CN117555874B - Log storage method, device, equipment and medium of distributed database - Google Patents
Log storage method, device, equipment and medium of distributed database Download PDFInfo
- Publication number
- CN117555874B CN117555874B CN202410041098.9A CN202410041098A CN117555874B CN 117555874 B CN117555874 B CN 117555874B CN 202410041098 A CN202410041098 A CN 202410041098A CN 117555874 B CN117555874 B CN 117555874B
- Authority
- CN
- China
- Prior art keywords
- data
- distributed
- log
- lock
- memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000000034 method Methods 0.000 title claims abstract description 41
- 238000003860 storage Methods 0.000 title claims abstract description 34
- 230000015654 memory Effects 0.000 claims abstract description 70
- 230000005540 biological transmission Effects 0.000 claims description 15
- 238000013136 deep learning model Methods 0.000 claims description 12
- 238000004590 computer program Methods 0.000 claims description 11
- 238000010801 machine learning Methods 0.000 claims description 11
- 230000000903 blocking effect Effects 0.000 claims description 5
- 238000012545 processing Methods 0.000 abstract description 14
- 238000007726 management method Methods 0.000 description 15
- 230000008569 process Effects 0.000 description 6
- 230000006870 function Effects 0.000 description 4
- 238000010586 diagram Methods 0.000 description 3
- 230000014759 maintenance of location Effects 0.000 description 3
- 238000012795 verification Methods 0.000 description 3
- 238000004891 communication Methods 0.000 description 2
- 230000008878 coupling Effects 0.000 description 2
- 238000010168 coupling process Methods 0.000 description 2
- 238000005859 coupling reaction Methods 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 230000007246 mechanism Effects 0.000 description 2
- 230000003287 optical effect Effects 0.000 description 2
- 230000002159 abnormal effect Effects 0.000 description 1
- 230000006978 adaptation Effects 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 230000009286 beneficial effect Effects 0.000 description 1
- 230000008094 contradictory effect Effects 0.000 description 1
- 238000007405 data analysis Methods 0.000 description 1
- 238000011161 development Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 230000004927 fusion Effects 0.000 description 1
- 230000003993 interaction Effects 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000001360 synchronised effect Effects 0.000 description 1
- 230000026676 system process Effects 0.000 description 1
- 239000002699 waste material Substances 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/1805—Append-only file systems, e.g. using logs or journals to store data
- G06F16/1815—Journaling file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F18/00—Pattern recognition
- G06F18/20—Analysing
- G06F18/25—Fusion techniques
- G06F18/253—Fusion techniques of extracted features
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Life Sciences & Earth Sciences (AREA)
- Artificial Intelligence (AREA)
- Bioinformatics & Cheminformatics (AREA)
- Bioinformatics & Computational Biology (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Evolutionary Biology (AREA)
- Evolutionary Computation (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a log storage method, device, equipment and medium of a distributed database, and relates to the technical field of computers. The method comprises the steps of obtaining first data, and storing the first data into at least two distributed memories; acquiring second data from a first distributed memory in at least two distributed memories, fusing the second data with third data according to the second data to acquire fourth data, wherein the third data is log data generated in the step of acquiring the first data and storing the first data in at least two distributed memories according to a preset form; establishing a first multithreading connection handle with fourth data according to the fourth data, and setting a first distributed lock; setting a first timeout time for the first distributed lock according to the first distributed lock; after the first thread writes the fourth data into the log memory, the first distributed lock is unlocked. Multiple multithreads Cheng Rong can be combined into one multithread, so that the data processing speed is improved, and the complexity of processing data is reduced.
Description
Technical Field
The present disclosure relates to the field of computer technologies, and in particular, to a method, an apparatus, a device, and a medium for storing logs in a distributed database.
Background
With the rapid development of the information age, mobile internet, internet of things, cloud computing and artificial intelligence have been rapidly developed in recent years, and a large amount of data is generated accordingly. Such data on the one hand provides a lot of valuable information for each industry and on the other hand the large amount of data also presents a great technical challenge in terms of data storage and data analysis.
Currently, a log storage method of a distributed database is usually executed by multiple threads, and a distributed lock is set for each thread, so that in the process of storing the log, the data processing speed is slower due to more threads.
Disclosure of Invention
The purpose of the application is to provide a log storage method, a device, equipment and a medium of a distributed database, which can synthesize a plurality of multithread Cheng Rong into one multithread, thereby improving the data processing speed and reducing the complexity of processing data.
The application provides a log storage method of a distributed database, which comprises the following steps: acquiring first data, and storing the first data to at least two distributed memories in a preset form; obtaining second data from a first distributed memory of the at least two distributed memories, the second data being log data generated by the first distributed memory; according to the second data, fusing the second data with third data to obtain fourth data, wherein the third data is log data generated in the step of obtaining first data, and storing the first data into at least two distributed memories according to a preset form; establishing a first multithreaded connection handle with the fourth data according to the fourth data; blocking the corresponding first thread according to the first multithreaded connection handle, and providing a distributed lock for the first thread Cheng Jiadi; setting a first timeout time for the first distributed lock according to the first distributed lock; and after the first thread writes the fourth data into a log memory, unlocking the first distributed lock.
Optionally, after the step of acquiring the second data from the first distributed storage in the at least two distributed storages, the method further includes: and obtaining fifth data from a second distributed memory in the at least two distributed memories, wherein the fifth data is log data generated by the second distributed memory.
Optionally, the step of fusing the second data with the third data according to the second data to obtain fourth data further includes: and according to the fifth data, fusing the second data, the third data and the fifth data to obtain fourth data.
Optionally, the step of fusing the second data with the third data according to the second data to obtain fourth data includes: and according to a first identifier of second data and a second identifier of third data, fusing the third data corresponding to the second identifier with the second data corresponding to the first identifier to obtain fourth data, wherein the third identifier of the fourth data corresponds to the first identifier and the second identifier.
Optionally, the step of setting a first timeout time for the first distributed lock according to the first distributed lock includes: and setting a first timeout time for the first distributed lock according to the timeout time of the distributed locks of other threads and/or the data size of the fourth data.
Optionally, the step of setting the first timeout time for the first distributed lock according to the timeout time of the distributed lock of the other threads and/or the data size of the fourth data includes: according to the overtime time of the distributed locks of other threads, predicting the unlocking time of the distributed locks of other threads through a machine learning model or a deep learning model; calculating the transmission time of the fourth data according to the data size of the fourth data and the data transmission rate and the data capacity of the transmission bandwidth; and setting a first timeout time for the first distributed lock according to the unlocking time of the distributed locks of the other threads and the transmission time of the fourth data.
Optionally, after the first data is obtained, the method further includes performing a trust check on the first data.
The application also provides a log storage device of the distributed database, which comprises: the data receiving module is used for receiving the log data generated by the first distributed memory, the log data generated by the step of acquiring the first data and storing the first data to at least two distributed memories in a preset form and the data fusing the second data and the third data; the distributed lock module establishes a first multithreaded connection handle with the fourth data, blocks a corresponding first thread and is a distributed lock for the first thread Cheng Jiadi; a machine learning model or a deep learning model module that sets a timeout time for the first distributed lock; and the reading and writing module is used for writing the fourth data into a log memory.
The application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the method of any one of the above.
The present application also provides a computer readable storage medium, on which a computer program is stored, the processor executing the computer program to implement the method of any one of the above.
The beneficial effects of this application include:
the application provides a log storage method, device, equipment and medium of a distributed database, which are characterized in that log data generated by a first distributed memory are fused with log data generated by the step of acquiring first data, the first data are stored to at least two distributed memories according to a preset form, so as to acquire fourth data, a first multithread connection handle is established for the fourth data, a first distributed lock is used for a first thread Cheng Shezhi, a first timeout time of the first distributed lock is set, a first thread is executed, the fourth data are written into the log memories, the separated multithreading and storage of the log data generated by the first distributed memory are used for acquiring the first data, and the first data are stored to at least two distributed memories according to the preset form to generate the log data multithreading, so as to be fused into a first multithreading for storing the fourth data. By adopting the arrangement mode, a plurality of multithreads Cheng Rong are combined into one multithread, so that the data processing speed is improved, and the complexity of processing data is reduced.
Drawings
For a clearer description of the technical solutions of the present application, the drawings that are needed in the embodiments will be briefly described below, it being understood that the following drawings only illustrate some embodiments of the present application and should therefore not be considered limiting in scope, and that other related drawings can be obtained from these drawings without the inventive effort for a person skilled in the art.
FIG. 1 is a flow chart of a log storage method of a distributed database according to the present application;
FIG. 2 is a second flow chart of a log storage method of the distributed database of the present application;
FIG. 3 is a flow chart III of a log storage method of the distributed database of the present application;
FIG. 4 is a flow chart diagram of a log storage method of a distributed database according to the present application;
FIG. 5 is a schematic diagram of a log storage method of the distributed database of the present application;
fig. 6 is a schematic structural diagram of a log storage device of the distributed database of the present application.
The realization, functional characteristics and advantages of the present application will be further described with reference to the embodiments, referring to the attached drawings.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and fully with reference to the accompanying drawings, in which it is evident that the embodiments described are only some, but not all, of the embodiments of the present invention. All other embodiments, which can be made by one of ordinary skill in the art based on the embodiments herein without making any inventive effort, are intended to be within the scope of the present application.
It should be noted that all directional indicators (such as up, down, left, right, front, and rear … …) in the embodiments of the present invention are merely used to explain the relative positional relationship, movement, etc. between the components in a particular posture (as shown in the drawings), and if the particular posture is changed, the directional indicator is correspondingly changed.
In the present application, unless explicitly specified and limited otherwise, the terms "coupled," "secured," and the like are to be construed broadly, and for example, "secured" may be either permanently attached or removably attached, or integrally formed; can be mechanically or electrically connected; either directly or indirectly, through intermediaries, or both, may be in communication with each other or in interaction with each other, unless expressly defined otherwise. The specific meaning of the above terms in the present invention can be understood by those of ordinary skill in the art according to the specific circumstances.
In addition, if there is a description of "first", "second", etc. in the embodiments of the present application, the description of "first", "second", etc. is for descriptive purposes only and is not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In addition, the meaning of "and/or" as it appears throughout includes three parallel schemes, for example "A and/or B", including the A scheme, or the B scheme, or the scheme where A and B are satisfied simultaneously. In addition, the technical solutions of the embodiments may be combined with each other, but it is necessary to base that the technical solutions can be realized by those skilled in the art, and when the technical solutions are contradictory or cannot be realized, the combination of the technical solutions should be regarded as not exist and not within the protection scope of the present application.
Currently, a log storage method of a distributed database is usually executed by multiple threads, and a distributed lock is set for each thread, so that in the process of storing the log, the data processing speed is slower due to more threads.
Example 1
Referring to fig. 1 and 5, the present application provides a log storage method of a distributed database, including the following steps:
step S110, first data are acquired, and the first data are stored in at least two distributed memories according to a preset form.
In a specific implementation, the distributed management platform acquires first data and stores the first data in at least two distributed memories according to a preset form. It should be noted that, the number of the distributed memories can be set according to actual requirements, and when the data volume is larger, the more the distributed memories are set; the storing according to the preset form may be to split the first data, and store the split first data in a plurality of distributed memories respectively, or may be to copy the first data into a plurality of copies, and store the plurality of copies of the first data in a plurality of distributed memories respectively.
Step S120, obtaining second data from a first distributed memory of the at least two distributed memories.
In a specific implementation, the first data is stored in at least two distributed memories according to a preset form, each memory generates a stored log data, and the distributed management platform acquires the second data from the first distributed memory in the at least two distributed memories, where it is noted that the second data is the log data generated by the first distributed memory. The log data refers to data for recording events, states and abnormal information generated in the running process of a system, an application program or network equipment. Such information typically includes time stamps, event types, event descriptions, source addresses, destination addresses, protocols, and the like.
And step S130, fusing the second data with the third data according to the second data to obtain fourth data.
In a specific implementation, the distributed management platform fuses the second data and the third data to obtain a merged fourth data. It should be noted that, the third data is log data generated in the step of obtaining the first data by the distributed management platform and storing the first data to at least two distributed memories according to a preset form.
Step S140, according to the fourth data, a first multithreading connection handle is established with the fourth data.
In an implementation, the distributed management platform establishes a first multi-threaded link handle for the merged fourth data. It should be noted that, the multithreaded connection handle refers to a database connection handle used in a multithreaded environment, and in a multithreaded application, each thread needs an independent database connection in order to be able to access the database concurrently. Conventional single-threaded connection handles can only be used by one thread, and problems such as race conditions and deadlocks can occur when multiple threads access a database simultaneously. The multithreaded connection handle can be shared by a plurality of threads, and each thread can independently access the database through the handle, so that the problems of competition conditions, deadlock and the like are avoided. The handle is an address, and the program transmits data among different functions through the handle, so that the address can be used for transmitting values, and the memory can be saved. Handles can be categorized into context handles, connection handles, statement handles. Environment handle: is used to store the global context of the data. Such as: environmental status, status diagnostics, etc. The environment handle should be created before the connection handle is created. Connection handle: one context handle may establish multiple connection handles, namely 1: n. Each connection handle may be connected to a data source, and each connection handle may be connected to multiple statement handles. Thus, mutual access between the plurality of databases is achieved in this manner. Statement handle: the address of an SQL sentence is stored, and the result of the execution of the SQL sentence can be also included.
Step S150, according to the first multithreaded connection handle, blocking the corresponding first thread, and providing a distributed lock for the first thread Cheng Jiadi.
Where a lock is a mechanism by which a computer coordinates multiple processes or threads to concurrently access a resource. In addition to the contention of traditional computing resources (e.g., CPU, RAM, I/O, etc.), in databases, data is also a resource that is shared by many users. How to guarantee the consistency and effectiveness of the concurrent access of data is a problem that all databases must solve, and lock conflicts are also an important factor affecting the performance of the concurrent access of the databases. For a stand-alone system, this may be achieved by conventional locking methods such as synchronized or reintrantlock, however, for a distributed cluster system, the problem is not solved by a simple local lock, so a distributed lock is required, and a three-party component or service is usually introduced to solve the problem, such as Redis, zookeeper.
Specifically, when a first thread is in a blocking state, it temporarily releases CPU resources, allowing other threads the opportunity to execute, at which time a distributed lock needs to be set for the first thread to ensure that the first thread can be executed.
Step S160, setting a first timeout time for the first distributed lock according to the first distributed lock.
It should be noted that, the timeout time of the distributed lock refers to an operation of automatically discarding the lock acquisition if the lock is not acquired within a certain time during the lock acquisition process. The timeout is to avoid the situation that the lock is always occupied and cannot be released due to a fault of a certain node or other reasons, and when the distributed lock is realized, a proper timeout is usually required to be set to ensure that the lock resource can be released in time under normal conditions, and meanwhile, frequent lock competition and resource waste caused by too short timeout time are avoided. The timeout period is set taking into account a number of factors including the usage scenario of the lock, system load, network delay, etc. In general, the timeout should be adjusted according to practical situations, and different timeout may be required for different application scenarios.
Specifically, a first timeout time is set for the first distributed lock according to the data amount of the fourth data, the system load and the network delay to ensure that the lock resources can be released in time under normal conditions.
In addition, it should be noted that when using distributed locks, the timeout time cannot completely avoid deadlock, so other fault tolerance mechanisms and exception handling strategies need to be considered when designing a distributed system to ensure reliability and stability of the system
Step S170, after the first thread writes the fourth data into the log memory, unlocking the first distributed lock.
In a specific implementation, after the first thread is executed and the fourth data is written into the log memory, the first distributed lock is unlocked, an unlocking code can be put into the script, and the script is executed by using a redistemplate. To ensure the atomicity of unlocking, the example procedure:
$token = rand(1,100000);
function lock(){return Redis::set("my:lock",$token,"nx","ex",10);}
function unlock(){$script = `if redis.call("get",KEYS[1]) == ARGV[1]
Thenreturn redis.call("del",KEYS[1])else
return 0end
return Redis::eval($script, "my:lock", $token)
} if (lock()) {// do something
unlock();
}
the token is a random number, when the token is locked, the token in the token is stored in the token of rediss, when the token is unlocked, the token in the token is first fetched, if the token is consistent with the token to be deleted, the lock is indicated as the previous set, otherwise, the lock is indicated as the expired, and the lock is indicated as the additional set, and no operation should be performed on the lock.
Example 2
Referring to fig. 2 and 5, in this embodiment, after the step of acquiring the second data from the first distributed storage in the at least two distributed storages, the method further includes the following steps:
step S121, obtaining fifth data from a second distributed storage in the at least two distributed storages.
In a specific implementation, the first data is stored in at least two distributed memories according to a preset form, each memory generates a stored log data, and the distributed management platform obtains fifth data from a second distributed memory in the at least two distributed memories, where it is noted that the fifth data is the log data generated by the second distributed memory.
In this embodiment, the step of fusing the second data with the third data according to the second data to obtain fourth data further includes the following steps:
step S131 fuses the second data, the third data and the fifth data according to the fifth data, so as to obtain fourth data.
In a specific implementation, the distributed management platform fuses the second data with the third data and the fifth data to obtain a merged fourth data. The data fusion is used for reducing the complexity of data processing in the subsequent data processing process, and changing the subsequent processing of multiple data into the processing of only one data.
Referring to fig. 3 and 5, in the present embodiment, the step of fusing the second data with the third data according to the second data to obtain fourth data includes the following steps:
step S132, fusing third data and second data corresponding to the first identification of the second identification according to the first identification of the second data and the second identification of the third data.
It should be noted that, when data are fused, the identifier of each data needs to be corresponding to ensure the integrity and consistency of the fused data.
Specifically, the distributed management platform fuses the second data with the first identifier and the third data with the second identifier to obtain fourth data. The first identifier corresponds to the second identifier, and the obtained fourth data has a third identifier corresponding to the first identifier and the second identifier.
Example 3
Referring to fig. 1 and fig. 4, in this embodiment, the step of setting a first timeout period for the first distributed lock according to the first distributed lock includes:
s161, setting a first timeout time for the first distributed lock according to the timeout time of the distributed locks of other threads and/or the data size of the fourth data.
And setting a first timeout time for the first distributed lock according to the timeout time of the distributed locks of other threads and/or the data size of the fourth data. In a specific implementation, when the fourth data amount is larger, the system processes the fourth data for a relatively longer time, and thus, a first timeout period is set for the first distributed lock for a relatively longer time. Further, when the distributed locks of other threads do not reach the timeout, the other threads occupy the CPU resource, so that the system processing data is crowded, when the timeout is reached, the distributed locks are released, and the other threads are released, so that the CPU resource can be released to execute the first thread. The timeout of the first distributed lock of the first thread cannot be shorter than the timeout of the distributed locks of the other threads.
In this embodiment, the step of setting the first timeout period for the first distributed lock according to the timeout periods of the distributed locks of other threads and/or the data size of the fourth data includes: according to the overtime time of the distributed locks of other threads, predicting the unlocking time of the distributed locks of other threads through a machine learning model or a deep learning model, and calculating the transmission time of fourth data according to the data size of the fourth data, the data transmission rate and the data capacity of the transmission bandwidth; and setting a first timeout time for the first distributed lock according to the unlocking time of the distributed locks of the other threads and the transmission time of the fourth data.
Specifically, when a first timeout time is set for a first distributed lock of a first thread, a machine learning model or a deep learning model is introduced, wherein the machine learning model or the deep learning model can predict the timeout time and the time for releasing the distributed lock of other threads of the system according to the timeout time and the time for releasing the distributed lock of other threads in the past so as to obtain a first time; the machine learning model or the deep learning model is further capable of predicting a time when the fourth data storage is completed based on the data size of the fourth data, the data capacity of the transmission bandwidth, and the data transmission rate to obtain a second time. The machine learning model or the deep learning model compares time values in the first time and the second time, when the time value of the first time is large, the first time data is sent to the distributed management platform, and the distributed management platform sets the first time as a first overtime time of the first distributed lock; when the time value of the second time is large, the second time data is sent to the distributed management platform, and the distributed management platform sets the second time as the first overtime time of the first distributed lock so as to prevent the first thread from releasing the first distributed lock and further releasing the first thread under the condition that the first thread is not executed yet because the time exceeds the first few-time, thereby ensuring that the first thread can execute and finish.
For example, when one other thread is executing, the machine learning model or the deep learning model predicts that the release time of the distributed lock of the other thread is 5 seconds, and sets the first timeout time of the first distributed lock of the first thread to be 5.5 seconds or 6 seconds; when two other threads are executing, the machine learning model or the deep learning model predicts that the release time of the distributed locks of the two other threads is 10 seconds, and then sets the first timeout time of the first distributed lock of the first thread to be 10.5 seconds or 11 seconds.
Example 4
On the basis of the above embodiment, in this embodiment, after the distributed management platform obtains the first data, performing a trust check on the first data is further included.
Specifically, the distributed management platform receives the connection of the first data by listening to a TCP port or a Unix socket, and after a connection is established, the distributed management platform performs the following operations:
first, the first data socket is set to a non-blocking mode, and then a readable file event is created for listening to the data transmission of the first data socket.
And then, before the data of the first data socket is sent, verification information is sent first, after the database receives the information, the information is verified, after the verification is passed, log information of the first data is received and processed, the verification is not passed, and the first data is disconnected.
Example 5
Referring to fig. 6, the present application further provides a log storage device of a distributed database, including: the data receiving module is used for receiving the log data generated by the first distributed memory, the log data generated by the step of acquiring the first data and storing the first data to at least two distributed memories in a preset form, and the data fusing the second data and the third data. The distributed lock module establishes a first multithreaded connection handle with the fourth data, blocks a corresponding first thread and is a distributed lock for the first thread Cheng Jiadi; the deep learning model module is used for setting timeout time for the first distributed lock; and the reading and writing module is used for writing the fourth data into a log memory.
Example 6
The application also provides a computer device comprising a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to realize the method.
Example 7
Further, the present embodiment provides a computer readable storage medium, on which a computer program is stored, and a processor executes the computer program to implement any one of the methods described above.
In some embodiments, the computer readable storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; but may be a variety of devices including one or any combination of the above memories. The computer may be a variety of computing devices including smart terminals and servers.
In the foregoing embodiments of the present disclosure, the descriptions of the various embodiments are emphasized, and for a portion of this disclosure that is not described in detail in this embodiment, reference is made to the related descriptions of other embodiments.
In the several embodiments provided in the present application, it should be understood that the disclosed technology content may be implemented in other manners. The above-described embodiments of the apparatus are merely exemplary, and the division of units may be a logic function division, and there may be another division manner in actual implementation, for example, multiple units or components may be combined or integrated into another system, or some features may be omitted, or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be through some interfaces, units or modules, or may be in electrical or other forms.
The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed over a plurality of units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
In addition, each functional unit in each embodiment of the present disclosure may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.
The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable non-volatile storage medium. Based on such understanding, the technical solution of the present disclosure may be embodied in essence or a part contributing to the prior art or all or part of the technical solution in the form of a software product stored in a non-volatile storage medium, including several instructions to cause a computer device (which may be a personal computer, a server or a network device, etc.) to perform all or part of the steps of the methods of the various embodiments of the present disclosure. And the aforementioned nonvolatile storage medium includes: a U-disk, a Read-Only Memory (ROM), a random access Memory (RAM, random Access Memory), a removable hard disk, a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely a preferred embodiment of the present disclosure and it should be noted that modifications and adaptations to those skilled in the art may be made without departing from the principles of the present disclosure and are intended to be comprehended within the scope of the present disclosure.
Claims (10)
1. A log storage method of a distributed database, comprising the steps of:
acquiring first data, and storing the first data to at least two distributed memories in a preset form;
obtaining second data from a first distributed memory of the at least two distributed memories, the second data being log data generated by the first distributed memory;
according to the second data, fusing the second data with third data to obtain fourth data, wherein the third data is log data generated in the step of obtaining first data, and storing the first data into at least two distributed memories according to a preset form;
establishing a first multithreaded connection handle with the fourth data according to the fourth data;
blocking the corresponding first thread according to the first multithreaded connection handle, and providing a distributed lock for the first thread Cheng Jiadi;
setting a first timeout time for the first distributed lock according to the first distributed lock;
and after the first thread writes the fourth data into a log memory, unlocking the first distributed lock.
2. The method of claim 1, further comprising, after the step of retrieving the second data from a first one of the at least two distributed memories:
and obtaining fifth data from a second distributed memory in the at least two distributed memories, wherein the fifth data is log data generated by the second distributed memory.
3. The method of claim 2, wherein the step of fusing the second data with third data to obtain fourth data according to the second data, further comprises:
and according to the fifth data, fusing the second data, the third data and the fifth data to obtain fourth data.
4. The method of claim 1, wherein the step of fusing the second data with third data to obtain fourth data according to the second data comprises:
and according to a first identifier of second data and a second identifier of third data, fusing the third data corresponding to the second identifier with the second data corresponding to the first identifier to obtain fourth data, wherein the third identifier of the fourth data corresponds to the first identifier and the second identifier.
5. The method of claim 1, wherein the step of setting a first timeout period for the first distributed lock according to the first distributed lock comprises:
and setting a first timeout time for the first distributed lock according to the timeout time of the distributed locks of other threads and/or the data size of the fourth data.
6. The method according to claim 5, wherein the step of setting the first timeout time for the first distributed lock according to the timeout time of the distributed lock of the other threads and/or the data size of the fourth data comprises:
according to the overtime time of the distributed locks of other threads, predicting the unlocking time of the distributed locks of other threads through a machine learning model or a deep learning model;
calculating the transmission time of the fourth data according to the data size of the fourth data and the data transmission rate and the data capacity of the transmission bandwidth;
and setting a first timeout time for the first distributed lock according to the unlocking time of the distributed locks of the other threads and the transmission time of the fourth data.
7. The method of claim 1, further comprising performing a trust check on the first data after the first data is obtained.
8. A log storage device for a distributed database, comprising:
the data receiving module is used for receiving the log data generated by the first distributed memory, acquiring the first data, storing the first data into at least two log data generated by the step of the distributed memory according to a preset form and fusing the second data with the third data;
the distributed lock module establishes a first multithreaded connection handle with fourth data, blocks a corresponding first thread and is a distributed lock for the first thread Cheng Jiadi;
a machine learning model or a deep learning model module that sets a timeout time for the first distributed lock;
and the reading and writing module is used for writing the fourth data into a log memory.
9. A computer device comprising a memory and a processor, the memory having stored therein a computer program, the processor executing the computer program to implement the method according to any of claims 1-7.
10. A computer readable storage medium, characterized in that the computer readable storage medium has stored thereon a computer program, which is executed by a processor to implement the method according to any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410041098.9A CN117555874B (en) | 2024-01-11 | 2024-01-11 | Log storage method, device, equipment and medium of distributed database |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202410041098.9A CN117555874B (en) | 2024-01-11 | 2024-01-11 | Log storage method, device, equipment and medium of distributed database |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN117555874A CN117555874A (en) | 2024-02-13 |
| CN117555874B true CN117555874B (en) | 2024-03-29 |
Family
ID=89823662
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202410041098.9A Active CN117555874B (en) | 2024-01-11 | 2024-01-11 | Log storage method, device, equipment and medium of distributed database |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117555874B (en) |
Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106708968A (en) * | 2016-12-01 | 2017-05-24 | 成都华为技术有限公司 | Distributed database system and data processing method in distributed database system |
| CN109446173A (en) * | 2018-09-18 | 2019-03-08 | 平安科技（深圳）有限公司 | Daily record data processing method, device, computer equipment and storage medium |
| CN112000698A (en) * | 2020-08-25 | 2020-11-27 | 青岛海尔科技有限公司 | Log recording method and device, storage medium and electronic device |
| CN112866319A (en) * | 2019-11-28 | 2021-05-28 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
| CN114840488A (en) * | 2022-07-04 | 2022-08-02 | 柏科数据技术（深圳）股份有限公司 | Distributed storage method, system and storage medium based on super-fusion structure |
| CN115756888A (en) * | 2022-11-15 | 2023-03-07 | 中科云谷科技有限公司 | Data processing method, processor, device and storage medium |
| CN116186082A (en) * | 2022-12-29 | 2023-05-30 | 中国邮政储蓄银行股份有限公司 | Data summarizing method based on distribution, first server and electronic equipment |
| CN116561091A (en) * | 2023-04-28 | 2023-08-08 | 济南浪潮数据技术有限公司 | Log storage method, device, equipment and readable storage medium |
| WO2023159976A1 (en) * | 2022-02-28 | 2023-08-31 | 华为技术有限公司 | Data segmented writing method, data reading method and apparatus |
| CN117270794A (en) * | 2023-11-22 | 2023-12-22 | 成都大成均图科技有限公司 | Redis-based data storage method, medium and device |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| GB201816837D0 (en) * | 2018-10-16 | 2018-11-28 | Microsoft Technology Licensing Llc | Database management |
-
2024
- 2024-01-11 CN CN202410041098.9A patent/CN117555874B/en active Active
Patent Citations (10)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN106708968A (en) * | 2016-12-01 | 2017-05-24 | 成都华为技术有限公司 | Distributed database system and data processing method in distributed database system |
| CN109446173A (en) * | 2018-09-18 | 2019-03-08 | 平安科技（深圳）有限公司 | Daily record data processing method, device, computer equipment and storage medium |
| CN112866319A (en) * | 2019-11-28 | 2021-05-28 | 顺丰科技有限公司 | Log data processing method, system and storage medium |
| CN112000698A (en) * | 2020-08-25 | 2020-11-27 | 青岛海尔科技有限公司 | Log recording method and device, storage medium and electronic device |
| WO2023159976A1 (en) * | 2022-02-28 | 2023-08-31 | 华为技术有限公司 | Data segmented writing method, data reading method and apparatus |
| CN114840488A (en) * | 2022-07-04 | 2022-08-02 | 柏科数据技术（深圳）股份有限公司 | Distributed storage method, system and storage medium based on super-fusion structure |
| CN115756888A (en) * | 2022-11-15 | 2023-03-07 | 中科云谷科技有限公司 | Data processing method, processor, device and storage medium |
| CN116186082A (en) * | 2022-12-29 | 2023-05-30 | 中国邮政储蓄银行股份有限公司 | Data summarizing method based on distribution, first server and electronic equipment |
| CN116561091A (en) * | 2023-04-28 | 2023-08-08 | 济南浪潮数据技术有限公司 | Log storage method, device, equipment and readable storage medium |
| CN117270794A (en) * | 2023-11-22 | 2023-12-22 | 成都大成均图科技有限公司 | Redis-based data storage method, medium and device |
Non-Patent Citations (2)
| Title |
|---|
| 一种使用Node.js构建的分布式数据流日志服务系统;张煜;《计算机系统应用》;20130215(第2期);70-73 * |
| 一种日志融合分析工具设计;张玉兵;《现代工业经济和信息化》;20171211;第7卷(第20期);29-30 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN117555874A (en) | 2024-02-13 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| Chen et al. | Fast and general distributed transactions using RDMA and HTM | |
| US8776063B2 (en) | Method and system for hardware feedback in transactional memory | |
| Detlefs et al. | Inheritance of synchronization and recovery properties in Avalon/C++ | |
| US20180173745A1 (en) | Systems and methods to achieve sequential consistency in replicated states without compromising performance in geo-distributed, replicated services | |
| US20200073758A1 (en) | Checkpointing for increasing efficiency of a blockchain | |
| EP1696611B1 (en) | Server data queuing system and method | |
| US8495266B2 (en) | Distributed lock | |
| US7536582B1 (en) | Fault-tolerant match-and-set locking mechanism for multiprocessor systems | |
| US20170220619A1 (en) | Concurrency Control Method and Apparatus | |
| JP6397053B2 (en) | System and method for providing distributed transaction locks in a transaction middleware machine environment | |
| CN110188110B (en) | Method and device for constructing distributed lock | |
| Dhoked et al. | An adaptive approach to recoverable mutual exclusion | |
| US8370317B2 (en) | Synchronizing shared resources in a collection | |
| CN116489179A (en) | Service processing system and service processing method | |
| US20070078852A1 (en) | Synchronizing shared resources in a collection | |
| CN109542922B (en) | Processing method for real-time service data and related system | |
| CN117555874B (en) | Log storage method, device, equipment and medium of distributed database | |
| CN112015563B (en) | Message queue switching method and device, electronic equipment and storage medium | |
| CN111930503A (en) | Resource lock acquisition method based on ETCD | |
| US7539678B2 (en) | Systems and methods for controlling access to an object | |
| CN114461409A (en) | Auxiliary analysis method, system and storage medium for abnormal blocking of mutual exclusion semaphore | |
| Bravo et al. | Reconfigurable atomic transaction commit | |
| WO2008039618A1 (en) | Persistent locks/resources for concurrency control | |
| Spector | Communication support in operating systems for distributed transactions | |
| CN112905361A (en) | Main task exception handling method and device, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |