CN112416863A - Data storage method and cache server - Google Patents
Data storage method and cache server Download PDFInfo
- Publication number
- CN112416863A CN112416863A CN202011120513.8A CN202011120513A CN112416863A CN 112416863 A CN112416863 A CN 112416863A CN 202011120513 A CN202011120513 A CN 202011120513A CN 112416863 A CN112416863 A CN 112416863A
- Authority
- CN
- China
- Prior art keywords
- file
- data
- kernel
- mode module
- communication connection
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/13—File access structures, e.g. distributed indices
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
- G06F16/162—Delete operations
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/172—Caching, prefetching or hoarding of files
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/17—Details of further file system functions
- G06F16/178—Techniques for file synchronisation in file systems
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/06—Protocols specially adapted for file transfer, e.g. file transfer protocol [FTP]
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Information Transfer Between Computers (AREA)
Abstract
The embodiment of the invention relates to the field of data processing, and discloses a data storage method and a cache server. The data storage method is applied to a cache server comprising a user mode program and a kernel mode module, and comprises the following steps: after establishing communication connection with a source server, the kernel mode module creates a file corresponding to the communication connection and sends path information of the file to a user mode program; when receiving data sent by a source server through communication connection, the kernel mode module copies the data into a file; and the user mode program controls the kernel mode module to store the data in the file into the storage device of the cache server based on the path information. In the application, after the data is copied from the kernel mode module to the user mode program, the user mode program analyzes the data and controls the kernel mode module to store the data into the storage device, so that processor resources and memory resources of the cache server are saved, and the speed of receiving the data is increased.
Description
Technical Field
The embodiment of the invention relates to the field of data processing, in particular to a data storage method and a cache server.
Background
With the advent of the multimedia age, the amount of data in networks has increased dramatically, requiring more and more files to be transmitted and stored in content distribution networks. In a content distribution network, a cache server undertakes multiple tasks of data receiving, processing and storing, and in order to cope with explosive increase of data volume, reducing time from when the cache server receives data to when the data is finally stored in a storage device, and reducing processor resource consumption and memory resource consumption of the cache server in the whole process have become important targets of performance optimization of the cache server.
At present, after receiving data to be stored in a storage device, a kernel state of a cache server needs to copy the data to a user state of the cache server, so that the user state can analyze the data, and the kernel state is controlled to store the data in the storage device by calling a file system write function in the kernel state, thereby realizing persistence of the data. However, the data is copied from the kernel mode to the user mode and then copied from the kernel mode to the user mode, so that the processor resource of the cache server is consumed, meanwhile, the transmission time of the data is increased, and the speed of completing the receiving of the data is reduced; meanwhile, the same data exists in the kernel mode and the user mode at the same time, and the memory resource of the cache server is occupied.
Disclosure of Invention
The embodiment of the invention aims to provide a data storage method and a cache server, which save processor resources of the cache server, reduce data transmission time and save memory resources of the cache server.
In order to solve the above technical problem, an embodiment of the present invention provides a data storage method, which is applied to a cache server including a user mode program and a kernel mode module, and includes the following steps: after the communication connection is established between the cache server and the source server, the kernel mode module creates a file corresponding to the communication connection and sends path information of the file to the user mode program; when receiving data sent by a source server through communication connection, the kernel mode module copies the data into a file; and the user mode program controls the kernel mode module to store the data in the file into the storage device of the cache server based on the path information.
An embodiment of the present invention further provides a cache server, including: a user mode program and a kernel mode module; the kernel mode module is used for establishing a file corresponding to the communication connection after the communication connection is established between the cache server and the source server, and sending the path information of the file to the user mode program; the kernel mode module is also used for copying data into a file when receiving the data sent by the source server through the communication connection; and the user mode program is used for controlling the kernel mode module to store the data in the file into the storage equipment of the cache server based on the path information.
An embodiment of the present invention further provides a cache server, including: at least one processor; and a memory communicatively coupled to the at least one processor; the processor comprises a user mode program and a kernel mode module; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data storage methods of the above embodiments.
Compared with the prior art, after the cache server establishes communication connection with the source server, the kernel mode module can create a file corresponding to the communication connection in the file system and send path information of the file to the user mode program; when receiving data sent by a source server through communication connection, the kernel mode module copies the data into a file, so that a user mode program can control the kernel mode module to store the data in the file into storage equipment of a cache server according to path information of the received file; the user mode program controls the kernel mode module to store the data in the file into the storage device of the cache server through the path information of the file sent by the kernel mode module, the received data only needs to be cached in the memory of the kernel mode module, and only the path information of the file needs to be interacted between the user mode program and the kernel mode module, so that the copy process of the data between the kernel mode module and the user mode program is reduced, and the processor resource and the memory resource of the cache server are saved; meanwhile, the process that the cache server receives the data and finally stores the data into the storage device for persistence is simplified, and the speed of receiving the data is increased to a certain extent.
In addition, the file includes at least one kernel page, and the kernel mode module copies data into the file, including: the kernel mode module copies the data into a kernel page of the file and sets the kernel page in which the data are cached to be dirty; the user mode program controls the kernel mode module to store the data in the file into the storage device of the cache server based on the path information, and the method comprises the following steps: and the user mode program controls the kernel mode module to store the data in the dirty kernel page into the storage device based on the path information. In this embodiment, a specific implementation manner is provided for the kernel mode module to copy the data into the file, when the data is cached in the kernel page of the file, the kernel page is dirty, and subsequently, only the data in the dirty kernel page may be stored in the storage device, so that the integrity of the data stored in the storage device is ensured.
In addition, the user mode program controls the kernel mode module to store the data in the dirty kernel page into the storage device based on the path information, and the method comprises the following steps: when the user mode program judges that the preset condition is met, controlling the kernel mode module to store the data in the dirty kernel page into the storage device based on the path information; the preset condition is any one of the following conditions: the available memory of the cache server is smaller than a preset threshold, the time for storing the dirty kernel page in the memory of the kernel mode module is larger than the preset time, and a storing instruction sent by the user side is received. The embodiment provides a specific scenario that the user-mode program controls the kernel-mode module to store data in the dirty kernel page into the storage device.
In addition, before the kernel-state module creates a file corresponding to the communication connection, the method further includes: before establishing communication connection with a source server, a user mode program sends a resource acquisition request to a kernel mode module; wherein, the resource acquisition request comprises path information; the kernel mode module creates a file corresponding to the communication connection, including: and the kernel mode module creates a file corresponding to the communication connection according to the path information in the resource acquisition request. The embodiment provides a method for acquiring path information of a file.
In addition, the kernel mode module creates a file corresponding to the communication connection, including: the kernel mode module generates path information of the file corresponding to the communication connection and creates the file corresponding to the communication connection according to the generated path information. The present embodiment provides another way of acquiring path information of a file.
In addition, the sending the path information of the file to the user mode program comprises the following steps: when the kernel mode module successfully creates the file corresponding to the communication connection, the kernel mode module sends the path information of the file to the user mode program; when the kernel mode module fails to create a file corresponding to the communication connection, first notification information for representing the file creation failure is sent to a user mode program; the data storage method further comprises the following steps: and when the user mode program receives the first notification information, controlling the kernel mode module to close the communication connection.
In addition, after the user mode program controls the kernel mode module to store the data in the file into the storage device of the cache server based on the path information, the method further includes: and when judging that the communication connection is abnormally closed, the kernel mode module deletes the data in the file stored in the storage device. In the embodiment, the data of the communication connection which is closed abnormally is prevented from being stored in the storage device of the cache server, so that the occupation of the storage resource of the cache server is reduced.
In addition, the data storage method further comprises: when the kernel state module is abnormal in the process of receiving data sent by a source server or storing data in a file into a storage device, synchronizing abnormal state information including error information to a file system; and when the user mode program reads the abnormal state information in the file system, judging whether to retry according to the error information in the abnormal state information, and controlling the kernel mode module to retry when judging to retry.
In addition, the sending the path information of the file to the user mode program comprises the following steps: the kernel mode module sends the control handle corresponding to the file to the user mode program; the control handle comprises path information and an operation handle; the user mode program controls the kernel mode module to store the data in the file into the storage device of the cache server based on the path information, and the method comprises the following steps: and the user mode program utilizes the operation handle to control the kernel mode module to store the data in the file into the storage device of the cache server based on the path information in the control handle.
In addition, the kernel-mode module includes: a persistence system and a file system; the persistence system is used for establishing a file corresponding to the communication connection in the file system after the communication connection is established between the cache server and the source server, and sending the path information of the file to the user mode program; the persistence system is configured to copy data to a file upon receiving data sent by the origin server over the communication connection.
Drawings
One or more embodiments are illustrated by way of example in the accompanying drawings, which correspond to the figures in which like reference numerals refer to similar elements and which are not to scale unless otherwise specified.
Fig. 1 is a schematic diagram of a cache server to which a data storage method according to a first embodiment of the present invention is applied;
FIG. 2 is a detailed flowchart of a data storage method according to a first embodiment of the present invention;
FIG. 3 is a detailed flowchart of a data storage method according to a second embodiment of the present invention;
FIG. 4 is a detailed flowchart of a data storage method according to a third embodiment of the present invention;
fig. 5 is a detailed flowchart of a data storage method according to a fourth embodiment of the present invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the embodiments of the present invention more apparent, embodiments of the present invention will be described in detail below with reference to the accompanying drawings. However, it will be appreciated by those of ordinary skill in the art that numerous technical details are set forth in order to provide a better understanding of the present application in various embodiments of the present invention. However, the technical solution claimed in the present application can be implemented without these technical details and various changes and modifications based on the following embodiments. The following embodiments are divided for convenience of description, and should not constitute any limitation to the specific implementation manner of the present invention, and the embodiments may be mutually incorporated and referred to without contradiction.
A first embodiment of the present invention relates to a data storage method applied to a cache server in a content distribution network, the cache server being configured to store data received from an origin server into a storage device of the cache server; the storage device may be a disk, a hard disk, etc. in the cache server. Referring to fig. 1, at least one application program generally runs in a user mode, which may be referred to as a user mode program 11, the cache server 1 further includes a kernel mode module 12 running in a kernel mode, an address space corresponding to the kernel mode module 12 stores running codes of a kernel, the kernel module, and data maintained by the kernel, the kernel mode module 12 may include a persistence system 121 and a file system 122, in an example, the kernel mode module 12 further includes a network protocol stack 123, and the kernel mode module 12 may establish a communication connection with the source server 2 through the network protocol stack 123. The privilege level of the user mode program 11 is the lowest, the privilege level of the kernel mode module 12 is the highest, and when the user mode program 11 needs to execute work exceeding its own privilege level, for example, the parent and child processes copy information, copy setting and directory table entry, etc., it will call the function interface in the kernel mode module 12 to implement.
The specific flow of the data storage method of the present embodiment is shown in fig. 2.
Specifically, when the cache server receives a data request sent by a user through a target application program in the terminal, or determines that a preset data update period of the target application program is reached, a user mode program corresponding to the target application program in the cache server sends a resource acquisition request representing data to be requested to a network protocol stack in the kernel mode module, after receiving the resource acquisition request, the network protocol stack initiates a request for establishing a communication connection to an origin server of the target application program, where the origin server may be an origin server or a transit node server (e.g., a parent node server of the cache server) storing the data. Taking the communication connection as the TCP connection as an example, when the cache server establishes the communication connection with the source server, the network protocol stack in the kernel mode module and the source server need to perform three-way handshake, and after the three-way handshake, the TCP connection is successfully established, that is, the cache server is successfully established in the communication connection with the source server. After the cache server establishes communication connection with the source server, the kernel mode module can initiate a resource request to the source server through the network protocol stack, the source server sends a resource request response to the kernel mode module when receiving the resource request, the network protocol stack in the kernel mode module informs a persistence system in the kernel mode module when receiving the resource request response, and the persistence system can create a file corresponding to the communication connection in the file system and send path information of the file to the user mode program.
In this embodiment, sending the path information of the file to the user mode program includes: the kernel mode module sends the control handle corresponding to the file to the user mode program; wherein the control handle contains path information and an operation handle. Specifically, the persistence system of the kernel mode module creates a file corresponding to the communication connection in the file system, generates a control handle, and sends the control handle to the user mode program; the control handle includes connection information of the communication connection and path information of a file corresponding to the communication connection, where the connection information of the communication connection includes an IP address of the TCP connection and information of a quadruple of TCP ports, for example, the TCP connection. The connection information of the communication connection also comprises an operation handle of a file corresponding to the communication connection, so that the user mode program can locate the communication connection according to the connection information of the communication connection in the control handle and write and manage the file of the communication connection in the file system according to the operation handle of the file corresponding to the communication connection contained in the control handle.
Specifically, after sending a resource request response to the source server, the cache server sends data corresponding to the resource request to the kernel-mode module through the established communication connection, and when receiving the data, the kernel-mode module copies the data to a file corresponding to the communication connection. When the source server sends data to the cache server through the communication connection, the source server performs operations such as subpackaging and packaging on original data, sends a plurality of obtained data packets to a network protocol stack of the kernel mode module, the kernel mode module assembles the data packets in the network protocol stack to obtain the original data and caches the original data, the persistence system adds the data corresponding to the cached communication connection and the path information of the file corresponding to the communication connection into the to-be-read linked list, wakes up a working thread corresponding to the file system, the working thread reads the data corresponding to the communication connection from the to-be-read linked list, and calls a writing interface of the file system according to the read path information of the file corresponding to the communication connection to write the data corresponding to the communication connection into the file corresponding to the communication connection.
And 103, the user mode program controls the kernel mode module to store the data in the file into a storage device of the cache server based on the path information.
Specifically, the file system of the kernel mode module provides a file writing interface, and the user mode program can utilize the kernel mode module to send the control handle of the user mode program including the operation handle corresponding to the file writing interface according to the path information of the file corresponding to the communication connection included in the control handle, and call the file writing interface of the file system to control the kernel mode module to store the data in the file to the storage device of the cache server, thereby implementing persistence of the data in the file.
In this embodiment, after the cache server establishes a communication connection with the source server, the kernel mode module may create a file corresponding to the communication connection in the file system, and send path information of the file to the user mode program; when receiving data sent by a source server through communication connection, the kernel mode module copies the data into a file, so that a user mode program can control the kernel mode module to store the data in the file into storage equipment of a cache server according to path information of the received file; the user mode program controls the kernel mode module to store the data in the file into the storage device of the cache server through the path information of the file sent by the kernel mode module, the received data only needs to be cached in the memory of the kernel mode module, and only the path information of the file needs to be interacted between the user mode program and the kernel mode module, so that the copy process of the data between the kernel mode module and the user mode program is reduced, and the processor resource and the memory resource of the cache server are saved; meanwhile, the process that the cache server receives the data and finally stores the data into the storage device for persistence is simplified, and the speed of receiving the data is increased to a certain extent.
A second embodiment of the present invention relates to a data storage method. The second embodiment is substantially the same as the first embodiment, and the main improvements are: a specific implementation manner of copying data into a file by a kernel mode module is provided.
The specific flow of the data storage method of the present embodiment is shown in fig. 3.
Specifically, a file created in the file system by the kernel mode module includes at least one kernel page, the running memory space occupied by each kernel page is the same, and when data sent by the source server through the communication connection is received, the persistence system of the kernel mode module copies the received data to the kernel page of the file, and sets the kernel page in which the data is cached to be dirty, that is, sets the dirty flag of the kernel page in which the data is stored to be set. When the storage of the current kernel page is full, the next kernel page is used for caching the rest data until all the data are cached in the kernel page, and then all the kernel pages copied with the data are set to be dirty. For example, the storage space of a single kernel page is 4KB, if the size of the received data is 9KB, three kernel pages are required to cache the data, the persistence system caches the data into the three kernel pages in sequence, and after the data caching is finished, the three kernel pages are dirty.
In step 203, the user mode program controls the kernel mode module to store the data in the dirty kernel page into the storage device based on the path information.
Specifically, the user mode program may call a write function of a file system in the kernel mode module according to the path information of the file corresponding to the communication connection, and control the file persistence system to store data in the kernel page that is dirty in the file corresponding to the communication connection in the storage device of the cache server. When all the data in the kernel page are stored in the storage device, the persistence system can restore the dirty mark of the kernel page to a non-dirty state, so that the storage resource consumption of the cache server caused by the fact that the data in the same kernel page are repeatedly written into the storage device is avoided.
In one example, when the user mode program determines that the preset condition is met, the kernel mode program controls the kernel mode module to store the data in the dirty kernel page into the storage device based on the path information. The preset conditions comprise any one or any combination of the following conditions: the available memory of the cache server is smaller than a preset threshold, the time for storing the dirty kernel page in the memory of the kernel mode module is larger than the preset time, and a storing instruction sent by the user side is received. Specifically, when the preset condition is that the available memory of the cache server is smaller than a preset threshold, the user mode program obtains the memory use condition of the cache server in real time, and when the current available running memory of the cache server is smaller than a preset first memory threshold, the user mode program controls the kernel mode module to store data in all dirty kernel pages in the file into the storage device, so that the running memory space of the cache server is released.
When the preset condition is that the time for storing the dirty kernel page in the internal memory of the kernel mode module is longer than the preset time, the user mode program monitors the time for storing the dirty kernel page in the file in the running internal memory of the kernel mode module in real time, and when the time for storing the dirty kernel page in the internal memory of the kernel mode module is longer than the preset time, the user mode program controls the kernel mode module to store the data in all the dirty kernel pages in the file in the storage device, so that the condition that the dirty kernel page is cached in the running internal memory for a long time and is not written into a disk is avoided.
When the preset condition is that a storage instruction sent by a user side is received, a user can trigger the operation of storing the dirty kernel pages into the storage device through the user side, and when a user mode program receives the storage instruction sent by the user side, the user mode program controls the kernel mode module to store data in all the dirty kernel pages in the file into the storage device.
In one example, after step 203, the method further includes:
and step 204, deleting the data in the file stored in the storage device by the kernel mode module when the kernel mode module judges that the communication connection is abnormally closed.
Specifically, the persistence system of the kernel-mode module can count the progress of received data in real time, determine that the communication connection is normally closed when a finish message sent by the source server after data transmission is completed is received, write all the received data sent by the source server into the created file, close the created file after all the data in the file are written into the storage device, respond to the finish message to the source server, and disconnect the communication connection. When the persistence system does not receive the finish message sent by the source server and the communication connection is closed, whether the size of the received data is equal to the size of the requested data is judged, if the size of the received data is smaller than the size of the requested data, the communication connection is judged to be closed abnormally, the data of the communication connection stored in the storage device is deleted, the data of the communication connection which is closed abnormally is prevented from being stored in the storage device of the cache server, and the occupation of storage resources of the cache server is reduced.
Compared with the first embodiment, the embodiment provides a specific implementation manner that the kernel mode module copies data into a file, when the data is cached into the kernel page of the file, the kernel page is set to be dirty, and subsequently, only the data in the dirty kernel page can be stored into the storage device, so that the integrity of the data stored into the storage device is ensured.
A third embodiment of the present invention relates to a data storage method. The third embodiment is substantially the same as the first embodiment, and mainly differs therefrom in that: and when the file is failed to be created, the user-mode program controls the kernel-mode module to close the communication connection.
The specific flow of the data storage method of the present embodiment is shown in fig. 4.
Step 301 comprises the following substeps:
and a substep 3011, after the communication connection between the cache server and the source server is established, the kernel mode module creates a file corresponding to the communication connection.
Specifically, after the cache server establishes a communication connection with the source server, the persistence system may create a file corresponding to the communication connection in the file system, and specific ways of creating the file include the following two ways: firstly, a user mode program adds path information in a resource acquisition request for representing data to be requested, which is sent to a network protocol stack in a kernel mode module, and a persistence system of the kernel mode module can create a file corresponding to communication connection in a file system according to the path information in the resource acquisition request; secondly, after the cache server establishes communication connection with the source server, the persistence system in the kernel mode module generates path information of a file corresponding to the communication connection, and creates the file corresponding to the communication connection in the file system according to the generated path information.
Sub-step 3012, determine whether creating the file corresponding to the communication connection is successful. If yes, go to substep 3013; if not, then go to sub-step 3014.
In sub-step 3013, the kernel mode module sends the path information of the file to the user mode program.
In sub-step 3014, the kernel mode module sends a first notification message indicating that the file creation failed to the user mode program.
Specifically, when the persistence system creates a file corresponding to the communication connection in the file system, if the creation of the file is successful, the persistence system sends the path information of the file to the user mode program. If the file is failed to be created due to the problems of full disk space or medium errors and the like, first notification information representing the file creation failure is sent to the user mode program.
And step 304, when the user mode program receives the first notification information, controlling the kernel mode module to close the communication connection.
Specifically, when receiving first notification information sent by the persistence system of the kernel mode module, the user mode program calls an interrupt file writing interface provided by the kernel mode module, and controls the file persistence system to stop receiving data and disconnect communication connection.
In one example, the first notification information sent by the persistence system of the kernel mode module further includes an error type of file creation failure, and the user mode program performs corresponding operation according to the error type of file creation failure when receiving the first notification information. For example, if the error type of the file creation failure is a temporary problem such as the memory space of the cache server is full, the user mode program may call a file pause write-in interface provided by the kernel mode module, control the kernel mode module to pause receiving data through the communication connection, and retry to create a file corresponding to the communication connection, if the number of times of the file creation failure is greater than the preset number of times, the user mode program calls an interrupt file write-in interface provided by the kernel mode module, and controls the file persistence system to stop receiving the data and disconnect the communication connection; if the error type of the file creation failure is the unrecoverable problem such as the storage device medium error, the user mode program directly calls an interrupt file writing interface provided by the kernel mode module to control the file persistence system to stop receiving the data and disconnect the communication connection. And after the communication connection is closed, the kernel mode module sends a result of the communication connection closure to the user mode program.
Compared with the first embodiment, when the file is failed to be created, the user mode program can control the kernel mode module to close the communication connection, so that the communication connection is prevented from being kept in the kernel mode module all the time, and unnecessary communication is reduced.
A fourth embodiment of the present invention relates to a data storage method, and is substantially the same as the first embodiment, and is mainly different in that: the user mode program can judge whether to retry or not according to the error information of the abnormal state when the abnormal state occurs.
A specific flow of the data storage method of the present embodiment is shown in fig. 5.
in step 404, when an abnormality occurs in the process of receiving data sent by the source server or storing data in a file into the storage device, the kernel state module synchronizes abnormal state information including error information to the file system.
Specifically, the persistence system of the kernel mode module updates the size of the currently received data to the file system in real time in the process of receiving data sent by the source server through the communication connection, and synchronizes abnormal state information including an abnormal error type to the file system when data reception is abnormal. In addition, the persistence system of the kernel mode module updates the size of data written into the storage device to the file system in the process of writing the data in the file into the storage device of the cache server, and synchronizes abnormal state information including the error type of the abnormality to the file system when the file writing is abnormal.
Specifically, the user mode program can read the data receiving progress and the file writing progress through the file system, and when abnormal state information in the file system is read, whether retry is performed is judged according to error information in the abnormal state information; taking the abnormal state information as an example of the abnormality occurring when the data in the file is written into the storage device, when the user mode program reads the abnormal state information, obtaining an error type in the abnormal state information, if the error type is a retriable error, for example, a temporary problem caused by insufficient memory of the cache server, the user mode program judges that the retry can be performed, the user mode program calls a file writing interface provided by the kernel mode module, and the persistence system of the kernel mode module writes the data in the file into the storage device; if the error type is an unrepeatable error, for example, the problem of unrecoverable caused by a disk medium error, the user mode program judges that the retry is impossible, directly calls an interface provided by the kernel mode module and used for interrupting the current file writing, and the persistence system immediately stops the operation of writing the data in the file into the storage device and disconnects the communication connection. The user mode module can preset a retry time threshold, if the number of times of abnormality occurrence during file writing of the persistence system is larger than the retry time threshold, the user mode module judges that retry is not performed any more, an interface provided by the kernel mode module and used for interrupting current file writing is directly called, and the persistence system immediately stops the operation of writing data in the file into the storage device and disconnects communication connection.
In one example, the kernel mode module further provides an interface for temporarily setting the current file writing and an interface for continuing the current file writing, so that the user mode program can adjust the writing sequence of the files according to the priority of the files to be written; for example, if a priority of a first file currently written into the storage device is low, if a new second file needs to be written into the storage device and the priority of the second file is high, the user mode program calls the interface for suspending the writing of the first file first and calls the second file writing interface to write the second file, the kernel mode module suspends the writing of the first file and writes the second file, when the writing of the second file is completed, the user mode program calls the interface for continuing the writing of the current file, and the kernel mode module continues the writing of the first file.
In this embodiment, the user mode program can determine whether or not to retry the program based on error information indicating an abnormal state when the abnormal state occurs, as compared with the first embodiment.
A fifth embodiment of the present invention relates to a cache server applied to a content distribution network, the cache server being configured to store data received from an origin server in a storage device of the cache server; the storage device may be a disk, a hard disk, etc. in the cache server. Referring to fig. 1, at least one application program generally runs in a user mode, which may be referred to as a user mode program 11, the cache server 1 further includes a kernel mode module 12 running in a kernel mode, an address space corresponding to the kernel mode module 12 stores running codes of a kernel, the kernel module, and data maintained by the kernel, the kernel mode module 12 may include a persistence system 121 and a file system 122, in an example, the kernel mode module 12 further includes a network protocol stack 123, and the kernel mode module 12 may establish a communication connection with the source server 2 through the network protocol stack 123. The privilege level of the user mode program 11 is the lowest, the privilege level of the kernel mode module 12 is the highest, and when the user mode program 11 needs to execute work exceeding its own privilege level, for example, the parent and child processes copy information, copy setting and directory table entry, etc., it will call the function interface in the kernel mode module 12 to implement.
The kernel mode module 12 is configured to create a file corresponding to the communication connection after the cache server 1 establishes the communication connection with the origin server 2, and send path information of the file to the user mode program 11.
The kernel mode module 12 is further configured to copy data into a file upon receiving the data sent by the source server 2 through the communication connection.
In one example, the kernel mode module 12 includes: a persistence system 121 and a file system 122.
The persistent system 121 is configured to create a file corresponding to the communication connection in the file system after the cache server 1 establishes the communication connection with the origin server 2, and send path information of the file to the user mode program 11.
The persistence system 121 is configured to copy data into a file upon receiving data sent by the origin server 2 over the communication connection.
The user mode program 12 is used for controlling the kernel mode module 11 to store the data in the file into a storage device (not shown in the figure) of the cache server 1 based on the path information.
Since the first to fourth embodiments correspond to the present embodiment, the present embodiment can be implemented in cooperation with the first to fourth embodiments. The related technical details mentioned in the first to fourth embodiments are still valid in this embodiment, and the technical effects that can be achieved in the first to fourth embodiments can also be achieved in this embodiment, and are not described herein again in order to reduce repetition. Accordingly, the related-art details mentioned in the present embodiment can also be applied to the first to fourth embodiments.
In this embodiment, after the cache server establishes a communication connection with the source server, the kernel mode module may create a file corresponding to the communication connection in the file system, and send path information of the file to the user mode program; when receiving data sent by a source server through communication connection, the kernel mode module copies the data into a file, so that a user mode program can control the kernel mode module to store the data in the file into storage equipment of a cache server according to path information of the received file; the user mode program controls the kernel mode module to store the data in the file into the storage device of the cache server through the path information of the file sent by the kernel mode module, the received data only needs to be cached in the memory of the kernel mode module, and only the path information of the file needs to be interacted between the user mode program and the kernel mode module, so that the copy process of the data between the kernel mode module and the user mode program is reduced, and the processor resource and the memory resource of the cache server are saved; meanwhile, the process that the cache server receives the data and finally stores the data into the storage device for persistence is simplified, and the speed of receiving the data is increased to a certain extent.
A sixth embodiment of the present invention relates to a cache server including: at least one processor; and a memory communicatively coupled to the at least one processor; the processor comprises a user mode program and a kernel mode module; the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data storage method of any one of the first to fourth embodiments.
Where the memory and processor are connected by a bus, the bus may comprise any number of interconnected buses and bridges, the buses connecting together one or more of the various circuits of the processor and the memory. The bus may also connect various other circuits such as peripherals, voltage regulators, power management circuits, and the like, which are well known in the art, and therefore, will not be described any further herein. A bus interface provides an interface between the bus and the transceiver. The transceiver may be one element or a plurality of elements, such as a plurality of receivers and transmitters, providing a means for communicating with various other apparatus over a transmission medium. The data processed by the processor is transmitted over a wireless medium via an antenna, which further receives the data and transmits the data to the processor.
The processor is responsible for managing the bus and general processing and may also provide various functions including timing, peripheral interfaces, voltage regulation, power management, and other control functions. And the memory may be used to store data used by the processor in performing operations.
It will be understood by those of ordinary skill in the art that the foregoing embodiments are specific examples for carrying out the invention, and that various changes in form and details may be made therein without departing from the spirit and scope of the invention in practice.
Claims (12)
1. A data storage method is characterized in that the method is applied to a cache server comprising a user mode program and a kernel mode module; the method comprises the following steps:
after the cache server establishes communication connection with a source server, the kernel mode module creates a file corresponding to the communication connection and sends path information of the file to the user mode program;
when data sent by the source server through the communication connection is received, the kernel mode module copies the data into the file;
and the user mode program controls the kernel mode module to store the data in the file into a storage device of the cache server based on the path information.
2. The data storage method of claim 1, wherein the file comprises at least one kernel page, and wherein the kernel mode module copies the data into the file, comprising:
the kernel mode module copies the data to the kernel page of the file and sets the kernel page cached with the data to be dirty;
the user mode program controls the kernel mode module to store the data in the file into a storage device of the cache server based on the path information, and the method comprises the following steps:
the user mode program controls the kernel mode module to store the data in the dirty kernel page into the storage device based on the path information.
3. The data storage method according to claim 2, wherein the user mode program controls the kernel mode module to store the data in the dirty kernel page into the storage device based on the path information, and the method comprises:
when the user mode program judges that a preset condition is met, controlling the kernel mode module to store the data in the dirty kernel page into the storage device based on the path information;
the preset condition is any one of the following conditions:
the available memory of the cache server is smaller than a preset threshold, the time for storing the dirty kernel page in the memory of the kernel mode module is larger than the preset time, and a storing instruction sent by a user side is received.
4. The data storage method of claim 1, before the kernel-mode module creates the file corresponding to the communication connection, further comprising:
before establishing the communication connection with the source server, the user mode program sends a resource acquisition request to the kernel mode module; wherein the resource acquisition request includes the path information;
the kernel mode module creates a file corresponding to the communication connection, including:
and the kernel mode module creates the file corresponding to the communication connection according to the path information in the resource acquisition request.
5. The data storage method of claim 1, wherein the kernel mode module creates a file corresponding to the communication connection, comprising:
and the kernel mode module generates path information of the file corresponding to the communication connection and creates the file corresponding to the communication connection according to the generated path information.
6. The data storage method of claim 1, wherein the sending the path information of the file to the user mode program comprises:
the kernel mode module sends the path information of the file to the user mode program when the file corresponding to the communication connection is successfully created;
when the kernel mode module fails to create the file corresponding to the communication connection, first notification information representing the file creation failure is sent to the user mode program;
the data storage method further comprises the following steps:
and when the user mode program receives the first notification information, controlling the kernel mode module to close the communication connection.
7. The data storage method according to claim 1, wherein after the user mode program controls the kernel mode module to store the data in the file into the storage device of the cache server based on the path information, the method further comprises:
and when judging that the communication connection is abnormally closed, the kernel mode module deletes the data in the file stored in the storage device.
8. The data storage method of claim 1, wherein the method further comprises:
when the kernel mode module is abnormal in the process of receiving the data sent by the source server or storing the data in the file into the storage device, the kernel mode module synchronizes abnormal state information including error information to the file system;
and when the user mode program reads abnormal state information in the file system, judging whether to retry according to the error information in the abnormal state information, and controlling the kernel mode module to retry when judging to retry.
9. The data storage method of claim 1, wherein the sending the path information of the file to the user mode program comprises:
the kernel mode module sends the control handle corresponding to the file to the user mode program; wherein the control handle contains the path information and an operation handle;
the user mode program controls the kernel mode module to store the data in the file into a storage device of the cache server based on the path information, and the method comprises the following steps:
and the user mode program controls the kernel mode module to store the data in the file into a storage device of the cache server by using the operation handle based on the path information in the control handle.
10. A cache server, comprising: a user mode program and a kernel mode module;
the kernel mode module is used for creating a file corresponding to the communication connection after the cache server establishes the communication connection with the source server, and sending the path information of the file to the user mode program;
the kernel mode module is further used for copying the data into the file when receiving the data sent by the source server through the communication connection;
and the user mode program is used for controlling the kernel mode module to store the data in the file into the storage equipment of the cache server based on the path information.
11. The cache server of claim 10, wherein the kernel-mode module comprises: a persistence system and a file system;
the persistence system is used for creating a file corresponding to the communication connection in the file system after the communication connection is established between the cache server and the source server, and sending the path information of the file to the user mode program;
the persistence system is configured to copy the data to the file upon receiving the data sent by the origin server over the communication connection.
12. A cache server, comprising: at least one processor; and the number of the first and second groups,
a memory communicatively coupled to the at least one processor; the processor comprises a user mode program and a kernel mode module;
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the data storage method of any one of claims 1 to 9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011120513.8A CN112416863A (en) | 2020-10-19 | 2020-10-19 | Data storage method and cache server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202011120513.8A CN112416863A (en) | 2020-10-19 | 2020-10-19 | Data storage method and cache server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN112416863A true CN112416863A (en) | 2021-02-26 |
Family
ID=74841308
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202011120513.8A Pending CN112416863A (en) | 2020-10-19 | 2020-10-19 | Data storage method and cache server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN112416863A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114296646A (en) * | 2021-12-24 | 2022-04-08 | 天翼云科技有限公司 | Caching method, device, server and storage medium based on IO service |
| CN115460470A (en) * | 2022-08-19 | 2022-12-09 | 武汉烽火技术服务有限公司 | Multicast data forwarding method, device, equipment and readable storage medium |
| WO2023273464A1 (en) * | 2021-06-30 | 2023-01-05 | 华为技术有限公司 | Data transmission method and electronic device |
| CN115826885A (en) * | 2023-02-21 | 2023-03-21 | 浪潮电子信息产业股份有限公司 | Data migration method and device, electronic equipment and storage medium |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120290107A1 (en) * | 2011-05-12 | 2012-11-15 | John Carlson | Apparatus and method for displaying state data of an industrial plant |
| CN103544324A (en) * | 2013-11-11 | 2014-01-29 | 北京搜狐新媒体信息技术有限公司 | Kernel-mode data access method, device and system |
| CN107608717A (en) * | 2016-07-12 | 2018-01-19 | 迈普通信技术股份有限公司 | User space formation gathering method, device and system |
| CN111552489A (en) * | 2020-03-31 | 2020-08-18 | 支付宝(杭州)信息技术有限公司 | User mode file system hot upgrading method, device, server and medium |
-
2020
- 2020-10-19 CN CN202011120513.8A patent/CN112416863A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20120290107A1 (en) * | 2011-05-12 | 2012-11-15 | John Carlson | Apparatus and method for displaying state data of an industrial plant |
| CN103544324A (en) * | 2013-11-11 | 2014-01-29 | 北京搜狐新媒体信息技术有限公司 | Kernel-mode data access method, device and system |
| CN107608717A (en) * | 2016-07-12 | 2018-01-19 | 迈普通信技术股份有限公司 | User space formation gathering method, device and system |
| CN111552489A (en) * | 2020-03-31 | 2020-08-18 | 支付宝(杭州)信息技术有限公司 | User mode file system hot upgrading method, device, server and medium |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2023273464A1 (en) * | 2021-06-30 | 2023-01-05 | 华为技术有限公司 | Data transmission method and electronic device |
| CN114296646A (en) * | 2021-12-24 | 2022-04-08 | 天翼云科技有限公司 | Caching method, device, server and storage medium based on IO service |
| CN114296646B (en) * | 2021-12-24 | 2023-06-23 | 天翼云科技有限公司 | Caching method and device based on IO service, server and storage medium |
| CN115460470A (en) * | 2022-08-19 | 2022-12-09 | 武汉烽火技术服务有限公司 | Multicast data forwarding method, device, equipment and readable storage medium |
| CN115460470B (en) * | 2022-08-19 | 2024-03-26 | 烽火通信科技股份有限公司 | Multicast data forwarding method, device, equipment and readable storage medium |
| CN115826885A (en) * | 2023-02-21 | 2023-03-21 | 浪潮电子信息产业股份有限公司 | Data migration method and device, electronic equipment and storage medium |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN112416863A (en) | Data storage method and cache server | |
| US6438707B1 (en) | Fault tolerant computer system | |
| US11403227B2 (en) | Data storage method and apparatus, and server | |
| CN110597910A (en) | Remote data synchronization method, device and system | |
| CN108989432B (en) | User-mode file sending method, user-mode file receiving method and user-mode file receiving and sending device | |
| US10901643B2 (en) | Using log objects in object storage for durability of file objects in volatile memory | |
| EP3891964A1 (en) | Cache warming: agility for a stateful service | |
| US10620871B1 (en) | Storage scheme for a distributed storage system | |
| CA3129982A1 (en) | Method and system for accessing distributed block storage system in kernel mode | |
| CN112667527A (en) | Method for ensuring data acquisition accuracy of client and client | |
| WO2020252724A1 (en) | Log processing method and device, and computer-readable storage medium | |
| CN113612814B (en) | Session information screening method, device, terminal and storage medium | |
| WO2023173718A1 (en) | Communication link update method and apparatus, and related device | |
| CN110119388A (en) | File read/write method, device, system, equipment and computer readable storage medium | |
| CN103714059A (en) | Data updating method and device | |
| EP4227802A1 (en) | Method, apparatus and computing device for configuring compute mode | |
| KR100261149B1 (en) | Pci ethernet controller and method for controlling the same | |
| CN116166470A (en) | Redis cluster clone replication method and device, medium and equipment | |
| CN115033337A (en) | Virtual machine memory migration method, device, equipment and storage medium | |
| CN113806295A (en) | File migration method, system, equipment and computer readable storage medium | |
| CN114721717B (en) | Dual-system-based mobile phone management system mdm instruction calling method and device | |
| EP4123470A1 (en) | Data access method and apparatus | |
| CN112003860B (en) | Memory management method, system and medium suitable for remote direct memory access | |
| KR100194773B1 (en) | Data consistency control method between mobile data processing system and mobile terminal | |
| JP2818575B2 (en) | Wireless base station remote reset system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |