CN112711567A

CN112711567A - Session file use space cleaning method and system

Info

Publication number: CN112711567A
Application number: CN202110013710.8A
Authority: CN
Inventors: 杨恒; 宣旭迎; 杨宏兵; 李洪新; 张承冰; 周石开; 周天睿; 袁鹏飞; 文中领; 周泽湘
Original assignee: Hunan Tongyou Feiji Technology Co ltd
Current assignee: Hunan Tongyou Feiji Technology Co ltd
Priority date: 2021-01-06
Filing date: 2021-01-06
Publication date: 2021-04-27
Anticipated expiration: 2041-01-06
Also published as: CN112711567B

Abstract

The invention provides a method and a system for cleaning a Session file use space. The method comprises the following steps: calculating the current system space occupancy rate according to the current session file occupancy space and the total system space; judging whether the current system space occupancy rate exceeds a cleaning triggering threshold value; and if the current system space occupancy rate exceeds the cleaning triggering threshold value, executing the space cleaning operation. The method and the system for clearing the Session file use space provided by the invention ensure that the file system keeps sufficient use space and the normal operation of the Web system is ensured.

Description

Session file use space cleaning method and system

Technical Field

The invention relates to the technical field of Web systems, in particular to a method and a system for cleaning a Session file use space.

Background

Sessions are applied in network applications, especially in Web systems; when a user is switched between different Web pages, the configuration required by the session is that the characteristic information of the user is usually stored in the session, and the whole session always exists; the Session mechanism enables the Web system to be as convenient as the CS application client, and the Session stored in the server side also ensures the security of the Web system.

The access mode is usually a file, the default condition is also the file, a user initially enters a Web system and may immediately require the user to verify account information for logging in, the Web service writes a Session into the file after the request is finished according to whether the Session is started, and the file is stored in a preset path; when the page is switched, the browser sends the ID parameter of Session to the Web service site, the Web service reads and analyzes the corresponding file according to the ID parameter, loads the user configuration information written last time, and the Web system presents interaction according to the user configuration information.

The Session is used for paying that the ID parameters are added to the browser, usually 32-bit random hexadecimal characters are used, and the Web service is added with a once-read file and a once-write file; what is obtained is that the stateless characteristic of the HTTP is solved, so that the Web system has the memory capacity, the URL request is solved without context, and the server side stores the state of the client side.

For Web systems, the pressure of one file read operation and one file write operation in one request to a disk is very small, and the size of a Session file is usually within one kilobyte.

However, for today in the big data era, the concurrency and user volume of the Web system must be considered from the technical pre-research and architecture; when the concurrency and the user amount are too large, the phenomenon that the Session files are overflowed is inevitable, the Session use space is suddenly expanded, the Web system cannot be normally used, and the server is paralyzed.

The prior art solutions to the above problems are roughly as follows:

the GC mechanism is as follows: sessions of the Web server usually have a GC mechanism, that is, garbage collection, execute a garbage collection action probabilistically according to a preset parameter, delete Session files exceeding a preset time, and well alleviate the problem of using space.

And (3) solving by a database: the Session information can be stored in a file, and the Session information can also be stored in a database, so that the database can solve the space problem.

And (3) process solution: the Session information can also be stored in the process of the Web service, namely, in the memory; whenever a Web service restarts, the space it uses can also be freed up.

Manual monitoring: and dispatching a specially-assigned person to monitor the running condition and the use space condition of the Web system in real time and process the running condition and the use space condition in time.

The GC mechanism can alleviate the problem of using space, but it does not really give a medicine to the symptom, even if the probability is set to one hundred percent to trigger, the preset maximum existence time is not easy to grasp, and it is not flexible, and perhaps the setting parameter is too small, which may cause the user who is visiting to drop the line, even the current user directly drops the line, and the situation that the user cannot log in occurs.

The database is really a solution that seems perfect using space; the database mode needs to specify the configuration to the connection closing and reading and writing of the database, and each interface of the Session is completed, so that the Session loses convenience; the access of the database is usually based on a file system, the direct file reading and writing is faster than the database query writing, the database also needs to be connected and closed, the advantages of the database are more embodied in the safety control of structured data and access, and the use of the database for Session is not beneficial to the performance of the characteristics of the database.

The performance of the process is improved, but the exchange of the internal memory for the hard disk space is undoubtedly not desirable, and the internal memory space is more precious; and the restart can cause data loss, which is more unsafe.

The manual mode is undoubtedly a cost increase for stability, but for a large user quantity Web system which has to contend for minutes, the manual mode is undoubtedly a water waggon, and manpower can only be used as a final guarantee.

The Web service is arranged in a network, and is under a normal condition when being subjected to malicious attack, the malicious attack usually simulates login action, the login is not required to be successful, only the login is required to be continuously tried, the Web service opens a Session and does not write any content, namely the Web service is closed, the Web service generates a large number of files with the size of 0, the malicious attack is to crack a password instead of generating the empty files, the files are generated in a short time and cannot exceed the preset maximum existence time and cannot be recycled by a GC;

these file sizes are indeed 0, since nothing is written, but the file must occupy a small portion of space, at least the file name, the file host, the time of the last modification of the file, etc.

As can be known from the linux kernel source code, a new empty file occupies one Inode, and fs/ext4/ext4.h of linux 4.19.154 define the Inode

struct ext4_inode{

__ le16 i _ mode; # File Authority

__ le16 i _ uid; # File owner ID

__ le32 i _ size; size of # File byte number

__ le32 i _ atime; time when # File was last accessed

__ le32 i _ ctime; # File creation time

__ le32 i _ mtime; time when # File is modified

__ le32 i _ dtime; time when # File is deleted

__ le16 i _ gid; group ID to which # File belongs

__ le16 i _ links _ count; # the number of times the inode of this File is connected

__ le32 i _ blocks; number of blocks of # File

......

__ le32 i _ block [ EXT2_ N _ BLOCKS ]; array of # blocks to store File data

......

Both Ext4 and xfs are 256 defaults, the structure of a file has no filename, and the filename is stored in a folder:

struct ext4_dir_entry_2{

__le32 inode； /*Inode number*/

__le16 rec_len； /*Directory entry length*/

__u8 name_len； /*Name length*/

__u8 file_type；

char name[EXT4_NAME_LEN]；/*File name*/

}；

it can be seen that, although an empty file, the Inode occupied by the empty file and the filename occupied by the folder also need space; the test results show that the xfs file system writes 10 ten thousand empty files occupying about 54.55MB, and the ext4 writes 10 ten thousand files occupying about 5M space.

For malicious attacks or empty files, the session can be solved without any measures.

Disclosure of Invention

The technical problem to be solved by the invention is to provide a method and a system for clearing the used space of the Session file, so that the file system can keep sufficient used space and the normal operation of the Web system can be ensured.

In order to solve the technical problem, the invention provides a method for cleaning a Session file use space, which comprises the following steps: calculating the current system space occupancy rate according to the current session file occupancy space and the total system space; judging whether the current system space occupancy rate exceeds a cleaning triggering threshold value; and if the current system space occupancy rate exceeds the cleaning triggering threshold value, executing the space cleaning operation.

In some embodiments, further comprising: generating a random number in real time before calculating the current system space occupancy rate according to the current session file occupancy space and the total system space; comparing the generated numerical value of the random number with a preset reference value of the random number; and determining whether to exit according to the comparison result.

In some embodiments, further comprising: reading an operating identifier RP before generating a random number in real time; if the RP already exists, the execution process exits.

In some embodiments, a space cleanup operation is performed, comprising: and judging whether the file is an empty file or not, and deleting the file if the file is the empty file.

In some embodiments, performing the space cleaning operation further comprises: and judging whether the existing time of the file is greater than the maximum existing time MLT, and if so, deleting the file.

In some embodiments, performing the space cleaning operation further comprises: recording the file name and the last modification time of the file into a linked list; sequencing the session files in the linked list from small to large according to the time of the last modification time; traversing the linked list; files in the linked list are deleted sequentially.

In some embodiments, performing the space cleaning operation further comprises: after recording the file name and the last modification time of the file into the linked list, judging whether the current available space is larger than the total space answer percentage or not before sequencing the session files in the linked list from small to large according to the last modification time, and if so, exiting the current process.

In addition, the invention also provides a system for cleaning the used space of the Session file, which comprises the following components: one or more processors; storage means for storing one or more programs which, when executed by the one or more processors, cause the one or more processors to implement the method for clearing the space used by the Session file as described above.

After adopting such design, the invention has at least the following advantages:

the Session automatic space cleaning system or scheme can flexibly control the use rate of the used space, and ensure that a user normally logs in the Web system and accesses the Web system.

Compared with the overtime recovery concerned by the GC recovery mechanism of the session, the automatic space cleaning of the session pays more attention to the problem of using space, and the problem that the Web system cannot be normally logged in and accessed due to insufficient using space is fundamentally avoided.

Compared with a database or memory mode, the automatic space cleaning mode is based on the original file mode of the session, belongs to default configuration, and is simpler and easier to use; and the automatic space cleaning belongs to an independent system, although the automatic space cleaning is related to session calling, the logic is clear, and the automatic space cleaning does not need to be coupled with a Web server or project codes.

The space cleaning system belongs to a subsystem of Web service, is started along with the starting of the Web service, and is automatically completed in monitoring, triggering and cleaning without manual nursing, so that the labor cost is saved.

The space cleaning system has special processing aiming at the empty file, and the space cleaning system can automatically process and clean the empty file no matter the empty file is attacked maliciously or useless empty file caused by coding specification.

Drawings

The foregoing is only an overview of the technical solutions of the present invention, and in order to make the technical solutions of the present invention more clearly understood, the present invention is further described in detail below with reference to the accompanying drawings and the detailed description.

FIG. 1 is a logical block diagram of a space cleanup system;

FIG. 2 is a system basic flow diagram of a space cleaning system;

FIG. 3 is a system detailed flow diagram of the space cleaning system;

FIG. 4 is a schematic flow diagram of a loop cleaning process;

FIG. 5 is a process flow diagram of the process management unit.

Detailed Description

The preferred embodiments of the present invention will be described in conjunction with the accompanying drawings, and it will be understood that they are described herein for the purpose of illustration and explanation and not limitation.

The scheme needs to define the feature configuration in advance so as to flexibly control the following steps:

trigger threshold cu (cleared used): default 80, value range 1-99, session specifies used space U of path, total space T of path, when U > T × (CU/100), space cleaning executes cleaning action.

Result threshold ca (clear available): defaulting to 80, taking the value range from 1 to 99, session specifies the latest available space A of the path, total space T, and when A > T (CA/100), the space cleaning is finished.

Space clearance factor cd (clear plasmid): the default value is 1, the value range is 1-99, and 0 means that space cleaning is never triggered; and taking the random number R of 1-99, wherein R is CD to trigger space cleaning.

LOG: the default value is 0, the value range is 0-15, 1 represents an information log, 2 represents an alarm log, 4 represents an error log, and 8 represents a debugging log; if the error LOG and debug LOG need to be enabled, the configuration LOG is 12, and 0 means that no LOG is enabled.

Log path ld (logdirection): default null character, log stored file path.

Process identification rp (running process): the cleaning system has no default value, and must be configured, exists at the beginning of the cleaning system, and is automatically deleted at the end, and is generally configured under a temporary directory.

Maximum lifespan MLT (session gc _ maxlifetime): default 1440, meaning 1440 seconds, 24 minutes, may be redefined or may follow the original session configuration.

The complete space cleaning system is shown in fig. 1 and comprises the following parts:

a random number unit (101) for generating a random number based on the parameter

The process management unit (102) generates a new process of an independent original process, manages a process identifier, writes a process number into the process identifier, and exits the called clear process identifier; receiving a process termination signal, and quitting the process; the space cleaning system is required to enter and exit through the process management unit.

And a space unit (103) for acquiring the used space, the total space, the remaining space and the like of the specified path.

And the file cleaning unit (104) is used for judging whether the file can be cleaned or not, the unit belongs to a circulation unit, and the circulation termination condition is that the file traversal of the specified path is completed or the process unit receives a process termination signal.

And a log unit (105) for recording the log output when the space cleaning system runs.

The technical scheme is as follows:

the total space of the Session directory is derived from the total space of the mount point of the file system where the Session directory is located.

When Session is started or initialized, the space cleaning system is called, the cleaning system creates a sub-process, the main process continues to execute downwards without waiting for the completion of the execution of the cleaning system, and the space cleaning system does not return any more but can exit from the main process; whether the space cleaning system is designed to be executed in a global part in project codes, or embedded in source codes before Web service compilation, or designed to be called as an independent system, the space cleaning system is a complete independent part and is not coupled with other parts as far as possible; the execution time of the space cleaning system becomes unpredictable due to the actual situation, and even if the file processing is fast, the file processing causes a little delay when the file amount is too large.

The space cleaning system only outputs logs of all levels to a file finally, the logs are divided into four levels of information, warning, error and debugging, all logs except grammatical errors of the language of the logs are contained, and the detailed scheme introduction is not particularly described.

The detailed technical scheme is as follows:

as shown in fig. 2, the space cleaning system mainly records logs after starting to call, performing process judgment, performing random number judgment, performing space judgment and file cleaning until quitting and all actions in the whole process.

Referring to FIG. 3, the detailed flow of the space cleanup system is as follows, and the left side of each flow is connected with the unit to be called:

step 1: the space cleanup system is entered or invoked.

Step 2: entering a process unit 102, judging an operating identification RP, if the RP exists, directly quitting the process and quitting, and indicating that the space cleaning process is in operation; otherwise, acquiring the current process number, and writing the process number into the running identifier RP; the process initializes a process-terminated global variable for receiving a termination signal sent by the user, and the space cleaning unit 104 may detect and read the process-terminated global variable in real time, respond to the termination information of the user, and terminate the process in time, as shown in fig. 5.

Step 3: calling the random number unit 101, generating random numbers R from 1 to 99, comparing with a space cleaning factor CD, if R < CD, calling the process unit to quit and clean the process identification, not cleaning and directly quitting, and if R > is equal to CD, continuing to execute downwards.

Step 4: calling a space unit 103, judging a trigger threshold CU, acquiring a total space T and a used space U of a designated storage path of the session, and when U is greater than T (CU/100), entering a file processing unit, wherein the used space is larger than the percentage of the total space of the CUs; otherwise, directly quitting the process.

Step 5: as shown in fig. 5, the file cleaning unit 104 is entered, and each loop determines whether a termination signal is received, and calls the process management unit to exit the process after receiving the termination signal.

The file cleaning unit 104 determines whether the file traversal is completed, and completes the traversal calling process to manage the exit process.

And the file cleaning unit 104 is used for judging whether the file is the current session file, and skipping the current cycle to enter the next cycle if the file is the currently accessed session file.

Step 6: the file cleaning unit 104 determines whether the file is an empty file, and deletes the empty file.

The file cleaning unit 104 deletes the file if the existing time (the current time minus the last modification time) of the file is greater than the maximum session maximum existing time MLT.

The file cleaning unit 104 records the file name and the last modification time of the file into the linked LIST.

Step 7: entering a file cleaning unit 104, calling a space unit, judging that the current available space A is larger than CA (A > T (CA/100)) of the total space, and calling a process management unit to quit the process; otherwise, the execution is continued.

Step 8: the linked LIST is sorted from small to large according to time.

Traversal chain LIST

And judging whether a termination signal is received or not, if so, calling the process management unit to quit the process, and if not, continuing to execute.

Sequential deletion of files

And repeating Step7 to judge whether the available space of the designated path meets the requirement, if so, calling the process management unit to quit the process, and if not, continuing to execute.

Step 9: the process is expected to be exited in Step8 eventually with the requirement of meeting the available space; if the last step is still executed, the process management exit process is still called, but abnormal exits are represented in the log.

In summary, the space cleanup program checks that the space usage reaches a predetermined threshold and will clean the space until the usage reaches a pre-configured threshold; after the cleaning logic is triggered, 2 times of circulation are carried out, the files are traversed for the first time, the empty files and the files with the existing time exceeding the maximum life time are cleaned, and a file linked list is recorded; if the available space can not reach the configuration CA in the first circulation cleaning, carrying out a second circulation to sequentially clean the files until the available space reaches the CA; in both cycles, a process termination signal is detected to stop cleaning.

The above description is only for the purpose of illustrating the preferred embodiments of the present invention and is not to be construed as limiting the present invention in any way, and it will be apparent to those skilled in the art that the above description of the present invention can be applied to various modifications, equivalent variations or modifications without departing from the spirit and scope of the present invention.

Claims

1. A method for clearing a Session file use space is characterized by comprising the following steps:

calculating the current system space occupancy rate according to the current session file occupancy space and the total system space;

judging whether the current system space occupancy rate exceeds a cleaning triggering threshold value;

and if the current system space occupancy rate exceeds the cleaning triggering threshold value, executing the space cleaning operation.

2. The method for clearing the space used by the Session file according to claim 1, further comprising:

generating a random number in real time before calculating the current system space occupancy rate according to the current session file occupancy space and the total system space;

comparing the generated numerical value of the random number with a preset reference value of the random number;

and determining whether to exit according to the comparison result.

3. The method for clearing the space used by the Session file according to claim 2, further comprising:

reading an operating identifier RP before generating a random number in real time;

if the RP already exists, the execution process exits.

4. The method for clearing space used by Session file according to claim 1, wherein performing a space clearing operation comprises:

and judging whether the file is an empty file or not, and deleting the file if the file is the empty file.

5. The method for clearing space used by Session file according to claim 1, wherein performing a space clearing operation further comprises:

and judging whether the existing time of the file is greater than the maximum existing time MLT, and if so, deleting the file.

6. The method for clearing space used by Session file according to claim 1, wherein performing a space clearing operation further comprises:

recording the file name and the last modification time of the file into a linked list;

sequencing the session files in the linked list from small to large according to the time of the last modification time;

traversing the linked list;

files in the linked list are deleted sequentially.

7. The method of claim 6, wherein performing a space cleanup operation further comprises:

after recording the file name and the last modification time of the file into the linked list, judging whether the current available space is larger than the total space answer percentage or not before sequencing the session files in the linked list from small to large according to the last modification time, and if so, exiting the current process.

8. A system for clearing a space used by a Session file, comprising:

one or more processors;

a storage device for storing one or more programs,

when executed by the one or more processors, cause the one or more processors to implement the method for clearing space used by a Session file according to any one of claims 1 to 7.