CN111680091A - Multithreading file registration method and device - Google Patents
Multithreading file registration method and device Download PDFInfo
- Publication number
- CN111680091A CN111680091A CN202010494421.XA CN202010494421A CN111680091A CN 111680091 A CN111680091 A CN 111680091A CN 202010494421 A CN202010494421 A CN 202010494421A CN 111680091 A CN111680091 A CN 111680091A
- Authority
- CN
- China
- Prior art keywords
- file
- registered
- target
- registration
- memory
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Software Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Databases & Information Systems (AREA)
- Data Mining & Analysis (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention provides a multithreading file registration method and a multithreading file registration device, which relate to the technical field of information processing, and comprise the following steps: acquiring basic parameters of a file to be registered, a memory file number threshold value and a registration thread number value; scanning a file list of a target position by using a single thread according to the number threshold of the memory files, and determining the information of a target file to be registered according to basic parameters; sending the information of the target file to be registered to a memory database so that the memory database generates state flag data of the target file to be registered; and generating a multithreading file registration result according to the target file information to be registered, the state flag data, the number value of the registration thread and the basic parameter. The invention can improve the file registration efficiency.
Description
Technical Field
The invention relates to the technical field of information processing, in particular to a multithreading file registration method and device.
Background
And file registration, which comprises the process of recording file information to a database and moving files from a source directory to a target directory. When daily downloading files of a plurality of source systems under a certain source directory is processed, the number of the files is large, and if special dates such as the end of a month, the end of a year, the rest days and the like are met, the number of large files is large, so that the file registration process is slow and time-consuming, and the loading speed of subsequent files is influenced. However, in the prior art, in the multithread registration process, the concurrence of the files is controlled by the database, and after the registrable files are obtained by each registration thread, the current file record of the database is inquired, so that not only is the connection of the database frequent, but also the multithread can obtain the same files at the same time, which causes repeated processing, and the file registration efficiency needs to be improved.
Disclosure of Invention
The invention provides a multithreading file registration method and device, which can improve the file registration efficiency.
In a first aspect, an embodiment of the present invention provides a multithreading file registration method, where the method includes: acquiring basic parameters of a file to be registered, a memory file number threshold value and a registration thread number value; scanning a file list of a target position by using a single thread according to the memory file number threshold, and determining target file information to be registered according to the basic parameters; sending the information of the target file to be registered to a memory database so that the memory database generates state mark data of the target file to be registered; and generating a multithreading file registration result according to the target file information to be registered, the state flag data, the registration thread numerical value and the basic parameter.
In a second aspect, an embodiment of the present invention further provides a multithreading file registration apparatus, where the apparatus includes: the parameter module is used for acquiring basic parameters of the file to be registered, the number threshold of the memory file and the number value of the registration thread; the scanning module is used for scanning a file list of a target position by using a single thread according to the memory file number threshold value and determining the information of a target file to be registered according to the basic parameters; the sending module is used for sending the information of the target file to be registered to a memory database so as to enable the memory database to generate state mark data of the target file to be registered; and the registration module is used for generating a multithreading file registration result according to the target file information to be registered, the state flag data, the registration thread numerical value and the basic parameter.
In a third aspect, an embodiment of the present invention further provides a computer device, including a memory and a processor, where the memory stores a computer program operable on the processor, and the processor implements the above-mentioned multi-threaded file registration method when executing the computer program.
In a fourth aspect, the present invention further provides a computer-readable medium having non-volatile program code executable by a processor, where the program code causes the processor to execute the above multithreading file registration method.
The embodiment of the invention has the following beneficial effects: the embodiment of the invention provides a multithreading file registration scheme, which can determine the information of a target file to be registered when a file list of a target position is scanned by using a single thread by acquiring basic parameters of the file to be registered, can control the scanning thread through a memory file number threshold value to avoid accumulation of a large amount of information, generates state mark data of the target file to be registered when the information of the target file to be registered is sent to a memory database, generates a multithreading file registration result according to the information of the target file to be registered, the state mark data, a number of values of a registration thread and the basic parameters, and controls the process of concurrent registration of a plurality of files through the state mark data and the number of values of the registration thread to ensure that the same file cannot be processed repeatedly. The embodiment of the invention can improve the efficiency of file registration.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the aforementioned and other objects, features and advantages of the present invention comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the embodiments or the prior art will be briefly described below, and it is obvious that the drawings in the following description are some embodiments of the present invention, and other drawings can be obtained by those skilled in the art without creative efforts.
FIG. 1 is a flowchart of a multithreading file registration method according to an embodiment of the present invention;
FIG. 2 is a block diagram of a multithreaded file registration apparatus according to an embodiment of the present invention;
FIG. 3 is a block diagram of another multi-threaded file registration apparatus according to an embodiment of the present invention;
fig. 4 is a block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
To make the objects, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some, but not all embodiments of the present invention. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.
At present, in order to improve the file registration efficiency, a common implementation scheme is to configure only a registration multithread pool and control the concurrence condition of multiple files through a database.
Based on this, the multithreading file registration method and device provided by the embodiment of the invention comprise a single-thread scanning function, the registrable files are put into the memory, concurrency is controlled according to the state of 0 or 1, and the maximum number of files is set, so that the memory is ensured not to accumulate a large number of records; not only is the database connection reduced, but also the speed of acquiring files by the registration thread is improved.
For the convenience of understanding the embodiment, a multithreading file registration method disclosed in the embodiment of the present invention is first described in detail.
An embodiment of the present invention provides a multithreading file registration method, which is shown in fig. 1 as a flowchart of the multithreading file registration method, and includes the following steps:
step S102, obtaining basic parameters of the file to be registered, a memory file number threshold value and a registration thread number value.
In the embodiment of the present invention, the basic parameters at least include a file name parameter, a coding mode parameter, a delimiter parameter, a control information line number parameter, and the like. The basic parameters may be used to determine information such as a source location where the file to be registered is located and a final location to which the file to be registered is to be moved, and may also be used to determine information such as a display effect of the file to be registered. The basic parameters of the file to be registered can be manually configured in advance in the configuration file of the database. The threshold of the number of the memory files can be used for determining the number of the files to be registered which can be stored in the memory. The number of registration threads can be used for determining the number of files to be registered which are simultaneously and concurrently registered. The threshold value of the number of the memory files and the numerical value of the number of the registered threads can be manually configured in the configuration files of the memory in advance.
It should be noted that, if the total number of files processed each time and the size of the file do not change much, it is not necessary to reconfigure the memory file number threshold and the registration thread number value before executing the present solution each time, and the memory file number threshold and the registration thread number value may be configured in the initial use without modification in the subsequent process.
And step S104, scanning a file list of the target position by using a single thread according to the memory file number threshold, and determining the information of the target file to be registered according to the basic parameters.
In the embodiment of the invention, after the parameter configuration is finished, the script for running the main flow is called, and the functions of file scanning and file registration are started. The file scan is a single thread. The method comprises the steps of continuously scanning a file list at a target position by using a single thread, filtering files being transmitted, determining files needing to be registered, namely target files to be registered, in the file list at the target position according to basic parameters, and using file names and the like corresponding to the target files to be registered for determining the target files to be registered and attributes of the target files as target file information to be registered.
Step S106, sending the information of the target file to be registered to the memory database so that the memory database generates the state flag data of the target file to be registered.
In the embodiment of the invention, each target file information to be registered can determine a target file to be registered, the target file information to be registered is sent to the memory database, and the memory database sets the state flag data of each target file to be registered as the initial state when storing the target file information to be registered.
Specifically, for a certain target file information to be registered, in the internal storage database, a key value may be set as the file name of the target file to be registered, a value is status flag data, and the initial status of the value may be set to "0".
It should be noted that, in the embodiment of the present invention, a map array may be defined in a memory, and this map array may be used as a memory database, and the registrable file list is placed in the memory through continuous scanning by a single thread, and the memory size is set through the memory file number threshold.
And step S108, generating a multithreading file registration result according to the target file information to be registered, the state flag data, the registration thread numerical value and the basic parameter.
In the embodiment of the invention, the file registration is a multithread, the number of threads which are simultaneously registered is determined according to the number of the registration threads, so that a plurality of files to be registered are acquired according to the information of the target files to be registered for registration, and whether the files to be registered are registered by the registration threads is determined according to the state mark data in the registration process, thereby preventing the same file from being repeatedly processed. The final position of the file to be registered of the moving target can be determined according to the basic parameters, the file to be registered of the moving target is moved according to the final position determined by the basic parameters, and the directory of the file to be registered of the moving target is changed, so that the file registration is completed with higher efficiency.
The embodiment of the invention provides a multithreading file registration scheme, which can determine the information of a target file to be registered when a file list of a target position is scanned by using a single thread by acquiring basic parameters of the file to be registered, can control the scanning thread through a memory file number threshold value to avoid accumulation of a large amount of information, generates state mark data of the target file to be registered when the information of the target file to be registered is sent to a memory database, generates a multithreading file registration result according to the information of the target file to be registered, the state mark data, a number of values of a registration thread and the basic parameters, and controls the process of concurrent registration of a plurality of files through the state mark data and the number of values of the registration thread to ensure that the same file cannot be processed repeatedly. The embodiment of the invention can improve the efficiency of file registration.
Considering that the file scanning speed is generally the file registration speed, in order to facilitate the control of the file scanning progress, the file list at the target position is scanned by using a single thread according to the memory file number threshold, and the method can be implemented according to the following steps:
calculating the number of files to be registered in the memory database according to the information of the target files to be registered; if the number of the files to be registered is larger than or equal to the memory file number threshold value, stopping scanning a file list at a target position by using a single thread; and if the number of the files to be registered is smaller than the threshold of the number of the memory files, scanning a file list at the target position by using a single thread.
In the embodiment of the invention, the information of the target files to be registered is stored in the memory database, the number of the files to be registered in the memory database can be calculated according to the information of the target files to be registered, the number of the files to be registered in the memory database is compared with the threshold value of the number of the memory files, if the number of the files to be registered is less than the threshold value of the number of the memory files, the scanning is continuously scanned or started, and if the number of the files to be registered is greater than or equal to the threshold value of the number of the memory files, the scanning thread is dormant, so that the scanning thread can timely register the scanned target files to be registered, and resources are fully utilized.
In order to reduce the repetition of the target file to be registered in the memory database, thereby reducing the repeated registration processing, improving the processing efficiency and saving resources, the information of the target file to be registered is sent to the memory database, and the method can be executed according to the following steps:
judging whether target file information to be registered exists in a memory database; if so, skipping target file information to be registered; and if not, sending the target file information to be registered to the memory database.
In the embodiment of the invention, if the information of the target file to be registered already exists in the memory database, the information of the target file to be registered is not repeatedly stored in the memory database, and the current information of the target file to be registered is not processed and skipped. And if the current target file information to be registered does not exist in the memory database, sending the current target file information to be registered to the memory database for storage.
In order to ensure that the same file cannot be processed repeatedly, reduce database connection and improve loading efficiency, the state mark data comprises an initial state and an occupied state; generating a multithreading file registration result according to the target file information to be registered, the state flag data, the number value of the registration thread and the basic parameter, wherein the multithreading file registration result can be executed according to the following steps:
acquiring information of a plurality of target files to be registered in a memory database according to the number of the registration threads; modifying the state identification data of the information of the target files to be registered from the initial state to an occupied state, and locking the target files to be registered in the occupied state; and generating a multithreading file registration result according to the basic parameters and the locked target file to be registered.
In the embodiment of the present invention, for example, if the number of the registration threads is N, N pieces of target file information to be registered in the memory database are obtained, and the state identification data of each piece of target file information to be registered is modified from the initial state to the occupied state, for example, if the initial state is represented by "0" and the occupied state is represented by "1", each piece of state identification data in the N pieces of target file information to be registered is changed from "0" to "1". And locking the target file to be registered in each occupation state to prevent other registration threads from accessing the target file to be registered again. And finally, sending the locked target file to be registered to a target database according to the basic parameters, changing the directory information of the locked target file to be registered, completing multithreading file registration and obtaining a multithreading file registration result.
Considering that there are many situations in the file registration process, in order to optimize the file registration result under different situations, the multithreading file registration result is generated according to the basic parameters and the locked target file to be registered, and the method can be executed according to the following steps:
sending the locked target file to be registered to a first target database according to the basic parameters, and judging whether the sending is successful; if so, changing the directory of the locked target file to be registered into a first target directory, and deleting the information of the target file to be registered corresponding to the locked target file to be registered; if not, judging whether the locked target file to be registered exists in the first target database; if yes, sending the locked target file to be registered to a second target database; and if the target file to be registered does not exist, sending the locked target file to be registered to a third target database.
In the embodiment of the invention, the position of the first target database corresponding to the locked target file to be registered and the corresponding first target directory can be determined according to the basic parameters. If the target file to be registered is successfully sent to the first target database, the first target database already has the target file to be registered, which indicates that the target file to be registered is successfully registered in the first target database. When the same file is sent to the first target database, the sending fails. And if the locked target file to be registered is sent to the first target database and the sending is successful, changing the directory of the target file to be registered into the first target directory, deleting the information of the target file to be registered corresponding to the file in the memory database, and finishing the file registration of the thread. If the locked target file to be registered is sent to the first target database, the sending is failed, and the reason of the failure needs to be judged, if the target file to be registered exists in the first target database, the successful registration is proved, the repeated registration is not carried out, the target file to be registered is sent to the second target database, if the target file to be registered is not registered in the first target database, and the target file to be registered is still inserted into the first target database to fail, the target file to be registered is indicated to possibly have problems, and the target file to be registered needs to be sent to the third target database, so that the registration file in the third target database can be manually processed in time.
It should be noted that, in the embodiment of the present invention, the oracle database may be used for the first target database, the second target database, and the third target database.
Considering the situation that the register thread timeout connection may exist, the scheme may further perform the following steps:
acquiring a time length threshold; and if the target registration thread with the connection duration being greater than or equal to the duration threshold exists, disconnecting the target registration thread, and modifying the state flag data corresponding to the target registration thread from the occupied state to the initial state.
In the embodiment of the present invention, the duration threshold may be set according to actual needs, which is not specifically limited in the embodiment of the present invention. Monitoring the connection duration of the registration thread, if a target registration thread with the connection duration being greater than or equal to the duration threshold exists, indicating that an exception occurs in the processing process, disconnecting the exception thread, and modifying the state flag data corresponding to the target file to be registered related to the exception thread from an occupied state to an initial state because the registration is not actually completed, for example, modifying the state identification data from '1' to '0' to wait for the next reprocessing.
In the embodiment of the invention, technicians need to configure file parameters, and set the size of the registered thread pool and the maximum number of files which can be stored in the memory. And evaluating the size of the thread pool, the size of the memory and the thread dormancy time according to the total number of files and the number of large files per day. Configuring file information in a database parameter table, defining a map array of a ConcurrentHashMap structure in a memory, and setting the size of the map; then scanning files in a source directory through a single thread without stopping to obtain a file list to be registered, adding the files into a map and initializing the state, and when the map exceeds a specified size, sleeping the thread; and the registration thread in the registration thread pool acquires a record from the map and marks the record at the same time, and deletes the record after the registration is finished. The process is illustrated below in one specific example:
1. parameters of configuration files in the database mainly comprise file names, encoding modes, separators and control information line numbers;
2. configuring the maximum number of memory files, the number of threads of a registered thread pool, the connection keeping time and the like in the configuration file;
3. after the parameter configuration is finished, calling a script for running a main flow, and starting file scanning and file registration functions:
1) the file scanning is a single thread, a file list under a source directory is scanned continuously by while circulation, transmitted files are filtered, and the files are put in if the current files do not exist in the memory map, the key value is a file name, and the value is in an initial state of 0; when the number of files in the memory exceeds the specified maximum number of files, the thread sleeps;
2) file registration is a multithread, a thread is obtained from a registration thread pool, a while cycle continuously obtains a registrable record (value is 0) from a memory map, and the value in the map is changed to 1 (synchronous lock) while the record is obtained; then reading database parameter table information, inserting records into a database, wherein the state is that the registration is successful 1000, if the insertion is successful, moving files to a target directory (directory to be loaded), if the insertion is unsuccessful, judging whether old files failed to be registered last time or repeated files according to the state 1000, if the old files are not successful, updating the database records and moving the files to the target directory (directory to be loaded), and if the repeated files are directly moved to the target directory (directory of repeated files); deleting the record in the memory map after the steps are completed; and if an exception occurs in the processing process, the value of the record in the internal map is recovered to be 0, and the record is waited to be reprocessed next time.
The invention provides a multithreading file registration method and a device, wherein the method divides the process of acquiring a registrable file and the file registration into two independent processes, the first process uses a single thread to continuously scan a source directory, and the registrable file is acquired and then is put into a memory; the second process uses multiple threads to continuously fetch files from the memory to complete registration. The single thread scanning module reads the file from the source directory and puts the file into the memory; and the multithreading registration module reads the memory record, calls the parameter module to obtain related parameters, records file information to an oracle database, moves the file from a source directory to a target directory of the mobile part, and deletes the memory record after the completion. The method realizes multi-file concurrent processing by using single-thread scanning and multi-thread registration, reduces database connection and improves file registration efficiency while ensuring that the same file cannot be processed repeatedly.
An embodiment of the present invention further provides a multithreading file registration apparatus, and referring to a structural block diagram of the multithreading file registration apparatus shown in fig. 2, the apparatus includes:
the parameter module 71 is configured to obtain a basic parameter of a file to be registered, a memory file number threshold, and a registration thread number value; the scanning module 72 is configured to scan a file list of a target location by using a single thread according to a threshold of the number of memory files, and determine information of a target file to be registered according to basic parameters; a sending module 73, configured to send the information of the target file to be registered to the memory database, so that the memory database generates status flag data of the target file to be registered; and the registration module 74 is configured to generate a multithreading file registration result according to the target file information to be registered, the state flag data, the registration thread numerical value, and the basic parameter.
In one embodiment, the scanning module is specifically configured to: calculating the number of files to be registered in the memory database according to the information of the target files to be registered; if the number of the files to be registered is larger than or equal to the memory file number threshold value, stopping scanning a file list at a target position by using a single thread; and if the number of the files to be registered is smaller than the threshold of the number of the memory files, scanning a file list at the target position by using a single thread.
In one embodiment, the sending module is specifically configured to: judging whether target file information to be registered exists in a memory database; if so, skipping target file information to be registered; and if not, sending the target file information to be registered to the memory database.
In one embodiment, the registration module is specifically configured to: acquiring information of a plurality of target files to be registered in a memory database according to the number of the registration threads; modifying the state identification data of the information of the target files to be registered from the initial state to an occupied state, and locking the target files to be registered in the occupied state; and generating a multithreading file registration result according to the basic parameters and the locked target file to be registered.
In one embodiment, the registration module is specifically configured to: sending the locked target file to be registered to a first target database according to the basic parameters, and judging whether the sending is successful; if so, changing the directory of the locked target file to be registered into a first target directory, and deleting the information of the target file to be registered corresponding to the locked target file to be registered; if not, judging whether the locked target file to be registered exists in the first target database; if yes, sending the locked target file to be registered to a second target database; and if the target file to be registered does not exist, sending the locked target file to be registered to a third target database.
In one embodiment, referring to the block diagram of another multithreaded file registration apparatus shown in fig. 3, the apparatus further includes a duration module 75 for: acquiring a time length threshold; and if the target registration thread with the connection duration being greater than or equal to the duration threshold exists, disconnecting the target registration thread, and modifying the state flag data corresponding to the target registration thread from the occupied state to the initial state.
The embodiment of the present invention further provides a computer device, referring to the schematic block diagram of the structure of the computer device shown in fig. 4, the computer device includes a memory 81 and a processor 82, the memory stores a computer program that can be executed on the processor, and the processor implements the steps of any one of the methods when executing the computer program.
It can be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the computer device described above may refer to the corresponding process in the foregoing method embodiment, and is not described herein again.
Embodiments of the present invention also provide a computer readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform any of the steps of the above-described method.
As will be appreciated by one skilled in the art, embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above-mentioned embodiments are only specific embodiments of the present invention, which are used for illustrating the technical solutions of the present invention and not for limiting the same, and the protection scope of the present invention is not limited thereto, although the present invention is described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: any person skilled in the art can modify or easily conceive the technical solutions described in the foregoing embodiments or equivalent substitutes for some technical features within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the embodiments of the present invention, and they should be construed as being included therein. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (11)
1. A multi-threaded file registration method, comprising:
acquiring basic parameters of a file to be registered, a memory file number threshold value and a registration thread number value;
scanning a file list of a target position by using a single thread according to the memory file number threshold, and determining target file information to be registered according to the basic parameters;
sending the information of the target file to be registered to a memory database so that the memory database generates state mark data of the target file to be registered;
and generating a multithreading file registration result according to the target file information to be registered, the state flag data, the registration thread numerical value and the basic parameter.
2. The method of claim 1, wherein scanning a list of files in a target location with a single thread according to the threshold number of memory files comprises:
calculating the number of files to be registered in a memory database according to the information of the target files to be registered;
if the number of the files to be registered is larger than or equal to the memory file number threshold value, stopping scanning a file list at a target position by using a single thread;
and if the number of the files to be registered is smaller than the memory file number threshold value, scanning a file list at a target position by using a single thread.
3. The method of claim 1, wherein sending the target file information to be registered to an in-memory database comprises:
judging whether the target file information to be registered exists in the memory database;
if yes, skipping the target file information to be registered;
and if not, sending the target file information to be registered to a memory database.
4. The method of claim 1, wherein the status flag data comprises an initial status and an occupied status;
generating a multithreading file registration result according to the target file information to be registered, the state flag data, the registration thread numerical value and the basic parameter, wherein the multithreading file registration result comprises the following steps:
acquiring information of a plurality of target files to be registered in a memory database according to the number values of the registration threads;
modifying the state identification data of the information of the target files to be registered from the initial state to an occupied state, and locking the target files to be registered in the occupied state;
and generating a multithreading file registration result according to the basic parameters and the locked target file to be registered.
5. The method of claim 4, wherein generating a multi-threaded file registration result according to the basic parameters and the locked target file to be registered comprises:
sending the locked target file to be registered to a first target database according to the basic parameters, and judging whether the sending is successful;
if so, changing the directory of the locked target file to be registered into a first target directory, and deleting the information of the target file to be registered corresponding to the locked target file to be registered;
if not, judging whether the locked target file to be registered exists in the first target database;
if yes, sending the locked target file to be registered to a second target database;
and if the target file to be registered does not exist, sending the locked target file to be registered to a third target database.
6. The method of claim 4 or 5, further comprising:
acquiring a time length threshold;
and if the target registration thread with the connection duration being greater than or equal to the duration threshold exists, disconnecting the target registration thread, and modifying the state flag data corresponding to the target registration thread from the occupied state to the initial state.
7. A multithreaded file registration apparatus, comprising:
the parameter module is used for acquiring basic parameters of the file to be registered, the number threshold of the memory file and the number value of the registration thread;
the scanning module is used for scanning a file list of a target position by using a single thread according to the memory file number threshold value and determining the information of a target file to be registered according to the basic parameters;
the sending module is used for sending the information of the target file to be registered to a memory database so as to enable the memory database to generate state mark data of the target file to be registered;
and the registration module is used for generating a multithreading file registration result according to the target file information to be registered, the state flag data, the registration thread numerical value and the basic parameter.
8. The apparatus of claim 7, wherein the scanning module is specifically configured to:
calculating the number of files to be registered in a memory database according to the information of the target files to be registered;
if the number of the files to be registered is larger than or equal to the memory file number threshold value, stopping scanning a file list at a target position by using a single thread;
and if the number of the files to be registered is smaller than the memory file number threshold value, scanning a file list at a target position by using a single thread.
9. The apparatus of claim 7, wherein the registration module is specifically configured to:
acquiring information of a plurality of target files to be registered in a memory database according to the number values of the registration threads;
modifying the state identification data of the information of the target files to be registered from the initial state to an occupied state, and locking the target files to be registered in the occupied state;
and generating a multithreading file registration result according to the basic parameters and the locked target file to be registered.
10. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method of any one of claims 1 to 6 when executing the computer program.
11. A computer-readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform the method of any of claims 1 to 6.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010494421.XA CN111680091B (en) | 2020-06-03 | 2020-06-03 | Multithread file registration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010494421.XA CN111680091B (en) | 2020-06-03 | 2020-06-03 | Multithread file registration method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111680091A true CN111680091A (en) | 2020-09-18 |
| CN111680091B CN111680091B (en) | 2023-04-28 |
Family
ID=72453046
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010494421.XA Active CN111680091B (en) | 2020-06-03 | 2020-06-03 | Multithread file registration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111680091B (en) |
Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110107198A1 (en) * | 2009-10-29 | 2011-05-05 | Canon Kabushiki Kaisha | Information processing apparatus, storage medium, and information processing method |
| CN102314572A (en) * | 2011-10-14 | 2012-01-11 | 迈普通信技术股份有限公司 | Registration information file generation method and device, and use authorization method of application software |
| US20130044117A1 (en) * | 2011-08-18 | 2013-02-21 | International Business Machines Corporation | Vector register file caching of context data structure for maintaining state data in a multithreaded image processing pipeline |
| CN103841196A (en) * | 2014-03-07 | 2014-06-04 | 长沙裕邦软件开发有限公司 | File uploading system and method based on multithreading |
| CN104834506A (en) * | 2015-05-15 | 2015-08-12 | 北京北信源软件股份有限公司 | Method for processing business application through multiple threads |
| CN109190333A (en) * | 2018-08-22 | 2019-01-11 | 山东超越数控电子股份有限公司 | A kind of high intensity method for protecting software |
-
2020
- 2020-06-03 CN CN202010494421.XA patent/CN111680091B/en active Active
Patent Citations (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110107198A1 (en) * | 2009-10-29 | 2011-05-05 | Canon Kabushiki Kaisha | Information processing apparatus, storage medium, and information processing method |
| US20130044117A1 (en) * | 2011-08-18 | 2013-02-21 | International Business Machines Corporation | Vector register file caching of context data structure for maintaining state data in a multithreaded image processing pipeline |
| CN102314572A (en) * | 2011-10-14 | 2012-01-11 | 迈普通信技术股份有限公司 | Registration information file generation method and device, and use authorization method of application software |
| CN103841196A (en) * | 2014-03-07 | 2014-06-04 | 长沙裕邦软件开发有限公司 | File uploading system and method based on multithreading |
| CN104834506A (en) * | 2015-05-15 | 2015-08-12 | 北京北信源软件股份有限公司 | Method for processing business application through multiple threads |
| CN109190333A (en) * | 2018-08-22 | 2019-01-11 | 山东超越数控电子股份有限公司 | A kind of high intensity method for protecting software |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111680091B (en) | 2023-04-28 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US20190129811A1 (en) | Query Fault Processing Method and Processing Apparatus | |
| CN110598054B (en) | Multithreading linked list processing method and device and computer readable storage medium | |
| CN109885565B (en) | Data table cleaning method and device | |
| CN112114903B (en) | File loading method and device | |
| US7269589B2 (en) | Database managing method and system having data backup function and associated programs | |
| CN111680091B (en) | Multithread file registration method and device | |
| CN111984662B (en) | Method and device for updating databases in batches | |
| CN111708618A (en) | Processing method and device based on Java multithreading | |
| US8719792B2 (en) | Correcting job control language job errors | |
| CN116126846A (en) | Data quality detection method, device and medium | |
| CN112115393B (en) | Page loading method, system, device and medium | |
| CN113836112A (en) | Data migration method, system, device and medium | |
| CN113434360A (en) | Method and system for monitoring operation | |
| CN106484776B (en) | Multithreading-based data processing method and device | |
| CN112819600A (en) | Timed task execution method, timed task execution device, timed task execution equipment and computer storage medium | |
| CN115658182B (en) | Method for automatically reloading XML mapping file of MyBatis | |
| CN117312320A (en) | Springbatch-based algorithm for performing fragmentation processing on large amount of data | |
| CN113239061B (en) | Intelligent court data updating method and system | |
| CN114647460A (en) | Application program starting method and device | |
| CN116719971B (en) | Pull chain table data loading method and device and electronic equipment | |
| US20210406067A1 (en) | Distributed storage method, electronic apparatus and non-transitory computer-readable storage medium | |
| CN116521404A (en) | Dependency diagnosis method, dependency diagnosis device, electronic equipment and storage medium | |
| CN115562842A (en) | Process interaction method and system based on Activiti | |
| CN115599861A (en) | Data verification method, storage medium and equipment for database synchronization process | |
| JP2006293582A (en) | Automatic job rerun system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |