CN111680091B - Multithread file registration method and device - Google Patents
Multithread file registration method and device Download PDFInfo
- Publication number
- CN111680091B CN111680091B CN202010494421.XA CN202010494421A CN111680091B CN 111680091 B CN111680091 B CN 111680091B CN 202010494421 A CN202010494421 A CN 202010494421A CN 111680091 B CN111680091 B CN 111680091B
- Authority
- CN
- China
- Prior art keywords
- file
- registered
- target
- registration
- files
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/25—Integrating or interfacing systems involving database management systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5011—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals
- G06F9/5016—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resources being hardware resources other than CPUs, Servers and Terminals the resource being the memory
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/52—Program synchronisation; Mutual exclusion, e.g. by means of semaphores
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention provides a multithreading file registration method and a device, which relate to the technical field of information processing, and the method comprises the following steps: acquiring basic parameters of a file to be registered, a memory file number threshold value and a registration thread number value; scanning a file list of a target position by using a single thread according to a memory file number threshold value, and determining target file information to be registered according to basic parameters; the method comprises the steps of sending target file information to be registered to a memory database, so that the memory database generates state mark data of the target file to be registered; and generating a multithreaded file registration result according to the target file information to be registered, the state flag data, the registration thread number value and the basic parameters. The invention can improve the file registration efficiency.
Description
Technical Field
The present invention relates to the field of information processing technologies, and in particular, to a method and an apparatus for registering a multithreaded file.
Background
File registration includes the process of recording file information to a database and moving files from a source directory to a target directory. When the daily downloading files of a plurality of source systems under a certain source directory are processed, the number of the files is large, and if special dates such as the end of a month, the end of a year, the date of interest and the like are encountered, the number of the large files is also large, so that the file registration process is slow and time-consuming, and the loading speed of the subsequent files is influenced. The multithread registration method can improve the file registration efficiency to a certain extent, however, in the multithread registration process in the prior art, files are controlled to be concurrent through a database, each registration thread queries the current file record of the database after acquiring the files which can be registered, so that the database is frequently connected, and the multithread possibly acquires the same files at the same time, so that repeated processing is caused, and the file registration efficiency is required to be improved.
Disclosure of Invention
The invention provides a multithreading file registration method and device, which can improve the file registration efficiency.
In a first aspect, an embodiment of the present invention provides a method for registering a multithreaded file, where the method includes: acquiring basic parameters of a file to be registered, a memory file number threshold value and a registration thread number value; scanning a file list of a target position by using a single thread according to the memory file number threshold value, and determining target file information to be registered according to the basic parameters; the target file information to be registered is sent to a memory database, so that the memory database generates state mark data of the target file to be registered, and the state mark data is an initial state at the moment; generating a multithreaded file registration result according to the target file information to be registered, the state flag data, the registration thread number value and the basic parameter;
the method for scanning the file list under the target position by using a single thread according to the memory file number threshold comprises the following steps: calculating the number of files to be registered in a memory database according to the target file information to be registered; if the number of the files to be registered is greater than or equal to the memory file number threshold, stopping scanning the file list at the target position by using a single thread; if the number of the files to be registered is smaller than the number threshold value of the memory files, scanning a file list at a target position by using a single thread;
generating a multithreaded file registration result according to the target file information to be registered, the state flag data, the registration thread number value and the basic parameter, wherein the multithreaded file registration result comprises: acquiring information of a plurality of target files to be registered in a memory database according to the number of the registration threads; modifying the state identification data of the information of the plurality of target files to be registered from an initial state to an occupied state, and locking the target files to be registered in the occupied state; and generating a multithreaded file registration result according to the basic parameters and the target file to be registered after locking. .
In a second aspect, an embodiment of the present invention further provides a multithreaded file registration apparatus, including: the parameter module is used for acquiring basic parameters of the files to be registered, a memory file number threshold value and a registration thread number value; the scanning module is used for scanning a file list of a target position by a single thread according to the memory file number threshold value and determining target file information to be registered according to the basic parameters; the sending module is used for sending the information of the target file to be registered to the memory database so that the memory database generates state mark data of the target file to be registered, and the state mark data is an initial state at the moment; the registration module is used for generating a multithreaded file registration result according to the target file information to be registered, the state flag data, the registration thread number value and the basic parameter;
the scanning module is specifically configured to: calculating the number of files to be registered in a memory database according to the target file information to be registered; if the number of the files to be registered is greater than or equal to the memory file number threshold, stopping scanning the file list at the target position by using a single thread; if the number of the files to be registered is smaller than the number threshold value of the memory files, scanning a file list at a target position by using a single thread;
the registration module is specifically configured to: acquiring information of a plurality of target files to be registered in a memory database according to the number of the registration threads; modifying the state identification data of the information of the plurality of target files to be registered from an initial state to an occupied state, and locking the target files to be registered in the occupied state; and generating a multithreaded file registration result according to the basic parameters and the target file to be registered after locking.
In a third aspect, an embodiment of the present invention further provides a computer device, including a memory, and a processor, where the memory stores a computer program that can run on the processor, and the processor implements the multithreaded file registration method when executing the computer program.
In a fourth aspect, embodiments of the present invention also provide a computer readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform the above-described multi-threaded file registration method.
The embodiment of the invention has the following beneficial effects: the embodiment of the invention provides a multithread file registration scheme, which can determine target file information to be registered when a single thread is utilized to scan a file list at a target position, can control a scanning thread through a memory file number threshold value, avoid accumulating a large amount of information, generate state mark data of the target file to be registered when the target file information to be registered is sent to a memory database, generate a multithread file registration result according to the target file information to be registered, the state mark data, a registration thread number value and the basic parameter, and control a process of concurrent registration of multiple files through the state mark data and the registration thread number value so as to ensure that the same file cannot be repeatedly processed. The embodiment of the invention can improve the efficiency of file registration.
Additional features and advantages of the invention will be set forth in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. The objectives and other advantages of the invention will be realized and attained by the structure particularly pointed out in the written description and claims hereof as well as the appended drawings.
In order to make the above objects, features and advantages of the present invention more comprehensible, preferred embodiments accompanied with figures are described in detail below.
Drawings
In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are needed in the description of the embodiments or the prior art will be briefly described, and it is obvious that the drawings in the description below are some embodiments of the present invention, and other drawings can be obtained according to the drawings without inventive effort for a person skilled in the art.
FIG. 1 is a flowchart of a method for registering multithreaded files according to an embodiment of the present invention;
FIG. 2 is a block diagram of a multithreaded file registration apparatus according to an embodiment of the present invention;
FIG. 3 is a block diagram of another multi-threaded file registration apparatus according to an embodiment of the present invention;
fig. 4 is a block diagram of a computer device according to an embodiment of the present invention.
Detailed Description
For the purpose of making the objects, technical solutions and advantages of the embodiments of the present invention more apparent, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings, and it is apparent that the described embodiments are some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
At present, in order to improve the file registration efficiency, a common implementation scheme is to only configure a registration multithreading pool, and control the concurrent situation of multiple files through a database.
Based on the above, the method and the device for registering the multithread file provided by the embodiment of the invention comprise a single-thread scanning function, wherein the registrable file is put into a memory, concurrency is controlled according to the state of 0 or 1, and the maximum number of files is set, so that the memory is ensured not to accumulate a large number of records; not only database connection is reduced, but also the speed of acquiring files by the registration thread is improved.
For the convenience of understanding the present embodiment, a detailed description will be first provided of a multithreaded file registration method disclosed in the present embodiment.
The embodiment of the invention provides a multithreading file registration method, which is shown in a flow chart of the multithreading file registration method in FIG. 1, and comprises the following steps:
step S102, obtaining basic parameters of files to be registered, a memory file number threshold value and a registered thread number value.
In the embodiment of the invention, the basic parameters at least comprise file name parameters, coding mode parameters, separator parameters, control information line number parameters and the like. The basic parameters can be used for determining information such as a source position where the file to be registered is located, a final position where the file to be registered is to be moved, and the like, and can also be used for determining information such as a display effect of the file to be registered. The basic parameters of the files to be registered can be manually configured in the configuration files of the database in advance. The memory file number threshold may be used to determine the number of files to be registered that may be stored in the memory. The registration thread number value may be used to determine the number of files to be registered that are concurrently registered. The memory file number threshold and the registered thread number value can be manually pre-configured in the configuration file of the memory.
It should be noted that if the total number of files and the size of the files processed each time are not changed greatly, the threshold value of the number of memory files and the number of registered threads need not to be reconfigured before the scheme is executed each time, and the method can be configured when being used for the first time, and the subsequent modification is not needed.
Step S104, scanning a file list of the target position by using a single thread according to the memory file number threshold value, and determining target file information to be registered according to the basic parameters.
In the embodiment of the invention, after parameter configuration is completed, a script for running a main flow is called, and file scanning and file registering functions are started. The file scan is a single thread. And continuously scanning a file list at a target position by using a single thread, filtering out files which are being transmitted, determining files which need to be registered in the file list at the target position according to basic parameters, namely target files to be registered, and taking file names and the like corresponding to the target files to be registered for determining the information of the target files to be registered and the attribute thereof as target file information to be registered.
Step S106, the information of the target file to be registered is sent to the memory database, so that the memory database generates state flag data of the target file to be registered.
In the embodiment of the invention, each target file to be registered can be determined, the target file to be registered is sent to the memory database, and the memory database sets the state flag data of each target file to be registered as an initial state when storing the target file to be registered.
Specifically, for a certain target file to be registered information, a key value may be set as a file name of the target file to be registered in the memory database, value is state flag data, and an initial state of value may be set to "0".
It should be noted that, in the embodiment of the present invention, a map array may be defined in a memory, and this map array may be used as a memory database, and may be placed in the memory by continuously scanning a registrable file list through a single thread, and the memory size may be set by using a memory file number threshold.
Step S108, generating a multithreaded file registration result according to the target file information to be registered, the state flag data, the registration thread number value and the basic parameters.
In the embodiment of the invention, the file registration is a multi-thread, and the number of threads which are concurrently registered is determined according to the number of the registered threads, so that a plurality of files to be registered are acquired according to the target file information to be registered for registration, and in the registration process, whether the files to be registered are registered by the registered threads is determined according to the state mark data, so that the same file can be prevented from being repeatedly processed. According to the basic parameters, the final position of the file to be registered of the moving target can be determined, the file to be registered of the moving target is determined according to the final position determined by the basic parameters, the catalog of the file to be registered of the target is changed, and the file registration is completed with higher efficiency.
The embodiment of the invention provides a multithread file registration scheme, which can determine target file information to be registered when a single thread is utilized to scan a file list at a target position, can control a scanning thread through a memory file number threshold value, avoid accumulating a large amount of information, generate state mark data of the target file to be registered when the target file information to be registered is sent to a memory database, generate a multithread file registration result according to the target file information to be registered, the state mark data, a registration thread number value and the basic parameter, and control a process of concurrent registration of multiple files through the state mark data and the registration thread number value so as to ensure that the same file cannot be repeatedly processed. The embodiment of the invention can improve the efficiency of file registration.
Considering that the file scanning speed is generally the file registration speed, in order to facilitate controlling the file scanning progress, the method for scanning the file list under the target position by using a single thread according to the threshold number of memory files can be performed according to the following steps:
calculating the number of files to be registered in the memory database according to the target file information to be registered; if the number of the files to be registered is greater than or equal to the number threshold value of the memory files, stopping scanning the file list at the target position by using a single thread; if the number of files to be registered is smaller than the number threshold of the memory files, a single thread is utilized to scan the file list at the target position.
In the embodiment of the invention, the information of the target files to be registered is stored in the memory database, the number of the files to be registered in the memory database can be calculated according to the information of the target files to be registered, the number of the files to be registered in the memory database is compared with the memory file number threshold, if the number of the files to be registered is smaller than the memory file number threshold, the scanning is continuously performed or started, and if the number of the files to be registered is greater than or equal to the memory file number threshold, the scanning thread is dormant, so that the registration thread can register the scanned target files to be registered in time, and resources are fully utilized.
In order to reduce the repetition of the target file to be registered in the memory database, thereby reducing repeated registration processing, improving processing efficiency and saving resources, the sending of the target file to be registered to the memory database can be performed according to the following steps:
judging whether the target file information to be registered exists in the memory database; if yes, skipping the information of the target file to be registered; if not, the target file information to be registered is sent to the memory database.
In the embodiment of the invention, if the target file information to be registered exists in the memory database, the target file information to be registered is not repeatedly stored in the memory database, and the current target file information to be registered is not processed and skipped. If the current target file information to be registered does not exist in the memory database, the current target file information to be registered is sent to the memory database for storage.
In order to ensure that the same file cannot be repeatedly processed, database connection is reduced, loading efficiency is improved, and state mark data comprise an initial state and an occupied state; generating a multithreaded file registration result according to the target file information to be registered, the state flag data, the registration thread number value and the basic parameters, wherein the multithreaded file registration result can be executed according to the following steps:
acquiring information of a plurality of target files to be registered in a memory database according to the number of the registration threads; modifying state identification data of the information of the plurality of target files to be registered from an initial state to an occupied state, and locking the target files to be registered in the occupied state; and generating a multithreaded file registration result according to the basic parameters and the target file to be registered after locking.
In the embodiment of the present invention, for example, if the number of the registration threads is N, N pieces of target file information to be registered in the memory database are obtained, and the state identification data of each piece of target file information to be registered is modified from an initial state to an occupied state, for example, if the initial state is represented by "0" and the occupied state is represented by "1", each piece of state identification data in the N pieces of target file information to be registered is changed from "0" to "1". And locking the target to-be-registered file in each occupied state to prevent other registration threads from accessing the target to-be-registered file again. And finally, sending the locked target file to be registered to a target database according to the basic parameters, changing the directory information of the locked target file to be registered, and finishing the multi-thread file registration to obtain a multi-thread file registration result.
In order to optimize the file registration results under different conditions, the multi-thread file registration results are generated according to the basic parameters and the target files to be registered after locking, and the multi-thread file registration results can be executed according to the following steps:
according to the basic parameters, the locked target file to be registered is sent to a first target database, and whether the sending is successful or not is judged; if so, changing the catalogue of the target file to be registered after locking into a first target catalogue, and deleting target file information to be registered corresponding to the target file to be registered after locking; if not, judging whether the locked target file to be registered exists in the first target database; if the target to be registered exists, the locked target to be registered file is sent to a second target database; and if the target to-be-registered file does not exist, sending the locked target to-be-registered file to a third target database.
In the embodiment of the invention, the position of the first target database corresponding to the locked target file to be registered and the corresponding first target directory can be determined according to the basic parameters. If the target to-be-registered file is successfully sent to the first target database, the first target database already has the target to-be-registered file, which indicates that the target to-be-registered file is successfully registered in the first target database. And then the same file is sent to the first target database, and the sending fails. If the locked target file to be registered is sent to the first target database and the sending is successful, the catalog of the target file to be registered is changed into the first target catalog, and the target file to be registered information corresponding to the target file in the memory database is deleted, so that the file registration of the thread is completed. If the locked target to-be-registered file is sent to the first target database and fails to be sent, the reason of the failure needs to be judged, if the target to-be-registered file exists in the first target database, the target to-be-registered file is proved to be successfully registered, repeated registration is not carried out, the target to-be-registered file is sent to the second target database, if the target to-be-registered file is not registered in the first target database and still fails to be inserted into the first target database, the target to-be-registered file possibly has a problem and needs to be sent to the third target database, and therefore the registered file in the third target database can be processed manually in time.
It should be noted that in the embodiment of the present invention, the oracle database may be used for the first target database, the second target database, and the third target database.
The scheme may also perform the following steps, considering that there may be a registration thread timeout connection:
acquiring a time length threshold; if the target registration thread with the connection time length being greater than or equal to the time length threshold exists, disconnecting the target registration thread, and modifying the state mark data corresponding to the target registration thread from the occupied state to the initial state.
In the embodiment of the present invention, the duration threshold may be set according to actual needs, which is not specifically limited in the embodiment of the present invention. Monitoring the connection time length of the registration thread, if a target registration thread with the connection time length being greater than or equal to the time length threshold exists, describing that an abnormality occurs in the processing process, disconnecting the abnormal thread, and modifying the state mark data corresponding to the target file to be registered, which is related to the abnormal thread, from an occupied state to an initial state, for example, changing the state mark data from '1' to '0', and waiting for being reprocessed next time, because the registration is not completed actually.
In the embodiment of the invention, technicians need to configure file parameters, set the size of a registered thread pool and the maximum number of files which can be stored in a memory. And evaluating the size of the thread pool, the size of the memory and the thread dormancy time according to the total number of the daily files and the number of the large files. Defining a map array of a ConcurrentHashMap structure in a memory according to configuration file information of a database parameter table, and setting the map size; then, continuously scanning the files under the source directory through a single thread to obtain a file list to be registered, adding the files into a map and initializing a state, and dormancy of the threads is performed after the map exceeds a specified size; the registration thread in the registration thread pool acquires a record from the map and marks the record at the same time, and the record is deleted after registration is completed. The method is described in one specific example below:
1. parameters of configuration files in the database mainly comprise file names, coding modes, separators and control information line numbers;
2. configuring the maximum number of memory files in the configuration file, registering the thread number of a thread pool, keeping the connection time and the like;
3. after parameter configuration is completed, calling a script for running a main flow, and starting file scanning and file registering functions:
1) The file scanning is a single thread, while circulates the file list under the endless scanning source directory, filter the file of the now that is being transmitted, and judge if the current file does not exist in the memory map and put into, the key value is the file name, value is the initial state 0; when the number of files in the memory exceeds the specified maximum number of files, the thread sleeps;
2) The file registration is a multi-thread, a thread is obtained from a registration thread pool, a certain record (value is 0) which can be registered is obtained from a memory map in a while circulation mode, and the value in the map is changed into 1 (synchronous lock) at the same time when the record is obtained; then reading the parameter table information of the database, inserting the record into the database, wherein the state is 1000 successfully registered, if the record is successfully inserted, moving the file to a target directory (directory to be loaded), if the record is unsuccessful, judging whether the record is an old file or a repeated file which fails to be registered last time according to whether the record is 1000 successfully inserted, updating the record of the database, moving the file to the target directory (directory to be loaded), and if the record is repeated, directly moving the file to the target directory (directory to be loaded); after the steps are completed, deleting the record in the memory map; if an exception occurs in the processing process, the record value in the internal map is restored to be 0, and the processing is waited for being reprocessed next time.
The invention provides a multithread file registration method and device, the method divides the process of obtaining the registrable file and the file registration into two independent processes, the first process uses the single thread to scan the source catalogue continuously, the registrable file is obtained and then put into the memory; the second process uses multithreading to continuously fetch files from memory to complete registration. The single-thread scanning module reads the file from the source catalog and puts the file into the memory; the multithreading registration module reads the memory record, invokes the parameter module to acquire related parameters, records file information to the oracle database, moves the file from the source directory to the target directory, and deletes the memory record after the completion. The method realizes multi-file concurrent processing by using single-thread scanning and multi-thread registration, reduces database connection and improves file registration efficiency while ensuring that the same file cannot be repeatedly processed.
The embodiment of the invention also provides a multithreaded file registration device, referring to a structural block diagram of the multithreaded file registration device shown in fig. 2, the device comprises:
the parameter module 71 is configured to obtain a basic parameter of a file to be registered, a memory file number threshold value, and a registered thread number value; the scanning module 72 is configured to scan the file list at the target location by using a single thread according to the number threshold of the memory files, and determine the target file information to be registered according to the basic parameters; a sending module 73, configured to send the information of the target file to be registered to the memory database, so that the memory database generates status flag data of the target file to be registered; the registration module 74 is configured to generate a multithreaded file registration result according to the target file information to be registered, the status flag data, the registration thread number value and the basic parameter.
In one embodiment, the scanning module is specifically configured to: calculating the number of files to be registered in the memory database according to the target file information to be registered; if the number of the files to be registered is greater than or equal to the number threshold value of the memory files, stopping scanning the file list at the target position by using a single thread; if the number of files to be registered is smaller than the number threshold of the memory files, a single thread is utilized to scan the file list at the target position.
In one embodiment, the sending module is specifically configured to: judging whether the target file information to be registered exists in the memory database; if yes, skipping the information of the target file to be registered; if not, the target file information to be registered is sent to the memory database.
In one embodiment, the registration module is specifically configured to: acquiring information of a plurality of target files to be registered in a memory database according to the number of the registration threads; modifying state identification data of the information of the plurality of target files to be registered from an initial state to an occupied state, and locking the target files to be registered in the occupied state; and generating a multithreaded file registration result according to the basic parameters and the target file to be registered after locking.
In one embodiment, the registration module is specifically configured to: according to the basic parameters, the locked target file to be registered is sent to a first target database, and whether the sending is successful or not is judged; if so, changing the catalogue of the target file to be registered after locking into a first target catalogue, and deleting target file information to be registered corresponding to the target file to be registered after locking; if not, judging whether the locked target file to be registered exists in the first target database; if the target to be registered exists, the locked target to be registered file is sent to a second target database; and if the target to-be-registered file does not exist, sending the locked target to-be-registered file to a third target database.
In one embodiment, referring to another multi-threaded file registration apparatus block diagram of FIG. 3, the apparatus further includes a duration module 75 for: acquiring a time length threshold; if the target registration thread with the connection time length being greater than or equal to the time length threshold exists, disconnecting the target registration thread, and modifying the state mark data corresponding to the target registration thread from the occupied state to the initial state.
The embodiment of the present invention further provides a computer device, referring to the schematic block diagram of the structure of the computer device shown in fig. 4, where the computer device includes a memory 81 and a processor 82, and the memory stores a computer program that can be run on the processor, and when the processor executes the computer program, the processor implements the steps of any of the methods described above.
It will be clearly understood by those skilled in the art that, for convenience and brevity of description, the specific working process of the computer device described above may refer to the corresponding process in the foregoing method embodiment, which is not repeated herein.
Embodiments of the present invention also provide a computer readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform the steps of any of the methods described above.
It will be appreciated by those skilled in the art that embodiments of the present invention may be provided as a method, system, or computer program product. Accordingly, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Furthermore, the present invention may take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, and the like) having computer-usable program code embodied therein.
The present invention is described with reference to flowchart illustrations and/or block diagrams of methods and computer program products according to embodiments of the invention. It will be understood that each flow and/or block of the flowchart illustrations and/or block diagrams, and combinations of flows and/or blocks in the flowchart illustrations and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
Finally, it should be noted that: the above examples are only specific embodiments of the present invention, and are not intended to limit the scope of the present invention, but it should be understood by those skilled in the art that the present invention is not limited thereto, and that the present invention is described in detail with reference to the foregoing examples: any person skilled in the art may modify or easily conceive of the technical solution described in the foregoing embodiments, or perform equivalent substitution of some of the technical features, while remaining within the technical scope of the present disclosure; such modifications, changes or substitutions do not depart from the spirit and scope of the technical solutions of the embodiments of the present invention, and are intended to be included in the scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the protection scope of the claims.
Claims (7)
1. A method for multi-threaded file registration, comprising:
acquiring basic parameters of a file to be registered, a memory file number threshold value and a registration thread number value;
scanning a file list of a target position by using a single thread according to the memory file number threshold value, and determining target file information to be registered according to the basic parameters;
the target file information to be registered is sent to a memory database, so that the memory database generates state mark data of the target file to be registered, and the state mark data is an initial state at the moment;
generating a multithreaded file registration result according to the target file information to be registered, the state flag data, the registration thread number value and the basic parameter;
the method for scanning the file list under the target position by using a single thread according to the memory file number threshold comprises the following steps:
calculating the number of files to be registered in a memory database according to the target file information to be registered;
if the number of the files to be registered is greater than or equal to the memory file number threshold, stopping scanning the file list at the target position by using a single thread;
if the number of the files to be registered is smaller than the number threshold value of the memory files, scanning a file list at a target position by using a single thread;
generating a multithreaded file registration result according to the target file information to be registered, the state flag data, the registration thread number value and the basic parameter, wherein the multithreaded file registration result comprises:
acquiring information of a plurality of target files to be registered in a memory database according to the number of the registration threads;
modifying the state identification data of the information of the plurality of target files to be registered from an initial state to an occupied state, and locking the target files to be registered in the occupied state;
and generating a multithreaded file registration result according to the basic parameters and the target file to be registered after locking.
2. The method of claim 1, wherein sending the target file to be registered information to an in-memory database comprises:
judging whether the target file information to be registered exists in the memory database;
if yes, skipping the file information to be registered of the target;
and if not, sending the target file information to be registered to a memory database.
3. The method of claim 1, wherein generating a multithreaded file registration result based on the base parameters and the locked target file to be registered comprises:
according to the basic parameters, the target files to be registered after locking are sent to a first target database, and whether the files to be registered are sent successfully or not is judged;
if so, changing the catalogue of the target file to be registered after locking into a first target catalogue, and deleting target file information to be registered corresponding to the target file to be registered after locking;
if not, judging whether the locked target file to be registered exists in the first target database;
if the target to be registered exists, the locked target to be registered file is sent to a second target database;
and if the target to-be-registered file does not exist, sending the target to-be-registered file after locking to a third target database.
4. The method as recited in claim 1, further comprising:
acquiring a time length threshold;
if a target registration thread with the connection time length being greater than or equal to the time length threshold exists, disconnecting the target registration thread, and modifying state mark data corresponding to the target registration thread from an occupied state to an initial state.
5. A multi-threaded file registration apparatus, comprising:
the parameter module is used for acquiring basic parameters of the files to be registered, a memory file number threshold value and a registration thread number value;
the scanning module is used for scanning a file list of a target position by a single thread according to the memory file number threshold value and determining target file information to be registered according to the basic parameters;
the sending module is used for sending the information of the target file to be registered to the memory database so that the memory database generates state mark data of the target file to be registered, and the state mark data is an initial state at the moment;
the registration module is used for generating a multithreaded file registration result according to the target file information to be registered, the state flag data, the registration thread number value and the basic parameter;
the scanning module is specifically configured to:
calculating the number of files to be registered in a memory database according to the target file information to be registered;
if the number of the files to be registered is greater than or equal to the memory file number threshold, stopping scanning the file list at the target position by using a single thread;
if the number of the files to be registered is smaller than the number threshold value of the memory files, scanning a file list at a target position by using a single thread;
the registration module is specifically configured to:
acquiring information of a plurality of target files to be registered in a memory database according to the number of the registration threads;
modifying the state identification data of the information of the plurality of target files to be registered from an initial state to an occupied state, and locking the target files to be registered in the occupied state;
and generating a multithreaded file registration result according to the basic parameters and the target file to be registered after locking.
6. A computer device comprising a memory, a processor, the memory having stored therein a computer program executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the method of any of the preceding claims 1 to 4.
7. A computer readable medium having non-volatile program code executable by a processor, the program code causing the processor to perform the method of any one of the preceding claims 1 to 4.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010494421.XA CN111680091B (en) | 2020-06-03 | 2020-06-03 | Multithread file registration method and device |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202010494421.XA CN111680091B (en) | 2020-06-03 | 2020-06-03 | Multithread file registration method and device |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111680091A CN111680091A (en) | 2020-09-18 |
| CN111680091B true CN111680091B (en) | 2023-04-28 |
Family
ID=72453046
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202010494421.XA Active CN111680091B (en) | 2020-06-03 | 2020-06-03 | Multithread file registration method and device |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111680091B (en) |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102314572A (en) * | 2011-10-14 | 2012-01-11 | 迈普通信技术股份有限公司 | Registration information file generation method and device, and use authorization method of application software |
| CN103841196A (en) * | 2014-03-07 | 2014-06-04 | 长沙裕邦软件开发有限公司 | File uploading system and method based on multithreading |
| CN104834506A (en) * | 2015-05-15 | 2015-08-12 | 北京北信源软件股份有限公司 | Method for processing business application through multiple threads |
| CN109190333A (en) * | 2018-08-22 | 2019-01-11 | 山东超越数控电子股份有限公司 | A kind of high intensity method for protecting software |
Family Cites Families (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP2011095997A (en) * | 2009-10-29 | 2011-05-12 | Canon Inc | Information processing apparatus, computer program, and information processing method |
| US8836709B2 (en) * | 2011-08-18 | 2014-09-16 | International Business Machines Corporation | Vector register file caching of context data structure for maintaining state data in a multithreaded image processing pipeline |
-
2020
- 2020-06-03 CN CN202010494421.XA patent/CN111680091B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102314572A (en) * | 2011-10-14 | 2012-01-11 | 迈普通信技术股份有限公司 | Registration information file generation method and device, and use authorization method of application software |
| CN103841196A (en) * | 2014-03-07 | 2014-06-04 | 长沙裕邦软件开发有限公司 | File uploading system and method based on multithreading |
| CN104834506A (en) * | 2015-05-15 | 2015-08-12 | 北京北信源软件股份有限公司 | Method for processing business application through multiple threads |
| CN109190333A (en) * | 2018-08-22 | 2019-01-11 | 山东超越数控电子股份有限公司 | A kind of high intensity method for protecting software |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111680091A (en) | 2020-09-18 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US11461195B2 (en) | Query fault processing method and processing apparatus | |
| US10346262B2 (en) | Job management apparatus, job management method and computer readable medium to generate and execute a retry job when an error occurs during a job step | |
| EP2187695A1 (en) | Method, device and system for realizing task in cluster environment | |
| CN112114903B (en) | File loading method and device | |
| WO2018149396A1 (en) | Service flow processing method and apparatus, readable storage medium and terminal device | |
| WO2019148657A1 (en) | Method for testing associated environments, electronic device and computer readable storage medium | |
| CN111680091B (en) | Multithread file registration method and device | |
| CN112328602B (en) | Method, device and equipment for writing data into Kafka | |
| CN111708618A (en) | Processing method and device based on Java multithreading | |
| US8205190B2 (en) | Data processing system and method | |
| CN111984662B (en) | Method and device for updating databases in batches | |
| CN113900952A (en) | Test case execution method and device, terminal equipment and readable storage medium | |
| CN106484776B (en) | Multithreading-based data processing method and device | |
| CN113792026A (en) | Deployment method and device of database script and computer readable storage medium | |
| CN115658182B (en) | Method for automatically reloading XML mapping file of MyBatis | |
| CN107656868B (en) | Debugging method and system for acquiring thread name by using thread private data | |
| CN111143277B (en) | Data processing method and device and electronic equipment | |
| JP2004318763A (en) | Job network management system and program | |
| CN111258850B (en) | Method and device for updating software information based on Linux system | |
| CN116521404A (en) | Dependency diagnosis method, dependency diagnosis device, electronic equipment and storage medium | |
| JP2006293582A (en) | Automatic job rerun system | |
| CN115599861A (en) | Data verification method, storage medium and equipment for database synchronization process | |
| JP2003345747A (en) | Method and device for managing processing performance, program and recording medium with program recorded thereon | |
| CN115562842A (en) | Process interaction method and system based on Activiti | |
| CN116107720A (en) | Job processing method, device and computer readable medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |