CN112565366B

CN112565366B - Distributed file importing method, device, equipment and storage medium

Info

Publication number: CN112565366B
Application number: CN202011353126.9A
Authority: CN
Inventors: 陈赞锦
Original assignee: Ping An Puhui Enterprise Management Co Ltd
Current assignee: Ping An Puhui Enterprise Management Co Ltd
Priority date: 2020-11-27
Filing date: 2020-11-27
Publication date: 2022-11-08
Anticipated expiration: 2040-11-27
Also published as: CN112565366A

Abstract

The invention relates to the field of big data, and discloses a distributed file importing method, a device, equipment and a storage medium, which are used for preventing the problem of content overflow, solving the problem of virus invading files and improving the work efficiency of file importing. The distributed file importing method comprises the following steps: acquiring a plurality of files to be uploaded, a plurality of file uploading requests and a plurality of service module groups; responding to a plurality of file uploading requests to obtain a plurality of files to be uploaded after viruses are scanned; triggering a plurality of preset task threads and a plurality of thread task locks, and distributing each main key to the corresponding task thread; acquiring a corresponding file to be uploaded after virus scanning according to each main key, and processing a plurality of files to be uploaded after virus scanning by combining a target task thread with a corresponding service module group to generate a plurality of files to be imported; and importing a plurality of files to be imported into a database or a downstream system. In addition, the invention also relates to a block chain technology, and a plurality of files to be uploaded can be stored in the block chain.

Description

Distributed file import method, device, equipment and storage medium

Technical Field

The present invention relates to the field of data processing technologies, and in particular, to a distributed file importing method, apparatus, device, and storage medium.

Background

With the development of science and technology, big data is widely applied to various fields, and importing data also becomes an important task. At present, data is imported by adopting a file import method, wherein the file import is a common service function and mainly is a processing mode for interacting a system and a user, acquiring external data and converting a data format.

At present, a file is imported, which mainly uses a front-end page to upload a file, a back-end service to receive and read the file, check the data, generate a data object, and process the data.

Disclosure of Invention

The invention provides a distributed file importing method, a distributed file importing device, distributed file importing equipment and a storage medium, which are used for preventing the problem of content overflow, solving the problem of virus invading files and improving the work efficiency of file importing.

The first aspect of the present invention provides a distributed file importing method, including: acquiring a plurality of files to be uploaded and a plurality of file uploading requests, and acquiring a plurality of service module groups configured by a terminal, wherein the plurality of files to be uploaded are a plurality of files after check bytes of the terminal, and one file uploading request corresponds to one file to be uploaded; when the terminal acquires a plurality of request character strings and a plurality of main keys from the plurality of files to be uploaded, the terminal responds to the plurality of file uploading requests to acquire a plurality of files to be uploaded after virus scanning, and stores the plurality of files to be uploaded after virus scanning in a file storage service module; receiving the multiple request character strings and the multiple main keys sent by a terminal, triggering multiple preset task threads and multiple thread task locks through the multiple request character strings, and distributing each main key to the corresponding task thread, wherein the multiple thread task locks are respectively used for locking the multiple preset task threads; acquiring a file to be uploaded after corresponding virus scanning in each task thread according to each main key, processing the file to be uploaded after virus scanning corresponding to the target main key through the target task thread in combination with the corresponding service module group, and generating a plurality of files to be imported; and importing the files to be imported into a database or a downstream system, wherein the downstream system is a system in butt joint with a server.

Optionally, in a first implementation manner of the first aspect of the present invention, when the terminal obtains multiple request character strings and multiple primary keys from multiple files to be uploaded, the terminal responds to the multiple file uploading requests to obtain multiple files to be uploaded after virus scanning, and stores the multiple files to be uploaded after virus scanning in the file storage service module includes: when the terminal acquires the request character string and the plurality of main keys from the file to be uploaded, a file storage service module is adopted to perform virus scanning on the plurality of files to be uploaded to obtain a plurality of scanning results; and obtaining a plurality of files to be uploaded after virus scanning according to the plurality of scanning results, and storing the plurality of files to be uploaded after virus scanning in the file storage service module.

Optionally, in a second implementation manner of the first aspect of the present invention, the obtaining, according to the multiple scanning results, multiple files to be uploaded after viruses are scanned, and storing, in the file storage service module, the multiple files to be uploaded after viruses are scanned includes: judging whether each scanning result comprises a virus file or not; if the target scanning result comprises a virus file, terminating the uploading process of the corresponding file to be uploaded; if the target scanning result does not include the virus file, determining the corresponding file to be uploaded as the file to be uploaded after virus scanning, obtaining a plurality of files to be uploaded after virus scanning, and storing the plurality of files to be uploaded after virus scanning in the file storage service module.

Optionally, in a third implementation manner of the first aspect of the present invention, the acquiring, in each task thread, a file to be uploaded after corresponding virus scanning according to each primary key, and processing, by combining the target task thread and the corresponding service module group, the file to be uploaded after virus scanning corresponding to the target primary key, to generate a plurality of files to be imported includes: in one task thread, calling a corresponding file to be uploaded after the target virus is scanned from the file storage service module through a back-end service module based on a target main key; judging whether the file to be uploaded after the target scans the viruses conforms to a preset processing rule or not; if the file to be uploaded after the target virus scanning meets the preset processing rule, the file to be uploaded after the target virus scanning is dispatched to the corresponding target service instance to be processed by combining with the corresponding target service module group, and a file to be imported is generated; and in other task threads, generating other files to be imported, and integrating the file to be imported and the other files to be imported to obtain a plurality of files to be imported.

Optionally, in a fourth implementation manner of the first aspect of the present invention, the determining whether the file to be uploaded after the target virus scanning meets a preset processing rule includes: reading byte data of the file to be uploaded after the target scans the viruses to obtain file byte data; judging whether the byte data of the file is larger than a byte threshold value or not; if the byte data of the file is larger than the byte threshold, judging that the file to be uploaded after the target virus scanning does not accord with a preset processing rule; and if the byte data of the file is less than or equal to the byte threshold, judging that the file to be uploaded after the target virus scanning meets a preset processing rule.

Optionally, in a fifth implementation manner of the first aspect of the present invention, if the file to be uploaded after the target virus scanning meets a preset processing rule, the sending the file to be uploaded after the target virus scanning to a corresponding target service instance for processing by combining a corresponding target service module group, and generating a file to be imported, includes: if the file to be uploaded after the target virus scanning meets the preset processing rule, extracting file reflection according to the file to be uploaded after the target virus scanning through a preset open source framework, wherein the file reflection is the class reflection of a JAVA program; determining a corresponding target service instance in a plurality of preset service instances according to the file reflection; reading the file data format of the file to be uploaded after the target scans viruses, and verifying the file data format by adopting a preset universal format verifier to obtain the verified file to be uploaded; extracting a check field from the checked file to be uploaded, extracting corresponding processing logic from the plurality of service module groups according to the check field, and processing the checked file to be uploaded through the processing logic to obtain a logically processed file to be uploaded; and constructing an object by adopting the target service instance based on the file to be uploaded after the logic processing to generate a file to be imported.

Optionally, in a sixth implementation manner of the first aspect of the present invention, the importing the multiple files to be imported into a database or a downstream system, where the downstream system is a system that interfaces with a server, and includes: when the files to be imported are imported into a database, analyzing the corresponding service instances to obtain corresponding service instance parents; embedding each file to be imported into a corresponding target service instance parent class, wherein the corresponding target service subclass inherits the attribute and the logic execution method of the target service instance parent class; reading a corresponding file to be imported through a target service subclass, and storing the corresponding file to be imported into a database; and when the files to be imported are imported into a downstream system, transmitting the files to be imported to a data interface corresponding to the downstream system, wherein the downstream system is a system in butt joint with a server.

A second aspect of the present invention provides a distributed file importing apparatus, including: the acquisition module is used for acquiring a plurality of files to be uploaded and a plurality of file uploading requests, and acquiring a plurality of service module groups configured by the terminal, wherein the plurality of files to be uploaded are a plurality of files after bytes are checked by the terminal, and one file uploading request corresponds to one file to be uploaded; the response module is used for responding to the file uploading requests to obtain a plurality of files to be uploaded after viruses are scanned and storing the files to be uploaded after the viruses are scanned in the file storage service module when the terminal obtains a plurality of request character strings and a plurality of main keys from the files to be uploaded; the distribution module is used for receiving the request character strings and the main keys sent by the terminal, triggering a plurality of preset task threads and a plurality of thread task locks through the request character strings, and distributing each main key to the corresponding task thread, wherein the thread task locks are respectively used for locking the preset task threads; the processing module is used for acquiring the corresponding file to be uploaded after virus scanning in each task thread according to each main key, processing the file to be uploaded after virus scanning corresponding to the target main key through the target task thread in combination with the corresponding service module group, and generating a plurality of files to be imported; and the import module is used for importing the files to be imported into a database or a downstream system, and the downstream system is a system in butt joint with the server.

Optionally, in a first implementation manner of the second aspect of the present invention, the response module includes: the scanning unit is used for scanning viruses of the files to be uploaded by adopting the file storage service module to obtain a plurality of scanning results when the terminal acquires the request character string and the main keys from the files to be uploaded; and the uploading unit is used for obtaining a plurality of files to be uploaded after virus scanning according to the plurality of scanning results and storing the plurality of files to be uploaded after virus scanning in the file storage service module.

Optionally, in a second implementation manner of the second aspect of the present invention, the uploading unit may further be specifically configured to: judging whether each scanning result comprises a virus file or not; if the target scanning result comprises a virus file, terminating the uploading process of the corresponding file to be uploaded; if the target scanning result does not comprise the virus file, determining the corresponding file to be uploaded as the file to be uploaded after virus scanning, obtaining a plurality of files to be uploaded after virus scanning, and storing the plurality of files to be uploaded after virus scanning in the file storage service module.

Optionally, in a third implementation manner of the second aspect of the present invention, the processing module includes: the calling unit is used for calling a file to be uploaded after a corresponding target scans viruses from the file storage service module through the back-end service module based on a target main key in one task thread; the judging unit is used for judging whether the file to be uploaded after the target scans the viruses conforms to a preset processing rule or not; the processing unit is used for sending the file to be uploaded after the target virus scanning to a corresponding target service instance for processing by combining a corresponding target service module group to generate a file to be imported if the file to be uploaded after the target virus scanning meets a preset processing rule; and the integration unit is used for generating other files to be imported in other task threads, and integrating the file to be imported and the other files to be imported to obtain a plurality of files to be imported.

Optionally, in a fourth implementation manner of the second aspect of the present invention, the determining unit may be further specifically configured to: reading byte data of the file to be uploaded after the target scans the viruses to obtain file byte data; judging whether the byte data of the file is larger than a byte threshold value; if the byte data of the file is larger than the byte threshold, judging that the file to be uploaded after the target virus scanning does not accord with a preset processing rule; and if the byte data of the file is less than or equal to the byte threshold, judging that the file to be uploaded after the target virus scanning meets a preset processing rule.

Optionally, in a fifth implementation manner of the second aspect of the present invention, the processing unit may further be specifically configured to: if the file to be uploaded after the target virus scanning meets the preset processing rule, extracting file reflection according to the file to be uploaded after the target virus scanning through a preset open source framework, wherein the file reflection is the class reflection of a JAVA program; determining a corresponding target service instance in a plurality of preset service instances according to the file reflection; reading the file data format of the file to be uploaded after the target scans viruses, and verifying the file data format by adopting a preset universal format verifier to obtain the verified file to be uploaded; extracting a check field from the checked file to be uploaded, extracting corresponding processing logic from the plurality of service module groups according to the check field, and processing the checked file to be uploaded through the processing logic to obtain a logically processed file to be uploaded; and adopting the target service instance to construct an object based on the file to be uploaded after the logic processing to generate a file to be imported.

Optionally, in a sixth implementation manner of the second aspect of the present invention, the import module may be further specifically configured to: when the files to be imported are imported into a database, analyzing the corresponding service instances to obtain corresponding service instance parents; embedding each file to be imported into a corresponding target service instance parent class, wherein the corresponding target service subclass inherits the attribute and the logic execution method of the target service instance parent class; reading a corresponding file to be imported through a target service subclass, and storing the corresponding file to be imported into a database; and when the files to be imported are imported into a downstream system, transmitting the files to be imported to a data interface corresponding to the downstream system, wherein the downstream system is a system in butt joint with a server.

A third aspect of the present invention provides a distributed file importing apparatus, including: a memory and at least one processor, the memory having instructions stored therein; the at least one processor calls the instructions in the memory to cause the distributed file import device to execute the distributed file import method described above.

A fourth aspect of the present invention provides a computer-readable storage medium having stored therein instructions, which, when run on a computer, cause the computer to execute the above-described distributed file import method.

According to the technical scheme provided by the invention, a plurality of files to be uploaded and a plurality of file uploading requests are obtained, and a plurality of service module groups configured by a terminal are obtained, wherein the plurality of files to be uploaded are a plurality of files after check bytes of the terminal, and one file uploading request corresponds to one file to be uploaded; when the terminal acquires a plurality of request character strings and a plurality of main keys from the plurality of files to be uploaded, the terminal responds to the plurality of file uploading requests to acquire a plurality of files to be uploaded after virus scanning, and stores the plurality of files to be uploaded after virus scanning in a file storage service module; receiving the multiple request character strings and the multiple main keys sent by a terminal, triggering multiple preset task threads and multiple thread task locks through the multiple request character strings, and distributing each main key to the corresponding task thread, wherein the multiple thread task locks are respectively used for locking the multiple preset task threads; acquiring a file to be uploaded after corresponding virus scanning in each task thread according to each main key, processing the file to be uploaded after virus scanning corresponding to the target main key through the target task thread in combination with the corresponding service module group, and generating a plurality of files to be imported; and importing the files to be imported into a database or a downstream system, wherein the downstream system is a system in butt joint with a server. In the embodiment of the invention, the size of the file is checked to prevent the problem of content overflow, viruses are scanned to solve the problem of file invasion, and the processing logic is obtained and the file is processed in the corresponding task thread by combining the service module group, so that the working efficiency of processing the file is improved, and the working efficiency of importing the file is improved.

Drawings

FIG. 1 is a schematic diagram of a network architecture according to an embodiment of the present invention;

FIG. 2 is a diagram of an embodiment of a distributed file import method according to an embodiment of the present invention;

FIG. 3 is a diagram of another embodiment of a distributed file import method according to an embodiment of the present invention;

fig. 4 is a schematic diagram of an embodiment of a distributed file importing apparatus according to an embodiment of the present invention;

fig. 5 is a schematic diagram of another embodiment of a distributed file importing apparatus according to an embodiment of the present invention;

fig. 6 is a diagram of an embodiment of a distributed file import device in the embodiment of the present invention.

Detailed Description

The embodiment of the invention provides a distributed file importing method, a distributed file importing device, a distributed file importing equipment and a storage medium, which are used for preventing the problem of content overflow, solving the problem of file intrusion by viruses and improving the working efficiency of file importing.

The terms "first," "second," "third," "fourth," and the like in the description and in the claims, as well as in the drawings, if any, are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that the embodiments described herein may be practiced otherwise than as specifically illustrated or described herein. Furthermore, the terms "comprises," "comprising," or "having," and any variations thereof, are intended to cover non-exclusive inclusions, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

The embodiment of the invention can be applied to a network architecture as shown in fig. 1, wherein in the network architecture, an H5 page is a pure page system, is deployed in nginx and is mainly responsible for user interaction; the front-end system of the back-end service mainly has the function of a gateway and controls the authority of the user; the back-end service module is responsible for verifying and analyzing the service logic kitchenware and the files and other processing are completed in the module; the file storage server module is used for transferring file transmission, scanning viruses of the files and preventing the files such as the viruses from entering the network architecture. The front-end system of the back-end service acquires a file sent by the H5 page, then utilizes the file storage server to scan the file and store the file, and finally processes and analyzes the file through the back-end server module.

For convenience of understanding, a specific flow of an embodiment of the present invention is described below, with reference to fig. 2, an embodiment of a distributed file importing method according to an embodiment of the present invention includes:

201. acquiring a plurality of files to be uploaded and a plurality of file uploading requests, and acquiring a plurality of service module groups configured by a terminal, wherein the plurality of files to be uploaded are a plurality of files after check bytes of the terminal, and one file uploading request corresponds to one file to be uploaded;

the server obtains a plurality of files to be uploaded and a plurality of file uploading requests after bytes are checked by the terminal, wherein one file uploading request corresponds to one file to be uploaded, and in addition, the server also obtains a plurality of service module groups configured by the terminal. It should be emphasized that, in order to further ensure the privacy and security of the files to be uploaded, the files to be uploaded may also be stored in a node of a block chain.

It should be noted that the modules in the service module group may be multiplexed, and the service module group configured by the terminal is different according to the difference between the file to be uploaded and the file uploading request, and the service module group is used for configuring the processing logic and supporting file parsing.

The file size is checked on the file through a hypertext Markup language5.0 (Hypertext Markup 5.0, H5) page at the terminal to obtain a file to be uploaded, and the file with bytes larger than a byte threshold is prohibited from being uploaded, so that the condition of memory overflow caused when a server analyzes the file is prevented. Typically, the byte threshold is 200M, and files larger than 200M are prohibited from being uploaded to the server. The H5 page can also be assembled with a service module to obtain a service module group, and the service module is in a service level and can be understood as an uploading function in a certain menu of the page. And after the H5 page checks the file size and assembles the service module, transmitting the file to be uploaded, the file uploading request and the configured service module group to a server.

It is to be understood that the execution subject of the present invention may be a distributed file importing apparatus, and may also be a terminal or a server, which is not limited herein. The embodiment of the present invention is described by taking a server as an execution subject.

202. When the terminal acquires a plurality of request character strings and a plurality of main keys from a plurality of files to be uploaded, a plurality of file uploading requests are responded, a plurality of files to be uploaded after virus scanning are acquired, and the files to be uploaded after the virus scanning are stored in a file storage service module;

after the terminal is configured with the service module group, the terminal can also obtain a plurality of request character strings and a plurality of main keys, when the terminal obtains the plurality of request character strings and the plurality of main keys, the server responds to a plurality of file uploading requests, uploads the files to the file storage server module for virus scanning, obtains a plurality of files to be uploaded after virus scanning, and finally stores the plurality of files to be uploaded after virus scanning in the file storage server module.

It should be noted that the request character string is token, the character string is a token of the request, and after the user logs in the account for the first time at the terminal, the terminal can take the request character string to request data to log in, and the user name and the password do not need to be written again, so that the docking time between the terminal and the server is saved. The primary key is a FileKey, which refers to the only primary key of a file, and through this primary key, a corresponding file to be uploaded can be found in the file storage service module, and the FileKey is usually a 32-bit character string, for example, 144815e5d2ac4f129ba22a13c6741df2.

Because the security of a plurality of files to be uploaded cannot be guaranteed, the files need to be scanned, and the files carrying viruses are scanned, so that the files are prevented from being invaded by the viruses.

203. Receiving a plurality of request character strings and a plurality of main keys sent by a terminal, triggering a plurality of preset task threads and a plurality of thread task locks through the plurality of request character strings, and distributing each main key to the corresponding task thread, wherein the plurality of thread task locks are respectively used for locking the plurality of preset task threads;

the method comprises the steps that a server receives a plurality of request character strings and a plurality of main keys sent by a terminal, matches corresponding task threads based on each request character string, extracts task codes of the task threads, and determines thread task locks corresponding to target task threads in a plurality of thread task locks based on the task codes. In order to ensure the consistency of the file processing state, each main key is allocated to the corresponding task thread, the corresponding thread task lock is adopted to lock the target task thread at the moment, the distribution lock Redis or the database lock is adopted as the thread person task lock, and the problem of resource occupation can be solved through the thread task lock, so that only one thread of the file to be uploaded after the virus is scanned corresponding to one main key at the same time is ensured to be accessed.

For example, the request string is "144815e5d2ac4f129ba22a13c6741df2", the string is analyzed, the request field is "daily sales journal", the task thread a is determined according to the field, the task code extracted based on the task thread a is "floor _ sales _ report", the server determines the thread task lock B according to the task code, then the primary key corresponding to the request string is allocated to the task thread, and the task thread a is locked by the thread task lock B.

204. Acquiring a file to be uploaded after corresponding virus scanning in each task thread according to each main key, processing the file to be uploaded after virus scanning corresponding to the target main key through the target task thread in combination with the corresponding service module group, and generating a plurality of files to be imported;

in order to save the processing time for analyzing files and ensure the consistency of file processing states, the scheme adopts a distributed mode to process a plurality of files, a server obtains a file to be uploaded after corresponding virus scanning in each task thread according to each main key through a back-end service module, if the file to be uploaded corresponding to the main key comprises a virus file, the file to be uploaded after the virus scanning cannot be obtained from a file storage service module according to the main key, and then the file to be uploaded after each virus scanning in each task thread is analyzed to generate a plurality of files to be imported.

It should be noted that the primary key is used to identify data in the data table, and the primary key may be composed of one or more fields.

205. And importing a plurality of files to be imported into a database or a downstream system, wherein the downstream system is a system in butt joint with a server.

After obtaining a plurality of files to be imported, the server imports the plurality of files to be imported into a database or a downstream system, wherein the downstream system is a system which is interfaced with the server and can be a financial system, a traffic system, an educational system and the like.

In the embodiment of the invention, the size of the file is checked to prevent the problem of content overflow, viruses are scanned to solve the problem of file invasion, and the processing logic is obtained and the file is processed in the corresponding task thread by combining the service module group, so that the working efficiency of processing the file is improved, and the working efficiency of importing the file is improved.

Referring to fig. 3, another embodiment of a distributed file importing method according to an embodiment of the present invention includes:

301. acquiring a plurality of files to be uploaded and a plurality of file uploading requests, and acquiring a plurality of service module groups configured by a terminal, wherein the plurality of files to be uploaded are a plurality of files after check bytes of the terminal, and one file uploading request corresponds to one file to be uploaded;

The file size is verified on the terminal through the H5 page, the file to be uploaded is obtained, and the file with bytes larger than the byte threshold value is prohibited from being uploaded, so that the condition that the memory overflows when the server analyzes the file is prevented. Typically, the byte threshold is 200M, and files larger than 200M are prohibited from being uploaded to the server. The H5 page can also be assembled with a service module to obtain a service module group, and the service module is in a service level and can be understood as an uploading function in a certain menu of the page. And after the H5 page checks the file size and assembles the service module, transmitting the file to be uploaded, the file uploading request and the configured service module group to a server.

302. When the terminal acquires the request character string and the plurality of main keys from the file to be uploaded, a file storage service module is adopted to perform virus scanning on the plurality of files to be uploaded to obtain a plurality of scanning results;

after the terminal is configured with the service module group, the terminal can also obtain a plurality of request character strings and a plurality of main keys, and when the terminal obtains the plurality of request character strings and the plurality of main keys, the server responds to a plurality of file uploading requests and uploads files to the file storage server module for virus scanning, so that a plurality of scanning results are obtained. The file storage service module is a module with file storage service, one function is file transmission transfer, and the other function is file scanning to prevent illegal files such as viruses from entering the next module.

303. Obtaining a plurality of files to be uploaded after virus scanning according to a plurality of scanning results, and storing the plurality of files to be uploaded after virus scanning in a file storage service module;

and the server stores the files to be uploaded after a plurality of viruses are scanned in a file storage server module according to a plurality of scanning results.

Specifically, the server judges whether each scanning result comprises a virus file; if the target scanning result comprises a virus file, the server terminates the uploading process of the corresponding file to be uploaded; if the target scanning result does not include the virus file, the server determines the corresponding file to be uploaded as the file to be uploaded after virus scanning, obtains a plurality of files to be uploaded after virus scanning, and stores the plurality of files to be uploaded after virus scanning in the file storage service module.

The server judges whether each scanning result comprises a virus file, if the target scanning result comprises the virus file, the uploading process of the corresponding file to be uploaded is ended, and the file to be uploaded is prevented from entering the next module in the server; if the target scanning result does not include the virus file, the corresponding file to be uploaded is indicated to be a safe file, at this time, the file to be uploaded needs to be determined as the file to be uploaded after virus scanning, and the file to be uploaded after virus scanning needs to be stored in the file storage service module, so that other modules in the server can download the file to be uploaded after virus scanning from the file storage service module.

304. Receiving a plurality of request character strings and a plurality of main keys sent by a terminal, triggering a plurality of preset task threads and a plurality of thread task locks through the plurality of request character strings, and distributing each main key to the corresponding task thread, wherein the plurality of thread task locks are respectively used for locking the plurality of preset task threads;

the server receives a plurality of request character strings and a plurality of main keys sent by the terminal, matches corresponding task threads based on each request character string, extracts task codes of the task threads, and determines thread task locks corresponding to target task threads in a plurality of thread task locks based on the task codes. In order to ensure the consistency of the file processing state, each main key is allocated to the corresponding task thread, the corresponding thread task lock is adopted to lock the target task thread at the moment, the distribution lock Redis or the database lock is adopted as the thread person task lock, and the problem of resource occupation can be solved through the thread task lock, so that only one thread of the file to be uploaded after the virus is scanned corresponding to one main key at the same time is ensured to be accessed.

305. Acquiring a file to be uploaded after corresponding virus scanning in each task thread according to each main key, processing the file to be uploaded after virus scanning corresponding to the target main key through the target task thread in combination with the corresponding service module group, and generating a plurality of files to be imported;

in order to save the processing time for analyzing files and ensure the consistency of file processing states, the scheme adopts a distributed mode to process a plurality of files, a server obtains a corresponding file to be uploaded after scanning viruses according to each main key in each task thread through a back-end service module, if the file to be uploaded corresponding to the main key comprises the virus file, the file to be uploaded after scanning the viruses cannot be obtained from a file storage service module according to the main key, and then each file to be uploaded after scanning the viruses is analyzed in each task thread to generate a plurality of files to be imported.

Specifically, in one task thread, the server searches a corresponding file to be uploaded after virus scanning in a file storage service module based on a target main key, and calls the file to be uploaded after virus scanning of the target into a back-end service module after the file to be uploaded is searched, wherein the back-end service module is mainly responsible for logic processing and file verification and analysis; then the server judges whether the file to be uploaded after the target scans the virus meets a preset processing rule or not; and if the file to be uploaded after the target scans the viruses conforms to the preset processing rule, dispatching the file to be uploaded after the target scans the viruses to a corresponding target server instance for processing, and generating a file to be imported, wherein the service instance is an analysis bean instance and is mainly used for processing business logic. And the server generates other files to be imported in other task threads, and integrates one file to be imported and other files to be imported to obtain a plurality of files to be imported.

The step of judging whether the file to be uploaded after the target virus scanning meets the preset processing rule by the server specifically comprises the following steps:

the server reads byte data of a file to be uploaded after a target scans viruses to obtain the byte data of the file; the server judges whether the byte data of the file is greater than a byte threshold value; if the byte data of the file is larger than the byte threshold, the server judges that the file to be uploaded after the target virus scanning does not accord with the preset processing rule; if the byte data of the file is smaller than or equal to the byte threshold, the server judges that the file to be uploaded after the target scans the viruses conforms to a preset processing rule.

For example, in the embodiment, the byte threshold is 200M, and if the byte data of the file is 300M, it is determined that the file to be uploaded after the target virus is scanned corresponding to the byte data of the file does not conform to the preset processing rule. In another embodiment, if the byte data of the file is 180M, it is determined that the file to be uploaded after the target virus is scanned corresponding to the byte data of the file conforms to the preset processing rule.

If the file to be uploaded after the target virus scanning meets the preset processing rule, the file to be uploaded after the target virus scanning is dispatched to the corresponding target server instance for processing, and therefore the generation of the file to be imported specifically comprises the following steps:

if the file to be uploaded after the target virus scanning meets a preset processing rule server, extracting file reflection according to the file to be uploaded after the target virus scanning through a preset open source framework, wherein the file reflection is the class reflection of a JAVA program; the server determines a corresponding target service instance in a plurality of preset service instances according to file reflection; the server reads the file data format of the file to be uploaded after the target scans viruses, and a preset universal format checker is adopted to check the file data format to obtain the checked file to be uploaded; the server extracts a check field from the checked file to be uploaded, extracts corresponding processing logic from the plurality of service module groups according to the check field, and processes the checked file to be uploaded through the processing logic to obtain the file to be uploaded after logic processing; and the server adopts the target service instance to construct an object based on the file to be uploaded after the logic processing, and generates a file to be imported.

If the file to be uploaded after the target scans the viruses conforms to the preset processing rule, processing the file to be uploaded after the target scans the viruses through a preset open source frame Spring frame to obtain a file reflection, wherein the file reflection is the class reflection of a JAVA program, and the JAVA program can obtain the class reflection of the JAVA program through the Spring frame because the JAVA program has a reflection mechanism; then the server determines a corresponding target service instance based on the file reflection; the server reads a file data format of a file to be uploaded after a target scans viruses, wherein the file data format can comprise the character size of digits, the digits are integers, the digits are decimal numbers and/or decimal digits of the digits, and the like, a universal format checker is adopted to check the data format, the standard of the check is a preset standard data format, and the file to be uploaded after the check is obtained; the server extracts a check field from the checked file to be uploaded, then matches corresponding processing logic according to the check field, processes the checked file to be uploaded through the processing logic to obtain the file to be uploaded after logic processing, for example, the check field is 'check age', the server matches the processing logic of 'check age', and processes the checked file to be uploaded through the logic to obtain the file to be uploaded after logic processing; and then the server adopts a target service instance to construct an object based on the file to be uploaded after the logical processing to obtain the file to be imported, wherein the object construction can be understood as that after each column corresponding to each header in the data table constructs data, a row of tables corresponding to the header are regarded as a constructed object, and thus the file to be imported is obtained.

306. And importing a plurality of files to be imported into a database or a downstream system, wherein the downstream system is a system in butt joint with a server.

Specifically, when a plurality of files to be imported are imported into the database, the server analyzes the corresponding service instance to obtain a corresponding service instance parent class; the server buries each file to be imported into a corresponding target service instance parent class, and the corresponding target service subclass inherits the attribute and the logic execution method of the target service instance parent class; reading a corresponding file to be imported through a target service subclass, and storing the corresponding file to be imported into a database; when a plurality of files to be imported are imported into a downstream system, the server transmits the files to be imported to a data interface corresponding to the downstream system, and the downstream system is a system in butt joint with the server.

If the file to be imported needs to be stored in the database, the server analyzes the corresponding service instance to obtain a service instance parent class, in this embodiment, the server instance used in the processing process can directly inherit the parent class without re-matching processing logic, only each file to be imported needs to be buried into the corresponding target service instance parent class, the corresponding target service subclass inherits the attribute and the logic execution method of the target service instance parent class, and the server imports the corresponding file to be imported into the database by using the attribute and the logic execution method of the service parent class; if a plurality of files to be imported are imported into the downstream system, the server only needs to transmit the plurality of files to be imported to the corresponding data interfaces, and other import processes are processed by the downstream system.

With reference to fig. 4, the distributed file importing method according to the embodiment of the present invention is described above, and a distributed file importing apparatus according to the embodiment of the present invention is described below, where an embodiment of the distributed file importing apparatus according to the embodiment of the present invention includes:

the obtaining module 401 is configured to obtain a plurality of files to be uploaded and a plurality of file uploading requests, and obtain a plurality of service module groups configured by a terminal, where the plurality of files to be uploaded are a plurality of files after check bytes of the terminal, and one file uploading request corresponds to one file to be uploaded; it should be emphasized that, in order to further ensure the privacy and security of the files to be uploaded, the files to be uploaded may also be stored in a node of a block chain.

A response module 402, configured to, when the terminal obtains multiple request character strings and multiple primary keys from the multiple files to be uploaded, respond to the multiple file uploading requests to obtain multiple files to be uploaded after virus scanning, and store the multiple files to be uploaded after virus scanning in a file storage service module;

an allocating module 403, configured to receive the multiple request character strings and the multiple primary keys sent by the terminal, trigger multiple preset task threads and multiple thread task locks through the multiple request character strings, and allocate each primary key to a corresponding task thread, where the multiple thread task locks are used to lock the multiple preset task threads respectively;

the processing module 404 is configured to obtain, in each task thread, a corresponding file to be uploaded after virus scanning according to each primary key, and process the file to be uploaded after virus scanning corresponding to the target primary key by combining the target task thread with the corresponding service module group, so as to generate a plurality of files to be imported;

an importing module 405, configured to import the multiple files to be imported into a database or a downstream system, where the downstream system is a system that interfaces with a server.

Referring to fig. 5, another embodiment of the distributed file importing apparatus according to the embodiment of the present invention includes:

Optionally, the response module 402 includes:

the scanning unit 4021 is configured to scan viruses of the multiple files to be uploaded by using a file storage service module to obtain multiple scanning results when the terminal obtains the request character string and the multiple primary keys from the files to be uploaded;

the uploading unit 4022 is configured to obtain a plurality of files to be uploaded after viruses are scanned according to the plurality of scanning results, and store the plurality of files to be uploaded after viruses are scanned in the file storage service module.

Optionally, the uploading unit 4022 may be further specifically configured to:

judging whether each scanning result comprises a virus file or not;

if the target scanning result comprises a virus file, terminating the uploading process of the corresponding file to be uploaded;

if the target scanning result does not include the virus file, determining the corresponding file to be uploaded as the file to be uploaded after virus scanning, obtaining a plurality of files to be uploaded after virus scanning, and storing the plurality of files to be uploaded after virus scanning in the file storage service module.

Optionally, the processing module 404 includes:

the calling unit 4041 is configured to, in one of the task threads, call, by the back-end service module, a file to be uploaded after a corresponding target scans viruses from the file storage service module based on the target primary key;

a judging unit 4042, configured to judge whether the file to be uploaded after the target virus scanning meets a preset processing rule;

the processing unit 4043, if the file to be uploaded after the target virus scanning meets a preset processing rule, is configured to dispatch the file to be uploaded after the target virus scanning to a corresponding target service instance for processing by combining a corresponding target service module group, so as to generate a file to be imported;

the integrating unit 4044 is configured to generate other files to be imported in other task threads, and integrate the file to be imported and the other files to be imported to obtain a plurality of files to be imported.

Optionally, the determining unit 4042 may be further specifically configured to:

reading byte data of the file to be uploaded after the target scans the viruses to obtain file byte data;

judging whether the byte data of the file is larger than a byte threshold value;

if the byte data of the file is larger than the byte threshold, judging that the file to be uploaded after the target virus scanning does not accord with a preset processing rule;

if the byte data of the file is smaller than or equal to the byte threshold, judging that the file to be uploaded after the target scans the viruses conforms to a preset processing rule.

Optionally, the processing unit 4043 may be further specifically configured to:

if the file to be uploaded after the target virus scanning meets the preset processing rule, extracting file reflection according to the file to be uploaded after the target virus scanning through a preset open source framework, wherein the file reflection is the class reflection of a JAVA program;

determining a corresponding target service instance in a plurality of preset service instances according to the file reflection;

reading the file data format of the file to be uploaded after the target scans viruses, and verifying the file data format by adopting a preset universal format verifier to obtain the verified file to be uploaded;

extracting a check field from the checked file to be uploaded, extracting corresponding processing logic from the plurality of service module groups according to the check field, and processing the checked file to be uploaded through the processing logic to obtain a logically processed file to be uploaded;

and adopting the target service instance to construct an object based on the file to be uploaded after the logic processing to generate a file to be imported.

Optionally, the import module 405 may be further specifically configured to:

when the files to be imported are imported into a database, analyzing the corresponding service instances to obtain corresponding service instance parents;

embedding each file to be imported into a corresponding target service instance parent class, wherein the corresponding target service subclass inherits the attribute and the logic execution method of the target service instance parent class;

reading a corresponding file to be imported through a target service subclass, and storing the corresponding file to be imported into a database;

and when the files to be imported are imported into a downstream system, transmitting the files to be imported to a data interface corresponding to the downstream system, wherein the downstream system is a system in butt joint with a server.

Fig. 4 and fig. 5 describe the distributed file importing apparatus in the embodiment of the present invention in detail from the perspective of the modular functional entity, and describe the distributed file importing device in the embodiment of the present invention in detail from the perspective of hardware processing.

Fig. 6 is a schematic structural diagram of a distributed file importing device according to an embodiment of the present invention, where the distributed file importing device 600 may have a relatively large difference due to different configurations or performances, and may include one or more processors (CPUs) 610 (e.g., one or more processors) and a memory 620, and one or more storage media 630 (e.g., one or more mass storage devices) storing applications 633 or data 632. Memory 620 and storage medium 630 may be, among other things, transient or persistent storage. The program stored in the storage medium 630 may include one or more modules (not shown), each of which may include a series of instruction operations in the distributed file importing apparatus 600. Further, the processor 610 may be configured to communicate with the storage medium 630 and execute a series of instruction operations in the storage medium 630 on the distributed file import apparatus 600.

The distributed file import device 600 may also include one or more power supplies 640, one or more wired or wireless network interfaces 650, one or more input-output interfaces 660, and/or one or more operating systems 631, such as Windows Server, mac OS X, unix, linux, freeBSD, and so forth. Those skilled in the art will appreciate that the distributed file import device architecture shown in fig. 6 does not constitute a limitation of the distributed file import device and may include more or fewer components than shown, or some components in combination, or a different arrangement of components.

The present invention further provides a distributed file importing apparatus, where the computer apparatus includes a memory and a processor, where the memory stores computer readable instructions, and the computer readable instructions, when executed by the processor, cause the processor to perform the steps of the distributed file importing method in the foregoing embodiments.

The present invention also provides a computer-readable storage medium, which may be a non-volatile computer-readable storage medium, and which may also be a volatile computer-readable storage medium, having stored therein instructions, which, when executed on a computer, cause the computer to perform the steps of the distributed file import method.

It is clear to those skilled in the art that, for convenience and brevity of description, the specific working processes of the above-described systems, apparatuses and units may refer to the corresponding processes in the foregoing method embodiments, and are not described herein again.

The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism, an encryption algorithm and the like. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.

The integrated unit, if implemented in the form of a software functional unit and sold or used as a stand-alone product, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present invention, which is substantially or partly contributed by the prior art, or all or part of the technical solution may be embodied in a software product, which is stored in a storage medium and includes instructions for causing a computer device (which may be a personal computer, a server, or a network device) to perform all or part of the steps of the method according to the embodiments of the present invention. And the aforementioned storage medium includes: various media capable of storing program codes, such as a usb disk, a removable hard disk, a read-only memory (ROM), a Random Access Memory (RAM), a magnetic disk, or an optical disk.

The above-mentioned embodiments are only used for illustrating the technical solutions of the present invention, and not for limiting the same; although the present invention has been described in detail with reference to the foregoing embodiments, it will be understood by those of ordinary skill in the art that: the technical solutions described in the foregoing embodiments may still be modified, or some technical features may be equivalently replaced; and such modifications or substitutions do not depart from the spirit and scope of the corresponding technical solutions of the embodiments of the present invention.

Claims

1. A distributed file import method is characterized by comprising the following steps:

acquiring a plurality of files to be uploaded and a plurality of file uploading requests, and acquiring a plurality of service module groups configured by a terminal, wherein the plurality of files to be uploaded are a plurality of files after check bytes of the terminal, and one file uploading request corresponds to one file to be uploaded;

when the terminal acquires a plurality of request character strings and a plurality of main keys from the plurality of files to be uploaded, responding to the plurality of file uploading requests to acquire a plurality of files to be uploaded after viruses are scanned, and storing the plurality of files to be uploaded after viruses are scanned in a file storage service module;

receiving the plurality of request character strings and the plurality of main keys sent by a terminal, triggering a plurality of preset task threads and a plurality of thread task locks through the plurality of request character strings, and distributing each main key to the corresponding task thread, wherein the plurality of thread task locks are respectively used for locking the plurality of preset task threads;

acquiring a file to be uploaded after corresponding virus scanning in each task thread according to each main key, processing the file to be uploaded after virus scanning corresponding to the target main key through the target task thread in combination with the corresponding service module group, and generating a plurality of files to be imported;

and importing the files to be imported into a database or a downstream system, wherein the downstream system is a system in butt joint with a server.

2. The method according to claim 1, wherein when the terminal obtains a plurality of request character strings and a plurality of primary keys from the plurality of files to be uploaded, the terminal responds to the plurality of file uploading requests to obtain a plurality of files to be uploaded after virus scanning, and stores the plurality of files to be uploaded after virus scanning in a file storage service module, and the method comprises:

when the terminal acquires the request character string and the plurality of main keys from the file to be uploaded, a file storage service module is adopted to perform virus scanning on the plurality of files to be uploaded to obtain a plurality of scanning results;

and obtaining a plurality of files to be uploaded after virus scanning according to the plurality of scanning results, and storing the plurality of files to be uploaded after virus scanning in the file storage service module.

3. The method of claim 2, wherein the obtaining a plurality of files to be uploaded after virus scanning according to the plurality of scanning results and storing the plurality of files to be uploaded after virus scanning in the file storage service module comprises:

judging whether each scanning result comprises a virus file or not;

if the target scanning result does not comprise the virus file, determining the corresponding file to be uploaded as the file to be uploaded after virus scanning, obtaining a plurality of files to be uploaded after virus scanning, and storing the plurality of files to be uploaded after virus scanning in the file storage service module.

4. The distributed file importing method according to claim 1, wherein the acquiring, in each task thread, the file to be uploaded after the corresponding virus scanning according to each primary key, and processing the file to be uploaded after the virus scanning corresponding to the target primary key by combining the target task thread with the corresponding service module group to generate a plurality of files to be imported includes:

in one task thread, calling a corresponding file to be uploaded after a target scans viruses from the file storage service module through a back-end service module based on a target main key;

judging whether the file to be uploaded after the target scans the viruses meets a preset processing rule or not;

if the file to be uploaded after the target virus scanning meets the preset processing rule, the file to be uploaded after the target virus scanning is dispatched to the corresponding target service instance to be processed by combining with the corresponding target service module group, and a file to be imported is generated;

and generating other files to be imported in other task threads, and integrating the file to be imported and the other files to be imported to obtain a plurality of files to be imported.

5. The distributed file importing method according to claim 4, wherein the determining whether the file to be uploaded after the target virus scanning meets a preset processing rule includes:

and if the byte data of the file is less than or equal to the byte threshold, judging that the file to be uploaded after the target virus scanning meets a preset processing rule.

6. The distributed file importing method according to claim 4, wherein if the file to be uploaded after the target virus scanning meets a preset processing rule, the file to be uploaded after the target virus scanning is dispatched to a corresponding target service instance to be processed by combining with a corresponding target service module group, and the generating of the file to be imported includes:

7. The distributed file importing method according to claim 1, wherein the importing the plurality of files to be imported into a database or a downstream system, the downstream system being a system interfacing with a server includes:

when the files to be imported are imported into a database, analyzing the corresponding service instances to obtain corresponding service instance parent classes;

reading the corresponding file to be imported through the target service subclass, and storing the corresponding file to be imported into a database;

8. A distributed file import apparatus, wherein the distributed file import apparatus includes:

the system comprises an acquisition module, a processing module and a processing module, wherein the acquisition module is used for acquiring a plurality of files to be uploaded and a plurality of file uploading requests, and acquiring a plurality of service module groups configured by a terminal, the plurality of files to be uploaded are a plurality of files after bytes are checked by the terminal, and one file uploading request corresponds to one file to be uploaded;

the response module is used for responding to the file uploading requests to obtain a plurality of files to be uploaded after viruses are scanned and storing the files to be uploaded after the viruses are scanned in the file storage service module when the terminal obtains a plurality of request character strings and a plurality of main keys from the files to be uploaded;

the distribution module is used for receiving the request character strings and the main keys sent by the terminal, triggering a plurality of preset task threads and a plurality of thread task locks through the request character strings, and distributing each main key to the corresponding task thread, wherein the thread task locks are respectively used for locking the preset task threads;

the processing module is used for acquiring the corresponding file to be uploaded after virus scanning in each task thread according to each main key, processing the file to be uploaded after virus scanning corresponding to the target main key through the target task thread in combination with the corresponding service module group, and generating a plurality of files to be imported;

and the import module is used for importing the files to be imported into a database or a downstream system, and the downstream system is a system in butt joint with the server.

9. A distributed file import apparatus, wherein the distributed file import apparatus includes: a memory and at least one processor, the memory having instructions stored therein;

the at least one processor invokes the instructions in the memory to cause the distributed file import device to perform the distributed file import method of any of claims 1-7.

10. A computer-readable storage medium having instructions stored thereon, wherein the instructions, when executed by a processor, implement the distributed file import method of any of claims 1-7.