CN107193881A - Upload files to method, device and the equipment of distributed file system - Google Patents
Upload files to method, device and the equipment of distributed file system Download PDFInfo
- Publication number
- CN107193881A CN107193881A CN201710282637.8A CN201710282637A CN107193881A CN 107193881 A CN107193881 A CN 107193881A CN 201710282637 A CN201710282637 A CN 201710282637A CN 107193881 A CN107193881 A CN 107193881A
- Authority
- CN
- China
- Prior art keywords
- file
- uploaded
- system banner
- obtaining
- upload
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/16—File or folder operations, e.g. details of user interfaces specifically adapted to file systems
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
- G06F16/18—File system types
- G06F16/182—Distributed file systems
Abstract
The present invention relates to a kind of method for uploading to distributed file system, including：Obtain file to be uploaded；System banner according to belonging to the file to be uploaded, obtains destination folder corresponding with the system banner in distributed file system；The file to be uploaded is uploaded into the destination folder.The invention further relates to a kind of device and equipment for uploading distributed file system.The above-mentioned method for uploading files to distributed file system, device and equipment, instead of manually handling, on the one hand can be avoided the increase of human cost, on the other hand can also effectively reduce operational error odds using automatic method.
Description
Technical field
The present invention relates to software field, method, device and the equipment of distributed system are more particularly to uploaded files to.
Background technology
As the development of cloud is, it is necessary to the data for being stored in server are continuously increased, individual server is locally stored
Capacity is difficult to meet storage demand, so introducing distributed type assemblies storage.And how by local file it is accurate, be stored in sorted order
In distributed file system HDFS cluster servers, specifically how the file in different files is uploaded into distributed document
Into a urgent problem in corresponding file in system HDFS.
Conventional method is manually manual mode to solve the above problems, but is manually handled, and on the one hand can be made
Into the increase of human cost, on the other hand it can also increase operational error odds.
The content of the invention
Based on this, it is necessary to provide a kind of automatic method for uploading files to distributed file system to replace manually
Processing, on the one hand can avoid the increase of human cost, on the other hand can also effectively reduce operational error odds.
A kind of method for uploading files to distributed file system, wherein, methods described includes：
Obtain file to be uploaded；
System banner according to belonging to the file to be uploaded, obtains corresponding with the system banner in distributed file system
Destination folder；
The file to be uploaded is uploaded into the destination folder.
The above-mentioned method for uploading files to distributed file system, replaces manually handling using automatic method,
On the one hand the increase of human cost can be avoided, on the other hand can also effectively reduces operational error odds.
As a kind of embodiment, wherein, it is described acquisition file to be uploaded the step of before also include：
Obtain and upload record；
If obtaining successfully, system banner and the corresponding file sequence number of system banner are obtained in the upload record；
The file to be uploaded is obtained according to the system banner and the corresponding file sequence number of system banner.
As a kind of embodiment, wherein, it is described according to the system banner and the corresponding file of system banner
The step of sequence number obtains the file to be uploaded includes：
Whether judge the file sequence number is initial value；
If so, then obtaining first file in the corresponding file of the system banner；
If it is not, then obtaining the file to be uploaded according to the file sequence number.
As a kind of embodiment, wherein, it is described that the file to be uploaded is uploaded into the destination folder
Step includes：
Whether the process for monitoring the upper transmitting file interrupts；
If so, then obtaining the sequence number of the system banner and upper transmitting file belonging to upper transmitting file；
The sequence number of system banner and upper transmitting file according to obtaining upper transmitting file, which is created or updated, uploads record.
As a kind of embodiment, wherein, it is described acquisition file to be uploaded the step of before also include：
Local directory is searched for, system banner set is obtained；
According to system banner set, create or update system banner caching.
As a kind of embodiment, wherein, the search local directory, the step of obtaining system banner set is wrapped
Include：
Opened every the very first time interval being pre-configured with and read local directory；
Whether judge the file type under local directory successively is file；
If so, then obtaining the title of the file, and the title of the file is added into the system banner set
In；
If it is not, then continuing to judge next file type under catalogue, until obtaining files all under local directory
Title.
As a kind of embodiment, wherein, it is described acquisition file to be uploaded the step of include：
Every the second time interval run-down system banner caching being pre-configured with；
System banner sequence opened file folder successively in being cached according to the system banner；
Check in the file of the opening with the presence or absence of the file for needing to upload；
If there is the file for needing to upload, the file to be uploaded is obtained；
If in the absence of the file for needing to upload, continuing to open next file, until the system banner sequence
Corresponding file is examined in row.
As a kind of embodiment, wherein, it is described acquisition file to be uploaded the step of include：Read text to be uploaded
Part；
Judge whether to read successfully；
If reading successfully, the transport stream of the file to be uploaded is obtained, local data is read；
If reading is unsuccessful, program sleep, the length of one's sleep is the 3rd time interval being pre-configured with, and obtains treat again
Transmitting file.
As a kind of embodiment, wherein, it is described that the file to be uploaded is uploaded into the destination folder
Step includes：
Set up connection pool and be connected to the distributed file system；
Obtain and open the upload path corresponding to the destination folder；
File is write into the corresponding upload path；
Judge whether the operation for uploading the file to be uploaded succeeds；
If uploading successfully, the file uploaded is deleted, the corresponding file path is closed；
If not uploading success, output error message simultaneously updates system journal.
A kind of device for uploading files to distributed file system, wherein, described device includes：
File uploading module, for obtaining file to be uploaded；
Destination folder acquisition module, for the system banner according to belonging to the file to be uploaded, obtains distributed text
Destination folder corresponding with the system banner in part system；
File uploading module, for uploading the file to be uploaded into the destination folder.
The above-mentioned device for uploading files to distributed file system, it is possible to achieve artificial hand is replaced using automatic method
Dynamic processing, on the one hand can avoid the increase of human cost, on the other hand can also effectively reduce operational error odds.
As a kind of embodiment, wherein, described device also includes：
Acquisition module is recorded, record is uploaded for obtaining；
Sequence number acquisition module, if for obtaining successfully, system banner and system mark are obtained in the upload record
Know corresponding file sequence number；
Ff module, for being treated according to the system banner and the corresponding file sequence number acquisition of system banner
Upper transmitting file.
As a kind of embodiment, wherein, the ff module also includes：
Sequence number judging unit, for judging whether the file sequence number is initial value；
Mark acquiring unit, for obtaining first file in the system banner；
Sequence number searching unit, for obtaining the file to be uploaded according to the file sequence number.
As a kind of embodiment, wherein, described device also includes：
Acquisition module is recorded, record is uploaded for obtaining；
Sequence number acquisition module, if for obtaining successfully, obtaining residing for the file to be uploaded in record is uploaded and being
The sequence number of system mark and file to be uploaded；
Ff module, for the system banner and the sequence number of file to be uploaded according to belonging to the file to be uploaded
Obtain the file to be uploaded.
As a kind of embodiment, wherein, the ff module also includes：
Sequence number judging unit, for judging whether the sequence number of the file to be uploaded is initial value；
Mark acquiring unit, for obtaining first file in the system banner；
Sequence number searching unit, for obtaining the file to be uploaded according to the sequence number of the upper transmitting file.
As a kind of embodiment, wherein, the file uploading module includes：
Monitoring unit is interrupted, whether the process for monitoring the upper transmitting file interrupts；
Mark acquiring unit, for if so, then obtaining the sequence number of the system banner and upper transmitting file belonging to upper transmitting file；
Updating block is recorded, is created for the system banner and the sequence number of upper transmitting file according to obtaining upper transmitting file
Or update upload record.
As a kind of embodiment, wherein, described device also includes：
Gather acquisition module, for searching for local directory, obtain system banner set；
Buffer update module, for according to system banner set, creating or updating system banner caching.
As a kind of embodiment, wherein, the set acquisition module includes：
Catalogue reading unit, for being opened every the very first time interval being pre-configured with and reading local directory；
Type judging unit, for judging whether the file type under local directory is file successively；
Gather updating block, for if so, then obtain the title of the file, and the title of the file is added
In the system banner set；
Catalogue judging unit, for if it is not, then continuing to judge next file type under catalogue, until obtaining local mesh
All system banners under record.
As a kind of embodiment, wherein, the file acquisition module includes：
Cache sweep unit, for every the second time interval run-down system banner caching being pre-configured with；
File opening unit, file is opened for the system banner sequence in being cached according to the system banner successively
Folder；
With the presence or absence of the file for needing to upload in file checking unit, the file for checking the opening, if in the presence of
The file for needing to upload, then obtain the file to be uploaded, if in the absence of the file for needing to upload, continuing to open
Next file, until file corresponding in the system banner sequence is examined.
As a kind of embodiment, wherein, the file acquisition module includes：
Document reading unit, for reading file to be uploaded；
Condition adjudgement unit, for judging whether to read successfully；
Data capture unit, if for reading successfully, obtaining the transport stream of the file to be uploaded, reads local number
According to；
Program sleep cell, if unsuccessful for reading, program sleep, the length of one's sleep is the 3rd time being pre-configured with
Interval, obtains file to be uploaded again.
As a kind of embodiment, wherein, the uploading module includes：
Connection pool sets up unit, for setting up connection pool and being connected to the distributed file system；
Path acquiring unit, for obtaining and opening the upload path corresponding to the destination folder；
File write unit, for writing file into the corresponding upload path；
Identifying unit is operated, whether the operation for judging to upload the file to be uploaded succeeds；
Path closing unit, if for uploading successfully, deleting the file uploaded, closes the corresponding file road
Footpath；
Daily record updating block, if for not uploading success, output error message simultaneously updates system journal.
A kind of equipment for uploading files to distributed file system, wherein, the equipment include processor, memory and
The computer program on the memory is stored in, the computer program realizes following steps when being executed by processor：
Obtain file to be uploaded；
System banner according to belonging to the file to be uploaded, obtains corresponding with the system banner in distributed file system
Destination folder；
The file to be uploaded is uploaded into the destination folder.
The above-mentioned equipment for uploading files to distributed file system, it is possible to achieve artificial hand is replaced using automatic method
Dynamic processing, on the one hand can avoid the increase of human cost, on the other hand can also effectively reduce operational error odds.
As a kind of embodiment, wherein, the step of the acquisition of computing device file to be uploaded before also
Including：
Obtain and upload record；
If obtaining successfully, system banner and the corresponding file sequence number of system banner are obtained in the upload record；
The file to be uploaded is obtained according to the system banner and the corresponding file sequence number of system banner.
As a kind of embodiment, wherein, the described of the computing device according to the system banner and is
The step of corresponding file sequence number of system mark obtains the file to be uploaded includes：
Whether judge the file sequence number is initial value；
If so, then obtaining first file in the corresponding file of the system banner；
If it is not, then obtaining the file to be uploaded according to the file sequence number.
As a kind of embodiment, wherein, the described of the computing device uploads into the destination folder
The step of file to be uploaded, includes：
Whether the process for monitoring the upper transmitting file interrupts；
If so, then obtaining the sequence number of the system banner and upper transmitting file belonging to upper transmitting file；
The sequence number of system banner and upper transmitting file according to belonging to obtaining upper transmitting file, which is created or updated, uploads record.
As a kind of embodiment, wherein, the step of the acquisition of computing device file to be uploaded it
It is preceding also to include：
Local directory is searched for, system banner set is obtained；
According to system banner set, create or update system banner caching.
As a kind of embodiment, wherein, the search local directory of the computing device obtains system mark
The step of knowing set includes：
Opened every the very first time interval being pre-configured with and read local directory；
Whether judge the file type under local directory successively is file；
If so, then obtaining the title of the file, and the title of the file is added into the system banner set
In；
If it is not, then continuing to judge next file type under catalogue, until obtaining files all under local directory
Title.
As a kind of embodiment, wherein, wrap the step of the acquisition of computing device file to be uploaded
Include：
Every the second time interval run-down system banner caching being pre-configured with；
System banner sequence opened file folder successively in being cached according to the system banner；
Check in the file of the opening with the presence or absence of the file for needing to upload；
If there is the file for needing to upload, the file to be uploaded is obtained；
If in the absence of the file for needing to upload, continuing to open next file, until the system banner sequence
Corresponding file is examined in row.
As a kind of embodiment, wherein, wrap the step of the acquisition of computing device file to be uploaded
Include：
Read file to be uploaded；
Judge whether to read successfully；
If reading successfully, the transport stream of the file to be uploaded is obtained, local data is read；
If reading is unsuccessful, program sleep, the length of one's sleep is the 3rd time interval being pre-configured with, and obtains treat again
Transmitting file.
As a kind of embodiment, wherein, the described of the computing device uploads into the destination folder
The step of file to be uploaded, includes：
Set up connection pool and be connected to the distributed file system；
Obtain and open the upload path corresponding to the destination folder；
File is write into the corresponding upload path；
Judge whether the operation for uploading the file to be uploaded succeeds；
If uploading successfully, the file uploaded is deleted, the corresponding file path is closed；
If not uploading success, output error message simultaneously updates system journal.
Brief description of the drawings
Fig. 1 is the application scenario diagram for the method for uploading that one of which embodiment is provided；
Fig. 2 is the flow chart for the method for uploading that one of which embodiment is provided；
Fig. 3 is the partial process view for the method for uploading that one of embodiment is provided；
Fig. 4 is the flow chart that the acquisition that one of which embodiment is provided uploads Files step；
Fig. 5 is flow chart the step of obtaining file to be uploaded that wherein another embodiment is provided；
Fig. 6 is stream the step of file to be uploaded is uploaded into destination folder that one of which embodiment is provided
Cheng Tu；
Fig. 7 is that the structure for the device for uploading files to distributed file system that one of which embodiment is provided is shown
It is intended to.
Embodiment
In order to make the purpose , technical scheme and advantage of the present invention be clearer, below in conjunction with drawings and Examples pair
The present invention is further elaborated.It should be appreciated that specific embodiment described herein is only to explain the present invention, not
For limiting the present invention.
Referring to Fig. 1, Fig. 1 is the application scenario diagram for the method for uploading that one of which embodiment is provided.Data can
With from different slave computers, for example：First slave computer 105, the second slave computer 107, the 3rd slave computer 109.It is appreciated that being
System can also access more slave computers again.The file that different slave computers is uploaded will be stored in local file server 103
Different files in, and file according to system directly from slave computer upload data in system banner position ordered
Name a, so file correspond to the system banner of a slave computer.With the increase of access slave computer, the quantity of file
Also it can increase.With it, with real time scan and the new file that slave computer is uploaded can be recognized, and according to different system banners
Upload in distributed file system in corresponding destination folder.Methods described can be applied in (SuSE) Linux OS, passed through
Linux system function and HDFS interfaces is called to realize.Methods described can use C Plus Plus, be programmed with reference to linux system real
It is existing.
Referring to Fig. 2, Fig. 2 is the side for uploading files to distributed file system that one of which embodiment is provided
The flow chart of method, methods described includes：
S202, obtains file to be uploaded.
Specifically, the file to be uploaded is to need to upload to the file of distributed file system.The distributed document
System can be Hadoop distributed systems (HDFS, Hadoop Distribute File System).
With the system mark in S204, the system banner according to belonging to the file to be uploaded, acquisition distributed file system
Know corresponding destination folder.
Specifically, the system banner belonging to file to be uploaded refers to the name for locally depositing the file of the file to be uploaded
Claim, destination folder corresponding with the system banner with distributed file system is obtained according to the system banner.Further
Ground, can use linux system programming to operate HDFS, initially set up distributed system connection pool, then connect described
Distributed system, then opens file path corresponding with destination folder in the distributed system, in the most backward path
Write file to be uploaded.
S206, the file to be uploaded is uploaded into the destination folder.
Specifically, the file to be uploaded is uploaded into the corresponding destination folder of distributed system.
The above-mentioned method for uploading files to distributed file system, replaces manually handling using automatic method,
On the one hand the increase of human cost can be avoided, on the other hand can also effectively reduces operational error odds.
Include obtaining before the step of method provided as one of which embodiment, acquisition file to be uploaded
Upload is taken to record；If obtaining successfully, system banner and the corresponding file sequence of system banner are obtained in the upload record
Number；The file to be uploaded is obtained according to the system banner and the corresponding file sequence number of system banner.
Specifically, it can realize that the progress of the upload to different files is automatically preserved using upload record,
Next time during startup program, it can be read according to record and upload record.It is described upload record be exactly in each file on
The sequence number of transmitting file, following form can be used by uploading record：System banner+sequence number, for example：A00000+1 represents A00000 texts
Part folder has uploaded to the file that reference number of a document is 1.The upload record can be stored in memory file, can also directly be preserved
In configuration file.If there is record is uploaded, then system banner and the system mark are further obtained in record is uploaded
Know corresponding file sequence number, file to be uploaded is then obtained according to the system banner and file sequence number, so that file will not be made
Some file of transmission generation is transmitted several times or some file is missed the mistakes such as transmission, ensure that the correct of file transmission
Property and validity.
The above-mentioned method for uploading files to distributed file system, can be realized to different files using record is uploaded
The progress of upload is automatically preserved, it is ensured that when program is restarted, can automatically continue upload.
The method provided as one of which embodiment, it is described to be treated into the corresponding folder described in upload
The step of transmitting file, includes：Whether the process for monitoring the upper transmitting file interrupts；If so, then obtaining the system belonging to upper transmitting file
The sequence number of mark and upper transmitting file；The sequence number of system banner and upper transmitting file according to belonging to obtaining upper transmitting file create or
Update and upload record.
Specifically, if there is special circumstances, the process of upper transmitting file for example needs to close or restart by terminal
Program.At this moment program is preserved during can system banner be cached system banner and the sequence of upper transmitting file in each file
Number it is stored in memory file.When the establishment is referred to without any upload record, program can automatically create upload record.It is described
Renewal refers to that when having uploaded record program can be updated to uploading record.
The method provided as one of which embodiment, it is described to be treated into the corresponding folder described in upload
The step of transmitting file, includes：Monitoring is whether file to be uploaded uploads success；If so, then obtaining the system mark belonging to upper transmitting file
Knowledge and the sequence number of upper transmitting file；The sequence number of system banner and upper transmitting file according to obtaining upper transmitting file is created or more
It is new to upload record.
Specifically, in the present embodiment, the upload record can monitor whether transmitting file uploads success, if described
Upload successfully, then record the sequence number of the system banner and upper transmitting file described in upper transmitting file, and create or update upper biography
Record.The upload record can be stored in memory file, can also be stored directly in configuration file, program can be each
The upload record in memory file or configuration file is read during startup first.
The method provided as one of which embodiment, the system mark according to belonging to the file to be uploaded
The step of sequence number of knowledge and file to be uploaded obtains the file to be uploaded also includes：Judge the sequence number of the file to be uploaded
Whether it is initial value；If so, then obtaining first file in the system banner；If it is not, then according to the upper transmitting file
Sequence number obtains the file to be uploaded.
Specifically, following form can be used by uploading record：System banner+sequence number.For example upload and be recorded as A00001+0
When, 0 for upload sequence number initial value, representation of file A00001 files not on be transmitted through file, then directly upload A00001 in
First file.
Also wrapped before the step of method provided as one of which embodiment, acquisition file to be uploaded
Include：
Local directory is searched for, system banner set is obtained；According to system banner set, create and update system banner and delay
Deposit.
Specifically, local directory is searched for, the file of upper transmitting file is needed to obtain all storages in catalogue, due to system
Mark and the title of file are one-to-one, therefore the folder name got is referred to as into system banner updated to system
Mark caching in, for record and easily calling system identify.The establishment refers to that program automatically creates system banner and delayed
Deposit.The renewal refers to that program can be updated according to the system banner newly obtained to system banner caching.
Specifically, can be with automatic identification wherein different file by searching for local directory, and not identical text is obtained automatically
The file to be uploaded uploaded is needed in part folder.Local file system is scanned, newly-increased text to be uploaded can be obtained automatically
Part, further reduces the whole artificial operation uploaded in flow.
Specifically, before search local directory, the step of can also increasing reading configuration file reads configuration file master
If in order to obtain the IP address of distributed file system server to be connected, the configuration file can also include being used to protect
Deposit the upload record information of file uploading position.
Specifically, before scanning local file system, the step of initialization log system can also be increased, the day aspiration
System is to export necessary program print information, conveniently check the running situation of system and be conveniently likely to occur in system and asked
Problem investigation is carried out during topic.
Specifically, before search local directory, the step of setting up connection pool can also be increased, the connection pool of setting up can
To be operated using linux system to HDFS interfaces, HDFS connection pools are set up.Set up the connection pool can optimize it is procedural
Energy.
The above-mentioned method for uploading files to distributed file system, can be by scanning and recording the side of local file system
Formula, recognizes wherein different files, meets the file uploaded to obtain, further reduces the step of manually handling.
Referring to Fig. 3, the partial process view for the method that Fig. 3 is provided by an embodiment.The search is local
Catalogue, the step of obtaining system banner set includes：
S302, opens every the very first time interval being pre-configured with and reads local directory.
Specifically, the local directory refers to the particular category that file to be uploaded is deposited in local system, such as HDFS mesh
Record etc..
Specifically, data may be from different slave computer modular systems, and the data of different slave computer modular systems will
It can be stored in different files, the title of these files in search local directory, if over new slave computer module
The access of system, may produce new system banner, all be corresponded every system banner with Folder Name.So can be with
Local directory is re-searched for every the time being pre-configured with, new system banner set is obtained.Further, for performance
The consideration of aspect, can make program sleep run-down local directory again after the regular hour.The length of one's sleep is according to need
Ask what is be pre-configured with.Further, the length of one's sleep was selected in the range of 5 minutes to 15 minutes.
S304, whether the type for judging the file under the local directory successively is file.
Specifically, the file that all types under local directory are file is found out.
S306, if so, then obtaining the title of the file, and adds the system banner by the title of the file
In set.
Specifically, if the type of local file is file, the title of the file is directly obtained, due to file
The title of folder is one-to-one with system banner, it is possible to the name of file directly is referred to as into system banner and adds system
Logo collection.
S308, if it is not, then continuing to judge next file type under catalogue, until file all under local directory
Title.
Specifically, above-mentioned steps can be operated by linux system programming to local system, and its basic process is：
Open local directory, read local directory, the file type under multilevel iudge current directory is catalogue or file, be catalogue then
System banner is found in expression, directly reads the system banner, and the system banner is added into system banner caching.Until local mesh
All file types are completed to judge under record, then scanning is completed, and closes local directory.
Referring to Fig. 4, Fig. 4 uploads the flow chart of Files step for the acquisition that one of which embodiment is provided.Institute
The step of stating acquisition file to be uploaded includes：
S402, caching is identified every the second time sweep primary system being pre-configured with.
Specifically, timing scan system banner is cached, and obtains the system banner sequence in system banner buffer queue.
S404, the system banner sequence opened file folder successively in being cached according to the system banner.
Specifically, loop checker system representation caches to search the file in each file.
S406, is checked in the file of the opening with the presence or absence of the file for needing to upload.
S408, if there is the file for needing to upload, obtains the file to be uploaded.
Specifically, if saving upload record, then easily can judge this document by uploading record information
With the presence or absence of the file for needing to upload in folder.For example：Recorded according to described upload, can read in this document folder and upload
The particular location of the file arrived, by the particular location it may determine that with the presence or absence of the text for needing to upload in this document folder
Part.
S410, if in the absence of the file for needing to upload, continuing to open next file, until the system mark
Know file corresponding in sequence to be examined.
Specifically, the corresponding file of system banner all in loop checking system mark caching.
Referring to Fig. 5, flow the step of obtaining file to be uploaded that Fig. 5 provides for one of which embodiment
Figure.The step includes：
S502, reads file to be uploaded.
Specifically, function is programmed by linux system and opens the specific file to be uploaded under system banner, started reading and treat
Upper transmitting file.
S504, judges whether to open successfully.
Specifically, start upper transmitting file, judge to open successfully or fail by function return value.
S506, if reading successfully, obtains the transport stream of the file to be uploaded.
Specifically, if reading the transport stream for successfully obtaining file, local data is further read, starts to upload.
S508, if reading is unsuccessful, program sleep, the length of one's sleep is the 3rd time being pre-configured with, and obtains treat again
Upper transmitting file.
Specifically, if opening failure, program is slept a period of time, the time can match somebody with somebody in advance according to demand
Put.Further, the time being pre-configured with can be selected in the range of 1 minute to 5 minutes.
Referring to Fig. 6, Fig. 6 uploads file to be uploaded into destination folder for what one of which embodiment was provided
The step of flow chart.The step includes：
S602, connects the distributed file system.
Specifically, call the HDFS interfaces of linux system to operate distributed file system, connect HDFS.
S604, obtains and opens the upload path corresponding to the destination folder.
Specifically, the upload path corresponding to corresponding folder in the HDFS interfaces opening HDFS of linux system is called.
S606, file is write into the corresponding upload path.
Specifically, upload path to the upload path obtained by step S604 and write file to be uploaded.
S608, judges whether the operation for uploading the file to be uploaded succeeds.
Specifically, judge to upload according to the return value for calling HDFS interfaces and whether succeed.
S610, if uploading successfully, deletes the file uploaded, closes the respective file path.
Specifically, if it is determined that uploading successfully, just delete the file uploaded, and close the respective file path,
It can also record or change simultaneously upload record.
S612, if not uploading success, output error message, the error message is used to update system journal.
Specifically, if not uploading success, output error message, Update log file is convenient according to journal file
Print error message, investigates failure cause.
Referring to Fig. 7, the dress for uploading files to distributed file system that Fig. 7 provides for one of which embodiment
The structural representation put, wherein, described device includes：
File uploading module, for obtaining file to be uploaded；
Destination folder acquisition module, for the system banner according to belonging to the file to be uploaded, obtains distributed text
Destination folder corresponding with the system banner in part system；
File uploading module, for uploading the file to be uploaded into the destination folder.
The above-mentioned device for uploading files to distributed file system, it is possible to achieve artificial hand is replaced using automatic method
Dynamic processing, on the one hand can avoid the increase of human cost, on the other hand can also effectively reduce operational error odds.
As a kind of embodiment, wherein, described device also includes：
Acquisition module is recorded, record is uploaded for obtaining；
Sequence number acquisition module, if for obtaining successfully, system banner and system mark are obtained in the upload record
Know corresponding file sequence number；
Ff module, for being treated according to the system banner and the corresponding file sequence number acquisition of system banner
Upper transmitting file.
As a kind of embodiment, wherein, the ff module also includes：
Sequence number judging unit, for judging whether the file sequence number is initial value；
Mark acquiring unit, for obtaining first file in the system banner；
Sequence number searching unit, for obtaining the file to be uploaded according to the file sequence number.
As a kind of embodiment, wherein, described device also includes：
Acquisition module is recorded, record is uploaded for obtaining；
Sequence number acquisition module, if for obtaining successfully, obtaining residing for the file to be uploaded in record is uploaded and being
The sequence number of system mark and file to be uploaded；
Ff module, for the system banner and the sequence number of file to be uploaded according to belonging to the file to be uploaded
Obtain the file to be uploaded.
As a kind of embodiment, wherein, the ff module also includes：
Sequence number judging unit, for judging whether the sequence number of the file to be uploaded is initial value；
Mark acquiring unit, for obtaining first file in the system banner；
Sequence number searching unit, for obtaining the file to be uploaded according to the sequence number of the upper transmitting file.
As a kind of embodiment, wherein, the file uploading module includes：
Monitoring unit is interrupted, whether the process for monitoring the upper transmitting file interrupts；
Mark acquiring unit, for if so, then obtaining the sequence number of the system banner and upper transmitting file belonging to upper transmitting file；
Updating block is recorded, is created for the system banner and the sequence number of upper transmitting file according to obtaining upper transmitting file
Or update upload record.
As a kind of embodiment, wherein, described device also includes：
Gather acquisition module, for searching for local directory, obtain system banner set；
Buffer update module, for according to system banner set, creating or updating system banner caching.
As a kind of embodiment, wherein, the set acquisition module includes：
Catalogue reading unit, for being opened every the very first time interval being pre-configured with and reading local directory；
Type judging unit, for judging whether the file type under local directory is file successively；
Gather updating block, for if so, then obtain the title of the file, and the title of the file is added
In the system banner set；
Catalogue judging unit, for if it is not, then continuing to judge next file type under catalogue, until obtaining local mesh
All system banners under record.
As a kind of embodiment, wherein, the file acquisition module includes：
Cache sweep unit, for every the second time interval run-down system banner caching being pre-configured with；
File opening unit, file is opened for the system banner sequence in being cached according to the system banner successively
Folder；
With the presence or absence of the file for needing to upload in file checking unit, the file for checking the opening, if in the presence of
The file for needing to upload, then obtain the file to be uploaded, if in the absence of the file for needing to upload, continuing to open
Next file, until file corresponding in the system banner sequence is examined.
As a kind of embodiment, wherein, the file acquisition module includes：
Document reading unit, for reading file to be uploaded；
Condition adjudgement unit, for judging whether to read successfully；
Data capture unit, if for reading successfully, obtaining the transport stream of the file to be uploaded, reads local number
According to；
Program sleep cell, if unsuccessful for reading, program sleep, the length of one's sleep is the 3rd time being pre-configured with
Interval, obtains file to be uploaded again.
As a kind of embodiment, wherein, the uploading module includes：
Connection pool sets up unit, for setting up connection pool and being connected to the distributed file system；
Path acquiring unit, for obtaining and opening the upload path corresponding to the destination folder；
File write unit, for writing file into the corresponding upload path；
Identifying unit is operated, whether the operation for judging to upload the file to be uploaded succeeds；
Path closing unit, if for uploading successfully, deleting the file uploaded, closes the corresponding file road
Footpath；
Daily record updating block, if for not uploading success, output error message simultaneously updates system journal.
A kind of equipment for uploading files to distributed file system, wherein, the equipment include processor, memory and
The computer program on the memory is stored in, the computer program realizes following steps when being executed by processor：
Obtain file to be uploaded；
System banner according to belonging to the file to be uploaded, obtains corresponding with the system banner in distributed file system
Destination folder；
The file to be uploaded is uploaded into the destination folder.
The above-mentioned equipment for uploading files to distributed file system, it is possible to achieve artificial hand is replaced using automatic method
Dynamic processing, on the one hand can avoid the increase of human cost, on the other hand can also effectively reduce operational error odds.
As a kind of embodiment, wherein, the step of the acquisition of computing device file to be uploaded before also
Including：
Obtain and upload record；
If obtaining successfully, system banner and the corresponding file sequence number of system banner are obtained in the upload record；
The file to be uploaded is obtained according to the system banner and the corresponding file sequence number of system banner.
As a kind of embodiment, wherein, the described of the computing device according to the system banner and is
The step of corresponding file sequence number of system mark obtains the file to be uploaded includes：
Whether judge the file sequence number is initial value；
If so, then obtaining first file in the corresponding file of the system banner；
If it is not, then obtaining the file to be uploaded according to the file sequence number.
As a kind of embodiment, wherein, the described of the computing device uploads into the destination folder
The step of file to be uploaded, includes：
Whether the process for monitoring the upper transmitting file interrupts；
If so, then obtaining the sequence number of the system banner and upper transmitting file belonging to upper transmitting file；
The sequence number of system banner and upper transmitting file according to belonging to obtaining upper transmitting file, which is created or updated, uploads record.
As a kind of embodiment, wherein, the step of the acquisition of computing device file to be uploaded it
It is preceding also to include：
Local directory is searched for, system banner set is obtained；
According to system banner set, create or update system banner caching.
As a kind of embodiment, wherein, the search local directory of the computing device obtains system mark
The step of knowing set includes：
Opened every the very first time interval being pre-configured with and read local directory；
Whether judge the file type under local directory successively is file；
If so, then obtaining the title of the file, and the title of the file is added into the system banner set
In；
If it is not, then continuing to judge next file type under catalogue, until obtaining files all under local directory
Title.
As a kind of embodiment, wherein, wrap the step of the acquisition of computing device file to be uploaded
Include：
Every the second time interval run-down system banner caching being pre-configured with；
System banner sequence opened file folder successively in being cached according to the system banner；
Check in the file of the opening with the presence or absence of the file for needing to upload；
If there is the file for needing to upload, the file to be uploaded is obtained；
If in the absence of the file for needing to upload, continuing to open next file, until the system banner sequence
Corresponding file is examined in row.
As a kind of embodiment, wherein, wrap the step of the acquisition of computing device file to be uploaded
Include：
Read file to be uploaded；
Judge whether to read successfully；
If reading successfully, the transport stream of the file to be uploaded is obtained, local data is read；
If reading is unsuccessful, program sleep, the length of one's sleep is the 3rd time interval being pre-configured with, and obtains treat again
Transmitting file.
As a kind of embodiment, wherein, the described of the computing device uploads into the destination folder
The step of file to be uploaded, includes：
Set up connection pool and be connected to the distributed file system；
Obtain and open the upload path corresponding to the destination folder；
File is write into the corresponding upload path；
Judge whether the operation for uploading the file to be uploaded succeeds；
If uploading successfully, the file uploaded is deleted, the corresponding file path is closed；
If not uploading success, output error message simultaneously updates system journal.
It should be understood by those skilled in the art that, embodiments herein can be provided as method, system or computer program
Product.Therefore, the application can be using the reality in terms of complete hardware embodiment, complete software embodiment or combination software and hardware
Apply the form of example.Moreover, the application can be used in one or more computers for wherein including computer usable program code
The computer program production that usable storage medium is implemented on (including but is not limited to magnetic disk storage, CD-ROM, optical memory etc.)
The form of product.
The application is the flow with reference to method, equipment (system) and computer program product according to the embodiment of the present application
Figure and/or block diagram are described.It should be understood that can be by every first-class in computer program instructions implementation process figure and/or block diagram
Journey and/or the flow in square frame and flow chart and/or block diagram and/or the combination of square frame.These computer programs can be provided
The processor of all-purpose computer, special-purpose computer, Embedded Processor or other programmable data processing devices is instructed to produce
A raw machine so that produced by the instruction of computer or the computing device of other programmable data processing devices for real
The device for the function of being specified in present one flow of flow chart or one square frame of multiple flows and/or block diagram or multiple square frames.
These computer program instructions, which may be alternatively stored in, can guide computer or other programmable data processing devices with spy
Determine in the computer-readable memory that mode works so that the instruction being stored in the computer-readable memory, which is produced, to be included referring to
Make the manufacture of device, the command device realize in one flow of flow chart or multiple flows and/or one square frame of block diagram or
The function of being specified in multiple square frames.
These computer program instructions can be also loaded into computer or other programmable data processing devices so that in meter
Series of operation steps is performed on calculation machine or other programmable devices to produce computer implemented processing, thus in computer or
The instruction performed on other programmable devices is provided for realizing in one flow of flow chart or multiple flows and/or block diagram one
The step of function of being specified in individual square frame or multiple square frames.
Each technical characteristic of embodiment described above can be combined arbitrarily, to make description succinct, not to above-mentioned reality
Apply all possible combination of each technical characteristic in example to be all described, as long as however, the combination of these technical characteristics is not deposited
In contradiction, the scope of this specification record is all considered to be.
Embodiment described above only expresses the several embodiments of the present invention, and it describes more specific and detailed, but simultaneously
Can not therefore it be construed as limiting the scope of the patent.It should be pointed out that coming for one of ordinary skill in the art
Say, without departing from the inventive concept of the premise, various modifications and improvements can be made, these belong to the protection of the present invention
Scope.Therefore, the protection domain of patent of the present invention should be determined by the appended claims.
Claims (19)
1. a kind of method for uploading files to distributed file system, it is characterised in that methods described includes：
Obtain file to be uploaded；
System banner according to belonging to the file to be uploaded, obtains corresponding with the system banner in distributed file system
Destination folder；
The file to be uploaded is uploaded into the destination folder.
2. according to the method described in claim 1, it is characterised in that also include before the step of acquisition file to be uploaded：
Obtain and upload record；
If obtaining successfully, system banner and the corresponding file sequence number of system banner are obtained in the upload record；
The file to be uploaded is obtained according to the system banner and the corresponding file sequence number of system banner.
3. method according to claim 2, it is characterised in that described corresponding according to the system banner and system banner
File sequence number obtain the file to be uploaded the step of include：
Whether judge the file sequence number is initial value；
If so, then obtaining first file in the corresponding file of the system banner；
If it is not, then obtaining the file to be uploaded according to the file sequence number.
4. method according to claim 2, it is characterised in that it is described uploaded into the destination folder it is described to be uploaded
The step of file, includes：
Whether the process for monitoring the upper transmitting file interrupts；
If so, then obtaining the sequence number of the system banner and upper transmitting file belonging to upper transmitting file；
The sequence number of system banner and upper transmitting file according to belonging to obtaining upper transmitting file, which is created or updated, uploads record.
5. according to the method described in claim 1, it is characterised in that also include before the step of acquisition file to be uploaded：
Local directory is searched for, system banner set is obtained；
According to system banner set, create or update system banner caching.
6. method according to claim 5, it is characterised in that the search local directory, obtains system banner set
Step includes：
Opened every the very first time interval being pre-configured with and read local directory；
Whether judge the file type under local directory successively is file；
If so, then obtaining the title of the file, and the title of the file is added in the system banner set；
If it is not, then continuing to judge next file type under catalogue, the name until obtaining files all under local directory
Claim.
7. method according to claim 5, it is characterised in that include the step of acquisition file to be uploaded：
Every the second time interval run-down system banner caching being pre-configured with；
System banner sequence opened file folder successively in being cached according to the system banner；
Check in the file of the opening with the presence or absence of the file for needing to upload；
If there is the file for needing to upload, the file to be uploaded is obtained；
If in the absence of the file for needing to upload, continuing to open next file, until in the system banner sequence
Corresponding file is examined.
8. according to the method described in claim 1, it is characterised in that include the step of acquisition file to be uploaded：Reading is treated
Upper transmitting file；
Judge whether to read successfully；
If reading successfully, the transport stream of the file to be uploaded is obtained, local data is read；
If reading is unsuccessful, program sleep, the length of one's sleep is the 3rd time interval being pre-configured with, and text to be uploaded is obtained again
Part.
9. according to the method described in claim 1, it is characterised in that it is described uploaded into the destination folder it is described to be uploaded
The step of file, includes：
Set up connection pool and be connected to the distributed file system；
Obtain and open the upload path corresponding to the destination folder；
File is write into the corresponding upload path；
Judge whether the operation for uploading the file to be uploaded succeeds；
If uploading successfully, the file uploaded is deleted, the corresponding file path is closed；
If not uploading success, output error message simultaneously updates system journal.
10. a kind of device for uploading files to distributed file system, it is characterised in that described device includes：
File uploading module, for obtaining file to be uploaded；
Destination folder acquisition module, for the system banner according to belonging to the file to be uploaded, obtains distributed field system
Destination folder corresponding with the system banner in system；
File uploading module, for uploading the file to be uploaded into the destination folder.
11. device according to claim 10, it is characterised in that described device also includes：
Acquisition module is recorded, record is uploaded for obtaining；
Sequence number acquisition module, if for obtaining successfully, system banner and system banner pair are obtained in the upload record
The file sequence number answered；
Ff module, for obtaining described to be uploaded according to the system banner and the corresponding file sequence number of system banner
File.
12. device according to claim 11, it is characterised in that the ff module also includes：
Sequence number judging unit, for judging whether the file sequence number is initial value；
Mark acquiring unit, for obtaining first file in the system banner；
Sequence number searching unit, for obtaining the file to be uploaded according to the file sequence number.
13. device according to claim 11, it is characterised in that the file uploading module includes：
Monitoring unit is interrupted, whether the process for monitoring the upper transmitting file interrupts；
Mark acquiring unit, for if so, then obtaining the sequence number of the system banner and upper transmitting file belonging to upper transmitting file；
Updating block is recorded, is created for the system banner and the sequence number of upper transmitting file according to obtaining upper transmitting file or more
It is new to upload record.
14. device according to claim 10, it is characterised in that described device also includes：
Gather acquisition module, for searching for local directory, obtain system banner set；
Buffer update module, for according to system banner set, creating or updating system banner caching.
15. device according to claim 14, it is characterised in that the set acquisition module includes：
Catalogue reading unit, for being opened every the very first time interval being pre-configured with and reading local directory；
Type judging unit, for judging whether the file type under local directory is file successively；
Gather updating block, for if so, then obtain the title of the file, and the title of the file added described
In system banner set；
Catalogue judging unit, for if it is not, then continuing to judge next file type under catalogue, until obtaining under local directory
All system banners.
16. device according to claim 14, it is characterised in that the file acquisition module includes：
Cache sweep unit, for every the second time interval run-down system banner caching being pre-configured with；
File opening unit, for the system banner sequence in system banner caching successively opened file folder；
With the presence or absence of the file for needing to upload in file checking unit, the file for checking the opening, if existing described
The file uploaded is needed, then obtains the file to be uploaded, if in the absence of the file for needing to upload, continuing to open next
Individual file, until file corresponding in the system banner sequence is examined.
17. device according to claim 10, it is characterised in that the file acquisition module includes：
Document reading unit, for reading file to be uploaded；
Condition adjudgement unit, for judging whether to read successfully；
Data capture unit, if for reading successfully, obtaining the transport stream of the file to be uploaded, reads local data；
Program sleep cell, if unsuccessful for reading, program sleep, the length of one's sleep is between the 3rd time being pre-configured with
Every obtaining file to be uploaded again.
18. device according to claim 10, it is characterised in that the uploading module includes：
Connection pool sets up unit, for setting up connection pool and being connected to the distributed file system；
Path acquiring unit, for obtaining and opening the upload path corresponding to the destination folder；
File write unit, for writing file into the corresponding upload path；
Identifying unit is operated, whether the operation for judging to upload the file to be uploaded succeeds；
Path closing unit, if for uploading successfully, deleting the file uploaded, closes the corresponding file path；
Daily record updating block, if for not uploading success, output error message simultaneously updates system journal.
19. a kind of equipment for uploading files to distributed file system, it is characterised in that the equipment includes processor, storage
Device and the computer program being stored on the memory, the computer program realize that right will when being executed by processor
Seek the step in method any one of 1-9.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710282637.8A CN107193881A (en) | 2017-04-26 | 2017-04-26 | Upload files to method, device and the equipment of distributed file system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201710282637.8A CN107193881A (en) | 2017-04-26 | 2017-04-26 | Upload files to method, device and the equipment of distributed file system |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN107193881A true CN107193881A (en) | 2017-09-22 |
Family
ID=59872314
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201710282637.8A Pending CN107193881A (en) | 2017-04-26 | 2017-04-26 | Upload files to method, device and the equipment of distributed file system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN107193881A (en) |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108920487A (en) * | 2018-05-04 | 2018-11-30 | 天津猎鹰网络技术有限公司 | The method and apparatus that data upload |
| CN109144946A (en) * | 2018-07-24 | 2019-01-04 | 中国建设银行股份有限公司 | A kind of document handling method and device |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103955544A (en) * | 2014-05-21 | 2014-07-30 | 北京奇艺世纪科技有限公司 | File uploading method and file uploading system |
| CN105100170A (en) * | 2014-05-21 | 2015-11-25 | 浙江海洋学院 | Automatic cloud storage synchronous management system for a plurality of cloud storage service platforms and method thereof |
| CN105227610A (en) * | 2014-06-30 | 2016-01-06 | 北京金山安全软件有限公司 | File uploading and storing method and device |
| CN105743978A (en) * | 2016-01-29 | 2016-07-06 | 武汉联影医疗科技有限公司 | Data transmitting method |
-
2017
- 2017-04-26 CN CN201710282637.8A patent/CN107193881A/en active Pending
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103955544A (en) * | 2014-05-21 | 2014-07-30 | 北京奇艺世纪科技有限公司 | File uploading method and file uploading system |
| CN105100170A (en) * | 2014-05-21 | 2015-11-25 | 浙江海洋学院 | Automatic cloud storage synchronous management system for a plurality of cloud storage service platforms and method thereof |
| CN105227610A (en) * | 2014-06-30 | 2016-01-06 | 北京金山安全软件有限公司 | File uploading and storing method and device |
| CN105743978A (en) * | 2016-01-29 | 2016-07-06 | 武汉联影医疗科技有限公司 | Data transmitting method |
Cited By (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN108920487A (en) * | 2018-05-04 | 2018-11-30 | 天津猎鹰网络技术有限公司 | The method and apparatus that data upload |
| CN109144946A (en) * | 2018-07-24 | 2019-01-04 | 中国建设银行股份有限公司 | A kind of document handling method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103984579B (en) | More equipment rooms share the method for current application program real-time running state | |
| US20090070355A1 (en) | Transitioning between historical and real time data streams in the processing of data change messages | |
| CN107710215A (en) | The method and apparatus of mobile computing device safety in test facilities | |
| CN108052374A (en) | A kind of method and device of deployment container micro services | |
| CN110096685A (en) | A kind of device identification generation method and device | |
| CN106503158A (en) | Method of data synchronization and device | |
| CN109327557A (en) | A kind of management method and device of virtual machine IP address | |
| CN103684900B (en) | Business method for inspecting and system | |
| CN104461929B (en) | Distributed data cache method based on blocker | |
| CN106021566A (en) | Method, device and system for improving concurrent processing capacity of single database | |
| CN114780370A (en) | Data correction method and device based on log, electronic equipment and storage medium | |
| CN108153793A (en) | A kind of original data processing method | |
| CN109977170A (en) | The synchronous method and system of network data inside and outside a kind of government affairs | |
| CN105893456B (en) | The isolated method and system of the computing basic facility of geography fence perception | |
| CN107193881A (en) | Upload files to method, device and the equipment of distributed file system | |
| CN106713045A (en) | Terminal device driving component configuration method and device | |
| CN106033438A (en) | Public sentiment data storage method and server | |
| CN110191364B (en) | Video data migration method, storage medium, device and system | |
| CN106528300A (en) | Console game synchronizing method, device and terminal | |
| CN107316176A (en) | The control method and device of declaration form processing | |
| CN103984633B (en) | A kind of bank main passes down the automatization test system of operation | |
| CN109657167B (en) | Data acquisition method, device, server and storage medium | |
| CN107272669A (en) | A kind of airborne Fault Management System | |
| CN106294387A (en) | Data migration method and device | |
| CN108984757A (en) | A kind of data lead-in method and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| RJ01 | Rejection of invention patent application after publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20170922 |