CN103678699A - Method and system for merging data in server - Google Patents
Method and system for merging data in server Download PDFInfo
- Publication number
- CN103678699A CN103678699A CN201310741755.2A CN201310741755A CN103678699A CN 103678699 A CN103678699 A CN 103678699A CN 201310741755 A CN201310741755 A CN 201310741755A CN 103678699 A CN103678699 A CN 103678699A
- Authority
- CN
- China
- Prior art keywords
- data file
- file
- cutting
- server
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/10—File systems; File servers
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/06—Digital input from, or digital output to, record carriers, e.g. RAID, emulated record carriers or networked record carriers
- G06F3/0601—Interfaces specially adapted for storage systems
- G06F3/0628—Interfaces specially adapted for storage systems making use of a particular technique
- G06F3/0638—Organizing or formatting or addressing of data
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Human Computer Interaction (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The invention discloses a method for merging data in a server. The method includes the following steps that a data file is received; the data file is placed into a corresponding internal queue; the data file in the internal queue is acquired, and the segmentation number is determined according to the size of the data file; the data file is segmented according to the segmentation number to acquire slices with different sizes; pre-established write-in threads are called respectively corresponding to the slices with different sizes, and the slices are sequentially added and written into an object file, corresponding to the data file, in a distributed-type file device in the server. By means of the method, the operation of series flow write-in does not need stopping, so that new slices are merged at the tail end of the file when the slices are opened all the time, and accordingly the speed of merging the slices into the data file can be improved.
Description
Technical field
The application relates to the method and system of data transmission, and the data that are specifically related in a kind of server merge method and system.
Background technology
Along with the Fast Growth of network traffics, many services that are not easy in the past to realize also can be completed easily.For example: broadband upload multimedia file (multimedia file).Because the transmission of magnanimity still can affect the load of server.Therefore at known multimedia server, also can possess the function of Bandwidth Management (Bandwidth Manager).Server can see through the modes such as bandwidth distribution or buffer memory and alleviate the data transmission amount of same time.
In addition, described transmission mode more combines a plurality of entity disks, and what make that data can be synchronous writes in described disk.And prior art writes the mode of disk to data, be that the synchronous origin source of data is write in server.Server can be kept in these data files.Afterwards, server by these data files in batches write entity disk.
Realizing of this mode writing is with low cost and quick.But the writing mode in past is to be a plurality of cutting fragments by the data file cutting of uploading.In server inside, these cutting fragments are reconfigured.The Writing Technology in past is that after each cutting fragment is write, worker thread can be closed byte stream (I/O Stream).When next cutting fragment will continue, write on after data file, another worker thread just can be opened byte stream.Although such processing mode can realize disposable data and write.But the switch of a plurality of worker threads can cause the delay writing.And make a mistake in the process writing, can make the current data file having recorded damage thereupon.
Therefore the time that data are transmitted again will be elongated, and can increase the load of the network bandwidth.In addition, same one data file is except will being again written into from client, and will again transmit in server inside.
Summary of the invention
The application's technical matters to be solved is to provide the merging method of the data in a kind of server, improves the merging processing speed of the cutting data in server.
In order to address the above problem, the application has separately disclosed a kind of transmission method of data cutting, comprises the following steps: received data file; Described data file is positioned in corresponding internal queues; Obtain the described data file in described internal queues and determine cutting quantity according to the file size of described data file; According to described cutting quantity, described data file is carried out to cutting, and obtain the cutting fragment of a plurality of different sizes; Corresponding to the cutting fragment of different sizes, call respectively the thread that writes being pre-created, described cutting fragment is sequentially added to the file destination corresponding with this data file in the decentralized file device that writes server.If there is not described file destination in the decentralized file device of server, create the described file destination corresponding with this data file.
Further, described in obtain the described data file in described internal queues, and determine cutting quantity according to the file size of described data file, specifically comprise: obtain the described data file in described internal queues; Read the file size of described data file; Corresponding to different file sizes and the data writing length restriction that writes thread being pre-created, determine cutting quantity.
Further, described method also comprises: described call be pre-created write thread before, judge whether the file destination corresponding with this data file exists; When the described file destination corresponding with this data file do not exist, create the file destination corresponding with this data file.
Describedly according to described cutting quantity, described data file is carried out to cutting, and obtain the cutting fragments of a plurality of different sizes, be specially: according to described cutting quantity, determine cut-off, described data file is carried out to pre-cutting; When the cutting position of described data file falls within the tail end of described internal queues, and before the cut-off of described data file is positioned at described tail end, revise the cutting position of described data file to described cut-off.
Comprise further: if the length of described cutting position and tail end is less than length threshold values, reduce the length of described cutting fragment.
Comprise further: if the length of described cutting position and tail end is greater than described length threshold values, extend the length of described cutting fragment.
In order to address the above problem, the data combination system that the application has disclosed in a kind of server comprises:
Local directory, receives the data file that client sends;
Scanning module, is connected in described local directory, and described scanning module monitors the whether unusual fluctuation of data file of described local directory;
Control module, be coupled to described scanning module, described control module comprises at least internal queues, described internal queues stores the path information of corresponding described data file, during the arbitrary described data file unusual fluctuation of described local directory, described scanning module makes described control module create new described internal queues according to the kind of the data file of unusual fluctuation or upgrades the content of known described data file;
Upload module, be connected in described local directory, during the arbitrary described data file unusual fluctuation of described local directory, described in upload a kind of document of the described data file of module detecting unusual fluctuation, and according to described kind of document, described data file is assigned to corresponding described internal queues;
At least one queue monitor module, be connected in described control module, described queue monitor module produces the queue monitor module of respective amount according to the quantity of described internal queues, described queue monitor module determines the cutting quantity of described data file according to the size of the described data file of described internal queues, and according to described cutting quantity, described data file is carried out to cutting to produce corresponding cutting fragment; And
A plurality of worker threads, are connected in described queue monitor module, and described worker thread reads corresponding described cutting fragment;
Wherein, described queue monitor mould is according to the assign the job writing position of the corresponding described file destination of thread of the file header of described file destination, worker thread, according to affiliated said write position and in the mode of additional treatments, writes described file destination the corresponding described cutting fragment of worker thread described in each.
Described queue monitor module takes out described data file from described internal queues according to the mode of first in first out to the data file of monitored described internal queues.
When described queue monitor module confirms that the cutting position of described data file falls within a tail end of described internal queues during in cutting, and before a cut-off of described data file is positioned at described tail end, the cutting position that described queue monitor module is revised described data file is to described cut-off.
Further, worker thread writes cutting fragment in described file destination in the mode walking abreast.
Further, described system also comprises a decentralized file device, is connected in worker thread, and worker thread is by cutting fragment writing in described decentralized file device sequentially.
Compared with prior art, the application can obtain and comprise following technique effect:
1) the application can not need to close to the merging of cutting fragment the action that crossfire writes, so new cutting fragment is incorporated in to the tail end of file at the state of cutting fragment one direct-open.Therefore can accelerate to merge the speed that cutting fragment is data file.
2) the application improves the transmitting procedure of data file, if data file is write fashionable making a mistake, the application does not need again from client, to obtain data file.
3), if the application makes a mistake in during transmission data file, the application can see through at least one heartbeat server and transmit messages to backstage, make that backstage personnel can be real-time to learn that what assembly occurs abnormal.
Certainly, arbitrary product of enforcement the application must not necessarily need to reach above-described all technique effects simultaneously.
Accompanying drawing explanation
Accompanying drawing described herein is used to provide further understanding of the present application, forms the application's a part, and the application's schematic description and description is used for explaining the application, does not form the improper restriction to the application.In the accompanying drawings:
Fig. 1 is the configuration diagram of the embodiment of the present application.
Fig. 2 is the operation workflow schematic diagram of the embodiment of the present application.
Fig. 3 A is that the first worker thread of the application and the cutting fragment of the second worker thread write schematic diagram.
Fig. 3 B is that the first worker thread of the application and the cutting fragment of the second worker thread write schematic diagram.
Embodiment
To coordinate drawings and Examples to describe the application's embodiment in detail below, by this application's implementation procedure how application technology means solve technical matters and reach technology effect can be fully understood and be implemented according to this.
As used some vocabulary in the middle of instructions and claim, censure specific components.Those skilled in the art should understand, and hardware manufacturer may be called same assembly with different nouns.This specification and claims are not used as distinguishing the mode of assembly with the difference of title, but the difference in function is used as the criterion of distinguishing with assembly.If " comprising " mentioned in the middle of instructions and claim is in the whole text an open language, therefore should be construed to " comprise but be not limited to "." roughly " refer to that in receivable error range, those skilled in the art can solve the technical problem within the scope of certain error, reach described technique effect substantially.In addition, " couple " word and comprise directly any and electric property coupling means indirectly at this.Therefore, if describe a first device in literary composition, be coupled to one second device, represent that described first device can directly be electrically coupled to described the second device, or be indirectly electrically coupled to described the second device by other devices or the means that couple.Instructions subsequent descriptions is for implementing the application's better embodiment, and right described description is to illustrate that the application's rule is object, not in order to limit the application's scope.The application's protection domain is when being as the criterion depending on the claims person of defining.
Data upload system in the application's server, please refer to Fig. 1.The application's transmission system comprises: local side 110 and many heartbeat servers 120.And local side 110 is consisted of following element: local directory 111, upper transmission module 112, scan module 113, control module 114, queue monitor module 115, worker thread 116 and decentralized file device 117.
And each item described in the application is carried out following steps:
Received data file;
Described data file is positioned in corresponding internal queues;
Obtain the described data file in described internal queues, and determine cutting quantity according to the file size of described data file;
According to described cutting quantity, described data file is carried out to cutting, and obtain the cutting fragment of a plurality of different sizes;
Corresponding to the cutting fragment of different sizes, call respectively the thread that writes being pre-created, described cutting fragment is sequentially added to the file destination corresponding with this data file in the decentralized file device 117 that writes server.
Upper transmission module 112 is connected in local directory 111 and control module 114.Scan module 113 is connected in local directory 111 and control module 114.For can be simply and whole function mode is clearly described, therefore the first heartbeat server 121 and the second heartbeat server 122 be set in this application.But in fact the quantity of heartbeat server 120 can determine according to factors such as the service ability of server, bandwidth or securities.
Control module 114 is connected in scan module 113, upper transmission module 112, queue monitor module 115, the first heartbeat server 121 and the second heartbeat server 122.Each local directory 111 can be deposited data file separately.For example: video file can be stored puts video local directory, and promotional literature is put as in advertisement local directory.Generally speaking, the data file that local directory 111 meetings reception client regularly sends.To some extent whether the file in scan module 113 supervision local directories 111 unusual fluctuation.Described file unusual fluctuation refers to increasing newly, move, revise or deleting of data.File in local directory 111 during unusual fluctuation, is gone up transmission module 112 and the data file of unusual fluctuation will be passed to control module 114 in the path in server to some extent.
In addition the kind of document of the data file of 111 unusual fluctuations of upper transmission module 112 detecting local directories.Described in the broadest sense kind of document may be audio file, movie file or text file etc.Furthermore, audio file may be subdivided into mp3, m4a, AAC or ARM etc. again; Movie file can be subdivided into mov, flv, m4v, mkv or mp4 etc. again.
If while there is not the internal queues 1141 of kind of data file in control module 114, upper transmission module can generate new internal queues 1141 by drive control module 114.Thereafter, if there is the data file of one species to be written into, upper transmission module 112 can be put into corresponding internal queues 1141 by the data file being newly written into.
Control module 114 can be classified successively and produce each self-corresponding internal queues according to received file path, and different internal queues can assign different queue monitor modules 115 to process.In Fig. 1, be to using the first internal queues, the second internal queues and the 3rd internal queues as signal.The cutting information of the file under each queue monitor module 115 can record.When described cutting information refers to file and is uploaded to local directory 111, file size is unusual fluctuation thereupon also.Before data file cutting, queue monitor module 115 is first taken out correct element (first element of queue) from first internal queues, then just carries out cutting.And being mode according to first in first out (First In First Out, FIFO), the data taking-up mode of internal queues realizes.
115 pairs of files of queue monitor module carry out isometric cutting.After the cutting that completes file, the fragment of each file cutting is defined as cutting fragment (slice).The stacking quantity of the internal queues 1141 under 115 detectings of queue monitor module surpasses upper prescribing a time limit, and queue monitor module 115 can be sent warning message to the first heartbeat server 121 and the second heartbeat server 122.
And the quantity of cutting fragment is to determine according to the performance of local side 110.For example: 1 worker thread 116 can be processed 50,000,000 cutting fragment, the data file of 200,000,000 is distributed to 116 processing of 4 worker threads so.In like manner, 500,000,000 data file, can distribute to 10 worker threads 116 and process.
The relevent information recording during for cutting is called cutting information.For example, each cutting length during cutting and the side-play amount in disk (offset) of document location.When scan module 113 finds that the data file in local directory 111 is deleted, scan module 113 can be deleted the corresponding internal queues 1141 in control module 114.
Then, different cutting fragments can be assigned each self-corresponding worker thread 116, as the data file that reads local directory 111.The realization of worker thread 116 can be realized by different program languages.Do not limit in this application the kind of program language, for example, JAVA language or ASP language.In fact, queue monitor module 115 is the data file of local directory 111 to be carried out to the estimation of cutting, in order to obtain each position that should described will be split of described data file.And in fact carry out file cutting processing to as if by queue monitor module 115, realized.Queue monitor module 115 is understood according to the length of cutting fragment after obtaining cutting fragment, and divides and task the affiliated partial document data of worker thread 116.Worker thread 116 is writing the data file of described part in decentralized file device 117.
Generally speaking, the first heartbeat server 121 or the second heartbeat server 122 surpass one default during after do not receive heartbeat requirement, heartbeat server 120 can be to sending warning message.But when worker thread 116 makes a mistake in the process writing.Worker thread 116 can send warning message to the first heartbeat server 121 and the second heartbeat server 122.Described worker thread 116 also can be notified other worker threads 116 with the time, requires other worker threads 116 to stop the processing that writes to decentralized file device 117, uses the data integrity of avoiding in decentralized file device 117.
In cutting fragment, write in the process of decentralized file device 117, the application's worker thread reads corresponding cutting fragment and according to the order that writes, adjusts the merging process of data file except meeting, please coordinate and shown in Fig. 2, understand this method:
Step S200: data file is uploaded to local directory;
Step S210: data file is positioned in internal queues by controlling module;
Step S220: queue monitor module is obtained the data file in internal queues and determined cutting quantity according to the file size of data file;
Step S230: queue monitor module carries out cutting according to cutting quantity to data file, and obtain a plurality of cutting fragments;
Step S240: configure a plurality of worker threads, the cutting fragment under each worker thread reads;
Step S250: judge whether there is file destination in decentralized file device;
Step S260: if there is not file destination in decentralized file device, create file destination in decentralized file device;
Step S270: if there is file destination in decentralized file device, queue monitor mould is according to the assign the job writing position of the corresponding file destination of thread of the file header of file destination; And
Step S280: worker thread, according to affiliated writing position and in the mode of additional treatments, writes file destination the corresponding cutting fragment of every worker thread.
For example, in server, have in advance internal queues, and be pre-created queue monitoring thread, dissimilar internal queues corresponding to data file.Server receives after the data file of uploading, and data file is positioned in corresponding internal queues.The corresponding establishment of different internal queues has different worker threads, for cutting data file, obtains data slot, and the data slot obtaining after cutting is written in the file destination on server.
When the data file that monitors queue when queue monitoring thread changes, whether the file destination for storing data files exists, if there is no first create file destination, and in file header data writing storage information, such as file type and memory location etc.; Then, queue monitoring thread starts a plurality of worker threads, obtains respectively the data file in internal queues, and determines cutting quantity according to the file size of data file, and data file is cut into a plurality of cutting fragments.Then, the cutting fragment that the parallel data file cutting by different queue of a plurality of worker threads obtains is written to file destination.By method of the present invention, do not need to close the action that crossfire writes, so the tail end that new cutting fragment is incorporated in to file at the state of cutting fragment one direct-open can accelerate to merge the speed that cutting fragment is data file.
Local directory 111 comes from the data file of client except receiving, and the data file in local directory 111 also may be modified, move or the unusual fluctuation such as deletion.Whether the data file that queue monitor module 115 is understood in real-time detecting local directory 111 has unusual fluctuation.When data file unusual fluctuation, queue monitor module 115 upwards transmission module 112 is sent the message of data file unusual fluctuation.
When having new data file to deposit local directory 111 in, scan module 113 can make control module set up new internal queues 1141, and deposits data file in new internal queues 1141.Generally speaking, the restricted number of internal queues 1141 depends on the service ability of server.If but have too much data file newly to create, may there is the stacking problem of internal queues 1141.Once there be excessive data file to be created, scan module 113 can see through control module 114 and send warning requirement to heartbeat server.
When the content of existing data file is to some extent during unusual fluctuation, scan module 113 can deposit data file after changing in internal queues 1141 again.If when local directory 111 is deleted data file, scan module 113 can require control module 114 that the internal queues of corresponding data file 1141 is deleted.
The kind of document of the data file of 111 unusual fluctuations of upper transmission module 112 detecting local directories.Whether upper transmission module 112 exists according to kind of document, and upper transmission module 112 requires control module 113 create new internal queues 1141 or data file is put into corresponding internal queues 1141.Upper transmission module 112 is obtained after the message of data file unusual fluctuation, and upper transmission module 112 can again be obtained the pathname of data file and pathname is stored in internal queues 1141.For different data files, unique internal queues 1141 can be set.When local directory 111 stores a large amount of data files, the quantity of the internal queues 1141 in control module 114 also can be grown up fast.
Because too much internal queues 1141 can cause excessive stackingly, can make the data file of handling that follow-up worker thread 116 cannot be instant.Whether the volume of services that therefore the application can monitor internal queues 1141 is over the pressure threshold value of setting.When volume of services overpressure threshold values, control module 114 can be sent warning message to the first heartbeat server 121 or the second heartbeat server 122.
Then, according to the quantity of cutting fragment, produce corresponding worker thread 116.The reference position of each worker thread 116 basis cutting fragment separately and the Partial Fragment that side-play amount obtains data file.Queue monitor module 115 turns cutting fragment by worker thread 116 and starts to read.Worker thread 116 writes the fragment obtaining in decentralized file device 117.For clearly demonstrating the application's process, at this, following element is done to further definition: the worker thread that reads two interdependent cutting fragments is defined as respectively the first worker thread 410 and the second worker thread 420.So be only to using two worker threads as explanation for conveniently illustrating at this, but in fact the quantity of worker thread is not limited to this.In addition be the file of energy distinguishes data file and decentralized file device 117, therefore the data file after merging is defined as file destination 430.
Please refer to Fig. 3 A and Fig. 3 B, be respectively the first worker thread of the application and the cutting fragment of the second worker thread writes schematic diagram.In the time of can creating file destination 430 in the process writing, queue monitor module 115 can first create a file header about each cutting fragment write sequence according to data file.The file size that each worker thread can be recorded according to file header is inserted cutting fragment on corresponding position.A plurality of worker threads 116 that belong to same group in the application are written in parallel in the mode that writes same file destination 430.And follow-up cutting fragment utilizes the mode of additional treatments (append) to write file destination 430 writing fashionable meeting.During due to additional treatments, do not need to close crossfire, so other front 116 journeys do not need to wait for the processing that can walk abreast.
For instance, the data file of wanting to write is video file, and each cutting fragment is the length of 10 minutes.Starting that original every 10 minutes cutting fragments are write writes file destination 430 and merges.The first worker thread 410 and the second worker thread 420 are after obtaining file header, and the first worker thread 410 and the second worker thread 420 can parallel writing in file destination 430.
In the process that writes decentralized file device 117, may make a mistake, so the application is when detecting the content of the cutting fragment writing when wrong, can send warning requirement to the first heartbeat server 121 or the second heartbeat server 122.
Also it should be noted that, term " comprises ", " comprising " or its any other variant are intended to contain comprising of nonexcludability, thereby the process, method, commodity or the system that make to comprise a series of key elements not only comprise those key elements, but also comprise other key elements of clearly not listing, or be also included as the intrinsic key element of this process, method, commodity or system.The in the situation that of more restrictions not, the key element being limited by statement " comprising ... ", and be not precluded within process, method, commodity or the system that comprises described key element and also have other identical element.
The embodiment that it will be understood by those skilled in the art that the application can be provided as method, system or computer program.Therefore, the application can adopt complete hardware implementation example, implement software example or in conjunction with the form of the embodiment of software and hardware aspect completely.And the application can adopt the form that wherein includes the upper computer program of implementing of computer-usable storage medium (including but not limited to magnetic disk memory, CD-ROM, optical memory etc.) of computer usable program code one or more.
The embodiment that the foregoing is only the application, is not limited to the application.To those skilled in the art, the application can have various modifications and variations.Any modification of doing within all spirit in the application and principle, be equal to replacement, improvement etc., within all should being included in the application's claim scope.
Claims (10)
1. the data merging method in a server, is characterized in that, comprises the following steps:
Received data file;
Described data file is positioned in corresponding internal queues;
Obtain the described data file in described internal queues, and determine cutting quantity according to the file size of described data file;
According to described cutting quantity, described data file is carried out to cutting, and obtain the cutting fragment of a plurality of different sizes;
Corresponding to the cutting fragment of different sizes, call respectively the thread that writes being pre-created, described cutting fragment is sequentially added to the file destination corresponding with this data file in the decentralized file device that writes server.
2. the data merging method in server as claimed in claim 1, is characterized in that, described method also comprises:
If there is not described file destination in the decentralized file device of server, create the described file destination corresponding with this data file.
3. the data merging method in server as claimed in claim 1, is characterized in that, described in obtain the described data file in described internal queues, and determine cutting quantity according to the file size of described data file, specifically comprise:
Obtain the described data file in described internal queues;
Read the file size of described data file;
Corresponding to different file sizes and the data writing length restriction that writes thread being pre-created, determine cutting quantity.
4. the data merging method in server as claimed in claim 1, is characterized in that, described method also comprises:
Described call be pre-created write thread before, judge whether the file destination corresponding with this data file exists;
When the described file destination corresponding with this data file do not exist, create the file destination corresponding with this data file.
5. the data merging method in the server as described in claim 1 to 4 any one, is characterized in that, describedly according to described cutting quantity, described data file is carried out to cutting, and obtains the cutting fragments of a plurality of different sizes, is specially:
According to described cutting quantity, determine cut-off, described data file is carried out to pre-cutting;
When the cutting position of described data file falls within the tail end of described internal queues, and before the cut-off of described data file is positioned at described tail end, revise the cutting position of described data file to described cut-off.
6. the data combination system in server, is characterized in that, comprising:
Local directory, receives the data file that client sends;
Scanning module, is connected in described local directory, and described scanning module monitors the whether unusual fluctuation of data file of described local directory;
Control module, be coupled to described scanning module, described control module comprises at least internal queues, described internal queues stores the path information of corresponding described data file, during the arbitrary described data file unusual fluctuation of described local directory, described scanning module makes described control module create new described internal queues according to the kind of the data file of unusual fluctuation or upgrades the content of known described data file;
Upload module, be connected in described local directory, during the arbitrary described data file unusual fluctuation of described local directory, described in upload a kind of document of the described data file of module detecting unusual fluctuation, and according to described kind of document, described data file is assigned to corresponding described internal queues;
At least one queue monitor module, be connected in described control module, described queue monitor module produces the queue monitor module of respective amount according to the quantity of described internal queues, described queue monitor module determines the cutting quantity of described data file according to the size of the described data file of described internal queues, and according to described cutting quantity, described data file is carried out to cutting to produce corresponding cutting fragment; And
A plurality of worker threads, are connected in described queue monitor module, and described worker thread reads corresponding described cutting fragment;
Wherein, described queue monitor mould is according to the assign the job writing position of the corresponding described file destination of thread of the file header of described file destination, worker thread, according to affiliated said write position and in the mode of additional treatments, writes described file destination the corresponding described cutting fragment of worker thread described in each.
7. the data combination system in server as claimed in claim 7, is characterized in that, described queue monitor module takes out described data file from described internal queues according to the mode of first in first out to the data file of monitored described internal queues.
8. the data combination system in server as claimed in claim 6, it is characterized in that, when described queue monitor module confirms that the cutting position of described data file falls within a tail end of described internal queues during in cutting, and before a cut-off of described data file is positioned at described tail end, the cutting position that described queue monitor module is revised described data file is to described cut-off.
9. the data combination system in server as claimed in claim 6, is characterized in that, further work thread writes cutting fragment in described file destination in the mode walking abreast.
10. the data combination system in server as claimed in claim 6, is characterized in that, further comprises a decentralized file device, is connected in worker thread, and worker thread is by cutting fragment writing in described decentralized file device sequentially.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310741755.2A CN103678699A (en) | 2013-12-27 | 2013-12-27 | Method and system for merging data in server |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201310741755.2A CN103678699A (en) | 2013-12-27 | 2013-12-27 | Method and system for merging data in server |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103678699A true CN103678699A (en) | 2014-03-26 |
Family
ID=50316243
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201310741755.2A Pending CN103678699A (en) | 2013-12-27 | 2013-12-27 | Method and system for merging data in server |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103678699A (en) |
Cited By (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103997514A (en) * | 2014-04-23 | 2014-08-20 | 汉柏科技有限公司 | File parallel transmission method and system |
| CN104360824A (en) * | 2014-11-10 | 2015-02-18 | 北京奇虎科技有限公司 | Data merging method and device |
| CN106156049A (en) * | 2015-03-27 | 2016-11-23 | 天脉聚源（北京）科技有限公司 | A kind of method and system of digital independent |
| CN112328375A (en) * | 2020-10-28 | 2021-02-05 | 华北电力大学（保定） | Correlation method and device for tracking data segments of distributed system |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1434606A (en) * | 2002-01-24 | 2003-08-06 | 华为技术有限公司 | Data segmenting and reloading method in ATM communication system |
| CN102790771A (en) * | 2012-07-25 | 2012-11-21 | 山东中创软件商用中间件股份有限公司 | File transmission method and system |
| CN103092840A (en) * | 2011-10-28 | 2013-05-08 | 上海邮电设计咨询研究院有限公司 | Method for acquiring self-increment mass data files from multiple sources |
-
2013
- 2013-12-27 CN CN201310741755.2A patent/CN103678699A/en active Pending
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1434606A (en) * | 2002-01-24 | 2003-08-06 | 华为技术有限公司 | Data segmenting and reloading method in ATM communication system |
| CN103092840A (en) * | 2011-10-28 | 2013-05-08 | 上海邮电设计咨询研究院有限公司 | Method for acquiring self-increment mass data files from multiple sources |
| CN102790771A (en) * | 2012-07-25 | 2012-11-21 | 山东中创软件商用中间件股份有限公司 | File transmission method and system |
Cited By (6)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN103997514A (en) * | 2014-04-23 | 2014-08-20 | 汉柏科技有限公司 | File parallel transmission method and system |
| CN104360824A (en) * | 2014-11-10 | 2015-02-18 | 北京奇虎科技有限公司 | Data merging method and device |
| CN104360824B (en) * | 2014-11-10 | 2017-12-12 | 北京奇虎科技有限公司 | The method and apparatus that a kind of data merge |
| CN106156049A (en) * | 2015-03-27 | 2016-11-23 | 天脉聚源（北京）科技有限公司 | A kind of method and system of digital independent |
| CN112328375A (en) * | 2020-10-28 | 2021-02-05 | 华北电力大学（保定） | Correlation method and device for tracking data segments of distributed system |
| CN112328375B (en) * | 2020-10-28 | 2022-03-22 | 华北电力大学（保定） | Correlation method and device for tracking data segments of distributed system |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110019240B (en) | Service data interaction method, device and system | |
| CN103699495A (en) | Transmission device and transmission system for splitting data | |
| CN103701907A (en) | Processing method and system for continuing to transmit data in server | |
| US20190188086A1 (en) | Redundancy reduction in blockchains | |
| US7925856B1 (en) | Method and apparatus for maintaining an amount of reserve space using virtual placeholders | |
| US9654433B2 (en) | Selective message republishing to subscriber subsets in a publish-subscribe model | |
| CN103678699A (en) | Method and system for merging data in server | |
| CN112965839B (en) | Message transmission method, device, equipment and storage medium | |
| US11315605B2 (en) | Method, device, and computer program product for storing and providing video | |
| CN112019605A (en) | Data distribution method and system of data stream | |
| US11743516B2 (en) | Content boundary based recordings | |
| CN106897431B (en) | Log export method and system | |
| CN110413588B (en) | Distributed object storage method and device, computer equipment and storage medium | |
| CN109787813B (en) | SDN-based service chain configuration method, device, system and medium | |
| CN103699666A (en) | Transmission method and transmission device for splitting data | |
| CN103685559A (en) | Method and system for processing data in server | |
| CN103701667A (en) | Method, device and system for monitoring heartbeat of server | |
| CN103684929A (en) | System and method for monitoring server status | |
| CN103092932A (en) | Distributed document transcoding system | |
| RU2698766C1 (en) | Method and device for transmitting, sending and receiving information | |
| CN115801765A (en) | File transmission method, device, system, electronic equipment and storage medium | |
| CN107210996B (en) | Service chain management method and device | |
| CN114897532A (en) | Operation log processing method, system, device, equipment and storage medium | |
| US9705833B2 (en) | Event driven dynamic multi-purpose internet mail extensions (MIME) parser | |
| CN112423162B (en) | Configuration synchronization method and system under distributed architecture |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C41 | Transfer of patent application or patent right or utility model | ||
| TA01 | Transfer of patent application right |
Effective date of registration: 20151228 Address after: Room six, building 19, building 68, No. 100089 South Road, Haidian District, Beijing Applicant after: LETV CLOUD COMPUTING CO., LTD. Address before: Room six, building 19, building 68, No. 100089 South Road, Haidian District, Beijing Applicant before: LeTV Information Technology (Beijing) Co., Ltd. |
|
| AD01 | Patent right deemed abandoned |
Effective date of abandoning: 20181109 |
|
| AD01 | Patent right deemed abandoned |