CN101651825A - Method and system for acquiring and uploading data - Google Patents
Method and system for acquiring and uploading data Download PDFInfo
- Publication number
- CN101651825A CN101651825A CN200810118353A CN200810118353A CN101651825A CN 101651825 A CN101651825 A CN 101651825A CN 200810118353 A CN200810118353 A CN 200810118353A CN 200810118353 A CN200810118353 A CN 200810118353A CN 101651825 A CN101651825 A CN 101651825A
- Authority
- CN
- China
- Prior art keywords
- data
- uploading
- temporary file
- buffer memory
- data acquisition
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses a method and a system for acquiring and uploading data, belongs to the technical field of data transmission, and is designed for solving the problem that acquired video and audio data according to the plan cannot realize quick and reliable uploading in the prior art. The method for acquiring and uploading the data comprises the following steps: acquiring the data and generating a temporary file; and caching partial data in the temporary file in turn, and uploading the partial data. The system for acquiring and uploading the data comprises a data acquiring and generatingunit used for acquiring the data and generating the temporary file, and a caching and uploading unit used for caching the partial data in the temporary file in turn and uploading the partial data.
Description
Technical field
The present invention relates to transfer of data, particularly a kind of data acquisition method for uploading and system.
Background technology
In recent years, along with the quick propelling of TV station's digitlization process and developing rapidly of Internet video application, the comprehensive solution of various multimedia digital management has occupied a large amount of market shares in traditional and emerging media marketplace.Loading system on the video acquisition, as one of system of core the most in the solution, finish various videos, the voice data of analog or digital signal are gathered from source device and caught, configuration generates compressed file according to difference, and uploads to other servers to finish the function of operations such as editor, warehouse-in, backup, retrieval.As seen, loading system is the foundation stone that other system is finished corresponding function in the solution on the video acquisition, and designing a last loading system of efficient, reliable collection all has very important meaning to performance and the stability that improves whole solution.
Loading system is general on traditional video acquisition adopts two kinds of methods to finish to upload: first kind is that looking of gathering according to plan, voice data directly upload to editor or warehouse-in server by host-host protocol, and on acquisition server, do not generate temporary file, the obvious so local I/O (I/O) that reduced, improved certain efficient, in case but network occurs unusually, gather file and in upload process, will produce loss of data, thereby the quality that causes gathering file descends, reliability is low; Second kind is that looking of gathering according to plan, voice data generate temporary file on acquisition server, when waiting this plan to finish, be after temporary file generates, again by FTP (FileTransfer Protocol, file transfer protocol (FTP)) or HTTP (Hypertext Transfer Protocol, HTML (Hypertext Markup Language)) with this temporary file upload to editor or the warehouse-in server.When network occurs when unusual, the data of losing in the upload process can send again by temporary file, thereby ensured the quality of gathering file, improved reliability, but the shortcoming of this method maximum is, video editing system must could obtain data and begin editor, the obvious like this time that increases whole workflow after acquisition system has been gathered whole file.
Summary of the invention
The invention provides a kind of data acquisition method for uploading and system, be used to solve according to plan looking of gathering, problem that voice data can not be realized fast and reliably upload.
For achieving the above object, the present invention adopts following technical scheme:
A kind of data acquisition method for uploading comprises:
Image data also generates temporary file;
Partial data in the described temporary file of buffer memory successively, and upload described partial data.
A kind of data acquisition uploading system comprises:
Data acquisition generation unit: be used for image data and generate temporary file;
Buffer memory uploading unit: be used for the partial data of the described temporary file of buffer memory successively, and upload described partial data.
Among the present invention, the generation of temporary file can be eliminated the interference that network brings unusually, improves the reliability of gathering file; In addition in the generative process of temporary file, the present invention is by buffer memory successively and upload partial data in the temporary file, has saved the time, can realize uploading fast of data.Therefore the present invention can make looking of gathering according to plan, voice data realize uploading fast and reliably.
Description of drawings
Fig. 1 is the flow chart of data acquisition method for uploading of the present invention;
Fig. 2 is for using the system structural framework schematic diagram of method shown in Figure 1;
Fig. 3 is the logical construction schematic diagram of memory buffer in the system shown in Figure 2 structural framing;
Fig. 4 is the algorithm basic flow sheet that adopts when transmitting data by memory buffer shown in Figure 3;
Fig. 5 is the structural representation of data acquisition uploading system of the present invention.
Embodiment
The invention provides a kind of data acquisition method for uploading and system that can realize fast and reliably upload, the present invention is described in detail below in conjunction with accompanying drawing
As shown in Figure 1 and Figure 2, data acquisition method for uploading of the present invention comprises:
Step 101: image data also generates temporary file;
Acquisition server generates temporary file in this locality after having gathered data, can improve the reliability of gathering file like this.
Step 102: the partial data in the described temporary file of buffer memory successively, and upload described partial data.
To edit as early as possible, put in storage or the purpose of other operations gathering file in order to reach, the present invention adopts a memory buffer to come data in the buffer memory temporary file, partial data in the buffer memory temporary file and uploading is successively each time all uploaded until the data of whole temporary file and is finished.Therefore in the process that temporary file generates, the present invention just can distribute by local area network (LAN) or wide area network the data in the temporary file fast to the server of editor, warehouse-in or other types.
Among the present invention, the generation of temporary file can be eliminated the interference that network brings unusually, improves the reliability of gathering file; In addition in the generative process of temporary file, the present invention is by buffer memory successively and upload partial data in the temporary file, has saved the time, can realize uploading fast of data.Therefore the present invention can make looking of gathering according to plan, voice data realize uploading fast and reliably.
In addition, partial data in the step 102 in the described temporary file of buffer memory carries out buffer memory by the circulation one-dimension array, it is the structure that the memory buffer structure optimization adopts simple circulation one-dimension array among the present invention, this is for following 2 considerations: the one, and media data has continuity, so use one-dimension array more convenient; The 2nd, buffering is based on funnel model, flow to data on one side, flow out data on one side, so adopt the circulation array, with the initial and end position of two pointers demarcation memory buffers, like this, in the process of dynamic data inflow and outflow, the mobile one-dimensional data of copy that need not be frequent has reduced the internal memory I/O of big data quantity (media data is all bigger), has improved efficient.
When memory buffer adopts the circulation one-dimension array, owner pointer and tail pointer are set respectively, and described initial and end position is decided to be data outflow end and data inflow end respectively in the initial and end position of circulation one-dimension array, at this moment,
Partial data in the step 102 in the described temporary file of buffer memory is specially:
Flow into end in data, reading section data from described temporary file, and store the tail pointer place of described circulation one-dimension array into;
Described tail pointer is according to the data volume of data streams read, mobile backward respective distance simultaneously.
Uploading described data in the step 102 is specially:
At the data outflow end,, and upload the data that read from the owner pointer reading of data of described circulation one-dimension array;
Described owner pointer is according to the data volume of data streams read, mobile backward respective distance simultaneously.
Said data are video data and/or voice data among the present invention.
Describe the transmission of data when adopting the circulation one-dimension array below in detail.
As shown in Figure 3, this circulation one-dimension array has divided several region into, and this sentences, and four districts are respectively I, II, III, IV are example.The initial and end pointer points to starting position, I district (the most left curve arrow sends the place) and starting position, III district respectively when initial, the data of having represented when initial in the buffering among the I and II district buffer memory, III and IV district are free buffer, and owner pointer has pointed to the data that arrive first.Then, inflow along with data, the III district has deposited the data of this inflow in, according to the data volume that flows into data, tail pointer has also pointed to the starting position of IV accordingly, and along with the outflow of data, the data of I district storage are handed out, according to the data volume that flows out data, owner pointer has also pointed to the position of II district beginning accordingly.Like this, II and III district buffer memory data, I and IV district become free buffer.Owing to used the circulation one-dimension array, so just store the I district after flowing into the full IV district of storage, the circulation of initial and end pointer is moved and is made that I and IV district logically are continuous.
Flow into end in data, can open the asynchronous reading of data from current temporary file of thread and store the tail pointer of memory buffer into, establish T
ReadThe time interval of expression reading of data, T
ReadBe given thresholding, the data volume that at every turn reads is the data that generated in blanking time.Because capture program is to gather spanned file according to the code check of configuration, the data bits that reads from temporary file is approximately R at every turn
Bit* T
Read, R
BitThe code check (bit leads) that temporary file is provided with is gathered in expression.
At the data outflow end, also open asynchronous after the owner pointer reading of data of memory buffer (one-dimension array promptly circulates), distributing of thread to transmission network.If T
WriteExpression sends the time interval of data, T
WriteBe given thresholding.Here, we distribute according to the principle of the fastest transmission, and the branch that is about to data in buffer maximum possible in the current memory buffer sends out, and establishes M
UseBe data in buffer amount in the current memory buffer, μ is instant transmission network bandwidth (bps), and then instant bandwidth is approximately T at full capacity
Write* μ is a Time Bandwidth fully loaded data volume because the data volume maximum of each distribution can not surpass, so each data volume that sends is approximately min (M
Use, T
Write* μ).
Unusual (such as shake occurring, the network bandwidth occupied by other system etc.) occurring and cause when transmission network is that Time Bandwidth μ reduces, thereby causes each data volume min (M that sends
Use, T
Write* μ) reduce, so, flow into end, if the data volume that reads next time less than the free space size of current memory buffer, then continues the data that read to the storage of the tail pointer place of memory buffer in data; Otherwise,, suspend reading of data according to the principle that prevents the memory cache overflow, guarantee that the phenomenon of overflow does not appear in buffering area, send enough data, vacate the free space of buffering up to the outflow end branch, the data volume that reads feasible next time is just restarted reading of data during less than free space size.
The basic flow sheet of algorithm can be with reference to figure 4 in the above-mentioned data transmission procedure.At first, calculate the free space in the current memory buffer, then, judge whether memory buffer is full, if then end to read the data in the temporary file, and after waiting for that trigger data flows into the end line journey, return initial step: calculate the free space in the current memory buffer; If not, then read the data in the temporary file, and after waiting for trigger data outflow end thread, according to the fastest transmission principle distributing data to transmission network, after waiting for that then trigger data flows into the end line journey, return initial step: calculate the free space in the current memory buffer.
To sum up, the method that data acquisition of the present invention is uploaded flows into and outflow end in data, carries out according to the principle of the fastest transmission, meets the demand that the data that as far as possible will generate fast are distributed to other servers; When network occurs when unusual in short-term, the data that buffering reads in the magnitude range of memory buffer free space, and data cached branch is sent out by the fastest transmission principle in the normal back of network recovery, vacate free space, reading of data guarantees that flow process is unobstructed as far as possible; When network occurs when long carrying out the operation that suspending/resuming reads according to the principle that prevents the buffer memory overflow when unusual, guaranteed that data can not lose, meet reliable demand to system.So, the flow process that method of the present invention can be supported fast, reliable video acquisition is uploaded.
Simultaneously as shown in Figure 5, data acquisition uploading system of the present invention comprises:
Data acquisition generation unit 501: be used for image data and generate temporary file;
Buffer memory uploading unit 502: be used for the partial data of the described temporary file of buffer memory successively, and upload described partial data.
Wherein, data acquisition generation unit 501 is an acquisition server, and buffer memory uploading unit 502 comprises buffer unit 5021 and uploading unit 5022.Buffer unit 5021: be used for by the circulation one-dimension array partial data of the described temporary file of buffer memory successively; Uploading unit 5022: be used to upload described partial data.
Described system also comprises: receiving element 503 is used for receiving data by transmission network from described buffer memory uploading unit.This receiving element 503 is one or more the combination in server, backup server and the retrieval server of editing server, warehouse-in, also can be the server of other type.In the present embodiment, described data are video data and/or voice data.
In the present embodiment, the generation of temporary file can be eliminated the interference that network brings unusually, improves the reliability of gathering file; In addition in the generative process of temporary file, the present invention is by buffer memory successively and upload partial data in the temporary file, has saved the time, can realize uploading fast of data.Therefore the present invention can make looking of gathering according to plan, voice data realize uploading fast and reliably.
The above; only be the specific embodiment of the present invention, but protection scope of the present invention is not limited thereto, anyly is familiar with those skilled in the art in the technical scope that the present invention discloses; can expect easily changing or replacing, all should be encompassed within protection scope of the present invention.Therefore, protection scope of the present invention should be as the criterion with the protection range of claim.
Claims (11)
1, a kind of data acquisition method for uploading is characterized in that, comprising:
Image data also generates temporary file;
Partial data in the described temporary file of buffer memory successively, and upload described partial data.
2, data acquisition method for uploading according to claim 1 is characterized in that, the partial data in the described temporary file of described buffer memory carries out buffer memory by the circulation one-dimension array.
3, data acquisition method for uploading according to claim 2 is characterized in that, the initial and end position of described circulation one-dimension array is respectively equipped with owner pointer and tail pointer, and described initial and end position is respectively the data outflow end and data flow into end.
4, data acquisition method for uploading according to claim 3 is characterized in that, the partial data in the described temporary file of described buffer memory is specially:
Flow into end in data, reading section data from described temporary file, and store the tail pointer place of described circulation one-dimension array into;
Described tail pointer is according to the data volume of data streams read, mobile backward respective distance simultaneously.
5, data acquisition method for uploading according to claim 4 is characterized in that, described method also comprises:
Flow into end in data, if the data volume that reads next time less than the free space size of described circulation one-dimension array, is then stored the data that read to the tail pointer place of described circulation one-dimension array from described temporary file; Otherwise, then suspend reading of data.
6, data acquisition method for uploading according to claim 4 is characterized in that, describedly uploads described data and is specially:
At the data outflow end,, and upload the data that read from the owner pointer reading of data of described circulation one-dimension array;
Described owner pointer is according to the data volume of data streams read, mobile backward respective distance simultaneously.
According to the described data acquisition method for uploading of arbitrary claim in the claim 1 to 6, it is characterized in that 7, described data are video data and/or voice data.
8, a kind of data acquisition uploading system is characterized in that, comprising:
Data acquisition generation unit: be used for image data and generate temporary file;
Buffer memory uploading unit: be used for the partial data of the described temporary file of buffer memory successively, and upload described partial data.
9, data acquisition uploading system according to claim 8 is characterized in that, described data acquisition generation unit is an acquisition server, and described buffer memory uploading unit comprises buffer unit and uploading unit, wherein,
Described buffer unit: be used for by the circulation one-dimension array partial data of the described temporary file of buffer memory successively;
Described uploading unit: be used to upload described partial data.
10, data acquisition uploading system according to claim 8 is characterized in that, described system also comprises:
Receiving element: be used for receiving data from described buffer memory uploading unit by transmission network;
Wherein, described receiving element is one or more the combination in server, backup server and the retrieval server of editing server, warehouse-in.
11, the described data acquisition uploading system of arbitrary claim in 10 according to Claim 8 is characterized in that described data are video data and/or voice data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200810118353A CN101651825A (en) | 2008-08-13 | 2008-08-13 | Method and system for acquiring and uploading data |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN200810118353A CN101651825A (en) | 2008-08-13 | 2008-08-13 | Method and system for acquiring and uploading data |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN101651825A true CN101651825A (en) | 2010-02-17 |
Family
ID=41673910
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN200810118353A Pending CN101651825A (en) | 2008-08-13 | 2008-08-13 | Method and system for acquiring and uploading data |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN101651825A (en) |
Cited By (12)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101860621A (en) * | 2010-06-07 | 2010-10-13 | 中兴通讯股份有限公司 | Mobile terminal file uploading method and mobile terminal |
| CN101895426A (en) * | 2010-07-20 | 2010-11-24 | 中兴通讯股份有限公司 | Data acquisition method and system in network resource evaluation |
| CN101917487A (en) * | 2010-08-23 | 2010-12-15 | 中国电信股份有限公司 | Method and system for optimizing communication between mobile terminal and server |
| CN104639890A (en) * | 2013-11-14 | 2015-05-20 | 株式会社日立制作所 | Facility monitoring control system |
| CN105721884A (en) * | 2016-04-26 | 2016-06-29 | 武汉斗鱼网络科技有限公司 | Privacy protection method and device for live broadcast |
| WO2017092603A1 (en) * | 2015-12-04 | 2017-06-08 | 阿里巴巴集团控股有限公司 | Data transmission method and device |
| CN106874145A (en) * | 2017-02-15 | 2017-06-20 | 山东浪潮商用系统有限公司 | A kind of asynchronous data backup method based on message queue |
| CN107018040A (en) * | 2017-02-27 | 2017-08-04 | 杭州天宽科技有限公司 | A kind of port data collection, the implementation method for caching and showing |
| CN108319251A (en) * | 2017-12-29 | 2018-07-24 | 中车唐山机车车辆有限公司 | Train data acquisition method and device |
| CN109714415A (en) * | 2018-12-26 | 2019-05-03 | 北京小米移动软件有限公司 | Data processing method and device |
| CN109713790A (en) * | 2018-12-19 | 2019-05-03 | 国网山东省电力公司无棣县供电公司 | A kind of power grid regulation information acquisition system |
| CN109727437A (en) * | 2018-12-19 | 2019-05-07 | 国网山东省电力公司无棣县供电公司 | A kind of long-distance wireless meter-reading system |
-
2008
- 2008-08-13 CN CN200810118353A patent/CN101651825A/en active Pending
Cited By (17)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101860621A (en) * | 2010-06-07 | 2010-10-13 | 中兴通讯股份有限公司 | Mobile terminal file uploading method and mobile terminal |
| WO2011153756A1 (en) * | 2010-06-07 | 2011-12-15 | 中兴通讯股份有限公司 | Method for uploading files of mobile terminal and mobile terminal thereof |
| CN101895426A (en) * | 2010-07-20 | 2010-11-24 | 中兴通讯股份有限公司 | Data acquisition method and system in network resource evaluation |
| CN101917487A (en) * | 2010-08-23 | 2010-12-15 | 中国电信股份有限公司 | Method and system for optimizing communication between mobile terminal and server |
| CN101917487B (en) * | 2010-08-23 | 2013-06-26 | 中国电信股份有限公司 | Method and system for optimizing communication between mobile terminal and server |
| CN104639890A (en) * | 2013-11-14 | 2015-05-20 | 株式会社日立制作所 | Facility monitoring control system |
| CN104639890B (en) * | 2013-11-14 | 2018-01-09 | 株式会社日立制作所 | System of monitoring and controlling plant |
| CN106844075A (en) * | 2015-12-04 | 2017-06-13 | 阿里巴巴集团控股有限公司 | A kind of data transferring method and equipment |
| WO2017092603A1 (en) * | 2015-12-04 | 2017-06-08 | 阿里巴巴集团控股有限公司 | Data transmission method and device |
| CN105721884A (en) * | 2016-04-26 | 2016-06-29 | 武汉斗鱼网络科技有限公司 | Privacy protection method and device for live broadcast |
| CN106874145A (en) * | 2017-02-15 | 2017-06-20 | 山东浪潮商用系统有限公司 | A kind of asynchronous data backup method based on message queue |
| CN107018040A (en) * | 2017-02-27 | 2017-08-04 | 杭州天宽科技有限公司 | A kind of port data collection, the implementation method for caching and showing |
| CN108319251A (en) * | 2017-12-29 | 2018-07-24 | 中车唐山机车车辆有限公司 | Train data acquisition method and device |
| CN109713790A (en) * | 2018-12-19 | 2019-05-03 | 国网山东省电力公司无棣县供电公司 | A kind of power grid regulation information acquisition system |
| CN109727437A (en) * | 2018-12-19 | 2019-05-07 | 国网山东省电力公司无棣县供电公司 | A kind of long-distance wireless meter-reading system |
| CN109714415A (en) * | 2018-12-26 | 2019-05-03 | 北京小米移动软件有限公司 | Data processing method and device |
| CN109714415B (en) * | 2018-12-26 | 2021-09-21 | 北京小米移动软件有限公司 | Data processing method and device |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN101651825A (en) | Method and system for acquiring and uploading data | |
| CN104935648B (en) | The CDN system and file of a kind of high performance-price ratio push away in advance, the method for fragment cache memory | |
| CN108848060B (en) | Multimedia file processing method, processing system and computer readable storage medium | |
| CN101060418B (en) | Special disk reading and writing system suitable for IPTV direct broadcast server with time shift | |
| CN101753947A (en) | Method and device for video-on-demand playback of video monitoring recording | |
| CN102468989B (en) | The method and system of network data | |
| CN101729836B (en) | Method for videoing bottom acquisition device | |
| CN103152423A (en) | Cloud storage system and data access method thereof | |
| CN102547444B (en) | Method for optimizing play of real-time video monitoring record | |
| CN108881942B (en) | Super-fusion normal state recorded broadcast system based on distributed object storage | |
| WO2009143741A1 (en) | Method, system and apparatus for playing media files on demand | |
| KR102274466B1 (en) | Video streaming method using real time caching technique and system thereof | |
| CN104639900B (en) | RTMP turns M3U8 video segment system and method | |
| CN101170719B (en) | Multi-file content processing method and system | |
| CN108960889A (en) | Voice says room progress monitoring method and device in house virtual three-dimensional space | |
| CN109189577A (en) | A kind of data prevent memory from overflowing method and apparatus when synchronous | |
| CN102243615B (en) | Memory buffer management method supporting audio/video frame storage | |
| CN110381128B (en) | Uploading method and cloud storage model suitable for streaming media file | |
| CN102137138A (en) | Method, device and system for cache collaboration | |
| CN109587232A (en) | Document breakpoint transmission method based on iOS network | |
| CN102325275A (en) | Packet discarding method, device and server for playing of streaming media | |
| CN111163297A (en) | Method for realizing high concurrency and quick playback of video monitoring cloud storage | |
| CN105472421A (en) | Video program collecting, editing and making method and system based on public cloud | |
| CN105978955A (en) | Mobile video synchronization system, method and mobile terminal | |
| CN110290396A (en) | A kind of method, apparatus, system, equipment and the storage medium of video processing |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C12 | Rejection of a patent application after its publication | ||
| RJ01 | Rejection of invention patent application after publication |
Application publication date: 20100217 |