CN103279521A - Video big data distributed decoding method based on Hadoop - Google Patents
Video big data distributed decoding method based on Hadoop Download PDFInfo
- Publication number
- CN103279521A CN103279521A CN2013102039001A CN201310203900A CN103279521A CN 103279521 A CN103279521 A CN 103279521A CN 2013102039001 A CN2013102039001 A CN 2013102039001A CN 201310203900 A CN201310203900 A CN 201310203900A CN 103279521 A CN103279521 A CN 103279521A
- Authority
- CN
- China
- Prior art keywords
- decoding
- video
- hadoop
- distributed
- hdfs
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
Images
Landscapes
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The invention discloses a video big data distributed decoding method based on Hadoop and belongs to the field of computers. The method mainly comprises the steps of uploading massive video files to a Hadoop distributed file system (HDFS) directly to be stored, mounting the HDFS onto a local file system with fuse_dfs so that files in the HDFS can be accessed with an uniform method, amending the data partitioning strategy of a MapReduce computing framework of the Hadoop and taking image frames as portioning boundaries so that the problems of frame splitting and decoding failure caused by byte-based partition can be solved, acquiring the public information needed by decoding through the HDFS mounted onto the local file system and accomplishing video big data distributed decoding by means of the MapReduce computing framework and an FFmpeg decoding bank, and inputting a decoding result as the Map of MapReduce for subsequent intelligent video analysis. Experimental results show that the method can effectively improve the decoding efficiency of massive video files, and the accuracy rate of the decoding reaches 100%.
Description
Technical field
The invention belongs to computer realm, be specifically related to the distributed coding/decoding method of the big data of a kind of video based on Hadoop.
Background technology
Video data comprises a large amount of useful informations, is a heat subject in the computer vision research field to video analysis.Take up space greatly because video has institute, in storage and transmission, all need to compress with conserve storage and transmission bandwidth.Yet intelligent video analysis at first needs compressed video is decoded, obtain original picture frame after, just can carry out follow-up analytical work.Traditional decoding scheme lays particular emphasis on the efficient and the accuracy rate that improve decoding algorithm based on one-of-a-kind system, and this still can tackle demand in the age that video data does not possess certain scale; When the scale of video data during considerably beyond the processing power of unit, this decoding scheme becomes the performance bottleneck of intelligent video analysis.
The Hadoop platform has been used to do distributed video decoding work, but require earlier with the video segmentation software video to be divided into one by one small video less than HDFS block size (being generally 64M), be uploaded to HDFS again, do not support the problem of video file with the dividing method of avoiding the HDFS inherence.Yet this method requires prior divided video, has not only increased the decoding workload, and when handling massive video data, makes this method be difficult to carry out.
For this reason, patent of the present invention has proposed a kind of new technical scheme, has solved the deficiency that exists in the existing magnanimity video distribution formula coding/decoding method, has effectively improved decoding efficiency, and decoding rate of accuracy reached to 100%.
Summary of the invention
Goal of the invention: at the deficiency of existing method existence, the invention provides the distributed coding/decoding method of the big data of a kind of video based on Hadoop, solve the inefficient problem of magnanimity video decode, be implemented in the decoding work of finishing the big data of video in the low hardware environment that disposes.
Technical scheme: for solving the problems of the technologies described above, the present invention adopts following technical scheme: the distributed coding/decoding method of the big data of a kind of video based on Hadoop specifically comprises the steps:
Step a: the magnanimity video file directly is uploaded among the Hadoop distributed file system HDFS stores;
Step b: with fuse_dfs HDFS is mounted to local file system, to visit the file among the HDFS with unified method;
Step c: revise the data segmentation strategy of MapReduce calculating framework among the Hadoop,, solved by byte and cut apart the frame division that causes and the problem that can't decode as partitioning boundary with picture frame;
Steps d: read the required public information of decoding from the HDFS that is mounted to local file system, the distributed decoding of magnanimity video is finished in recycling MapReduce calculating framework and FFmpeg decoding storehouse;
Step e: the Map input of decoded result as MapReduce, be used for follow-up intelligent video analysis.
With respect to prior art, the present invention has following beneficial effect:
1, the present invention directly is stored in the magnanimity video among the distributed file system HDFS, need not the big data of prior divided video, has made things convenient for the storage of magnanimity video.
2, the present invention has revised the data segmentation strategy among the Hadoop,, has solved Hadoop and has cut apart the frame division that causes, the problem that can't decode by byte as partitioning boundary with picture frame.
3, the present invention is based on the Hadoop platform, the video decode task is distributed in the cluster environment of being made up of the computing machine of low configuration carries out, both saved system operation cost, and improved decoding efficiency again, decoded picture frame can be directly used in follow-up intelligent video analysis simultaneously.
Description of drawings
Fig. 1: the one-piece construction figure of method
Fig. 2: with the MapReduce logical division process flow diagram of frame of video position as partitioning boundary
Fig. 3: with the MapReduce logical division result of frame of video position as partitioning boundary
The distributed decoding of Fig. 4 and the contrast of unit decode time
The contrast of the distributed decoding of Fig. 5 and unit decoding efficiency
Embodiment
Below in conjunction with drawings and Examples technology of the present invention is described further, should understand these embodiment only is used for explanation the present invention and is not used in and limits the scope of the invention, after reading the present invention, those skilled in the art all fall within the application's gained claim institute restricted portion to the modification of the various equivalent form of values of invention.
As shown in Figure 1, the invention provides the distributed coding/decoding method of the big data of a kind of video based on Hadoop, specifically comprise the steps:
Step a: the magnanimity video file directly is uploaded among the distributed file system HDFS of Hadoop and stores:
Step b: with fuse_dfs HDFS is mounted to local file system, to visit the file among the HDFS with unified method;
Step c: revise the data segmentation strategy of MapReduce calculating framework among the Hadoop,, solved by byte and cut apart the frame division that causes and the problem that can't decode as partitioning boundary with picture frame:
C1: realized user-defined ImageInputFormat class, such is inherited from the FileInputFormat class.In the ImageInputFormat class, the createRecordReader method is returned an ImageRecordReader object; The isSplitable method is returned true, supposes that video file is always divisible;
C2: realized user-defined ImageRecordReader class, such is inherited from the RecordReader class.In the ImageRecordReader class, the initialize method is at first created a HFFmpegFrameGrabber decoder object, and the reference position of this object is made as the reference position of this split; By HFFmpegFrameGrabber object acquisition next frame image, judge whether this frame is key frame, if be key frame, the start frame of this frame as this split then; If be not key frame, continue to obtain downwards up to obtaining key frame then as the start frame of this split;
C3: in the ImageRecordReader class, the nextKeyValue method judges that at first whether the current location of HFFmpegFrameGrabber object has exceeded the end position of this split, if do not exceed, continues to obtain the next frame image; If exceed, judge whether next frame is key frame, if be key frame, stops to obtain, this split finishes; If be not key frame, continue to obtain downwards up to obtaining key frame, stop then obtaining, this split finishes.The logical division flow process as shown in Figure 2, logical division result as shown in Figure 3,1,6,11,16,21 is key frame among Fig. 3, all the other are non-key frame.
Steps d: obtain the required public information of decoding by the HDFS that is mounted to local file system, the distributed decoding of magnanimity video is finished in recycling MapReduce calculating framework and FFmpeg decoding storehouse;
D1: having realized the HFFmpegFrameGrabber class, is a realization of HframeGrabber interface, and the coding/decoding method that such has mainly encapsulated Ffmpeg decoding storehouse obtains the required public information of decoding by the HDFS that is mapped to this locality and carries out distributed decoding.In the HFFmpegFrameGrabber class, the setBytePos method is used for arranging the start byte position of demoder; The getImage method is used for obtaining next frame image and decoding; The isKeyFrame method is used for judging whether present frame is key frame;
D2: realized the FrameNumWritable class, inherited from WritableComparable that such is used for the play position that identification frames is positioned at sequence of frames of video, as key (Key) type of Map input; Realized the ImageWritable class, inherited from Writable that such is used for depositing the view data of frame, as value (Value) type of Map input;
D3: after the image decoding, at first the frame number in this split is stored in the FrameNumWritable object as the unique identification of frame the reference position of split and frame, and the view data IplImage of frame is stored in the ImageWritable object; Then these two objects are passed to the Map method as key (Key) and value (Value) respectively, as the input of Map method;
Step e: the Map input of decoded result as MapReduce, be used for follow-up intelligent video analysis
E1: after the user realizes the MapReduce application program of oneself, in the Map method, obtain decoded view data by key (Key) and value (Value), be used for follow-up intelligent video analysis;
After the e2:Map distributed treatment finished, the output result imported as Reduce and does the merger processing, finally finishes calculating.
Embodiment:
In the present embodiment, the Hadoop cluster is made up of 15 PCs, and every PC CPU is Intel (R) Pentium (R) 4CPU2.80GHz, in save as 1.5G, hard disk is 80G, wherein 1 as cluster Master, 14 as cluster Slaves; The video that is used for decoding is that size is respectively 15M, 30M, 60M, 100M, 300M, 500M, 1G, 2G, 6G, 12G, the AVI video of 24G.This example carries out the decoding of distributed decoding and unit respectively with the video of above-mentioned different sizes, and decode time, decoding efficiency and the accuracy rate of two kinds of decoding processes have been done contrast.
The time contrast of distributed decoding and unit decoding as shown in Figure 4.As can be seen from Figure 4, along with video increases gradually, distributed decode time is far smaller than the unit decode time.The efficient contrast of distributed decoding and unit decoding as shown in Figure 5.As can be seen from Figure 5, increase along with video, unit decoding efficiency level off, and distributed decoding efficiency during less than 60M, is lower than the unit decoding efficiency at video, at video during greater than 60M, progressively be higher than the unit decoding efficiency, and begin during greater than 6G to descend at video, the reason of decline is because experiment condition is limited, is subjected to the influence of clustered node quantity and cluster processing power.
Distributed decoding accuracy rate test data is the following video file of 1G, and the decoded video of unit is designated as E, and distributed decoded video is designated as A.Test result is as shown in table 1, and attribute is respectively the number percent that comprises frame among the E among the number percent that comprises frame among the A in the frame number, E of frame number, the A of decoding back E, the A in the table.In the implementation process, unit and distributed decoded frame number are identical, and A and E comprise frame number percent mutually and be 100%.Hence one can see that, is accurately based on the distributed decoding of Hadoop, the situation of frame and wrong frame occurs leaking.
Table 1
| Video size (MB) | The frame number of E | The frame number of A | E comprises A | A comprises |
| 15 | 1796 | 1796 | 100% | 100% |
| 30 | 4196 | 4196 | 100% | 100% |
| 60 | 7796 | 7796 | 100% | 100% |
| 100 | 15584 | 15584 | 100% | 100% |
| 300 | 39537 | 39537 | 100% | 100% |
| 500 | 71272 | 71272 | 100% | 100% |
| 1024 | 143401 | 143401 | 100% | 100% |
Explanation is at last, above embodiment is only unrestricted in order to technical scheme of the present invention to be described, although with reference to preferred embodiment the present invention is had been described in detail, those of ordinary skill in the art is to be understood that, can make amendment or be equal to replacement technical scheme of the present invention, and not breaking away from aim and the scope of technical solution of the present invention, it all should be encompassed in the middle of the claim scope of the present invention.
Claims (5)
1. the distributed coding/decoding method of the big data of the video based on Hadoop specifically comprises the steps:
Step a: the magnanimity video file directly is uploaded among the Hadoop distributed file system HDFS stores;
Step b: with fuse_dfs HDFS is mounted to local file system, to visit the file among the HDFS with unified method;
Step c: revise the data segmentation strategy of MapReduce calculating framework among the Hadoop,, solved by byte and cut apart the frame division that causes and the problem that can't decode as partitioning boundary with picture frame;
Steps d: obtain the required public information of decoding by the HDFS that is mounted to local file system, the distributed decoding of magnanimity video is finished in recycling MapReduce calculating framework and FFmpeg decoding storehouse;
Step e: the Map input of decoded result as MapReduce, can be used in follow-up intelligent video analysis.
2. according to the distributed coding/decoding method of the described big data of a kind of video based on Hadoop of claim 1, it is characterized in that: among the described step a, be the magnanimity video file directly to be uploaded among the distributed file system HDFS of Hadoop store.
3. according to the distributed coding/decoding method of the described big data of a kind of video based on Hadoop of claim 1, it is characterized in that: among the described step c, be to have revised the data segmentation strategy among the Hadoop, with picture frame as partitioning boundary, namely cut apart for each, can judge that this is cut apart obtain decodable frame, cuts apart the frame division that causes, the problem that can't decode thereby solved Hadoop by byte.
4. according to the distributed coding/decoding method of the described big data of a kind of video based on Hadoop of claim 1, it is characterized in that: in the described steps d, the public information that video decode is required, header etc. for example, be stored in the local mount point of describing among the step b, the distributed decoding of magnanimity video has been realized in recycling MapReduce calculating framework and FFmpeg decoding storehouse.The video format of support of the present invention has AVI, MPEG-4, RMVB, FLV, MOV, ASF, WMV, MKV, TS, VCD, DVD, MPEG-1, MPEG-2.
5. according to the distributed coding/decoding method of the described big data of a kind of video based on Hadoop of claim 1, it is characterized in that: among the described step e, the Map input of decoded result as MapReduce, key (key) is frame number, value (value) is image information, and these information can be used in intelligent video analysis such as foreground detection, motion tracking, summary generation.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013102039001A CN103279521A (en) | 2013-05-28 | 2013-05-28 | Video big data distributed decoding method based on Hadoop |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN2013102039001A CN103279521A (en) | 2013-05-28 | 2013-05-28 | Video big data distributed decoding method based on Hadoop |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN103279521A true CN103279521A (en) | 2013-09-04 |
Family
ID=49062040
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN2013102039001A Pending CN103279521A (en) | 2013-05-28 | 2013-05-28 | Video big data distributed decoding method based on Hadoop |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103279521A (en) |
Cited By (13)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104394415A (en) * | 2014-12-09 | 2015-03-04 | 中国电子科技集团公司第二十八研究所 | Method for distributed decoding of video big data |
| CN104519323A (en) * | 2014-12-26 | 2015-04-15 | 桂林远望智能通信科技有限公司 | Personnel and vehicle target classification system and method |
| CN104794231A (en) * | 2015-05-04 | 2015-07-22 | 罗敬宁 | Unstructured data processing method and device based on Map/Reduce |
| CN104853218A (en) * | 2015-05-12 | 2015-08-19 | 浪潮集团有限公司 | Method for realizing image JPEG encoding by using distributed big data architecture |
| CN104935785A (en) * | 2015-05-15 | 2015-09-23 | 江苏博智软件科技有限公司 | Hadoop based distributed video big data prospect detection and motion tracking method |
| CN105072456A (en) * | 2015-07-14 | 2015-11-18 | 中国科学院信息工程研究所 | Hadoop-based ciphertext video flow processing method, device, server and system |
| CN105721933A (en) * | 2016-03-23 | 2016-06-29 | Tcl集团股份有限公司 | Method for creating advertisement video information base, advertisement video identification method, apparatus for creating advertisement video information base and advertisement video identification apparatus |
| CN105975865A (en) * | 2016-05-27 | 2016-09-28 | 北京工业大学 | Hadoop platform measuring method based on dependable computing |
| CN106302609A (en) * | 2015-06-08 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of access method and device |
| CN106844562A (en) * | 2016-12-30 | 2017-06-13 | 北京航天泰坦科技股份有限公司 | Geographical image transparent caching mechanism based on FUSE |
| CN107463483A (en) * | 2017-07-31 | 2017-12-12 | 上海斐讯数据通信技术有限公司 | A kind of magnanimity mobile terminal measures statistical method and device online |
| CN110956202A (en) * | 2019-11-13 | 2020-04-03 | 重庆大学 | Image training method, system, medium and intelligent device based on distributed learning |
| CN111800635A (en) * | 2020-06-17 | 2020-10-20 | 西安理工大学 | Speculative parallel video decoding method based on AVS standard |
Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080250227A1 (en) * | 2007-04-04 | 2008-10-09 | Linderman Michael D | General Purpose Multiprocessor Programming Apparatus And Method |
| CN103116625A (en) * | 2013-01-31 | 2013-05-22 | 重庆大学 | Volume radio direction finde (RDF) data distribution type query processing method based on Hadoop |
-
2013
- 2013-05-28 CN CN2013102039001A patent/CN103279521A/en active Pending
Patent Citations (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20080250227A1 (en) * | 2007-04-04 | 2008-10-09 | Linderman Michael D | General Purpose Multiprocessor Programming Apparatus And Method |
| CN103116625A (en) * | 2013-01-31 | 2013-05-22 | 重庆大学 | Volume radio direction finde (RDF) data distribution type query processing method based on Hadoop |
Non-Patent Citations (1)
| Title |
|---|
| 张浩: "MapReduce编程模型在云海量视频转码中的研究", 《中国优秀硕士学位论文全文数据库信息科技辑》 * |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104394415A (en) * | 2014-12-09 | 2015-03-04 | 中国电子科技集团公司第二十八研究所 | Method for distributed decoding of video big data |
| CN104519323A (en) * | 2014-12-26 | 2015-04-15 | 桂林远望智能通信科技有限公司 | Personnel and vehicle target classification system and method |
| CN104794231A (en) * | 2015-05-04 | 2015-07-22 | 罗敬宁 | Unstructured data processing method and device based on Map/Reduce |
| CN104794231B (en) * | 2015-05-04 | 2018-05-25 | 罗敬宁 | A kind for the treatment of method and apparatus of the unstructured data based on Map/Reduce |
| CN104853218A (en) * | 2015-05-12 | 2015-08-19 | 浪潮集团有限公司 | Method for realizing image JPEG encoding by using distributed big data architecture |
| CN104935785A (en) * | 2015-05-15 | 2015-09-23 | 江苏博智软件科技有限公司 | Hadoop based distributed video big data prospect detection and motion tracking method |
| US11221997B2 (en) | 2015-06-08 | 2022-01-11 | Advanced New Technologies Co., Ltd. | On-demand creation and access of a virtual file system |
| CN106302609B (en) * | 2015-06-08 | 2020-02-28 | 阿里巴巴集团控股有限公司 | Access method and device |
| CN106302609A (en) * | 2015-06-08 | 2017-01-04 | 阿里巴巴集团控股有限公司 | A kind of access method and device |
| CN105072456A (en) * | 2015-07-14 | 2015-11-18 | 中国科学院信息工程研究所 | Hadoop-based ciphertext video flow processing method, device, server and system |
| CN105721933A (en) * | 2016-03-23 | 2016-06-29 | Tcl集团股份有限公司 | Method for creating advertisement video information base, advertisement video identification method, apparatus for creating advertisement video information base and advertisement video identification apparatus |
| CN105975865B (en) * | 2016-05-27 | 2019-06-07 | 北京工业大学 | A kind of Hadoop platform measure based on trust computing |
| CN105975865A (en) * | 2016-05-27 | 2016-09-28 | 北京工业大学 | Hadoop platform measuring method based on dependable computing |
| CN106844562A (en) * | 2016-12-30 | 2017-06-13 | 北京航天泰坦科技股份有限公司 | Geographical image transparent caching mechanism based on FUSE |
| CN107463483A (en) * | 2017-07-31 | 2017-12-12 | 上海斐讯数据通信技术有限公司 | A kind of magnanimity mobile terminal measures statistical method and device online |
| CN107463483B (en) * | 2017-07-31 | 2021-03-19 | 台州市吉吉知识产权运营有限公司 | Mass mobile terminal online quantity statistical method and device |
| CN110956202A (en) * | 2019-11-13 | 2020-04-03 | 重庆大学 | Image training method, system, medium and intelligent device based on distributed learning |
| CN110956202B (en) * | 2019-11-13 | 2023-08-01 | 重庆大学 | Image training method, system, medium and intelligent device based on distributed learning |
| CN111800635A (en) * | 2020-06-17 | 2020-10-20 | 西安理工大学 | Speculative parallel video decoding method based on AVS standard |
| CN111800635B (en) * | 2020-06-17 | 2022-01-07 | 西安理工大学 | Speculative parallel video decoding method based on AVS standard |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103279521A (en) | Video big data distributed decoding method based on Hadoop | |
| CN110019240B (en) | Service data interaction method, device and system | |
| US11836533B2 (en) | Automated reconfiguration of real time data stream processing | |
| US11374995B2 (en) | Multimedia file processing | |
| CN111258978B (en) | Data storage method | |
| US10356150B1 (en) | Automated repartitioning of streaming data | |
| CN106649828B (en) | Data query method and system | |
| CN109241159B (en) | Partition query method and system for data cube and terminal equipment | |
| BR112014000597B1 (en) | METHOD AND SYSTEM FOR DOWNLOADING A MULTIMEDIA FILE | |
| CN105144121A (en) | Caching content addressable data chunks for storage virtualization | |
| CN104506950A (en) | Multithread download method and download device in network streaming media play, and download equipment | |
| WO2013078583A1 (en) | Method and apparatus for optimizing data access, method and apparatus for optimizing data storage | |
| CN104469396A (en) | Distributed transcoding system and method | |
| US20220207043A1 (en) | Entity data services for virtualized computing and data systems | |
| CN112000703B (en) | Data warehousing processing method and device, computer equipment and storage medium | |
| CN103699656A (en) | GPU-based mass-multimedia-data-oriented MapReduce platform | |
| US20170344560A1 (en) | Distributed file system with integrated file object conversion | |
| CN104778252B (en) | The storage method and device of index | |
| Luo et al. | Big-data analytics: challenges, key technologies and prospects | |
| CN104394415A (en) | Method for distributed decoding of video big data | |
| CN105554069B (en) | A kind of big data processing distributed cache system and its method | |
| US9380126B2 (en) | Data collection and distribution management | |
| US11250001B2 (en) | Accurate partition sizing for memory efficient reduction operations | |
| CN104679575A (en) | Control system and control method for input and output flow | |
| CN112953944A (en) | Audio-video transcoding method based on MapReduce |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C02 | Deemed withdrawal of patent application after publication (patent law 2001) | ||
| WD01 | Invention patent application deemed withdrawn after publication |
Application publication date: 20130904 |