CN113485607B - Public general video capturing method - Google Patents
Public general video capturing method Download PDFInfo
- Publication number
- CN113485607B CN113485607B CN202110769744.XA CN202110769744A CN113485607B CN 113485607 B CN113485607 B CN 113485607B CN 202110769744 A CN202110769744 A CN 202110769744A CN 113485607 B CN113485607 B CN 113485607B
- Authority
- CN
- China
- Prior art keywords
- video
- capture
- clip
- data
- plug
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/02—Protocols based on web technology, e.g. hypertext transfer protocol [HTTP]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/951—Indexing; Web crawling techniques
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/90—Details of database functions independent of the retrieved data types
- G06F16/95—Retrieval from the web
- G06F16/953—Querying, e.g. by the use of web search engines
- G06F16/9535—Search customisation based on user profiles and personalisation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
- G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
- G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
- G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
- G06F3/0482—Interaction with lists of selectable items, e.g. menus
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Databases & Information Systems (AREA)
- General Engineering & Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Data Mining & Analysis (AREA)
- Human Computer Interaction (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention discloses an open universal video capture method, which comprises S1, installing and operating a chroma um browser supporting H.264 decoding; starting a development mode of a chromium browser, and installing a video capture plug-in; s2, starting a database and message middleware service, and initializing the state of a local user configuration directory pool; s3, starting video capture scheduling service and interface layer service; s4, initiating a video data capture request based on the HTTP protocol from the client to the interface layer; s5, the interface layer forwards the video data capturing request to a video capturing scheduling service, the video capturing scheduling service starts a chromium browser process, loads a video capturing plug-in, and opens the address of a video fragment page; and the video capture plug-in injects and executes Javascr i pt codes from the video clip page to finish the automatic click and play of contents such as the video clip. The advantages are that: the method can capture video data of various mainstream media formats, reject advertisements according to filtering rules, merge and transcode the streaming media video, and reduce the video capture difficulty.
Description
Technical Field
The invention relates to the technical field of crawler grabbing, in particular to an open universal video grabbing method.
Background
With the increasing progress of internet technology, the popularization of novel media forms such as short videos and small videos drives information transmission and influence, and various video data are important components of news media products. The traditional video data capturing is established on capturing and analyzing webpage source codes, the new technologies such as Html5 streaming media and the like are very easy to catch, meanwhile, the workload of capturing and analyzing data of various websites and App is very large, and a new video capturing method is needed to solve the problem.
Disclosure of Invention
The invention aims to provide a public universal video capture method, so that the problems in the prior art are solved.
In order to achieve the purpose, the technical scheme adopted by the invention is as follows:
an open universal video capture method comprises the following steps,
s1, installing and running a chromium browser supporting H.264 decoding; starting a development mode of a chrome browser, and installing a video capture plug-in;
s2, starting a database and message middleware service, and initializing the state of a local user configuration directory pool;
s3, starting video capture scheduling service and interface layer service;
s4, initiating a video data capture request based on the HTTP protocol from the client to the interface layer;
s5, the interface layer forwards the video data capturing request to a video capturing scheduling service, the video capturing scheduling service starts a chrome browser process, loads a video capturing plug-in, and opens the address of a video fragment page; the video capture plug-in module injects and executes Javascript codes from a video clip page to finish automatic click playing of the video clip; after the video clip starts to play, the video capture plug-in acquires the playing address, type, size and title information of the video clip by intercepting the field value of the HTTP network response data head, and feeds back the information of the video clip to the video capture scheduling service; and the video capture scheduling service filters and combines the corresponding video clips to generate video data according to the information of the video clips, and the video data is persisted to a database and fed back to the client through an interface layer to complete the capture of the video data.
Preferably, the code of the video capture plug-in and the environment configuration of the browser are packaged into user configuration template data, when the video capture scheduling service is started, the user configuration template data are copied into a plurality of user configuration data, and the use state record is stored in the message middleware service, so that each video data capture request can independently run under the condition of multiple concurrent requests and is not mutually influenced.
Preferably, step S5 specifically includes the following steps,
s51, the interface layer forwards the video data capturing request to a video capturing scheduling service;
s52, starting a chrome browser process by the video capture scheduling service, designating a temporary user data directory containing related video capture plug-ins, loading the corresponding video capture plug-ins, and opening the address of a video clip page;
s53, injecting Javascript codes when the video clip page begins to load, and setting the request context information of the video clip; triggering automatic playing after the Video clip page begins to be loaded, automatically clicking a playing button or searching a Video tag object, and playing the Video clip by using an autoplay method;
s54, after the loading of the video clip page is finished, notifying a content page Javascript of the video capture plug-in by a Javascript code triggering message injected into the video clip page to obtain the request context information of the video clip, notifying a background page Javascript of the video capture plug-in by the content page Javascript of the video capture plug-in, and storing the request context information of the video clip;
s55, based on the request context information of the video clip, the background page Javascript of the video capture plug-in analyzes and sniffs the type, size and title information of the video clip from the HTTP network response data header;
s56, the video capture plug-in sends a POST request of HTTP to the video capture scheduling service, the video capture scheduling service filters and combines corresponding video clips according to the information of the video clips to generate video data, and the video data are durably transmitted to a database and returned to an interface layer;
and S57, the interface layer returns the video data to the client to complete the capture of the video data.
Preferably, the request context information of the video clip is the capture task ID of the corresponding video clip.
Preferably, the value of the content-type field of the HTTP network response data header is matched with a pre-configured audio video classification to judge whether the video clip is audio or video, and if the video clip is audio, the video clip is filtered and the type of the next video clip is judged; if the video clip is a video, the address, type, size and title of the video clip are obtained.
Preferably, the video capture scheduling service filters and combines the corresponding video clips according to the information of the video clips to generate the video data by the specific processes of,
matching and screening the video clips with the advertisement formats in the video clips according to a preset filtering rule, removing the video clips with the advertisement formats and reserving the video clips with the feature film formats; and merging and transcoding all the video clips with the feature formats through the ffmpeg library to generate video data.
Preferably, the pre-configured filtering rule is dictionary information containing website domain names, advertisement video formats, content identification formats and advertisement waiting markets.
Preferably, the HTTP network response data header content-distribution field value is matched with a preconfigured video type to obtain the type of the video clip.
Preferably, the preconfigured video types include flv, f4v, mp3, mp4, m3u8, ts, m4s, wav, ogg, ogv, and webm.
Preferably, the size of the video clip is obtained using an HTTP network response data header Content-Length field value.
The invention has the beneficial effects that: the video capture plug-in can be used for automatically carrying out molecular and playing videos to capture video data of various mainstream media formats, advertisements can be removed according to filtering rules, the stream media videos are combined and transcoded, and the video capture difficulty is reduced.
Drawings
FIG. 1 is a schematic flow chart of a capture method in an embodiment of the invention;
FIG. 2 is a timing diagram illustrating video capture according to an embodiment of the present invention;
fig. 3 is a video capture Python backend architecture diagram in an embodiment of the invention.
Detailed Description
In order to make the objects, technical solutions and advantages of the present invention more apparent, the present invention is further described in detail below with reference to the accompanying drawings. It should be understood that the detailed description and specific examples, while indicating the invention, are intended for purposes of illustration only and are not intended to limit the scope of the invention.
As shown in fig. 1 to 3, in the present embodiment, an openable universal video capture method is provided, which includes the following steps,
s1, installing and running a chromium browser supporting H.264 decoding; starting a development mode of a chrome browser, and installing a video capture plug-in;
s2, starting a database and message middleware service, and initializing the state of a local user configuration directory pool;
s3, starting video capture scheduling service and interface layer service;
s4, initiating a video data capture request based on the HTTP protocol from the client to the interface layer;
s5, the interface layer forwards the video data capturing request to a video capturing scheduling service, the video capturing scheduling service starts a chrome browser process, loads a video capturing plug-in, and opens the address of a video fragment page; the video capture plug-in module injects and executes Javascript codes from a video clip page to finish automatic click playing of the video clip; after the video clip starts to play, the video capture plug-in acquires the playing address, type, size and title information of the video clip by intercepting the field value of the HTTP network response data head, and feeds back the information of the video clip to the video capture scheduling service; and the video capture scheduling service filters and combines the corresponding video clips to generate video data according to the information of the video clips, and the video data is persisted to a database and fed back to the client through an interface layer to complete the capture of the video data.
In this embodiment, a chroma browser supporting h.264 decoding is installed and run, and the problem that a third-party plug-in is disabled is solved by setting a group policy.
In this embodiment, the code of the video capture plug-in and the environment configuration of the browser are packaged into user configuration template data, when the video capture scheduling service is started, the user configuration template data is copied into a plurality of user configuration data, and a use state record is stored in the message middleware service, so that each video data capture request can run independently without mutual influence under the condition of multiple concurrent requests.
In this embodiment, the video capture scheduling service and the interface layer service are layered independently, and the interface layer architecture is located on the upper layer of the video capture scheduling service, has scalable expansion capability, and is convenient for docking and packaging into a general API.
The interface layer service is defined based on an Openfaas framework; the interface layer service is a service package using an HTTP protocol, comprises two APIs (application program interfaces) of video data capture and video result query, and can be dynamically and telescopically expanded and conveniently connected with other systems.
The video capturing and scheduling service is realized based on Python, and the video capturing and scheduling service realizes capturing message monitoring, thread pool management, Web service monitoring, task scheduling management and the like, and is responsible for communication between interface layer messages and browser plug-in messages, creation and release of browser processes and data persistence functions.
In this embodiment, step S5 specifically includes the following steps,
s51, the interface layer forwards the video data capturing request to a video capturing scheduling service;
s52, starting a chrome browser process by the video capture scheduling service, designating a temporary user data directory containing related video capture plug-ins, loading the corresponding video capture plug-ins, and opening the address of a video clip page;
s53, injecting Javascript codes when the video clip page begins to load, and setting the request context information of the video clip; triggering automatic playing after the Video clip page begins to be loaded, automatically clicking a playing button or searching a Video tag object, and playing the Video clip by using an autoplay method;
s54, after the loading of the video clip page is finished, notifying a content page Javascript of the video capture plug-in by a Javascript code triggering message injected into the video clip page to obtain the request context information of the video clip, notifying a background page Javascript of the video capture plug-in by the content page Javascript of the video capture plug-in, and storing the request context information of the video clip;
s55, based on the request context information of the video clip, the background page Javascript of the video capture plug-in analyzes and sniffs the type, size and title information of the video clip from the HTTP network response data header;
s56, the video capture plug-in sends a POST request of HTTP to the video capture scheduling service, the video capture scheduling service filters and combines corresponding video clips according to the information of the video clips to generate video data, and the video data are durably transmitted to a database and returned to an interface layer;
and S57, the interface layer returns the video data to the client to complete the capture of the video data.
And the request context information of the video clip is the grabbing task ID of the corresponding video clip.
Matching the content-type field value of the HTTP network response data head with a pre-configured audio video classification, judging whether the video clip is audio or video, and if the video clip is audio, filtering out and judging the type of the next video clip; if the video clip is a video, the address, type, size and title of the video clip are obtained. The pre-configured audio and video are classified into an argument and a video for distinguishing whether the video clip is specifically audio or video.
Matching the content-displacement field value of the HTTP network response data head with a preset video type to acquire the type of the video clip. The preconfigured video types include flv, f4v, mp3, mp4, m3u8, ts, m4s, wav, ogg, ogv, and webm.
And acquiring the size of the video clip by using the HTTP network response data header Content-Length field value.
In this embodiment, the specific process of the video capture scheduling service filtering and merging the corresponding video segments according to the information of the video segments to generate video data is,
matching and screening the video clips with the advertisement formats in the video clips according to a preset filtering rule, removing the video clips with the advertisement formats and reserving the video clips with the feature film formats; and merging and transcoding all the video clips with the feature formats through the ffmpeg library to generate video data.
The pre-configured filtering rule is dictionary information containing website domain names, advertisement video formats, content identification formats and advertisement waiting markets.
For example, in some video website data, the advertisement is located at the first or the first few, the video format is flv, and the feature format is m3u8 or ts. Then, the advertisements with flv video format are screened and removed by using the pre-configured filtering rule, and the remaining m3u8/m4s/ts and other positive films with streaming media format are merged and transcoded by the ffmpeg library to generate video data, and then the video data is persisted in the database.
By adopting the technical scheme disclosed by the invention, the following beneficial effects are obtained:
the invention provides an open universal video capture method, which can automatically perform molecular and video playing by using a video capture plug-in to capture video data of various mainstream media formats, can eliminate advertisements according to a filtering rule, and can merge and transcode streaming media videos to reduce the video capture difficulty.
The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and improvements can be made without departing from the principle of the present invention, and such modifications and improvements should also be considered within the scope of the present invention.
Claims (8)
1. An open universal video capture method is characterized in that: comprises the following steps of (a) carrying out,
s1, installing and running a chromium browser supporting H.264 decoding; starting a development mode of a chrome browser, and installing a video capture plug-in;
s2, starting a database and message middleware service, and initializing the state of a local user configuration directory pool;
s3, starting video capture scheduling service and interface layer service;
s4, initiating a video data capture request based on the HTTP protocol from the client to the interface layer;
s5, the interface layer forwards the video data capturing request to a video capturing scheduling service, the video capturing scheduling service starts a chrome browser process, loads a video capturing plug-in, and opens the address of a video fragment page; the video capture plug-in module injects and executes Javascript codes from a video clip page to finish automatic click playing of the video clip; after the video clip starts to play, the video capture plug-in acquires the playing address, type, size and title information of the video clip by intercepting the field value of the HTTP network response data head, and feeds back the information of the video clip to the video capture scheduling service; the video capturing scheduling service filters and combines the corresponding video clips to generate video data according to the information of the video clips, the video data are durably transmitted to a database and fed back to a client through an interface layer, and capturing of the video data is completed;
the method comprises the steps that codes of video capture plug-ins and environment configuration of a browser are packaged into user configuration template data, when a video capture scheduling service is started, the user configuration template data are copied into a plurality of user configuration data, and a use state record is stored in a message middleware service, so that each video data capture request can run independently under the condition of multiple concurrent requests and is not mutually influenced;
the step S5 specifically includes the following contents,
s51, the interface layer forwards the video data capturing request to a video capturing scheduling service;
s52, starting a chrome browser process by the video capture scheduling service, designating a temporary user data directory containing related video capture plug-ins, loading the corresponding video capture plug-ins, and opening the address of a video clip page;
s53, injecting Javascript codes when the video clip page begins to load, and setting the request context information of the video clip; triggering automatic playing after the Video clip page begins to be loaded, automatically clicking a playing button or searching a Video tag object, and playing the Video clip by using an autoplay method;
s54, after the loading of the video clip page is finished, notifying a content page Javascript of the video capture plug-in by a Javascript code triggering message injected into the video clip page to obtain the request context information of the video clip, notifying a background page Javascript of the video capture plug-in by the content page Javascript of the video capture plug-in, and storing the request context information of the video clip;
s55, based on the request context information of the video clip, the background page Javascript of the video capture plug-in analyzes and sniffs the type, size and title information of the video clip from the HTTP network response data header;
s56, the video capture plug-in sends a POST request of HTTP to the video capture scheduling service, the video capture scheduling service filters and combines corresponding video clips according to the information of the video clips to generate video data, and the video data are durably transmitted to a database and returned to an interface layer;
and S57, the interface layer returns the video data to the client to complete the capture of the video data.
2. The disclosed universal video capture method of claim 1, wherein: and the request context information of the video clip is the grabbing task ID of the corresponding video clip.
3. The disclosed universal video capture method of claim 1, wherein: matching the content-type field value of the HTTP network response data head with a pre-configured audio video classification, judging whether the video clip is audio or video, and if the video clip is audio, filtering out and judging the type of the next video clip; if the video clip is a video, the address, type, size and title of the video clip are obtained.
4. The disclosed universal video capture method of claim 1, wherein: the video capture scheduling service filters and combines the corresponding video clips according to the information of the video clips to generate video data by the specific process that,
matching and screening the video clips with the advertisement formats in the video clips according to a preset filtering rule, removing the video clips with the advertisement formats and reserving the video clips with the feature film formats; and merging and transcoding all the video clips with the feature formats through the ffmpeg library to generate video data.
5. The disclosed universal video capture method of claim 1, wherein: the pre-configured filtering rule is dictionary information containing website domain names, advertisement video formats, content identification formats and advertisement waiting markets.
6. The disclosed universal video capture method of claim 3, wherein: matching the content-displacement field value of the HTTP network response data head with a preset video type to acquire the type of the video clip.
7. The disclosed universal video capture method of claim 3, wherein: the preconfigured video types include flv, f4v, mp3, mp4, m3u8, ts, m4s, wav, ogg, ogv, and webm.
8. The disclosed universal video capture method of claim 3, wherein: and acquiring the size of the video clip by using the HTTP network response data header Content-Length field value.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110769744.XA CN113485607B (en) | 2021-07-07 | 2021-07-07 | Public general video capturing method |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202110769744.XA CN113485607B (en) | 2021-07-07 | 2021-07-07 | Public general video capturing method |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN113485607A CN113485607A (en) | 2021-10-08 |
| CN113485607B true CN113485607B (en) | 2022-02-15 |
Family
ID=77937974
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202110769744.XA Active CN113485607B (en) | 2021-07-07 | 2021-07-07 | Public general video capturing method |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN113485607B (en) |
Families Citing this family (2)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN114900719B (en) * | 2022-04-29 | 2024-01-30 | 广东博思信息技术股份有限公司 | Method and system for playing real-time video of monitoring equipment based on HTML5 |
| CN116909260B (en) * | 2023-09-12 | 2023-12-01 | 常州星宇车灯股份有限公司 | Intelligent driving domain controller test verification method for simulating HIL (high-performance liquid chromatography) rack |
Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102098327A (en) * | 2010-12-15 | 2011-06-15 | 百度在线网络技术（北京）有限公司 | Method and device for downloading online video sniffer |
| CN103905915A (en) * | 2010-12-15 | 2014-07-02 | 百度在线网络技术（北京）有限公司 | Online video sniffing downloading method and device |
| CN104469467A (en) * | 2014-12-10 | 2015-03-25 | 北京奇虎科技有限公司 | Video playing method, device and system in browser client |
| CN112487268A (en) * | 2020-12-14 | 2021-03-12 | 安徽经邦软件技术有限公司 | Data crawling implementation method based on distributed crawler technology |
Family Cites Families (1)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| WO2008021344A2 (en) * | 2006-08-14 | 2008-02-21 | Fliqz, Inc. | System and method for providing video media on a website |
-
2021
- 2021-07-07 CN CN202110769744.XA patent/CN113485607B/en active Active
Patent Citations (4)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102098327A (en) * | 2010-12-15 | 2011-06-15 | 百度在线网络技术（北京）有限公司 | Method and device for downloading online video sniffer |
| CN103905915A (en) * | 2010-12-15 | 2014-07-02 | 百度在线网络技术（北京）有限公司 | Online video sniffing downloading method and device |
| CN104469467A (en) * | 2014-12-10 | 2015-03-25 | 北京奇虎科技有限公司 | Video playing method, device and system in browser client |
| CN112487268A (en) * | 2020-12-14 | 2021-03-12 | 安徽经邦软件技术有限公司 | Data crawling implementation method based on distributed crawler technology |
Also Published As
| Publication number | Publication date |
|---|---|
| CN113485607A (en) | 2021-10-08 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| US8924383B2 (en) | Broadcast video monitoring and alerting system | |
| US10698952B2 (en) | Using digital fingerprints to associate data with a work | |
| CN113485607B (en) | Public general video capturing method | |
| US8589973B2 (en) | Peer to peer media distribution system and method | |
| US9769232B2 (en) | Apparatus and method for managing media content | |
| KR101977516B1 (en) | Media distribution and management platform | |
| WO2017063399A1 (en) | Video playback method and device | |
| US8041781B2 (en) | System and method for providing web system services for storing data and context of client applications on the web | |
| US8046436B2 (en) | System and method of providing context information for client application data stored on the web | |
| US8046438B2 (en) | System and method of restoring data and context of client applications stored on the web | |
| CN101489125A (en) | Video controlling method based on XML and system thereof | |
| EP1650927A2 (en) | Broadcast video monitoring and alerting system | |
| US20090254632A1 (en) | Methods and apparatus for delivering auxiliary data to device | |
| US7996779B2 (en) | System and method of providing a user interface for client applications to store data and context information on the web | |
| US20080228903A1 (en) | System and method of serving advertisements for web applications | |
| US20080307105A1 (en) | Streaming media archiver for live events | |
| WO2017130035A1 (en) | A system and methods thereof for auto-playing video content on mobile devices | |
| US8490131B2 (en) | Automatic capture of data for acquisition of metadata | |
| US8046437B2 (en) | System and method of storing data and context of client application on the web | |
| US10075485B2 (en) | Animated snapshots | |
| US20190266646A1 (en) | Real-Time Advertisement Preparation System | |
| Wei et al. | MediaAlert-a broadcast video monitoring and alerting system for mobile users | |
| US11405698B2 (en) | Information processing apparatus, information processing method, and program for presenting reproduced video including service object and adding additional image indicating the service object | |
| CN111447490A (en) | Streaming media file processing method and device | |
| CN115499702B (en) | Display method and device of preview image, electronic equipment and storage medium |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |