CN103984745A - Distributed video vertical searching method and system - Google Patents
Distributed video vertical searching method and system Download PDFInfo
- Publication number
- CN103984745A CN103984745A CN201410222453.9A CN201410222453A CN103984745A CN 103984745 A CN103984745 A CN 103984745A CN 201410222453 A CN201410222453 A CN 201410222453A CN 103984745 A CN103984745 A CN 103984745A
- Authority
- CN
- China
- Prior art keywords
- video
- module
- distributed
- index
- data
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/71—Indexing; Data structures therefor; Storage structures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/73—Querying
- G06F16/738—Presentation of query results
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Multimedia (AREA)
- Data Mining & Analysis (AREA)
- Databases & Information Systems (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Software Systems (AREA)
- Computational Linguistics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
- Two-Way Televisions, Distribution Of Moving Picture Or The Like (AREA)
Abstract
The invention provides a distributed video vertical searching method and system. The method includes an offline processing step and an online processing step. The offline processing step includes video capture, video storage and video information index building. The online processing step includes user interaction and video indexing. The method and system has the advantage that a distributed video vertical searching technical scheme based on Hadoop and Elastic Search is provided based on the video field, an effectively distributed searching scheme is provided for massive video data, batch fast collection, real-time searching and the like of massive videos are achieved, high-efficiency real-time search is achieved, and quick collection work for offline processing of the massive videos is achieved.
Description
Technical field
The present invention relates to video search field, relate in particular to distributed video method for vertical search and system.
Background technology
Along with the fast development of web2.0 and internet, the stock number above internet increases rapidly, presents a kind of explosive growth speed.How from information resources magnanimity, miscellaneous, finding the video information oneself needing, to search engine, bring challenges, is also the deficiency in current technology and the problem that will solve.
Summary of the invention
In order to solve the problems of the prior art, the invention provides a kind of distributed video method for vertical search.
The invention provides a kind of distributed video method for vertical search, it is characterized in that, comprise processed offline step and online treatment step;
Processed offline step comprises:
A. video acquisition step: obtain video data resource;
B. video storage step: the video data resource of obtaining in video acquisition step is deposited in database;
C. the index creation step of video information: create distributed index;
Online treatment step comprises:
User interactions step: search inputting interface is provided and returns and show interface;
Video frequency searching step: carry out video frequency searching according to the searched key word of search inputting interface input, the video data retrieving is shown to interface shows by returning.
As a further improvement on the present invention,
In described video acquisition step, the API providing by internet video open platform obtains video resource, this process operation is on Hadoop cluster, by Map task, accessing video open platform server, obtain the data of video JSON form and resolve to the form of video class of oneself definition, circulation is obtained until Map task finishes;
In described video storage step, video storage process operation is on Hadoop cluster and HBase cluster, by Reduce process, the video information of obtaining in video acquisition step is resolved, obtain again the storage entrance of distributed data base HBase, video information is deposited in the video library of predefined, circulate until Reduce task finishes;
In the index creation step of video information, the index creation of video information runs on HBase cluster and ElasticSearch cluster, first obtain the video information in HBase database, index creation interface by encapsulation ElasticSearch creates distributed index burst and index copy again, index stores is on ElasticSearch cluster, until video data establishment is complete in HBase video library, this process finishes;
Video frequency searching step runs on ElasticSearch cluster.
As a further improvement on the present invention, in described video acquisition step, comprise the steps:
A1. video slicing, arranges burst number;
A2. generating video linking URL;
A3. according to the URL generating, the server of access open platform, obtains the video data that URL is corresponding, and its form is JSON type;
A4. according to the video data of the JSON type of obtaining, utilize corresponding JSON to resolve interface JSON data are resolved, and be converted into the video class oneself defining;
In described video storage step, comprise the steps:
B1. complete the resolving of video class, video information is resolved to corresponding video attribute information;
B2. receiver, video attribute information, and call the writing in the database that incoming interface deposits corresponding video information in of distributed data base HBase.
As a further improvement on the present invention, in the index creation step of described video information, comprise the steps:
C1. connect HBase distributed data base, obtain video information;
C2. the video information of obtaining is packaged into the data of JSON type;
C3. obtain the video information of JSON type, through a hash process, corresponding video distribution is set up to index in the middle of corresponding index burst, after index completes, carry out the establishment of copy;
C4. judge in database, whether video data completes, in this way, finish; Otherwise jumping to step C1 continues to carry out.
As a further improvement on the present invention, in described online treatment step, comprise the steps:
The first step: user passes through user's query interface input inquiry keyword easily, submits to retrieval server;
Second step: the searching keyword that retrieval server is responsible for user to submit to is distributed to each node server;
The 3rd step: each node server receives after the retrieval request that retrieval server sends over, and automatically retrieves index burst on this node, completes the process of parallel search, and result is returned;
The 4th step: retrieval server receives the video data that each node server returns, and its video data is merged, for returning to user;
The 5th step: the video data that retrieval server is returned is shown to user.
The present invention also provides a kind of distributed video vertical search system, comprises processed offline unit and online processing unit;
Processed offline unit comprises:
Video acquisition module: for obtaining video data resource;
Video storage module: deposit in database for the video data resource that video acquisition module is obtained;
The index creation module of video information: for creating distributed index;
Online processing unit comprises:
User interactive module: for search inputting interface being provided and returning and show interface;
Video frequency searching module: for carrying out video frequency searching according to the searched key word of search inputting interface input, the video data retrieving is shown to interface shows by returning.
As a further improvement on the present invention, in described video acquisition module, the API providing by internet video open platform obtains video resource, this process operation is on Hadoop cluster, by Map task, accessing video open platform server, obtains the data of video JSON form and resolves to the form of video class of oneself definition, and circulation is obtained until Map task finishes;
In described video storage module, video storage process operation is on Hadoop cluster and HBase cluster, by Reduce process, the video information of obtaining in video acquisition step is resolved, obtain again the storage entrance of distributed data base HBase, video information is deposited in the video library of predefined, circulate until Reduce task finishes;
In the index creation module of video information, the index creation of video information runs on HBase cluster and ElasticSearch cluster, first obtain the video information in HBase database, index creation interface by encapsulation ElasticSearch creates distributed index burst and index copy again, index stores is on ElasticSearch cluster, until video data establishment is complete in HBase video library, this process finishes;
Video frequency searching module runs on ElasticSearch cluster.
As a further improvement on the present invention, in described video acquisition module, comprise:
Burst module: for video slicing, burst number is set;
Generation module: for generating video linking URL;
Acquisition module: for according to the URL generating, the server of access open platform, obtains the video data that URL is corresponding, and its form is JSON type;
Conversion module: for according to the video data of the JSON type of obtaining, utilize corresponding JSON to resolve interface JSON data are resolved, and be converted into the video class oneself defining;
In described video storage module, comprise:
Parsing module: for completing the resolving of video class, video information is resolved to corresponding video attribute information;
Deposit module in: for receiver, video attribute information, and call the writing in the database that incoming interface deposits corresponding video information in of distributed data base HBase.
As a further improvement on the present invention, the index creation module in described video information comprises:
Link block: for connecting HBase distributed data base, obtain video information;
Packetization module: for the video information of obtaining being packaged into the data of JSON type;
Processing module: for obtaining the video information of JSON type, through a hash process, corresponding video distribution is set up to index in the middle of corresponding index burst, carry out the establishment of copy after index completes;
Judge module: for judging that whether database video data completes, and in this way, finishes; Otherwise jumping to step C1 continues to carry out.
As a further improvement on the present invention, at described online processing unit, comprise:
Load module: user passes through user's query interface input inquiry keyword easily, submits to retrieval server;
Distribution module: the searching keyword of being responsible for user to submit to for retrieval server is distributed to each node server;
Retrieval module: receive after the retrieval request that retrieval server sends over for each node server, automatically retrieve index burst on this node, complete the process of parallel search, and result is returned;
Return to module: for retrieval server, receive the video data that each node server returns, and its video data is merged, for returning to user;
Display module: be shown to user for the video data that retrieval server is returned.
The invention has the beneficial effects as follows: the present invention is based on video field, a kind of technical scheme of the distributed video vertical search based on Hadoop and ElasticSearch is proposed, for massive video data, a kind of effective distributed search scheme is proposed, solve the problem such as batch Quick Acquisition, real-time search of magnanimity video, thereby realized the Quick Acquisition work of efficient real-time search, processed offline magnanimity video.
Accompanying drawing explanation
Fig. 1 is logic schematic diagram of the present invention.
Fig. 2 is that Map/Reduce video information of the present invention gathers schematic diagram.
Fig. 3 is that distributed index of the present invention creates schematic diagram.
Fig. 4 is query processing schematic diagram of the present invention.
Embodiment
As shown in Figure 1, the invention discloses a kind of distributed video method for vertical search, comprise processed offline step and online treatment step;
Processed offline step comprises:
A. video acquisition step: obtain video data resource;
B. video storage step: the video data resource of obtaining in video acquisition step is deposited in database;
C. the index creation step of video information: create distributed index;
Online treatment step comprises:
User interactions step: search inputting interface is provided and returns and show interface;
Video frequency searching step: carry out video frequency searching according to the searched key word of search inputting interface input, the video data retrieving is shown to interface shows by returning.
As one embodiment of the present of invention:
In described video acquisition step, the API providing by internet video open platform obtains video resource, this process operation is on Hadoop cluster, by Map task, accessing video open platform server, obtain the data of video JSON form and resolve to the form of video class of oneself definition, circulation is obtained until Map task finishes;
In described video storage step, video storage process operation is on Hadoop cluster and HBase cluster, by Reduce process, the video information of obtaining in video acquisition step is resolved, obtain again the storage entrance of distributed data base HBase, video information is deposited in the video library of predefined, circulate until Reduce task finishes;
In the index creation step of video information, the index creation of video information runs on HBase cluster and ElasticSearch cluster, first obtain the video information in HBase database, index creation interface by encapsulation ElasticSearch creates distributed index burst and index copy again, index stores is on ElasticSearch cluster, until video data establishment is complete in HBase video library, this process finishes;
Video frequency searching step runs on ElasticSearch cluster.
As shown in Figure 2, in described video acquisition step, comprise the steps:
A1.Hadoop video slicing process, before operating in Map/Reduce, arranges burst number, and job invocation needed task to carry out burst (piece that is divided into fixed size) before Hadoop cluster;
A2. generating video linking URL, for subsequent step access services device is laid the groundwork;
A3. according to the URL generating, the server of access open platform, obtains the video data that URL is corresponding, and its form is JSON type;
A4. according to the video data of the JSON type of obtaining, utilize corresponding JSON to resolve interface JSON data are resolved, and be converted into the video class oneself defining;
So far, Map process finishes, and will enter Reduce data storage procedure afterwards.
In described video storage step, comprise the steps:
B1. complete the resolving of video class, video information is resolved to corresponding video attribute information;
B2. receiver, video attribute information, and call the writing in the database that incoming interface deposits corresponding video information in of distributed data base HBase.
As shown in Figure 3, in the index creation step of described video information, comprise the steps:
C1. connect HBase distributed data base, ergodic data storehouse, obtains video information;
C2. the data type of supporting due to ElasticSearch is JSON, so the video information of obtaining in step C1 need to be packaged into the data of JSON type;
C3. complete the Sharding process of distributed index, according to system setting, it is 2 that supposing the system is arranged to burst number, copy is 1, from step C2, obtain so the video information of a JSON type, through a hash process, corresponding video distribution is set up to index in the middle of corresponding index burst, after index completes, carry out the establishment of copy;
C4. judge in database, whether video data completes, in this way, finish; Otherwise jumping to step C1 continues to carry out.
As shown in Figure 4, in described online treatment step, comprise the steps:
In step S1: user passes through user's query interface input inquiry keyword easily, submits to retrieval server;
In step S2: the searching keyword that retrieval server is responsible for user to submit to is distributed to each node server;
In step S3: each node server receives after the retrieval request that retrieval server sends over, and automatically retrieves index burst on this node, completes the process of parallel search, and result is returned;
In step S4: retrieval server receives the video data that each node server returns, and its video data is merged, for returning to user;
In step S5: the invention provides to user's friendly result and show interface, the video data that retrieval server is returned shows user with the form of graphic interface.
In Fig. 4, index server is node server.
The invention also discloses a kind of distributed video vertical search system, comprise processed offline unit and online processing unit;
Processed offline unit comprises:
Video acquisition module: for obtaining video data resource;
Video storage module: deposit in database for the video data resource that video acquisition module is obtained;
The index creation module of video information: for creating distributed index;
Online processing unit comprises:
User interactive module: for search inputting interface being provided and returning and show interface;
Video frequency searching module: for carrying out video frequency searching according to the searched key word of search inputting interface input, the video data retrieving is shown to interface shows by returning.
In described video acquisition module, the API providing by internet video open platform obtains video resource, this process operation is on Hadoop cluster, by Map task, accessing video open platform server, obtain the data of video JSON form and resolve to the form of video class of oneself definition, circulation is obtained until Map task finishes;
In described video storage module, video storage process operation is on Hadoop cluster and HBase cluster, by Reduce process, the video information of obtaining in video acquisition step is resolved, obtain again the storage entrance of distributed data base HBase, video information is deposited in the video library of predefined, circulate until Reduce task finishes;
In the index creation module of video information, the index creation of video information runs on HBase cluster and ElasticSearch cluster, first obtain the video information in HBase database, index creation interface by encapsulation ElasticSearch creates distributed index burst and index copy again, index stores is on ElasticSearch cluster, until video data establishment is complete in HBase video library, this process finishes;
Video frequency searching module runs on ElasticSearch cluster.
In described video acquisition module, comprise:
Burst module: for video slicing, burst number is set;
Generation module: for generating video linking URL;
Acquisition module: for according to the URL generating, the server of access open platform, obtains the video data that URL is corresponding, and its form is JSON type;
Conversion module: for according to the video data of the JSON type of obtaining, utilize corresponding JSON to resolve interface JSON data are resolved, and be converted into the video class oneself defining;
In described video storage module, comprise:
Parsing module: for completing the resolving of video class, video information is resolved to corresponding video attribute information;
Deposit module in: for receiver, video attribute information, and call the writing in the database that incoming interface deposits corresponding video information in of distributed data base HBase.
Index creation module in described video information comprises:
Link block: for connecting HBase distributed data base, obtain video information;
Packetization module: for the video information of obtaining being packaged into the data of JSON type;
Processing module: for obtaining the video information of JSON type, through a hash process, corresponding video distribution is set up to index in the middle of corresponding index burst, carry out the establishment of copy after index completes;
Judge module: for judging that whether database video data completes, and in this way, finishes; Otherwise jumping to step C1 continues to carry out.
At described online processing unit, comprise:
Load module: user passes through user's query interface input inquiry keyword easily, submits to retrieval server;
Distribution module: the searching keyword of being responsible for user to submit to for retrieval server is distributed to each node server;
Retrieval module: receive after the retrieval request that retrieval server sends over for each node server, automatically retrieve index burst on this node, complete the process of parallel search, and result is returned;
Return to module: for retrieval server, receive the video data that each node server returns, and its video data is merged, for returning to user;
Display module: be shown to user for the video data that retrieval server is returned.
As shown in Figure 1, the present invention includes a multi-search engine, described multi-search engine adopts Hadoop, HBase, ElasticSearch as architecture;
The Map/Reduce of employing Hadoop completes collection and the resolving of video information, utilize distributed data base HBase to complete storage, utilize ElasticSearch to complete establishment and the retrieving of distributed index, utilize JSP technology to carry out completing user reciprocal process, by JSP technology, come the query interface of completing user and result to show interface.
In the present invention, suppose that cluster has 5 ordinary PC, server name of the present invention: master, slave1, slave2, slave3, slave4, wherein, master is the master server of server, take on the work that task scheduling, index burst are distributed, created index, simultaneously as querying server, receive inquiry request and be submitted to other node server; Slave node server as data parallel computing, data storage, querying server, really process inquiry request, returns to Query Result.
The present invention is divided into off-line process and online processing procedure two parts generally.Off-line process comprises the gatherer process of video, the Index process of the storing process of video, video; And online processing procedure is mainly user's reciprocal process and the distributed real-time query process of back-end data.Due to the non real-time property of Hadoop and the high speed processing ability of large data, the present invention utilizes the mass data processing ability of Hadoop and gatherer process and the storing process that Map/Reduce multiple programming framework completes the video in off-line procedure.Because ElasticSearch provides an extendible distributed search scheme of increasing income and the intimate real-time function of ElasticSearch, the present invention completes establishment and the real-time retrieval process of distributed index with ElasticSearch.
Mass data processing ability and ElasticSearch that the present invention makes full use of Hadoop are close to real-time search capability.
Video acquisition module is comprised of information acquisition interface, and Map/Reduce interface, the JSON format analysis interface of its lower encapsulation Hadoop, run on Hadoop cluster.
Video storage module, its lower encapsulation HBase writes incoming interface, Map/Reduce interface, runs on Hadoop, HBase cluster.
The index creation module of video information, its lower encapsulation HBase fetch interface, ElasticSearch index interface, run on HBase cluster, Hadoop cluster, ElasticSearch cluster.
Video frequency searching module, its lower encapsulation ElasticSearch query interface, JSON Data Analysis interface, run on ElasticSearch cluster.
The present invention's operation at least needs 5 ordinary PC or server, because extendability of the present invention is fine, for mass data, in theory, can improve the performance of distributed search system by increasing the quantity of main frame in cluster.
In addition, the present invention is to provide a kind of solution of general distributed vertical search, be incessantly applicable to video field.
Technique effect of the present invention is very obvious: whole system is divided into off-line process and online processing procedure in real time.Off-line process mainly realizes by cluster high speed capability and the distributed batch processing ability of Hadoop; Online processing procedure is mainly completed by the intimate real-time search capability of ElasticSearch cluster.
The present invention is based on video field, a kind of technical scheme of the distributed video vertical search based on Hadoop and ElasticSearch is proposed, for massive video data, a kind of effective distributed search scheme is proposed, solve the problem such as batch Quick Acquisition, real-time search of magnanimity video, thereby realized the Quick Acquisition work of efficient real-time search, processed offline magnanimity video.
Above content is in conjunction with concrete preferred implementation further description made for the present invention, can not assert that specific embodiment of the invention is confined to these explanations.For general technical staff of the technical field of the invention, without departing from the inventive concept of the premise, can also make some simple deduction or replace, all should be considered as belonging to protection scope of the present invention.
Claims (10)
1. a distributed video method for vertical search, is characterized in that, comprises processed offline step and online treatment step;
Processed offline step comprises:
A. video acquisition step: obtain video data resource;
B. video storage step: the video data resource of obtaining in video acquisition step is deposited in database;
C. the index creation step of video information: create distributed index;
Online treatment step comprises:
User interactions step: search inputting interface is provided and returns and show interface;
Video frequency searching step: carry out video frequency searching according to the searched key word of search inputting interface input, the video data retrieving is shown to interface shows by returning.
2. distributed video method for vertical search according to claim 1, is characterized in that:
In described video acquisition step, the API providing by internet video open platform obtains video resource, this process operation is on Hadoop cluster, by Map task, accessing video open platform server, obtain the data of video JSON form and resolve to the form of video class of oneself definition, circulation is obtained until Map task finishes;
In described video storage step, video storage process operation is on Hadoop cluster and HBase cluster, by Reduce process, the video information of obtaining in video acquisition step is resolved, obtain again the storage entrance of distributed data base HBase, video information is deposited in the video library of predefined, circulate until Reduce task finishes;
In the index creation step of video information, the index creation of video information runs on HBase cluster and ElasticSearch cluster, first obtain the video information in HBase database, index creation interface by encapsulation ElasticSearch creates distributed index burst and index copy again, index stores is on ElasticSearch cluster, until video data establishment is complete in HBase video library, this process finishes;
Video frequency searching step runs on ElasticSearch cluster.
3. distributed video method for vertical search according to claim 2, is characterized in that, in described video acquisition step, comprises the steps:
A1. video slicing, arranges burst number;
A2. generating video linking URL;
A3. according to the URL generating, the server of access open platform, obtains the video data that URL is corresponding, and its form is JSON type;
A4. according to the video data of the JSON type of obtaining, utilize corresponding JSON to resolve interface JSON data are resolved, and be converted into the video class oneself defining;
In described video storage step, comprise the steps:
B1. complete the resolving of video class, video information is resolved to corresponding video attribute information;
B2. receiver, video attribute information, and call the writing in the database that incoming interface deposits corresponding video information in of distributed data base HBase.
4. distributed video method for vertical search according to claim 3, is characterized in that, in the index creation step of described video information, comprises the steps:
C1. connect HBase distributed data base, obtain video information;
C2. the video information of obtaining is packaged into the data of JSON type;
C3. obtain the video information of JSON type, through a hash process, corresponding video distribution is set up to index in the middle of corresponding index burst, after index completes, carry out the establishment of copy;
C4. judge in database, whether video data completes, in this way, finish; Otherwise jumping to step C1 continues to carry out.
5. distributed video method for vertical search according to claim 4, is characterized in that, in described online treatment step, comprises the steps:
The first step: user passes through user's query interface input inquiry keyword easily, submits to retrieval server;
Second step: the searching keyword that retrieval server is responsible for user to submit to is distributed to each node server; The 3rd step: each node server receives after the retrieval request that retrieval server sends over, and automatically retrieves index burst on this node, completes the process of parallel search, and result is returned;
The 4th step: retrieval server receives the video data that each node server returns, and its video data is merged, for returning to user;
The 5th step: the video data that retrieval server is returned is shown to user.
6. a distributed video vertical search system, is characterized in that, comprises processed offline unit and online processing unit;
Processed offline unit comprises:
Video acquisition module: for obtaining video data resource;
Video storage module: deposit in database for the video data resource that video acquisition module is obtained;
The index creation module of video information: for creating distributed index;
Online processing unit comprises:
User interactive module: for search inputting interface being provided and returning and show interface;
Video frequency searching module: for carrying out video frequency searching according to the searched key word of search inputting interface input, the video data retrieving is shown to interface shows by returning.
7. distributed video vertical search system according to claim 6, is characterized in that:
In described video acquisition module, the API providing by internet video open platform obtains video resource, this process operation is on Hadoop cluster, by Map task, accessing video open platform server, obtain the data of video JSON form and resolve to the form of video class of oneself definition, circulation is obtained until Map task finishes;
In described video storage module, video storage process operation is on Hadoop cluster and HBase cluster, by Reduce process, the video information of obtaining in video acquisition step is resolved, obtain again the storage entrance of distributed data base HBase, video information is deposited in the video library of predefined, circulate until Reduce task finishes;
In the index creation module of video information, the index creation of video information runs on HBase cluster and ElasticSearch cluster, first obtain the video information in HBase database, index creation interface by encapsulation ElasticSearch creates distributed index burst and index copy again, index stores is on ElasticSearch cluster, until video data establishment is complete in HBase video library, this process finishes;
Video frequency searching module runs on ElasticSearch cluster.
8. distributed video vertical search system according to claim 7, is characterized in that, in described video acquisition module, comprises:
Burst module: for video slicing, burst number is set;
Generation module: for generating video linking URL;
Acquisition module: for according to the URL generating, the server of access open platform, obtains the video data that URL is corresponding, and its form is JSON type;
Conversion module: for according to the video data of the JSON type of obtaining, utilize corresponding JSON to resolve interface JSON data are resolved, and be converted into the video class oneself defining;
In described video storage module, comprise:
Parsing module: for completing the resolving of video class, video information is resolved to corresponding video attribute information;
Deposit module in: for receiver, video attribute information, and call the writing in the database that incoming interface deposits corresponding video information in of distributed data base HBase.
9. distributed video vertical search system according to claim 8, is characterized in that, in the index creation module of described video information, comprises:
Link block: for connecting HBase distributed data base, obtain video information;
Packetization module: for the video information of obtaining being packaged into the data of JSON type;
Processing module: for obtaining the video information of JSON type, through a hash process, corresponding video distribution is set up to index in the middle of corresponding index burst, carry out the establishment of copy after index completes; Judge module: for judging that whether database video data completes, and in this way, finishes; Otherwise jumping to step C1 continues to carry out.
10. distributed video method for vertical search according to claim 9, is characterized in that, at described online processing unit, comprises:
Load module: user passes through user's query interface input inquiry keyword easily, submits to retrieval server;
Distribution module: the searching keyword of being responsible for user to submit to for retrieval server is distributed to each node server;
Retrieval module: receive after the retrieval request that retrieval server sends over for each node server, automatically retrieve index burst on this node, complete the process of parallel search, and result is returned;
Return to module: for retrieval server, receive the video data that each node server returns, and its video data is merged, for returning to user;
Display module: be shown to user for the video data that retrieval server is returned.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410222453.9A CN103984745B (en) | 2014-05-23 | 2014-05-23 | Distributed video method for vertical search and system |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410222453.9A CN103984745B (en) | 2014-05-23 | 2014-05-23 | Distributed video method for vertical search and system |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103984745A true CN103984745A (en) | 2014-08-13 |
| CN103984745B CN103984745B (en) | 2018-02-16 |
Family
ID=51276718
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410222453.9A Expired - Fee Related CN103984745B (en) | 2014-05-23 | 2014-05-23 | Distributed video method for vertical search and system |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103984745B (en) |
Cited By (20)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317966A (en) * | 2014-11-18 | 2015-01-28 | 国家电网公司 | Dynamic indexing method applied to quick combined querying of big electric power data |
| CN104778252A (en) * | 2015-04-16 | 2015-07-15 | 天脉聚源(北京)传媒科技有限公司 | Index storage method and index storage device |
| CN105357504A (en) * | 2015-12-17 | 2016-02-24 | 深圳市科漫达智能管理科技有限公司 | Recording and playback method and device for video stream data |
| CN105959709A (en) * | 2016-04-26 | 2016-09-21 | 北京数智源科技股份有限公司 | Multimedia video fusion application cloud platform |
| CN106055622A (en) * | 2016-05-26 | 2016-10-26 | 浪潮软件集团有限公司 | Data searching method and system |
| CN106202207A (en) * | 2016-06-28 | 2016-12-07 | 中国电子科技集团公司第二十八研究所 | A kind of index based on HBase ORM and searching system |
| CN106557591A (en) * | 2016-12-01 | 2017-04-05 | 深圳中兴网信科技有限公司 | Search method and retrieval device |
| CN106897736A (en) * | 2017-01-17 | 2017-06-27 | 华南理工大学 | A kind of multi-field non-cooperating distributed search result emerging system and its fusion method |
| CN106980699A (en) * | 2017-04-14 | 2017-07-25 | 中国科学院深圳先进技术研究院 | A kind of data processing platform (DPP) and system |
| CN107066581A (en) * | 2017-04-14 | 2017-08-18 | 北京邮电大学 | Distributed traffic monitor video data storage and quick retrieval system |
| CN107688643A (en) * | 2017-08-29 | 2018-02-13 | 环球智达科技（北京）有限公司 | Search method based on keyword |
| CN108153883A (en) * | 2017-12-26 | 2018-06-12 | 北京百度网讯科技有限公司 | Searching method and device, computer equipment, program product and storage medium |
| CN108366217A (en) * | 2018-03-14 | 2018-08-03 | 成都创信特电子技术有限公司 | Monitor video acquisition and storage method |
| CN108717382A (en) * | 2018-05-11 | 2018-10-30 | 北京奇虎科技有限公司 | Audio-video document processing method, device and terminal device based on JSON structures |
| CN109657072A (en) * | 2018-12-13 | 2019-04-19 | 北京百分点信息科技有限公司 | A kind of intelligent search WEB system and method applied to government's aid decision |
| CN110134851A (en) * | 2019-05-05 | 2019-08-16 | 北京科技大学 | A kind of search engine system and construction method based on field Intranet |
| CN110188111A (en) * | 2019-05-30 | 2019-08-30 | 上海优扬新媒信息技术有限公司 | A kind of off-line data batch updating method, apparatus and distributed memory system |
| CN110555152A (en) * | 2018-03-31 | 2019-12-10 | 甘肃万维信息技术有限责任公司 | distributed search system based on Elasticissearch framework |
| CN112015856A (en) * | 2020-08-26 | 2020-12-01 | 海看网络科技(山东)股份有限公司 | Method for realizing pinyin retrieval based on elastic search in IPTV |
| CN112131449A (en) * | 2020-09-21 | 2020-12-25 | 西北大学 | Implementation method of cultural resource cascade query interface based on elastic search |
Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101867793A (en) * | 2010-05-14 | 2010-10-20 | 蔡晓东 | Distribution type intelligent video searching system and using method |
| CN102063476A (en) * | 2010-12-13 | 2011-05-18 | 百度时代网络技术（北京）有限公司 | Video searching method and system |
| CN103020236A (en) * | 2012-12-15 | 2013-04-03 | 安科智慧城市技术（中国）有限公司 | Method, system and distributed database system for retrieving recorded video |
-
2014
- 2014-05-23 CN CN201410222453.9A patent/CN103984745B/en not_active Expired - Fee Related
Patent Citations (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101867793A (en) * | 2010-05-14 | 2010-10-20 | 蔡晓东 | Distribution type intelligent video searching system and using method |
| CN102063476A (en) * | 2010-12-13 | 2011-05-18 | 百度时代网络技术（北京）有限公司 | Video searching method and system |
| CN103020236A (en) * | 2012-12-15 | 2013-04-03 | 安科智慧城市技术（中国）有限公司 | Method, system and distributed database system for retrieving recorded video |
Non-Patent Citations (3)
| Title |
|---|
| 武毅: "基于 Lucene.Net 的全文检索研究与应用", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 牛雷: "分布式多媒体平台中视频搜索技术的研究与应用", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
| 韩云辉: "基于Lucene的数字版权资源库的构建与应用研究", 《中国优秀硕士学位论文全文数据库 信息科技辑》 * |
Cited By (30)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104317966B (en) * | 2014-11-18 | 2017-08-08 | 国家电网公司 | A kind of dynamic index method inquired about for electric power big data Rapid Combination |
| CN104317966A (en) * | 2014-11-18 | 2015-01-28 | 国家电网公司 | Dynamic indexing method applied to quick combined querying of big electric power data |
| CN104778252B (en) * | 2015-04-16 | 2018-12-21 | 天脉聚源(北京)传媒科技有限公司 | The storage method and device of index |
| CN104778252A (en) * | 2015-04-16 | 2015-07-15 | 天脉聚源(北京)传媒科技有限公司 | Index storage method and index storage device |
| CN105357504A (en) * | 2015-12-17 | 2016-02-24 | 深圳市科漫达智能管理科技有限公司 | Recording and playback method and device for video stream data |
| CN105959709A (en) * | 2016-04-26 | 2016-09-21 | 北京数智源科技股份有限公司 | Multimedia video fusion application cloud platform |
| CN106055622A (en) * | 2016-05-26 | 2016-10-26 | 浪潮软件集团有限公司 | Data searching method and system |
| CN106202207A (en) * | 2016-06-28 | 2016-12-07 | 中国电子科技集团公司第二十八研究所 | A kind of index based on HBase ORM and searching system |
| CN106202207B (en) * | 2016-06-28 | 2020-04-21 | 中国电子科技集团公司第二十八研究所 | HBase-ORM-based indexing and retrieval system |
| CN106557591A (en) * | 2016-12-01 | 2017-04-05 | 深圳中兴网信科技有限公司 | Search method and retrieval device |
| CN106897736A (en) * | 2017-01-17 | 2017-06-27 | 华南理工大学 | A kind of multi-field non-cooperating distributed search result emerging system and its fusion method |
| CN106897736B (en) * | 2017-01-17 | 2020-05-22 | 华南理工大学 | Multi-field non-cooperative distributed retrieval result fusion system and fusion method thereof |
| CN106980699B (en) * | 2017-04-14 | 2020-02-14 | 中国科学院深圳先进技术研究院 | Data processing platform and system |
| CN107066581B (en) * | 2017-04-14 | 2019-10-08 | 北京邮电大学 | The storage of distributed traffic monitor video data and quick retrieval system |
| CN107066581A (en) * | 2017-04-14 | 2017-08-18 | 北京邮电大学 | Distributed traffic monitor video data storage and quick retrieval system |
| CN106980699A (en) * | 2017-04-14 | 2017-07-25 | 中国科学院深圳先进技术研究院 | A kind of data processing platform (DPP) and system |
| CN107688643A (en) * | 2017-08-29 | 2018-02-13 | 环球智达科技（北京）有限公司 | Search method based on keyword |
| CN108153883A (en) * | 2017-12-26 | 2018-06-12 | 北京百度网讯科技有限公司 | Searching method and device, computer equipment, program product and storage medium |
| CN108153883B (en) * | 2017-12-26 | 2022-02-18 | 北京百度网讯科技有限公司 | Search method and apparatus, computer device, program product, and storage medium |
| CN108366217B (en) * | 2018-03-14 | 2021-04-06 | 成都创信特电子技术有限公司 | Monitoring video acquisition and storage method |
| CN108366217A (en) * | 2018-03-14 | 2018-08-03 | 成都创信特电子技术有限公司 | Monitor video acquisition and storage method |
| CN110555152A (en) * | 2018-03-31 | 2019-12-10 | 甘肃万维信息技术有限责任公司 | distributed search system based on Elasticissearch framework |
| CN108717382A (en) * | 2018-05-11 | 2018-10-30 | 北京奇虎科技有限公司 | Audio-video document processing method, device and terminal device based on JSON structures |
| CN108717382B (en) * | 2018-05-11 | 2021-07-13 | 北京奇虎科技有限公司 | JSON structure-based audio and video file processing method and device and terminal equipment |
| CN109657072A (en) * | 2018-12-13 | 2019-04-19 | 北京百分点信息科技有限公司 | A kind of intelligent search WEB system and method applied to government's aid decision |
| CN110134851A (en) * | 2019-05-05 | 2019-08-16 | 北京科技大学 | A kind of search engine system and construction method based on field Intranet |
| CN110134851B (en) * | 2019-05-05 | 2021-10-15 | 北京科技大学 | Search engine system based on domain intranet and construction method |
| CN110188111A (en) * | 2019-05-30 | 2019-08-30 | 上海优扬新媒信息技术有限公司 | A kind of off-line data batch updating method, apparatus and distributed memory system |
| CN112015856A (en) * | 2020-08-26 | 2020-12-01 | 海看网络科技(山东)股份有限公司 | Method for realizing pinyin retrieval based on elastic search in IPTV |
| CN112131449A (en) * | 2020-09-21 | 2020-12-25 | 西北大学 | Implementation method of cultural resource cascade query interface based on elastic search |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103984745B (en) | 2018-02-16 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103984745A (en) | Distributed video vertical searching method and system | |
| CN109344223B (en) | Building information model management system and method based on cloud computing technology | |
| Das et al. | Big data analytics: A framework for unstructured data analysis | |
| CN106126641B (en) | A kind of real-time recommendation system and method based on Spark | |
| Lee et al. | SQL-to-NoSQL schema denormalization and migration: a study on content management systems | |
| CN107402995A (en) | A kind of distributed newSQL Database Systems and method | |
| CN104252536B (en) | A kind of internet log data query method and device based on hbase | |
| CN104516979B (en) | A kind of data query method and system based on quadratic search | |
| CN104102710A (en) | Massive data query method | |
| CN102368262A (en) | Method and equipment for providing searching suggestions corresponding to query sequence | |
| CN104424199A (en) | Search method and device | |
| CN105138561B (en) | A kind of darknet space data acquisition method and device | |
| CN104462222A (en) | Distributed storage method and system for checkpoint vehicle pass data | |
| CN106407371A (en) | User comment data displaying method and system, server and client | |
| CN102968465A (en) | Network information service platform and search service method based on network information service platform | |
| CN107203532A (en) | Construction method, the implementation method of search and the device of directory system | |
| CN103914488A (en) | Document collection, identification, association, search and display system | |
| CN103914487A (en) | Document collection, identification and association system | |
| CN102508884A (en) | Method and device for acquiring hotpot events and real-time comments | |
| Kuderu et al. | Relational database to NoSQL conversion by schema migration and mapping | |
| CN103218396B (en) | The management and running visual analysis method of static Web page is generated according to visitation frequency feature | |
| Wu et al. | An Auxiliary Decision‐Making System for Electric Power Intelligent Customer Service Based on Hadoop | |
| CN105426431A (en) | Search system for distributed resource site and implementation method thereof | |
| CN102467502A (en) | Retrieval method and system | |
| CN105589863B (en) | Searching method, data processing method, device and system |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant | ||
| CF01 | Termination of patent right due to non-payment of annual fee |
Granted publication date: 20180216 Termination date: 20190523 |
|
| CF01 | Termination of patent right due to non-payment of annual fee |