WO2010146522A2 - Systems and methods for streaming and archiving video with geographic anchoring of frame contents - Google Patents
Systems and methods for streaming and archiving video with geographic anchoring of frame contents Download PDFInfo
- Publication number
- WO2010146522A2 WO2010146522A2 PCT/IB2010/052639 IB2010052639W WO2010146522A2 WO 2010146522 A2 WO2010146522 A2 WO 2010146522A2 IB 2010052639 W IB2010052639 W IB 2010052639W WO 2010146522 A2 WO2010146522 A2 WO 2010146522A2
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- video sequence
- frames
- data
- geographic
- database
- Prior art date
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/50—Information retrieval; Database structures therefor; File system structures therefor of still image data
- G06F16/58—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
- G06F16/583—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually using metadata automatically derived from the content
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/20—Information retrieval; Database structures therefor; File system structures therefor of structured data, e.g. relational data
- G06F16/29—Geographical information databases
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
- G06F16/70—Information retrieval; Database structures therefor; File system structures therefor of video data
- G06F16/78—Retrieval characterised by using metadata, e.g. metadata not derived from the content or metadata generated manually
Definitions
- the present invention relates to systems and methods for streaming and archiving video with geographic anchoring of frame contents.
- the location data for the aforementioned geo-tagging of video is typically derived from sensors associated with the imaging device, such as GPS sensors, inertial sensors and/or inclination sensors.
- sensors associated with the imaging device such as GPS sensors, inertial sensors and/or inclination sensors.
- Such sensors provide various levels of precision as to the location of the imaging device, and possibly also the direction in which the imaging device is facing, but do not provide accurate information about the geographical coverage of the video content.
- the user of a database may need to sift through many videos taken from locations which theoretically could have viewed the location of interest including videos which were taken from nearby locations but not facing in the correct direction.
- US Patent No. 6504571 discloses a system and method for querying a digital image database based on various parameters including geographic location data.
- the indexing of the video is based on the location of the imaging device, there is a wide margin of uncertainty regarding the geographical content of each image, leading to location mismatches between a requested location and coverage of video sequences retrieved from a database.
- the user typically has no way of knowing whether a particular location of interest is included within the video or not without checking all videos recorded within a certain range of the location of interest, or at least, a large subset of such videos which were taken in the right general direction. In the case of a large database, such a task may be impractical.
- the present invention relates to systems and methods for streaming and archiving video with geographic anchoring of frame contents.
- a method comprising the steps of: (a) processing frames from a source video sequence by matching image content of the frames to image data from a geographic database to derive the geographic locations of pixels within at least part of the frames; (b) compressing the source video sequence by a lossy video compression technique to generate a compressed video sequence; and (c) encoding the compressed video sequence together with data indicative of the geographic locations of pixels as a composite data stream.
- the data indicative of the geographic locations includes parameters of a transformation for mapping between image content of the frames and the geographic database.
- the data indicative of the geographic locations includes at least part of an elevation map.
- the data indicative of the geographic locations for a subset of the frames includes data for defining geographic locations relative to geographic locations defined in others of the frames.
- the composite data stream is a video data format including metadata.
- the source video sequence is a real-time video sequence output by a video sensor, the method further comprising transferring the composite data stream over a wide area network as a substantially real-time video sequence for providing geographically-anchored-video functionality at a remote location.
- a system comprising: (a) an image processor having an input for receiving frames of a source video sequence from a video source, the image processor being configured to process the frames by matching image content of the frames to image data from a geographic database to derive the geographic locations of pixels within at least part of the frames; and (b) an encoder in data communication with the image processor and configured to: (i) compress the source video sequence by a lossy video compression technique to generate a compressed video sequence; and (ii) encode the compressed video sequence together with data indicative of the geographic locations of pixels as a composite data stream.
- a data product comprising: (a) a compressed video format containing a compressed video sequence derived by compression of a source video sequence by a lossy compression technique; and (b) data combined with the video sequence as part of a composite data stream, the data being indicative of geographic locations of pixels within at least part of frames of the compressed video sequence as derived by matching image content of frames of the source video Io image data from a geographic database.
- a method comprising the steps of: (a) receiving a composite data stream comprising: (i) a compressed video format containing a compressed video sequence derived by compression of a source video sequence by a lossy compression technique, and (ii) data indicative of geographic locations of pixels within at least part of frames of the compressed video sequence as derived by matching image content of frames of the source video to image data from a geographic database; (b) displaying the compressed video sequence to a user; (c) receiving from the user in input indicative of a position within the displayed video; and (d) employing the data to derive a geographical location corresponding to the position within the displayed video.
- a method comprising the steps of: (a) processing frames from a source video sequence by matching image content of the frames to image data from a geographic database to derive the geographic locations of pixels within at least part of the frames, and hence a geographical footprint of at least one of the frames; (b) compressing the source video sequence by a lossy video compression technique to generate a compressed video sequence; (c) storing at least part of the compressed video sequence in an image database; and (d) generating at least one indexing entry associated with at least one frame of the compressed video sequence, the indexing entry including the geographical footprint of at least one frame of the compressed video sequence, thereby allowing retrieval of frames from the database based on image content location.
- a database comprising: (a) a main database containing a plurality of compressed video sequences derived by compression of source video sequences by a lossy compression technique; and (b) a database index including at least one indexing entry associated with at least one frame of the compressed video sequences, the indexing entry including a geographical footprint of at least one frame of the compressed video sequence derived by matching image content of frames of the source video sequence to image data from a geographic database, thereby allowing retrieval of frames from the database based on image content location.
- FIG. 1 is a schematic overview of a system, constructed and operative according to an embodiment of the present invention, and for implementing a method according to the present invention
- FIG. 2 is a block diagram of an MPEG transport stream multiplexing with metadata useful for implementing the present invention
- FIG. 3 is a block diagram of selectice VOD archiving according to an aspect of the present invention
- FIG. 4 is a block diagram for video and frames retrieval from a VOD system according to a further aspect of the present invention.
- the present invention has a number of aspects, each of patentable significance in its own right, which relate to transmission, storage, retrieval and/or display of images (still or video sequences) which is tied to geographic information.
- a first aspect of the invention relates to a method, system and corresponding data structure for encoding images, preferably video sequences, with information sufficient to define the geographic coverage of the content of the images.
- a second aspect of the present invention relates to an image storage and/or searching system and method which enables selective retrieval of images of a desired geographical location. Both of these aspects will be described by reference to FIG. 1 which shows an overview of a system 10 for implementing an embodiment of the present invention combining these two aspects. In this context, it is important to distinguish between information relating to camera position and information relating to the content of the images. As pointed out above, tagging of images with the camera position can give a general indication of where the images were taken, but is insufficient to accurately and reliably identify the geographic coverage of the images.
- the information for encoding within the video sequence is derived from registration of the images by a registration module 12, directly or via an intermediate reference image, to images of a geographic image database 14, thereby giving an accurate and reliable indication of the actual geographical coverage of the image.
- the resulting registration data may be provided together with the video to a local player 16, corresponding to the existing functionality of on-site geo- registered video.
- an embodiment of the present invention makes this same functionality available at a remote site by encoding geographical coverage information together with a compressed video data stream at encoder 18, all as described in more detail below.
- this geographical coverage information encoded within the video sequence renders the video sequence "portable" in the sense that the geographical information becomes available to a subsequent recipient of the video sequence without the recipient necessarily requiring access to the geographic database infrastructure, intermediate reference images or scene- matching processing systems.
- the video sequence can then be compressed using high compression ratio lossy compression techniques, such as the various MPEG standards, for network transmission while maintaining the functionality of a geo-registered video sequence.
- the video sequence can then be used in many different ways, and in many different contexts, some of which will be detailed below in the context of additional aspects of the present invention.
- the video sequence encoded according to this aspect of the present invention may be transmitted to one or more remote viewing station (at remote player 20) where various interactive functionality relating to geographical location within the images can be provided (in a manner similar to the real-time on-site displays of military systems described above, optionally substantially in real time), it may be stored in a database 22 for retrieval (the entire sequence, selected clips or individual frames) according to a geographical location based query, and it may be input into any other system for further processing based on the geographical anchoring of the image content.
- the information defining the geographic coverage of the content of the images can be provided in various different forms.
- the information is preferably sufficient to determine geographic location on a pixel-by-pixel basis within the image.
- the information may include parameters of a transformation which maps each pixel (or group of pixels) to a geographical location.
- geographical coverage of an image may be de ⁇ ned by a small number of geographically anchored points (typically, pixels) for the image. Typically, four points are sufficient.
- the transformation or anchor point locations are typically used together with information about the relief of the terrain (for example, a digital terrain map ("DTM") or a digital model of an urban landscape).
- DTM digital terrain map
- This relief information may be provided to the end user from a separate source (geographic database). Alternatively, in some implementations, the relief information may be included as metadata also embedded in the video sequence.
- the information for defining the geographic coverage of the content of the images may be encoded for certain frames by definition relative to another frame. For example, optical flow or feature tracking techniques may be used to link pixels of one frame to corresponding features of a nearby frame which are themselves anchored by one of the above techniques.
- each frame of the video sequence is provided with geographical anchoring information (e.g., transformation parameters, specific geographic information and optionally additional information about the video content), although implementations in which anchoring information is provided only for intermittent frames also fall within the scope of the present invention.
- geographically anchored frames may be at fixed intervals, or may be chosen according to any other criteria, for example, when the overlap with the previous anchored frame falls below a certain level. If so desired, the use of such criteria may facilitate reconstruction of geographic data for the intervening frames based on techniques such as optical flow or feature tracking performed at the playback device. These techniques are computationally relatively light.
- the information for defining the geographic location of the content of the images is preferably encoded within the data structure of the video sequence itself.
- a wide range of data structures could be used for encoding the images (frames) of the video sequence together with the additional information of the invention.
- a standard format of a type allowing insertion of metadata is used.
- An example of such a format is MPEG2-TS or MPEG4-TS with "private streams" (see FIG. 2).
- a method and system for generating video sequences encoded according to this aspect of the present invention typically includes components for performing the registration process to generate a transformation mapping images to geographically anchored reference images.
- the system preferably includes additional processing modules, where required, for converting the derived frame-to- reference transformation into the data type and format required.
- the system may also generate a separate output of the geographical footprint of some or all of the images, and a corresponding index sufficient to identify the image, for storage in database index 24.
- the source for the video sequences processed may be real-time sampled video, or may be prerecorded video from any source.
- an index 24 is preferably assembled containing a geographical "footprint" for each indexed image.
- the footprint is defined by a polygon, typically a rectangle, trapezoid or trapezium, which corresponds substantially to the geographical area covered by the image.
- parts of the image in the far-field i.e., with low spatial resolution, may be excluded from the polygon of the indexed footprint.
- the definition of the footprint may be simplified to a standardized polygon, for example a rectangle, inset within the full footprint, and corresponding to the central frame or otherwise defined primary region of coverage of the image.
- an index of image footprints By maintaining such an index of image footprints, it becomes possible to send a query from remote player 20 to search index 24 and retrieve from an associated database, or from any of a plurality of storage devices covered by the index, any or all images which include a specific location of interest within their footprint area for viewing on remote player 20.
- An example of such a video-on-demand query is illustrated schematically in FIG. 4.
- an image returned as matching the required location according to this aspect of the invention is known to have the location of interest within its footprint. This makes it possible to pinpoint images of a certain location, building or other point of interest within a store of images so large that it would otherwise be impractical to search by camera location alone.
- the index preferably also includes various additional searchable data.
- the index may advantageously contain any one or more of the following types of data associated with the image identifiers and footprint data: time and date on which the image was taken; camera location and/or other data allowing searching for views from a given direction (for example, the front or back of a building); image resolution data; image quality data; and type of imaging sensor used (for example, thermal images or visible light images).
- Footprint indexing is preferably done on a frame-by-frame basis, or at least for intermittent frames distributed through a video sequence, thereby allowing selective retrieval or selective viewing of relevant frames from within an extended video sequence.
- indexing of video sequences as a unit also falls within the scope of the present invention and may be preferred in certain circumstances.
- a video sequence is indexed as a unit, it is typically preferable to generate a combination footprint which indicates the geographical area covered by the entirety of the sequence, or an approximation thereto.
- the footprint data may be generated directly by the image registration system described above and provided as a separate output.
- a separate indexing processor module may be used to derive the footprint information from the frame content location encoded video stream described above.
- the footprint may also be a useful tool in management of a large image database, allowing implementation of a range of rules 26 for selective storage or prioritizing database content.
- the footprints can be processed to determine what frames include information relating to locations or regions of interest and to selectively store such frames only.
- the footprints can be used to eliminate excessive redundancy of data, and optionally to discard frames whose content has been superseded by one or more subsequently added frames.
- footprint information is available for each frame, such rules can be tailor made in a highly specific manner, for example, keeping all historical data for geographical locations of particularly high priority, some lesser degree of coverage (e.g., most up-to-date information only, or highest resolution information only, or earliest plus latest data) for regions of lesser interest.
- An example of a selective archiving process according to an aspect of the present invention is illustrated schematically in FIG. 3. Many other such variants will be clear to one ordinarily skilled in the art.
Abstract
Description
Claims
Priority Applications (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AU2010261433A AU2010261433B2 (en) | 2009-06-14 | 2010-06-14 | Systems and methods for streaming and archiving video with geographic anchoring of frame contents |
SG2011092392A SG176835A1 (en) | 2009-06-14 | 2010-06-14 | Systems and methods for streaming and archiving video with geographic anchoring of frame contents |
US13/378,051 US9020038B2 (en) | 2009-06-14 | 2010-06-14 | Systems and methods for streaming and archiving video with geographic anchoring of frame contents |
EP10789090.7A EP2443827A4 (en) | 2009-06-14 | 2010-06-14 | Systems and methods for streaming and archiving video with geographic anchoring of frame contents |
Applications Claiming Priority (3)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
IL199336 | 2009-06-14 | ||
IL199366 | 2009-06-14 | ||
IL19933609 | 2009-06-14 |
Publications (2)
Publication Number | Publication Date |
---|---|
WO2010146522A2 true WO2010146522A2 (en) | 2010-12-23 |
WO2010146522A3 WO2010146522A3 (en) | 2011-03-31 |
Family
ID=43569900
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/IB2010/052639 WO2010146522A2 (en) | 2009-06-14 | 2010-06-14 | Systems and methods for streaming and archiving video with geographic anchoring of frame contents |
Country Status (5)
Country | Link |
---|---|
EP (1) | EP2443827A4 (en) |
AU (1) | AU2010261433B2 (en) |
IL (1) | IL206371A (en) |
SG (1) | SG176835A1 (en) |
WO (1) | WO2010146522A2 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113205010A (en) * | 2021-04-19 | 2021-08-03 | 广东电网有限责任公司东莞供电局 | Intelligent disaster-exploration on-site video frame efficient compression system and method based on target clustering |
Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20020122564A1 (en) * | 2001-03-05 | 2002-09-05 | Rhoads Geoffrey B. | Using embedded identifiers with images |
US6504571B1 (en) * | 1998-05-18 | 2003-01-07 | International Business Machines Corporation | System and methods for querying digital image archives using recorded parameters |
US20060251321A1 (en) * | 2005-05-04 | 2006-11-09 | Arben Kryeziu | Compression and decompression of media data |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6282362B1 (en) * | 1995-11-07 | 2001-08-28 | Trimble Navigation Limited | Geographical position/image digital recording and display system |
US6741790B1 (en) * | 1997-05-29 | 2004-05-25 | Red Hen Systems, Inc. | GPS video mapping system |
US7254249B2 (en) * | 2001-03-05 | 2007-08-07 | Digimarc Corporation | Embedding location data in video |
-
2010
- 2010-06-14 SG SG2011092392A patent/SG176835A1/en unknown
- 2010-06-14 AU AU2010261433A patent/AU2010261433B2/en active Active
- 2010-06-14 WO PCT/IB2010/052639 patent/WO2010146522A2/en active Application Filing
- 2010-06-14 IL IL206371A patent/IL206371A/en active IP Right Grant
- 2010-06-14 EP EP10789090.7A patent/EP2443827A4/en not_active Withdrawn
Patent Citations (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6504571B1 (en) * | 1998-05-18 | 2003-01-07 | International Business Machines Corporation | System and methods for querying digital image archives using recorded parameters |
US20020122564A1 (en) * | 2001-03-05 | 2002-09-05 | Rhoads Geoffrey B. | Using embedded identifiers with images |
US20060251321A1 (en) * | 2005-05-04 | 2006-11-09 | Arben Kryeziu | Compression and decompression of media data |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN113205010A (en) * | 2021-04-19 | 2021-08-03 | 广东电网有限责任公司东莞供电局 | Intelligent disaster-exploration on-site video frame efficient compression system and method based on target clustering |
Also Published As
Publication number | Publication date |
---|---|
IL206371A (en) | 2014-03-31 |
AU2010261433A1 (en) | 2012-01-19 |
AU2010261433B2 (en) | 2016-05-26 |
EP2443827A2 (en) | 2012-04-25 |
WO2010146522A3 (en) | 2011-03-31 |
IL206371A0 (en) | 2010-12-30 |
EP2443827A4 (en) | 2016-11-23 |
SG176835A1 (en) | 2012-01-30 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9191693B2 (en) | Information processing device, information processing system, and program | |
US8200712B2 (en) | System and method for generating a virtual tour on a display device | |
US6282362B1 (en) | Geographical position/image digital recording and display system | |
US8189690B2 (en) | Data search, parser, and synchronization of video and telemetry data | |
US20070173956A1 (en) | System and method for presenting geo-located objects | |
US8321395B2 (en) | Associating digital images with waypoints | |
US7675549B1 (en) | Imaging architecture for region and time of interest collection and dissemination | |
CN101640775B (en) | Video recording method, photo taking method and mobile terminal | |
US9020038B2 (en) | Systems and methods for streaming and archiving video with geographic anchoring of frame contents | |
TW201142751A (en) | Video processing system generating corrected geospatial metadata for a plurality of georeferenced video feeds and related methods | |
IL269051A (en) | Real time frame alignment in video data | |
US20100191765A1 (en) | System and Method for Processing Images | |
KR100692792B1 (en) | Location based service system and method grasping terminal location using location based image data, and mobile terminal applied to the same | |
US20040066391A1 (en) | Method and apparatus for static image enhancement | |
AU2010261433B2 (en) | Systems and methods for streaming and archiving video with geographic anchoring of frame contents | |
US20160127617A1 (en) | System for tracking the position of the shooting camera for shooting video films | |
Lewis | Linking spatial video and GIS | |
US20230237796A1 (en) | Geo-spatial context for full-motion video | |
JP2008225353A (en) | Image display system, image display method, and program | |
JP2007067974A (en) | Video monitoring system and video monitoring method | |
WO2023147376A1 (en) | Geo-spatial context for full-motion video | |
CN114268757A (en) | Real-time positioning video generation method | |
Lewis et al. | Role of Spatial Video in GIS | |
CN115914789A (en) | Infrared thermal imaging endoscope system | |
KR20170102083A (en) | System and method for providing contents |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 10789090 Country of ref document: EP Kind code of ref document: A2 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2010261433 Country of ref document: AU |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
WWE | Wipo information: entry into national phase |
Ref document number: 9344/CHENP/2011 Country of ref document: IN |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2010789090 Country of ref document: EP |
|
ENP | Entry into the national phase |
Ref document number: 2010261433 Country of ref document: AU Date of ref document: 20100614 Kind code of ref document: A |
|
WWE | Wipo information: entry into national phase |
Ref document number: 13378051 Country of ref document: US |