EP2076808A1 - Method for acquiring, processing and presenting images and multimedia navigating system for performing such method - Google Patents

Method for acquiring, processing and presenting images and multimedia navigating system for performing such method

Info

Publication number
EP2076808A1
EP2076808A1 EP06832343A EP06832343A EP2076808A1 EP 2076808 A1 EP2076808 A1 EP 2076808A1 EP 06832343 A EP06832343 A EP 06832343A EP 06832343 A EP06832343 A EP 06832343A EP 2076808 A1 EP2076808 A1 EP 2076808A1
Authority
EP
European Patent Office
Prior art keywords
panoramic
video
images
camera
image
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Withdrawn
Application number
EP06832343A
Other languages
German (de)
French (fr)
Inventor
Mauro Marenzi
Claudio Colosio
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Geomondo Srl
Mykugi Srl
Original Assignee
Geomondo Srl
Mykugi Srl
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Geomondo Srl, Mykugi Srl filed Critical Geomondo Srl
Publication of EP2076808A1 publication Critical patent/EP2076808A1/en
Withdrawn legal-status Critical Current

Links

Classifications

    • GPHYSICS
    • G02OPTICS
    • G02BOPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B13/00Optical objectives specially designed for the purposes specified below
    • G02B13/06Panoramic objectives; So-called "sky lenses" including panoramic objectives having reflecting surfaces
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C11/00Photogrammetry or videogrammetry, e.g. stereogrammetry; Photographic surveying
    • G01C11/02Picture taking arrangements specially adapted for photogrammetry or photographic surveying, e.g. controlling overlapping of pictures
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34Route searching; Route guidance
    • G01C21/36Input/output arrangements for on-board computers
    • G01C21/3602Input other than that of destination using image analysis, e.g. detection of road signs, lanes, buildings, real preceding vehicles using a camera
    • GPHYSICS
    • G01MEASURING; TESTING
    • G01CMEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C21/00Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00
    • G01C21/26Navigation; Navigational instruments not provided for in groups G01C1/00 - G01C19/00 specially adapted for navigation in a road network
    • G01C21/34Route searching; Route guidance
    • G01C21/36Input/output arrangements for on-board computers
    • G01C21/3626Details of the output of route guidance instructions
    • G01C21/3635Guidance using 3D or perspective road maps
    • G01C21/3638Guidance using 3D or perspective road maps including 3D objects and buildings
    • G06T3/12
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50Constructional details
    • H04N23/55Optical parts specially adapted for electronic image sensors; Mounting thereof
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60Control of cameras or camera modules
    • H04N23/698Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • HELECTRICITY
    • H04ELECTRIC COMMUNICATION TECHNIQUE
    • H04NPICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00Details of television systems
    • H04N5/222Studio circuitry; Studio devices; Studio equipment
    • H04N5/262Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2628Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Abstract

A method is disclosed for acquiring, processing and presenting panoramic images in a visor according to a function of three-dimensional rendering, such method being adapted to directly correlate spherical coordinates (6) of a three-dimensional map (7) to Cartesian coordinates of 'a map (5) . Α system is further described for managing images of a urban territory, that operates with such method, comprising a video-camera (1) coupled with a parabolic mirror (2) by means of a framework (3) integral with means moving along a urban route and comprising a visor of panoramic recordings of the video-camera (1) related to acquisition data of a satellite tele-detecting system.

Description

METHOD FOR ACQUIRING, PROCESSING AND PRESENTING IMAGES AND MULTIMEDIA NAVIGATING SYSTEM FOR PERFORMING SUCH METHOD
The present invention deals with a method for acquiring, processing and presenting images and the application of such method in terrestrial navigation, in particular in urban territories.
The invention refers to: systems for taking photographs (G03B: APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR); methods for using photographic systems (G06K9/00: Methods or arrangements for reading or recognising printed or written characters or for recognising patterns, e.g. fingerprints); measuring and navigating systems (GOlC: MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY) ; methods for transforming three-dimensional image models (G06T15/00: Three dimensional (3D) image rendering, e.g. from a model to a bit-mapped image) . The prior art is represented in particular by EP-A-O 908 053, that deals with an apparatus for capturing and displaying 360° panoramic images (panoramic images), equipped with an apparatus for correcting astigmatism optic aberration, the so-corrected image being captured as annular image, then transformed through a transformation mechanism.
US-A-5 563 650 deals with a method for producing panoramic images captured by means of a fish-eye optics and transformed into digital form by removing distortions through software. The method further correlates panoramic' images to spots of a layout and a series of information such as address, data, orientation, direction, street names and names of members living in displayed houses.
US 2004/0169724 Al uses a panoramic image to create spherical environment maps correcting perspective distortions, obtaining a plurality of images, or their portions, free from distortions.
The above described state of the art has the problem of having to real-time correct perspective distortions resulting from the capture of panoramic images through optics of the fish-eye type or the reflection of panoramic images through convex mirrors. A correction of the perspective distortion, in fact, implies the use of an additional unwrapping software, to perform which a dedicated hardware must be used. The post processing activity linked to the need of correcting perspective distortions typical of panoramic images obtained through particular optics such as fish-eye or convex mirrors, therefore implies a burdensome CPU computation activity; it also implies the loss of image quality due to the treatment of digitised image pixels (source video) detected by an optic chamber of the digital type and handled in terms of pixel translation and interpolation. This problem is more strongly felt in case of a motion picture shooting in which the disturbance induced by vibrations of the camera supporting device is not easily . filtered, resulting into an unpleasant effect on the final yield of the produced image.
The difficulty linked to real-time uses of a panoramic image taken by a video-camera has been confronted and solved through an innovative method for acquiring, processing and presenting images, subject matter of the present document. A further object of the invention is developing a multimedia navigating system coupled with such method.
This method is a method for acquiring images or sequences of panoramic images and for processing the same according to a spherical mapping whose space coordinates are bi-univocally related to two-dimensional coordinates of a plane mapping obtained from the image captured by the video-camera, and for displaying panoramic photographs in a visor shaped as a two-dimensional or three-dimensional map, pointing out position and direction thereof with respect to a geo-stationary reference.
The above and other objects and advantages of the invention, as will result from the following description, are obtained with a method as claimed in claim 1. Preferred embodiments and non-trivial variations of the present invention are the subject matter of the dependent claims.
The present invention will be better described by some preferred embodiments thereof, provided as a non-limiting example, with reference to the enclosed drawings, in which:
- figure 1 shows the system for capturing panoramic images at 360° composed of a video-camera coupled with a parabolic mirror; figure 2 shows an orthogonal projection view of the parabolic mirror that reflects a panoramic image at 360° as annular image;
- figure 3 shows the two-dimensional mapping of the annular image taken by the video-camera;
- figure 4 shows an axonometric view of the spherical virtual space on which the annular image is projected (image rendering form the model to the bit-mapped image) ;
- figure 5 shows the image extracted from the spherical virtual space and projected on a visor; - figure 6 and figure 7 show the handling of a portion of the two-dimensional mapping of the annular image taken by the video-camera;
- figure 8 shows the portion of spherical virtual space used for projecting the portion of image as seen in figure 7 ; and
- figure 9 shows the yield of a multimedia application that integrates the panoramic images to a two-dimensional map.
A video-camera 1 is aligned with a convex mirror 2 through a rigid framework 3. The convex mirror 2 in this case is parabolic. The video-camera 1 can be of a commercial type - provided with low video resolution - or of an industrial type - provided with high quality video resolution - equipped with a driver that is able to process a plane image depending on the electro-optical sensor coupled with the video-camera itself. The image 4 reflected by the parabolic mirror 2 is an image of the panoramic type characterised by a view field of about 110° of angular opening, suitably slanted on its top and bottom parts with respect to the line of framed horizon. The video-camera 1 receives the image 4 reflected and projected by the parabolic mirror 2 associating the points of a Cartesian plane 5 to an array of active electro-optical elements. The e ro-o ic array of H x V pixels in which: H = V guarantees a management of square-shaped images with a resolution power equal to 2.
The method for acquiring and managing the panoramic image of the present invention substantially consists in performing a rendering in a three-dimensional environment starting from a two-dimensional map built in the Cartesian plane 5 (three-dimensional (3D) image rendering, e.g. from a model to a bit-mapped image) . In this case, such three- dimensional environment corresponds to a mapping in spherical coordinates 6. The rendering of the image 4 built in the Cartesian plane 5 and projected in the virtual space in spherical coordinates 6 is represented by image 7.
In case of a convex mirror 2 equipped with spherical surface, the algorithm regulating the bi-univocal correspondence between the two maps, respectively built in plane 5 and in the sphere in spherical coordinates 6, is given by: u = α*cos ( β) ; v = α*sin (β) ; in which angle α moves in elevation, angle β moves in azimuth.
In case of a convex mirror 2 equipped with a parabolic surface, the algorithm regulating the bi-univocal correspondence between the two maps in plane 5 and in the sphere in spherical coordinates 6 is given by: u = f (α) *cos (β) ; v = f (α) *sin(β) ; in which angle α moves in elevation, angle β moves in azimuth, f(α) represents a correction factor depending on the parabolic mirror curvature.
Figure 5 shows the panoramic photograph resulting from the image 4, this latter one processed in three-dimensional mapping 7 and therefore projected through a visor.
The bi-univocal correspondence between map associated with the Cartesian plane 5 and three-dimensional map 7 also allows partially managing the image 4. In fact, it is possible to cut a portion of image 4, namely to perform a zooming, cutting a portion 8 of Cartesian plane 5, and to re-build the three-dimensional rendering, obtaining the three-dimensional map shown in figure 8.
The method for acquiring and rendering in a panoramic image in a three-dimensional environment according to the above procedure enables a series of real-time functions with a small use of CPU memory and computation resources. First of all, the method is free from post-processing unwrapping processing. The two-dimensional mapping 5 of a square portion 1:1 and power resolution 2 is the excellent format for exploiting the hardware acceleration of 3D video cards, optimising the relationship between speed (frames/second) and image quality. The direct rendering in spherical coordinates does not require high performance hardware, operating on conventional platforms (Windows, Mac) ; moreover, it does not require the installation of additional software; after installing a Shockwave plug-in, it is possible to directly navigate in Internet.
From the point of view of optical yield, the video recording of the panoramic image is realistic and free from perspective distortions. The method also allows: using video sources with integrated audio, displaying video images recorded on moving means; performing the zooming on portions of panoramic images; saving panoramic images in a video format or in sequences of frames; managing the frames by using conventional commands of video-recorders and DVD players DVD; positioning dynamic hotspots directly in fixed video points; displaying three-dimensional elements; providing the video with special three-dimensional effects of the fog effect or particle, LensFlare effect types; processing the frames with video effects such as de- saturation, blur, contrast; interacting with other external programs such as satellite maps and GPS navigators, two- and three-dimensional geographic maps, videogames.
The method for acquiring and managing the panoramic image of the present invention has been successfully used within displaying systems for urban territory and guided navigation through a geo-stationary reference.
In both systems, the urban territory is captured as images or a sequence of panoramic (video) images pf external and internal environments, complying with a sequence of image acquisition as guarantee of the integral coverage of the captured _ territory; in this case, the optical system of the type comprising the video-camera 1 coupled with a a parabolic mirror 2 assembled on the framework 3 is applied to a moving means along a urban route; the optical device operates at a height from the ground of at least 150 centimetres, under good meteorologic conditions in order to realise a perfect brightness yield. A software program correlates panoramic recordings of video-camera 1 with acquisition data of a GPS satellite antenna, realising a data base of images coupled with geo- referred longitude, latitude and direction data.
A further software program checks the display of two- dimensional and three-dimensional panoramic images in a visor by showing geo-referenced data coupled with each image. The system (player) allows navigating through cursor keys or interface keys or through active keys on the reference geographic map.
The thereby obtained geographic map is equipped with a special position pointing device that is able to point out the framing direction of the panoramic image or a portion thereof, as pointed out in the yield images in figure 9. The player can contain, inside the displayed panoramic photograph window, some pieces of information, texts and hotspots keys that can be activated by a computer through mouse or touch screen.
These hotspots allow accessing related texts, by- opening Internet windows, or by opening other media such as video, photo, etc.
The player is written according to the most common programming languages (C, Java, Visual Basic, etc.) and, for the WEB environment, it exploits plug-ins for browsers like Shockwave, Flash, Java, ActiveX. Moreover, the player can operate in a standalone mode or can be managed through a latest-generation cellular phone in a Java and Symbian environment .

Claims

CIAIMS
1. Method for acquiring, processing and presenting panoramic images in a visor according to a three- dimensional rendering function, characterised in that it is adapted to directly correlate spherical coordinates (6) of a three-dimensional map (7) to Cartesian coordinates of a map (5) .
2. Method according to claim 1, characterised in that it is adapted to capture the panoramic images by means of a video-camera (1) coupled with a convex mirror (2), both being made integral by means of a framework (3) .
3. Method according to claim 2, characterised in that the algorithm regulating a bi-univocal correspondence between the two maps in the plane (5) and in the sphere in spherical coordinates (β) is function of the convex mirror (2) curvature.
4. Method according to any one of claims 1 to 3, characterised in that the convex mirror (2) has a parabolic geometry.
5. Method according to claim 3, characterised in that Ia two-dimensional map (5) has sizes with a 1:1 proportion and its optical resolution is a power equal to 2.
6. Method according to any one of the previous claims, characterised in that it does not provide for using an additional unwrapping software or other software for correcting perspective distortions that are typical of panoramic images obtained through particular optics, such as "fish-eye" or convex mirrors.
7. System for managing images of a urban territory of a type operating according to the method according to any one of the previous claims, comprising a video-camera (1) couplet with a parabolic mirror (2) by means of the framework (3), said framework (3) being integral with means moving along a urban route, characterised in that it comprises a visor of panoramic recordings of the video- camera (1) related with acquisition data of a satellite tele-detecting system.
8. System according to claim 7, characterised in that panoramic recordings compose a database of images coupled with geo-referenced longitude, latitude and direction data.
9. System according to any one of claims 7 or 8, characterised in that the panoramic recordings visor is adapted to show a special position pointing device that is able to point out the framing direction of the panoramic image or a portion thereof.
10. System according to any one of claims 7 to 9, characterised in that it is adapted to include, within the displayed panoramic photograph window, some pieces of information, texts and hotspots keys that can be activated by a computer through mouse or touch screen.
11. System according to claim 10, characterised in that said hotspots allow accessing related texts, by opening Internet windows, or by opening other media such as video, photo, etc.
EP06832343A 2006-08-17 2006-11-29 Method for acquiring, processing and presenting images and multimedia navigating system for performing such method Withdrawn EP2076808A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
IT000609 IT1378233B1 (en) 2006-08-17 2006-08-17 METHOD OF ACQUISITION, PROCESSING AND PRESENTATION OF IMAGES AND MULTIMEDIA NAVIGATION SYSTEM COMBINED WITH THIS METHOD
PCT/IT2006/000826 WO2008020461A1 (en) 2006-08-17 2006-11-29 Method for acquiring, processing and presenting images and multimedia navigating system for performing such method

Publications (1)

Publication Number Publication Date
EP2076808A1 true EP2076808A1 (en) 2009-07-08

Family

ID=37877420

Family Applications (1)

Application Number Title Priority Date Filing Date
EP06832343A Withdrawn EP2076808A1 (en) 2006-08-17 2006-11-29 Method for acquiring, processing and presenting images and multimedia navigating system for performing such method

Country Status (3)

Country Link
EP (1) EP2076808A1 (en)
IT (1) IT1378233B1 (en)
WO (1) WO2008020461A1 (en)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107040792A (en) * 2017-05-05 2017-08-11 暴风集团股份有限公司 Panoramic video player method, panoramic video playing device and player

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
ITTO20080310A1 (en) * 2008-04-22 2008-07-22 Geomondo S R L APPLICATION OF IDENTIFICATION, GEO-LOCATION AND MANAGEMENT OF POINTS OF INTEREST (POI)
US8701040B2 (en) 2008-09-29 2014-04-15 Microsoft Corporation Panoramic graphical user interface
FR2995091B1 (en) * 2012-08-28 2015-08-14 Commissariat Energie Atomique IMAGING DEVICE WITH A WIDE ANGLE OF VIEW
ITTO20120877A1 (en) * 2012-10-09 2013-01-08 Geomondo S R L METHOD OF VIRTUAL EXPLORATION OF THE THREE-DIMENSIONAL SPACE OF AN EXHIBITION FAIR
CN110060201B (en) * 2019-04-15 2023-02-28 深圳市数字城市工程研究中心 Hot spot interaction method for panoramic video
CN112242002B (en) * 2020-10-09 2022-07-08 同济大学 Object identification and panoramic roaming method based on deep learning

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5050966A (en) * 1988-07-06 1991-09-24 Kaiser Aerospace & Electronics Corporation Optical combiner collimating apparatus
US5563650A (en) * 1992-11-24 1996-10-08 Geeris Holding Nederland B.V. Method and device for producing panoramic images, and a method and device for consulting panoramic images
JPH10504115A (en) * 1994-08-10 1998-04-14 ヴァーチャリティ (アイピー) リミテッド Head mounted display optics
US6459451B2 (en) * 1996-06-24 2002-10-01 Be Here Corporation Method and apparatus for a panoramic camera to capture a 360 degree image
JP3054146B1 (en) * 1999-01-06 2000-06-19 宜昇科技股▲分▼有限公司 Panoramic image device
JP3804916B2 (en) * 2001-02-09 2006-08-02 シャープ株式会社 Imaging system, program used for controlling image data thereof, method for correcting distortion of captured image in imaging system, and storage medium storing procedure thereof
US7567274B2 (en) * 2002-12-09 2009-07-28 Frank Edughom Ekpar Method and apparatus for creating interactive virtual tours

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2008020461A1 *

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107040792A (en) * 2017-05-05 2017-08-11 暴风集团股份有限公司 Panoramic video player method, panoramic video playing device and player

Also Published As

Publication number Publication date
WO2008020461A1 (en) 2008-02-21
ITTO20060609A1 (en) 2006-11-16
IT1378233B1 (en) 2010-07-30

Similar Documents

Publication Publication Date Title
US9858717B2 (en) System and method for producing multi-angle views of an object-of-interest from images in an image dataset
US7415356B1 (en) Techniques for accurately synchronizing portions of an aerial image with composited visual information
US8611643B2 (en) Spatially registering user photographs
US9880010B2 (en) Method of and arrangement for mapping range sensor data on image sensor data
JP5118787B1 (en) Orthorectification of oblique images joined to the direct view and their applications
US7583858B2 (en) Image processing based on direction of gravity
WO2008152740A1 (en) Digital aerial photographing three-dimensional measurement system
US20060078215A1 (en) Image processing based on direction of gravity
US20070098238A1 (en) Imaging methods, imaging systems, and articles of manufacture
US20120300020A1 (en) Real-time self-localization from panoramic images
WO2008020461A1 (en) Method for acquiring, processing and presenting images and multimedia navigating system for performing such method
US20070070233A1 (en) System and method for correlating captured images with their site locations on maps
US20070127848A1 (en) Apparatus, system and method for mapping information
EP1331605A2 (en) Omnidirectional visual system and method
Boukerch et al. Development of panoramic virtual tours system based on low cost devices
WO2009069165A2 (en) Transition method between two three-dimensional geo-referenced maps
JP5262232B2 (en) Image orientation display method and apparatus, and photograph
Zheng et al. Scanning scene tunnel for city traversing
JP3660108B2 (en) Image storage method and machine-readable medium
JP2005115500A (en) Information processor and information processing method
JP2004310686A (en) Image processing method and device
JPH1118007A (en) Omnidirectional image display system
CN111581322A (en) Method, device and equipment for displaying interest area in video in map window
JP2003051021A (en) Information updating device, information acquisition device and information update processing device
KR101205961B1 (en) Apparatus for caculating 3D spatial coordinates of digital images

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20090305

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR

AX Request for extension of the european patent

Extension state: AL BA HR MK RS

17Q First examination report despatched

Effective date: 20100520

DAX Request for extension of the european patent (deleted)
STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: THE APPLICATION IS DEEMED TO BE WITHDRAWN

18D Application deemed to be withdrawn

Effective date: 20120601