HK1136899A - Method of and apparatus for producing a multi-viewpoint panorama - Google Patents

Abstract

A method of producing multi-viewpoint panorama of a roadside is disclosed. The method comprises: - acquiring a set of laser scan samples obtained by at least one terrestrial based laser scanner mounted on a moving vehicle, wherein each sample is associated with location data and orientation data; - acquiring at least one image sequence, wherein each image sequence is obtained by means of a terrestrial based camera mounted on the moving vehicle, wherein each image of the at least one image sequences is associated with location and orientation data; - extracting a surface from the set of laser scan samples and determining the location of said surface in dependence of the location data associated with the laser scan samples; - producing a multi-viewpoint panorama for said surface from the at least one image sequence in dependence of the location of the surface and the location and orientation data associated with each of the images.

Description

Method and apparatus for generating multi-view panorama

Technical Field

The invention relates to a method for generating a multi-view panorama. The invention further relates to a method of generating a roadside panorama from a multi-view panorama. The invention further relates to an apparatus for multi-view panoramas, a computer program product and a processor readable medium carrying the computer program product. The invention further relates to a computer implemented system using said roadside panorama.

Background

Today, people use navigation devices to navigate themselves along roads or use map displays on the internet. The navigation device displays in its display a planar perspective, an angular perspective (bird's eye view) or a variable-scale "2D" map of the location. Only information about roads or some simple attribute information about areas, such as lakes and parks, is displayed in the display. Such information is in fact an abstract representation of the location and does not show what is visible to a person or camera located at the location shown (actually or virtually) in the display. Some internet applications display overhead pictures taken from satellites or airplanes and still rarely display a limited set of photographs taken from the road, perhaps near the user's location (real or virtual) and facing in the same general direction that the user intends to look at.

More accurate and realistic roadside views are needed in future navigation devices and internet applications. The roadside view enables the user to see what is visible in a particular location and to verify very easily when driving whether the navigation device is using the correct location or whether the place of interest being asked on the internet is really where he wants to go or simply views the area in more detail for entertainment or commercial reasons. In the display, the user can then immediately see whether the building seen on the display corresponds to a building that he can see at the roadside or a building that is imagined according to memory or other description. Panoramic images generated from images captured from different perspectives are considered multi-viewpoint or multi-view. Another type of panoramic image is a slit scan panoramic image. In its simplest form, a strip panorama exhibits an orthographic projection along the horizontal axis and a perspective projection along the vertical axis.

Systems for generating multi-view panoramas are known from the teaching of ashham acalla (Aseem Agarwala) et al (photography long scenes with multi-view panoramas) (ACM graphic journal (proceedings sigraph 2006), 2006). A system for generating a multi-view panorama of a long, substantially planar scene (e.g., the front of a building along a city street) is generated from a relatively sparse set of photographs captured using a handheld camera. The user must recognize the principal plane of the scene being photographed. The system then automatically computes the panorama using markov random field optimization.

Another technique for depicting real images of surroundings is to develop a full 3D model of the area and then apply real textures to the outside dimensions of each building. An application (e.g. an application in a navigation unit or on the internet) may then use 3D rendering software to construct a real picture of the surrounding object.

Disclosure of Invention

The present invention seeks to provide an alternative method of generating a multi-view panorama and an alternative way of providing a high quality, easily understandable set of images representing a virtual surface at near-photographic quality, which are easy to manipulate to obtain a pseudo-real perspective image without the added cost and complexity of developing a 3D model.

According to the invention, the method comprises:

obtaining a set of laser scan samples obtained by a laser scanner mounted on a moving vehicle, wherein each sample is associated with location data;

acquiring at least one sequence of images, wherein each sequence of images has been obtained with a land-based camera mounted on the moving vehicle, wherein each image of the at least one sequence of images is associated with position and orientation data;

extracting a surface from the set of laser scan samples and determining a location of the surface from the location data associated with the laser scan samples;

generating a multi-view panorama of the polygon from the at least one sequence of images as a function of the position of the surface and the position and orientation data associated with each of the images.

The invention is based on identifying a mobile mapping vehicle traveling on the earth's surface, recording a sequence of images of geographic locations collected of the surface with a land-based camera. Further, the mobile mapping vehicle records laser scan samples that enable software to generate a 3D representation of the environment of the mobile mapping vehicle from distance information from the laser scanner samples. The position and orientation of the vehicle is determined with the aid of a GPS receiver and inertial measurement devices, such as one or more gyroscopes and/or accelerometers. Furthermore, the position and orientation of the camera relative to the vehicle and thus relative to the 3D representation of the environment is known. To be able to produce a visually attractive multi-view panorama, the distance between the camera and the surface of the panorama must be known. The panorama can represent a view of the roadside that changes from the building surface all the way to the roadside panorama of the street. This can be done by existing image processing techniques. However, this requires a significant amount of computer processing power. According to the invention, the surface is determined by processing the laser scanner data. This requires much less processing power to determine the position of the surface than using only image processing techniques. Subsequently, a multi-view panorama can be generated by projecting the recorded images or segments of images onto the determined surface.

The geographic locations of the cameras and laser scanners are accurately known with on-board positioning systems (e.g., GPS receivers) and other additional position and orientation determination equipment (e.g., inertial navigation systems-INS).

Another improvement of the present invention is the ability to provide imagery displaying certain realistic 3D images without the processing time necessary to compute the 3D model and without the processing time necessary to render a full 3D model. The 3D model contains a plurality of polygons or surfaces. Rendering a full 3D model requires evaluating, for each of the polygons, whether the polygon can be seen when viewing the 3D model from a particular side. If a polygon is visible, the polygon will be projected on the image. The multi-view panorama according to the present invention is only one surface of the entire street-facing surface.

Other embodiments of the invention are defined in the following claims.

In one embodiment of the invention, generating comprises:

detecting one or more obstacles obstructing a view of a portion of the surface in all images of the at least one sequence of images;

projecting a view of one of the one or more obstacles to a multi-view panorama. The laser scanner samples enable us to detect for each image which obstacles are in front of the camera and in front of the position of the plane of the multi-view panorama to be generated. These features enable us to detect which parts of the plane are not visible in any of the images and should fill up an obstacle. This allows us to minimize the number of obstacles visible in the panorama in front of the front and thereby exclude as many obstacles as possible from the multi-viewpoint panorama so as not to obstruct the view of a portion of the surface in all of the images. This enables us to provide street-side multi-view panoramas with good visual quality.

In another embodiment of the present invention, the generating further comprises:

determining, for each of the detected obstacles, whether it is fully visible in any of the images;

projecting a view of the detected object from one of the at least one image to a multi-view panorama if the detected obstacle is fully visible in the at least one image. These features allow us to reduce the number of obstacles that will be partially visualized in the panorama. This improves the attractiveness of the multi-view panorama.

In one embodiment of the invention, the multi-view panorama is preferably generated from a partial image of the image having an associated view angle that is most perpendicular to the polygon. This feature enables us to produce the best quality multi-view panoramas from the images.

In one embodiment of the invention, the roadside panorama is generated by combining the multi-view panoramas. A common surface is determined for a roadside panorama that is parallel to, but at a distance from, a line (e.g., the centerline of a road). Projecting a multi-view panorama having a different location than the common surface on the common surface so as to represent each of the multi-view panoramas as if the each of the multi-view panoramas were seen at a distance equal to a distance between the surface and the line. Accordingly, a panorama is generated that visualizes the object in the multi-view panorama having a different location than the common surface, now as seen from the same distance. As many obstacles as possible have been removed from the multi-view panorama for best visual quality, resulting in a roadside panorama in which many obstacles along the road will not be visualized.

Roadside panoramas in accordance with the present invention provide the ability to display imagery of some realistic 3D view of a street without the processing time necessary to render a full 3D model of buildings along the street. Providing a 3D view of the street using a 3D model of the street requires determining for each building or portion of each building along the street whether it can be seen and then rendering each 3D model of the building or portion thereof as a 3D view. Images showing some realistic 3D view of the street can be easily provided by roadside panoramas according to the present invention. The roadside panorama represents a building along the street when projected onto a common surface. The surface can be easily transformed into a pseudo perspective image by projecting the pixel columns of the roadside panorama onto the 3D view in sequence (starting with the pixel column having the position farthest from the viewing position up to the pixel column having the position closest to the viewing point). In this way, real perspective images may be generated for the surfaces of the left and right roadside panoramas, resulting in a pseudo-real view of the street. When using a 3D model of a building along a street, only two images representing two surfaces are required instead of a plurality of polygons.

The present invention may be implemented using software, hardware or a combination of software and hardware. When all or part of the present invention is implemented in software, the software may reside on a processor-readable storage medium. Examples of suitable processor-readable storage media include floppy disks, hard disks, CD ROMs, DVDs, memory ICs, and so forth. When the system includes hardware, the hardware may include: an output device (e.g., a monitor, speaker, or printer); input devices (e.g., a keyboard, a pointing device, and/or a microphone); and a processor in communication with the output device; and a processor-readable storage medium in communication with the processor. The processor-readable storage medium stores code that is capable of programming the processor to perform the acts to implement the invention. The processes of the present invention may also be implemented on a server accessible via a telephone line or other network or internet connection.

Drawings

The invention will be discussed in more detail below using a number of exemplary embodiments with reference to the attached drawings, which are intended to illustrate the invention and not to limit its scope, which is defined by the appended claims and equivalents thereof

FIG. 1 shows an MMS system with a camera and laser scanner;

FIG. 2 shows a graphical representation of position and orientation parameters;

FIG. 3 shows a block diagram of a computer arrangement with which the present invention may be implemented;

FIG. 4 is a flow diagram of an example embodiment of a process for generating road information according to the present invention;

FIG. 5 shows a histogram based on a laser scan sample;

FIG. 6 shows exemplary results of polygon detection;

FIG. 7 shows a perspective view of a projection of a source image onto a virtual plane;

FIG. 8 shows a top view of a projection of a source image onto a virtual plane;

FIG. 9 shows a side view of a projection of a source image onto a virtual plane;

FIG. 10 shows top views of two cameras recording the same plane at different positions;

FIG. 11 shows a perspective view from the situation shown in FIG. 10;

FIG. 12 illustrates the process of constructing a panorama from two images;

FIG. 13 shows top views of two cameras recording the same plane at different positions;

FIG. 14 shows a perspective view from the situation shown in FIG. 13;

figures 15a-d show the application of a panorama,

figures 16a-e illustrate a second embodiment of finding the area in the source image that produces the multi-view panorama,

FIG. 17 shows a flow chart of an algorithm to assign portions of source images to be selected in the source images; and

fig. 18 shows another example of a roadside panorama.

Detailed Description

Fig. 1 shows an MMS system in the form of a car 1. The vehicle 1 has one or more cameras 9(I) (I ═ 1, 2, 3.. I) and one or more laser scanners 3(J) (J ═ 1, 2, 3.. J). The viewing angle or the one or more cameras 9(i) may be in any direction relative to the driving direction of the car 1 and may thus be a front-view camera, a side-view camera, a rear-view camera, or the like. Preferably, the angle between the driving direction of the car 1 and the viewing angle of the camera is in the range of 45 to 135 degrees on either side. The car 1 may be driven by a driver along a road of interest. In an exemplary embodiment, two side view cameras are mounted on the car 1, with the distance between the two cameras being 2 meters and the viewing angle of the cameras being perpendicular to the direction of travel of the car 1 and parallel to the earth's surface. In another exemplary embodiment, two cameras have been mounted on the automobile 1, the cameras having a horizontal and forward viewing angle from one side of the automobile, which are about 45 ° and 135 °, respectively. In addition, a third side view camera having an upward viewing angle of 45 ° may be mounted on the automobile. This third camera is used to capture the upper part of the building on the curb.

The vehicle 1 has a plurality of wheels 2. Furthermore, the vehicle 1 has a highly accurate position determination device. As shown in fig. 1, the position determining apparatus includes the following components:

a GPS (global positioning system) unit connected to the antenna 8 and arranged to communicate with a plurality of satellites SLi (i 1, 2, 3.) and to calculate position signals from signals received from the satellites SLi. The GPS unit is connected to the microprocessor mup. Based on the signals received from the GPS unit, the microprocessor μ P can determine suitable display signals to be displayed on the monitor 4 in the car 1, informing the driver of the location of the car and in which direction the car may be travelling. Instead of a GPS unit, a differential GPS unit may be used. Differential Global Positioning System (DGPS) is an enhancement to the Global Positioning System (GPS) that uses a network of fixed ground-based reference stations to broadcast the difference between the position indicated by the satellite system and a known fixed position. These stations broadcast the difference between the measured satellite pseudoranges and the actual (internally computed) pseudoranges, and the receiver station may correct its pseudoranges by the same amount.

DMI (distance measuring instrument). This appliance is a log that measures the distance traveled by the automobile 1 by sensing the number of rotations of one or more of the wheels 2. The DMI is also connected to a microprocessor μ P to allow the microprocessor μ P to take into account the distance measured by the DMI while calculating a display signal from the output signal from the GPS unit.

IMU (inertial measurement unit). Such an IMU may be implemented as 3 gyroscopic units arranged to measure rotational and translational accelerations in 3 orthogonal directions. The IMU is also connected to a microprocessor μ P to allow the microprocessor μ P to take into account the measurements of the DMI while calculating a display signal from the output signal from the GPS unit. The IMU may also include a dead reckoning sensor.

It should be noted that many combinations of global satellite navigation systems and onboard inertial and dead reckoning systems may be found by those skilled in the art to provide accurate position and orientation of the vehicle and thus of the equipment (which is installed with reference to the known position and orientation of the vehicle).

The system as shown in fig. 1 is a so-called "mobile mapping system" which collects geographical data, for example by taking pictures with one or more cameras 9(i) mounted on the car 1. The camera is connected to a microprocessor mup. The camera 9(i) in front of the car may be a stereo camera. The camera may be arranged to generate a sequence of images in which images have been captured at a predefined frame rate. In an exemplary embodiment, one or more of the cameras are cameras arranged to capture a picture at each predefined displacement or each time interval of the car 1. The predefined displacement is selected such that at a position at a predefined distance perpendicular to the driving direction is captured as at least two consecutive pictures of the side looking camera. For example, a picture may be captured after every 4 meters of travel, creating an overlap at a distance of 5 meters in each image of a plane parallel to the direction of travel.

While the automobile 1 is traveling along a roadside building, the laser scanner 3(j) scans a laser sample. It is also connected to the microprocessor μ P and sends these laser samples to the microprocessor μ P.

It is generally desirable to provide position and orientation measurements as accurately as possible from the 3 measurement units GPS, IMU and DMI. These position and orientation data are measured when the camera 9(i) takes a picture and the laser scanner 3(j) scans the laser sample. The pictures and laser samples are stored for later use in a suitable memory of the μ P in conjunction with corresponding position and orientation data of the car 1 collected while taking these pictures. The picture includes information about road information, such as the center of the road, road surface edges, and road width. Since the position and orientation data associated with the laser sample and picture are obtained from the same position determining device, an accurate match can be made between the picture and laser sample.

Fig. 2 shows which location signals are available from the three measurement units GPS, DMI and IMU shown in fig. 1. Figure 2 shows that the microprocessor mup is arranged to calculate 6 different parameters, i.e. 3 distance parameters x, y, z relative to an origin in a predetermined coordinate system and respectively ω_x、ω_y、ω_zAnd 3 angular parameters representing rotation about the x-axis, y-axis, and z-axis, respectively. The z direction coincides with the direction of the gravity vector. Global UTM coordinate systemThe system may be used as a predetermined coordinate system.

It is generally desirable to provide position and orientation measurements as accurately as possible from the 3 measurement units GPS, IMU and DMI. These position and orientation data are measured as the camera 9(i) takes an image and the laser scanner 3(j) scans the laser sample. Both the image and the laser sample are stored for later use in suitable memory of the microprocessor in conjunction with corresponding position and orientation data of the car 1 at the time these pictures and laser samples were taken and the position and orientation of the camera and laser scanner relative to the car 1.

The picture and laser sample include information about an object on the curb, such as a facade of a building block. In one embodiment, the laser scanner 3(j) is arranged to generate an output with a minimum resolution of 50Hz and 1 degree in order to generate a sufficiently dense output for the method. Laser scanners (e.g., MODELMS 291-S05) produced by SICK are capable of producing such outputs.

The microprocessor and the memory 9 in the car 1 may be implemented as a computer arrangement. An example of such a computer arrangement is shown in fig. 3.

In fig. 3, a schematic diagram of a computer arrangement 300 is given, which comprises a processor 311 for performing arithmetic operations. In the embodiment shown in fig. 1, the processor will be a microprocessor μ P.

The processor 311 is connected to a number of memory components, including a magnetic hard disk 312, Read Only Memory (ROM)313, Electrically Erasable Programmable Read Only Memory (EEPROM)314, and Random Access Memory (RAM) 315. Not all of these memory types need necessarily be provided. Moreover, these memory components need not be physically located close to the processor 311, but may be located remote from the processor 311.

The processor 311 is also connected to means for inputting instructions, data, etc. by a user, such as a keyboard 316 and a mouse 317. Other input means known to those skilled in the art may also be provided, such as a touch screen, a trackball and/or a sound transducer.

A read unit 319 is provided which is connected to the processor 311. The reading unit 319 is arranged to read data from, and possibly write data on, a removable data carrier or a removable storage medium, such as a floppy disk 320 or a CDROM 321. Other removable data carriers may be magnetic tapes, DVDs, CD-R, DVD-R, memory sticks, etc., as known to those skilled in the art.

The processor 311 may be connected to a printer 323 to print out data on paper and to a display 318, such as a monitor or LCD (liquid crystal display) screen or any other type of display known to those skilled in the art.

The processor 311 may be connected to a microphone 329.

Further, the processor 311 may be connected to a communication network 327, such as a Public Switched Telephone Network (PSTN), a Local Area Network (LAN), a Wide Area Network (WAN), the Internet, etc., by way of the I/O component 325. The processor 311 may be arranged to communicate with other communication arrangements over the network 327. The I/O means 325 are further adapted to connect the position determining device (DMI, GPS, IMU), the camera 9(I) and the laser scanner 3(j) to the computer arrangement 300.

The data carriers 320, 321 may comprise a computer program product in the form of data and instructions arranged to provide the processor with the ability to perform a method according to the invention. However, such computer program product may alternatively be downloaded via the telecommunications network 327.

The processor 311 may be implemented as a stand-alone system, or as a plurality of parallel-operating processors each arranged to implement subtasks of a larger computer program, or as one or more main processors with several sub-processors. Portions of the functionality of the present invention may even be implemented by remote processors in communication with the processor 311 through a telecommunications network 327.

The components contained in the computer system of FIG. 3 are those typically found in general purpose computer systems and are intended to represent a broad variety of such computer components as are well known in the art.

Thus, the computer system of FIG. 3 may be a personal computer, a workstation, a minicomputer, a mainframe computer, and the like. The computers may also include different bus configurations, networked platforms, multi-processor platforms, and the like. Various operating systems may be used, including UNIX, Solaris, Linux, Windows, Macintosh OS, and other suitable operating systems.

For post-processing of the images and scans taken by the camera 9(i) and laser scanner 3(j) and the position/orientation data, an arrangement similar to that in fig. 3 will be used, but this arrangement will not be located in the car 1 but may conveniently be located in a building for off-line post-processing. The images and scans taken by the camera 9(i) and scanner 3(j) and the associated position/orientation data are stored in one or more memories 312-315. This storage may be accomplished by first storing the images and scans and the associated position/orientation data on a DVD, memory stick or the like or transmitting it (possibly wirelessly) from memory 9. The associated position and orientation data defining the trajectory of the car 1 may be stored as raw data, including a time stamp. Further, each image and laser scanner sample has a timestamp. The time stamps enable us to accurately determine the position and orientation of the camera 9(i) and laser scanner 3(j) at the time the image and laser scanner samples were captured, respectively. In this way, the time stamp defines the spatial relationship between the view shown in the image and the laser scanner sample. The associated position and orientation data may also be stored as data linked by the database architecture used with the respective image and laser scanner samples.

In the present invention, a multi-viewpoint panorama is generated by using both an image taken by the camera 9(i) and a scan scanned by the laser scanner 3 (j). The method uses a unique combination of techniques from both the image processing and laser scanning fields. The present invention can be used to generate a multi-view panorama that changes from the immediate street view of a building to the full roadside view of the street.

FIG. 4 shows a flow diagram of an example implementation of a process for generating roadside information in accordance with the present invention. Fig. 4 shows the following actions:

A. action 42: laser spot map creation

B. Act 44: extracting plane coordinates of object from laser point map

C. Action 46: source image portion selection (using shadow map)

D. Act 48: constructing a panorama from selected portions of the source image these actions will be explained in detail below.

A. Action 42: laser spot map creation

A good method for finding a plane point is to use histogram analysis. The histogram contains the number of laser scan samples that the laser scanner 3(j) scans at a certain distance, as seen in the direction perpendicular to the trajectory traveled by the MMS system and summed up along a certain distance traveled by the car 1. The laser scanner may scan in a surface perpendicular to the earth's surface in angular directions spanning 180 °, for example. For example, the laser scanner may scan 180 samples, each sample being offset by 1 ° from its neighboring sample. In addition, laser scanning sample pieces were made at least every 20 cm. The driving speed of the vehicle should not be faster than 54km/h by the laser scanner rotating 75 times a second. Most of the time, the MMS system will follow a route along a line that is guided along a certain road (only when changing lanes or turning for some reason, the path traveled will show a deviation from this route.

The laser scanner 3(j) is in one embodiment a 2D laser scanner. The 2D laser scanner 3(j) provides a set of three data (so-called laser samples) including the measurement time, the measurement angle and the distance from the laser scanner 3(j) to the nearest solid visible at this angle. By combining the car 1 position and orientation (which is captured by the position determining device in the car), the relative position and orientation of the laser scanner relative to the car 1, and the laser samples, a laser spot map is created as shown in fig. 5. The laser spot map shown in fig. 5 is obtained by a laser scanner scanning in a direction perpendicular to the traveling direction of the automobile. If more than one laser scanner is used to generate the laser spot map, the laser scanners may have angles of, for example, 45 °, 90 °, and/or 135 °. If only one laser scanner is used, the laser scanner scanning perpendicular to the direction of travel provides the best resolution in the laser spot map space for finding the vertical plane parallel to the direction of travel.

In fig. 5, two histograms are shown:

1. distance histogram 61-this histogram 61 shows the number of laser scan samples as a function of distance to the car 1 added over a certain travel distance (e.g., 2 meters), including samples close to the car 1. When making laser scan slices every 20cm, 10 slices of laser scan samples will be accounted for. The figure shows a peak value close to the car 1 indicating the laser "echo" close to the car 1. This peak is associated with many echoes present close to the car 1 due to the angular sweep performed by the laser scan. Furthermore, at larger distances there is a second peak, which is related to the vertical surface of the object identified at said larger distance from the car 1.

2. The distance histogram 63 only shows the second peak at a certain distance from the car 1, which indicates only one object. This histogram is achieved by eliminating the higher density of laser scan samples in the direct vicinity of the car 1 due to the angular distribution of the laser scan. The effect of this elimination is that someone will better see objects at a certain distance away from the car 1, i.e. the front of the building 65. The elimination further has the effect of reducing the effect of obstacles in the histogram. This reduces the chance that an obstacle will be erroneously identified as a vertical plane.

The peaks on the histogram 63 indicate the presence of a flat solid surface parallel to the direction of vehicle travel. The approximate distance between the automobile 1 and the front face 65 may be determined by any available method. For example, the methods set forth in co-pending patent application PCT/NL2006/050264, which is incorporated herein by reference, may be used for such purposes. Alternatively, the GPS (or other) data indicative of the trajectory on which the car 1 is traveling may be compared to data showing the location of the footprint of the building and thereby render approximate distance data between the car 1 and the front face 65. The local maximum peak in a region is identified as the base point of the front face 65 by analyzing the histogram data for this approximate distance in that region. All laser scan samples within a vertical distance of 0.5m before this local maximum peak, for example, are considered architectural details of the front face 65 and labeled as "planar points". Laser scanned samples with a larger vertical distance than the maximum peak are discarded or can be marked as "flat spots". All other samples are laser scanned samples having a position between the position of the local maximum peak and the position of the car 1, which is considered a "ghost point" and so marked. It should be noted that a distance of 0.5m is given as an example only. Other distances may be used if desired.

The histogram analysis is performed every 2 meters along the trajectory of the car 1. In this way, the laser spot map is divided into 2 meter slices. In each slice, the histogram determines whether to mark the laser scan sample as a "flat spot" or a "ghost spot".

B. Act 44: extracting plane coordinates of object from laser point map

The laser samples labeled "plane points" are used to extract plane coordinates from the laser point map. The present invention treats a surface representing a street surface (typically a facade of a building) in a 3D space. The invention is illustrated by an example in which the surface is a polygon, which is a vertical rectangle representing the facade of a building. It should be noted that the method can be applied to any 'vertical' surface. Thus, the term "polygon" in the following description should not be limited to a closed plan view defined by straight sides, but may in principle be any "vertical" surface. By 'vertical' surface is meant any commonly configured surface that is visible to the camera.

Polygons are extracted from the laser scanner data labeled "flat spots". Many existing techniques are available for finding planes or surfaces, including methods based on the RANSAC (random sample consensus) algorithm.

The simple RANSAC algorithm is used directly on 3D points labeled as "plane points". For vertical-only planes, a simplified embodiment of the present invention first projects all non-ground points on some horizontal plane by discarding the height values of the 3D points. The 2D points of the horizontal plane are then detected using RANSAC or hough transform (Hugh transform). These lines are used to derive the lower and upper position of the plane along the lines.

The algorithm described above requires additional processing to find the polygon bounding the plane. Prior art methods for finding polygons bounding a plane are known. In one example, all laser points from the plane below a given threshold are projected onto the plane. This plane is analogous to a 2D image to which clustering techniques and image segmentation algorithms can be applied to obtain polygons representing, for example, boundaries of building facades. Fig. 6 shows exemplary results of polygon detection. The laser scanner map shown in fig. 6 is obtained by combining laser scanner samples from two laser scanners. One with an angle of 45 deg. to the direction of travel of the car 1 and the other with an angle of 135 deg. to the direction of travel of the car 1. Thus, two polygons of the plane of the side facades 602, 604 can be extracted immediately after the polygon of the plane of the front facade 600 of the building. For each detected plane, the polygon is described by plane coordinates, which are the 3D positions of the corners of the plane in a predetermined coordinate system.

It should be noted that the polygons of the plane may also be retrieved using geo-referenced 3D locations (available from commercial databases) about the building and determine whether the laser scanner sample from the laser scanner map is a "plane point" or a "ghost point".

It should be noted that when a multi-view panorama is generated for street-facing surfaces of only one building, the orientation of the base points of the street-facing surfaces may not necessarily be parallel to the direction of travel.

The street-side multi-view panorama can be used to generate a roadside multi-view panorama. Roadside panoramas are a composite of multiple multi-viewpoint panoramas of a building. The characteristics of the roadside panorama according to the present invention are:

the panoramic view represents a virtual common constructed vertical surface;

each pixel column of the panorama represents a vertical surface at a predetermined vertical distance from the trajectory of the car, the centerline of the street, or any other representation of a line along the street, and

each pixel of the panorama represents a region of the surface, where the region has a fixed height.

If a roadside panorama of a street is generated, the surface of the panorama is typically considered parallel to the direction of travel, the centerline of a road extending along the road, or any other feature. Thus, the surface of a curbside panorama of a curved street will follow the curvature of the street. Each point of the panorama is considered as seen in an orientation perpendicular to the surface. Thus, for roadside panoramas of streets, distances up to the most common surface are searched in the laser scanner map or predefined values have been given to the distances. This distance defines the pixel resolution of the panorama in the horizontal and vertical directions. The vertical resolution depends on the distance, while the horizontal resolution depends on the distance in combination with the curvature of the line along the street. However, the vertical distance between the driving direction of the car and the base point of the vertical surface found by the histogram analysis may contain a discontinuity. This may occur when two adjacent buildings do not have the same building line (i.e., are not aligned on the same plane). To obtain the roadside panorama defined above, the multi-view panorama of each building surface will be transformed into a multi-view panorama as if the building surface were seen from distances up to the most common surface. In this way, each pixel will represent a region of equal height.

In known panoramas, two objects having the same size but at different distances will be displayed in the panoramas at different sizes. According to an embodiment of the present invention, a roadside panorama will be generated, wherein two similar objects having different vertical distances with respect to the driving direction will have the same size at the multi-viewpoint panorama. Thus, when the roadside panorama is generated, each frontal panorama will be scaled such that each pixel of the roadside panorama will have the same resolution. Thus, in a roadside panorama generated by the above-described method, a building having an actual height of 10 meters at a distance of 5 meters will have the same height in the roadside panorama as a building having an actual height of 10 meters at a distance of 10 meters.

Roadside panoramas with the above characteristics show the front of buildings along the street as if the buildings had the same building lines, but in fact they would not have the same building lines. The important visual objects of the panorama are in the same plane. This enables us to transform a forward looking panorama into a perspective view without annoying visual distortion. This has the advantage that the panorama can be used in an application running on a system as shown in fig. 3 or on any kind of mobile device with minimal image processing capabilities, such as a navigation device. With a panorama where the front of the building parallel to the direction of the street is scaled to have the same building lines, a near reality view of the panorama can be presented from any viewing angle. A near reality view is an easily understood view that may represent reality but does not correspond to reality.

C. Action 46: source image portion selection (using shadow map)

The multi-view panorama obtained by the present invention is composed of a set of images from the image sequence obtained by the camera 9 (i). Each image has associated position and orientation data. The method described in the unpublished patent application PCT/NL2006/050252 is for determining which source images have a viewing window that includes at least a portion of the surface determined in act 44. First, a source image having a viewing window comprising at least a portion of the surface for which a panorama has to be generated is selected from at least one sequence of source images generated by a camera. This may be done because each source image has an associated position and orientation of the camera that captured the source image.

In the present invention, the surface corresponds to a substantially vertical plane. Knowing the position and orientation of the camera, as well as the viewing angle and viewing window, the projection of the viewing window onto the surface can be determined. Those skilled in the art, who are aware of goniometry mathematics, can rewrite the orthorectification method described in the unpublished application PCT/NL2006/050252 to a method for projecting a viewing window having an arbitrary viewing angle onto an arbitrary surface. The projection of a polygon or surface area onto the viewing window of a camera having both an arbitrary position and orientation is performed by three operations: rotation at the camera focus, scaling and translation.

Fig. 7 shows a perspective view of a projection of a source image 700, which is equal to a camera viewing window on a virtual surface 702. The virtual surface 702 corresponds to a polygon and has coordinates (xt1, yt1, zt1), (xt2, yt2, zt2), (xt3, yt3, zt3), and (xt4, yt4, zt 4). Reference 706 indicates the focus of the camera. The focal point 706 of the camera has coordinates (xf, yf, zf). The edge of the source image 700 defines the viewing window of the camera. A straight line through the focal point 706 of the camera defines a projection from a pixel of the virtual surface 702 onto a pixel of the source image 700 through the intersection of both the viewing window and the virtual surface 702. Furthermore, the intersection of the line passing through the focal point 706 of the camera and the virtual surface 702 and the laser scanner sample labeled "ghost point" define points of the virtual plane that are not visible in the viewing window. In this way, a shadow 708 of the obstacle 704 may be projected on the virtual surface 702. The shadow of an obstacle is a set of contiguous pixels in front of a virtual surface (e.g., a front surface). Since the position of the virtual surface corresponds to the position of the street level, the shadow can be accurately projected on the virtual surface. It should be noted that balconies extending 0.5 meters all the way from the street level are considered part of the common structural surface. Thus, the details of the perspective of the balcony in the source image will be projected on the multi-viewpoint panorama. The details of the perspective view are that the sides of the balcony, which will not be visualized in the pure forward view image of the building, are perpendicular to the street level.

The above projection method is used to select a source image of at least a portion of a viewing surface. After selecting a source image for viewing at least a portion of the surface, a laser scanner sample having a position between a position of a focal point of a camera and a position of the surface is selected in a laser scanner map. These are laser scanner samples labeled as "ghost point" samples. The selected laser scan sample represents an obstruction that prevents the camera from recording the object represented by the virtual surface 702. Selected laser scanner samples are clustered by known algorithms to form one or more solid obstacles. Then, a shadow of the obstacle is generated on the virtual surface 702. This is done by extending a straight line through the focal point 706 and the solid obstacle all the way to the location of the virtual surface 702. The position where a line along the boundary of an obstacle hits the virtual surface 702 corresponds to the boundary point of the shadow of the obstacle.

As can be seen in fig. 7, an object 704 (i.e., a tree) is seen in the image in front of the surface 702. If the position of the object 704 is known relative to the virtual surface 702 and the focal point 706 of the camera, the shadow 708 of the object 704 on the virtual surface 702 can be easily determined.

According to the invention, a geo-located multi-view panorama of a surface retrieved from a laser scanner map or 3D information about the facade of a building is created using the surface or information from a commercial database. The method according to the invention combines the 3D information of the camera 9(i) position and orientation, the focal length and resolution of the image (═ pixel size), the 3D information of the detected plane and the 3D position of the ghost point sample of the laser scanner map. The combination of the camera's position and orientation information with the laser scanner map enables the method to determine, for each individual image:

1) whether a source image captured by a camera includes at least a portion of a surface; and

2) which object prevents the camera from visualizing the image information to be at the portion of the surface.

The result of the combination enables the method to determine on which parts of the image the front face represented by the virtual plane is visible. Thus determining which images are available for generating a multi-view panorama. Images with viewing windows that could otherwise capture at least a portion of a virtual surface but are unable to capture any portion of the virtual surface due to a large obstruction in front of the camera would be discarded. The "ghost points" between the position of the surface and the camera position are projected on the source image. This enables the method to find surfaces or areas (shadow bands) where obstacles are visible on the source image and thus on the final multi-view panorama.

It should be noted that the examples used to illustrate the invention use polygons as virtual surfaces. Simple examples have been used to reduce the complexity of the examples. However, those skilled in the art will immediately recognize that the present invention is not limited to flat surfaces but may be used with any smooth surface, for example, a vertical curved surface.

Fig. 8 and 9 show top and side views, respectively, of an obstacle 806 projected on the source image 800 and on the virtual surface 804. The position of the obstacle 806 is obtained from the laser scanner map. Thus, according to the present invention, instead of obtaining the position of an object through a complex image processing algorithm that uses image segmentation and triangulation algorithms on more than one image to detect and determine the position of planes and obstacles in the image, the 3D information from the laser scanner map is used in conjunction with the position and orientation data of the camera. Using a laser scanner map in conjunction with the position and orientation data of the camera provides a simple and accurate method of determining the position of obstacles in an image that prevent the camera from visualizing the area of the surface of an object behind the obstacle. Goniometry is used to determine the shadow 802 of the obstacle 806 on the source image 800 and the shadow 808 of the obstacle 806 on the virtual surface 804, which describes the street-level position and orientation of the object (i.e., the building front). The shadow 808 on the virtual surface will be referred to as a shadow band in the following description of the present invention.

The multi-view panorama is constructed by finding the areas of the source image that best visualize the surface that has been found in the laser scanner map and projecting the areas on the multi-view panorama. The visualized obstacles of the source images or the areas of the visualized obstacles with the smallest shadows (═ areas) on the multi-view panorama should be selected and combined to obtain the multi-view panorama.

Two possible implementations for finding the partial source images in the source image used to generate the multi-view panorama will be disclosed.

The first embodiment is for finding the area.

The above object has been achieved in the first embodiment by generating a shadow map for each source image of a portion of a visualization surface. The shadow map is a binary image, wherein the size of the image corresponds to the area of the source image that when projected on a plane visualizes the plane and wherein for each pixel it is indicated whether it visualizes the surface or obstacle in the source image. Subsequently, all shadow maps are superimposed on the main shadow map corresponding to the surface. In this way, one master shadow map is made for the surface and thus for the multi-view panorama to be generated.

In one embodiment, a master shadow map is generated, wherein a shadow band in such master shadow map indicates that at least one of the selected source images visualizes an obstacle when a region of the at least one selected source image corresponding to the shadow band is projected on the multi-view panorama. In other words, this master shadow map identifies which areas of the front face are not obstructed by any obstacles in the image. It should be noted that the size and resolution of the master shadow map is similar to the size and resolution of the multi-view panorama to be generated.

The main shadow map is used to divide the multi-view panorama into several segments. The segments are obtained by cutting the main shadow map into the segments by finding the best "saw path", wherein the path on the main shadow map does not divide the shadow band into two parts. The segmentation defines how the panorama must be constructed. It should be noted that the sawing path always spans the area of the main shadow map that has been obtained by the superposition of the shadow maps of the at least two images. Having the path between the shadow bands ensures that the seam between the segments of the panoramic image is in the visible part of the front face and not likely in the area of the obstacle to be projected on the front face. This enables the method to select the best image for projecting the region corresponding to a segment on the panorama. The best image may be an image having no shadow band or an image having a minimum shadow band area in the area corresponding to the segment. An additional criterion for determining the best position of the "sawing path" may be the viewing angle of the orientation of the at least two images with respect to the plane of the panorama to be generated. Since the at least two images have different positions, the viewing angle with respect to the front will be different. It has been found that the most vertical images will provide the best visual quality in a panoramic view.

Each segment may be defined as a polygon, with the edges of the polygon defined by 3D positions in a predefined coordinate system. Since the "sawing path" spans the pixels in all of the at least two source images that visualize the surface corresponding to the plane, the method allows to create a smooth zone between the two segments. The smoothing reduces visual interference in the multi-view panorama. This aspect of the invention will be elucidated later. The width of the smooth zone may be used as another criterion for finding the best "sawing path". The width of the smooth region may be used to define a minimum distance between the saw path and the shadow band. If the closest distance between the edge lines of the two shadow bands is less than the predefined distance, a segment with two shadow bands will be created. Furthermore, the pixels of the source image for said smooth regions should not represent obstacles. The pixels for the smooth region are the edges of the pixels around the shadow. Thus, the width of the smooth region defines the minimum distance between the edge line of the shadow band and the polygon defining the segment that includes the shadow band. It should be noted that if the obstacle causing the shadow band is partially visible in the image, the distance between the edge line of the shadow band and the polygon defining the segment may be zero.

A multi-view panorama is generated by combining portions of the source images associated with the segments. To obtain an optimal visualization of the multi-view panorama, for each segment we have to select the source image of the segment for which the multi-view panorama generation of the object has to be visualized in the most appropriate way.

The area of the source image that has to be used for generating the corresponding segment of the panorama is determined in the following manner:

1. selecting a source image having an area of the entire area of the visualization segment;

2. selecting from the source images in the previous action the source image that contains the least number of shadow-labeled pixels in the associated segment in the shadow map associated with the source image.

The first action ensures that only the pixels of the source image corresponding to the segment are taken from one source image. This reduces the amount of visible interference, such as partially visualizing an obstacle. For example, a car parked in front of an area of a building corresponds to a segment that can be seen in three images, one visualization front end, one visualization back end and one visualization whole car, then a segment from an image that visualizes whole car would be taken in that case. It should be noted that selecting other images may result in a panorama visualizing more details of the object to be represented by the panorama hidden behind the car in the selected image. It has been found that it is more attractive to fully visualize an image of an obstacle than to partially visualize the image of the obstacle. It should further be noted that there may be images that visualize the entire area without the presence of a car, however their viewing angle is less advantageous than the other three images. In that case, this image will be selected because it contains the least number (zero) of pixels marked as shadows in the associated segment in the shadow map associated with the image.

Furthermore, when there are two images visualizing the entire region without any object (zero pixels marked as shadows), the image with the closest vertical viewing angle will be selected for visualizing the region in the multi-view panorama.

A second action, subsequent to the first action, ensures the selective visualization of the source image of a large part of the object represented by the panorama. Thus, for each segment, a source image is selected that visualizes the smallest shaded band region in the region corresponding to the segment.

If there is no image showing the entire area corresponding to a segment, the segment must be sawn into subsegments. In that case, the image boundaries may be used as a saw path. The previous steps will be repeated for the subsections to select the image with the most favorable region for visualizing the region in the multi-view panorama. The parameters that determine the most favorable region are the number of pixels marked as shadows and the viewing angle.

In other words, the source images for a multi-view panorama are combined in the following way:

1. performing stitching in a portion of the multi-view panorama located between shadow bands defined by a main shadow map when shadow bands in the main shadow map are discontinuous;

2. when the shadow bands of obstacles visible in a selected source image projected on a multi-view panorama overlap or are discontinuous, the area of the multi-view panorama is divided into sections by the following rules:

a) and selecting a source image containing a full shadow band to be placed in the multi-view panorama. When there is more than one source image containing a full shadow band, the source image that visualizes the segment at the perspective closest to the vertical vector is selected. In other words, the forward-looking source image visualizing the segment is preferably a source image viewed at an above angle;

b) when there is no image covering the full shadow band, the segment is taken from the most vertical part of the source images in which it is visualized.

A second embodiment is used to find the region.

The second embodiment will be elucidated by means of fig. 16 a-f. Fig. 16a shows top views of two camera positions 1600, 1602 and a surface 1604. Located between the two camera positions 1600, 1602 and the surface 1604 are a first obstruction 1606 and a second obstruction 1608. The first obstruction 1606 is visible in the viewing windows of both camera positions and only the second obstruction 1608 is visible through the first camera position 1600. Three (shaded) regions may be derived by projecting the shadow of the obstacle on surface 1604. Region 1610 is obtained by projecting the shadow of the second obstacle onto the surface from first camera position 1600. Region 1612 and region 1614 have been obtained by projecting the shadow of the first obstacle onto the surface from the second and first camera positions, respectively. A shadow map will be generated for the source images captured from the first and second camera positions 1600, 1602, respectively. For each portion of the visualization surface 1604 of the source image, a shadow map will be generated. This shadow map (referenced in the same coordinate system as the multi-view panorama of the surface 1604 to be generated) indicates, for each pixel, whether the pixel visualizes the surface 1604 or does not visualize the surface due to an obstacle.

Fig. 16b shows a left shadow map 1620 corresponding to a source image captured from the first camera position 1600 and a right shadow map 1622 corresponding to a source image captured from the second camera position 1602. Which regions of the left-shaded map display surface 1604 that are visualized in the source image do not contain visual information of the surface 1604. Area 1624 is a shadow corresponding to second obstacle 1608 and area 1626 is a shadow corresponding to first obstacle 1606. First barrier 1606 is seen to be higher than second barrier 1608. The right shaded map 1622 displays only one area 1628, the area 1628 containing no visual information of the surface 1604. Area 1628 corresponds to the shadow of first barrier 1606.

The shadow maps are combined to produce a master shadow map. The master shadow map is a map associated with the surface for which the multi-view panorama must be generated. However, according to a second embodiment, for each pixel in the main shadow map it is determined whether it can be visualized by at least one source image. The purpose of the master shadow map is to find the area of the panorama that does not visualize the surface but will visualize obstacles in front of the surface.

Fig. 16c shows a main shadow map 1630 that has been obtained by combining the shadow maps 1620 and 1622. This combination can be done accurately because the position and orientation of each camera is accurately recorded. The region 1640 is a region of the surface 1604 that cannot be visualized by either a source image captured from the first camera position 1600 or the second camera position 1602. The pixels of this area 1640 are critical because they will always display an obstacle and never display the surface 1604. The pixels in region 1640 obtain corresponding values, e.g., "critical". Region 1640 will display a portion of first obstacle 1606 or a portion of second obstacle 1608 in the multi-viewpoint panorama of surface 1604. Each of the other pixels will obtain a value indicating that a value of an associated pixel of the multi-view panorama is available from at least one source image to visualize the surface. In fig. 16c, regions 1634, 1636 and 1638 indicate regions corresponding to regions 1624, 1626 and 1628 in the shadow map of the respective source image. The regions 1634, 1636 and 1638 obtain values indicating that values for associated pixels of the multi-view panorama are available from at least one source image to visualize the surface.

The master shadow map 1630 is then used to generate a usage map for each source image. The usage map has a size equal to the shadow map of the source image. The usage map indicates, for each pixel:

1) whether the values of the corresponding pixels in the source image should be used to generate the multi-view panorama,

2) whether the values of the corresponding pixels in the source image should not be used for generating the multi-view panorama, an

3) Whether the values of the corresponding pixels in the source image are available for generating the multi-view panorama.

Such a map may be generated by verifying, for each shadow band in the shadow map of the source images, whether a corresponding region in the master shadow map includes at least one pixel indicating that the pixel is unable to visualize surface 1604 in the multi-view panorama through any of the source images. If so, the region corresponding to the entire shadow band will be marked as "should be used". If not, the area corresponding to the entire shadow will be marked as "should not be used". The remaining pixels will be marked as "usable". FIG. 16d shows a left usage map 1650 that has been obtained by combining the information in the shadow map 1620 with the main shadow map 1630. Region 1652 corresponds to the shadow of second obstacle 1608. This region 1652 has obtained the value "should be used" because region 1624 in shadow map 1620 has one or more corresponding pixels labeled "critical" in the main shadow map. This means that if one pixel of region 1652 must be used to generate the multi-view panorama, all other pixels of the region must be used. Region 1654 corresponds to the shading of first obstacle 1606. The region 1654 has obtained a value of "should not be used" because the region 1626 in the corresponding shadow map 1620 does not have any pixels marked as "critical" in the corresponding region 1636 in the main shadow map, which means that the first obstacle 1606 can be removed from the multi-view panorama by selecting the corresponding region in the source image captured by the second camera 1602. Thus, the region in the source image corresponding to region 1654 should not be used to generate the multi-view panorama of surface 1604. The right use map 1656 of FIG. 16d has been obtained by combining the information in the shadow map 1622 with the main shadow map 1630. Region 1658 corresponds to the shading of second obstacle 1606. This region 1658 has obtained the value "should be used" because region 1628 in shadow map 1622 has one or more corresponding pixels marked as "critical" in the main shadow map. This means that if one pixel of region 1658 must be used to generate the multi-view panorama, all other pixels of the region must be used.

Maps 1650 and 1656 are used to select which portions of the source image must be used to generate the multi-view panorama. One embodiment of an algorithm to assign portions of source images to be selected in a source image will be given. It should be clear to those skilled in the art that other possible algorithms may be used. A flow chart of the algorithm is shown in fig. 17. The algorithm begins with retrieving an empty selection map that indicates, for each pixel of the multi-view panorama, which source image should be used to generate the multi-view panorama of the surface 1604 and usage maps 1650, 1656 associated with each source image.

Pixels of the selection map to which the source image has not yet been assigned are then selected 1704. In act 1706, a source image having a corresponding pixel in its associated usage map labeled "should use" or "available" is searched. Preferably, if the corresponding pixel in all usage maps is marked as "usable", the source image having the most vertical viewing angle relative to that pixel is selected. Furthermore, to optimize the visibility of the surface 1604 in the panoramic view, in the event that the corresponding pixel in one of the usage maps is marked as "must use," a source image having the smallest area marked as "must use" in the usage map that covers the area marked as "critical" in the main shadow map is preferably selected with the main shadow map.

After the source image is selected, in act 1708, a usage map of the selected image is used to determine which area of the source around the selected pixel should be used to generate a panorama. This can be done by a growing algorithm. For example, by selecting all neighboring pixels in the usage map that are marked as "should be used" and available (and where the source image has not been assigned to the corresponding pixel in the selection map).

The next act 1710 determines whether all pixels have been assigned a source image. If not, act 1704 is performed again by selecting the pixels to which the source image has not been assigned and subsequent acts will be repeated until a source image is to be assigned to each pixel.

Fig. 16e shows two images that identify which portions of the source image are selected for producing the multi-view panorama of surface 1604. The combination of the partial source images is shown in fig. 16f, which corresponds to a selected map 1670 of the multi-view panorama of surface 1604. The left image 1660 of fig. 16e corresponds to the source image captured by the first camera 1600 and the right image 1662 corresponds to the source image captured by the second camera 1602. Pixels in the left segment 1672 of the selection map 1670 are assigned to a corresponding area in the source image captured from the first camera position 1600, which corresponds to area 1664 in the left image 1660 of FIG. 16 e. Pixels in the right segment 1674 of the selection map 1670 are assigned to corresponding regions in the source image captured from the second camera location 1602. This area corresponds to area 1666 in right image 1662 of FIG. 16 e.

When the above algorithm is applied, a pixel is selected in the left part of the selection map, for example the upper left pixel. The pixels are present in only one source image. In act 1708, the neighboring region may grow until it is defined by the edge of the selection map and the pixel marked "will not be used". In this manner, area 1664 is selected and the pixels of segment 1672 are assigned the first source image in selection map 1670. Subsequently, a new pixel to which the source image has not yet been assigned is selected. This pixel is located in area 1666. Subsequently, a neighboring area of the pixel is selected. The edges of the area 1666 are defined by the source image edges and the assigned pixels in the selection map 1670 that have been assigned to other source images (i.e., to images captured by the first camera).

Selecting pixels from the source image that correspond to segments 1672 and 1674 will produce a multi-view panorama where the first obstacle 1606 is not visible and the second obstacle is safely visible.

In the right image of fig. 16e, area 1668 identifies which corresponding pixels are available to generate the area of the multi-view panorama of surface 1604. This area may be obtained by extending act 1708 according to the following criteria: the growing process stops when the width of the overlapping edge with the other source images exceeds a predefined threshold (e.g., 7 pixels) or at the pixels marked as "should be used" or "should not be used" in the usage map. Area 1668 is such an overlapping edge. This is illustrated in FIG. 16e by area 1676. This area can be used as a smooth area. This enables the method to mask irregularities between two adjacent source images, such as color differences between the images. In this way, the color may be smoothly changed from the background color of the first image to the background color of the second color. This reduces the number of abrupt color changes in regions that should normally have the same color.

The two embodiments described above for selecting a source image portion generate a map for a multi-view panorama, where each pixel is assigned to a source image. This means that all information visible in the multi-view panorama will be obtained by projecting the corresponding source image portion on the multi-view panorama. Both embodiments attempt to eliminate as many obstacles as possible by selecting portions of the source image that show surfaces in the source image rather than obstacles. If only the projection of pixels of a part of the source image on the panorama is applied, some parts of the surface are not visualized in any source image and thus obstacles or parts of obstacles will be visualized. However, the two embodiments may be adapted to first derive features of regions of the surface that are not visible from either of the source images. These areas correspond to the shadows in the main shadow map of the second embodiment. Some of the characteristics that can be derived are height, width, shape, size. If the features of the region match predefined criteria, pixels in the multi-view panorama corresponding to the region may be derived from pixels in the multi-view panorama surrounding the region. For example, if the width of the region does not exceed a predetermined number of pixels (e.g., the shade of a street light pole) in the multi-view panorama, the pixel value may be obtained by assigning an average or interpolation of neighboring pixels. It should be clear that other threshold functions may be applied.

Furthermore, algorithms may be applied that determine whether the generated obstacles are significant enough to be reproduced with a certain degree of fidelity. For example, a tree that blocks the front is displayed in two images, only a small portion being seen in one image at the edge of the image and the entire tree being seen in the other image. The algorithm may be arranged to determine whether including the fraction in the panorama would not seem silly. If so, the small portion is displayed, resulting in a panoramic view showing the largest portion of the front face and small visual irregularities due to the tree. If not, the entire tree would be included, resulting in a panorama revealing a smaller portion of the front face, but without visual irregularities about the tree. In these ways, the number of visible obstacles and the corresponding size in the multi-view panorama can be further reduced. This enables the method to provide a panoramic view with the best visual effect. The function may be performed on the corresponding shadow map.

D. Act 48: forming a panorama from selected portions of a source image

After generating a segment map corresponding to the multi-view panorama and selecting for each segment a source image that should be used to project the area of the source image corresponding to the segment, the area of the source image associated with the segment is projected on the panorama. This process may be compared to the orthorectification method described in the unpublished patent application PCT/NL2006/050252, which may be described as performing three operations on a region of the source image, namely rotation, scaling and translation at the camera's full focus, all algorithms well known in image processing. All segments together form a combined picture, which is a multi-view panorama, because images with different positions are used (multi-view).

Visual irregularities at the intersection of one segment to another can be reduced or eliminated by defining smooth zones along the boundaries of the two segments.

In one embodiment, the values of the pixels of the smooth region are obtained by averaging the values of corresponding pixels in the first and second source images. In another embodiment, the pixel values are obtained by the following formula: value_pan＝α×value_image1+(1-α)×value_image2Wherein value_pan、value_image1And value_image2Is a pixel value in the multi-view panorama, the first image and the second image, respectively, and alpha is in the range of 0 to 1Wherein α is 1 when the smooth region touches the first image and α is 0 when the smooth region touches the second image, α being linearly changeable from one side of the smooth region to the other side. In this case, value_panIs the average of the values of the first and second images in the middle of the smooth region, which is usually where the stitching is. It should be noted that the parameter a may have any other suitable course when varying from 0 to 1.

Many other algorithms to obtain a smooth intersection from one segment to another are known in the art of image processing.

The above method will be illustrated by some simple examples.

Fig. 10 shows top views of two cameras 1000, 1002 at different positions A, B and recording the same plane 1004. The two cameras 1000, 1002 are mounted on a moving vehicle (not shown) and the vehicle moves from position a to position B. Arrow 1014 indicates the direction of travel. In the given example, the source image sequence includes only two source images of the visualization plane 1004. A source image is obtained from the first camera 1000 when the vehicle is at position a. Another source image is obtained from the second camera 1002 when the vehicle is at position B. Fig. 11 shows a perspective image from the situation shown in fig. 10. The left and right perspective images correspond to source images captured by the first camera 1000 and the second camera 1002, respectively. The two cameras have different viewing angles with respect to the direction of travel of the vehicle. FIG. 10 shows an obstruction 1006, such as a cylinder, located between positions A and B and plane 1004. Thus, the portion 1008 of the plane 1004 is not visible in a source image captured by the first camera 1000 and the portion 1010 of the plane 1004 is not visible in a source image captured by the second camera 1002.

The shadow map associated with a source image captured by the camera 1000 has shadows on the right half and the shadow map associated with a source image captured by the camera 1000 has shadows on the left half. Fig. 10 shows a top view of the main shadow map of plane 1004. The shadow map includes two intermittent shadows 1008 and 1010. According to the invention, the place 1012 where the main shadow map is stitched is between the two shadows 1008 and 1010. In FIG. 11, polygons 1102 and 1104 represent two segments in which plane 1004 is divided.

As described above, the method according to the invention analyzes for each segment the corresponding area in the shadow map of each source image. The source image that visualizes the segment with the smallest shaded area will be selected. In the given example, a source image that does not contain a shadow in the corresponding segment will be selected to represent the segment. Thus, the left part of the plane 1004 indicated by the polygon 1102 in fig. 11 will be obtained from the image captured by the first camera 1000 and the right part of the plane 1004 indicated by the polygon 1104 in fig. 11 will be obtained from the image captured by the first camera 1002.

Fig. 12 illustrates the process of constructing a panoramic view of the plane 1004 in fig. 10 from the two images shown in fig. 11 after selecting, for each segment, the corresponding source image to visualize the corresponding segment. In one embodiment, the segments defined by polygons 1102 and 1104 are projected onto a multi-view panorama of plane 1004.

The two segments do not match perfectly where 1202 they are spliced. The reason may be the difference in resolution, color and other visual parameters of the two source images at the place 1202 of the stitching. When the pixel values of the two segments at both sides of the place 1202 of stitching are derived directly from only one of the respective images, the user may notice the irregularity in the panorama. To reduce the visibility of the defects, a smooth zone 1204 may be defined around the place 1202 of the splice.

Fig. 13 and 14 show another simple example similar to the example given above for elucidating the invention. In this example, another obstacle obstructs the visualization plane 1304. Fig. 13 shows top views of two cameras 1300, 1302 at different positions C, D and recording the same plane 1304. The two cameras 1300, 1302 are mounted on a moving vehicle (not shown) and the vehicle moves from position C to position D. Arrow 1314 indicates the direction of travel. In the given example, the source image sequence comprises only two source images of the visualization plane 1304. A source image is obtained from the first camera 1300 when the vehicle is at position C. Another source image is obtained from the second camera 1302 while the vehicle is at position D. Fig. 14 shows a perspective image from the situation shown in fig. 13. The left and right perspective images shown in fig. 14 correspond to source images captured by the first camera 1300 and the second camera 1302, respectively. The two cameras have different viewing angles with respect to the direction of travel of the vehicle. FIG. 13 shows an obstacle 1306, for example a cylinder, located between positions C and D and plane 1004. Thus, the portion 1304 of the plane 1308 is not visible in a source image captured by the first camera 1300 and the portion 1304 of the plane 1310 is not visible in a source image captured by the second camera 1302.

Fig. 13 shows an overhead view of the main shadow map associated with plane 1304. The master shadow map displays shadows 1008 and 1010 with overlapping areas. Because there are only two images of the visualization plane 1304, the region of the plane associated with the shadow corresponding to the overlap cannot be seen in either of the images. The area of the panorama of plane 1304 corresponding to the overlap will visualize the corresponding portion of obstruction 1306. The master shadow map may now be divided into three portions, one of which includes the shadow. The edge lines of the polygon defining the segment containing the shadow are preferably spaced apart from the edge lines of the shadow by a minimum distance. This allows us to define a smooth zone. References 1312 and 1316 indicate the left and right edges of the segment along the line. When the two source images fully visualize the segment (a scene easily visible in fig. 14), the segment will be taken from the source image having the most perpendicular viewing angle with respect to the plane. In the given example, the segment will be taken from a source image captured by the second camera 1302. When the segment containing the obstruction and the rightmost portion of the plane are to be taken from the same source image, the edge line with the reference 1316 can be removed and does not have to define a smooth region there. Thus, two segments are eventually left to form the panorama of plane 1304. In fig. 14, polygons 1302 and 1304 represent two segments of a source image that are used to form plane 1304. Reference 1312 indicates the edge lines where smooth zones may be defined.

The above method is performed automatically. It may happen that the quality of the multi-view panorama is such that some correction is required to perform the image processing tools and object recognition tools of the present invention. For example, polygons found in a laser scanner map correspond to two adjacent buildings, whereas for each building facade a panorama must be generated. In that case, the method comprises some verification and manual adaptation actions to achieve the possibility of confirming or adapting the intermediate result. These actions may also be adapted to accept intermediate or final results of the road information generation. Furthermore, the superposition of polygons representing building surfaces and/or shadow maps in relation to one or more consecutive source images may be used for a requestor to perform verification.

The multi-view panoramas produced by the present invention are stored in a database along with associated position and orientation data in a suitable coordinate system. The panorama can be used to map out pseudo-reality, easy to understand views, and to generate views of cities around the world in applications such as Google Earth (Google Earth), Google Street View (Google Street View), and Microsoft's virtual Earth (Microsoft) or can be conveniently stored or arranged on a navigation device.

As described above, the multi-view panorama is used to generate a roadside panorama.

Fig. 15a-15d show the application of roadside panoramas produced by the present invention. The application enhances the visual output of current navigation systems and navigation applications on the internet. The device executing the application does not require dedicated image processing hardware to produce the output. FIG. 15a shows a pseudo-perspective view of a street, which can be easily produced without the use of complex 3D models of buildings at the curb. The pseudo-perspective view has been obtained by processing the map generated similarity (map generated similarity) of the left and right roadside panoramas of the street and the road surface (earth surface) between two multi-viewpoint panoramas. The map and two images may have been obtained by processing the sequence of images and position/direction of travel data that have been recorded during the mobile mapping activity time, or may have used images of virtual planes and combined with data derived from a digital map database. FIG. 15b shows a roadside panorama to the left side of a street and FIG. 15c shows a roadside panorama to the right side of the street. FIG. 15d shows segments expanded from a map database or also from an orthorectified image of a street (also collected from a mobile mapping vehicle). It can be seen that a pseudo-real view of a street can be produced with a very limited number of planes. References 1502 and 1506 indicate portions of images that have been obtained by making pseudo-perspective views of the panoramas of fig. 15b and 15c, respectively. The portions 1502 and 1506 can be easily generated by transforming the panoramas of 15b and 15c into perspective images by sequentially projecting the pixel columns of the roadside panoramas on the pseudo-real view starting with the pixel column having the position farthest from the viewing position and going to the pixel column having the position closest to the viewing point. Reference 1504 indicates the portion of the image that has been obtained by making an extension of the map database or the perspective of an orthorectified image of the road surface.

It should be noted that in pseudo perspective images, all buildings on one side of the road have the same building line and therefore they cannot be a full perspective view. In practice, each building may have its own building line. In a panorama captured by a slit scan camera, the buildings will then have different sizes. Using this type of panorama in this application will produce a perspective image that looks strange. The different vertical distances between the building and the road will be interpreted in the fluoroscopic image as different heights and sizes of the building. In this case, the invention is able to produce a fairly realistic view image with a fraction of the processing power required for a more complete 3D representation. According to the method according to the invention, roadside panoramas of streets are generated in two steps. First, for buildings along the street, one or more multi-view panoramas will be made. Second, a roadside panorama is generated by projecting the one or more multi-viewpoint panoramas onto one common smooth surface. In one embodiment, the common smooth surface is parallel to a line along the road, e.g., a trajectory line, a centerline, a sideline of an automobile. By "smooth" it is meant that the distance between the surface and the line along the road may vary, but not abruptly.

In the first action, a multi-viewpoint panorama is generated for each smooth surface along the curb. A smooth surface may be formed by one or more adjacent building facades having the same building line. Furthermore, in this action, as many obstacles in front of the surface as possible will be removed. The removal of the obstacle can be accurately performed only when the determined position of the surface corresponds to the actual position of the facade of the building. The orientation of the surface along the roadway may vary. Furthermore, the vertical distance between the direction of the road and the surface of two adjacent multi-viewpoint panoramas along the street may vary.

In a second action, a roadside panorama is generated from the multi-view panorama generated in the first action. The multi-view panorama is assumed to be a smooth surface along the road, where each pixel is considered to represent the surface as seen from a defined distance normal to the surface. In the roadside panorama according to the present invention, the vertical resolution of each pixel of the roadside panorama is similar. For example, a pixel represents a rectangle having a height of 5 cm. The roadside panorama used in the application is a virtual surface, with each multi-view panorama along the buildings of the roadside scaled such that it has a similar vertical resolution at the virtual surface. Accordingly, streets with houses having equal street level but different building lines will be visualized in the panorama as if the houses had the same building lines and similar street level.

For the roadside panorama described above, the depth information may be associated along the horizontal axis of the panorama. This enables an application running on a system with some powerful image processing hardware to generate a 3D representation from the panoramic view according to the actual position of the building.

In current digital map databases, streets and roads are stored as road segments. The visual output of the present application using digital maps can be improved by associating in a database left and right roadside panoramas, and optionally orthorectified images of the street's road surface, with each segment. In the digital map, the position of the multi-view panorama can be defined by absolute coordinates or coordinates with respect to predefined coordinates of the segments. This enables the system to accurately determine the position of the pseudo-perspective view of the panorama in output relative to the street.

Streets with intersections or junctions will be represented by several segments. The intersection or confluence point will be the start or end point of the segment. When the database includes associated left and right roadside panoramas for each segment, a perspective view as shown in FIG. 15a can be readily generated by making a perspective view of the left and right roadside panoramas associated with segments of the street that are visible and at a reasonable distance. Fig. 15a is a perspective image generated for the case where the car has a driving direction parallel to the direction of the street. Arrow 1508 indicates the orientation and position of the car on the road. When the panorama is generated for the most common plane, the panorama will start with the leftmost building and end with the rightmost building of the curb corresponding to the road segment. Thus, there is no panoramic view of the space between the buildings at the intersection. In one embodiment, these portions of the fluoroscopic image will not be archived with information. In another embodiment, these portions of the fluoroscopic image will be archived with the corresponding portions of the panorama associated with the segments connected to the intersection or meeting point and the expanded map data or ortho-corrected surface data. In this way, both sides of the building at the corners of the intersection will be displayed in the fluoroscopic image.

In navigation systems without dedicated image processing hardware, the display may still be refreshed frequently, e.g., every second, depending on the distance traveled while driving the vehicle. In that case, the perspective view would be generated and output each second based on the actual GPS position and orientation of the navigation device.

Furthermore, the multi-view panorama according to the present invention is suitable for use in applications for easily providing a pseudo-real view of the surroundings of a street, address or any other point of interest. For example, an output rendering route planning system can be easily enhanced by adding a geo-referenced roadside panorama according to the present invention, where the front face of the building has been scaled to equalize the resolution of the pixels of the building. Such a panorama corresponds to a panorama of a street, where all buildings along the street have the same building line. The user searches for a location. The corresponding map is then presented in a window on the screen. Subsequently, in another window on the screen (or temporarily on the same window), an image (like that of 15b or 15 c) is presented according to the roadside oriented perpendicular to the road corresponding to said position. In another implementation, the orientation of the map on the screen can be used to define the perspective in which orientation the panorama should be given. All pixels of the roadside panorama are considered to represent the street-side at the location of the surface of the roadside panorama. The roadside panorama only includes visual information assumed on the surface. Thus, a pseudo-real perspective view can be easily made for any arbitrary viewing angle of the roadside panorama. The map may be rotated on the screen by a rotation function of the system. At the same time, a corresponding perspective pseudo-reality image corresponding to the rotation made can be generated. For example, when the direction of a street is from the left to the right of the screen representing a corresponding portion of a digital map, only a portion of the panorama as shown in fig. 15b will be displayed. The portion may be displayed without transforming the image, as it is assumed that the display represents a roadside view, which is perpendicular to the direction of the street. Further, the position selected by the portion from the user corresponds to a predetermined region of the left and right panoramas. When the street is oriented from the bottom to the top of the screen, a perspective view like fig. 15a would be generated by combining the left and right roadside panoramas and optionally the orthorectified images of the road surface.

The system may also include a flip function to rotate the map over 180 ° with one instruction and view the other side of the street.

The glance function of the system may be used to walk in the direction of streets on the map and simultaneously display a corresponding visualization of the streets in accordance with the orientation of the map on the screen. Whenever a pseudo-reality image is presented as the image used, the left and right roadside panoramas and the orthorectified road surface image (if needed) represent the rectified image. The corrected image is an image in which each pixel represents a pure front view of the building facade and a top view of the road surface.

Fig. 15b and 15c show roadside panoramas of a street, where all houses have the same ground level. However, it will be apparent to those skilled in the art that the above-described method will typically produce a roadside panorama where houses having different ground planes will be displayed as roadside panoramas of different heights in the roadside panorama. Fig. 18 shows such a roadside panorama. In the roadside panorama, only pixels corresponding to a surface representing the multi-view panorama should be displayed on a display. Thus, when the roadside panorama is reproduced on the display, the pixels in areas 1802 and 1804 should not be counted. Preferably, the regions 1802 and 1804 will be given values, patterns or textures that enable detection of where along the edge of the region of the object along the roadside. For example, the pixels in the regions 1802 and 1804 will obtain values that are not normally present in an image, or in each column of pixels, the values of the pixels begin with a first predefined value and end with a pixel having a second predefined value, where the first predefined value is different than the second predefined value. It should be noted that a building on a mountain may have a street level in which the ground level has a slope. This will also be seen in the street-side multi-view panorama and the roadside panorama containing the multi-view panorama.

There are applications that visualize the height information of roads when generating a perspective image of a digital map on a screen. Roadside panoramas as shown in fig. 18 are well suited for use in those applications to provide a pseudo-real perspective view of the street. The height of the road surface will in most cases match the ground level next to the street. A street-side multi-view panorama may have been projected onto a surface associated with the roadside panorama. In such a case, the height of the road surface may not match the height of the ground plane next to the street. The application may be provided with an algorithm that detects the difference between the height of the road surface and the ground plane of the street level in the multi-view panorama. Thus, the application is arranged to determine in each pixel column the vertical position corresponding to the lowest position of the pixels of the object represented by the roadside panorama by detecting the position of the top pixel of area 1802. Since each pixel represents an area having a predetermined height, the difference in height between the road surface and the ground plane may be determined. This difference along the street is then used to correct the street-level height in the panorama and generate a pseudo perspective image of the road surface and the curb, where the height of the road surface matches the height of the ground plane of the street level.

There are applications that use maps that do not contain the height of the road. Therefore, it is only suitable for generating a perspective view of a horizontal map. The combination of the roadside panoramas of fig. 18 will produce a perspective image in which the ground level of the building varies along the road. This inconsistency may not seem realistic. Two embodiments will be given in which these applications can provide a pseudo-real perspective image.

In a first embodiment, the application will derive height information from roadside panoramas and use the height information to augment a perspective view of a horizontal map. Thus, the application is arranged to determine in each pixel column the vertical position corresponding to the lowest position of the pixels of the object represented by the roadside panorama by detecting the position of the top pixel of area 1802. Since each pixel represents a region having a predetermined height, the difference in height along the street may be determined. This difference along the street is then used to generate a pseudo perspective image of the road surface, which visualizes the corresponding difference in height along the street. In this way, the roadside panorama can be combined with a road surface, where in the pseudo-real perspective image the road surface and a surface of a roadside view will be contiguous. It will be apparent to those skilled in the art that if road surfaces with different heights have to be produced from the roadside ground plane shown in fig. 18, gradually increasing/decreasing road surfaces should be produced. Preferably, a smoothing function is applied to the ground level along the street derived from the roadside panorama. The result of applying this smoothing function is that the height of the road surface changes smoothly, which is a more realistic view of the road surface.

In a second embodiment, the application would remove the area 1802 from the roadside panorama and use the image thus obtained in combination with a horizontal map, as opposed to the first embodiment. Removal of the area 1802 will result in an image similar to the roadside panorama shown in fig. 15 c. By removing the height information from the roadside panorama, a pseudo-real perspective image is generated representing a horizontal road surface where buildings along the road all have the same ground level. In that case, the ground plane of the front face in the roadside panorama has a slope that can be seen in the pseudo-real perspective image by distortion of the visual rectangularity of doors and windows.

The foregoing detailed description of the invention has been presented for purposes of illustration and description. It is not intended to be exhaustive or to limit the invention to the precise form disclosed, and obviously many modifications and variations are possible in light of the above teaching. For example, instead of using source images of two or more cameras, a panoramic view of a building surface may be generated using only a sequence of images of one camera. In that case, for the front face at a predefined distance perpendicular to the trajectory of the moving vehicle, the two consecutive images should have sufficient overlap, for example, > 60%.

The described embodiments were chosen in order to best explain the principles of the invention and its practical application to thereby enable others skilled in the art to best utilize the invention in various embodiments and with various modifications as are suited to the particular use contemplated. The scope of the invention is intended to be defined by the appended claims.

Claims (18)

1. A method of generating a multi-view panorama of a roadside, comprising:

obtaining a set of laser scan samples obtained by at least one land-based laser scanner mounted on a moving vehicle, wherein each sample is associated with position data and orientation data;

acquiring at least one sequence of images, wherein each sequence of images is obtained with a land-based camera mounted on the moving vehicle, wherein each image of the at least one sequence of images is associated with position and orientation data;

extracting a surface from the set of laser scan samples and determining a location of the surface from the location data associated with the laser scan samples;

generating a multi-view panorama of the surface from the at least one sequence of images as a function of the position of the surface and the position and orientation data associated with each of the images.

2. The method of claim 1, wherein generating comprises:

detecting an obstruction obstructing viewing of a portion of the surface in a first image of the at least one sequence of images;

selecting an area of a second image that visualizes the portion of the surface; and

generating the portion of the multi-view panorama using the region of the second image.

3. The method of claim 1, wherein generating comprises:

detecting one or more obstacles obstructing a view of a portion of the surface in all images of the at least one sequence of images;

projecting a view of one of the one or more obstacles to the multi-view panorama.

4. The method of claim 3, wherein generating further comprises:

determining, for each of the detected obstacles, whether it is fully visible in any of the images;

projecting a view of the detected object from one of the at least one image to the multi-view panorama if the detected obstacle is fully visible in the at least one image.

5. The method according to any one of claims 1-4, wherein the panorama is preferably generated from a portion of an image having an associated viewing angle that is most perpendicular to the surface.

6. The method of claim 1, wherein generating comprises:

generating a master shadow map of the surface;

and generating the multi-view panorama according to the main shadow map.

7. The method of claim 6, wherein generating a master shadow map includes:

selecting an image having a viewing window comprising at least a portion of the surface;

generating a shadow map for each selected image by projecting shadows in front of the surface that are visualized for obstacles in the corresponding selected image; and

combining the shadow maps of the selected images to obtain the master shadow map.

8. The method of claim 6 or 7, wherein generating further comprises:

dividing the main shadow map into a plurality of segments;

determining, for each segment, a corresponding image having no obstructions in its associated viewing window; and

projecting a region associated with the segment on the multi-view panorama using the corresponding image.

9. The method of claim 8, wherein generating further comprises:

if no corresponding image of a segment has been found, an image with the entire obstruction in its associated viewing window is used.

10. The method of claim 8 or 9, wherein generating further comprises:

if no corresponding image of a segment has been found, the image with the associated view angle that is most perpendicular to the surface is used.

11. The method of claim 1, 2 or 3, wherein the surface is extracted by performing a histogram analysis on the set of laser scan samples.

12. A method of generating a roadside panorama comprising:

retrieving a plurality of multi-view panoramas that may have been generated by any of claims 1 to 10 and associated location information;

determining a location of a virtual surface of the roadside panorama; and

projecting the plurality of multi-view panoramas on the virtual surface.

13. An apparatus for performing the method of any one of claims 1-11, the apparatus comprising:

an input device;

a processor-readable storage medium; and

a processor in communication with the input device and the processor-readable storage medium;

an output device for enabling connection with a display unit;

the processor-readable storage medium stores code to program the processor to perform a method comprising:

obtaining a set of laser scan samples obtained by at least one land-based laser scanner mounted on a moving vehicle, wherein each sample is associated with position data and orientation data;

acquiring at least one sequence of images, wherein each sequence of images is obtained with a land-based camera mounted on the moving vehicle, wherein each image of the at least one sequence of images is associated with position and orientation data;

extracting a surface from the set of laser scan samples and determining a location of the surface from the location data associated with the laser scan samples;

generating a multi-view panorama of the surface from the at least one sequence of images as a function of the position of the surface and the position and orientation data associated with each of the images.

14. A computer program product comprising instructions which, when loaded on a computer arrangement, allow said computer arrangement to perform any of the methods according to claims 1-11.

15. A processor readable medium carrying a computer program product, which when loaded on a computer arrangement allows said computer arrangement to perform any of the methods according to claims 1-11.

16. A processor readable medium carrying a multi-view panorama that has been obtained by performing any of the methods of claims 1-11.

17. A computer-implemented system for simultaneously providing on-screen a selected location in a map and street and a pseudo-reality view from the location, comprising

A map including the selected location;

the at least one roadside panorama of claim 11;

a map generation component for displaying a display map comprising the selected location in the street in a variable orientation on a screen; and

a view generation component for generating a pseudo-reality view of the selected location from the at least one roadside panorama in accordance with the variable orientation.

18. The computer-implemented system of claim 17, wherein the map and the pseudo-reality view are combined into one pseudo-perspective view.