Classifications

• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T13/00—Animation
  • G06T13/80—2D [Two Dimensional] animation, e.g. using sprites
• • G—PHYSICS
  • G06—COMPUTING; CALCULATING OR COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T1/00—General purpose image data processing
  • G06T1/0007—Image acquisition

Definitions

• the field of the present invention relates to animation.
• apparatus and methods are described herein for generating and storing an image database and for interactively generating animation sequences using images in the datab-ase.
• a picture is a two-dimensional view of one or more objects.
• Film, or animation is a linear sequence of pictures. Film or animation may be viewed as a one-dimensional array of two-dimensional pictures.
• the present invention discloses apparatus and methods for creating two- or higher- dimensional arrays of pictures that would allow a viewer to create his/her own film or animation of the object.
• the present invention provides apparatus and methods to create animations of objects that are very large relative to the amount of detail one might want to see in the animation, by using an image database. This is the case, for instance, when creating an aerial animation of a geographical area. The earth is very large as compared to the size of an individual house that one might want to view in an animation. This is also the case when creating a microscopic animation of a skin surface. Here the size of the object is also very large as related to the amount detail one might wish to see. Each image will only show a very small detailed part of the object surface.
• the details that one might want to see may not be visible in a straight-on picture (i.e., acquired from directly above).
• the image database is generated using a combination of oblique cameras that are targeted at an angle greater than about 0° and less than 90° downward from horizontal toward the object, so that vertically oriented details of the object may be seen in an animation.
• PANORAMIC ANIMATIONS Microsoft's SurroundVideo® and Apple Computer's QuickTime® VR use a system where a camera is placed inside a scene. The camera is rotated and images are acquired from all directions. The images are stored in a computer and then assembled so that a single seamless image strip is created showing all directions. The image is projected onto a cylindrical object around the viewer. The viewer can now interactively create panoramic animations by moving around and viewing in all directions.
• An undesirable artifact of this technology is that wide-angle type views .are generated of the view. This means that the true size and shape of objects is distorted and less comprehensible to the viewer. In the present invention montages of images that are not wide angle are created, so no wide angle artifacts are generated.
• the image database may be generated in the present invention using a large number of camera positions, thereby allowing camera movements from one point to another in an animation. In this way the viewer can create an animation of images acquired from different points, thereby showing an animation of movement over the object surface.
• SurroundVideo® and QuickTime® VR only images acquired from a single point within the object are animated.
• SurroundVideo® and QuickTime® VR acquire the images from inside the object, whereas in the present invention images are acquired at a distance from the object.
• Stereo images can be used in still images or in animated images.
• the present invention may be implemented with regular images and may also be implemented with stereo images.
• 3D MODELING Three-dimensional (3D) models can be created in a computer, and perspective images can be generated and animated from the 3D models.
• 3D models contain a three-dimensional geometrical description of the object, or the surfaces of the object.
• Perspective images can be created from any position in the 3D Model.
• the perspective images can be sequenced to create an animation.
• the image database used in the present invention is a collection of two-dimensional images. It may employ a 2D array of grid points with a reference to the images that have been generated at each such grid point relative to the object.
• the problem of 3D models is that it is exceedingly difficult to create a model of a large area that is completely accurate.
• an optimal scheme may be devised to photographically map entire cities or areas and create a mosaic of images that are comprehensible to the average viewer. Images can be selected from the database and scaled in such a way that a nearly optically continuous image sequence is presented to a viewer as an animation.
• the image database requires a large amount of computer storage space, typically much more than a 3D model.
• a multi-perspective image database is also distinctly different from photogrammetry.
• a multi-perspective image database it is essential to create oblique images from an object.
• images used for photogrammetry should have as little perspective as possible.
• each image in the database contains perspective, it is possible to create a sequence of images that gives the feeling of moving around or over an object. This is not possible with images that show little or no perspective.
• the change in perspective in the sequence of images makes it possible to see the exact relationship between details of the object. In a city database, for instance, the exact position and height of a tree next to a house can be seen. Without perspective images, and without a nearly optically continuous image sequence, this is not possible.
• the sequence of images can produce a feeling of motion through the scene or movement of objects in the scene.
• the sequence of images can only be played forward and backward, and the viewer has no control over what is viewed, in what order, or from what direction.
• a viewer has full control of the .animation, and may move and view in any direction, thereby creating his or her own "film" of the target object shown in the image database. This offers a great potential for data analysis, travel information, city planning, space exploration, computer gaming, and many other fields.
• any viewer may "fly” over the object or city from anywhere in the world. With sufficient computer network connection bandwidth and/or network access speed, the viewer may remotely “fly” over a city interactively in real time.
• Certain aspects of the present invention may overcome one or more aforementioned drawbacks of the previous art and/or advance the state-of-the-art of object animation apparatus and methods, and in addition may meet one or more of the following objects:
• To provide apparatus and methods for generating an image database the image database comprising images of a target object recorded from a variety of viewing positions and in a variety of viewing directions;
• the image database comprising images of a target object recorded from a variety of viewing positions and in a variety of viewing directions;
• the imaging assembly comprising at least one camera
• the imaging assembly comprising at least one wide-angled camera;
• the imaging assembly comprising a plurality of cameras pointing radially outw.ard and downward toward the target object;
• an 25 apparatus comprising: an imaging assembly comprising at least one camera; means for moving the imaging assembly relative to the target object; means for determining a position of the imaging assembly relative to the target object; means for storing an image recorded by the camera; and means for storing image parameters and the position of the imaging assembly corresponding to each image.
• One or more of the foregoing objects may be achieved in the 30 present invention by a method comprising the steps of: a) recording an image of the target object with a camera on an imaging assembly; b) determining the position of the imaging assembly relative to the target object; c) storing the image recorded by the camera; d) storing image parameters and the position of the imaging assembly corresponding to the image; e) moving the imaging assembly to a new position relative to the target object; and f) repeating steps a) through e).
• an apparatus comprising: means for storing the image database; means for storing image parameters and imaging assembly position for each image in the image database; means for selecting a trajectory of viewing position, viewing direction, viewing distance, and/or viewing time by a user; means for selecting, cropping or splicing, scaling, and sequencing images from the image database using the user-selected trajectory, stored image parameters, and stored imaging assembly positions; and means for presenting the selected, scaled, and sequenced images from the image database as an animation sequence.
• One or more of the foregoing objects may be achieved in the present invention by a method comprising the steps of: a) selecting a trajectory of viewing position, viewing direction, viewing distance, and/or viewing time by a user; b) selecting, cropping or splicing, scaling, and sequencing images from the image database using the user-selected trajectory, stored image parameters, and stored imaging assembly positions; and c) presenting the selected, scaled, and sequenced images from the image database as an animation sequence.
• an imaging assembly may comprise a plurality of cameras, all pointed toward the target object at about the same vertical angle of greater than about 0° and less than 90° downward from horizontal.
• the fields-of-view of the cameras preferably cover a full circle when viewed from above.
• An imaginary grid of points may be defined over the object.
• a viewer may interactively "move" over the grid and change the viewing direction and image size by selecting a trajectory of viewing position, viewing direction, viewing distance, and/or viewing time.
• Images may be selected, cropped or spliced, and scaled so that a near optically-continuous sequence of images is created.
• the images may be played back at a speed sufficiently great that a viewer may perceive "movement" relative to the object.
• the images may be played back preferably at a rate greater than about 6 images per second, most preferably at about 24 images per second (a common frame rate for a motion picture). Additional objects and advantages of the present invention may become apparent upon referring to the preferred and alternative embodiments of the present invention as illustrated in the drawings and described in the following written description .and/or claims.
• Figure 1 is a block diagram showing an overview of apparatus and methods according to the present invention.
• Figure 2A shows an isometric view of an imaging assembly according to the present invention
• Figure 2B shows a side view of a single camera/gyro-stabilizer assembly.
• Figures 3 A .and 3B show, respectively, top .and isometric views of the camera directions of an imaging assembly according to the present invention.
• Figure 4 shows a top view of the fields-of-view of cameras on an imaging assembly according to the present invention.
• Figure 5 shows a flow diagram for generation of a multi-perspective image database according to the present invention.
• Figure 6 shows a top view of multiple camera directions from each of multiple camera positions on grid points, according to the present invention.
• Figure 7 shows a top view of a target object detail with camera positions and directions corresponding to images showing the detail, according to the present invention.
• Figure 8 shows a flow diagram for generation of an animation sequence from a multi- perspective image database, according to the present invention.
• Figure 9 A shows a top view of a target object detail with camera positions and directions corresponding to images showing the detail
• Figures 9B, 9C, .and 9D show images in the multi-perspective image database showing the detail
• Figures 9E, 9F, and 9G show images clipped and scaled to form part of an animation sequence showing the detail.
• Figure 10 is a flow diagram showing linkage of data, images, maps, audio, and/or video to a multi-perspective image database according to the present invention.
• Figure 11 is a flow diagram showing analysis and display of data linked to a multi- perspective image database according to the present invention.
• Figure 12 is a flow diagram showing editing of data linked to a multi-perspective image database according to the present invention. MODES FOR CARRYING OUT THE INVENTION
• the viewer may perceive apparent motion and/or evolution of one or more objects in the scene, and/or apparent motion of the viewer's viewing position and/or viewing direction relative to the scene.
• a preferred convention for such Cartesian coordinates has the xy-plane oriented substantially horizontally and substantially coinciding with the surface of the target object and the z-axis substantially perpendicular to the surface of the object, although other conventions may be used without departing from inventive concepts disclosed and/or claimed herein.
• camera setup alternatively "imaging assembly” - a set of cameras used to create a multi- perspective image database, and typically comprising a plurality of oblique cameras and one vertical camera clip an image - alternatively “crop an image” - select a part of an image for viewing continuous image sequence - also "near-continuous image sequence", “optically-continuous image sequence”, or “near-optically-continuous image sequence” - a series of images where the changes in perspective .and scale of each consecutive image are (nearly) continuous, so that when played back at sufficient speed it produces the appearance of smooth camera movement.
• the term “near-continuous” is used because from a mathematical point of view, the sequences generated by the present invention are only approximately continuous, not exactly continuous.
• An object of the present invention is to produce a sequence that appears nearly continuous to the human eye. Such a sequence may be played back at a rate of preferably greater than about 6 images per second, most preferably about 24 images per second (a common frame rate for motion pictures). Even at relatively slow playback speeds (similar to a slide show), such a sequence may yield a perception of motion relative to the target object.
• image database alternatively "multi-perspective image database” or simply “database” - collection of all images. However, other databases are referred to herein containing other types of data.
• image parameter - camera position (X CA M, y m, z ⁇ u), camera vertical angle ( ⁇ CAM), camera azimuthal angle (CAM), camera constant (c), camera field-of-view ( ⁇ CA M, A ⁇ > C AM), .and/or other parameters describing the recording of a given image of the target object.
• a database of these parameters for each image in the image database may be generated and stored for use in subsequent selection, manipulation, sequencing, and/or playback of images in an animation sequence.
• object detail - alternatively "object detail”, “target object detail”, or “detail of the target object” - a detail or feature of the target object which appears in one or more images of a multi- perspective image database geographical coordinates - alternatively x GE0 , y ⁇ o, .and z 0 ⁇ o, "object coordinates", “target object coordinates”, or “physical coordinates” - the three-dimensional Cartesian coordinates of a particular detail of the target object, preferably using the same convention set forth for the camera coordinates, although other conventions may be used without departing from inventive concepts disclosed and/or claimed herein.
• geo-reference may be used when the target object is the earth's surface or a portion thereof, or may be used in the more general case of any target object for which an image database has been generated.
• geo-reference - a link between a location on a map or image and data related to a detail of the object corresponding to the location positioning system - any apparatus and/or method used to determine a position relative to the target object (i.e., for determining object coordinates).
• a positioning system may include a system for measuring and/or storing an elevation profile for the object surface.
• grid a systematic field of points in a substantially horizontal plane, typically laid out as
• the grid spacing may be small relative to the size of the target object and to the size of the area of the target object corresponding to an image.
• grid size alternatively "grid spacing" - the distance between two adjacent grid points
• the horizontal plane which substantially coincides with the surface of the object is also referred to as the .xy-plane, although other conventions may be used without departing from inventive concepts disclosed and/or claimed herein.
• a "horizontal plane” may be defined l o locally, and may not be parallel to or coincide with a "horizontal plane” at a distant location.
• an "average horizontal plane” may be defined locally which most nearly approximates the rough surface locally.
• Hotspot an area or collection of areas on an image sharing a common attribute. With the use of 15 hotspots, data can be linked to images. Hotspots can also be used to display certain characteristics of the data superimposed on the image. Selection and activation of a hotspot by a pointing device in a graphical user interface (GUI) environment may result in a certain action such as the retrieval of data, actions of the software, etc.
• image coordinates - alternatively x M and y M - two-dimensional Cartesian coordinates used to 20 specify a location in an image.
• the xy-plane corresponds to the plane of the two-dimensional image, with the origin at the center of the image, the x-axis horizontal, and the y-axis vertical, although other conventions may be used without departing from inventive concepts disclosed .and/or claimed herein.
• Internet any worldwide computer network that allows computers to communicate with each 25 other using standardized communication protocols
• intranet a computer network that allows computers from a selected group (a company, a department, an organization) to communicate with each other using standard communication protocols
• a vertical angle of 0° corresponds to a camera directed horizontally, while a vertical angle of 90° corresponds to a camera directed vertically downward.
• a "vertical axis" may be defined locally, and may not be parallel to a
• a vertical axis at a distant location.
• a vertical axis may be defined substantially pe ⁇ endicular to an "average horizontal plane", which may be defined locally and which most nearly approximates the rough surface locally.
• vertical plane - a plane perpendicular to a horizontal plane.
• a vertical plane is parallel to the vertical axis.
• Vertical planes may be defined locally for curved and/or rough target object surfaces.
• vertical camera - a camera pointed at the target object with a vertical angle of about 90°, i.e., vertically downward. The images created by such a camera will appear flat and show little perspective.
• WAN alternatively "wide area network” - a network of computers at multiple locations z-direction - the direction perpendicular to a horizontal plane, i.e., directly upward or downward.
• the imaging assembly is considered “above” the target object (positive value for Z CAM ) -and pointed “down” at the target object, although other conventions may be used without departing from inventive concepts disclosed and/or claimed herein.
• zoom-in - to enlarge the size of the object in .an image on the screen so it appears larger and closer zoom-out - to decrease the size of the object in .an image on the screen so it appears smaller and farther away.
• any of the terms "means for storing the image database”, “means for storing image parameters”, “means for selecting a trajectory”, “means for selecting, scaling, and sequencing images”, “means for presenting”, and any other “means” described herein shall denote any device, machine, apparatus, hardware, software, combination thereof, and/or equivalent thereof capable of performing the specified functions, including but not limited to one or more computers, storage media, servers, server software, browser software, user interface hardware and/or software, terminals, networked devices, the Internet, the World Wide Web, an intranet, an extr-anet, a LAN, a WAN, modems, memory units, storage devices, processors, distributed computing resources, integrated circuits, ASICs, functional equivalents thereof, and/or combinations thereof.
• any medium suitable for storage of information, data, and or images, whether in digitally-encoded form or in non-digital form, or any combination of and/or functional equivalents of such media may be utilized for any of the various storage functions described herein.
• such storage media may be distributed over a plurality of storage locations.
• network connection shall encompass any apparatus, hardware, software, and/or combination thereof capable of providing any of the various connections described herein, including but not limited to an Internet connection, an Internet service provider (ISP), browser software, user interface hardware and/or software, a modem hook-up, a cable, a phone line, a satellite link, a wireless link, a microwave link, television, a BBS system, a local area network, a wide area network, an intranet, an extranet, direct linkage of a plurality of computers, connections between devices within a single computer, terminal connections, program instructions, multiple user accounts on a single computer, multiple storage areas on a single computer, combinations thereof, functional equivalents thereof, and future apparatus, methods, systems, and/or protocols for performing analogous functions.
• ISP Internet service provider
• Images of the target object may be recorded at block 10 by an imaging assembly comprising one or more cameras. Images may be recorded from multiple viewing locations and in multiple viewing directions relative to the target object.
• a positioning system may be employed at block 20 to record a camera position corresponding to each image recorded, and one or more of the camera position, camera direction, and/or other image parameters may be stored for each image at block 40.
• the images themselves may be stored at block 30 as an image database.
• the positioning system may be employed to control the recording of images. Data related to details of the target object may be linked to the image database via the image parameters at block 50.
• a viewer may select a viewing trajectory at block 70.
• the selected trajectory and image parameters may be employed at block 60 to select, scale and sequence images retrieved from the image database.
• the image sequence may then be presented as an animation sequence at block 90.
• the viewer may: view the animation sequence; select a subsequent trajectory based on the animation sequence; and/or view, analyze, and/or edit linked data at block 80. Any connections represented by arrows in Figure 1 may be provided by a network or other connection.
• FIG. 2A and 2B A preferred embodiment of an imaging assembly 100 according to the present invention is shown in Figures 2A and 2B, and comprises: a plurality of oblique cameras 110 pointing radially outward from a central point 101 and downward toward the object, each preferably having substantially the same vertical angle of greater than about 0° and less than 90°, preferably between about 15° and about 75°, and most preferably between about 15° and about 30°; and one vertical camera 120 pointing vertically downward at the object with about a 90° vertical angle.
• the viewing directions of the cameras are illustrated in Figures 3 A and 3B.
• the vertical camera may be used to record reference and images.
• the fields-of- view 300 of the oblique cameras should preferably cover an entire 360° range of azimuthal angles around a central point and should preferably be substantially uniformly spaced around the central point.
• an imaging assembly may comprise twelve oblique cameras each having a field-of-view of about 40°, although many other configurations having various numbers of cameras and various fields-of-view may be employed without departing from inventive concepts disclosed and/or claimed herein.
• the flow diagram of Figure 5 illustrates a preferred method for generating a multi- perspective image database for an object (an area of a city, for example).
• a fine imaginary 2D grid may be defined (408) to overlay the object, as shown in Figure 6.
• Images may be recorded with the imaging assembly positioned at each xy grid position 501 at a height z C m above the object.
• the horizontal distance between adjacent grid points should preferably be relatively small compared to the size of the object.
• the entire camera setup is prepared (406) aimed at the object so that one vertical image and multiple oblique images (represented by arrows 502) may be recorded at each grid point.
• All oblique cameras should preferably have the same focal length (or equivalently, the same camera constant, roughly the lens-to-focal-plane distance) and the same field-of-view. All oblique cameras are preferably aimed at the object under subst-antially the same vertical angle of greater than about 0° and less than 90°, preferably between about 15° and about 75°, most preferably between about 15° and about 30°, to record suitable perspective images. Since the focal length, the camera height, and the vertical angle are substantially the same for all oblique cameras, all perspective images will be substantially the same scale and have substantially the same perspective distortion. The area covered by an image is determined by the distance from the camera to the object, and the camera field-of-view.
• the height of the imaging assembly above the object surface and the camera field-of-view should preferably be chosen so that each oblique image covers an area of the object greater than about the grid spacing, most preferably greater than about three times the grid spacing. This ensures that a detail 601 on the target object will be visible in multiple images 502 recorded from multiple grid points 501, as illustrated in Figure 7.
• the imaging assembly In order to generate an entire multi-perspective image database, as illustrated in the flow diagram of Figure 4, the imaging assembly must be moved to each of the grid points, at which each camera records an image. For the example of an area of a city, this may be done by mounting the imaging assembly in an aircraft and flying (412, 414) over the city along the imaginary gridlines, and recording (422) images with each camera as the aircraft crosses each grid point (416).
• the imaging assembly may be gyro-stabilized (by gyro-stabilizers 130) to insure that the cameras are always pointing in the same direction. Navigation of the aircraft along the imaginary gridlines and accurate recording of the camera position for each image may be facilitated by use of a global positioning system (GPS) or other positioning system (404, 410, 416). For an imaging assembly comprising twelve oblique cameras and one vertical camera, thirteen images will be recorded (422) at each grid point. Each image is stored (424, 426) in the image database, and image parameters are also stored (418, 420) for each image.
• GPS global positioning system
• image parameters preferably include, but are not limited to, camera position (X CA M, ycAM, Z CA M), camera vertical angle ( ⁇ CAM), camera azimuthal angle ( ⁇ CAM), camera constant (c), and camera field-of- view ( ⁇ CAM , ⁇ C ⁇ M).
• the images and image parameters are preferably stored in digital form in a searchable format, for example, as a database in a computer memory or on one or more computer storage media. Without departing from inventive concepts disclosed .and/or claimed herein, images, image parameters, data, and/or information may be stored in any suitable format, digital and/or non-digital, on any suitable storage medium.
• image recording and the flight may be ended (428, 430).
• Generation of a database of images as described above may be repeated at different points in time, and the time at which each image is recorded may be stored as an image parameter (t C M).
• the camera height, Z C AM may often be defined with respect to some reference point of the t-arget object, but not necessarily the surface of the target object.
• Z CAM is typically defined as the camera elevation above sea level. If the terrain imaged is not flat, the height of the camera above the ground may vary, changing the perspective parameters of the images.
• a preferred embodiment of the present invention may therefore include as a component of the positioning system means for determining the elevation of the surface of the target object. Photogrammetry techniques may be employed using images recorded by the vertical camera to calculate an elevation profile for the area for which the image database is generated.
• the imaging assembly may employ a range-finder or equivalent device for measuring the elevation profile of the area, preferably concurrently with the recording of the images for the image database.
• the elevation profile may be stored and used in subsequent processing of images in the database, as set forth hereinbelow.
• a series of images (9E, 9F, and 9G, for example) of a detail 601 of the object may be created that show the detail in the same size, and that form a near- optically-continuous sequence.
• This may be accomplished according to the present invention by selecting (806, 808) images (9B, 9C, and 9D, for example), clipping (810) them (i.e. selecting a part of the image) or splicing them (i.e., assembling adjacent images), and scaling (812) the clipped or spliced images.
• the continuity of the images may not be mathematically perfect, but perfect continuity is not necess.ary to produce .an animation suitable for viewing by a human viewer when the images are played back.
• the images in the animation sequence may preferably be played back (819) at greater than about six images per second, most preferably at about 24 images per second (a common frame rate for commercial motion pictures).
• the animation may be played back at rates slower than about six images per second. Such a slow playback may still give a viewer the perception of motion relative to the target object, even if it appears as a "slide show" instead of a smooth animation.
• a single camera may be used to record images in multiple camera directions.
• the camera direction may be changed after recording each of multiple images at a given camera position, and the process repeated at successive camera positions.
• a single camera may be employed having a wide-angled, or "fish-eye" lens. The camera may be directed vertically downward, so that the camera field-of-view may cover an entire 360° range of azimuthal angles. A single wide-angled image may therefore cover .an area of the target object equivalent to the combined areas covered by a plurality of images recorded by an imaging assembly as described above.
• Each wide-angled image may be divided into a plurality of images prior to storage, or may be stored as a single image and processed during subsequent generation of an animation sequence.
• one or more cameras may be used in conjunction with one or more auxiliary optics, such as a plurality of mirrors, prisms, and/or lenses set at a variety of positions and/or orientations, to record images covering an area of the target object functionally equivalent to the combined areas covered by a plurality of images recorded by an imaging assembly as described above.
• Each image thus recorded may be divided into a plurality of images prior to storage, or may be stored as a single image and processed during subsequent generation of an animation sequence.
• trajectories 883 of viewing position, viewing direction, viewing distance, and/or viewing time relative to the target object may be simulated in an animation by selecting different image sequences from the l.arge database of images.
• Such trajectories may include as examples, but are not limited to: circular motion around an object detail with the view directed toward the detail; circuit motion with the view directed radially outward from the center of the circular motion; motion along an arbitrary curvilinear path with the view directed in the direction of motion; motion along an arbitrary curvilinear path with an independently arbitrarily varying viewing direction; a view from a fixed position with a fixed viewing direction showing images recorded at successively later times, thereby showing the temporal evolution of the object details visible in the view.
• softw.are may be employed to select the images from the database, and clip or splice and scale them so that the viewer experiences near-continuous movement when the image sequence is played back.
• the images are selected and displayed at a speed of preferably greater than about six images per second, most preferably about 24 images per second, the viewer experiences the feeling of smooth motion over the target object.
• An entire trajectory may be specified prior to any selection and/or presentation of images as an animation sequence.
• the viewer may preferably interactively control the trajectory of viewing position, viewing direction, viewing distance, and/or viewing time, thereby allowing the viewer to virtually "fly over" the target object, looking in .any desired direction -as he/she does so.
• Images may be selected, manipulated, and presented as an animation sequence and, concurrently, subsequent portions of the trajectory may be selected.
• the viewer may zoom in and out during a trajectory so that he/she may view an animation of relatively small details of the object, or an animation of a relatively large portion of the object.
• Pre-selected, default, and/or automatic trajectories may also be employed to generate animations.
• An animation generated for a trajectory may be stored for later play-back.
• a smaller vertical angle for the oblique cameras yields better viewing of vertical details of the target object (building facades and hills in the city example).
• the smaller the camera vertical angle at a given camera height the farther away the camera is from the viewed object detail, and the more the camera needs to zoom in to get good images of the detail.
• the more the camera is zoomed in the smaller the resulting field- of-view, and the more cameras are needed.
• Additional data may be available about details of the target object for which a multi- perspective image database has been created. For a city, for example, address information, phone directory information, tax data, property information, business and/or Chamber of Commerce data, census data, Standard Metropolitan Statistical Area (SMSA) and/or other government data, and/or other alphanumeric data may be available.
• SMSA Standard Metropolitan Statistical Area
• Maps may be available as CAD drawings, as scanned images, or in other formats. Images acquired from the ground may also be available for many objects, and in some cases there may be accompanying digitized video or sound. Some or all of this information may have associated geographical coordinates (x G o, j or equivalently longitude and latitude). Since the image parameters are known, the image position (x M , y m ) for any such detail information/data can be calculated for each image. This enables generation (either manually or, preferably, automatically) of one or more links (i.e., geo- references) between the information/data and each image in the multi-perspective image database in which the relevant location appears, as illustrated in the flow diagram of Figure 10.
• links i.e., geo- references
• the image location of each address can be calculated for each image and so-called "hotspots" generated on the images.
• GUI graphical user interface
• the corresponding address, or any other information/data associated with the image location may be displayed, either superimposed on the image or in a separate display.
• the viewer may be provided with an address finder, whereby entering an address into the system may automatically trigger the system to display an image or an animation of the lot and/or house, building, or other structure at the address.
• the linked data can be used to do any type of data sorting and/or analysis, and the geo-references may be used to display the results of the analysis on an image or an animation of images, as shown in the flow diagram of Figure 11.
• Analysis results may be color-coded and graphically overlaid on each image of the multi-perspective image database in real time as the viewer navigates through the database. For example, the viewer may graphically view the results of an analysis while viewing an animation and experiencing a three-dimensional "feel" for the entire target object.
• vertical reference images recorded by the vertical camera of the imaging assembly, new geo-references may be created for each image, or the positions of existing geo- references may be improved.
• the viewer may draw an outline of an object detail onto the different views simultaneously and attach data to the object detail. In this way the viewer can very finely define the size and shape of the detail in different dimensions, and the position of the object detail can be generated automatically for all images on which the detail appears.
• Viewers may also edit data that is linked to the image database, as shown in the flow diagram of Figure 12. This can be done without affecting the images themselves. Changes in the data may be displayed superimposed on an image or animation, allowing the viewer to interactively view his/her changes visually.
• a multi-perspective image database for an object or geographical area may be stored on a disk drive or other digital storage medium that is connected to a computer, to a computer network, or to the Internet.
• the storage medium with the image database may be referred to as the Server ( Figure 8, 802).
• the navigation and viewing hardware/software (801) used by the viewer to move through the images can run on the Server, or on a separate computer or terminal with a network connection to the Server directly, to a common computer network with the Server, or to the Internet.
• Navigation and viewing requests may be sent (803, 806) to the Server and the Server may select (806, 808), manipulate (clip 810 or splice, scale 812), and transmit (814, 817) the images to the viewer that comprise the desired view or animation.
• the viewer may navigate and view an animation of an object, city, geographical area (for which an image database is stored on the Server) from anywhere in the world.
• the viewing and navigating hardware/software, the image database, and Server software may also run on a single computer so that viewers may view ariimations on computers that are mobile, or computers that are not connected to the Internet or to a network.
• a multi-perspective image database typically will be large, possibly hundreds of gigabytes for a single target object. Therefore, it may be advantageous to store the images centrally, and to transfer images to individual viewers over a network or via the Internet.
• the apparatus and methods described above pertain primarily to a two-dimensional multi- perspective image database.
• a two-dimensional image database With such a two-dimensional image database, perceived movement in an animation (alternatively, the trajectory of viewing positions) occurs only within the horizontal plane from which the images are recorded, although a primitive sense of vertical movement can be achieved by zooming in or out.
• a three-dimensional multi-perspective image database may be generated.
• a multi-dimensional multi-perspective image database may be generated also having a temporal dimension.
• a time- interval multi-perspective image database thus generated may be very useful for monitoring changes in a target object or geographical area over time.
• a regular grid pattern for acquisition of images for the database is not strictly necessary.
• an image database may be generated and employed in which images are recorded from an arbitr.ary set of viewing positions, in an arbitrary set of viewing directions (not necessarily the same for all viewing positions), and at an arbitrary set of viewing times (not necessarily the same for all viewing positions or viewing directions), provided that the image database contains sufficiently many views to cover adequately the target object for the desired viewing and/or data analysis, display, and/or editing.
• acquisition of images from regular grid points offers the possibility of more efficient storage, selection, and/or processing of images and/or data.
• fish-eye or wide-angle lenses may be used for recording images for the image database. Additional viewing and navigating hardware/software may be employed to create perceived camera movement within an animation wherein the vertical angle of the view changes, thereby enabling animations wherein a viewer may simulate looking forward, shifting the view to look downward, and swinging further to look backward.
• stereo or holographic images may be stored in a multi-perspective image database and used to generate stereo or holographic animations, respectively.
• the apparatus and methods disclosed herein are particularly well suited for generating an image database for a city or other geographical area, and for implementation using digitally recorded and stored images.
• the cameras used in the imaging assembly preferably may be digital cameras, and preferably may be linked to a computer.
• the computer may preferably be equipped with reception hardware/software for receiving and processing information from a global positioning system (GPS), and preferably may contain or be linked to a high volume digital storage medium, such as a large hard disk, for storage of images and image parameters.
• GPS global positioning system
• the imaging assembly may be moved along grid lines over the area. At each grid point the computer triggers all cameras simultaneously, and the images are immediately downloaded to the computer and named (or numbered).
• the names of the images and their corresponding image parameters are simultaneously stored in a database.
• Image parameters such as x C , y AM, a . nd z CAM may be obtained from, or calculated from data obtained from, the GPS.
• images may be recorded by non- digital still or motion photography and converted to digital format and stored at a later time.
• any means for determining the position of the imaging assembly relative to the target object i.e., positioning system
• the GPS-based scheme disclosed herein is exemplary only.
• the images may be stored in non-digital form, i.e., as physical photographs, slides, holograms, stereo images, or other form.
• a mechanical device may be employed to store the pictures, slides, or other format, select from among them for an animation sequence, clip and/or scale projections of those images selected, and project the sequence as an animation.
• storage of the images, image parameters, data, and/or other information, whether in digital or non-digital form, may be on any storage medium suitable for storing images, image parameters, data, and/or other information.
• a multi -perspective image database may be created from within a target object.
• a multi-perspective image database may be generated from a 3D computer model. Images for the database may be rendered on the computer from each grid point with 3D rendering software and then stored. Animations may be generated from the rendered and stored images as described herein. This technique may be more advantageous than real time 3D rendering when storage space may be abundant, but processing power may be limited.
• a multi-perspective image database may be generated using a plurality of static imaging assemblies at a plurality of locations relative to the target object, rather than moving an imaging assembly to a plurality of locations.
• a target object includes surface details which move during acquisition of a multi- perspective image database (cars moving along city streets, for ex-ample)
• those moving surface details may be observed to appear, disappear, or move irregularly in an animation generated from the image database.
• transient surface details may be removed from images in the database. This may be accomplished automatically for digitally stored images through the use of optical pattern recognition software to detect .and remove the moving surface details.
• EXAMPLE CREATING A MULTI-PERSPECTIVE IMAGE DATABASE FOR A CITY.
• the following is exemplary only and shall not be construed so as to limit the scope of the present invention.
• Mount an imaging assembly comprising digital cameras and mounted on a gyro-stabilizer on the underside of a relatively slow-flying airplane, such as a Cessna 172 (which can fly at about 70 mph).
• a Cessna 172 which can fly at about 70 mph.
• an unmanned drone aircraft or miniature aircraft may be used, either automated or by remote control.
• Aim all twelve (or 16, 32, etc.) oblique cameras with substantially the same vertical angle of between about 15° and about 75° downward from horizontal.
• the multi-perspective image database may be linked to any address database (such as a county address database) that has longitude .and latitude information (or contains State Plane coordinates).
• This link may be used to locate a given address on any image in which it appears.
• the viewer may be provided with an address finder that will generate an animation of a house, building or other object, after entering the address.
• the address finder and the resulting animation may be provided over the Internet, thereby enabling the process to be done remotely.
• This type of link between geographical data and a location on a map or image is an example of a geo-reference.
• the vertical images in the database may be used to link data to all images. For example, using the geo-reference link to the address database, determine for each house its exact longitude and latitude. Locate and measure the horizontal size of each building or house from the vertical images, and calculate the height of each building or house shown in each oblique image in the multi-perspective image database in which it appears. These calculations are possible since all image parameters are known and have been stored (elevation profile data for the terrain may be required). Store the calculated size information in the address database. It is now possible to calculate both the approximate location and size of each building or house in each image, thereby enabling creation of so-called "hotspots" on each image and linkage of those hotspots to information in the address database.
• a "hotspot file” may be created for each image listing which structures are shown on which pixels of the image.
• the hotspot files may be stored with the database.
• a viewer may click on any building in any image to retrieve the available data for that building, or a viewer may search for houses which fit viewer-specified search parameters, and a search program may highlight houses in each image which fit the search parameters. Differing ranges of such search parameters may be highlighted in different ways (by color-coding, for example) to facilitate analysis of the data. For example, a viewer may define different colors for different price ranges for a house. After determining the color for each house, its corresponding hotspot in each image may be drawn in the appropriate color.
• the analysis of the property values may be shown in real time as the viewer interactively "flies" over the virtual city.
• the color-coded hotspots may be drawn on each image before it is displayed as part of an animation, thereby not affecting the "3D feel" of the animation.
• Lines indicating roads may be drawn on the vertical reference images, and used to graphically create a database of road identities and locations for each vertical image in the image database.
• Road identities and locations may alternatively be obtained from digitally-stored and/or CAD-generated maps.
• the position of a road may be calculated in each oblique image in the multi-perspective image database and this location overlaid on the images while moving through the image database, by highlighting the road in a certain color in each image, for example.
• the pixel location in each image for each latitude and longitude point is known or can be calculated.
• the image database may therefore be linked to a GPS receiver/card in a computer in a car.
• the system may update and display the location of the car in real time as the viewer/driver simultaneously "moves" through the multi-perspective image database of the city and drives around the city.
• EXAMPLE CITY DATABASE SIZE.
• the following is exemplary only and shall not be construed so as to limit the scope of the present invention.
• MB megabytes
• a medium-sized city such as the Eugene-Springfield (OR) area comprises roughly 200 square kilometers, requiring over 1,000,000 images occupying more than 150 gigabytes (GB) of storage space.
• GB gigabytes
• Image database size may be reduced by using a lower image resolution and/or by using 8-bit images, at the potential expense of degraded image quality. Compression algorithms may be used, at the potential expense of slower access to images in the database.
• many city image databases would probably not fit onto current CD-ROM media or DVD-formatted media.
• distribution of images over the Internet or over a computer network from a central storage medium is a preferred means for enabling viewer access to the image database, particularly since most viewers would only require access to a relatively small fraction of the total number of images available in the database.
• a traveler may enter an address or city and start flying over the city, checking out hotels, restaurants, places to visit, and other travel information;
• Public utilities may visually check the physical environment and existing structures and utility infrastructure when scheduling repair jobs or planning construction. Images, animations, and/or information from the database may be accessed from an office or from the field, and information may be added regarding work progress, repair reports, account activity, etc.;
• Delivery or parcel services may use the image database to help find addresses and choose optimal routing;
• Architects and/or city planners may use animations to get a feel for the existing buildings and environment, to superimpose designs onto images or animations, and/or use animations and images to determine location of certain objects;
• Emergency services and law enforcement personnel may check out a building or site in an animation from the dispatch room or from the emergency vehicle before arriving at the site.
• Databases for emergency call (i.e., 911 call) data, property ownership data, crime statistics, floor plans, ground photos, and/or hazardous materials data may be linked to the image database.
• Pre- fire, pre-raid, or pre-emergency plans may be formulated with the aid of images and animations from the database; 6.
• Insurance companies may view a property in an animation in the course of processing an insurance application or claim;
• Traffic planners may link the image database to traffic planning software, run analyses, and view the results in an animation
• Real estate professionals may view/display properties for rent or sale in animations, and may link the image database to databases containing information on such properties, such as floor plans, price, tax, and market data, and ground photos;
• Animations may be used during litigation or a trial to show crime scene or illustrate a sequence of events
• An image database may be linked to a demographic database and used to display results of analyses superimposed on images or animations;
• a vehicle may be linked to a positioning system and tracked in real time on an image or animation, either in the vehicle or at a remote location.
• OTHER USE EXAMPLES The following is exemplary only and shall not be construed so as to limit the scope of the present invention.
• a multi -perspective image database may have uses in other areas, including but not limited to:
• a multi-perspective image database may be generated for a planetary surface and used to create animations for virtual exploration and research; 2.
• a multi-perspective image database may be generated for an undersea area and used to create animations for virtual exploration and research;
• a multi -perspective image database may be used by geologists and/or cartographers.
• a multi-perspective image database may be generated for a microscopic surface and used to create animations for virtual exploration and research in such diverse areas as medical applications, microbiology, materials science, etc.
• the image database may comprise images recorded by any method of microscopy, including but not limited to optical microscopy, phase contrast microscopy, scanning tunneling microscopy, atomic force microscopy, near-field scanning microscopies, fluorescence microscopy, electron microscopy;
• a multi-perspective image database may be generated for management of natural and industrial resources, including but not limited to forests, f.arml.ands, watersheds, mining areas, oil and natural gas fields, pipelines, etc., and the image database may be linked to databases for inventory, growth, harvest, seasonal changes, etc., and used by government agencies, private industrial companies, or environmental activist organizations;
• a multi-perspective image database may be used to plan construction or expansion of highways; 7.
• a multi-perspective image database may be generated for a virtual world and used as a backdrop for computer gaming, allowing a player to play the game from any animated view of the virtual world.
• An array of stationary imaging assemblies may be installed around and above an arena, stadium, or sports facility and used to continuously record images during sporting or other events. Viewers may view the image sequences in real time to view the event live and may continuously and interactively change their view of the event while watching from a remote location (to "follow the play", for example). Earlier images may be reviewed, allowing "instant replay” of any portion of the event from any angle, with obvious implications for sports officiating, for example.
• X REF X CAM ⁇ Z CAM ⁇ an "cAM C0S ⁇ cAM Z CAM C0S ⁇ CAM CTR XcAM + tanfl CAM
• the corresponding image coordinates (x m y M ) may be calculated for a given image as: s-c-sm ⁇ C.AM / IM
• This algorithm may also be used to determine whether a given object detail appears in a given image. If calculated values of x M andy ⁇ , fall outside the range of the image in question, then the object detail does not appear in that image.
• an irregular target object surface may be approximately characterized by a two-dimensional array of object surface height values (i.e., an elevation profile).
• an elevation profile may be independently available, or may be generated within the scope of the present invention, as described hereinabove, by photogrammetry (using vertical reference images) or by direct measurement (range-finding or other equivalent methods).

Applications Claiming Priority (3)

Publications (2)

Family

ID=22082430

Family Applications (1)

Country Status (3)

Families Citing this family (3)

Citations (2)

Family Cites Families (6)

• 1998
  • 1998-12-07 WO PCT/US1998/025955 patent/WO1999033026A1/en not_active Application Discontinuation
  • 1998-12-07 AU AU17151/99A patent/AU1715199A/en not_active Abandoned
  • 1998-12-07 EP EP98961969A patent/EP1040450A4/de not_active Withdrawn

Patent Citations (2)

Non-Patent Citations (1)

Also Published As

Similar Documents

Legal Events