EP1195057A1

EP1195057A1 - Apparatus, media and method for capturing and processing spherical images

Info

Publication number: EP1195057A1
Application number: EP99918469A
Authority: EP
Inventors: Martin L. Bauer; Bruce Cole; Kimberly S. Evans; Craig Grantham; Laban P. Jackson; Christopher M. King; Sean Kitzmiller; Daniel P. Kuban; H. Lee Martin; Michael J. Tourville; Steven D. Zimmerman; James L. Hatmaker; Sean W. Mcginnis
Original assignee: Interactive Pictures Corp
Current assignee: Interactive Pictures Corp
Priority date: 1999-04-08
Filing date: 1999-04-08
Publication date: 2002-04-10
Also published as: WO2000062542A1; EP1195057A4; AU3638499A

Abstract

Computer software (150) and an associated digital camera system (100) for capturing digital wide angle such as hemispherical images, for processing two or more wide angle or hemispherical images to seam the wide angle or hemispherical images together to form a spherical image, for linking sound data files (105), location data (110), time and date data (100b) and horizontal reference data to the captured image data, for selecting for display (175) or for printing (165) undistorted portions of the spherical images (170), for posting the spherical images to a web site (160) or transmitting the spherical images via the Internet (155), for time and date stamping (100b) digital wide angle images as they are captured along with horizontal reference data, location data, for example, from the global positioning system (110), and environmental data.

Description

APPARATUS, MEDIA AND METHOD FOR CAPTURING AND PROCESSING SPHERICAL IMAGES BACKGROUND OF THE INVENTION

1. Technical -Field The present invention relates generally to the field of digital image capture and processing and, more particularly, to apparatus, media and a method for capturing digital wide angle such as hemispherical images, for processing two or more wide angle or hemispherical images to seam the wide angle or hemispherical images together to form a spherical image, for linking sound data files, location data, time and date data and horizontal reference data to the captured image data, for selecting for display or for printing undistorted portions of the spherical images, and for posting the spherical images to a web site or transmitting the spherical images via the Internet.

2. Description of the Related Arts

In US Patent No. 5,185,667, a hemispherical image processing system is described. The system comprises a camera, a fish eye lens and a digital processing system. The digital processing system operates on a data file containing data representing a distorted fish eye lens image and performs a transformation on the data to permit display of a perspective corrected image portion of the distorted fish eye image in real time responsive to pan, tilt, rotate and zoom inputs provided by a user. In US Patent No. 5,384,588 and 5,877,801, a wide angle image transmission system is described. For example, a camera equipped with a wide angle lens may capture images having a great field of view. Multiple remote users may input commands in real time in such a manner that each remote user may view different perspective corrected portions of the same hemispherical image. When a camera, for example, is capturing a single hemispherical image of a baseball game, one remote user may zoom in on the pitcher and view a perspective corrected image of the pitcher while another viewer may zoom in on the batter and view a perspective corrected image of the batter. Meanwhile, the camera has not moved as it captures the whole field of view of the baseball game. In US Patent 5,764,276, there is described a system of obtaining spherical views of an image as earlier described by US Patent 5,130,794. One may obtain a spherical image by capturing two hemispherical images. Two fisheye objective lenses such as the NIKON lens having a 220 degree field of view cooperate to capture a spherical image. The '276 patent further describes a camera positioning and rotator device for properly positioning a camera on a tripod and the digital processing of the spherical image to provide perspective corrected views in response to a user using a cursor for controlling pan, tilt, rotate and zoom.

Cameras are now digital and capture images in the form of digital data. Most cameras are not equipped with wide angle or fisheye lenses. Fish eye lenses, in particular, are very large and very expensive (such as the NIKON 220 degree lens described above). The Internet is now a convenient network for the inexpensive and popular transmission of digital data. The transmission of e-mail has become commonplace and its use is encouraged by camera manufacturers for the transmission of digital images so, for example, family members may view digital pictures taken minutes earlier at a wedding.

On the other hand, camera manufacturers do not currently supply inexpensive fish eye lenses. Complex image processing software is required for dewarping perspective corrected views in real time and for seaming wide angle images together.

Consequently, there remains in the art a need to decrease the costs and simplify the processes associated with permitting a photographer to take advantage of the wide angle image technology briefly described above. SUMMARY OF THE INVENTION

According to the principles of the present invention, a photographer may obtain a digital camera equipped with a normal field of view lens and a converter/adapter for such a lens that permits the photographer to capture a greater than 180 degree field of view with a single picture. Moreover, the photographer may utilize a rotating adapter for a tripod to which the digital camera, normal lens and fish eye lens converter/adapter are attached. The rotating tripod adapter mounts the camera in such a way as to support the focal or nodal point of the lens over the rotating axis of the adapter. The camera may turn and temporarily lock into positions exactly 180 degrees apart so as to capture opposing views as taught by pending US Application Serial No. 08/767,376 filed December 16, 1996 and entitled "Method and Mechanism for

Automatic Opposing Alignment of Photographic Image Capture. " The digital camera of the present invention is further provided with a digital clock for date and time stamping each wide angle image as it is captured, horizontal reference data sensors for sensing and recording pan, tilt and rotation reference data for each wide angle image as it is captured and location data sensing circuitry such as global positioning system data sensors for obtaining and recording location data such as latitude and longitude data for each wide angle image as it is captured for forwarding to image processing apparatus according to the present invention.

A wedding picture may be taken by asking the wedding party to gather around a digital camera equipped and mounted to the tripod as described above. One digital image is captured, the camera is then immediately rotated 180 degrees and a second digital image is captured. Pending US Application Serial No., filed concurrently herewith, (attorney docket no. 1096.79941), describes an imaging system in which both images are captured simultaneously and several wedding images captured in video sequence for viewing as home video virtual reality experiences. The hemispherical images captured in sequence are preferably time and date stamped and stored with horizontal reference data and camera location data temporarily in the digital camera for downloading to a digital image processor such as a personal computer on which is installed software according to the present invention. The hemispheres are then downloaded into memory of the personal computer on which has been preinstalled software from a media for performing camera system calibration, automatic seaming, previewing the seamed spherical images, associating background or introductory audio files with seamed images, posting images to a web site with downloadable viewer software so they may be viewed by a remote viewer, bundling the images with a viewer, for example, in an e-mail so a family member recipient of the e-mail may view them, and customizing the spherical image by permitting the user to edit text or otherwise customize a tripod cap portion of the image. Each step of the image process may be provided in a convenient screen comprising a mode map area, a work area and a navigation area for navigating through the mode. Also, the present invention involves a method and system for doing business involving the purchase of keys for storing of seamed images.

These and other features of the present invention will become better understood through the drawings and detailed description of the invention which follows. BRIEF DESCRIPTION OF THE DRAWINGS

Figure 1 is a system overview showing the individual elements of apparatus, media and method for capturing and processing spherical images including a wide angle digital image capture system, digital audio data files, global positioning system data, a data processing system such as a personal computer, and the various output capabilities including Internet e-mail, image posting to a web site, printing conventional still images via a printer (black and white or color) in a selected aspect ratio, displaying the images and storing the images in a photo album memory. Figure la provides a block schematic diagram of a digital camera imaging system according to the present invention.

Figure 2 show individual elements of the wide angle digital image capture system including a fisheye or wide angle camera lens adapter 220-1 (Figure 2e), a rotator 210 (Figure 2b), a tripod 200 (Figure 2a), a digital camera 215, 220 or 225 equipped with the fisheye lens converter/adapter 220-1 or a digital camera already equipped with a fisheye lens.

Figures 3a, 3b and 3c show in sequence the steps of taking opposing hemispherical images from a table or desk top wherein Figure 3a shows an assembled Olympus D340-L digital camera arrangement, the camera with a fisheye converter/adapter mounted to a rotator, and the rotator coupled to the tripod 200;

Figure 3b shows a camera rotated on rotator 210 to capture a first hemisphere; and Figure 3c shows the camera rotated 180 degrees to capture a second hemisphere.

Figure 4 is a screen print of a typical computer display screen for selecting a camera during software installation. Figure 5 provides an explanation of a calibration screen for calibrating a camera imaging system including a tripod and rotator.

Figure 6 provides one example of a preview screen for previewing a spherical image formed by seaming two hemispheres.

Figure 7 is a first screen of several screens for an image processing mode for processing digital image data, the first screen directed to collecting digital wide angle image data for one, two or more images. Figure 8 shows the result from Figure 7 of selecting one image and using that one image to form a 360 degree round image.

Figure 9 shows the more typical example where a 360 degree round image is to be formed from two opposing hemispheres. Figure 10 provides an overview of image downloading screens for downloading image data from digital camera memory, with and without TWAIN software.

Figure 11 is a preview IPIX image screen for selecting desired IPIX image format for preview and adding keys as may be necessary for storing IPIX images in the selected format. Figure 12 is an add key screen accessed from the preview IPIX screen of Figure

11.

Figure 13 is a sound screen for selecting background and introduction sound data files for association with the IPIX image currently being processed.

Figure 14 is a biographical information screen that may be completed by the user or photographer to, for example, name the IPIX image, associate key words with the image and identify the photographer.

Figure 15 is a tripod cap screen, the tripod cap being a wasted portion of an IPIX image which may be used to convey information or to link the IPIX image being processed to a URL. Figure 16 is a close-up view of a default tripod cap supplied with the image as viewed from the preview screen of Figure 6.

Figure 17 is a horizontal reference data screen whereby the user may manually enter pan tilt and rotate data associated with a particular image as referenced to true plumb or select horizontal (normal) or vertical camera orientation. Figure 18 is an image processing selection screen whereby a user may select one or more forms of image seam processing and at least one associated parameter.

Figure 19 is the camera configuration screen for selecting a camera configuration.

Figure 20 is another copy of the preview screen of Figure 6 useful for explaining the process of establishing zoom parameters and setting an initial IPIX image point of view. Figure 21 shows the screen for storing an IPIX image after it has been processed and identifying file names for each desired format.

Figure 22 is an example of a combined mode screen where all the screens of Figures 7-19 are combined into one screen which may be referred to herein as an expert mode screen.

Figure 23 is an example of a panorama print screen accessible from an IPIX image viewer. DETAILED DESCRIPTION OF THE DRAWINGS

Apparatus for capturing and processing spherical images according to the present invention is shown in Figure 1. A wide angle digital camera system 100 according to the present invention comprises the several elements shown in Figure 2 (digital camera, wide angle lens converter/adapter, tripod and camera mount and rotator) and may further including digital audio files 105 previously captured or recorded and stored simultaneous with the capture of a digital image and global positioning input data 110, preferably stored at the time of image capture.

It is also preferable to record data representing pan, tilt and rotate coordinates referred to herein of the digital camera and camera rotator of Figure 2 with respect to true plumb as horizontal reference data 100a at the time of image capture. Typically, the digital camera may be assumed to be pointed in a plane horizontal with the plane of the ground or the floor. However, such is not always the case when capturing an image. According to the present invention, such reference level measurements may be made manually, however, with improvements in digital cameras, such measurements may be automatically registered as digital data along with the image captured in memory. For example, a small gyroscope and sensor may be used to register the coordinates in relation to camera direction and location at the time of image capture and in each opposing direction of camera aim as would be known from the surveying, aviation and maritime arts.

Similarly, time of day and date clock data 100b may be preferably recorded simultaneously with image capture. Time of day and date may replace the use of image number to record image sequences and, of course, provides greater detail.

Today's digital camera systems may not incorporate a digital clock for time and date stamping of images as they are taken. Nevertheless, such time and date data is preferably recorded in conjunction with or auxiliary to image capture. So while today it may be necessary to manually record the time and date of image capture and continue to use an image sequence number as provided by conventional camera film, digital cameras of the future may simultaneously record time and date as each picture is taken with the digital image data files as time and date data.

Digital audio files 105 may be prerecorded, post-recorded or computer composed or generated audio for providing accompanying music or explanatory language or both (voice over music) for a captured digital image. The digital files may also comprise simultaneously recorded audio and a digital microphone and digital recording functionality may be built in to digital camera system 100. Preferably, a digital or electret microphone may record audio data and be mounted and directed in the same direction as a camera lens to capture the sounds coming from the direction of image capture as would be known from the movie film and television programming studio arts. For a spherical image according to the present invention, it is preferable to provide for a stereo sound file, recording both the forward and opposing camera directions as left and right stereo audio channels.

Likewise, global positioning system data 110 may be captured simultaneously with the image or by prerecording or post-recording the location data as is known from the surveying art. The object is to record the precise latitude and longitude global coordinates of each image as it is captured. Having such data, one can easily associate front and back hemispheres with one another for the same image set (especially when considered with time and date data). The path of image taking from one picture to the next can be permanently recorded and used, for example, to reconstruct a picture tour taken by a photographer when considered with the date and time of day stamps.

Other data may be automatically recorded in memory as well (not shown) including names of human subjects, brief description of the scene, temperature, humidity, wind velocity, altitude and other environmental factors. These auxiliary digital data files associated with each image captured would only be limited in type by the provision of appropriate sensing and/or measuring equipment and the access to digital memory at the time of image capture. One or more or all of these capabilities may be built into wide angle digital camera system 100.

As has been introduced in US Patents 5,764,276; 5,877,801 and US Patent Application Serial No. 08/767,376 filed December 16, 1996, the object of the image capture system 100 of the present invention is to capture two or more wide angle images, for example and preferably, two hemispherical images taken with a fish eye lens camera and seam the two hemispheres together to form a spherical environment. A hemisphere as described and used herein is an approximately 360 degrees round by 180 degree image (sometimes considered as 180 degrees by 180 degrees in latitude and longitude) that will contain distortion caused by the fish eye lens or lens system and, when viewed as substantially large flat image portions, will be perspectively incorrect.

Two hemispheres when digitally seamed together using software media 150 according to the present invention form a spherical environment. The user of software 150 may view perspectively correct smaller portions and zoom in on those portions from any direction as if the user were in the environment, causing a virtual reality experience. When a fish eye lens is used such as a NIKON fisheye lens, the field of view that is captured is 220 degrees, which is 40 degrees more than a hemisphere. There is considerable overlap in opposing images that are captured. The overage permits inaccuracy in someone's inadvertently not capturing perfectly opposing views. However, these 220 degree field of view lenses are very large and very expensive. According to the present invention, a far less expensive and smaller 185 degree lens converter/adapter may be used to adapt a normal flat image lens purchased with a digital camera to capture a 185 degree field of view image. However, better registration of opposing hemispheres is required. With 185 degrees field of view, there is only 5 degrees beyond 180 degrees of overlap with the opposing hemisphere. Consequently, the camera system of the present invention uses a tripod and rotator mount for the camera that permits almost perfectly opposing hemispheres to be captured.

Digital processing system 125 need not be a large computer. For example, the digital processor 125 may comprise an IBM/PC-compatible computer equipped with a Microsoft WINDOWS 95 or 98 or WINDOWS NT 4.0 or later operating system.

Preferably, the system 125 comprises a quad-speed or faster CD-ROM drive, although other media may be used such as Iomega ZIP discs or conventional floppy discs. An Apple Computer manufactured processing system 125 should have a MACINTOSH Operating System 7.5.5 or later operating system with QUICKTIME 3.0 software or later installed. The user should assure that there exists at least 100 megabits of free hard disk space for operation. An Intel Pentium 133 MHz or 603e PowerPC 180 MHz or faster processor is recommended so the spherical images, hereafter referred to as IPIX (TM) images where IP--X is a trademark or Interactive Pictures Corporation of Oak Ridge, Tennessee, may be seamed together and stored as quickly as possible. Also, a minimum of 32 megabits of random access memory is recommended. The personal computer should have a display 125-1 for adequately displaying high resolution IPIX spherical images. The display 125-1 or 175 should be capable of displaying 800 x 600 pixels or higher of resolution with at least 256 different colors. Processing system 125 should also comprise a keyboard input 125-3, a mouse and pad 125-4 for inputting data selections. Of course, voice recognition may be supported in future releases of the processing software stored on media 150. Processor housing 125-

2 comprises hard disc memory, cd-rom and other media drives, random access memory and processors as is well known in the art.

Image processing software is typically produced as software media 150 and sold for loading on digital signal processing system 125. Once the software according to the present invention is properly installed, a user may load the digital memory of processing system 125 with digital image data from digital camera system 100, digital audio files 105 and global positioning data and all other data described above as desired and utilize the software to seam each two hemisphere set of digital images together to form IPIX images. Once the images are seamed and the user is pleased with the result, the user may output the IPIX images in various ways as will be further described herein. The IPIX images may be transmitted with a viewer as Internet e-mail 155 to addressed terminals where the e-mails may be opened and enjoyed by the recipients. The IPIX images may be posted on the user's web site 160. IPIX image portions that have been personally selected for content by the user may be transmitted to a preferably color printer for copying. (One family wedding IPIX image may yield a virtually unlimited collection of conventional flat still images of the wedding.)

The viewer may store the IPIX image in a database 180, for example, as a data folder collection of files called wedding. Also, the viewer may immediately or at any time view the IPIX image portions of his choice by panning, rotating, tilting and zooming to the image of choice and displaying the image on display 125-1 or display

175. Other outputs of the image data may come to mind of one of ordinary skill in the art.

Figure la provides a block schematic diagram of a digital camera imaging system according to the present invention. A fisheye lens adapter 220-1 converts a typical digital camera lens 220-5 to a wide angle lens. In many instances the adapter

220-1 may slidably fit and be friction attached to lens 220-5. Digital camera body 225 may contain the elements 100a, 250, 255, 260, 265, 270, 275, 280 and 285 comprising a block schematic diagram of electronic digital circuitry. Some of the elements may be externally mounted but operate in synchronization with the image capture operation as will be described further below. Controller 250 typically comprises a small microprocessor or application specific integrated circuit. Programmable read only memory, array memory or other program memory houses the programs that operate the camera. Clock 260 is typically a digital clock that clocks the microprocessor operation. According to the present invention and possibly coupled with appropriate computer programs of program memory 270, clock 260 provides for time and date stamping of digital images as they are captured. Shutter 265 operates to permit an image to be captured via lens system 220-1, 220-5 and transmitted to digital image memory 275 which also stores other concurrently stored data such as the time and date stamp data. In accordance with the present invention, horizontal reference data 100a is sensed by well known sensors in the art and also captured for each image that represents the camera's pan, tilt and rotation angles with respect to predetermined tripod reference points 180 degrees apart and true vertical or plumb where pan is look around, tilt is up and down and rotate is turning the camera sideways. Location data such as latitude and longitude for each image is captured and stored for each image by location data sensor 255 which may be a global positioning system. Interfaces 280 and

285 are provided for interfacing the captured image and other data with a computer system 125.

In operation, when the user operates the shutter or preferably sets a timer so that the user may get in the picture and the timer expires, several operations occur simultaneously: time and date stamp, location data stamp, and horizontal reference data stamping occur simultaneously with image capture. In an embodiment not shown, digital camera 100 may be equipped with a means of entering biographical information as well as will be further described herein. Moreover, the digital camera may be equipped with at least one microphone for recording a simultaneous audio clip with the digital image of predetermined duration. Referring to Figure 2, there are shown the collection of elements comprising a typical wide angle digital camera system 100 according to the present invention. Figure 2a shows a tripod 200 according to the present invention. Ideally, a digital camera 225 should be mounted on a monopod, a one legged camera mount. Using a monopod reduces the size of a typically undesirable "footprint'' made when capturing two images, a tripod cap image portion. An IPIX spherical image captures the sky, the full surroundings but, because of the tripod, leaves a small circular area at the bottom of an image that comprises the tripod 200 and not the floor or ground which might be a more desirable part of the image than the tripod 200. (The monopod or tripod appearance at the bottom of an image, the small wasted portion of the IPK image, will be hereinafter be referred to as the tripod cap.)

Tripod 200 comprises conventional elements including three tripod legs, 201a, 201b and 201c. These are typically extendable. It is desirable in shooting a normal landscape and to simulate a true virtual reality experience to capture an image at the same level a person would capture the image at eye level. So the legs preferably extend to approximately five feet. Extendable height adjustment 202a in its non- extended position may preferably comprise a suction cup 202a or other device, for securely fastening the tripod on table tops or to level surfaces when the surface is not level. Fastener 204 such as a screw may be used to hold the adjustment 202 in place. Tripod cap 203 is adapted to receive a rotator assembly 210 according to Figure 2b. Referring to Fig. 2b, there is shown a rotator 210 according to the present invention. The base 210a of rotator 210 is fixed to tripod cap 203 by a fastener that may be a screw mount, bayonet mount or other fastener known in the art. In a preferred embodiment, the rotator 210 attaches to tripod 200 through screw heads of rotator base 210a. The rotator 210 is held by the rotator base 210a, turned clockwise and may spin slightly as it is screwed securely to the tripod 200. Once the rotator 210 is attached to the tripod 200, the rotator arm 210b should be turned until it clicks softly into a detente position. Base 210a and rotating arm 210b are permitted to rotate one with respect to the other to two predetermined detente positions which are, in a preferred mode, exactly 180 degrees apart. A camera, for example, camera 215, 220 or 225, are mounted at mounting point 210c which is a predetermined distance d from the axis of rotation of rotator 210. Referring briefly to Figures 2f and 3a to 3c, it can be seen how the camera lens is positioned directly over the rotator arm 210b and mounted so as to point along its length. Moreover, the focal node of the lens is mounted directly over the axis of rotation. Since cameras may vary as to the distance between the camera body and the lens, mount 210c may be slidably adjustable in the lengthwise direction of the rotator arm 210b to permit the lens to be mounted over the axis of rotation. The camera body is attached to the rotator arm 210b using, for example, a thumb screw. The camera should be snug along a back stop. It is important, in order to obtain quality IPIX images, to be sure that the tripod and rotator are firmly assembled together with the digital camera and are fixed and do not move with relation to the surface on which they rest, as one hemisphere is captured and then the second hemisphere is captured. For wide angle lens cameras, there would be more detente positions and more image seaming as necessary to capture a full 360 by 360 image if desired. For example, three detente positions may be 120 degrees apart and four detente positions may be 90 degrees apart. Stereoscopic IPIX image capture is described by Patent Cooperation Treaty application PCT/US98/21304 entitled "Method for Creation and Interactive Viewing of Totally Immersive Stereoscopic Images."

Typical digital cameras 215, 220 and 225 are shown in Figures 2c, 2d and 2f respectively. Camera 215 is an Olympus D-340L with a fisheye lens converter 215-1, 215-2 shown mounted over the conventional Olympus lens beneath. Mounting bracket 215-2 must be used to mount the fisheye lens converter 215-1 to the camera body.

With the Kodak DC200 camera 220 shown in Figure 2d, the fisheye lens converter 220-1 slidably mounts over the existing lens and is held in place by friction. Figure 2f shows a NIKON Coolpix 900s with the NIKON fisheye lens converter mounted over the NIKON lens. The camera is shown already mounted to a rotator 210. Other cameras may be used to advantage as well so long as a fisheye or wide angle lens converter may be adaptably fit to the camera or the fisheye or wide angle lenses are available for the camera. The digital cameras ideally capture digital images with as high a resolution as possible. Resolution is typically quantified in terms of a pixel array such as 600 by 800. The larger the number of pixels in each of horizontal and vertical dimensions, the better the resolution and the better quality of IPIX image that one can zoom into. In accordance with US Application serial no. 08/835,210, entitled

"Method and Apparatus for Providing a High Resolution Image to Produce a Realistic Immersion Experience," as one zooms into a portion of an image such as a painting on a wall, a high resolution flat still image may be replaced for the perspective corrected portion of the image zoomed into. Figure 2e shows a 185 degree field of view fisheye lens adapter 220-1 and carton 220-2 manufactured by FIT Corporation of Tokyo, Japan. The lens converter converts the optics of a standard lens to be able to capture a full 185 degree field of view, permitting overlapping hemispherical images to be captured for seaming together as an IPIX spherical image. Now the practice of capturing hemispherical images will be discussed with reference to Figures 3a, 3b and 3c. An IPIX spherical image is an interactive, spherical image with a complete field of view, from earth to sky, floor to ceiling, horizon to horizon. An IPIX image is created from two opposing photographs captured by a fisheye lens. The present invention remaps the images, seaming them into an immersive whole. IP-X images can be linked together, linked to traditional photographs and rendering, enhanced with audio or other file types and incorporated into other applications as will be further described herein. IPIX images can be applied in a variety of ways including but not limited to virtual world creation, computer-based training, real estate sales and marketing, museum tours and travel logs. As will be described herein, IPIX images may be posted on a World Wide Web site for viewing by industry-standard web browsers or with an embedded IPIX viewer, e-mailed for remote viewing by a local -DP-X viewer, imported into Adobe Photoshop software applications for enhancement or touch-up, embedded into a Microsoft PowerPoint or MacroMedia Director and Authorware presentations or viewed from an Microsoft Visual Basic, Visual C or Visual C++ application program. Once two hemispheres are captured to form an IPIX spherical image, software media 150 is run on a processor 125 to seam the hemispheres together, color match the images to adjust for different exposures in the opposing views, blend the seam to create a single image out of the two hemispheres, save the IPK image on one or more sizes and resolutions, configure the IPIX image with sound files and brand the IPIX with personal information or environmental information through the design of a tripod cap

(the bottom of the image).

An IPIX image has two hemispheres, front and back. It is preferable to shoot consecutive image pairs without moving the tripod. Composing an IPIX image involves two major decisions; the image's point of view and where a seam will fall. The point of view should be decided by deciding where to place the camera and tripod/rotator. IPIX images place the viewer in the center of a picture and can make objects appear farther away than they really are.

The seam location can be chosen to avoid human subjects or the work of art center of an image. Desirably the seam should fall in an area of not much interest. For example, the camera is taking the picture of the family at dinner with the camera mounted in the center of the table. Place the camera so it takes half the people in one image (with no one at the seam) and half the people in the other image. The scene should be free of unwanted objects. It is preferable to use a high quality mode if the camera provides for it and to disable the flash. Most cameras have a timer to give the user a few seconds to either be completely in or out of the picture as desired and avoids camera movement (since no one is pushing on a shutter button).

After taking one hemisphere, the camera is rotated to the opposite position. Referring to Figure 3a, the camera 215 is shown mounted on the tripod/rotator 200. The suction cup 202a may be used to hold the tripod to the table top. Figure 3b shows the camera tripod 200 and rotator 210 rotated to a first detente position facing right.

Then one rotates the rotator in direction 310 (or the opposite direction) to obtain the opposite position shown in Figure 3c facing left.

IPIX images can be taken without a tripod or rotator but are susceptible to blurring from camera movement, or there may be problems with aligning the two hemispheres if there is insufficient overlap. One way is to mark a spot on the ground and an arrow so the camera user can try to emulate a tripod/rotator as closely as possible, holding the camera level in both directions.

Now installation of software media 150 will be briefly described followed by calibration of the software with the camera of the user's choice. Thereafter, the features of the method and media will be further described in connection with a discussion of the operation of the software on a pair of hemispheres.

Installation of the image processing software of the present invention, hereinafter referred to as IPIX Wizard software, involves well known software media 150 installation steps. One inserts the IPIX Wizard CD-ROM into the CD-ROM drive of data processing system 125 and a Setup program should start automatically. One may click on "next" or "agree" through a succession of Welcome and Readme screens.

One finally arrives at a destination screen wherein the user must identify a location in memory for the software to reside.

Once a destination is selected, the screens according to Figure 4 appear on display 125-1. Here the user chooses the type of camera the user will be using with the software. Preferably, a choice is required before the user may "continue. " After a choice of camera is made, the IPIX Wizard files and IPIX Viewer files may be copied to the hard drive 125-2. If an Olympus camera was selected, TWAIN drivers will be installed by a secondary installation program. TWAIN drivers and their importance to receiving digital image files will be explained further herein. Further Readme and registration screens may follow as is well known in the art.

Camera calibration requires the input of data files for two hemispheres captured using the camera selected during installation of the software media 150 for the IPIX Wizard software. Figure 5 shows a calibration screen for leading a user through the calibration process. The calibration process calculates the seam location from two images taken with the camera system including rotator and tripod the user wishes to calibrate. IPIX Wizard software can contain default configurations for each of the camera types supported by the software. However, when the user calibrates their camera with IPIX Wizard software, a new calibrated camera configuration file is generated. The camera calibration file appears in Figure 19 of the IPIX Wizard mode and Figure 22 of the Expert mode as will be described further herein. Camera calibration can be accessed in one of two ways: from the Preferences screen or when IPIX Wizard software prompts the user to calibrate. As already indicated, it is necessary to download hemispheres from the camera to the computer. The camera manual will have instructions for attaching most cameras to a computer for downloading. On an Open Hemispheres screen one can click to locate the images (it does not matter which image is selected first) and these are entered into the calibration screen of Figure 5 as images 501 and 502. Box 503 shows the user's camera type and configuration. There are two choices for calibration: quick and thorough per box 504. It is recommended that the user try a quick calibration and, if the result is acceptable, it is likely that the thorough could not have done any better. The quick calibration may be as much as twenty-five times faster than the thorough calibration. A time of calibration screen may appear showing an estimate of the amount of time remaining in the calibration.

The program first makes an initial guess at the center of each hemisphere to locate the border. Then the program places one image border in a linear or circular buffer and attempts to match the image border of the other hemisphere to it. When there is an optimum match, then the seaming is complete. For example, a first pixel at the center of the initial center image guess is checked and then several new neighbors checked in either the x or y coordinate system direction to see if an optimum has been reached yet. In one embodiment a circle find algorithm may improve the guess made of the center of a circle and other embodiments, two linear buffers may be used or circular buffers with skips for unimportant portions of the image (such as the tripod cap portions).

Once calibration is complete, one can preview the image formed by the seaming process. If the results are unacceptable, one may use the thorough calibration selection. A typical preview screen is shown in Figure 6. The image seam is shown at the arrow in the previewed perspectively correct view intentionally selected at the border between two hemispheres 602 and 603.

Now the operation of IPIX Wizard software will be further explained with reference to the several screens of Figures 7-22. Figures 7-21 represent the IPI-X Wizard mode and Figure 22 represents all the IPIX Wizard mode screens in one and is referred to herein as the Expert mode. Once a user becomes knowledgeable of the processes described below for the IPIX Wizard mode, then, the user may use the Expert mode screen of Figure 22. The first screen of the IPIX Wizard mode is an open hemispheres screen according to Figures 7-10 which involve opening one or more hemispheres to be seamed. There exist three portions of this screen. An IPIX Wizard map shows the particular screen of the IPIX Wizard mode that is currently displayed or active. By clicking on one or more of the choices, one may access screens of the IPIX Wizard mode out of sequence as desired. The Open Hemispheres screen is presently highlighted in Figure 7. There is also a work area 702 and a navigation area 703. The work area is now used to input two hemispheres for operation. Portion 704 permits the choice of a single hemisphere or two hemispheres. A single hemisphere can be seamed to its reverse image for a visual effect if desired as taught by US Application Serial No. 08/516,629 filed August 18, 1995. This option is shown in Figure 8 as "Mirror this hemisphere" choice 801. On the other hand, for a true virtual reality experience, two hemispheres should be selected and loaded per Figure 9. In Figure 9, the user can click on double arrow button 905 to swap front and back images if appropriate. The file names are shown in the boxes below the respective front and back hemisphere images. "No thumbnail" in Figure 7 represents that no image has been selected yet. One can browse to locate the image via browse buttons 903 and 904 or type the file name into the respective boxes 901 or 902. The "Acquire" button 906 is used to acquire images from your camera, provided your camera has a TWAIN driver. TWAIN stands for Technology (or Toolkit) without an interesting name. A TWAIN driver is a piece of software that interacts with a scanner or digital camera to allow computer 125 to communicate with the device. If the camera does not have a TWAIN driver, then direct camera access cannot be obtained; the user may download the images before starting the IPIX Wizard mode. Figure 10 shows what will happen if the user clicks on "acquire" and there is or is not a TWAIN driver installed. If the TWAIN exists for the camera, for example, the Kodak DC 200, then images may be directly selected.

Referring to Figure 11, Preview IPIX is highlighted in the map area 701. There are four choices provided for image preview - high resolution, low resolution, JAVA and e-mail. (JAVA is a registered trademark of Sun Microsystems, Inc. of

Mountain View, California). The high resolution choice saves an IPIX image in a 1024 x 1024 pixel file. The low resolution choice saves the image as 512 x 512 pixels. The JAVA option saves a 1400 x 700 pixel image.

The JAVA format may be posted to a web site with a runtime downloadable JAVA viewer applet. The difference between JAVA applet viewer software and browser software such as a Netscape or Microsoft Explorer (ActiveX) plug-in viewer is that the JAVA applet is typically smaller and simpler. Also, JAVA-capable browsers such as Internet Explorer and Netscape Navigator download the applet with the web page upon which it resides, much the same way as the browser downloads the other elements of a web page. The JAVA applet simplifies the user experience by eUminating the need for the user to download and install a plug-in with their browser.

The e-mail option saves an IPIX image as an executable file bundled with a viewer. The result does not need to be e-mailed to someone but can be used to create local presentations. If transmitted as an e-mail, the IPIX image can be viewed on a PC running Windows 95/98 and NT or by an Apple MACINTOSH OS computer. Other formats in other resolutions are also possible and may come to mind of one of ordinary skill in the art and may be preferred in some applications. IPIX image files have a .IPX extension by convention and the e-mail IPIX image file has a .EXE extension. Add keys button 1101 is also shown. Figure 12 relates to the on-line purchase of keys for IPIX images. If the user needs keys (needed to save an IPIX image), the user uses this screen to purchase keys. A key is not required to run the IPIX Wizard image build process, but a key must be available at the end of the process to save the seamed IPIX image to memory.

Selection of the Add Keys button 1101 on the Preview IPIX screen (Figure 11) will install new keys or verify current IPIX image key count. Figure 12 shows the Add

Keys screen. A site ID code 1201 may be unique to each computer installed with IPIX Wizard software. The code 1201 is required to purchase additional keys. For example, the site code 1201 may comprise twelve characters, ten unique to the site and two for uniquely identifying the software. IPIX keys field 1202 (shown as 0 keys remaining) shows the number of remaining keys from those already purchased and remaining. IPIX software typically may be purchased with a predetermined number of keys. When new keys are purchased, a 30 digit code 1203 appears in the screen that may be used to activate the new keys that have been downloaded. The user enters the code and clicks on ok to activate the keys. A user may also redeem key certificates at an IPIX web site store by entering the certificate number at the web site. Figure 13 relates to the addition or association of sound files to an IPIX image.

In the Wizard map area 701, Sound is highlighted. In the depicted screen, area 1301 permits the selection of an audio background to the IPIX image. This may, for example, be a prestored MIDI (musical instrument digital interface) file or other file that has been computer composed or generated (for example, MIDI) or recorded contemporaneous with the image capture or before or after image capture. It may be concert music or other audio data background to the playing of the image. Area 1302 permits the selection of a sound file to be played as an introduction to the IPIX image. A user might want to prerecord or voice synthesize a description of the image before it is shown to a viewer, for example, a VOX or WAV file. WAV file support is built into Windows 95 and VOX files are compressed audio data files (optimized for speech) and downloaded easily.

Figure 14 relates to the customization of an IPIX image by adding bio or other personal information. Bio Information is highlighted in the map area 701. Also, Bio 1400 is one of four tabs that may be selected for customizing an IPIX image. There are three ways shown for adding personal bio information. The user may name the

IPIX image in area 1401, identify key words in area 1402 and name the photographer in area 1402. The latter area may be utilized to generate along with time and date data a copyright notice protecting the IPIX image. Other ways may come to mind of a user. For example, one may manually input time and date data, location data, environmental data (temperature, humidity and so on) as described above.

Figure 15 is entitled tripod cap in map area 701. Also, tripod cap is selectable as tab 1501. Tripod cap relates to that portion of an IPIX spherical image taken up by the tripod, typically at the bottom of an IPIX image. One may create their own tripod cap. For example, one may draw a square and use a program such as MS Paint software to design a custom cap to be saved as a .bmp or TIFF file. Using the browse button, the user selects the customized cap they have created. Also, a user may place a link on their tripod cap that, when actuated, takes control of the web browsers to the specified hotspot at URL link area 1502. In alternative embodiments, the link may be to any file type, for example, to link audio to the image or "hotspot" from one IP--X image to another IPIX image in an image sequence. Figure 16 shows a close-up of the tripod cap provided automatically in an IPIX image. The default tripod cap may be, for example, 350 x 350 pixels.

Figure 17 relates to Ref pan/tilt rotate in map area 701 and Ref PTR in the customizing tabs 1701. The choices include Horizontal, Vertical and User defined. The typical tripod arrangement will be horizontal. That is the camera is set up vertically to shoot horizontally. If the camera points at the sky for a first hemisphere and the ground for the second hemisphere, then, vertical orientation should be shown. If the user is not on even ground, then the user may input the coordinates for pan, tilt and rotate. It is also suggested according to the present invention to embed a sensor in the body of a camera for automatically recording pan, tilt and rotate measurements in relation to each hemisphere captured. The sensor may be taken from the well known aviation or maritime arts.

Figure 18 relates to image options in map area 701 or image processing 1801 in the customizing tabs section. There exist four choices: blend seam and a selectable range of pixel counts, color balance hemispheres, image smoothing and adjust image alignment. When an IPIX image is created, the two hemispheres' seams are blended together. The user may increase or decrease the number of pixels being blended together. Increasing or decreasing may improve the final image quality. A default value may be 15 pixels with reference to the overlap of five degrees if the lens as a 185 degree field of view. Seam blending finds pixels which are assumed to match in the overlap area between the two hemispheres. It blends the two hemispheres by taking a weighted average of the matching pixels. The weighting is applied as a function of the distance from the seam or edge of the hemisphere in towards the selected pixel value, for example, 15 pixels.

Image smoothing is really interpolated scaling. For the various available formats, the source hemispheres are scaled either in a fast pixel picking interpolative mode or a slower pseudo-bilinear interpolative mode known in the art. Also, a color balancing routine may be run to smooth the relative color exposures between front and back images. The color balancing routine can eliminate large color shifts between hemispheres. The color balancing algorithm finds pixels that are assumed to match in the overlap area between the two hemispheres. It averages the difference in each of the three color components (red, green and blue) between the matching pixels. It then scales the colors in each hemisphere in such a manner as to eliminate the perceived difference between the matching pixels. The result of color balancing is compensation for mismatched exposures between hemispheres.

Omitting these steps will result in faster seaming time but decrease image quality. Image smoothing can provide a smoother IPIX image but may double the creation time. If the user wants to calibrate the hemispheres before building the IPIX image from them, the calibrate routine may be called by selecting "Adjust image alignment."

Figure 19 shows the camera configuration screen. The camera configuration is identified in box 1901.

Figure 20 shows a typical image preview screen that is displayed of an IPIX image after the steps above have been performed of seaming, blending, balancing and the like. One may pan, tilt, rotate and zoom within the image but to save the image to a file requires a key. Within this screen, the user may set minimum and maximum zoom (magnification levels) values from predetermined parameters, change the initial viewpoint (the view that is seen when the IPIX is first opened), save the IPDC image and reset the image to a default. To change a viewpoint, the user pans or tilts around the image to find a viewpoint and selects the Set View button 2001. At any time the user may return to the initial view by clicking "go to initial view" 2002 or right clicking with the mouse. The Set Zoom Min 2003 and Set Zoom Max 2004 buttons are used to change the minimum and maximum magnification values from predetermined zoom values to new minimum and maximum limits. Go to Zoom buttons 2005 and 2006 allow return to the predetermined zoom limits. Finally, reset to default button 2007 allows the user to reset everything to default values.

Figure 21 allows one to save an IPIX image in one of many formats. The first major area is the save area 2100 used to identify and store the IPIX image file. One can check any one or more of several formats: Initial view thumbnail 2101, hemisphere thumbnail 2102, high resolution IPIX 2103, low resolution IPIX 2104, JAVA IPIX 2105, e-mail IPIX 2106 and e-mail IPIX 2107 with spin (pan and tilt). One or more of these boxes may be selected for the price of one key. Thumbnails do not cost a key. Cancel allows the user not to save the IPIX image that was created.

Once many IPIX images have been saved, an album (folder) can be created in memory of an event or a vacation or whatever. G.P.S. location data and time and date stamps may be used to sequentially tell a story as appropriate accompanied by music and introductory audio. Each IPIX image may have an infinite number of different views. For example, a wedding picture of A and B may be taken comprising a single

IPIX image at the church. A print at printer 165 may select the married couple or the whole wedding party including bride's maids and ushers. Other views may be selected of some of the guests in different groupings. A high quality color printer 165 is preferred and produces as many flat still images 170 as the viewer wishes from the same IPIX.

The print process will be further described with reference to Figure 23. A user accesses a view and pans, tilts, rotates and zooms to a desired perspective corrected view. A parameters pull-down menu may be used to select an aspect ratio and a print size such as 8" by 10" or other size (such as is known in the art). By aspect ratio is intended herein the ratio of image width to image height of a flat image. For example, a typical aspect ratio may be 1 to 1, 4 by 3, or 5 by 3. A panoramic aspect ratio may have a higher aspect ratio than 5 by 3. By size is intended the print size of the image such as eight inches by ten inches.

In accordance with the present invention, a warning screen may be used to warn a user that the dots per inch resolution of the printer may not provide a presentable image if too large a size is selected. For example, if the printer resolution is only 100 dots per inch and an eight inch by ten inch image size is selected, the image will not be clear and the warning message may be indicated.

Warning messages may also be provided if too large of the surface area of a sphere has been selected for printing. In mapping a sphere to a cylinder and all the vertical dimension is selected, the upper and lower portions of an image will be stretched. As aspect ratio changes and image size increases to capture more of the vertical or horizontal, and more of the sphere is intentionally captured for printing, the resulting view will not be perspective corrected in the sense that a human viewer would not believe that the image is perspectively correct. For example, if one attempts to print a spherical view of the earth, the resulting view of the sphere mapped to a cylinder would show the North and South Poles as exaggerated in size (a typical equirectangular image mapping). With reference to the Earth, one would be able to select approximately 45 degrees above and below the equator (90 degrees total) without stretching the resulting image too greatly at the top and bottom. The resulting printed image would appear perceptively correct to a human. Consequently, an error message may be used to indicate that a selected image for printing will not be perspectively correct to a viewer if printed when the selected vertical range is above 90 degrees.

In one embodiment, the user pulls down a menu including Print from clicking on the File button 2301 of Figure 23. The Print menu may include, for example, a panorama choice, a fit the page choice (the largest possible image) and a small size choice (such as a standard photographic size like three inches by five inches which also represents a five by three aspect ratio).

Figure 23 provides an example of a screen which one would see if one selects panorama viewing from an IPIX viewer software screen. When a user arrives at this screen they may pan, tilt, rotate and zoom through the IPIX spherical image to select a central most image section for panoramic printing. Thus, the screen of Figure 23 may depict in the background the selected center of the panoramic image. A 360 degree image is generated in the horizontal dimension for the printed panorama but a range of values for field of view 2302 from 60 degrees to 130 degrees is provided for selection in the vertical dimension. As one changes the selected field of view value, the screen may contain in a window (not shown) a preview of the selected panorama image (not shown).

Many cameras today permit the capture of a panoramic or wide angle image having, for example, a variable aspect ratio in the horizontal dimension. While a 360 degree image is generated in the present, depicted embodiment, the user in another embodiment may select the horizontal dimension as well. The horizontal dimension may be selected from viewing and cropping a preview, printable image or by selecting a horizontal dimension parameter by selecting degrees (for example, 240 degrees), by aspect ratio or by image print size, for example, three inches by eight inches, or combinations of these approaches. Also, IPIX images may be operated upon, for example, to extract a selected perspectively correct or other portion and save the file so that it may be edited or otherwise processed for display or otherwise (add computer generated graphics, used as a background set for live or cartoon characters and the like). The saved file may be retrieved by a second software package for computer graphics generation and addition to the image and the like or such applications may be included in the IPIX Wizard software.

Finally, a sequence of IPIX images may be linked and stored to form a conventional video file, for example, by storing an expanding file over time as a viewer pans, tilts, rotates and zooms their way through a single IPIX image file or one that has been linked by hotspots to other IPIX image files. A tour may comprise a collection of IPIX images and audio files in sequence, may be generated from a single

IPIX image or may be collected over time as a real time 360 degree video experience.

Thus there has been described apparatus, media and a method for capturing digital wide angle such as hemispherical images, for processing two or more wide angle or hemispherical images to seam the wide angle or hemispherical images together to form a spherical image, for linking sound data files, location data, time and date data and horizontal reference data to the captured image data, and for selecting for display or for printing undistorted portions of the spherical images. Posting the spherical images to a web site or transmitting the spherical images via the Internet are further described by an appendix which assumes the reader is familiar with HTML

(Hypertext Markup Language). All patents and patent applications referenced herein should be deemed to be incorporated by reference as to their entire contents. The following United States patent applications are related by subject matter to certain aspects of the present invention: US Patent Application serial no.'s 08/373,434 filed January 17, 1995; 08/516,629 filed August 18, 1995; 08/835,210 filed April 7, 1997; 08/863,584 filed May 27, 1997; and 09/093,337 filed June 9, 1998. Any and all

United States patents or patent applications referenced herein should be deemed to be incorporated by reference as to their entire contents. Further variations and modifications of the present invention may come to mind of one of ordinary skill in the art through an appreciation of the above description of the invention which should not be deemed to be limited by the scope of the claims which follow.

APPENDIX SETTING UP A WEB SITE

To post IPIX images to the World Wide Web, a user must create a Web page with special HTML code, and, when using an IPIX Plug-in software module, configure a server MIME type. This section assumes the user has a basic understanding of the Hypertext Markup Language (HTML).

MIME TYPES

When IPIX image files are used on Web pages, the approximate MIME type must be configured on the Web Server. On the local computer, the MIME type is configured when the IPIX Plug-in software module is installed. The MIME type must also be set on the hosting server (a one time configuration) IPIX images will not be displayed properly. What are MIME Types?

IPIX image files present a new file type to systems that have not run them before. Whenever a new file type is introduced, the operating system must be instructed how to handle it and what application is used process it.

MIME types are used to help define the new file type and application software used to process it. This applies to both Web browsers and Web servers. IPIX Image MIME Types On Web Servers

When a Web server is not properly configured for the IPIX IMAGE MIME type, users experience difficulties in downloading and viewing IPIX image files. When the .IPX extension is detected but is not on the Web server MIME type list, a default application type and subtype will be employed. The browser will attempt to assign an application to open the IPIX image file. Because no default application can open an IPIX image file, the user will not be able to view the IPIX image on their browser.

Configuring a MIME type for a Web server is usually not difficult. • The server maintains a file indicating the types of documents that correspond to various file extensions. The name of this file differs with each Web server. • When the server gets a request for a URL leading to a file which has one of these extensions, it first sends a single line to the browser which states the major and minor MIME types.

The Webmaster or Internet Service Provider will be able to set up the MIME types on the server hosting the user's Web pages.

The MIME types that should be added to the Web server are:

To establish the MIME types:

• Install the IPIX Plug-in software module on the user's computer to establish the MIME types on the local machine.

• Make sure the computers hosting the user's Web pages have the proper MIME types set. Again, ask the Web master or Internet Service Provider (ISP) to add the appropriate MIME types to all of their servers.

If an IPIX image displays in Internet Explorer, but not in Netscape (via Netscape plug-in viewer software), the MIME types are probably not set up properly.

THE IPIX PLUG-IN AND VIEWERS

Interactive Pictures Corporation has developed automated and semi-automated viewer downloading and installation. The degree of automation depends upon browser type and operating system. There should exist both a visible link to the IPIX Plug-in software module download, and using an embedded object tag with the IPIX image on the Web page.

These tags are not necessary if the user is using JAVA IPIX images. JAVA IPIX images contain a built-in IPIX Viewer, and do not require the IPIX Plug- in. (JAVA is a registered trademark of Sun Microsystems of Mountain View, CA)

Linking to the IPIX Plug-in Download

To aid site visitors without an IPIX Plug-in, provide a link to the Interactive Pictures Corporation's Plug-in download area: http://www.ipix.com/cgi- bin/download.cgi. One option is to use an IPIX (Interactive Pictures Corporation) logo as a graphic linked to the download page.

• With clickable text:

< A HREF="http://www.ipix.com/cgi-bin/download.cgi" > Click here. </A>

• With a clickable graphic:

< A -BKEF="http://www.ipix.com/cgi-bin/download.cgi" > <BMG SRC="images/ipixlogo.gif" BORDER=0 ALIGN = "MIDDLE" > < / A >

Embed Object Tag for Plug-ins Page

In addition to providing a link to the download page, embedding the IPIX image with an object tag can help users obtain the correct Plug-in. This code sample will open a link to the Plug-in page for people who do not have the IPIX Plug-in installed.

• The first three lines of code are ActiveX commands (where ActiveX is the name of the plug-in viewer software) for Microsoft Internet Explorer that identify the type of Object (IpixXl) the Class and version number, and the location of the IPIX Plug-in software module to be installed (Codebase).

• PARAM NAME provides the ActiveX component with the IPIX image file name.

• The Last four lines associate the file type with IPIX file, and the location of the IPIX Plug-in page if the application associate with x-ipix is not found.

<OBJECT ID="IpixXr WIDTH=300 HEIGHT=400 CLASSID="C-LSD: 11260943-42 IB- 11D0-8EAC-0000C07D88CF"

CODEBASE="http://www.ipix.com/viewers/ipixx.cab#version=2,2,0, >

< !- For MSIE 3,4+ - >

< !~ For Netscape 3,4 - >

< EMBED SRC ="yourimage.ipix" HEIGHT=300 WIDTH=400

PALETTE="foreground" BORDER =0 type = "application/x-ipix" plugins-Page = "http://www.ipix.com/cgi-bin/download.cgi" >

What happens when the IPIX Plug-in Software Module is downloaded?

It depends upon what browser and operating system are on the computer. If the user has Netscape 4.x or Internet Explorer 3.x/4.x on a Windows-based PC, then download and installation is almost invisible. The Plug-in is installed with the browser still -running. If the user is using a Macintosh browser, or an older version of Netscape or Internet Explorer, the user will have to download the installation program to their hard drive, close the browser, and then run the installation.

FILE LOCATIONS AND NAMING CONVENTIONS

When the user is placing files on the server, the user must meet the following requirements:

• The IPIX images must be stored in the same directory with any associated scripts, sound files, and multimedia components used by an IPIX image.

• The IPIX images must be in the same directory in the HTML or in a subdirectory.

• It is strongly advised to keep all file names and directory names in lowercase characters.

• When the user displays an IPIX image in a frame set, you must use the frame name in all UPPERCASE letters. This is a requirement for the frames to function properly.

• The naming convention used for the JAVA IPIX Viewer is "IpixViewer.class," where the I and V are capitalized.

Absolute path names are supported, but relative paths are recommended. Performance is usually improved when relative paths are employed. There are three ways to place IPIX files into HTML code for viewing:

Controlled IPIX Image Size

When the user wants to control the IPIX image size within the browser window, employ the EMBED SRC tag.

To set IPIX image size, include the HEIGHT and the WIDTH dimensions within the EMBED SRC tag.

It is also essential to include the PALETTE attribute with the value foreground: [PALETTE="foreground"].

Full Screen Image

The simplest way to post an IPIX image for full screen viewing is to use the HREF Anchor tag. When this method is used, the IPIX will fill the browser window.

The following HTML code samples can provide some guidance when one is building Web pages. To view an IPIX image:

• With a clickable graphic another Web page:

< A HREF "http://ww.ipix.com/ipixx/14344.ipx" > Click here < IMG SRC="images/yourimage.gif" WIDTH =100

HEIGHT=86 BORDER=0< > /A>

• With a clickable graphic to a relative address:

< A HREF "ipix/sample.ipx" > Click here <IMG SRC="images/yourimage.giP WIDTH=100 HEIGHT=86 BORDER=0< >/A>

• Without a clickable graphic:

< A HREF ="http://www.ipix.com/ipix/sample.ipx" Click here</A> <IMG SRC="images/yourimage.gir WIDTH =100 HEIGHT =86 BORDER =0>

• Using FRAMES. The HTML code using a FRAME which contains an IPIX link:

JAVAScript Window

It is possible to view IPIX images in separate windows. The size of this window is controlled. This is accomplished using JAVAScript.

Some browsers, including Internet Explorer 3.x and AOL 4.0, may have problems viewing pages using JAVAScript.

The following code may be used for this:

< !-TfflS SCRIPT OPENS IPIX WINDOWS- > </HEAD>

< SCRIPT LANGUAGE="JavaScript" > function Openlpix Window (ipix_path)

{ window.open('ipix/' +ipix_path, 'IPDCWin', ^,height=300,width=400');

}

< /SCRIPT >

</HEAD>

<BODY>

<A

HREF = "javascript: OpenlpixWindow ('yourimage.ipx');" >

Click to View IPJX Image < /A >

</BODY>

WORKING WITH JAVA IPIX IMAGES

JAVA IPIX images are designed for viewing with the IPIX JAVA Viewer. It is unnecessary for a customer to download the IPIX Plug-in software module to view a JAVA IPIX image. They are, however, generated in a lower resolution to reduce file size and may not be acceptable in all cases.

The IPIX JAVA Viewer files consist of the JAVA class files and an archive of the JAVA applet (Ipix Viewer .jar). JAR files work with Microsoft Internet Explorer and Netscape Navigator 4.0 and higher. If someone viewing your page has a 3.x version of a browser, the JAR file will not be recognized, and instead, all of the class files will be loaded. This has the same effect as the JAR file, but it takes a little bit longer to display the page.

For the latest IPIX JAVA Viewer updates, visit the Interactive Picturers' FTP site at ftp://ftp.omniview.com/public/viewers/java.

JAVA Requirements on the Web Server

There are four elements that must be placed on the Web server to properly post a JAVA IPIX image:

• The HTML file that links to the JAVA IPIX image;

• The IPIX JAVA Viewer itself, named IpixViewer.jar; • The JAVA class files; and

• The JAVA IPIX image intended for viewing.

The JAVA JPIX images and the JAVA Viewer may be in the same folder/directory and on the same server.

JAVA Web Page Requirements

To use JAVA JPIX images with the IPIX JAVA Viewer on a Web page follow these guidelines:

• The Spin value can be on or off. The JAVA applet allows a page designer to set the JAVA image to an auto-spin mode by simply changing the value of the Spin parameter to on.

• The image size can be changed, and the Width value may be a multiple of four (more or less).

Creating a JAVA Web Page

Capitalization is very important. If the user does not have the proper capitalization, the page will not work correctly. This information is discussed in "File Locations and Naming Conventions" earlier in this chapter.

When the user creates their page and post it to an Internet site, the user will need to upload the class files. These files are included with your IPIX Wizard. If the user installed their IPIX software to the default location, then the JAVA Viewer files are located in the IPIX software folder, and on the IPIX CD. Resizing The Page

Resizing a Web page containing a JAVA IPIX image may cause problems with image display. When the user resizes the image, the browser automatically refreshes the Web page to reformat it to the new display size. The IPIX image may display as a black screen. To correct this, the user should hold down the Shift button on their keyboard and click the Refresh button on the browser. The IPIX image should display properly after the page reloads.

POSTING IPIX IMAGES TO THE WEB These steps will walk the user through posting IPIX images to a Web page. Everything from MIME types to JAVA/Non-JAVA JPIX images are covered.

STEP 1: ESTABLISHINGMIMETYPE SETTINGS

Make sure the server system or Internet Service Provider has the necessary MIME type settings. For IPIX images to work with the user's server system, these MIME type settings are necessary: application/x-ipscript file extension=ips application/x-ipix file extension = ipx

For more information on MIME types, see "MIME Types" , on page 28 of this Appendix.

STEP 2: GATHERING ALL NECESSARY FILES

Make sure all of the IPIX image files— IPIX images (.IPX), IPScripts (.IPS), and audio files (.MID, .VOX, .WAV) are together in one directory.

STEP 3: EMBEDDING THE IPIX IMAGE

Embed the IPIX image onto the user's Web page. The code used to embed an IPIX image on a Web page varies depending on how one prefers the IPIX picture to appear. If the user would like the IPIX image to open to 640 x 480, the user should use the following code:

For JAVA IPJX Images, go to Step 5 (Using JAVA IPIX images).

<a href="yourimage.ipix" > something clickable like text or a button < /a >

If the user would like to control the opening size of the IPIX image, use the following code, varying the width and height values:

< !- For Netscape 3,4 - >

< embed src="yourimage.ipx" border =0 width =320 height=240 palette = "FOREGROUND" < > /embed > < /OBJECT >

STEP 4: CONTROLLING THE IMAGE SIZE

To open an IPIX image full screen:

<a href="yourimage.ipx" > something clickable like text or a button < /a >

To control the size of the IPIX image:

< OBJECT ID="IpixXl" WIDTH=370 HEIGHT=290 CLASSID ="CLSID: 11260943-421B-11D0-8EAC-0000C07D88CF"

CODEBASE= "http://www.ipix.eom/viewers/ipixx.cab#version =2,2,0, 1 " > < !-For MSIE 3,4+ - >

< embed src="yourimage.ipx" border=0 width=320 height =240 palette = "FOREGROUND" < > /embed > < /OBJECT >

STEP 5: USING JAVA IPIX IMAGES

Embed the JAVA JPIX image onto the user Web page using the following JAVA applet. With this code, the user must have the .jar and the .class files in the same folder as the HTML page that uses this code.

< applet code ="Ipix Viewer. class" archive="IpixViewer.jar" align = "baseline" width ="320" height ="240" name ="Ipix Viewer" >

< /applet >

If the user has multiple JAVA IPIX images on their Web site, the user can add a codebase line to the user's applet. Using the codebase line allows the user to keep the .jar and .class files in one place and to access this one location from any page on the user's Web site.

< applet code = "Ipix Viewer . class" archive = "Ipix Viewer .jar" codebase = "http://ww . ourdomain.co-m/jav ^,' align = "baseline" width ="320" height ="240" name="IpixViewer" >

< /Applet > Explanation of Terms:

The following items are explanations of terms used in the above sample of HTML code.

• APPLET tag is required to call the JAVA applet.

• ARCHIVE indicates the name and location of the JAVA applet. The Ipix Viewer .jar is a compressed form of the JAVA applet.

• CODEBASE indicates the absolute or relative location of the JAR and class files. This tag is only required if the IPIX JAVA Viewer and the IPIX images are in different locations.

• WIDTH=320 This indicates the width of the applet in pixels. 320 is the suggested size. The Width value may be divisible by four (more or less). • HEIGHT =240 This indicates the height of the applet in pixels. 240 is the suggested height.

• PARAM NAME = "URL" This parameter indicates the name and the location of the JAVA IPIX image. Here the parameter is VALUE="yourimage.ipx". This indicates that the JAVA JPIX image is named yourimage.ipx and is located in the same directory as the HTML file that is calling it.

• PARAM NAME = "Spin" This parameter turns on and off the automatic spin feature of the JAVA IPIX image. The VALUE may be set to ON or OFF. When the Spin parameter is not specified, it defaults to off.

For more information on the applet tag, visit Sun's JAVA Web site at: http://java.sun.com/docs/books/tutorial/applet appetsonly/appletTag.html.

Claims

WHAT WE CLAIM IS:

1. Apparatus for processing a plurality of digital wide angle images comprises a digital camera system and an image processor for seaming together said plurality of wide angle images to form a 360 degree round image and for associating audio files to said 360 degree round image, said plurality of digital wide angle images having an audio channel associated with each said wide angle image such that, with two such digital wide angle images, a first digital wide angle image has an associated left stereo channel and a second digital wide angle image has an associated right stereo channel.

2. Apparatus as recited in claim 1 wherein said audio files include an introductory audio prior to viewing said 360 degree round image.

3. Apparatus as recited in claim 1 wherein said audio files represent background audio during viewing of said 360 degree round image.

4. Apparatus as recited in claim 1 wherein said digital wide angle images comprise hemispherical images.

5. Apparatus as recited in claim 1 wherein said digital camera system comprises a time clock for digitally time stamping said digital wide angle images upon image capture.

6. Apparatus as recited in claim 4 wherein said digital camera system comprises a standard lens and a fisheye lens converter.

7. Apparatus as recited in claim 1 wherein said digital camera system comprises a location sensor for outputting location data for storage with said wide angle image as it is captured.

8. Apparatus as recited in claim 1, said image processor for receiving data for customizing a wasted image portion according to user input.

9. Apparatus as recited in claim 1, said image processor for receiving data for identifying a hotspot for association with a selected image portion.

10. Apparatus as recited according to claim 5 further comprising a user interface for inputting personal data, the image processor for constructing copyright indicia for said 360 degree round image responsive to said user interface and said clock.

11. Apparatus as recited in claim 1 further comprising a printer, said image processor, responsive to user inputs in real time for selecting a portion of said 360 degree round image, for outputting said selected image portion for flat still image printing.

12. Apparatus as recited in claim 11 wherein said flat still image comprises a panoramic image portion of said 360 degree round image.

13. Apparatus as recited in claim 1 for outputting said 360 degree round image for posting to a web site.

14. Apparatus as recited in claim 1 for outputting said 360 degree round image as a self-executable file.

15. Apparatus as recited in claim 14 wherein said self-executable file includes viewer software bundled together with the image.

16. Apparatus for outputting a 360 degree round image for posting to a web site for association with a viewer software applet.

17. Apparatus for processing a plurality of digital wide angle images comprising a digital camera system and an image processor for seaming together said plurality of wide angle images to form a 360 degree round image and for outputting said 360 degree round image for posting to a web site for association with a viewer software applet.

18. Apparatus as recited in claim 17, said image processor for further posting said image to a web site with associated audio files.

19. A method for processing a plurality of digital wide angle images for use with a digital image processor, at least two of said digital wide angle image having an associated audio file, the method comprising the steps of seaming together said plurality of wide angle images to form a 360 degree round image and associating said audio files to said 360 degree round image as left and right stereo audio files in relation to a selected portion of said 360 degree round image for viewing.

20. A method as recited in claim 19 further comprising the step of receiving data for customizing a wasted image portion according to user input.

21. A method as recited in claim 19 further comprising the step of receiving data for identifying a hotspot for association with a selected image portion, the hotspot linking said 360 degree round image to a second 360 degree round image having different associated stereo audio files.

22. A method as recited in claim 19 further comprising the steps of receiving personal and date stamp data and constructing copyright indicia for said 360 degree round image responsive to user command.

23. A method for processing a plurality of digital wide angle images for use with a digital image processor comprising the steps, responsive to user inputs in real time, of selecting a perspectively conect portion of a 360 degree round image, the selected image portion having selected image aspect ratio and size, and outputting said selected image portion.

24. A method as recited in claim 23 further comprising the step of outputting said 360 degree round image for posting to a web site.

25. A method as recited in claim 23 further comprising the step of outputting said 360 degree round image as a self-executable file for viewing.

26. A digital camera comprising a controller, a digital image memory and a digital clock for outputting time of day and date data, said camera being equipped with a fisheye lens for capturing first and second opposing hemispheres from the same camera location for forming a spherical image from said first and second opposing hemispheres, said controller, responsive to actuation of a shutter, for simultaneously recording the time of day and date for association with said captured digital spherical image in digital image memory.

27. A digital camera comprising a controller, a digital image memory and a location data sensor for outputting camera location data, said camera being equipped with a fish-eye lens for capturing first and second opposing hemispheres from the same camera location for forming a spherical image from said first and second opposing hemispheres, said controller, responsive to actuation of a shutter, for simultaneously recording the location data for association with said captured digital spherical image in digital image memory.

28. A digital camera comprising a controller, a digital image memory and a horizontal reference data sensor for outputting horizontal reference data, said controller, responsive to actuation of a shutter, for simultaneously recording the horizontal reference data for association with a captured digital image in said digital image memory.

29. A digital camera system comprising a digital camera including a time of day and date clock, a lens and a wide angle lens adapter, said digital camera for mounting to a camera rotator having 180 degree opposing detente positions, said rotator for mounting to a tripod having three legs and a fourth central leg, the fourth central leg having attachment means for attaching the tripod to other than horizontal surfaces.

30. A digital medium for storing digital data representing a method for processing a plurality of digital wide angle images for use with a digital image processor, the method comprising the steps of seaming together said plurality of wide angle images to form a 360 degree round image and associating audio files to said 360 degree round image, said plurality of wide angle images having an audio channel associated with each said wide angle image such that with two such wide angle images, a first image has an associated left stereo channel and a second image has an associated wight stereo channel.

31. A method for processing a plurality of digital wide angle images into a 360 degree round image comprising the steps of storing in memory for association with each 360 degree round image an initial point of view and minimum and maximum magnification parameters which are different from default values for said initial point of view and maximum and minimum magnification parameters.

32. A method for processing a plurality of digital wide angle images into a 360 degree round image comprising the step of storing a selectable parameter value for a seam width between digital wide angle images and for selection data representing color balancing and image smoothing processes.

33. A method for purchasing keys for permitting storage of 360 degree round images generated by an image processing system, the method comprising the steps of displaying a number of keys remaining to a user, displaying a unique site code for said image processing system and transmitting said unique site code to a key distribution center, and responsive to outputting an order for additional keys, displaying a unique key access code. AMENDED CLAIMS

[received by the International Bureau on 3 January 2000 (03.01.00); original claims 1, 19 and 30 amended; new claim 34 added; remaining claims unchanged ( 6 pages)]

1. Apparatus for processing a plurality of digital wide angle images comprises a digital camera system and an image processor for seaming together said plurality of wide angle images to form a single 360 degree round spherical image being capable of viewing as a sequence of perpsectively correct, flat image portions and for associating audio files to said single 360 degree round spherical image, said plurality of digital wide angle images having an audio channel associated with each said wide angle image such that, with two such digital wide angle images, a first digital wide angle image has an associated left stereo channel and a second digital wide angle image has an associated right stereo channel.

7. Apparatus as recited in claim 1 wherein said digital camera system comprises a location sensor for outpuπing location data for storage with said wide angle image as it is captured.

8. Apparatus as recited in claim 1. said image processor for receiving data for customizing a wasted image portion according to user input.

19. A method for processing a plurality of digital wide angle images for use with a digital image processor, at least two of said digital wide angle image having an associated audio file, the method comprising the steps of seaming together said plurality of wide angle images to form a single 360 degree round spherical image being capable of viewing as a sequence of perspectively correct, flat image portions and associating said audio files to said single 360 degree round spherical image as left and right stereo audio files in relation to a selected portion of said single 360 degree round spherical image for viewing.

23. A method for processing a plurality of digital wide angle images for use with a digital image processor comprising the steps, responsive to user inputs in real time, of selecting a perspectively correct portion of a 360 degree round image, the selected image portion having selected image aspect ratio and size, and outputting said selected image portion.

26. A digital camera comprising a controller, a digital image memory and a digital clock for outputting time of day and date data, said camera being equipped with a fish- eye lens for capturing first and second opposing hemispheres from the same camera location for forming a spherical image from said first and second opposing hemispheres, said controller, responsive to actuation of a shutter, for simultaneously recording the time of day and date for association with said captured digital spherical image in digital image memory.

27. A digital camera comprising a controller, a digital image memory and a location data sensor for outputting camera location data, said camera being equipped with a fish- eye lens for capturing first and second opposing hemispheres from the same camera location for forming a spherical image from said first and second opposing hemispheres, said controller, responsive to actuation of a shutter, for simultaneously recording the location data for association with said captured digital spherical image in digital image memory.

30. A digital medium for storing digital data representing a method for processing a plurality of digital wide angle images for use with a digital image processor, the method comprising the steps of seaming together said plurality of wide angle images to form a single 360 degree round spherical image being capable of viewing as a sequence of perspectively correct, flat image portions and associating audio files to said single 360 degree round spherical image, said plurality of wide angle images having an audio channel associated with each said wide angle image such that with two such wide angle images, a first image has an associated left stereo channel and a second image has an associated right stereo channel.

31. A method for processing a plurality of digital wide angle images into a 360 degree round image comprising the steps of storing in memory for association with each 360 degree round image an imtial point of view and minimum and maximum magnification parameters which are different from default values for said initial point of view and maximum and minimum magnification parameters.

33. A method for purchasing keys for permitting storage of 360 degree round images generated by an image processing system, the method comprising the steps of displaying a number of keys remaining to a user, displaying a unique site code for said image processing system and transmitting said unique site code to a key distribution center, and responsive to outputting an order for additional keys, displaying a unique key access code.

34. A method of decrementing one key from at least one purchased key comprising the steps of seaming together wide angle images to form a single 360 degree round spherical image and decrementing one key responsive to storing the single 360 degree round spherical image.

STATEMENT UNDER ARTICLE 19

Responsive to the International Search Report in the above-titled International application issued on 04 November 1999 and prior to publication, please amend the application to replace sheets 6- jΛ with new sheets (pages) 26-32 comprising claims 1-34 and the recommended Abstract. Claims 1, 19 and 30 have been amended to recite "a single 360 degree round spherical image being capable of viewing as a sequence of perspectively correct, flat image portions^ Also, a new independent claim 34 has been added to claim applicants' method of decrementing one from one purchased key. No changes were made to original sheets 32-43 (an Appendix); these sheets have been renumbered as sheets 33-44.

REMARKS

These remarks are made responsive to the Citations and Explanations section of the PCT International Search Report issued 04 November 1999 by Authorized Officer Tommy Chin (by James R. Matthew). The Officer indicates that Ritchey (US 5,130,794) is a document of particular relevance with respect to claims 1-4, 12, 19, 23, 26, 27 and 30. Applicants note a description of microphones 67 in Figure 6 and element 31 of Figure 8. Ritchey suggests that microphones 67 "face outward" (column 12, line 52) - on the "left and right side of the housing 31 such that stereo sound is recorded" (column 12, lines 54-55). Ritchey fails to describe associating audio files with a single 360 degree round spherical image, said single image being capable of viewing as a sequence of perspectively correct, flat image portions of the spherical image.

Independent claims other than claims 1, 19 and 30 have not been amended. These other independent claims including claims 16, 17, 23, 26-29 and 31-33 are directed to a number of concepts. Allegedly, for example, recording time of day and date (claim 26) and location (claim 27) of a captured spherical image are shown by Ritchey. Ritchey discloses a conventional camcorder. The Examiner may be assuming that camcorders have time and date stamping. If so, the Examiner is encouraged to identify a teaching of same. Location (claim 27) is not something known to applicants and could not be located in Ritchey. Moreover, posting spherical images to a web site, preselecting aspect ratio or ordering keys off a web site, among other claimed features could not be located in the applied references.

Applicant has added a new claim 34 directed to a method decrementing one key when a spherical image is stored. None of the cited references show such a method.

Applicant respectfully requests that claims 1-34 be examined as this application either enters the national stage or applicants demand preliminary examination.