WO2005081057A1 - Method and apparatus for providing a combined image - Google Patents

Method and apparatus for providing a combined image Download PDF

Info

Publication number
WO2005081057A1
WO2005081057A1 PCT/SG2005/000044 SG2005000044W WO2005081057A1 WO 2005081057 A1 WO2005081057 A1 WO 2005081057A1 SG 2005000044 W SG2005000044 W SG 2005000044W WO 2005081057 A1 WO2005081057 A1 WO 2005081057A1
Authority
WO
WIPO (PCT)
Prior art keywords
images
stitching
cameras
image
combined image
Prior art date
Application number
PCT/SG2005/000044
Other languages
French (fr)
Inventor
Toh Onn Desmond Hii
Original Assignee
Creative Technology Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Creative Technology Ltd filed Critical Creative Technology Ltd
Priority to AU2005215585A priority Critical patent/AU2005215585A1/en
Priority to GB0616491A priority patent/GB2430104A/en
Publication of WO2005081057A1 publication Critical patent/WO2005081057A1/en

Links

Classifications

    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B37/00—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe
    • G03B37/04—Panoramic or wide-screen photography; Photographing extended surfaces, e.g. for surveying; Photographing internal surfaces, e.g. of pipe with cameras or projectors providing touching or overlapping fields of view
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/239—Image signal generators using stereoscopic image cameras using two 2D image sensors having a relative position equal to or related to the interocular distance
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/243—Image signal generators using stereoscopic image cameras using three or more 2D image sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N5/00—Details of television systems
    • H04N5/222—Studio circuitry; Studio devices; Studio equipment
    • H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
    • H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation

Definitions

  • This invention relates to a method and apparatus for providing a combined image and refers particularly, though not exclusively, to such a method and apparatus for providing a combined image from a plurality of images.
  • Panoramic images are images over a wide angle. In normal photography panoramic images are normally taken by having a sequence of successive images that are subsequently joined, or stitched together, to form the combined image. When the images are taken simultaneously using a plurality of cameras, the images are normally displayed separately. For video camera security, video conferencing, and other similar applications, this means multiple cameras, and multiple displays, must be used for continuous panoramic imaging.
  • one or more of the cameras may be a pan/tilt camera. This requires the pan/tilt cameras to have an operator to move the camera's field of vision, or a servomotor to move the camera.
  • the servomotor may be operated remotely and/or automatically.
  • the camera is covering only a part of its maximum field of view at any one time. The consequence is that another part of its maximum field of view is not covered at any one time. This is unsatisfactory.
  • wide-angle lenses may be used to reduce the impact of the loss of coverage, the distortion introduced, particularly at higher off-axis angles, is also unsatisfactory.
  • a wide-angle lens also requires a higher resolution image sensor to maintain the same resolution.
  • a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images comprising: (a) generating the plurality of images in each of the plurality of cameras;
  • a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images comprising:
  • a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images comprising:
  • a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras, each of the plurality of cameras having an image system for taking an image of the plurality of images comprising: (a) generating the plurality of images in each of the plurality of cameras; (b) performing overlap calculations to determine overlap regions of the plurality of images;
  • step (d) using the results of step (b) for all subsequent pluralities of images from the plurality of cameras.
  • a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images comprising: (a) generating the plurality of images in each of the plurality of cameras; (b) selecting a presentation style for the combined image; and (c) stitching the plurality of images to form the combined image in the presentation style, stitching being by a stitcher disguised as a virtual camera.
  • a method of producing a combined video image from a plurality of video images each produced by one of a plurality of video cameras each having an image system for taking an image of the plurality of images comprising:
  • a penultimate aspect of the invention provides a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising the steps: (a) generating the plurality of images in each of the plurality of cameras; (b) performing overlap calculations to determine overlap regions of the plurality of images; (a) using the overlap calculations to perform colour correction in the plurality of images; and
  • a final aspect of the invention provides apparatus for providing a combined image, the apparatus comprising
  • Each camera may have a buffer, and they may be in a common body, or may be separate.
  • Figure 1 is a perspective view of a preferred form of combined camera
  • Figure 2 is a perspective view of a second form of a combined camera
  • Figure 3 is a block diagram of the apparatus of Figures 1 and 2;
  • Figure 4 is a flow chart of the virtual camera of Figure 2.
  • Figure 5 is a representation of various presentation styles.
  • one approach to create a real-time combined video stream is to use multiple cameras 10. Although three are shown, this is for convenience. The number used may be any appropriate number from two up. If enough cameras were used, the field of view could be 360° in one plane. It could be spherical.
  • the image sensors 12 in a multiple-camera can either be separate entities as shown in Figure 1, or combined into a single camera body 14 as shown in Figure 2. Either way, each image sensor 12 of the multiple cameras provides a partial view of the target scene.
  • the fields of view of each camera 10 overlaps with the field of view of the adjacent camera 10, and the video streams from each camera are stitched together using a stitcher into a single, combined video. If the cameras 10 are separate entities as shown in Figure 1 they may be separate but relatively close as if in a cluster; or may be separate and remote from each other. If remote, it is still preferred for the fields of view to overlap.
  • the multiple- camera configuration As compared to a single camera with mechanical pan tilt motor, the multiple- camera configuration has the advantage of no moving parts which makes it free from mechanical failure. It has the additional benefit of capturing the entire scene all the time, behaving like a wide-angle lens camera, but without the associated distortion and loss of image data, particularly at wide, off-axis angles. Unlike a single wide-angle lens camera, which has a single image sensor, the multiple- camera configuration is scalable to wider view, and provides higher resolution due to the usage of multiple image sensors.
  • a multiple-camera system is useable using existing cameras and video applications, such as video conferencing and web casting applications, on a standard computer.
  • existing video applications can be used.
  • One way for it to work with existing video applications is to disguise a stitcher as a virtual camera ( Figure 3) that can process the individual images from the cameras 10 to form the combined image, and present it to a generic video application. In this way special hardware and/or software may be avoided.
  • OS computer operating systems
  • Every camera has a custom "device driver”, which provides a common interface to which the OS can communicate.
  • the OS provides a common interface to its applications for them to send queries and commands to the camera.
  • Such layered architecture provides a standard way for the applications to access the cameras. Using a common driver interface is important for these applications to work independently of the camera vendor. It also enables these applications to continue to function with future cameras, as long as the cameras respect the common driver interface.
  • the virtual camera 32 does not exist in a physical sense.
  • the virtual camera 32 Instead of providing a video stream from an image sensor, which it lacks, the virtual camera 32 obtains the video streams 34 from other real cameras 30, 31 directly from their device drivers 33 or by using the common driver interface. It then combines and repackages these video streams into a single video stream, which it offers through its own common driver interface 33.
  • a combined camera 32 is a virtual camera, which stitches the input video streams 34 into a combined video stream.
  • the virtual camera 32 is a video processor capable of processing one or more input video streams, and outputs a single video stream.
  • the virtual camera 32 appears as a regular camera, with a wide viewing angle. In this way, the image data from more than one camera 30, 31 can be processed by the virtual camera 32 such that the computer's video application 35 sees it as a single camera.
  • the number of cameras involved is not limited and may be two, three, four, five, six, and so forth.
  • the panorama captured by their combined field of view is not limited and may extend to 360°, and even to a sphere.
  • the combined virtual camera 32 is essentially a stitcher. In real time it takes overlapping images, one from each camera, and combines them into one combined image.
  • the images come from the buffers 41, 42, 43... from each camera 30, 31....
  • Each image is warped (44) into an intermediate coordinate, such as the cylindrical or spherical co-ordinates, so that stitching can be reduced to a simple two-dimensional search. It then determines the overlap region of these images (45). Using the overlap region, colour correction can be performed (46) to ensure colour consistency across the images. The same colour correction, or substantially the same colour correction, is used for all subsequent images.
  • the final images are then blended (47) together to form the final panorama.
  • the combined virtual camera performs the overlap calculation (45) only once, and assumes that the camera positions remain the same throughout the session.
  • Some video applications have format restriction. For example H.261 based video conferencing applications only accept CIF and QCIF resolution. The size and aspect ratio of the resulting combined image is likely to be different from the standard video formats. An additional stage to transform the image to the required format may also be performed, which typically involves scaling and panning.
  • Figure 5 illustrates a number of different presentation styles.
  • Figure 5(a) is the original combined image.
  • the letterbox and pan & scan style of Figures 5(b) and 5(c) respectively resemble the approaches taken by the Digital Versatile Disc (DVD) format, to display a 16:9 image on a standard 4:3 display.
  • the horizontal compression style of Figure 5(d) may be useful for recording the combined video as it captures the entire view, at the expense of some loss in image detail.
  • a separate user interface may be provided to the user to enable the selection of different presentation styles.
  • pan & scan (48) the user can interactively pan the panorama to select a region of interest.
  • automatic panning and switching between styles can be employed at pre-set time intervals.
  • Multiple styles can also be created simultaneously.
  • the horizontal compressed style may be used for recording the video, while the pan & scan may be used for display.
  • the virtual camera may perform the stereoscopic image formation such as, for example, by interlacing odd and even rows, and stacking the images for a top-to-bottom stereoscopy.
  • the virtual camera may be used to combine or merge video from different cameras; and it may be used for the generation of lenticular stereoscopic image/video.
  • the virtual camera 32 is able to convert multiple video streams into a single stream in a stereo format by performing interlacing, resizing, and translation. Resizing is preferably performed with proper filtering such as, for example, "Cubic” and “Lanczos” interpolations for upsizing, and "Box” or “Area Filter” for downsizing. Row-interlace stereoscopy format interlaces the stereo pair with odd rows representing the left eye, and even rows representing the right eye. This can be viewed using de-multiplexing equipment such as, for example, "Stereographic's SimulEyes", and that is compatible with standard video signals.
  • the virtual camera 32 performs the interlacing, which involves copying pixels, and possibly resizing each line:
  • Above-Below stereoscopy format requires the vertically resizing and translation of the source images, the top for the left eye, and the bottom for the right eye.
  • Side-by-Side format can also be used.
  • the virtual camera 32 performs scaling and translation to combine the two video streams into a single stereo video stream.
  • a device capable of decoding the selected format can be used to view the stereo pair using stereo glasses.
  • the cameras 10 may be digital still cameras, or digital motion picture cameras.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Studio Devices (AREA)
  • Image Processing (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Stereoscopic And Panoramic Photography (AREA)

Abstract

Disclosed is a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras. Each camera has an image system for taking an image of the plurality of images. The method comprises generating the plurality of images in each of the plurality of cameras and stitching the plurality of images to form the combined image using a stitcher disguised as a virtual camera.

Description

Method and Apparatus for Providing a Combined Image
Field of the Invention
This invention relates to a method and apparatus for providing a combined image and refers particularly, though not exclusively, to such a method and apparatus for providing a combined image from a plurality of images.
Definitions
Throughout this specification the use of "combined" is to be taken as including a reference to the creation of a panoramic image, as well as a stereoscopic image, lenticular stereoscopic image/video, and video post-production to merge two or more video image streams into a single video stream.
Background to the Invention
Panoramic images are images over a wide angle. In normal photography panoramic images are normally taken by having a sequence of successive images that are subsequently joined, or stitched together, to form the combined image. When the images are taken simultaneously using a plurality of cameras, the images are normally displayed separately. For video camera security, video conferencing, and other similar applications, this means multiple cameras, and multiple displays, must be used for continuous panoramic imaging.
Alternatively or additionally, one or more of the cameras may be a pan/tilt camera. This requires the pan/tilt cameras to have an operator to move the camera's field of vision, or a servomotor to move the camera. The servomotor may be operated remotely and/or automatically. However, when such a system is used, the camera is covering only a part of its maximum field of view at any one time. The consequence is that another part of its maximum field of view is not covered at any one time. This is unsatisfactory.
Although wide-angle lenses may be used to reduce the impact of the loss of coverage, the distortion introduced, particularly at higher off-axis angles, is also unsatisfactory. A wide-angle lens also requires a higher resolution image sensor to maintain the same resolution.
Summary of the Invention
In accordance with one aspect of the present invention there is provided a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising: (a) generating the plurality of images in each of the plurality of cameras;
(b) stitching the plurality of images to form the combined image using a stitcher disguised as a virtual camera.
According to another aspect of the invention there is provided a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising:
(a) generating the plurality of images in each of the plurality of cameras;
(b) using a virtual camera to perform a stitching operation on the plurality of images to form the combined image.
According to a further aspect of the invention there is provided a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising:
(a) generating the plurality of images in the plurality of cameras;
(b) warping each of the plurality of images into an intermediate co-ordinate; and
(c) stitching the plurality of images into the combined image using a two dimensional search, stitching being by a stitcher disguised as a virtual • camera.
In accordance with yet another aspect of the invention there is provided a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras, each of the plurality of cameras having an image system for taking an image of the plurality of images, the method comprising: (a) generating the plurality of images in each of the plurality of cameras; (b) performing overlap calculations to determine overlap regions of the plurality of images;
(c) stitching the plurality of images to form the combined image; and
(d) using the results of step (b) for all subsequent pluralities of images from the plurality of cameras.
In accordance with an additional aspect of the invention there is provided a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising: (a) generating the plurality of images in each of the plurality of cameras; (b) selecting a presentation style for the combined image; and (c) stitching the plurality of images to form the combined image in the presentation style, stitching being by a stitcher disguised as a virtual camera.
In accordance with a further additional aspect of the invention there is provided a method of producing a combined video image from a plurality of video images each produced by one of a plurality of video cameras each having an image system for taking an image of the plurality of images, the method comprising:
(a) warping each of the plurality of video images into an intermediate coordinate;
(b) determining overlap regions of the warped plurality of video images; (c) stitching the warped plurality of video images to form the combined video image, stitching being by a stitcher disguised as a virtual camera; and (d) processing the combined video image for one or more of: display and storage.
A penultimate aspect of the invention provides a method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising the steps: (a) generating the plurality of images in each of the plurality of cameras; (b) performing overlap calculations to determine overlap regions of the plurality of images; (a) using the overlap calculations to perform colour correction in the plurality of images; and
(b) performing substantially the same colour correction for all subsequent pluralities of images from the plurality of cameras.
A final aspect of the invention provides apparatus for providing a combined image, the apparatus comprising
(a) a plurality of cameras each having an image system;
(b) a stitcher for producing the combined image by performing a stitching operation on a plurality of images, each of the plurality of images being produced by one of the plurality of cameras; and
(c) the stitcher being disguised as a virtual camera.
Each camera may have a buffer, and they may be in a common body, or may be separate.
Brief Description of the Drawings
In order that the invention may be fully understood and readily put into practical effect, there shall now be described by way of non-limitative example only preferred embodiments of the present invention, the description being with reference to the accompanying illustrative drawings in which:
Figure 1 is a perspective view of a preferred form of combined camera;
Figure 2 is a perspective view of a second form of a combined camera; Figure 3 is a block diagram of the apparatus of Figures 1 and 2;
Figure 4 is a flow chart of the virtual camera of Figure 2; and
Figure 5 is a representation of various presentation styles.
Detailed Description of the Preferred Embodiments
As shown in Figures 1 and 2, one approach to create a real-time combined video stream is to use multiple cameras 10. Although three are shown, this is for convenience. The number used may be any appropriate number from two up. If enough cameras were used, the field of view could be 360° in one plane. It could be spherical. The image sensors 12 in a multiple-camera can either be separate entities as shown in Figure 1, or combined into a single camera body 14 as shown in Figure 2. Either way, each image sensor 12 of the multiple cameras provides a partial view of the target scene. Preferably the fields of view of each camera 10 overlaps with the field of view of the adjacent camera 10, and the video streams from each camera are stitched together using a stitcher into a single, combined video. If the cameras 10 are separate entities as shown in Figure 1 they may be separate but relatively close as if in a cluster; or may be separate and remote from each other. If remote, it is still preferred for the fields of view to overlap.
As compared to a single camera with mechanical pan tilt motor, the multiple- camera configuration has the advantage of no moving parts which makes it free from mechanical failure. It has the additional benefit of capturing the entire scene all the time, behaving like a wide-angle lens camera, but without the associated distortion and loss of image data, particularly at wide, off-axis angles. Unlike a single wide-angle lens camera, which has a single image sensor, the multiple- camera configuration is scalable to wider view, and provides higher resolution due to the usage of multiple image sensors.
A multiple-camera system is useable using existing cameras and video applications, such as video conferencing and web casting applications, on a standard computer. In this way existing video applications can be used. One way for it to work with existing video applications is to disguise a stitcher as a virtual camera (Figure 3) that can process the individual images from the cameras 10 to form the combined image, and present it to a generic video application. In this way special hardware and/or software may be avoided.
Most computer operating systems (OS) provide a standard method for its applications to access an attached camera. Typically, every camera has a custom "device driver", which provides a common interface to which the OS can communicate. In turn, the OS provides a common interface to its applications for them to send queries and commands to the camera. Such layered architecture provides a standard way for the applications to access the cameras. Using a common driver interface is important for these applications to work independently of the camera vendor. It also enables these applications to continue to function with future cameras, as long as the cameras respect the common driver interface. The virtual camera 32 does not exist in a physical sense. Instead of providing a video stream from an image sensor, which it lacks, the virtual camera 32 obtains the video streams 34 from other real cameras 30, 31 directly from their device drivers 33 or by using the common driver interface. It then combines and repackages these video streams into a single video stream, which it offers through its own common driver interface 33. A combined camera 32 is a virtual camera, which stitches the input video streams 34 into a combined video stream. As such the virtual camera 32 is a video processor capable of processing one or more input video streams, and outputs a single video stream.
From a video application's 35 perspective, the virtual camera 32 appears as a regular camera, with a wide viewing angle. In this way, the image data from more than one camera 30, 31 can be processed by the virtual camera 32 such that the computer's video application 35 sees it as a single camera. The number of cameras involved is not limited and may be two, three, four, five, six, and so forth. The panorama captured by their combined field of view is not limited and may extend to 360°, and even to a sphere.
As shown in Figure 4, the combined virtual camera 32 is essentially a stitcher. In real time it takes overlapping images, one from each camera, and combines them into one combined image. The images come from the buffers 41, 42, 43... from each camera 30, 31.... Each image is warped (44) into an intermediate coordinate, such as the cylindrical or spherical co-ordinates, so that stitching can be reduced to a simple two-dimensional search. It then determines the overlap region of these images (45). Using the overlap region, colour correction can be performed (46) to ensure colour consistency across the images. The same colour correction, or substantially the same colour correction, is used for all subsequent images. The final images are then blended (47) together to form the final panorama.
To achieve real-time performance, the combined virtual camera performs the overlap calculation (45) only once, and assumes that the camera positions remain the same throughout the session.
Some video applications have format restriction. For example H.261 based video conferencing applications only accept CIF and QCIF resolution. The size and aspect ratio of the resulting combined image is likely to be different from the standard video formats. An additional stage to transform the image to the required format may also be performed, which typically involves scaling and panning.
Figure 5 illustrates a number of different presentation styles. Figure 5(a) is the original combined image. The letterbox and pan & scan style of Figures 5(b) and 5(c) respectively resemble the approaches taken by the Digital Versatile Disc (DVD) format, to display a 16:9 image on a standard 4:3 display. The horizontal compression style of Figure 5(d) may be useful for recording the combined video as it captures the entire view, at the expense of some loss in image detail.
A separate user interface may be provided to the user to enable the selection of different presentation styles. For pan & scan (48), the user can interactively pan the panorama to select a region of interest. Alternatively, automatic panning and switching between styles can be employed at pre-set time intervals. Multiple styles can also be created simultaneously. For example, the horizontal compressed style may be used for recording the video, while the pan & scan may be used for display.
By having multiple viewpoints, a perfect stitch may be possible. However, at the overlapping region, double or missing images may result. The problem may be more serious for near objects than distant objects. For surveillance application, which has mostly distant objects, the problems may be reduced. For close-up applications such as, for example, video conferencing, three cameras may be used, so that the centre camera has the full picture of the human head and shoulder. Each camera should preferably send thirty frames each second.
For real-time stereoscopy, the virtual camera may perform the stereoscopic image formation such as, for example, by interlacing odd and even rows, and stacking the images for a top-to-bottom stereoscopy. For post-processing of video, the virtual camera may be used to combine or merge video from different cameras; and it may be used for the generation of lenticular stereoscopic image/video.
The virtual camera 32 is able to convert multiple video streams into a single stream in a stereo format by performing interlacing, resizing, and translation. Resizing is preferably performed with proper filtering such as, for example, "Cubic" and "Lanczos" interpolations for upsizing, and "Box" or "Area Filter" for downsizing. Row-interlace stereoscopy format interlaces the stereo pair with odd rows representing the left eye, and even rows representing the right eye. This can be viewed using de-multiplexing equipment such as, for example, "Stereographic's SimulEyes", and that is compatible with standard video signals. The virtual camera 32 performs the interlacing, which involves copying pixels, and possibly resizing each line:
Line 1 [ Left eye Line 1 ]
Line 2 [ Right eye Line 2 ] Line 3 [ Left eye Line 3 ]
Line 4 [ Right eye Line 4 ]
Above-Below stereoscopy format requires the vertically resizing and translation of the source images, the top for the left eye, and the bottom for the right eye. In the same way, the Side-by-Side format can also be used. In these cases, the virtual camera 32 performs scaling and translation to combine the two video streams into a single stereo video stream. At the receiving end, a device capable of decoding the selected format can be used to view the stereo pair using stereo glasses.
The cameras 10 may be digital still cameras, or digital motion picture cameras.
Whilst there has been described in the foregoing description a preferred embodiment of the present invention, it will be understood by those skilled in the technology that may variations or modifications in details of one or more of design, construction and operation maybe made without departing from the present invention.

Claims

The Claims
1. A method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising: (a) generating the plurality of images in each of the plurality of cameras; (b) stitching the plurality of images to form the combined image using a stitcher disguised as a virtual camera.
2. A method as claimed in claim 1 , wherein stitching is by warping each of the plurality of images into an intermediate co-ordinate, and stitching the plurality of images into the combined image using a two dimensional search.
3. A method as claimed in claim 1, further comprising performing overlap calculations to determine overlap regions of the plurality of images, the overlap calculation being used for all subsequent pluralities of images from the plurality of cameras.
4. A method as claimed in claim 1 , further comprising selecting a presentation style for the combined image.
5. A method as claimed in claim 3, further comprising selecting a presentation style for the combined image.
6. A method as claimed in claim 3, wherein stitching is by warping each of the plurality of images into an intermediate co-ordinate, and stitching the plurality of images into the combined image using a two dimensional search.
7. A method as claimed in claim 4, wherein stitching is by warping each of the plurality of images into an intermediate- co-ordinate, and stitching the plurality of images into the combined image using a two dimensional search.
8. A method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising: (a) generating the plurality of images in each of the plurality of cameras; (b) using a virtual camera to perform a stitching operation on the plurality of images to form the combined image.
9. A method as claimed in claim 8, wherein stitching is by warping each of the plurality of images into an intermediate co-ordinate, and stitching the plurality of images into the combined image using a two dimensional search.
10. A method as claimed in claim 8, further comprising performing overlap calculations to determine overlap regions of the plurality of images, the overlap calculation being used for all subsequent pluralities of images from the plurality of cameras.
11. A method as claimed in claim 10, further including: (a) using the overlap calculations to perform colour correction in the plurality of images; and (b) maintaining the colour correction for all subsequent pluralities of images from the plurality of cameras.
12. A method as claimed in claim 10, further comprising selecting a presentation style for the combined image.
13. A method as claimed in claim 11, further comprising selecting a presentation style for the combined image.
14. A method as claimed in claim 11 , wherein stitching is by warping each of the plurality of images into an intermediate co-ordinate, and stitching the plurality of images into the combined image using a two dimensional search.
15. A method as claimed in claim 12, wherein stitching is by warping each of the plurality of images into an intermediate co-ordinate, and stitching the plurality of images into the combined image using a two dimensional search.
16. A method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising: (a) generating the plurality of images in each of the plurality of cameras; (b) warping each of the plurality of images into an intermediate co- ordinate; and (c) stitching the plurality of images into the combined image using a two dimensional search, stitching being by a stitcher disguised as a virtual camera.
17. A method as claimed in claim 16, further comprising performing overlap calculations to determine overlap regions of the plurality of images, the overlap calculation being used for all subsequent pluralities of images from the plurality of cameras.
18. A method as claimed in claim 16, further comprising selecting a presentation style for the combined image.
19. A method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising: (a) generating the plurality of images in each of the plurality of cameras; (b) performing overlap calculations to determine overlap regions of the plurality of images; (c) stitching the plurality of images to form the combined image, stitching being by a stitcher disguised as a virtual camera; and (d) using the results of step (b) for all subsequent pluralities of images from the plurality of cameras.
20. A method as claimed in claim 19, wherein stitching is by warping each of the plurality of images into an intermediate co-ordinate, and stitching the plurality of images into the combined image using a two dimensional search.
21. A method as claimed in claim 19, further comprising selecting a presentation style for the combined image.
22. A method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising: (a) generating the plurality of images in each of the plurality of cameras; (b) selecting a presentation style for the combined image; and (c) stitching the plurality of images to form the combined image in the presentation style, stitching being by a stitcher disguised as a virtual camera.
23. A method as claimed in claim 22, wherein stitching is by warping each of the plurality of images into an intermediate co-ordinate, and stitching the plurality of images into the combined image using a two dimensional search.
24. A method as claimed in claim 22, further comprising performing overlap calculations to determine overlap regions of the plurality of images, the overlap calculations being used for all subsequent pluralities of images from the plurality of cameras.
25. A method of producing a combined video image from a plurality of video images each produced by one of a plurality of video cameras each having an image system for taking an image of the plurality of images, the method comprising: (a) warping each of the plurality of video images into an intermediate co-ordinate; (b) determining overlap regions of the warped plurality of video images; (c) stitching the warped plurality of video images to form the combined video image, stitching being by a stitcher disguised as a virtual camera; and (d) processing the combined video image for one or more of: display and storage.
26. A method as claimed in claim 25, further comprising performing overlap calculations to determine overlap regions of the plurality of images, the overlap calculations being used for all subsequent pluralities of images from the plurality of cameras.
27. A method as claimed in claim 25, further comprising selecting a presentation style for the combined image.
28. A method for providing a combined image from a plurality of images each produced by one of a plurality of cameras each having an image system for taking an image of the plurality of images, the method comprising: (a) generating the plurality of images in each of the plurality of cameras; (b) performing overlap calculations to determine overlap regions of the plurality of images; (c) using the overlap calculations to perform colour correction in the plurality of images; and (d) performing substantially the same colour correction for all subsequent pluralities of images from the plurality of cameras.
29. A method as claimed in claim 28, wherein stitching is by warping each of the plurality of images into an intermediate co-ordinate, and stitching the plurality of images into the combined image using a two dimensional search.
30. A method as claimed in claim 28, further comprising selecting a presentation style for the combined image.
31. A method as claimed in claim 28, wherein stitching is by a stitcher disguised as a virtual camera.
32. A method as claimed in claim 29, further comprising selecting a presentation style for the combined image.
33. A method as claimed in claim 30, further comprising selecting a presentation style for the combined image.
34. A method as claimed in claim 29, wherein stitching is by a stitcher disguised as a virtual camera.
35. Apparatus for producing a combined image, the apparatus comprising: (a) a plurality of cameras each having an image system; (b) a stitcher for performing a stitching operation on a plurality of images, each of the plurality of images being produced by one of the plurality of cameras, to produce the combined image; (c) the stitcher being disguised as a virtual camera.
36. Apparatus as claimed in claim 36, wherein each camera includes a buffer.
37. Apparatus as claimed in claim 36, wherein the plurality of cameras is in a common body.
38. Apparatus as claimed in claim 36, wherein each of the plurality of cameras is in a separate body.
SβSmblE SHEET (RULE 26)
PCT/SG2005/000044 2004-02-19 2005-02-17 Method and apparatus for providing a combined image WO2005081057A1 (en)

Priority Applications (2)

Application Number Priority Date Filing Date Title
AU2005215585A AU2005215585A1 (en) 2004-02-19 2005-02-17 Method and apparatus for providing a combined image
GB0616491A GB2430104A (en) 2004-02-19 2005-02-17 Method and apparatus for providing a combined image

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US10/783,279 US20050185047A1 (en) 2004-02-19 2004-02-19 Method and apparatus for providing a combined image
US10/783,279 2004-02-19

Publications (1)

Publication Number Publication Date
WO2005081057A1 true WO2005081057A1 (en) 2005-09-01

Family

ID=34861191

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/SG2005/000044 WO2005081057A1 (en) 2004-02-19 2005-02-17 Method and apparatus for providing a combined image

Country Status (6)

Country Link
US (1) US20050185047A1 (en)
CN (1) CN1922544A (en)
AU (1) AU2005215585A1 (en)
GB (1) GB2430104A (en)
TW (1) TW200529098A (en)
WO (1) WO2005081057A1 (en)

Families Citing this family (49)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7868847B2 (en) * 2005-05-24 2011-01-11 Mark W Miles Immersive environments with multiple points of view
US8509563B2 (en) * 2006-02-02 2013-08-13 Microsoft Corporation Generation of documents from images
CN101479767A (en) * 2006-06-30 2009-07-08 Nxp股份有限公司 A method and device for video stitching
US8139026B2 (en) 2006-08-02 2012-03-20 Research In Motion Limited System and method for adjusting presentation of text and images on an electronic device according to an orientation of the device
US8493323B2 (en) * 2006-08-02 2013-07-23 Research In Motion Limited System and method for adjusting presentation of moving images on an electronic device according to an orientation of the device
EP2259163A3 (en) * 2006-08-02 2011-03-16 Research In Motion Limited System and method for adjusting presentation of moving images on an electronic device according to an orientation of the device
US7843487B2 (en) * 2006-08-28 2010-11-30 Panasonic Corporation System of linkable cameras, each receiving, contributing to the encoding of, and transmitting an image
US8607281B2 (en) * 2006-09-07 2013-12-10 Porto Vinci Ltd. Limited Liability Company Control of data presentation in multiple zones using a wireless home entertainment hub
US9319741B2 (en) 2006-09-07 2016-04-19 Rateze Remote Mgmt Llc Finding devices in an entertainment system
JP2008219428A (en) * 2007-03-02 2008-09-18 Fujifilm Corp Imaging apparatus
FR2913779B1 (en) * 2007-03-13 2009-11-13 Gint DEVICE FOR THE ACQUISITION OF CUSTOMIZATION IMAGE.
US8009200B2 (en) * 2007-06-15 2011-08-30 Microsoft Corporation Multiple sensor input data synthesis
US8300083B2 (en) * 2007-07-20 2012-10-30 Hewlett-Packard Development Company, L.P. Position relationships associated with image capturing devices
TW200907557A (en) * 2007-08-08 2009-02-16 Behavior Tech Computer Corp Camera array apparatus and the method for capturing wide-angle video over a network
EP2174486A2 (en) * 2007-08-09 2010-04-14 Koninklijke Philips Electronics N.V. Method and device for creating a modified video from an input video
KR101204888B1 (en) * 2007-08-31 2012-11-26 삼성테크윈 주식회사 Digital photographing apparatus, method for controlling the same, and recording medium storing program to implement the method
SG150414A1 (en) * 2007-09-05 2009-03-30 Creative Tech Ltd Methods for processing a composite video image with feature indication
AU2008310672A1 (en) * 2007-10-12 2009-04-16 Polycom, Inc. Integrated system for telepresence videoconferencing
US20100033570A1 (en) * 2008-08-05 2010-02-11 Morgan Plaster Driver observation and security system and method therefor
CN101771830B (en) * 2008-12-30 2012-09-19 华为终端有限公司 Three-dimensional panoramic video stream generating method and equipment and video conference method and equipment
EP2385705A4 (en) 2008-12-30 2011-12-21 Huawei Device Co Ltd Method and device for generating stereoscopic panoramic video stream, and method and device of video conference
US8767081B2 (en) * 2009-02-23 2014-07-01 Microsoft Corporation Sharing video data associated with the same event
CN101527828B (en) * 2009-04-14 2011-08-10 华为终端有限公司 Image acquisition equipment
CN101556758B (en) * 2009-04-23 2010-12-01 杭州镭星科技有限公司 Method for realizing displaying of high dynamic luminance range images by a plurality of projecting cameras
JP5235798B2 (en) * 2009-06-22 2013-07-10 富士フイルム株式会社 Imaging apparatus and control method thereof
TWI532009B (en) 2010-10-14 2016-05-01 華晶科技股份有限公司 Method and apparatus for generating image with shallow depth of field
US9055189B2 (en) 2010-12-16 2015-06-09 Microsoft Technology Licensing, Llc Virtual circular conferencing experience using unified communication technology
US8908054B1 (en) * 2011-04-28 2014-12-09 Rockwell Collins, Inc. Optics apparatus for hands-free focus
US8698874B2 (en) * 2011-06-10 2014-04-15 Microsoft Corporation Techniques for multiple video source stitching in a conference room
US8953079B2 (en) * 2012-12-31 2015-02-10 Texas Instruments Incorporated System and method for generating 360 degree video recording using MVC
CA2856896A1 (en) * 2013-07-18 2015-01-18 Spo Systems Inc. Limited Virtual video patrol system and components therefor
CN104516482A (en) * 2013-09-26 2015-04-15 北京天盛世纪科技发展有限公司 Shadowless projection system and method
US20150124120A1 (en) * 2013-11-05 2015-05-07 Microscan Systems, Inc. Machine vision system with device-independent camera interface
ES2654598T3 (en) * 2013-12-23 2018-02-14 Coherent Synchro, S.L. System to generate a composite video image and procedure to obtain a composite video image
US20150271400A1 (en) * 2014-03-19 2015-09-24 Htc Corporation Handheld electronic device, panoramic image forming method and non-transitory machine readable medium thereof
CN104243920B (en) * 2014-09-04 2017-09-26 浙江宇视科技有限公司 A kind of image split-joint method and device encapsulated based on basic flow video data
CN104360488B (en) * 2014-11-14 2017-05-24 山东理工大学 Display method for compact three-dimensional display system
TWI548281B (en) * 2014-11-21 2016-09-01 財團法人工業技術研究院 Wide view monitoring system and method thereof
CN104680078B (en) * 2015-01-20 2021-09-03 中兴通讯股份有限公司 Method for shooting picture, method, system and terminal for viewing picture
CN104680077B (en) * 2015-01-20 2021-10-12 中兴通讯股份有限公司 Method for encrypting picture, method for viewing picture, system and terminal
CN104615917A (en) * 2015-01-20 2015-05-13 中兴通讯股份有限公司 Picture camouflaging method, picture viewing method, system and terminal
US9769419B2 (en) 2015-09-30 2017-09-19 Cisco Technology, Inc. Camera system for video conference endpoints
CN105530473A (en) * 2015-12-09 2016-04-27 北京永泰安达科技有限公司 Fast evidence collection system for panoramic eye criminal investigation
CN105657290A (en) * 2016-01-29 2016-06-08 宇龙计算机通信科技(深圳)有限公司 Dual-camera based scanning method and device
US11019257B2 (en) 2016-05-19 2021-05-25 Avago Technologies International Sales Pte. Limited 360 degree video capture and playback
DE102017009145A1 (en) * 2016-10-14 2018-04-19 Avago Technologies General Ip (Singapore) Pte. Ltd. Capture and playback 360-degree video
US10297059B2 (en) 2016-12-21 2019-05-21 Motorola Solutions, Inc. Method and image processor for sending a combined image to human versus machine consumers
CN109300145B (en) * 2018-08-20 2020-06-16 彭楷文 Self-adaptive intelligent camouflage system
KR20220025600A (en) * 2020-08-24 2022-03-03 삼성전자주식회사 Method and apparatus for generating image

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5650814A (en) * 1993-10-20 1997-07-22 U.S. Philips Corporation Image processing system comprising fixed cameras and a system simulating a mobile camera
US5657073A (en) * 1995-06-01 1997-08-12 Panoramic Viewing Systems, Inc. Seamless multi-camera panoramic imaging with distortion correction and selectable field of view
WO1998047291A2 (en) * 1997-04-16 1998-10-22 Isight Ltd. Video teleconferencing
WO1999006943A1 (en) * 1997-08-01 1999-02-11 Sarnoff Corporation Method and apparatus for performing local to global multiframe alignment to construct mosaic images
WO2002013515A2 (en) * 2000-08-07 2002-02-14 Koninklijke Philips Electronics N.V. Fast digital pan tilt zoom video system

Family Cites Families (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5023725A (en) * 1989-10-23 1991-06-11 Mccutchen David Method and apparatus for dodecahedral imaging system
KR940017747A (en) * 1992-12-29 1994-07-27 에프. 제이. 스미트 Image processing device
US5495576A (en) * 1993-01-11 1996-02-27 Ritchey; Kurtis J. Panoramic image based virtual reality/telepresence audio-visual system and method
US5850352A (en) * 1995-03-31 1998-12-15 The Regents Of The University Of California Immersive video, including video hypermosaicing to generate from multiple video views of a scene a three-dimensional video mosaic from which diverse virtual video scene images are synthesized, including panoramic, scene interactive and stereoscopic images
US6549650B1 (en) * 1996-09-11 2003-04-15 Canon Kabushiki Kaisha Processing of image obtained by multi-eye camera
JPH10336705A (en) * 1997-06-02 1998-12-18 Canon Inc Compound eye camera
US6545702B1 (en) * 1998-09-08 2003-04-08 Sri International Method and apparatus for panoramic imaging

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5650814A (en) * 1993-10-20 1997-07-22 U.S. Philips Corporation Image processing system comprising fixed cameras and a system simulating a mobile camera
US5657073A (en) * 1995-06-01 1997-08-12 Panoramic Viewing Systems, Inc. Seamless multi-camera panoramic imaging with distortion correction and selectable field of view
WO1998047291A2 (en) * 1997-04-16 1998-10-22 Isight Ltd. Video teleconferencing
WO1999006943A1 (en) * 1997-08-01 1999-02-11 Sarnoff Corporation Method and apparatus for performing local to global multiframe alignment to construct mosaic images
WO2002013515A2 (en) * 2000-08-07 2002-02-14 Koninklijke Philips Electronics N.V. Fast digital pan tilt zoom video system

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
FOOTE J. ET AL: "FlyCam: Practical Panoramic Video and Automatic Camera Control", PROC.IEEE INTERNATIONAL CONFERENCE ON MULTIMEDIA AND EXPO., vol. 3, 2000, pages 1419 - 1422, XP010512772, DOI: doi:10.1109/ICME.2000.871033 *
MAJUMDER A. ET AL: "Immersive Teleconferencing: A New Algorithm to Generate Seanless Panoramic Video Imagery.", PROC.ACM INTERNATIONAL CONFERENCE ON MULTIMEDIA., 1999, pages 169 - 178, XP001033456 *

Also Published As

Publication number Publication date
GB2430104A (en) 2007-03-14
TW200529098A (en) 2005-09-01
CN1922544A (en) 2007-02-28
GB0616491D0 (en) 2006-09-27
US20050185047A1 (en) 2005-08-25
AU2005215585A1 (en) 2005-09-01

Similar Documents

Publication Publication Date Title
US20050185047A1 (en) Method and apparatus for providing a combined image
US11109066B2 (en) Encoding and decoding of volumetric video
US10645369B2 (en) Stereo viewing
US7224382B2 (en) Immersive imaging system
US20200186784A1 (en) Modular Configurable Camera System
KR101003277B1 (en) Method and system for producing seamless composite images having non-uniform resolution from a multi-imager
EP2603834B1 (en) Method for forming images
US20170227841A1 (en) Camera devices with a large field of view for stereo imaging
CN104685860A (en) Generating images from light fields utilizing virtual viewpoints
KR20180053367A (en) Systems and methods for creating a surround view
US20100097444A1 (en) Camera System for Creating an Image From a Plurality of Images
CN105794197A (en) Portable device capable of generating panoramic file
JP2017505565A (en) Multi-plane video generation method and system
JP2014212519A (en) Stereoscopic panoramas
JP3691444B2 (en) Stereo imaging device
US11812009B2 (en) Generating virtual reality content via light fields
JP2014529930A (en) Selective capture and display of a portion of a native image
JP2019145059A (en) Information processing unit, information processing system, information processing method and program
WO2018109265A1 (en) A method and technical equipment for encoding media content
JPH1042307A (en) Key system and synthetic image forming method
WO2018109266A1 (en) A method and technical equipment for rendering media content
KR20190118803A (en) Stereoscopic image generating apparatus
WO2019008233A1 (en) A method and apparatus for encoding media content
KR101229421B1 (en) 3d stereo image capture apparatus based on multi segmented methdo and the metohd thereof
Carr et al. Portable multi-megapixel camera with real-time recording and playback

Legal Events

Date Code Title Description
AK Designated states

Kind code of ref document: A1

Designated state(s): AE AG AL AM AT AU AZ BA BB BG BR BW BY BZ CA CH CN CO CR CU CZ DE DK DM DZ EC EE EG ES FI GB GD GE GH GM HR HU ID IL IN IS JP KE KG KP KR KZ LC LK LR LS LT LU LV MA MD MG MK MN MW MX MZ NA NI NO NZ OM PG PH PL PT RO RU SC SD SE SG SK SL SY TJ TM TN TR TT TZ UA UG US UZ VC VN YU ZA ZM ZW

AL Designated countries for regional patents

Kind code of ref document: A1

Designated state(s): GM KE LS MW MZ NA SD SL SZ TZ UG ZM ZW AM AZ BY KG KZ MD RU TJ TM AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LT LU MC NL PL PT RO SE SI SK TR BF BJ CF CG CI CM GA GN GQ GW ML MR NE SN TD TG

121 Ep: the epo has been informed by wipo that ep was designated in this application
DPEN Request for preliminary examination filed prior to expiration of 19th month from priority date (pct application filed from 20040101)
WWE Wipo information: entry into national phase

Ref document number: 0616491.7

Country of ref document: GB

Ref document number: 0616491

Country of ref document: GB

Ref document number: 200580005316.9

Country of ref document: CN

Ref document number: 2005215585

Country of ref document: AU

NENP Non-entry into the national phase

Ref country code: DE

WWW Wipo information: withdrawn in national office

Country of ref document: DE

ENP Entry into the national phase

Ref document number: 2005215585

Country of ref document: AU

Date of ref document: 20050217

Kind code of ref document: A

WWP Wipo information: published in national office

Ref document number: 2005215585

Country of ref document: AU

122 Ep: pct application non-entry in european phase