EP2253131B1 - Panoramic camera with multiple image sensors using timed shutters - Google Patents
Panoramic camera with multiple image sensors using timed shutters Download PDFInfo
- Publication number
- EP2253131B1 EP2253131B1 EP09708701.9A EP09708701A EP2253131B1 EP 2253131 B1 EP2253131 B1 EP 2253131B1 EP 09708701 A EP09708701 A EP 09708701A EP 2253131 B1 EP2253131 B1 EP 2253131B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- image
- image sensor
- camera
- camera apparatus
- capturing
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/698—Control of cameras or camera modules for achieving an enlarged field of view, e.g. panoramic image capture
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/40—Extracting pixel data from image sensors by controlling scanning circuits, e.g. by modifying the number of pixels sampled or to be sampled
- H04N25/41—Extracting pixel data from a plurality of image sensors simultaneously picking up an image, e.g. for increasing the field of view by combining the outputs of a plurality of sensors
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
- H04N25/50—Control of the SSIS exposure
- H04N25/53—Control of the integration time
- H04N25/531—Control of the integration time by controlling rolling shutters in CMOS SSIS
Definitions
- the present invention relates to the field of panoramic still and motion photography.
- Imaging systems exist that include more than one camera in a rosette formation attached to a vehicle. Those imaging systems may be used to capture panoramic images, for example, along a street. Each camera includes an entrance pupil. Having multiple entrance pupils at different locations can cause spatial parallax. Parallax refers to a perceived shift of an imaged object against a background caused by the different viewpoints of the entrance pupils of the cameras. Parallax is a particular problem when stitching together images from multiple cameras, since it can cause ghosting of foreground objects when background objects are aligned in the region where adjacent images overlap.
- Mirrors have been used to avoid parallax in panoramic imaging.
- Disney's Circle- Vision 360[deg.] system uses mirrors to view the world through multiple co- located entrance pupils. This can result in zero parallax.
- making and using the necessary large mirrors is difficult. Also, this technique limits the vertical field of view.
- Each camera in the rosette formation includes an image sensor that converts an optical signal into an electrical signal to form an image.
- image sensors Two types of image sensors avoid the need for a mechanical shutter-an interline-shutter charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor.
- CCD mechanical shutter-an interline-shutter charge-coupled device
- CMOS complementary metal oxide semiconductor
- the CMOS sensor typically has a rolling shutter, which exposes different lines of the scene at slightly different times, in a sequence that rolls across the image.
- the rolling shutter can lead to image distortion, such as warping, when the camera is moving relative to the scene or subject. Some work has been done to compensate for the image distortion.
- Wilburn et al. describes using a rolling shutter timing offset to process and correct distortions induced by motion of a subject of an image. See Wilburn et al., "High-Speed Videography Using a Dense Camera Array", 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'04), Vol. 2, pp. 294-341 .
- the interline-shutter CCD avoids the image distortion Issues of the rolling shutter CMOS sensor.
- the interline-shutter CCD can suffer from blooming and streaking. Blooming and streaking occur when a portion of the sensor is over-exposed, causing light to spillover to adjacent pixels or into the readout CCD structure.
- US 2007/0030342 A1 refers to an apparatus having a mechanism for capturing frames by video cameras and a processing unit for spatio-temporal interpolation to generate a visual output.
- WO 2007/045714 refers to a camera that obtains digital images or video with reduced motion distortion by adjusting the scan direction of an image sensor.
- a camera apparatus for panoramic photography includes a first image sensor positioned to capture a first image.
- the first image sensor has a rolling-shutter readout arranged In portrait orientation.
- the camera apparatus also includes second image sensor positioned to capture a second image.
- the second image sensor has a rolling-shutter readout arranged in portrait orientation.
- the camera apparatus includes a controller configured to signal the second image sensor to start capturing the second image before the first image sensor finishes capturing the first image. At least a portion of the first image is in front of the second image relative to a forward direction of the camera apparatus.
- a method for panoramic photography includes the steps of: capturing a first image with a first image sensor and starting to capture a second image with a second image sensor prior to completion of the capturing of the first image.
- the first and second image sensors have a rolling-shutter readouts arranged in landscape orientation. At least a portion of the first image is in front of the second image relative to a forward direction of a camera apparatus comprising the first and second image sensors.
- a camera apparatus for motion photography includes a first camera having a first entrance pupil. During forward motion of the camera apparatus, the first camera captures a first image.
- a camera apparatus also includes a second camera having a second entrance pupil. During the forward motion of the camera apparatus, the second camera captures a second image. Motion of the camera apparatus results in motion parallax. The timing of when the first camera captures a first image relative to when the second camera captures the second image uses the motion parallax to reduce the effect of spatial parallax between the first camera and the second camera.
- a method for motion photography with a camera apparatus includes: capturing a first image with a first camera in the camera apparatus at a first time during forward motion of the camera apparatus, and capturing a second image with a second camera in the camera apparatus at a second time the during forward motion of the camera apparatus.
- Motion of the camera apparatus results in motion parallax.
- the timing of when the capturing (a) occurs relative to when the capturing (b) occurs uses the motion parallax to reduce the effect of spatial parallax between the first camera and the second camera.
- embodiments of the present invention reduce distortion and spatial parallax.
- Embodiments of present invention reduce distortion and spatial parallax in panoramic images taken from a camera rosette affixed to a vehicle.
- Each camera in the camera rosette includes an image sensor.
- the image sensor may expose to capture an image while the vehicle is moving. This movement can cause distortion in the resulting image.
- the CMOS sensor is arranged in "portrait" orientation. As is described in detail below, this reduces the image distortion.
- the motion parallax of the vehicle is employed to cancel out spatial parallax.
- exposure of the cameras in the camera rosette is timed such that the entrance pupils of the cameras are in approximately the same location when each sensor is exposed. This embodiment is described in detail below.
- references to "one embodiment”, “an embodiment”, “an example embodiment”, etc. indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
- camera rosette refers to two or more cameras arranged to capture a panoramic image. Each camera captures an image. The images may be stitched together to form the panoramic image. This stitching may, for example, be done by well-known software techniques.
- a camera refers to a device that captures a photographic image.
- a camera includes an image sensor, and a lens having an entrance pupil.
- image sensor refers to a device that converts optical signals into electrical signals to capture an image when exposed.
- FIG. 1A includes a diagram 100 illustrating a camera that includes a rolling-shutter CMOS image sensor.
- Diagram 100 shows a camera and its image sensor at two points in time. At a first point in time, the camera is at a position 102 and its image sensor is at a position 104. At a second point in time, the camera is at a position 112 and its image sensor is at a position 114.
- the dot in the figure represents the entrance pupil of the lens, and rays are shown projecting through it in the manner of a pinhole camera for simplicity.
- the camera may be attached to a vehicle, and movement of the vehicle may have caused the camera to move.
- the image sensor may be a rolling shutter CMOS sensor.
- a rolling shutter CMOS sensor may include groups of photodiodes that capture light and convert the light into electrical signals during exposure.
- a rolling shutter CMOS sensor may expose different groups of photodiodes at different times.
- a rolling shutter CMOS sensor has a readout time. The readout time is the time necessary to read out all the groups of photodiodes at the ends of their respective exposure times.
- the CMOS sensor is in "landscape" orientation, meaning that the groups of photodiodes are arranged as rows running parallel to the ground.
- the rolling shutter sensor may be exposed while the vehicle is in motion.
- a rolling shutter sensor may require 100 ms of readout time to capture an image. While the image is captured, the vehicle and camera may have moved forward 1 m. The movement causes the objects captured by the image sensor to change as shown at 106. This causes distortion in the resulting image. Because different rows of photodiodes in the CMOS sensors are exposed at different times, the image may appear warped, e.g. vertical features may be slanted.
- An image sensor that does not have a rolling shutter and is exposed in its entirety, such as an interline-shutter CCD sensor, may reduce or avoid warping. However, as mentioned earlier, interline-shutter CCD sensors may suffer from blooming and streaking.
- FIG. 1B includes a diagram 150 illustrating a camera having a rolling-shutter sensor, such as a CMOS sensor, in portrait orientation.
- Portrait orientation means that the groups of photodiodes are arranged in columns running perpendicular to the ground.
- Diagram 150 shows a camera at different positions at different points in time.
- the camera may be affixed to a vehicle, and the vehicle may be in motion.
- the columns of photodiodes are exposed back-to-front in object space and front-to-back in image space.
- the camera exposes a forward-most photodiode column 156 to capture an image including a back-most field of view 154.
- the camera exposes a photodiode column 158, which is behind photodiode column 156.
- This exposure captures a field of view 172, which is to the front of field of view 154.
- columns continue to expose in a front-to-back manner, which captures objects from back-to-front.
- the camera exposes a back-most photodiode column 168 to capture an image including a forward-most field of view 170.
- Positioning the image sensor in portrait orientation avoids the warping of vertical features that occurs in landscape orientation. Instead, objects may appear stretched by an amount that depends on their distance. Stretching is not as visually unappealing as warping. In fact, stretching may be a positive feature. Stretching makes foreground objects thinner with respect to background objects, rather than fatter, when the rolling shutter is arranged in the direction described. As result, the foreground objects occlude less of the scene behind them. This may be a benefit in many applications related to panoramic imaging.
- Panoramic imaging often includes taking images from multiple cameras oriented in different directions. The multiple images may then be stitched together into a panorama using well-known software techniques.
- an object may exist in the field of view of more than one camera.
- motion photography objects may move or change over time relative to the cameras. If multiple cameras expose the object at different times, then the object may appear differently in the multiple images to stitch together into a single panorama.
- the resulting panorama may have ghosting or other undesirable image problems.
- exposure needs to be coordinated between the multiple cameras to capture objects at the same time or close to the same time. This becomes even more challenging with rolling shutter image sensors because different portions of each image sensor are exposed as different times.
- embodiments of the present invention time exposure of multiple image sensors as illustrated in FIGs. 2A-C .
- FIGs. 2A-C include diagrams illustrating a portion of a camera rosette, which may be used in panoramic imaging.
- Each camera in the camera rosette includes a rolling-shutter image sensor, such as a rolling-shutter CMOS sensor, in portrait orientation as described with respect to FIG. 1B .
- FIG. 2A includes a diagram 200.
- Diagram 200 shows a camera rosette with cameras 202 and 210. For clarity, only two cameras are shown. However, in practice the rosette may contain a different number of cameras.
- Camera 202 has photodiode column 204 exposed to a field of view 220.
- FIG. 2B includes a diagram 230.
- Diagram 230 shows the rosette shown in FIG. 2A at a later point in time.
- Camera 202 has a total field of view 224, and camera 210 has a total field of view 234.
- camera 202 scans across field of view 224 back-to-front (in object space)
- camera 202 begins to expose photodiode columns from a group of photodiode columns 206.
- Group of columns 206 captures objects within field of view 222.
- Field of view 222 overlaps with camera 210's total field of view 234 for distant objects.
- camera 210 begins to expose photodiode columns from a group of photodiode columns 208.
- Camera 210 exposes photodiode columns in group of columns 208 back-to-front (in object space) to capture field of view 226.
- Field of view 226 overlaps with camera 202's total field of view 224 for distant objects.
- camera 202 exposes photodiode columns in group of columns 206 simultaneously with camera 210 exposing photodiode columns in group of columns 208.
- camera 202 finishes its exposure.
- camera 210 continues to expose photodiode columns, as shown FIG. 2C .
- FIG. 2C shows a diagram 240.
- Diagram 240 shows the rosette shown in FIG. 2B at a later point in time after camera 202 has completed its exposure.
- a photodiode column 232 is exposed, capturing a field of view 228.
- camera 210 continues its exposure, it may overlap with an field of view with another camera (not shown) and the process may continue for an entire camera rosette as is described with respect to FIG. 3 .
- overlap field of view 222 for distant objects between camera 202 and camera 210 may be 10% of total field of view 224 of camera 202.
- the delay between the time when camera 202 starts exposure and the time when camera 210 starts exposure may be 90% of camera 202 readout time.
- Camera 202 and 210 may or may not have the same readout time. In an example where the readout time is 100 ms, the delay between the time when camera 202 starts exposure and the time when camera 210 starts exposure may be 90 ms.
- the camera rosette may have at least one controller that synchronizes the cameras to capture images at specified time offsets. So, the controller may be preprogrammed with the delays indicating when each camera starts to capture an image.
- the delay time may be offset according to the vehicle's velocity. As is described below with respect to FIG. 5 , this has the effect of using motion parallax from movement of the vehicle to cancel out spatial parallax caused by the multiple cameras.
- a camera rosette may have a total of between six and ten cameras, inclusive, positioned to capture a 360 degree panorama.
- the camera rosette may have at least eight cameras, such as the rosette illustrated with respect to FIG. 3 .
- FIG. 3 shows a diagram 300 illustrating the operation of a camera rosette according to an embodiment of the present invention in further detail.
- a camera rosette may include nine cameras--eight in a horizontal ring, and one directed straight up.
- Diagram 300 shows the eight horizontal cameras.
- Each camera in the camera rosette includes a rolling shutter sensor in portrait orientation with columns of photodiodes.
- the camera rosette begins capturing a panoramic image at the rear-most cameras 305 and 304. Both of the cameras 305 and 304 start by exposing the back most photodiodes columns (in object-space) and scan forward as described with respect to FIG. 1B . Cameras 306, 303 start capturing images at a time defined by a time delay as described with respect to FIG. 2B .
- the rosette continues to scan forward until the forward-most cameras 301, 308 have completed scanning.
- Each camera starts at a time defined by the start of the adjacent camera to its rear plus a delay.
- a camera 302 starts at a time defined by the start of camera 303 plus a delay.
- the delay may correspond to the size of the overlap in the field of view of adjacent cameras, as described with respect to FIG. 2B .
- the delay may also be offset according to the velocity of the vehicle, as is described with respect to FIG. 5 .
- Each of the eight cameras in FIG. 3 is at a different location and has an entrance pupil at a different location. Having entrance pupils at different locations causes spatial parallax as illustrated in FIGs. 4A-B .
- FIGs. 4A-B show how the positioning of the entrance pupils of two cameras impacts spatial parallax.
- FIG. 4A shows an example imaging apparatus 400.
- Imaging apparatus 400 includes a camera 402 and a camera 404.
- Camera 402 has an entrance pupil 406, and camera 404 has an entrance pupil 408. Entrance pupil 406 and entrance pupil 408 are separated by a distance 426.
- Each camera 402, 404 captures an image including an object 410. However there is a wide difference in the background of object 410 as it appears in cameras 402, 404. This is shown by the large parallax angle 422.
- FIG. 4B shows an example imaging apparatus 450.
- Imaging apparatus 450 includes a camera 416 and a camera 414.
- Camera 416 has an entrance pupil 418
- camera 414 has an entrance pupil 420.
- Entrance pupil 418 and entrance pupil 420 are separated by a distance 428.
- Distance 428 is shorter than distance 426 in FIG. 4A .
- Each camera 416, 414 captures an image including an object 412.
- motion parallax is caused by a camera being in different locations at different times.
- motion parallax caused by motion of a camera rosette by cancel out or reduce the effect of spatial parallax caused by the entrance pupils of the different cameras on the camera rosette being located at different locations. This embodiment is illustrated in FIG. 5 .
- FIG. 5 shows a diagram 500 illustrating timing the exposure of cameras in a camera rosette to reduce parallax, according to an embodiment of the present invention.
- Diagram 500 shows how to time the exposure of the cameras during motion of the vehicle so that each of the cameras is exposed when its entrance pupil is located at approximately the same location. As result, in diagram 500 the motion parallax caused by the moving vehicle reduces the spatial parallax due to multiple cameras.
- Diagram 500 shows a portion of a camera rosette at three positions 510, 520, 530 at three different points in time.
- the camera rosette shown has three cameras 502, 504, and 506 shown.
- Cameras 502, 504, and 506 have entrance pupils 508, 514, and 512 respectively.
- the image sensors have rolling shutters.
- the embodiment is useful with rolling shutters.
- the image capture is described as happening at an instant, even though there will be some exposure duration and a small motion blur as a result.
- camera 502 captures an image.
- camera 502's entrance pupil 508 is at a location 524. Each subsequent camera is timed to take an image when that camera's entrance pupil is approximately located at location 524.
- the camera rosette reaches position 520.
- entrance pupil 514 of camera 504 is approximately at location 524.
- camera 504 captures an image.
- entrance pupil 512 of camera 506 reaches approximately location 524, and camera 506 captures an image.
- the approximation may be due to inaccuracies in the vehicles velocity, or changes to the vehicle's direction. In an example, the approximation may be based on an average vehicle velocity as opposed to the actual velocity over the relevant period. Also, the approximation may be due to the fact that cameras are positioned two or three dimensionally on the camera rosette, whereas the movement of the rosette (e.g., on a vehicle) may be in only one direction. For example, in FIG. 5 entrance pupil 508 is not at precisely the same location as entrance pupil 514 because entrance pupil 508 are entrance pupil 514 are at different locations on the axis perpendicular to the axis of motion.
- a virtual plane is perpendicular to the direction of motion of a camera rosette.
- Each camera in a camera rosette may expose when an entrance pupil of the camera passes through the plane. In this way, each camera exposes when its entrance pupil is at approximately the same location, reducing parallax.
- the time to trigger each subsequent camera may be calculated according to vehicle velocity and a distance between the cameras, such as a distance 522.
- the delay from a time when camera 504 captures an image and a time when camera 506 captures an image is distance 522 divided by the vehicle's velocity. Supposing distance 522 was 10 cm and the vehicle velocity was 10 m/s, then the delay between the time when camera 504 captures an image and the time when camera 506 captures an image would be 10 ms. In other words, the further rear camera (camera 506) starts to capture an image 10 ms after the more forward camera (camera 504).
- a vehicle velocity used to time the cameras may be an average vehicle velocity.
- the vehicle velocity may be determined in real time by a speedometer, a wheel encoder, or a GPS sensor. The velocity may be signaled to controllers in the cameras, which adjust the timing accordingly.
- FIG. 2B describes timing camera exposure in a camera rosette where each camera includes a rolling-shutter CMOS sensor in portrait orientation.
- each camera is timed to start exposure when the previous camera begins to scan an region where their fields of view overlap at a distance.
- t is the delay time
- R 1 is a rolling-shutter readout time of the first image sensor
- F 1 is a percentage of the field of view of the first image sensor which does not overlap with the field of view of the second image sensor
- D 12 is a distance between an entrance pupil of the first image sensor and an entrance pupil of the second image sensor
- v is a speed of the camera apparatus. It would be recognized that small changes such as small offsets and coefficients may be added to this equation.
- overlap field of view 222 between camera 202 and camera 210 may be 10% of the total field of view of camera 202. In that example, 90% of camera 202's field of view does not overlap with camera 210's field of view.
- the delay between the time when camera 202 starts exposure and the time when camera 210 starts exposure may be 90 ms. In other words, the more forward camera (camera 210) starts to capture an image 90 ms after the further rear camera starts to capture an image (camera 202).
- cameras with overlapping fields of view capture images of objects in their overlap region at the same time, or from the same vehicle position.
- This timing is a first-order correction for parallax, in that it prevents the large parallax errors that would result if the objects in the overlap area were captured at very different times and very different vehicle positions, as would happen if the cameras started their rolling shutters simultaneously. However, there remains a smaller parallax error due to the entrance pupils of the cameras not being in the same place.
- the delay time is offset to further reduce spatial parallax.
- the further rear camera (camera 506) starts to capture an image 10 ms after the more forward camera (camera 504).
- the more forward camera (camera 210) starts to capture an image 90 ms after the further rear camera starts to capture an image (camera 202).
- the 90 ms delay is offset by 10 ms. Therefore, the more forward camera starts to capture an image 80 ms after the further rear camera starts to capture an image.
- This embodiment has reduced spatial parallax, accounting jointly for rolling-shutter readout delay, vehicle velocity, and entrance pupil separation.
- the timing offsets for reduced spatial parallax may only be applied to some of the cameras in a camera rosette.
- the offset may be applied to only those cameras pointed sideways from the vehicle motion.
- the cameras pointed sideways may be the cameras where typical vehicle motion moves the location of the entrance pupil of the more rear camera into the location where the camera took an image in, for example, 0.01 s or less.
- spatial parallax of forward-looking and backward-looking portions of the panorama may not be cancelled by motion parallax, but objects in these directions tend to be distant enough to not have a large parallax problem.
- a rolling shutter with a very fast readout time may be useful. As the shutter readout time becomes faster, the delay computed to account for the rolling shutter (as described in FIGs. 2A-C ) decreases. So, at a particular readout time, the delay for the rolling shutter and the offset to account for spatial parallax may cancel each other out.
- This readout time is the time it takes for the vehicle to move forward a distance equal to the distance between entrance pupils.
- a readout time of a first image sensor may be 10 ms. Ninety percent of the first image sensor's field of view may not overlap with a second, next forward-most image sensor. If the vehicle is stationary, the delay time between exposure of the first and second image sensor would be 9 ms.
- the delay offset due to motion of the vehicle would also be 9 ms.
- the offsets would effectively cancel each other out.
- an image sensor with a variable rolling readout time may be used. In that instance, the readout time may be adjusted according to a speed of the vehicle.
- FIG. 6 shows a camera rosette 604 affixed to a vehicle 602.
- Camera rosette 604 includes numerous cameras, such as a camera 606.
- camera rosette 604 may be used to capture panoramic images of buildings running along a street.
- controller 608 controls the timing of the exposure of each camera.
- Controller 608 may receive an input indicating a speed of the vehicle.
- controller 608 may be preprogrammed with timing offsets, such as offsets based on an average speed of the vehicle.
- a single controller 608 may be used.
- each camera may have its own controller.
- Controller 608 may be implemented in hardware, software, firmware, or any combination thereof. Controller 608 may be, for example, a general purpose computer running software to control the cameras' exposure. Controller 608 may have a processor and memory.
- camera apparatus for motion photography comprising a first camera having a first entrance pupil wherein, during forward motion of the camera apparatus, the first camera captures a first image; and a second camera having a second entrance pupil wherein, during the forward motion of the camera apparatus, the second camera captures a second image, wherein motion of the camera apparatus results in motion parallax, and wherein the timing of when the first camera captures a first image relative to when the second camera captures the second image uses the motion parallax to reduce the effect of spatial parallax between the first camera and the second camera.
- the first camera captures a first image when the first entrance pupil is at a first location and the second camera captures a second image when the second entrance pupil is at a second location, wherein the second location is approximately equal to the first location.
- the first and second cameras are attached to a vehicle.
- the apparatus comprises at least six additional image sensors, wherein a computer is able to stitch together images taken from the at least eight total cameras to create a panoramic image.
- Another aspect of the present invention provides a method for motion photography with a camera apparatus, comprising:
- the capturing of the first image comprises capturing the first image when a first entrance pupil of the first camera is at a first location
- the capturing the second image comprises capturing the second image when the second entrance pupil is at a second location, wherein the second location is approximately equal to the first location
- Another aspect of the present invention provides apparatus for motion photography with a camera apparatus, comprising a means for capturing a first image with a first camera in the camera apparatus at a first time during forward motion of the camera apparatus; and a means for capturing a second image with a second camera in the camera apparatus at a second time during forward motion of the camera apparatus, wherein motion of the camera apparatus results in motion parallax, wherein the timing of when the capturing of the first image occurs relative to when the capturing of the second image occurs uses the motion parallax to reduce the effect of spatial parallax between the first camera and the second camera.
Description
- The present invention relates to the field of panoramic still and motion photography.
- Imaging systems exist that include more than one camera in a rosette formation attached to a vehicle. Those imaging systems may be used to capture panoramic images, for example, along a street. Each camera includes an entrance pupil. Having multiple entrance pupils at different locations can cause spatial parallax. Parallax refers to a perceived shift of an imaged object against a background caused by the different viewpoints of the entrance pupils of the cameras. Parallax is a particular problem when stitching together images from multiple cameras, since it can cause ghosting of foreground objects when background objects are aligned in the region where adjacent images overlap.
- Mirrors have been used to avoid parallax in panoramic imaging. For example, Disney's Circle- Vision 360[deg.] system uses mirrors to view the world through multiple co- located entrance pupils. This can result in zero parallax. However, making and using the necessary large mirrors is difficult. Also, this technique limits the vertical field of view.
- Each camera in the rosette formation includes an image sensor that converts an optical signal into an electrical signal to form an image. Two types of image sensors avoid the need for a mechanical shutter-an interline-shutter charge-coupled device (CCD) or a complementary metal oxide semiconductor (CMOS) sensor.
- The CMOS sensor typically has a rolling shutter, which exposes different lines of the scene at slightly different times, in a sequence that rolls across the image. The rolling shutter can lead to image distortion, such as warping, when the camera is moving relative to the scene or subject. Some work has been done to compensate for the image distortion. Wilburn et al. describes using a rolling shutter timing offset to process and correct distortions induced by motion of a subject of an image. See Wilburn et al., "High-Speed Videography Using a Dense Camera Array", 2004 IEEE Computer Society Conference on Computer Vision and Pattern Recognition (CVPR'04), Vol. 2, pp. 294-341.
- The interline-shutter CCD avoids the image distortion Issues of the rolling shutter CMOS sensor. However, the interline-shutter CCD can suffer from blooming and streaking. Blooming and streaking occur when a portion of the sensor is over-exposed, causing light to spillover to adjacent pixels or into the readout CCD structure.
- What is needed is an imaging apparatus that reduces distortion and spatial parallax, while avoiding the blooming and streaking issues associated with CCDs.
-
US 2007/0030342 A1 refers to an apparatus having a mechanism for capturing frames by video cameras and a processing unit for spatio-temporal interpolation to generate a visual output. -
WO 2007/045714 refers to a camera that obtains digital images or video with reduced motion distortion by adjusting the scan direction of an image sensor. - The present invention relates to the field of panoramic still and motion photography. In a first embodiment, a camera apparatus for panoramic photography includes a first image sensor positioned to capture a first image. The first image sensor has a rolling-shutter readout arranged In portrait orientation. The camera apparatus also includes second image sensor positioned to capture a second image. The second image sensor has a rolling-shutter readout arranged in portrait orientation. Finally, the camera apparatus includes a controller configured to signal the second image sensor to start capturing the second image before the first image sensor finishes capturing the first image. At least a portion of the first image is in front of the second image relative to a forward direction of the camera apparatus.
- In a second embodiment, a method for panoramic photography includes the steps of: capturing a first image with a first image sensor and starting to capture a second image with a second image sensor prior to completion of the capturing of the first image. The first and second image sensors have a rolling-shutter readouts arranged in landscape orientation. At least a portion of the first image is in front of the second image relative to a forward direction of a camera apparatus comprising the first and second image sensors.
- In a third embodiment, a camera apparatus for motion photography includes a first camera having a first entrance pupil. During forward motion of the camera apparatus, the first camera captures a first image. A camera apparatus also includes a second camera having a second entrance pupil. During the forward motion of the camera apparatus, the second camera captures a second image. Motion of the camera apparatus results in motion parallax. The timing of when the first camera captures a first image relative to when the second camera captures the second image uses the motion parallax to reduce the effect of spatial parallax between the first camera and the second camera.
- In a fourth embodiment, a method for motion photography with a camera apparatus includes: capturing a first image with a first camera in the camera apparatus at a first time during forward motion of the camera apparatus, and capturing a second image with a second camera in the camera apparatus at a second time the during forward motion of the camera apparatus. Motion of the camera apparatus results in motion parallax. The timing of when the capturing (a) occurs relative to when the capturing (b) occurs uses the motion parallax to reduce the effect of spatial parallax between the first camera and the second camera.
- In this way, embodiments of the present invention reduce distortion and spatial parallax.
- Further embodiments, features, and advantages of the invention, as well as the structure and operation of the various embodiments of the invention are described in detail below with reference to accompanying drawings.
- The accompanying drawings, which are incorporated herein and form a part of the specification, illustrate the present invention and, together with the description, further serve to explain the principles of the invention and to enable a person skilled in the pertinent art to make and use the invention.
-
FIG. 1A includes a diagram illustrating a camera that includes a rolling-shutter CMOS sensor. -
FIG. 1B includes a diagram illustrating a camera having a rolling-shutter CMOS sensor with groups of photodiodes arranged in columns substantially perpendicular to the ground, according to an embodiment of the present invention. -
FIGs. 2A-C include diagrams illustrating a portion of a camera rosette which may be used in panoramic imaging, according to an embodiment of the present invention. -
FIG. 3 includes a diagram showing the operation of the camera rosette shown inFIGs. 2A-C in further detail. -
FIGs. 4A-B show how positioning of the entrance pupils of a camera impacts parallax. -
FIG. 5 shows a diagram illustrating timing the exposure of cameras in a camera rosette to reduce spatial parallax, according to an embodiment of the present invention. -
FIG. 6 shows a diagram of a camera rosette affixed to a vehicle. - The drawing in which an element first appears is typically indicated by the leftmost digit or digits in the corresponding reference number. In the drawings, like reference numbers may indicate identical or functionally similar elements.
- Embodiments of present invention reduce distortion and spatial parallax in panoramic images taken from a camera rosette affixed to a vehicle. Each camera in the camera rosette includes an image sensor. The image sensor may expose to capture an image while the vehicle is moving. This movement can cause distortion in the resulting image. In an embodiment of the present invention, the CMOS sensor is arranged in "portrait" orientation. As is described in detail below, this reduces the image distortion.
- Having multiple cameras in different locations on the camera rosette can cause spatial parallax. In a further embodiment of the present invention, the motion parallax of the vehicle is employed to cancel out spatial parallax. In that embodiment, as the vehicle moves, exposure of the cameras in the camera rosette is timed such that the entrance pupils of the cameras are in approximately the same location when each sensor is exposed. This embodiment is described in detail below.
- In the detailed description of the invention that follows, references to "one embodiment", "an embodiment", "an example embodiment", etc., indicate that the embodiment described may include a particular feature, structure, or characteristic, but every embodiment may not necessarily include the particular feature, structure, or characteristic. Moreover, such phrases are not necessarily referring to the same embodiment. Further, when a particular feature, structure, or characteristic is described in connection with an embodiment, it is submitted that it is within the knowledge of one skilled in the art to effect such feature, structure, or characteristic in connection with other embodiments whether or not explicitly described.
- The term "camera rosette" used herein refers to two or more cameras arranged to capture a panoramic image. Each camera captures an image. The images may be stitched together to form the panoramic image. This stitching may, for example, be done by well-known software techniques.
- The term "camera" used herein refers to a device that captures a photographic image. A camera includes an image sensor, and a lens having an entrance pupil.
- The term "image sensor" used herein refers to a device that converts optical signals into electrical signals to capture an image when exposed.
-
FIG. 1A includes a diagram 100 illustrating a camera that includes a rolling-shutter CMOS image sensor. Diagram 100 shows a camera and its image sensor at two points in time. At a first point in time, the camera is at aposition 102 and its image sensor is at aposition 104. At a second point in time, the camera is at aposition 112 and its image sensor is at aposition 114. The dot in the figure represents the entrance pupil of the lens, and rays are shown projecting through it in the manner of a pinhole camera for simplicity. - In an example, the camera may be attached to a vehicle, and movement of the vehicle may have caused the camera to move. As mentioned above, the image sensor may be a rolling shutter CMOS sensor. A rolling shutter CMOS sensor may include groups of photodiodes that capture light and convert the light into electrical signals during exposure. A rolling shutter CMOS sensor may expose different groups of photodiodes at different times. A rolling shutter CMOS sensor has a readout time. The readout time is the time necessary to read out all the groups of photodiodes at the ends of their respective exposure times. In the example shown in
FIG. 1A , the CMOS sensor is in "landscape" orientation, meaning that the groups of photodiodes are arranged as rows running parallel to the ground. - The rolling shutter sensor may be exposed while the vehicle is in motion. In an example, a rolling shutter sensor may require 100 ms of readout time to capture an image. While the image is captured, the vehicle and camera may have moved forward 1 m. The movement causes the objects captured by the image sensor to change as shown at 106. This causes distortion in the resulting image. Because different rows of photodiodes in the CMOS sensors are exposed at different times, the image may appear warped, e.g. vertical features may be slanted. An image sensor that does not have a rolling shutter and is exposed in its entirety, such as an interline-shutter CCD sensor, may reduce or avoid warping. However, as mentioned earlier, interline-shutter CCD sensors may suffer from blooming and streaking.
- To deal with the problem of warping due to rolling shutter sensors, embodiments of the present invention have rolling shutter image sensors arranged in "portrait" orientation as illustrated in
FIG. 1B. FIG. 1B includes a diagram 150 illustrating a camera having a rolling-shutter sensor, such as a CMOS sensor, in portrait orientation. Portrait orientation means that the groups of photodiodes are arranged in columns running perpendicular to the ground. - Diagram 150 shows a camera at different positions at different points in time. As mentioned earlier, the camera may be affixed to a vehicle, and the vehicle may be in motion. In an embodiment of the invention, the columns of photodiodes are exposed back-to-front in object space and front-to-back in image space.
- At a
position 152, the camera exposes aforward-most photodiode column 156 to capture an image including a back-most field ofview 154. Then, at aposition 160, the camera exposes aphotodiode column 158, which is behindphotodiode column 156. This exposure captures a field ofview 172, which is to the front of field ofview 154. As the camera moves to aposition 162 and aposition 164, columns continue to expose in a front-to-back manner, which captures objects from back-to-front. Finally, when the camera is at aposition 166, the camera exposes aback-most photodiode column 168 to capture an image including a forward-most field ofview 170. - Positioning the image sensor in portrait orientation avoids the warping of vertical features that occurs in landscape orientation. Instead, objects may appear stretched by an amount that depends on their distance. Stretching is not as visually unappealing as warping. In fact, stretching may be a positive feature. Stretching makes foreground objects thinner with respect to background objects, rather than fatter, when the rolling shutter is arranged in the direction described. As result, the foreground objects occlude less of the scene behind them. This may be a benefit in many applications related to panoramic imaging.
- Panoramic imaging often includes taking images from multiple cameras oriented in different directions. The multiple images may then be stitched together into a panorama using well-known software techniques. When multiple cameras are used in this way, an object may exist in the field of view of more than one camera. In motion photography, objects may move or change over time relative to the cameras. If multiple cameras expose the object at different times, then the object may appear differently in the multiple images to stitch together into a single panorama. The resulting panorama may have ghosting or other undesirable image problems. To deal with this, exposure needs to be coordinated between the multiple cameras to capture objects at the same time or close to the same time. This becomes even more challenging with rolling shutter image sensors because different portions of each image sensor are exposed as different times. To deal with this, embodiments of the present invention time exposure of multiple image sensors as illustrated in
FIGs. 2A-C . -
FIGs. 2A-C include diagrams illustrating a portion of a camera rosette, which may be used in panoramic imaging. Each camera in the camera rosette includes a rolling-shutter image sensor, such as a rolling-shutter CMOS sensor, in portrait orientation as described with respect toFIG. 1B . -
FIG. 2A includes a diagram 200. Diagram 200 shows a camera rosette withcameras Camera 202 hasphotodiode column 204 exposed to a field ofview 220. -
FIG. 2B includes a diagram 230. Diagram 230 shows the rosette shown inFIG. 2A at a later point in time.Camera 202 has a total field ofview 224, andcamera 210 has a total field ofview 234. Ascamera 202 scans across field ofview 224 back-to-front (in object space),camera 202 begins to expose photodiode columns from a group ofphotodiode columns 206. Group ofcolumns 206 captures objects within field ofview 222. Field ofview 222 overlaps withcamera 210's total field ofview 234 for distant objects. - When
camera 202 begins to expose photodiode columns from group ofcolumns 206,camera 210 begins to expose photodiode columns from a group ofphotodiode columns 208.Camera 210 exposes photodiode columns in group ofcolumns 208 back-to-front (in object space) to capture field ofview 226. Field ofview 226 overlaps withcamera 202's total field ofview 224 for distant objects. Effectively,camera 202 exposes photodiode columns in group ofcolumns 206 simultaneously withcamera 210 exposing photodiode columns in group ofcolumns 208. Whencamera 202 completes exposure of group ofcolumns 206,camera 202 finishes its exposure. However,camera 210 continues to expose photodiode columns, as shownFIG. 2C . -
FIG. 2C shows a diagram 240. Diagram 240 shows the rosette shown inFIG. 2B at a later point in time aftercamera 202 has completed its exposure. In diagram 240, aphotodiode column 232 is exposed, capturing a field ofview 228. Ascamera 210 continues its exposure, it may overlap with an field of view with another camera (not shown) and the process may continue for an entire camera rosette as is described with respect toFIG. 3 . - The embodiment illustrated in
FIGs. 2A-C is described with respect to an illustrative example. In an example, overlap field ofview 222 for distant objects betweencamera 202 andcamera 210 may be 10% of total field ofview 224 ofcamera 202. In that example, the delay between the time whencamera 202 starts exposure and the time whencamera 210 starts exposure may be 90% ofcamera 202 readout time.Camera camera 202 starts exposure and the time whencamera 210 starts exposure may be 90 ms. - The camera rosette may have at least one controller that synchronizes the cameras to capture images at specified time offsets. So, the controller may be preprogrammed with the delays indicating when each camera starts to capture an image.
- In another embodiment, the delay time may be offset according to the vehicle's velocity. As is described below with respect to
FIG. 5 , this has the effect of using motion parallax from movement of the vehicle to cancel out spatial parallax caused by the multiple cameras. - As mentioned above, the operation of the two cameras described with respect to
FIGs. 2A-C may continue through an entire camera rosette. In one embodiment, a camera rosette may have a total of between six and ten cameras, inclusive, positioned to capture a 360 degree panorama. In a preferred embodiment, the camera rosette may have at least eight cameras, such as the rosette illustrated with respect toFIG. 3 . -
FIG. 3 shows a diagram 300 illustrating the operation of a camera rosette according to an embodiment of the present invention in further detail. In a preferred embodiment, a camera rosette may include nine cameras--eight in a horizontal ring, and one directed straight up. Diagram 300 shows the eight horizontal cameras. Each camera in the camera rosette includes a rolling shutter sensor in portrait orientation with columns of photodiodes. - The camera rosette begins capturing a panoramic image at the
rear-most cameras cameras FIG. 1B .Cameras FIG. 2B . - The rosette continues to scan forward until the
forward-most cameras camera 302 starts at a time defined by the start ofcamera 303 plus a delay. The delay may correspond to the size of the overlap in the field of view of adjacent cameras, as described with respect toFIG. 2B . The delay may also be offset according to the velocity of the vehicle, as is described with respect toFIG. 5 . - Each of the eight cameras in
FIG. 3 is at a different location and has an entrance pupil at a different location. Having entrance pupils at different locations causes spatial parallax as illustrated inFIGs. 4A-B . -
FIGs. 4A-B show how the positioning of the entrance pupils of two cameras impacts spatial parallax.FIG. 4A shows anexample imaging apparatus 400.Imaging apparatus 400 includes acamera 402 and acamera 404.Camera 402 has anentrance pupil 406, andcamera 404 has anentrance pupil 408.Entrance pupil 406 andentrance pupil 408 are separated by adistance 426. Eachcamera object 410. However there is a wide difference in the background ofobject 410 as it appears incameras large parallax angle 422. -
FIG. 4B shows anexample imaging apparatus 450.Imaging apparatus 450 includes acamera 416 and acamera 414.Camera 416 has anentrance pupil 418, andcamera 414 has anentrance pupil 420.Entrance pupil 418 andentrance pupil 420 are separated by adistance 428.Distance 428 is shorter thandistance 426 inFIG. 4A . Eachcamera object 412. There is a small difference in the background ofobject 412 as it appears incameras cameras FIG. 4A . This is shown by theparallax angle 424. Therefore,parallax angle 422 inFIG. 4B is less thanparallax angle 424 inFIG. 4A . - Thus, two cameras being in different places can cause spatial parallax. Similarly, motion parallax is caused by a camera being in different locations at different times. According to an embodiment of the present invention, motion parallax caused by motion of a camera rosette by cancel out or reduce the effect of spatial parallax caused by the entrance pupils of the different cameras on the camera rosette being located at different locations. This embodiment is illustrated in
FIG. 5 . -
FIG. 5 shows a diagram 500 illustrating timing the exposure of cameras in a camera rosette to reduce parallax, according to an embodiment of the present invention. As shown inFIG 4A-B , as camera entrance pupils get closer together, the spatial parallax between the cameras decreases. Diagram 500 shows how to time the exposure of the cameras during motion of the vehicle so that each of the cameras is exposed when its entrance pupil is located at approximately the same location. As result, in diagram 500 the motion parallax caused by the moving vehicle reduces the spatial parallax due to multiple cameras. - Diagram 500 shows a portion of a camera rosette at three
positions cameras Cameras entrance pupils - In the embodiment in
FIG. 5 , the image sensors have rolling shutters. The embodiment is useful with rolling shutters. However, in an example one may also reduce parallax with image sensors having "global" shutter, such as an interline CCD or an image sensor using a mechanical shutter. The image capture is described as happening at an instant, even though there will be some exposure duration and a small motion blur as a result. Atposition 510,camera 502 captures an image. Atposition 510,camera 502'sentrance pupil 508 is at alocation 524. Each subsequent camera is timed to take an image when that camera's entrance pupil is approximately located atlocation 524. - During a later point in time as the vehicle is moving, the camera rosette reaches
position 520. Atposition 520,entrance pupil 514 ofcamera 504 is approximately atlocation 524. At this point,camera 504 captures an image. Finally, atposition 530entrance pupil 512 ofcamera 506 reaches approximatelylocation 524, andcamera 506 captures an image. - The approximation may be due to inaccuracies in the vehicles velocity, or changes to the vehicle's direction. In an example, the approximation may be based on an average vehicle velocity as opposed to the actual velocity over the relevant period. Also, the approximation may be due to the fact that cameras are positioned two or three dimensionally on the camera rosette, whereas the movement of the rosette (e.g., on a vehicle) may be in only one direction. For example, in
FIG. 5 entrance pupil 508 is not at precisely the same location asentrance pupil 514 becauseentrance pupil 508 areentrance pupil 514 are at different locations on the axis perpendicular to the axis of motion. - In another embodiment, a virtual plane is perpendicular to the direction of motion of a camera rosette. Each camera in a camera rosette may expose when an entrance pupil of the camera passes through the plane. In this way, each camera exposes when its entrance pupil is at approximately the same location, reducing parallax.
- As noted earlier, the time to trigger each subsequent camera may be calculated according to vehicle velocity and a distance between the cameras, such as a
distance 522. In the example shown, the delay from a time whencamera 504 captures an image and a time whencamera 506 captures an image isdistance 522 divided by the vehicle's velocity. Supposingdistance 522 was 10 cm and the vehicle velocity was 10 m/s, then the delay between the time whencamera 504 captures an image and the time whencamera 506 captures an image would be 10 ms. In other words, the further rear camera (camera 506) starts to capture an image 10 ms after the more forward camera (camera 504). - As mentioned earlier, a vehicle velocity used to time the cameras may be an average vehicle velocity. Alternatively, the vehicle velocity may be determined in real time by a speedometer, a wheel encoder, or a GPS sensor. The velocity may be signaled to controllers in the cameras, which adjust the timing accordingly.
- In a preferred embodiment, the features shown in diagram 500 may be used in conjunction with the features described with respect to
FIGs. 2A-C. FIG. 2B describes timing camera exposure in a camera rosette where each camera includes a rolling-shutter CMOS sensor in portrait orientation. In that embodiment, each camera is timed to start exposure when the previous camera begins to scan an region where their fields of view overlap at a distance. Describing the embodiment with respect toFIG. 2B , a delay between whencamera 202 starts exposure and whencamera 210 starts exposure substantially satisfies the equation: - where t is the delay time, R1 is a rolling-shutter readout time of the first image sensor, F1 is a percentage of the field of view of the first image sensor which does not overlap with the field of view of the second image sensor, D12 is a distance between an entrance pupil of the first image sensor and an entrance pupil of the second image sensor, and v is a speed of the camera apparatus. It would be recognized that small changes such as small offsets and coefficients may be added to this equation.
- In an example given with respect to
FIG. 2B , overlap field ofview 222 betweencamera 202 andcamera 210 may be 10% of the total field of view ofcamera 202. In that example, 90% ofcamera 202's field of view does not overlap withcamera 210's field of view. In an example wherecamera 202's readout time is 100 ms, the delay between the time whencamera 202 starts exposure and the time whencamera 210 starts exposure may be 90 ms. In other words, the more forward camera (camera 210) starts to capture an image 90 ms after the further rear camera starts to capture an image (camera 202). As a result, cameras with overlapping fields of view capture images of objects in their overlap region at the same time, or from the same vehicle position. This timing is a first-order correction for parallax, in that it prevents the large parallax errors that would result if the objects in the overlap area were captured at very different times and very different vehicle positions, as would happen if the cameras started their rolling shutters simultaneously. However, there remains a smaller parallax error due to the entrance pupils of the cameras not being in the same place. - In an example of the embodiment incorporating both features in diagram 500 and
FIGs. 2A-C , the delay time is offset to further reduce spatial parallax. In the example given with respect toFIG. 5 , the further rear camera (camera 506) starts to capture an image 10 ms after the more forward camera (camera 504). In the example inFIGs. 2A-C , the more forward camera (camera 210) starts to capture an image 90 ms after the further rear camera starts to capture an image (camera 202). In the combined embodiment, the 90 ms delay is offset by 10 ms. Therefore, the more forward camera starts to capture an image 80 ms after the further rear camera starts to capture an image. This embodiment has reduced spatial parallax, accounting jointly for rolling-shutter readout delay, vehicle velocity, and entrance pupil separation. - In another embodiment, the timing offsets for reduced spatial parallax may only be applied to some of the cameras in a camera rosette. In an example, the offset may be applied to only those cameras pointed sideways from the vehicle motion. The cameras pointed sideways may be the cameras where typical vehicle motion moves the location of the entrance pupil of the more rear camera into the location where the camera took an image in, for example, 0.01 s or less. In this example, spatial parallax of forward-looking and backward-looking portions of the panorama may not be cancelled by motion parallax, but objects in these directions tend to be distant enough to not have a large parallax problem.
- As an alternative embodiment, a rolling shutter with a very fast readout time may be useful. As the shutter readout time becomes faster, the delay computed to account for the rolling shutter (as described in
FIGs. 2A-C ) decreases. So, at a particular readout time, the delay for the rolling shutter and the offset to account for spatial parallax may cancel each other out. This readout time is the time it takes for the vehicle to move forward a distance equal to the distance between entrance pupils. In an example, a readout time of a first image sensor may be 10 ms. Ninety percent of the first image sensor's field of view may not overlap with a second, next forward-most image sensor. If the vehicle is stationary, the delay time between exposure of the first and second image sensor would be 9 ms. However, if the vehicle is moving at 9 m/s and the distance between the first and second image sensor is 10 cm, then the delay offset due to motion of the vehicle would also be 9 ms. Thus, the offsets would effectively cancel each other out. Similarly, an image sensor with a variable rolling readout time may be used. In that instance, the readout time may be adjusted according to a speed of the vehicle. -
FIG. 6 shows acamera rosette 604 affixed to avehicle 602.Camera rosette 604 includes numerous cameras, such as acamera 606. In an example,camera rosette 604 may be used to capture panoramic images of buildings running along a street. - Each of the cameras in the camera rosette is coupled to a controller 608. As mentioned earlier, controller 608 controls the timing of the exposure of each camera. Controller 608 may receive an input indicating a speed of the vehicle. Alternatively, controller 608 may be preprogrammed with timing offsets, such as offsets based on an average speed of the vehicle. A single controller 608 may be used. Alternatively, each camera may have its own controller.
- Controller 608 may be implemented in hardware, software, firmware, or any combination thereof. Controller 608 may be, for example, a general purpose computer running software to control the cameras' exposure. Controller 608 may have a processor and memory.
- The Summary and Abstract sections may set forth one or more but not all exemplary embodiments of the present invention as contemplated by the inventor(s), and thus, are not intended to limit the present invention and the appended claims in any way.
- The foregoing description of the specific embodiments will so fully reveal the general nature of the invention that others can, by applying knowledge within the skill of the art, readily modify and/or adapt for various applications such specific embodiments, without undue experimentation, without departing from the general concept of the present invention. Therefore, such adaptations and modifications are intended to be within the meaning and range of equivalents of the disclosed embodiments, based on the teaching and guidance presented herein. It is to be understood that the phraseology or terminology herein is for the purpose of description and not of limitation, such that the terminology or phraseology of the present specification is to be interpreted by the skilled artisan in light of the teachings and guidance.
- The breadth and scope of the present invention should not be limited by any of the above-described exemplary embodiments, but should be defined only in accordance with the following claims.
- Once aspect of the present invention provides camera apparatus for motion photography, comprising a first camera having a first entrance pupil wherein, during forward motion of the camera apparatus, the first camera captures a first image; and a second camera having a second entrance pupil wherein, during the forward motion of the camera apparatus, the second camera captures a second image, wherein motion of the camera apparatus results in motion parallax, and wherein the timing of when the first camera captures a first image relative to when the second camera captures the second image uses the motion parallax to reduce the effect of spatial parallax between the first camera and the second camera.
- In one embodiment during forward motion of the camera apparatus, the first camera captures a first image when the first entrance pupil is at a first location and the second camera captures a second image when the second entrance pupil is at a second location, wherein the second location is approximately equal to the first location.
- In one embodiment the first and second cameras are attached to a vehicle.
- In one embodiment the apparatus comprises at least six additional image sensors, wherein a computer is able to stitch together images taken from the at least eight total cameras to create a panoramic image.
- Another aspect of the present invention provides a method for motion photography with a camera apparatus, comprising:
- (a) capturing a first image with a first camera in the camera apparatus at a first time during forward motion of the camera apparatus; and
- (b) capturing a second image with a second camera in the camera apparatus at a second time during forward motion of the camera apparatus, wherein motion of the camera apparatus results in motion parallax, wherein the timing of when the capturing (a) occurs relative to when the capturing (b) occurs uses the motion parallax to reduce the effect of spatial parallax between the first camera and the second camera.
- In one embodiment the capturing of the first image (a) comprises capturing the first image when a first entrance pupil of the first camera is at a first location, and wherein the capturing the second image (b) comprises capturing the second image when the second entrance pupil is at a second location, wherein the second location is approximately equal to the first location.
- Another aspect of the present invention provides apparatus for motion photography with a camera apparatus, comprising a means for capturing a first image with a first camera in the camera apparatus at a first time during forward motion of the camera apparatus; and a means for capturing a second image with a second camera in the camera apparatus at a second time during forward motion of the camera apparatus, wherein motion of the camera apparatus results in motion parallax, wherein the timing of when the capturing of the first image occurs relative to when the capturing of the second image occurs uses the motion parallax to reduce the effect of spatial parallax between the first camera and the second camera.
Claims (14)
- A camera apparatus for panoramic photography, comprising:first and second image sensors (202, 210)a means for capturing a first image with the first image sensor (202), the first image sensor (202) having a rolling-shutter readout arranged to sequentially activate columns of photodiodes running perpendicular to the ground; anda means for starting to capture a second image with the second image sensor (210) at a delay time prior to completion of the capturing the first image, the delay time being set dependent upon an amount of overlap of a field of view (224) of the first image sensor (202) and a field of view (234) of the second image sensor (210) such that ghosting between the first and second images is prevented, wherein the second image sensor (210) has a rolling-shutter readout arranged to sequentially activate columns of photodiodes running perpendicular to the ground,wherein at least a portion of the first image is in front of the second image relative to a forward direction of the camera apparatus running parallel to the ground.
- The apparatus of claim 1, wherein the means for starting to capture a second image is configured to signal the second image sensor (210) to start capturing the second image approximately when the first image sensor (202) starts to capture an overlap of the fields of view of the first and second image sensors (202, 210).
- The apparatus of claim 1, wherein the delay time is offset in dependence upon a velocity value of the camera apparatus whereby motion parallax resulting from motion of the camera apparatus cancels out spatial parallax between the first and second image sensors (202, 210).
- The apparatus of claim 3, wherein the velocity value of the camera apparatus is an approximate or average velocity of the camera apparatus.
- The apparatus of claim 3, wherein the delay time is offset by a time between when an entrance pupil of the first image sensor (202) passes a location and when an entrance pupil of the second image sensor (210) passes the location during forward motion of the camera apparatus.
- The apparatus of claim 3, wherein the delay time approximately satisfies the following equation:wherein t is the delay time,wherein R1 is a rolling-shutter readout time of the first image sensor (202),wherein F1 is a percentage of the field of view (224) of the first image sensor (202) which does not overlap with the field of view (234) of the second image sensor (210),wherein D12 is a distance between an entrance pupil of the first image sensor (202) and an entrance pupil of the second image sensor (210), andwherein v is a speed of the camera apparatus.
- The apparatus of claim 1, wherein the first and second image sensors (202, 210) are attached to a vehicle.
- The apparatus of claim 1, wherein the first image sensor (202) is arranged to expose at least a first and second column of the first image sensor (202), the first column capturing a more rearward portion of the first image and the second column capturing a more forward portion of the first image relative to forward motion of the camera apparatus,
wherein the first column of the first image sensor (202) is arranged to start exposure before the second column of the first image sensor (202),
wherein the second image sensor (210) is arranged to expose at least a first and second column of the second image sensor (210), the first column capturing a more rearward portion of the second image and the second column capturing a more forward portion of the second image relative to the forward motion of the camera apparatus, and
wherein the first column of the second image sensor (210) is arranged to start exposure before the second column of the second image sensor (210). - The apparatus of claim 1, further comprising at least six additional image sensors.
- The apparatus of claim 1, wherein the first and second image sensors (202, 210) are CMOS image sensors.
- A method for panoramic photography, comprising:capturing a first image with a first image sensor (202), the first image sensor (202) having a rolling-shutter readout arranged to sequentially activate columns of photodiodes running perpendicular to the ground; andstarting to capture a second image with a second image sensor (210) at a delay time prior to completion of the capturing of the first image, the delay time set dependent upon an amount of overlap of a field of view of the first image sensor (202) and a field of view of the second image sensor (210) to prevent ghosting between the first and second images, wherein the second image sensor (210) has a rolling-shutter readout arranged to sequentially activate columns of photodiodes running perpendicular to the ground , wherein at least a portion of the first image is in front of the second image relative to a forward direction running parallel to the ground of a camera apparatus comprising the first and second image sensors (202, 210).
- The method of claim 11, wherein the delay time is offset in dependence upon a velocity value of the camera apparatus, whereby motion parallax resulting from motion of the camera apparatus to cancel out spatial parallax between the first and second image sensors (202, 210).
- The method of Claim 12, wherein the delay time is offset by a time between when an entrance pupil of the first image sensor (202) passes a location and when an entrance pupil of the second image sensor (210) passes the location during forward motion of the camera apparatus.
- The method of Claim 12, wherein the delay time approximately satisfies the following equation:wherein t is the delay time,wherein R1 is a rolling-shutter readout time of the first Image sensor (202),wherein F1 is a percentage of the field of view (224) of the first image sensor (202) which does not overlap with the field of view (234) of the second image sensor (210),wherein D12 is a distance between an entrance pupil of the first image sensor (202) and an entrance pupil of the second image sensor (210), andwherein v is a speed of the camera apparatus.
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US2723708P | 2008-02-08 | 2008-02-08 | |
PCT/US2009/000804 WO2009099667A1 (en) | 2008-02-08 | 2009-02-09 | Panoramic camera with multiple image sensors using timed shutters |
Publications (2)
Publication Number | Publication Date |
---|---|
EP2253131A1 EP2253131A1 (en) | 2010-11-24 |
EP2253131B1 true EP2253131B1 (en) | 2014-06-25 |
Family
ID=40561834
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP09708701.9A Active EP2253131B1 (en) | 2008-02-08 | 2009-02-09 | Panoramic camera with multiple image sensors using timed shutters |
Country Status (7)
Country | Link |
---|---|
US (5) | US8493436B2 (en) |
EP (1) | EP2253131B1 (en) |
JP (2) | JP5337170B2 (en) |
CN (1) | CN102037720B (en) |
AU (1) | AU2009210672B2 (en) |
CA (1) | CA2714492C (en) |
WO (1) | WO2009099667A1 (en) |
Families Citing this family (54)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP2253131B1 (en) * | 2008-02-08 | 2014-06-25 | Google, Inc. | Panoramic camera with multiple image sensors using timed shutters |
US7974314B2 (en) * | 2009-01-16 | 2011-07-05 | Microsoft Corporation | Synchronization of multiple data source to a common time base |
US8698875B2 (en) * | 2009-02-20 | 2014-04-15 | Google Inc. | Estimation of panoramic camera orientation relative to a vehicle coordinate frame |
CN101852979A (en) * | 2009-03-30 | 2010-10-06 | 鸿富锦精密工业(深圳)有限公司 | Panoramic camera |
US8862987B2 (en) * | 2009-03-31 | 2014-10-14 | Intel Corporation | Capture and display of digital images based on related metadata |
US20100271533A1 (en) * | 2009-04-28 | 2010-10-28 | Crispin Porter & Bogusky | Image Recordation Device with a Plurality of Lenses |
JP2011119785A (en) * | 2009-11-30 | 2011-06-16 | Fujitsu Ltd | Camera control apparatus, and method for controlling camera |
CN101877775A (en) * | 2010-04-06 | 2010-11-03 | 中兴通讯股份有限公司 | Telepresence system and camera group thereof |
WO2012089895A1 (en) * | 2010-12-31 | 2012-07-05 | Multitouch Oy | Rolling shutter compensation in camera array |
US20120320204A1 (en) * | 2011-06-20 | 2012-12-20 | 3M Innovative Properties Company | Asset assessment system |
TWI526706B (en) * | 2011-10-05 | 2016-03-21 | 原相科技股份有限公司 | Image system |
JP5860663B2 (en) * | 2011-10-18 | 2016-02-16 | 日立オートモティブシステムズ株式会社 | Stereo imaging device |
US8937643B1 (en) * | 2012-02-28 | 2015-01-20 | Carlos Esteban | Offset rolling shutter camera model, and applications thereof |
US20130278715A1 (en) * | 2012-03-16 | 2013-10-24 | Mark Nutsch | System and method for discreetly collecting 3d immersive/panoramic imagery |
US9476970B1 (en) * | 2012-03-19 | 2016-10-25 | Google Inc. | Camera based localization |
NL2008639C2 (en) * | 2012-04-13 | 2013-10-16 | Cyclomedia Technology B V | Device, system and vehicle for recording panoramic images, and a device and method for panoramic projection thereof. |
US8941777B1 (en) * | 2012-10-31 | 2015-01-27 | Google Inc. | System and method for capturing images of a geographic area |
FR2998126B1 (en) * | 2012-11-15 | 2014-12-26 | Giroptic | METHOD AND DEVICE FOR CAPTURING AND CONSTRUCTING A FLOW OF PANORAMIC OR STEREOSCOPIC IMAGES |
CN104969238B (en) | 2013-01-15 | 2019-06-18 | 移动眼视力科技有限公司 | Vehicle and imaging system for vehicle |
US10179543B2 (en) | 2013-02-27 | 2019-01-15 | Magna Electronics Inc. | Multi-camera dynamic top view vision system |
US9065985B2 (en) | 2013-03-15 | 2015-06-23 | Tolo, Inc. | Diagonal collection of oblique imagery |
KR102077498B1 (en) * | 2013-05-13 | 2020-02-17 | 한국전자통신연구원 | Movement path extraction devices of mutual geometric relations fixed camera group and the method |
US9742991B2 (en) * | 2013-05-24 | 2017-08-22 | Robert Frank Latorre | 360 degree photobooth kiosk |
US9851527B2 (en) * | 2013-10-18 | 2017-12-26 | Light Labs Inc. | Methods and apparatus for capturing and/or combining images |
US8811812B1 (en) | 2014-03-27 | 2014-08-19 | Michael Shawn Lawler | Camera rig |
WO2015171544A1 (en) | 2014-05-06 | 2015-11-12 | Niazi Zakariya | Imaging system, method, and applications |
JP6385212B2 (en) * | 2014-09-09 | 2018-09-05 | キヤノン株式会社 | Image processing apparatus and method, imaging apparatus, and image generation apparatus |
EP3034359B1 (en) | 2014-12-15 | 2018-05-16 | Ricoh Company, Ltd. | Monitoring system, and vehicle mountable with monitoring system |
EP3043202B1 (en) | 2015-01-09 | 2019-07-24 | Ricoh Company, Ltd. | Moving body system |
KR102362138B1 (en) | 2015-07-23 | 2022-02-14 | 삼성전자주식회사 | Image sensor module and image sensor device including the same |
DE102015214900A1 (en) * | 2015-08-05 | 2017-02-09 | Robert Bosch Gmbh | Method and device for generating delay signals for a multi-camera system and generating fused image data for a multi-camera system for a vehicle and multi-camera system |
RU2592855C1 (en) * | 2015-08-25 | 2016-07-27 | Вячеслав Михайлович Смелков | Device for panoramic television-computer monitoring |
US10694249B2 (en) | 2015-09-09 | 2020-06-23 | Vantrix Corporation | Method and system for selective content processing based on a panoramic camera and a virtual-reality headset |
US11287653B2 (en) | 2015-09-09 | 2022-03-29 | Vantrix Corporation | Method and system for selective content processing based on a panoramic camera and a virtual-reality headset |
US10506006B2 (en) | 2015-09-09 | 2019-12-10 | Vantrix Corporation | Method and system for flow-rate regulation in a content-controlled streaming network |
US10419770B2 (en) | 2015-09-09 | 2019-09-17 | Vantrix Corporation | Method and system for panoramic multimedia streaming |
US11108670B2 (en) | 2015-09-09 | 2021-08-31 | Vantrix Corporation | Streaming network adapted to content selection |
US10033928B1 (en) * | 2015-10-29 | 2018-07-24 | Gopro, Inc. | Apparatus and methods for rolling shutter compensation for multi-camera systems |
WO2017149875A1 (en) | 2016-02-29 | 2017-09-08 | ソニー株式会社 | Image capture control device, image capture device, and image capture control method |
US10044963B2 (en) * | 2016-03-30 | 2018-08-07 | Panasonic Intellectual Property Management Co., Ltd. | Imaging apparatus |
US10110813B1 (en) * | 2016-04-27 | 2018-10-23 | Ambarella, Inc. | Multi-sensor camera using rolling shutter sensors |
JP2018046430A (en) | 2016-09-15 | 2018-03-22 | ソニー株式会社 | Information processing device, method, and program |
US10527925B2 (en) | 2017-05-16 | 2020-01-07 | Nico Toutenhoofd | Fully-spherical imaging system, camera support for same, and associated methods |
KR101964919B1 (en) * | 2017-05-26 | 2019-08-13 | 주식회사 만도 | Method and Apparatus for controlling parking of vehicle |
DE102017215347A1 (en) * | 2017-09-01 | 2019-03-07 | Conti Temic Microelectronic Gmbh | Method for the predictable exposure control of at least a first vehicle camera |
JP2019062479A (en) * | 2017-09-27 | 2019-04-18 | 京セラ株式会社 | Imaging apparatus and movable body |
US10841496B2 (en) * | 2017-10-19 | 2020-11-17 | DeepMap Inc. | Lidar to camera calibration based on edge detection |
US10582181B2 (en) | 2018-03-27 | 2020-03-03 | Honeywell International Inc. | Panoramic vision system with parallax mitigation |
US11181619B2 (en) | 2018-06-14 | 2021-11-23 | Waymo Llc | Camera ring structure for autonomous vehicles |
KR102569375B1 (en) | 2018-10-24 | 2023-08-22 | 삼성전자주식회사 | Electronic device and controlling method thereof |
KR20200053125A (en) | 2018-11-08 | 2020-05-18 | 삼성전자주식회사 | Electronic device and control method thereof |
CN112308783A (en) | 2019-07-24 | 2021-02-02 | 株式会社理光 | Rolling effect correction method and device and computer readable storage medium |
JP2022021367A (en) * | 2020-07-22 | 2022-02-03 | キヤノン株式会社 | Imaging system, manufacturing system, information processing method, method for manufacturing article, program, and recording medium |
US11665330B2 (en) | 2021-01-27 | 2023-05-30 | Dell Products L.P. | Dynamic-baseline imaging array with real-time spatial data capture and fusion |
Family Cites Families (86)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPS51142212A (en) * | 1975-06-02 | 1976-12-07 | Hokkaido Daigaku | Tridimensional television system |
US4532550A (en) * | 1984-01-31 | 1985-07-30 | Rca Corporation | Exposure time control for a solid-state color camera |
US5023723A (en) * | 1988-04-18 | 1991-06-11 | Canon Kabushiki Kaisha | Image sensing apparatus having plural image sensors and plural shutters |
JP2674135B2 (en) * | 1988-09-20 | 1997-11-12 | 株式会社ニコン | Electronic still camera |
US5990941A (en) * | 1991-05-13 | 1999-11-23 | Interactive Pictures Corporation | Method and apparatus for the interactive display of any portion of a spherical image |
US5495576A (en) * | 1993-01-11 | 1996-02-27 | Ritchey; Kurtis J. | Panoramic image based virtual reality/telepresence audio-visual system and method |
EP0888017A2 (en) * | 1993-08-26 | 1998-12-30 | Matsushita Electric Industrial Co., Ltd. | Stereoscopic image display apparatus and related system |
US7630806B2 (en) * | 1994-05-23 | 2009-12-08 | Automotive Technologies International, Inc. | System and method for detecting and protecting pedestrians |
US6118475A (en) * | 1994-06-02 | 2000-09-12 | Canon Kabushiki Kaisha | Multi-eye image pickup apparatus, and method and apparatus for measuring or recognizing three-dimensional shape |
US5473364A (en) * | 1994-06-03 | 1995-12-05 | David Sarnoff Research Center, Inc. | Video technique for indicating moving objects from a movable platform |
EP1494476A1 (en) * | 1994-09-09 | 2005-01-05 | Canon Kabushiki Kaisha | Image processing method and apparatus |
US5703961A (en) * | 1994-12-29 | 1997-12-30 | Worldscape L.L.C. | Image transformation and synthesis methods |
US6327381B1 (en) * | 1994-12-29 | 2001-12-04 | Worldscape, Llc | Image transformation and synthesis methods |
US5657073A (en) * | 1995-06-01 | 1997-08-12 | Panoramic Viewing Systems, Inc. | Seamless multi-camera panoramic imaging with distortion correction and selectable field of view |
US6141034A (en) * | 1995-12-15 | 2000-10-31 | Immersive Media Co. | Immersive imaging method and apparatus |
US6108440A (en) * | 1996-06-28 | 2000-08-22 | Sony Corporation | Image data converting method |
WO1999058927A1 (en) * | 1998-05-08 | 1999-11-18 | Sony Corporation | Image generating device and method |
US20010015751A1 (en) * | 1998-06-16 | 2001-08-23 | Genex Technologies, Inc. | Method and apparatus for omnidirectional imaging |
EP2259220A3 (en) * | 1998-07-31 | 2012-09-26 | Panasonic Corporation | Method and apparatus for displaying image |
US7015954B1 (en) * | 1999-08-09 | 2006-03-21 | Fuji Xerox Co., Ltd. | Automatic video system using multiple cameras |
JP3298851B2 (en) * | 1999-08-18 | 2002-07-08 | 松下電器産業株式会社 | Multi-function vehicle camera system and image display method of multi-function vehicle camera |
US6831677B2 (en) * | 2000-02-24 | 2004-12-14 | Yissum Research Development Company Of The Hebrew University Of Jerusalem | System and method for facilitating the adjustment of disparity in a stereoscopic panoramic image pair |
US6993159B1 (en) * | 1999-09-20 | 2006-01-31 | Matsushita Electric Industrial Co., Ltd. | Driving support system |
US6535114B1 (en) * | 2000-03-22 | 2003-03-18 | Toyota Jidosha Kabushiki Kaisha | Method and apparatus for environment recognition |
US6669346B2 (en) * | 2000-05-15 | 2003-12-30 | Darrell J. Metcalf | Large-audience, positionable imaging and display system for exhibiting panoramic imagery, and multimedia content featuring a circularity of action |
US6778207B1 (en) * | 2000-08-07 | 2004-08-17 | Koninklijke Philips Electronics N.V. | Fast digital pan tilt zoom video |
US6853809B2 (en) * | 2001-01-30 | 2005-02-08 | Koninklijke Philips Electronics N.V. | Camera system for providing instant switching between wide angle and full resolution views of a subject |
US20020124260A1 (en) * | 2001-03-02 | 2002-09-05 | Creative Design Group, Inc. | Video production system for vehicles |
US7012637B1 (en) * | 2001-07-27 | 2006-03-14 | Be Here Corporation | Capture structure for alignment of multi-camera capture systems |
US6947059B2 (en) * | 2001-08-10 | 2005-09-20 | Micoy Corporation | Stereoscopic panoramic image capture device |
US20040164249A1 (en) * | 2003-02-26 | 2004-08-26 | Crosetto Dario B. | Method and apparatus for determining depth of interactions in a detector for three-dimensional complete body screening |
US7224382B2 (en) * | 2002-04-12 | 2007-05-29 | Image Masters, Inc. | Immersive imaging system |
US20040001138A1 (en) * | 2002-06-27 | 2004-01-01 | Weerashinghe W.A. Chaminda P. | Stereoscopic panoramic video generation system |
KR20050046822A (en) * | 2002-10-18 | 2005-05-18 | 사르노프 코포레이션 | Method and system to allow panoramic visualization using multiple cameras |
US7388598B2 (en) * | 2003-02-14 | 2008-06-17 | Image Ensemble, Inc. | 3D camera system and method |
JP4346938B2 (en) * | 2003-04-09 | 2009-10-21 | キヤノン株式会社 | Image processing apparatus, method, program, and storage medium |
US20040212723A1 (en) * | 2003-04-22 | 2004-10-28 | Malcolm Lin | Image pickup apparatus and operating method |
JP2005045552A (en) * | 2003-07-22 | 2005-02-17 | Canon Inc | Imaging device and its method |
US7436429B2 (en) * | 2003-11-24 | 2008-10-14 | The Boeing Company | Virtual pan/tilt camera system and method for vehicles |
US20070182812A1 (en) * | 2004-05-19 | 2007-08-09 | Ritchey Kurtis J | Panoramic image-based virtual reality/telepresence audio-visual system and method |
US7697028B1 (en) * | 2004-06-24 | 2010-04-13 | Johnson Douglas M | Vehicle mounted surveillance system |
US8027531B2 (en) * | 2004-07-21 | 2011-09-27 | The Board Of Trustees Of The Leland Stanford Junior University | Apparatus and method for capturing a scene using staggered triggering of dense camera arrays |
US7629995B2 (en) * | 2004-08-06 | 2009-12-08 | Sony Corporation | System and method for correlating camera views |
US7750936B2 (en) * | 2004-08-06 | 2010-07-06 | Sony Corporation | Immersive surveillance system interface |
US20060028550A1 (en) * | 2004-08-06 | 2006-02-09 | Palmer Robert G Jr | Surveillance system and method |
JP4531484B2 (en) * | 2004-08-16 | 2010-08-25 | パナソニック株式会社 | Camera system |
JP4325557B2 (en) * | 2005-01-04 | 2009-09-02 | ソニー株式会社 | Imaging apparatus and imaging method |
JP4551990B2 (en) | 2005-02-03 | 2010-09-29 | 名古屋市 | Panorama video creation method and creation device |
KR100715026B1 (en) * | 2005-05-26 | 2007-05-09 | 한국과학기술원 | Apparatus for providing panoramic stereo images with one camera |
JP4512690B2 (en) | 2005-06-16 | 2010-07-28 | 独立行政法人産業技術総合研究所 | Monitoring system and method by image processing |
JP2007028008A (en) * | 2005-07-13 | 2007-02-01 | Konica Minolta Photo Imaging Inc | Imaging apparatus, imaging system, and operating program thereof |
TW200715830A (en) * | 2005-10-07 | 2007-04-16 | Sony Taiwan Ltd | Image pick-up device of multiple lens camera system to create panoramic image |
EP1938577B1 (en) | 2005-10-21 | 2013-08-14 | Nokia Corporation | A method and a device for reducing motion distortion in digital imaging |
JP4780385B2 (en) * | 2005-11-17 | 2011-09-28 | アイシン精機株式会社 | Perimeter monitoring system |
JP4724539B2 (en) * | 2005-11-24 | 2011-07-13 | キヤノン株式会社 | ACTUAL EXPOSURE TIME CALCULATION METHOD AND IMAGING DEVICE, OPTICAL DEVICE, CAMERA SYSTEM AND ACTUAL EXPOSURE TIME MEASURING DEVICE |
US8325220B2 (en) * | 2005-12-02 | 2012-12-04 | Koninklijke Philips Electronics N.V. | Stereoscopic image display method and apparatus, method for generating 3D image data from a 2D image data input and an apparatus for generating 3D image data from a 2D image data input |
JP2007214620A (en) | 2006-02-07 | 2007-08-23 | Sony Corp | Image processing apparatus, image processing method, and program |
US7834910B2 (en) * | 2006-03-01 | 2010-11-16 | David M. DeLorme | Method and apparatus for panoramic imaging |
US20070237364A1 (en) * | 2006-03-31 | 2007-10-11 | Fuji Photo Film Co., Ltd. | Method and apparatus for context-aided human identification |
DE102006019124B4 (en) * | 2006-04-25 | 2012-08-30 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Image acquisition system for all-round visibility |
JP2007295429A (en) * | 2006-04-27 | 2007-11-08 | Fujifilm Corp | Digital still camera and its control method |
US7567293B2 (en) * | 2006-06-07 | 2009-07-28 | Onlive, Inc. | System and method for performing motion capture by strobing a fluorescent lamp |
SG138491A1 (en) * | 2006-06-21 | 2008-01-28 | Generic Power Pte Ltd | Method and apparatus for 3-dimensional vision and inspection of ball and like protrusions of electronic components |
US7953295B2 (en) * | 2006-06-29 | 2011-05-31 | Google Inc. | Enhancing text in images |
US20080018774A1 (en) * | 2006-07-21 | 2008-01-24 | Matsushita Electric Industrial Co., Ltd. | Image capture device |
US20080027599A1 (en) * | 2006-07-28 | 2008-01-31 | James Logan | Autonomous vehicle and systems and methods for the operation thereof |
US20080117288A1 (en) * | 2006-11-16 | 2008-05-22 | Imove, Inc. | Distributed Video Sensor Panoramic Imaging System |
US8094182B2 (en) * | 2006-11-16 | 2012-01-10 | Imove, Inc. | Distributed video sensor panoramic imaging system |
US7813843B2 (en) * | 2007-01-04 | 2010-10-12 | Cisco Technology, Inc | Ad-hoc mobile IP network for intelligent transportation system |
US7990394B2 (en) * | 2007-05-25 | 2011-08-02 | Google Inc. | Viewing and navigating within panoramic images, and applications thereof |
US20100133424A1 (en) * | 2007-05-26 | 2010-06-03 | Norman Matheson Lindsay | Electro-optical sensors |
US8072448B2 (en) * | 2008-01-15 | 2011-12-06 | Google Inc. | Three-dimensional annotations for street view data |
US20110178756A1 (en) * | 2008-02-05 | 2011-07-21 | Utah State University Research Foundation | Integrated, Predictive, Radiance Sensor Apparatus and Method |
EP2253131B1 (en) * | 2008-02-08 | 2014-06-25 | Google, Inc. | Panoramic camera with multiple image sensors using timed shutters |
JP4582198B2 (en) * | 2008-05-30 | 2010-11-17 | ソニー株式会社 | Solid-state imaging device, imaging device, and driving method of solid-state imaging device |
WO2009149413A1 (en) * | 2008-06-06 | 2009-12-10 | Real D | Blur enhancement of stereoscopic images |
JP5132497B2 (en) * | 2008-09-16 | 2013-01-30 | キヤノン株式会社 | IMAGING DEVICE AND IMAGING DEVICE CONTROL METHOD |
US8893026B2 (en) * | 2008-11-05 | 2014-11-18 | Pierre-Alain Lindemann | System and method for creating and broadcasting interactive panoramic walk-through applications |
WO2010075202A2 (en) * | 2008-12-26 | 2010-07-01 | Datalogic Scanning, Inc. | Systems and methods for imaging |
JP2010204304A (en) * | 2009-03-02 | 2010-09-16 | Panasonic Corp | Image capturing device, operator monitoring device, method for measuring distance to face |
CN102450001B (en) * | 2009-05-29 | 2015-08-12 | 惠普开发有限公司 | Reduce the method for the viewpoint associated artifact in multi-projector system, equipment and optical projection system |
JP4903888B2 (en) * | 2010-08-09 | 2012-03-28 | 株式会社ソニー・コンピュータエンタテインメント | Image display device, image display method, and image correction method |
US8937643B1 (en) * | 2012-02-28 | 2015-01-20 | Carlos Esteban | Offset rolling shutter camera model, and applications thereof |
US8941777B1 (en) * | 2012-10-31 | 2015-01-27 | Google Inc. | System and method for capturing images of a geographic area |
KR102023587B1 (en) * | 2015-05-27 | 2019-09-23 | 구글 엘엘씨 | Camera Rig and Stereoscopic Image Capture |
US10671082B2 (en) * | 2017-07-03 | 2020-06-02 | Baidu Usa Llc | High resolution 3D point clouds generation based on CNN and CRF models |
-
2009
- 2009-02-09 EP EP09708701.9A patent/EP2253131B1/en active Active
- 2009-02-09 CN CN2009801107588A patent/CN102037720B/en active Active
- 2009-02-09 CA CA2714492A patent/CA2714492C/en active Active
- 2009-02-09 JP JP2010545890A patent/JP5337170B2/en active Active
- 2009-02-09 AU AU2009210672A patent/AU2009210672B2/en active Active
- 2009-02-09 US US12/368,014 patent/US8493436B2/en active Active
- 2009-02-09 WO PCT/US2009/000804 patent/WO2009099667A1/en active Application Filing
-
2013
- 2013-01-10 JP JP2013002323A patent/JP2013066247A/en active Pending
- 2013-02-28 US US13/780,493 patent/US20130169745A1/en not_active Abandoned
-
2016
- 2016-01-26 US US15/006,449 patent/US9794479B2/en active Active
-
2017
- 2017-09-21 US US15/711,128 patent/US10397476B2/en active Active
-
2019
- 2019-06-24 US US16/449,917 patent/US10666865B2/en active Active
Also Published As
Publication number | Publication date |
---|---|
JP2011512735A (en) | 2011-04-21 |
WO2009099667A1 (en) | 2009-08-13 |
US20160142632A1 (en) | 2016-05-19 |
US10666865B2 (en) | 2020-05-26 |
US8493436B2 (en) | 2013-07-23 |
CA2714492C (en) | 2014-07-15 |
US20090201361A1 (en) | 2009-08-13 |
CN102037720B (en) | 2013-11-06 |
EP2253131A1 (en) | 2010-11-24 |
JP2013066247A (en) | 2013-04-11 |
JP5337170B2 (en) | 2013-11-06 |
CN102037720A (en) | 2011-04-27 |
AU2009210672A1 (en) | 2009-08-13 |
US20180077351A1 (en) | 2018-03-15 |
US9794479B2 (en) | 2017-10-17 |
CA2714492A1 (en) | 2009-08-13 |
US10397476B2 (en) | 2019-08-27 |
US20190313022A1 (en) | 2019-10-10 |
AU2009210672B2 (en) | 2013-09-19 |
US20130169745A1 (en) | 2013-07-04 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10666865B2 (en) | Panoramic camera with multiple image sensors using timed shutters | |
US9185297B2 (en) | Image capture system, control method thereof and image capture apparatus | |
EP2518995B1 (en) | Multocular image pickup apparatus and multocular image pickup method | |
US7340160B2 (en) | Imaging apparatus | |
JP4869795B2 (en) | Imaging control apparatus, imaging system, and imaging control method | |
KR20110102313A (en) | Stereoscopic image processing device, method, recording medium and stereoscopic imaging apparatus | |
CN108737696B (en) | Image pickup apparatus, control method, and non-transitory storage medium | |
JP5167783B2 (en) | Focus detection apparatus and imaging apparatus | |
JP4636979B2 (en) | Imaging apparatus and control method thereof | |
US11330156B2 (en) | Image capturing device | |
JPH09289611A (en) | Digital camera | |
JP5393877B2 (en) | Imaging device and integrated circuit | |
US20220038646A1 (en) | Synchronized camera system having two different cameras | |
JPH11258491A (en) | Focus detecting device and its method and storage medium readable through computer | |
US7782381B2 (en) | Generating an instant review image using fractional readouts | |
JP6439412B2 (en) | Image processing apparatus and image processing method | |
JP7100535B2 (en) | Image pickup device and image processing device, their control method, program, storage medium | |
JP2024004307A (en) | Imaging device and control method thereof, program, and storage medium | |
JP3260745B2 (en) | 3D display device for camera | |
JPH10170816A (en) | Focusing information detector, focus detector, and camera using the same | |
JP2009094780A (en) | Imaging device | |
JP2019062493A (en) | Controller, imaging apparatus, control method, and program | |
JP2006217204A (en) | Imaging apparatus |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
17P | Request for examination filed |
Effective date: 20100908 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA RS |
|
DAX | Request for extension of the european patent (deleted) | ||
17Q | First examination report despatched |
Effective date: 20120110 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
INTG | Intention to grant announced |
Effective date: 20140110 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MK MT NL NO PL PT RO SE SI SK TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: REF Ref document number: 675291 Country of ref document: AT Kind code of ref document: T Effective date: 20140715 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R096 Ref document number: 602009024887 Country of ref document: DE Effective date: 20140807 |
|
REG | Reference to a national code |
Ref country code: NL Ref legal event code: T3 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: NO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140925 Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140926 |
|
REG | Reference to a national code |
Ref country code: AT Ref legal event code: MK05 Ref document number: 675291 Country of ref document: AT Kind code of ref document: T Effective date: 20140625 |
|
REG | Reference to a national code |
Ref country code: LT Ref legal event code: MG4D |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: HR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20141027 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 7 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20141025 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R097 Ref document number: 602009024887 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed |
Effective date: 20150326 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20150209 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CH Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20150228 Ref country code: LI Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20150228 Ref country code: MC Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: MM4A |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20150209 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 8 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 9 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT; INVALID AB INITIO Effective date: 20090209 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R082 Ref document number: 602009024887 Country of ref document: DE Representative=s name: MARKS & CLERK (LUXEMBOURG) LLP, LU Ref country code: DE Ref legal event code: R081 Ref document number: 602009024887 Country of ref document: DE Owner name: GOOGLE LLC (N.D.GES.D. STAATES DELAWARE), MOUN, US Free format text: FORMER OWNER: GOOGLE INC., MOUNTAIN VIEW, CALIF., US |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 10 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: CD Owner name: GOOGLE INC., US Effective date: 20180213 Ref country code: FR Ref legal event code: CJ Effective date: 20180213 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: MK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20140625 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R079 Ref document number: 602009024887 Country of ref document: DE Free format text: PREVIOUS MAIN CLASS: H04N0005232000 Ipc: H04N0023600000 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: NL Payment date: 20230226 Year of fee payment: 15 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: FR Payment date: 20230223 Year of fee payment: 15 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: GB Payment date: 20230227 Year of fee payment: 15 Ref country code: DE Payment date: 20230223 Year of fee payment: 15 |
|
P01 | Opt-out of the competence of the unified patent court (upc) registered |
Effective date: 20230505 |