EP2135442A1 - Caméra multiobjectif pouvant fonctionner dans divers modes - Google Patents

Caméra multiobjectif pouvant fonctionner dans divers modes

Info

Publication number
EP2135442A1
EP2135442A1 EP08725358A EP08725358A EP2135442A1 EP 2135442 A1 EP2135442 A1 EP 2135442A1 EP 08725358 A EP08725358 A EP 08725358A EP 08725358 A EP08725358 A EP 08725358A EP 2135442 A1 EP2135442 A1 EP 2135442A1
Authority
EP
European Patent Office
Prior art keywords
image
block
capture
stage
focus
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
EP08725358A
Other languages
German (de)
English (en)
Other versions
EP2135442B1 (fr
Inventor
Wilbert Frank Janson Jr.
Kenneth Alan Parulski
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Eastman Kodak Co
Original Assignee
Eastman Kodak Co
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Eastman Kodak Co filed Critical Eastman Kodak Co
Publication of EP2135442A1 publication Critical patent/EP2135442A1/fr
Application granted granted Critical
Publication of EP2135442B1 publication Critical patent/EP2135442B1/fr
Not-in-force legal-status Critical Current
Anticipated expiration legal-status Critical

Links

Classifications

    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B15/00—Special procedures for taking photographs; Apparatus therefor
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/673—Focus control based on electronic image sensor signals based on contrast or high frequency components of image signals, e.g. hill climbing method
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/681—Motion detection
    • H04N23/6812—Motion detection based on additional sensors, e.g. acceleration sensors
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/69—Control of means for changing angle of the field of view, e.g. optical zoom objectives or electronic zooming
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/70—Circuitry for compensating brightness variation in the scene

Definitions

  • the present invention relates to a digital camera that produces digital image files and, more particularly, to a digital camera that uses multiple lenses and image sensors to provide various imaging modes, such as wide angle and telephoto imaging modes.
  • BACKGROUND OF THE INVENTION Currently, most digital cameras use a zoom taking lens and a single color image sensor to capture still and motion images. The captured images are then processed to produce digital image files, which are stored in a digital memory in the camera. The digital image files can then be transferred to a computer, displayed, printed, and shared via the Internet.
  • a digital camera needs to provide a precise automatic lens focusing system (i.e., an autofocus system).
  • the autofocus system must be capable of quickly obtaining the correct focus in order to minimize the "shutter delay" between the time the shutter button is pressed and the still image is captured.
  • the autofocus system must also work in a continuous image capture mode wherein video images are captured. For instance, in a video mode the focus should be adjusted in real-time while video images are being continuously captured.
  • Many digital cameras and scanners capture images using an image sensor and a taking lens system with an adjustable focus.
  • the focus distance of such an adjustable focus taking lens system can automatically be set to one of a plurality of different settings by sensing, control, and drive systems, which are adapted to provide optimal focus of what is determined to be a subject area in a scene.
  • Lens systems that provide automatically adjustable focus settings based on a focus measurement and an adjustable focus lens are referred to herein as autofocus systems.
  • Digital cameras typically use one of two types of autofocus systems: a rangefinder system and a "through-the-lens" focus system.
  • a rangefinder system uses rangef ⁇ nding sensors such as a sonic rangefinder or a dual lens rangefinder to determine the distance from a camera to one or more portions of a scene within a field of view of the rangefinder system.
  • a sonic rangefinder measures the phase offset between a projected sonic signal and a reflected sonic signal to infer the distance to objects in the scene.
  • Dual lens rangefinders contain two lenses that are separated by a distance along with two matching sensor areas that capture matched pairs of images. Dual lens rangefinders are commonly used on digital cameras in the form of dual lens rangefinder modules which contain two lenses separated by a distance along with two matching sensor areas that capture matched pairs of low resolution images.
  • Dual lens rangefinder-based autofocus systems include active and passive systems. Active systems actively project light onto the scene, while passive systems work with the available light from the scene. Dual lens rangefinder modules can be purchased from Fuji Electric in several models such as the FM6260W. A dual lens rangefinder module for optical apparatus such as a camera is described in U.S. Patent No. 4,606,630, which was issued to Haruki et al. on August 19, 1986 (and assigned to Fuji Electric). According to the description of the prior art in this patent, matched pairs of low resolution images are analyzed for correlation between the two images to determine the offset between the two images caused by the separation between the two lenses.
  • FIG. 27 A diagram illustrative of the principle of the operation of a conventional rangefinder is shown herein in FIG. 27.
  • light from an object 151 is incident on two small lenses 152 and 153 which have a sufficiently short focal length f that light rays received from the object through different spaced paths 154 and 155 produce corresponding spaced images 157 and 158 in a focal plane 156 which is common to the lenses 152 and 153.
  • the centers of the images 157 and 158 are located at reference positions 170 and 180 in FIG. 27, but when the object 151 is located at a closer distance, the centers of the images are shifted apart to positions 171 and 181.
  • b is the distance between the optical axes of the small lenses, that is, the base length.
  • two optical sensor arrays 190 and 191 are provided in the focal plane 156 as shown in FIG. 27.
  • These optical sensor arrays each comprise a plurality of optical sensors, for instance CCD devices, and an analog photoelectric signal is generated by each optical sensor corresponding to the light intensity at the portion of the image which is incident on the sensor.
  • Haruki et al. shows a conventional circuit, as well as a higher speed rangefinding circuit according to the patent, for obtaining the sum x of the shifted distances by comparing two image signal trains comprising the digital image signals from the left and right optical sensor arrays.
  • the offset information x is used along with the lens separation distance b and the focal length f to calculate the distanced to the scene by triangulation.
  • the calculated distance d to the scene is used to guide the positioning of an adjustable focus lens to produce the best image quality.
  • this adjustment may be based on a calibration curve established between the distance to the scene as measured by the dual lens rangefinder module and a series of best focused images as produced by a "through the lens" autofocus system.
  • the calibration curve is stored as an equation or a look-up table in a microprocessor in the camera.
  • Rangefinder-based autofocus systems have the advantage of being very fast, some having a response time that can be in the range of 0.01-0.05 second.
  • the focus quality produced by some rangefinder-based autofocus systems can vary when they are used in different operating conditions.
  • sonic autofocus systems cannot focus through a glass window as the glass stops the projected sonic signal, thereby causing the autofocus system to focus onto the glass.
  • the accuracy of dual lens rangefinders are typically influenced by changes in environmental conditions such as temperature and/or humidity.
  • the problem with dual lens rangefinder modules is that the calibration between the dual lens rangefinder module and the adjustable focus lens position is not stable within the normal operating environment for digital cameras.
  • dual lens rangefinder modules are not typically used independently for autofocus in digital cameras but instead are used as a rough focus adjustment that is supplemented by a "through the lens" autofocus system.
  • the "through-the-lens" autofocus system determines a focus state through an analysis of a series of autofocus images captured with the adjustable focus lens system positioned at a plurality of different focus distances.
  • a plurality of autofocus images (e.g., 5-20) are captured with the adjustable focus lens in a series of different positions in a so-called “hill climb” method.
  • This type of autofocus is known as “hill climbing” autofocus because it generates a sequence of values that increase in level until they pass over a peak, i.e., a "hill".
  • the lens focus position is adjusted automatically until the contrast of the edge detail in the image, or a particular area of the image, is maximized.
  • the contrast present in each of the autofocus images is compared and the autofocus image with the greatest contrast is deemed to have been captured with the best focus conditions (often the best focus lens position is further refined by interpolating the contrast values between images).
  • filters to separate not only the higher frequency component of the video signal, but also the lower frequency component.
  • a lens may be quickly driven in coarse adjustment steps in a low frequency range furthest from the maximum focus, and then driven in finer adjustment steps in a high frequency range nearer to the maximum focus.
  • FIG. 28 A flow diagram of a conventional "hill climbing" contrast autofocus algorithm is shown in FIG. 28. This algorithm uses the "hill climbing" contrast autofocus method discussed above and shown in the diagram of FIG.
  • the abscissa indicates the focusing position of a lens along a distance axis
  • the ordinate indicates the focusing evaluation value
  • the curves A and B indicate the focusing evaluation values for high and low frequency components, respectively, relative to a particular in-focus position P.
  • the best starting point for the algorithm depends on the hyperfocal distance of the current lens setting, which is a function of the focal length setting and the f-number. A distance of about 2 meters is typically a good starting point. Then a low frequency bandpass filter is loaded (stage 197) and the focus values are read out. The algorithm employs a comparison stage 198 to set the direction of lens adjustment toward increasing focus values, and to determine when the lens is stepped over the "hill". The depth of field, which depends on the present focal length and f-number, sets the number of steps, i.e., the next near focus position, which should be taken before capturing the next frame when using the low frequency bandpass filter.
  • a high frequency bandpass filter is loaded (stage 199), and the lens is moved in the opposite direction until the peak of the higher "hill” is found (curve A in FIG. 29).
  • the peak focus value may use either the weighted average or peak value from numerous pixels.
  • the two types of autofocus systems are used together in a hybrid system in which the rangefinder based autofocus system is used to provide a fast estimation of the adjustable focus lens location that is then followed by the use of the "through-the-lens" autofocus system to refine the focus setting.
  • the rangefinder based autofocus system is used to provide a fast estimation of the adjustable focus lens location that is then followed by the use of the "through-the-lens" autofocus system to refine the focus setting.
  • U.S. Patent No. 6,864,474 entitled “Focusing Apparatus for Adjusting Focus of an Optical Instrument” and which issued March 8, 2005 in the name of Misawa, describes the coordinated use of a rangefinder-based autofocus system with a "through-the-lens" autofocus system.
  • the focus position of the adjustable focus taking lens is determined by both the rangefinder-based autofocus system and the "through-the-lens” autofocus system.
  • the difference between the adjustable focus taking lens position determined by the rangefinder-based autofocus system and the adjustable focus taking lens position determined by the "through-the-lens” autofocus system is stored for future reference.
  • the stored difference information is used to refine the number of autofocus images captured and analyzed by the "through-the-lens" autofocus system in the "hill climb” method to determine the adjustable focus lens position for best focus, thereby reducing the number of autofocus images captured and processed in cases where the rangefinder system is accurate and increasing the number of autofocus images captured and processed in cases where the rangefinder is inaccurate.
  • the method described by Misawa assumes that the performance of the rangefinder, adjustable focus taking lens system and control system are consistent over time, do not fluctuate with variations in environmental conditions, and do not otherwise change or drift over time.
  • the "hill climb” method typically operates over incremental distances near the subject presently focused upon. Then, in refocusing an image, the “hill climb” method typically determines whether any lens movement is stepping "up or down the hill” and resets the lens for a new maximum. In practice, this means that, if the lens movement is stepping "down the hill", the lens motion is immediately reversed so as to seek the new maximum for the existing subject.
  • a focusing device includes a movable focusing lens adapted to be moved to different positions across the entire focusing range, a conversion element for converting light incident on and transmitted through the focusing lens into a signal, and a lens driving mechanism for moving the focusing lens.
  • the focusing device further includes a focus evaluation value calculation unit for calculating a focus evaluation value for each position of the focusing lens based on the signal from the conversion element.
  • the focus evaluation value calculation unit extracts only the signals corresponding to the pixels in a focus area defined, e.g., at the center of an image, which is further divided into nine "tiles", that is, blocks that are obtained by dividing the focus area into a small number of rows and columns used as observation areas.
  • the determination is negative, the focusing lens is moved by a preset step width, and the calculation is repeated.
  • the focusing lens is always moved stepwise from an infinite far position to a nearest position, and for each step of movement, a focus evaluation value is calculated for each tile.
  • the focusing lens position producing the maximum peak is determined as the in-focus lens position.
  • a lens driving output is then applied to the lens driving mechanism so that the lens moves to the determined in-focus position.
  • a digital camera can use multiple image sensors with different focal length lenses, as described in commonly assigned U.S. Patent Application Serial No. 1 1/062,174, entitled “Digital Camera Using Multiple Lenses and Image Sensors to Provide an Improved Zoom Range", which was filed February 18, 2005 in the names of Labaziewicz et al., the disclosure of which is incorporated herein by reference.
  • the Kodak Easyshare V610 dual lens digital camera includes a 38-114 mm (35 mm equiv.) f/3.9-f/4.4 lens and a 130-380 mm (35 mm equiv.) f/4.8 lens, in order to provide a 10x optical zoom range.
  • only one of the two image sensors is used at a time. The two image sensors do not simultaneously capture images.
  • U.S. Patent Application Publication No. US 2003/0020814 which was published January 30, 2003, discloses an image capturing apparatus having a plurality of capturing optical systems, each coupled to a CCD image sensor, including a first system having a shorter focal length and a second system having a longer focal length.
  • the two lenses can provide different focal lengths ranges, including one system with a fixed-focus lens and the other system with a zoom lens, or they can both be fixed focus lenses set to two different focus distance settings.
  • a selection unit automatically selects the capture signals from one of the capturing optical systems based on capture conditions, such as measured distance or luminance, determined by a capture condition acquiring unit.
  • a rough autofocus search (where a stepping motor may be driven at intervals of several steps) is made by the ordinary mode optical system (where the focal depth is relatively large). This rough search results in a reduced focal distance range that includes the focusing position.
  • the telescopic mode optical system is driven to an autofocus search start position at one end of the reduced focal distance range.
  • a fine autofocus search is performed by the telescopic mode optical system (where the focal depth is relatively shorter), but only in the reduced focal distance range determined by the ordinary mode autofocus search.
  • each of the optical systems could be used for capturing an image and for focus adjustment, or one optical system could be employed for capturing an image and focus adjustment and the other optical system could be devoted only to focus adjustment of the image-capturing optical system.
  • the non-capturing optical system determines the focusing position first, the capturing optical system is driven to that position and a fine adjustment is then made by the capturing optical system.
  • a problem with these prior art systems is that either a separate autofocus sensor must be used (thus increasing the cost) or else there is typically a significant "shutter delay" as the autofocus is performed using the same sensor that is used to capture the image.
  • the separate autofocus sensor is usually a rangefinder and, as mentioned above, the calibration between the dual lens rangefinder module and the adjustable focus lens position is not stable within the normal operating environment for digital cameras.
  • the process can be relatively slow in determining a focus setting due to the large number of autofocus images that must be captured and compared.
  • the problem can be somewhat alleviated according to the aforementioned U.S. Patent Application Publication No. US 2003/0160886, but difficulties remain in rapidly achieving focus as the subject changes or moves, or in rapidly interchanging the focusing requirements of the optical systems when the operator elects to change the capture function from one photographing system to the other.
  • a further problem is that during video capture, many of the frames are out of focus due to the use of an autofocus system that uses the "hill climb method" to focus.
  • a digital camera uses multiple lenses and image sensors to provide various imaging modes, such as wide angle and telephoto imaging modes.
  • various imaging modes such as wide angle and telephoto imaging modes.
  • the user should be given a preview of the capture possibilities for each of the modes.
  • the camera should be designed so as to allow for a situation where the user fails to choose an imaging mode. What is therefore needed is a digital camera with dual capture systems that provides a preview snapshot of the pictures from the dual image capture stages, however with an improved capability for defaulting to a predetermined imaging mode in the absence of a user selection.
  • the object of this invention is to present both images from a dual-lens camera to a user for selection, and to automatically shift to one as a default choice in the absence of a user selection.
  • the present invention which is directed to overcoming one or more of the problems set forth above, pertains to a method for operating a digital camera having first and second imaging stages for forming separate images of a scene.
  • the invention comprises the steps of: designating one of the imaging stages as a default imaging stage; capturing and displaying first and second images from the first and second imaging stages, respectively; timing display of the images for a predetermined time period; and enabling an operator selection of one of the imaging stages as a primary capture unit which is to be primarily used for capturing an image of the scene that is stored by the digital camera, whereby the default imaging stage is automatically selected as the primary capture unit if the operator selection does not occur within the predetermined time period.
  • the first and second imaging stages may include zoom lenses, and the method further may include the step of setting the first and second imaging stages to respective default zoom positions, whereby the step of capturing and displaying the images displays the images obtained at their respective default zoom positions.
  • the imaging stage not selected as the primary capture unit may be designated as a secondary capture unit for influencing, adjusting, analyzing or augmenting the image formed by the primary capture unit, e.g., where the secondary stage provides scene analysis data for setting the capture parameters for the primary capture unit or where the secondary capture unit provides image data for augmenting the image captured by the primary capture unit.
  • the default imaging stage may be automatically selected as the primary capture unit if the operator selection has not occurred when the shutter button is actuated.
  • the invention also pertains to an electronic camera having first and second imaging stages for capturing separate images of a scene, one of the stages being designated as a default imaging stage.
  • a processor enables capture and display of the separate images, and further responds to an operator selection of one of the imaging stages as a primary capture unit which is to be primarily used for capturing an image of the scene that is stored by the digital camera. If the operator selection does not occur within a predetermined time period, or if the camera is actuated before the time has run out, the processor automatically selects the default imaging stage as the primary capture unit.
  • the invention generally provides the advantage of a dual capture camera system that provides a preview snapshot of both pictures from the dual image capture stages so that the camera user can easily perceive the imaging advantages of each stage and accordingly make a choice.
  • this is coupled with the advantage of defaulting to a predetermined imaging mode in the absence of a user selection within a predetermined time, or if the camera's capture function is actuated by the user before the time has run out.
  • FIG. 1 depicts a block diagram of a first embodiment of a digital camera using a first zoom lens with a first image sensor, and a second zoom lens with a second image sensor according to the invention.
  • FIGS. 2 A and 2B are two perspective views of the digital camera shown in FIG. 1.
  • FIG. 3 depicts a flow diagram showing a method for performing autofocus and for capturing digital still images using the digital camera shown in FIG 1.
  • FIG. 4 depicts a flow diagram showing a method for performing autofocus using a rangefinder method with two image capture stages.
  • FIG. 5 depicts a flow diagram showing a method for performing autofocus using a "hill climb method” with two image capture stages.
  • FIG. 6 depicts a flow diagram showing a method for producing an autofocus rangefinder calibration curve.
  • FIG. 7 depicts a flow diagram showing a method for producing an autofocus "hill climb method” calibration curve.
  • FIG. 8 depicts a flow diagram showing a method for performing autofocus and for capturing digital video images using the digital camera shown in FIG 1.
  • FIG. 9 depicts a flow diagram showing a method for performing autofocus with a rangefinder method with two image capture stages.
  • FIG. 10 depicts a flow diagram showing a method for performing autofocus with a "hill climb method” with two image capture stages.
  • FIG. 1 1 depicts a method for producing a range map with two image capture stages.
  • FIG. 12 depicts a block diagram of a digital camera using a first fixed focal length lens with a first image sensor, and a second (zoom) lens with a second image sensor according to another embodiment of the invention.
  • FIG. 13 depicts a block diagram of a digital camera using a first fixed focal length lens with a first image sensor, a second (zoom) lens with a second image sensor, and a third (zoom) lens with a third image sensor according to another embodiment of the invention.
  • FIG. 14 depicts a flow diagram showing a method for enhancing the depth of field of an image by using images from both image capture stages.
  • FIG. 15A and FIG. 15B depict a diagram of a mobile phone camera with two image capture stages according to another embodiment of the invention.
  • FIG. 16A and FIG. 16B depict a diagram of a stage containing two image capture stages in a single stage for a mobile phone camera.
  • FIG. 17A and FIG. 17B are representations of images captured with two image capture stages showing the offset between the two images that is used to determine the distance to portions of the scene.
  • FIG. 18 depicts a flow diagram of a method for determining GPS locations of portions of the scene.
  • FIG. 19 depicts a flow diagram for selecting one of the imaging stages in a dual lens camera system as the primary capture unit, while relegating the other imaging stage to other functions, and defaulting to a predetermined capture unit in the absence of a user selection, all in accordance with the invention.
  • FIG. 20 depicts a flow diagram illustrating the usage of the non-primary capture unit for scene analysis.
  • FIG. 21 depicts a flow diagram illustrating the usage of the non-primary capture unit for scene analysis whenever the change in scene conditions exceeds a threshold.
  • FIG. 22 depicts a flow diagram illustrating the usage of both the primary capture unit and the non-primary capture unit for scene analysis.
  • FIG. 23 depicts a flow diagram illustrating the reversal of the functions of the capture units, that is, the current scene analysis and primary capture units are reset to be the primary capture unit and scene analysis capture unit, respectively.
  • FIG. 24 depicts a flow diagram illustrating the usage of a preview image taken by the primary capture stage to set the capture unit parameters for the primary capture stage, specifically during the capture process.
  • FIG. 25 depicts a flow diagram illustrating the usage of a preview image taken by the primary capture stage and a scene analysis image taken by the non- capture stage to set the capture unit parameters for the primary capture stage, specifically during the capture process.
  • FIG. 26 depicts a flow diagram illustrating the usage of a primary image taken by the primary capture stage and an augmentation image taken by the non- capture stage to produce an enhanced image.
  • FIG. 27 depicts a diagram illustrative of the operation of a conventional dual lens rangefinder.
  • FIG. 28 depicts a flow diagram of a conventional "hill climbing" contrast autofocus algorithm.
  • FIG. 29 illustrates the relationship between the focus value obtained from the filters used to isolate components of the image and the lens position for a "hill climbing" autofocus method.
  • Each of the several embodiments described herein include an image capture assembly, such as a digital camera - still or video - or a digital scanner, having multiple image capture stages, each composed of a lens and an image sensor, wherein the lenses of the multiple image capture stages have different focal lengths to provide an extended optical zoom range for the image capture assembly.
  • the present description contemplates the use of the multiple image capture stages to additionally provide an enhanced autofocus capability.
  • dedicated autofocus modules can be eliminated thereby reducing the cost and size of the image capture assembly while improving the sharpness of the captured still and video images, as well as increasing the speed of response of the autofocus system.
  • one image capture stage may be used to capture digital still images or video images while another image capture stage is simultaneously being used for another purpose, such as enhanced autofocus, generation of a secondary image, production of a range map, and the like.
  • a first imaging stage containing a first (e.g., zoom) lens is used to capture a still image or a series of images as in a video sequence, while a second imaging stage simultaneously provides images for the purpose of autofocus of the first imaging stage.
  • the focus conditions of the lens in the second imaging stage can be adjusted over a wider range, e.g., around a peak contrast position (e.g., "hill climb' autofocus) or from the near focus position to the infinity focus position (e.g., "whole way” autofocus) to determine the new best focus setting for the lens in the first imaging stage without detrimentally affecting the focus quality of the images captured by the first imaging stage.
  • a peak contrast position e.g., "hill climb' autofocus
  • the near focus position to the infinity focus position e.g., "whole way” autofocus
  • the camera When the user adjusts the zoom position on the image capture assembly to a second optical zoom range, the camera automatically switches to use the second imaging stage (containing, e.g., a second zoom lens) to capture the images, and begins using the first imaging stage for autofocus of the second imaging stage. Because the two lenses have different focal lengths, they have different magnifications. Therefore, the digital filters used to determine the autofocus may be adjusted, as a function of this difference in magnification, in order to compensate for the differences in magnification.
  • the second imaging stage containing, e.g., a second zoom lens
  • two image capture stages are used together to form a high resolution rangefinder similar to a dual lens rangefinder but with higher resolution, which is provided by the two high resolution image capture stages and a larger separation distance between the two lenses in the two image capture stages.
  • the first image capture stage may be used to provide an initial accurate autofocus using a "hill climb" contrast comparison method; subsequently, the first image capture stage is used in conjunction with the second image capture stage as a high resolution rangefinder operating in a differential mode to detect any changes in the distance to the scene that require refocusing prior to capture of a digital still image or prior to or during a video capture.
  • the two image capture stages are both autofocused at the same time using a "hill climb" contrast comparison method prior to capture of a digital still image or capture of a video image by the first image capture stage.
  • the second image capture stage then continues to check the focus by measuring the contrast in the image; when a change in contrast is detected, a second autofocus operation using the "hill climb” contrast comparison method is performed using the second image capture stage to determine the change in focus condition.
  • the focus of the first image capture stage is then changed by an amount proportional to the change in focus determined by using the second image capture stage.
  • a differential focus change measurement is performed from a position established by a "hill climb" autofocus to improve the accuracy of the auto focusing process.
  • the first imaging stage including the first zoom lens and its associated image sensor
  • the second imaging stage including the second zoom lens and its associated image sensor
  • the camera automatically switches to the second imaging stage, using the second zoom lens and its associated sensor to capture the motion sequence, and begins using the first imaging stage, with its first lens and associated image sensor, to simultaneously determine if the focus of the second zoom lens should be adjusted as a result of subject motion.
  • the two image capture stages are configured as a high resolution rangefinder in order to determine the distances to different portions of the scene in the form of a range map.
  • the range map is then used to modify the captured image signal or the output image for a variety of purposes, such as (without limitation): to improve image processing and enable improved image quality; to improve object identification within the image; to enable object extraction from an image; to enable motion tracking of objects within multiple images; to enable reduced depth of field images by blurring of objects outside of the desired depth of field; to improve the dynamic range within images; to reduce exposure issues introduced by use of a flash; and to improve scene balance within the image.
  • a first imaging stage including a first zoom lens and a first image sensor
  • a second imaging stage including a second zoom lens and a second image sensor
  • the sensor output from the second imaging stage is used as a secondary output image for modifying the primary output image, thereby generating an enhanced primary image signal.
  • the secondary still image is used to provide an enhancement signal that may, e.g., sharpen portions of the primary still image that are positioned near the secondary focus distance or may modify the dynamic range of the primary still image.
  • the images from both imaging stages may be used to generate a range map identifying the distances to the different portions of the scene.
  • the camera further includes a GPS unit for providing GPS coordinates for the location of the camera and an electronic compass for providing the pointing direction of the camera.
  • the GPS coordinates for the location of the camera, the pointing direction of the camera, and distance offsets from the range map may be used in order to generate GPS coordinates for portions of the scene.
  • an electronic camera has first and second imaging stages for capturing separate images of a scene, one of the stages being designated as a default imaging stage.
  • a processor enables capture and display of the separate images, and further responds to an operator selection of one of the imaging stages as a primary capture unit which is to be primarily used for capturing an image of the scene that is stored by the digital camera. If the operator selection does not occur within a predetermined time period, or if the camera is actuated before the time has run out, the processor automatically selects the default imaging stage as the primary capture unit.
  • FIG. 1 depicts a block diagram of an image capture assembly 1OA according to the first embodiment of the present invention.
  • the image capture assembly 1OA is preferably a portable battery operated device, small enough to be easily handheld by a user when capturing and reviewing images.
  • the image capture assembly 1OA is a digital camera that produces both still images and motion video images that are stored on a removable memory card 54.
  • the digital camera may produce and store only motion images or only still images.
  • the image capture assembly 1OA includes two imaging stages 1 and 2, both with zoom lenses 3 and 4. (These stages will hereinafter be referred to in the specification as image capture stages, although in most cases only one stage - at a time - is capturing an image that is stored on the removable memory card 54.)
  • the first zoom lens 3 is controlled by a first lens focus adjuster, e.g., zoom and focus motors 5a, and provides an image to a first image sensor 12.
  • the second zoom lens 4 is controlled by a second lens focus adjuster, e.g., zoom and focus motors 5b, and provides an image to a second image sensor 14.
  • An adjustable aperture and shutter assembly in each zoom lens (not shown) is used to control the exposure to image sensors 12 and 14.
  • FIGS. 2 A and 2B show perspective views of the image capture assembly 1OA described in relation to FIG. 1.
  • FIG. 2 A is a frontal view of the image capture assembly 1 OA, showing the first zoom lens 3, the second zoom lens 4 and a flash 48.
  • FIG. 2B is a rear view of the camera 1OA, showing a color LCD image display 70 and a number of user controls 42, including a shutter button 42a for enabling an image capture sequence, a zoom button 42c for enabling a selection of a zoom setting, and a multi-position selector 42d for navigating through images, menu choices and the like that are displayed on the color LCD image display 70.
  • the image capture stages 1 and 2 comprise the zoom lenses 3 and 4 and the image sensors 12 and 14, as shown in FIG 1. While zoom lenses 3 and 4 are offset vertically, as shown in FIG. 2a, the zoom lenses 3 and 4 could be offset in other directions, such as horizontally. In addition, according to another embodiment of the invention, one (or both) of the zoom lenses 3 and 4 could be replaced with a fixed focal length lens. In all cases, the optical axes of the zoom lenses 3 and 4 and the sensors 12 and 14 are generally aligned with respect to each other so as to be viewing substantially the same scene, albeit typically with different fields of view. The configuration of the optical components are further described in the aforementioned, commonly assigned U.S. Patent Application
  • the configuration of the zoom lenses 3 and 4 in the image capture stages 1 and 2 can include folded optical paths to change the overall dimensions of the image capture stages 1 and 2; however, folded optical paths are not necessary for practice of the present invention.
  • the image sensors 12 and 14 are single-chip color megapixel CCD sensors, using the well-known Bayer color filter pattern in order to capture color images, although other sensors, such as CMOS sensors, and other color filter arrays, such as stripe filters, may be used equally well without limitation according to the invention.
  • the image sensors 12 and 14 may have a variety of aspect ratios, for example, a 4:3 image aspect ratio and a variety of resolutions, for example, a total of 6.1 MP effective megapixels (million pixels), with, in this particular case, 2848 active columns of pixels x 2144 active rows of pixels. It should also be understood that the image sensors 12 and 14 do not have to have the same specifications. For instance, in some embodiments disclosed in the aforementioned, commonly assigned U.S. Patent Application Serial No.
  • the size, resolution, color filter array, spectral sensitivity and aspect ratio of the image sensors 12 and 14 can be different.
  • a control processor and timing generator 40 controls the first image sensor 12 by supplying signals to clock drivers 13, and controls the second image sensor 14 by supplying signals to clock drivers 15.
  • the control processor and timing generator 40 also controls the zoom and focus motors 5a and 5b, an auto exposure detector 46, user controls 42, first and second digital multiplexer control elements 34 and 36, and the flash 48 for emitting light to illuminate the scene.
  • the user controls 42 are used to control the operation of the digital camera 1OA, as also described earlier in reference to FIG. 2B.
  • An analog output signal 12e from the first image sensor 12 is amplified and converted to a first digital image signal by a first analog signal processor 22.
  • the digitized first digital image signal is provided to a first input of the first digital multiplexer control element 34 and to a first input of the second digital multiplexer control element 36.
  • An analog output signal 14e from the second image sensor 14 is amplified and converted to a second digital image signal by a second analog signal processor 24.
  • the digitized second digital image signal is provided to a second input of the digital multiplexer control element 34 and a second input of a second digital multiplexer control element 36.
  • the function of the first multiplexer 34 is to select either the first sensor output 12e from the first image sensor 12, or the second sensor output 14e from the second image sensor 14 as the image capture signal.
  • the function of the second multiplexer 36 is to select either the second sensor output 14e from the second image sensor 14 or the first sensor output 12e from the first image sensor 12 as the autofocus image signal, which is provided to an image processor 50.
  • the control processor and timing generator 40 controls the digital multiplexers 34 and 36 in order to select one of the sensor outputs (12e or 14e) as the captured image signal, and to select the other sensor output (14e or 12e) as the autofocus image signal.
  • the digital data provided by the first digital multiplexer control element 34 is temporarily stored in a DRAM buffer memory 38 and subsequently processed by the image processor 50 to produce a processed digital image file, which may contain a still digital image or a video image.
  • the digital data provided by the second digital multiplexer control element 36 is provided to the image processor 50, which performs autofocus calculations as will be described later in reference to FIGS. 4, 5, 6 and 7.
  • the image processor 50 produces the focus detection signals that drive the first and second focus adjusters, that is, the zoom and focus motors 5a and 5b.
  • the control processor and timing generator 40 in combination with the image processor 50, either (a) selects the sensor output 12e from the first imaging stage 1 as the captured image signal and uses the sensor output 14e from the second imaging stage 2 to generate the focus detection signal for the selected imaging stage 1 or (b) selects the sensor output 14e from the second imaging stage 2 as the captured image signal and uses the sensor output 12e from the first imaging stage 1 to generate the focus detection signal for the selected imaging stage 2.
  • the focus detection signal is applied to the zoom and focus motors 5a and 5b of the selected imaging stage in order to adjust the focus of the image providing the sensor output for the captured image signal.
  • the processing performed by the image processor 50 is controlled by firmware stored in a firmware memory 58, which may be flash EPROM memory or any other form of appropriate memory.
  • the processor 50 processes the digital input image from the DRAM buffer memory 38, using a RAM memory 56 to store intermediate results during the processing stage.
  • the processed digital image file is provided to a memory card interface 52, which stores the digital image file on the removable memory card 54.
  • Removable memory cards 54 are one type of removable digital image storage medium, and are available in several different physical formats.
  • the removable memory card 54 can include (without limitation) memory cards adapted to well-known formats, such as the Compact Flash, SmartMedia, MemoryStick, MMC, SD, or XD memory card formats.
  • the image processor 50 also receives input from a global positioning system (GPS) unit 57, which enables the image processor 50 to determine the GPS coordinates (i.e., location) of the camera at any appropriate time, e.g., when an image is captured.
  • GPS global positioning system
  • the image processor also receives directional input from an electronic compass 59, which enables the image processor 50 to determine which direction the camera is pointed, e.g., when an image is captured.
  • the image processor 50 performs various other image processing functions, including color interpolation followed by color and tone correction, in order to produce rendered sRGB image data.
  • the rendered sRGB image data is then JPEG compressed and stored as a JPEG image file on the removable memory card 54.
  • the rendered sRGB image data may also be provided to a host PC 66 via a host interface 62 communicating over a suitable interconnection 64, such as a SCSI connection, a USB connection or a Firewire connection.
  • the JPEG file preferably uses the so- called "Exif ' image format defined in "Digital Still Camera Image File Format (Exif)" version 2.2 by the Japan Electronics and Information Technology
  • This format includes an Exif application segment that stores particular image metadata, including the date/time the image was captured, as well as the lens f/number, GPS location and pointing direction when the image was captured and other camera settings.
  • the image processor 50 while typically a programmable image processor, can alternatively be, for example, a hard-wired custom integrated circuit (IC) processor, a general purpose microprocessor, or a combination of hard-wired custom IC and programmable processors.
  • IC integrated circuit
  • the image processor 50 can alternatively be, for example, a hard-wired custom integrated circuit (IC) processor, a general purpose microprocessor, or a combination of hard-wired custom IC and programmable processors.
  • IC integrated circuit
  • the image processor 50 can alternatively be, for example, a hard-wired custom integrated circuit (IC) processor, a general purpose microprocessor, or a combination of hard-wired custom IC and programmable processors.
  • one or more of the functions shown as separate blocks in FIG 1 such as the digital multiplexer control elements 34 and 36, the DRAM buffer memory 38, and the RAM memory 58, can be incorporated in an IC containing the image processor 50.
  • control processor 40 and the image processor 50 may be merged as needed for purposes of certain applications and discussions, such as in the reference to a processing stage in this description and in certain of the claims, where, e.g., selection of a sensor output (as by the control processor 40) and generation of a focus signal (as by the image processor 50) are referred to.
  • the recitation of a processing stage is intended to encompass the recited functions, whether they are to be found in one or more actual processing elements, circuits, or the like.
  • the digital camera 1OA is included as part of a camera phone.
  • the image processor 50 also interfaces to a cellular processor 90, which uses a cellular modem 92 to transmit digital images to a cellular network (not shown) using radio frequency transmissions via an antenna 94.
  • the two image capture stages 1 and 2, and the zoom and focus motors 5a and 5b may be part of an integrated assembly.
  • the clock drivers 13 and 15, as well as the analog signal processors 22 and 24 and the analog/digital converters included therewith, may be part of the integrated assembly.
  • FIG. 3 depicts a flow diagram showing a method for capturing digital images using the image capture assembly of FIG. 1.
  • the zoom lenses 3 and 4 are set to their default positions, which is preferably a wide angle position where the output of the first image sensor 12 is used to capture images in a preview mode for display on the color LCD image display 70 to enable the user to compose the images to be captured.
  • the user typically presses the zoom button 42c in order to set a desired field of view for the digital camera 1 OA.
  • the zoom position setting is compared to a value X at which the image capture function switches from the first image capture stage to the second image capture stage.
  • the first image capture stage 1 is used to capture images in the preview mode, while the second image capture stage 2 is used to capture autofocus images.
  • the first image capture stage 1 continues to capture images for preview on the display 70 (block 110) while, in block 106, the second image capture stage 2 is used to capture autofocus images for autofocus of the first image capture stage 1 , which are processed by the image processor 50 and used in block 108 to focus the first image capture stage 1.
  • block 112 if the zoom button 42c is not pressed, and in block 114 if the capture button is pressed, a digital image is captured in block 116 with the first image capture stage 1.
  • control is returned to block 102, and if the capture button is not pressed in block 114, control is returned to block 106.
  • the zoom position setting is greater than X (a positive response to block 102)
  • the second image capture stage 2 is used to capture images in the preview mode, while the first image capture stage 1 is used to capture autofocus images.
  • the second image capture stage 2 continues to capture images for preview on the display 70 (block 130) while, in block 126, the first image capture stage 1 is used to capture autofocus images for autofocus of the second image capture stage 2, which are processed by the image processor 50 to generate a focus detection signal that is used in block 128 to focus the second image capture stage 2.
  • FIG. 4 A flow chart of an autofocus process using the two image capture stages shown in FIG. 3 in a rangefinder configuration is shown in FIG. 4, where the rangefinder method is used in blocks 108 and 128 of FIG. 3 to autofocus the images from the first and second image capture stages, hi block 250, the user determines the zoom position by adjusting the zoom control 42c on the camera, which in turn dictates, as described above, which image capture stage will be used to capture the final image and which image capture stage will be used only for autofocus images (block 252).
  • the image capture stage that will not be used for capture of the final image is zoomed to a position closest to the transition zoom position between the two zoom lens systems in the two image capture stages (block 254).
  • the focus lenses of the two image stages are moved to their respective hyperfocal positions, wherein the greatest focus range is produced (block 256).
  • the capture button has 3 positions: SO is the neutral position that the button maintains prior to the operator touching the capture button; Sl is the intermediate position in which the camera performs autofocus and autoexposure; S2 is the final position in which the camera performs a final autoexposure and captures the final image.
  • Autofocus images are then captured by both image stages (block 260) with their zoom lenses at their respective zoom positions.
  • the autofocus image from the image stage in the lower zoom position i.e., where the zoom position is less than X (see block 102 in FIG. 3) is then cropped and upsampled so that corresponding features in the two autofocus images span the same number of pixels (block 262).
  • the cropped and upsampled autofocus image is then correlated with the other autofocus image to identify the pixel shift between the two autofocus images (block 264) and thereby produce the focus detection signal.
  • Fig. 17A shows a representation of the autofocus image 350 as captured from the higher zoom position image stage.
  • Fig 17B shows a representation of a cropped and upsampled autofocus image 352 from the image stage in the lower zoom position. These representations show the offset between the two images that is used to determine the distance to portions of the scene.
  • a calibration factor is then applied to the focus detection signal to determine the distance that the focus lens must be moved to produce a best focus condition for the image capture.
  • the focus detection signal is applied to the zoom and focus motor 5a and the focus lens is then moved the determined distance in the image capture stage that will be used for the final image capture to produce the condition for best focus 268.
  • the capture button When the user pushes the capture button from Sl to S2, the image capture stage capture the final image.
  • FIG. 5 in a well known "hill climb” contrast comparison method is shown in FIG. 5, where the "hill climb” contrast comparison method is used in blocks 108 and 128 of FIG. 3 to autofocus the images from the first and second image capture stages.
  • the user selects a zoom position. The zoom position determines which imaging stage will be used as the capture stage (block 252). Then, in block 254, the image capture stage not used for capture is zoomed to the point closest to the user selected zoom position from block 250.
  • the user pushes the capture button 42a from the SO position to the Sl position to initiate an autofocus sequence. Then, in block 272, both image capture stages are autofocused by the "hill climb” method.
  • video images are continuously captured by the capture stage (block 276).
  • focus is continually checked in block 278 with the image capture stage not being used for capture using the "hill climb" contrast comparison method. Then, in the decision block 280, if the focus is good, control is returned to block 276 and video images are continuously captured by the capture stage. If, in the decision block 280, if the focus is not good, then the focus lens adjustment is identified (block 282) that is needed to produce best focus on the image capture stage not being used for capture. In block 284, the autofocus "hill climb" method calibration curve and the identified focus lens adjustment is used to determine the movement of the focus lens needed in the capture stage to produce best focus, thereby producing a focus change detection signal.
  • the focus change detection signal is applied to the zoom and focus motor 5a or 5b and the focus lens in the capture stage is moved to the new best focus position, and control is returned to block 276 and video images are continuously captured by the capture stage.
  • focus is continually checked in block 278 with the image capture stage not being used for capture using the "whole way" autofocus method. Accordingly, in block 284, an autofocus "whole way” method calibration curve and the identified focus lens adjustment is used to determine the movement of the focus lens needed in the capture stage to produce best focus, thereby producing a focus change detection signal.
  • the focus change detection signal is applied to the zoom and focus motor 5a or 5b and the focus lens in the capture stage is moved to the new best focus position, and control is returned to block 276 and video images are continuously captured by the capture stage.
  • Calibration curves are used in both FIGS. 4 and 5 to determine the movement of the focus lens needed in the capture stage to produce best focus.
  • FIGS. 6 and 7 depict flow diagrams for calculating these curves. More specifically, FIG. 6 depicts calculation of the autofocus rangefinder calibration curve used in block 266 of FIG. 4.
  • a series of image sets are captured with objects at known distances, using the shorter focal length first image capture stage and the longer focal length second image capture stage at a series of focus lens positions.
  • the autofocus image from the lower focal length first image stage is cropped and upsampled so that corresponding features in the two autofocus images span the same number of pixels as shown in Figs. 17A and 17B.
  • the images from the second image capture stage are correlated to corresponding portions of the images from the cropped and upsampled image from the first image capture stage to determine the pixel offset between the images in each image set.
  • the data of pixel offset between images in each image set versus known distance to objects is stored as the autofocus rangefinder calibration curve for use in block 266 of FIG. 4.
  • FIG. 7 depicts calculation of the "hill climb” calibration curve used in block 284 of FIG. 5.
  • a series of image sets are captured with objects at known distances, using the first image capture stage and the second image capture stage - wherein autofocus is done by the "hill climb” method for each image.
  • the focus lens positions is compared for the two image capture stages versus the distance to the focused objects in the image sets.
  • the data of focus lens positions of the first image capture stage versus the focus lens positions of the second image capture stage for the same distance to focused objects in the images is stored as an autofocus "hill climb” method calibration curve for use in block 284 of FIG. 5.
  • FIG. 5 depicts calculation of the "hill climb" calibration curve used in block 284 of FIG. 5.
  • FIG. 8 depicts a flow diagram showing a method for capturing video images using the image capture assembly of FIG. 1. Much of the flow diagram duplicates the functional elements shown in FIG. 3, and will not be repeated here where the same functions and reference characters are indicated. If the zoom position setting is less than X (a negative response to block 102), then the first image capture stage 1 is used to capture video images, while the second image capture stage 2 is used to capture autofocus images. The focus of the first image capture stage is performed as described in FIG. 3, except it is now for a video image. Subsequently, in block 112 in FIG. 8, if the zoom button 42c is not pressed, and in block 114 if the capture button is pressed, a video image is captured in block 118 by the first image capture stage 1.
  • the video image is checked for focus quality in block 119, and if there is a need to refocus, control is returned to block 106 and the focus change detection signal is generated, which is used in block 108 to drive the focus motor 5a for the first image capture stage 1. If there is no need to refocus, then control is returned to block 112. If the zoom position setting is greater than X (a positive response to block
  • the second image capture stage 2 is used to capture video images, while the first image capture stage 1 is used to capture autofocus images.
  • the focus of the second image capture stage is performed as described in FIG. 3, except it is now for a video image.
  • the zoom button 42c is not pressed, and if in block 134 if the capture button is pressed, a video image is captured in block 138 with the second image capture stage 2.
  • the video image is checked for focus quality in block 139, and if there is a need to refocus, control is returned to block 126 and the focus change detection signal is generated, which is used in block 128 to drive the focus motor 5b for the second image capture stage 2. If there is no need to refocus, then control is returned to block 132.
  • FIG. 9 A flow chart of an autofocus process using the two image capture stages shown in FIG. 8 in a rangefinder configuration is shown in FIG. 9, where the rangefinder method is used in blocks 108 and 128 of FIG. 8 to autofocus the images from the first and second image capture stages.
  • a first autofocus image is captured with the lower focal length image capture stage.
  • the autofocus image from the image stage in the lower focal length is cropped and upsampled so that corresponding features in the two autofocus images span the same number of pixels.
  • a second autofocus image is captured with the longer focal length image capture stage.
  • the second autofocus image is correlated in block 444 with the cropped and upsampled image to determine the pixel offset between the images for different portions of the images. Then, in block 446, the focus correction needed is determined from the offset and the autofocus rangefinder calibration curve (which was calculated in FIG. 6).
  • FIG. 10 A flow chart of an autofocus process using the two image capture stages shown in FIG. 8 in a "hill climb” contrast comparison method is shown in FIG. 10, where the "hill climb" contrast comparison method is used in blocks 108 and 128 of FIG. 8 to autofocus the images from the first and second image capture stages.
  • a first image is captured with the first image capture stage, wherein the autofocus is done by the "hill climb” method.
  • the image from the first image capture stage is provided as a preview image on the display.
  • a second image is captured with the second image capture stage, wherein the autofocus is done by the "hill climb” method.
  • the focus conditions are changed for the first image capture stage based on the change of the focus conditions for the second image capture stage and the autofocus "hill climb" method calibration curve (which was calculated in FlG. 7). If the focus conditions have not changed (negative response to block 472), control is returned to block 468 and another image is captured with the second image capture stage. Alternatively, and in order to maintain focus with a minimal amount of hunting, focus is continually checked in block 470 with the second image capture stage using the "whole way" autofocus method. Accordingly, if the focus conditions have changed (positive response to block 472), the focus conditions are changed for the first image capture stage based on the change of the focus conditions for the second image capture stage and an autofocus "whole way” method calibration curve. If the focus conditions have not changed (negative response to block 472), control is returned to block 468 and another image is captured with the second image capture stage.
  • FIG. 11 depicts a flow diagram showing a method for processing images captured using the image capture assemblies of FIGS. 3 or 8, wherein a range map is produced.
  • Methods to produce a rangemap are well known to those skilled in the art; for example, a description of a method for producing a rangemap or depth map from a disparity map produced from the pixel offset information for a set of images captured by multiple cameras with similar fields of view is described in United States Patent Application Publication Number 2006/0193509 (published August 31, 2006 in the names of Antonio Criminisi et al., and entitled "Stereo-based Image Processing"), which is incorporated herein by reference.
  • two or more image capture devices are included in a single electronic camera. Since the two or more image capture devices have different focal lengths, at least one of the images must be modified to make the two or more images comparable to enable the pixel offsets to be determined.
  • a first autofocus image is captured with the lower focal length image capture stage and, in block 442, the autofocus image from the image capture stage in the lower zoom position is cropped and upsampled so that corresponding features in the two autofocus images span the same numbers of pixels.
  • a second autofocus image is captured with the higher focal length image capture stage.
  • the second autofocus image is correlated with the cropped and upsampled image to determine the pixel offset between the images for different portions of the images.
  • the pixel offsets are then converted in block 482 to distances from the image capture device using the autofocus rangefinder calibration curve.
  • a map is then produced in block 484 showing the distances to different portions of the images.
  • the distance from the image capture device to portions of the scene can be calculated from the measured pixel offsets between corresponding portions of the first and second autofocus images, that is, between corresponding portions of the cropped and upsampled image obtained in block 442 from the first autofocus image and the second autofocus image obtained from block 448.
  • the relationship between the pixel offset as experienced on the image sensors/?, the pixel size m, the spacing between lenses s, the effective focal length of the lens/and the distance d to the portion of the scene is given as
  • Table 1 shows typical pixel offsets for an image capture device as described.
  • the relationship between pixel offset and distance to portions of the scene is calibrated against objects in a scene with known distances to compensate for any unexpected variations in dimensions and any angular tilt between the two lens assemblies.
  • the range map is then used to modify the captured image signal or the output image for a variety of purposes, such as (without limitation): a) to improve object identification within the image by identifying the continuous boundaries of the object so the shape of the object can be defined; b) to enable object extraction from an image by identifying the continuous boundaries of the object so it can be segmented within the image; c) to enable motion tracking of objects within multiple images by identifying objects so they can be tracked as the same object between images; d) to enable dynamic depth of field images by blurring of portions of the image that correspond to areas of the scene that lie outside of the desired depth of field; e) to improve the dynamic range within images by applying gain adjustments to objects as a whole; f) to reduce exposure issues introduced by use of a flash by reducing the gain on the portions of the image that correspond to objects in the foreground and increasing the gain on objects in the background; g) to improve scene balance within the image by enabling objects in the foreground to be emphasized.
  • the exposure for various locations can be improved by applying gain adjustments to selected object portions of the image: in particular, e.g., to the cloud detail, the snow detail, and the fur on the black dog.
  • the range map can be used to improve dynamic range within the output image by applying gain adjustments to objects as a whole within the output image, and independently of their position in the range map.
  • the depth of field can be adjusted so that, e.g., the dog is in focus, the mountains are in focus and so are those great flowers.
  • the range data can be used to isolate the mountains and the flowers, which can then be blurred, and further to isolate the dog, which is sharpened to obtain a nice sharp image.
  • the user can make a dynamic depth of field, that is, with mixed regions of the ranges in focus.
  • the range map can be used to enable dynamic depth of field images by blurring portions of the output image, independently of their position in the range map, that correspond to areas of the scene that lie outside of a desired depth of field for a featured portion of the image.
  • the dog and mountains albeit they are at opposite range extremes, could be brought in focus because they are the regions of interest and the flowers in the mid range can be blurred smoothly.
  • FIGS. 12 and 13 show block diagrams of digital cameras using different variations of the optical image capture stages.
  • FIG. 12 depicts a block diagram of a digital camera having a first image capture stage 71 and a second image capture stage 2.
  • the first stage 71 includes a first fixed focal length lens 73 and the first image sensor 12, and the second stage 2 includes the second (zoom) lens 4 and second image sensor 14.
  • the focusing is carried out between the focus adjustments of the first and second image capture stages 71 and 2.
  • the focal length of the fixed focal length lens typically generates an ultra- wide angle field of view, e.g., 22mm equiv., it may have a fixed focus set to a distance near the lens hyperfocal distance of, e.g., 8 feet, so that objects from 4 feet to infinity are in focus. Therefore, the fixed focal length lens does not need to include a focus adjustment.
  • the fixed focal length lens includes an adjustable aperture and shutter assembly (not shown) to control the exposure of the image sensor.
  • FIG. 13 depicts a block diagram of a digital camera having a first image capture stage 71 , a second image capture stage 2, and a third image capture stage 74.
  • the first stage 71 includes a first fixed focal length lens 73 with a first image sensor 12, the second stage 2 includes a second (zoom) lens 4 with a second image sensor 14, and the third image capture stage 74 includes a third (zoom) lens 75 with a third image sensor 16.
  • the first lens 73 is typically a ultra-wide angle lens
  • the second (zoom) lens 4 is typically a wide angle zoom lens
  • the third (zoom) lens 75 is typically a telephoto zoom lens. Since, as mentioned in the preceding paragraph, the first fixed focal length lens 73, being an ultra- wide angle lens, is typically not focused, the focusing is carried out between the second and third image capture stages 2 and 74.
  • FIG. 14 depicts a flow diagram showing a method for enhancing the depth of field of an image by using images from both image capture stages from FIG.l.
  • the zoom position is set to a default position when the camera is powered on.
  • the zoom position setting is compared to a value X at which the image capture function switches from the first image capture stage to the second image capture stage.
  • the zoom position setting is less than X (a negative response to block 502), then the first image capture stage 1 is used to capture images in the preview mode, while the second image capture stage 2 is used to capture autofocus images.
  • the first image capture stage 1 continues to capture images for preview on the display 70 (block 506) while the second image capture stage 2 is used to capture autofocus images for autofocus of the first image capture stage 1 , which are processed by the image processor 50 and used to focus the first image capture stage 1.
  • the zoom button not be pressed (negative response to block 508), and when the shutter button 42a is pressed, a primary still image is captured in block 510 using the first image capture stage set to a primary focus position.
  • a secondary still image is captured using the second image capture stage set to a secondary focus position.
  • the secondary still image is used to enhance the depth of field of the primary image, for instance, where the secondary still image is used to provide an enhancement signal that can be used to sharpen portions of the primary still image that are positioned near the secondary focus distance.
  • the zoom position setting is greater than X (a positive response to block 502)
  • the second image capture stage 2 is used to capture images in the preview mode, while the first image capture stage 1 is used to capture autofocus images.
  • the second image capture stage 2 continues to capture images for preview on the display 70 (block 526) while the first image capture stage 1 is used to capture autofocus images for autofocus of the second image capture stage 2, which are processed by the image processor 50 and used to focus the second image capture stage 2.
  • a primary still image is captured in block 530 using the second image capture stage set to a primary focus position.
  • a secondary still image is captured using the first image capture stage set to a secondary focus position.
  • the secondary still image is used to enhance the depth of field of the primary image, for instance, where the secondary still image is used to sharpen portions of the primary still image that are positioned near the secondary focus distance.
  • the enhancement signal is generated by the camera to sharpen portions of the primary still image that are positioned near the secondary focus distance.
  • the primary and secondary still images may submitted to an external processor, such as the host PC 66 illustrated in FIG. 1 , and the enhancement signal is generated by the external processor to sharpen portions of the primary still image that are positioned near the secondary focus distance.
  • the concept of multiple lenses and multiple sensors, and the use of an integrated image capture assembly may be adapted for use in a cell phone of the type having a picture taking capability. Accordingly, and as shown in FIG.
  • a cell phone 600 includes a phone stage comprising a microphone 602 for capturing the voice of a caller, related electronics (not shown) for processing the voice signals of the caller and the person called, and a speaker 604 for reproducing the voice of the one called.
  • a keypad 606 is provided for entering phone numbers and image capture commands, and a (LCD) display 608 for showing phone-related data and for reproducing images captured by the phone or received over the cellular network.
  • the rear view of the cell phone 600 shown in FIG. 15B identifies some of the internal components, including a cellular image capture assembly 610 connected via the image processor 50 (as shown in FIG. 1) to a cellular processing stage comprising the cellular processor 90 and the modem 92.
  • the cellular processor 90 receives and processes the image data from the image processor 50 and the voice data captured by the microphone 602, and transfers the image and voice data to the cellular modem 92.
  • the cellular modem 92 converts the digital image and voice data into the appropriate format for transmission by the antenna 94 to a cellular network.
  • the assembly 610 comprises an integrated packaging of the optical and imaging components on a common substrate 620. More specifically, the assembly 610 includes a first fixed focal length lens 612 and a first image sensor 614, and a second fixed focal length lens 616 and a second image sensor 618.
  • the first lens 612 preferably a fixed focal length wide angle lens (such as a 40 mm equiv. lens), forms an image on the first image sensor 614
  • the second lens 616 preferably a fixed focal length telephoto lens (such as 100 mm equiv. lens), forms an image on the second image sensor 618. Both of the lenses are oriented in the same direction in order to form images of the same portion of the overall scene in front of them, albeit with different fields of view.
  • Each lens 612 and 616 and each associated image sensor 614 and 618 are mounted to the substrate 620 with an IR cut filter 622 in between to reduce the incidence of IR radiation on the image pixels.
  • Electronic components 624 such as resistors, capacitors and power management components, are also mounted on the substrate 620.
  • the image signals are taken from the substrate 620 via a flex connector 626.
  • the data taken from the assembly 610 may be raw image data, or if suitable processors (not shown) are on board the substrate 620, the data could be YUV image data or JPEG image data.
  • the image processor 50 may provide digital zooming between the wide angle and the telephoto focal lengths; the user may initiate such zooming via a user interface displayed on the (LCD) display 608 and by keying appropriate buttons on the keypad 606.
  • the wide angle image sensor 614 may have high resolution, e.g., higher than that of the telephoto image sensor 618, in order to provide a higher quality source image for the digital zooming.
  • the image processor 50 either (a) selects the sensor output from the wide angle lens 612 as the captured image signal and uses the sensor output from the telephoto lens 616 to generate a focus detection signal for the wide angle lens 612 or (b) selects the sensor output from the telephoto lens 616 as the captured image signal and uses the sensor output from the wide angle lens 612 to generate the focus detection signal for the telephoto lens 616.
  • the focus detection signal is then applied to the auto focus subsystem 628 of the telephoto lens 616 in order to adjust the focus of the image providing the sensor output for the captured image signal.
  • the wide angle lens 612 could instead be a zoom lens, such as a wide angle to normal angle zoom lens.
  • the wide angle lens 612 is set to its hyperfocal distance, which means it is in focus from a few feet to infinity without need for any focus adjustment by the user.
  • the telephoto lens 616 is automatically focused by an auto focus subsystem 628. This is required because the hyperfocal distance increases as the focal length increases, and so the focus needs to be adjusted in order to obtain proper focus for objects at typical (e.g. 4' to 12') distances. By using only one focusing subsystem 628 for the telephoto lens 616, the cost and size can be reduced.
  • z dimension 630 which must be held to a very small figure consistent with a cell phone layout and architecture. This may be obtained by careful choice of the telephoto focal length and the size of the sensor. For example, the size of the sensor 616, and consequently the size of the image that must be produced to fill the sensor, may be made small enough to reduce the focal length to an acceptable z dimension 630.
  • the two lenses may have approximately identical focal lengths, with the imaging arrays being of different sizes.
  • each lens is designed to fill the area of the imaging array and each lens-array combination will have substantially the same actual focal length, i.e., the same lens to array distance.
  • the 35mm equiv. of each lens will be different; consequently, each lens will have a different field of view.
  • an analog output signal from the first image sensor 614 is amplified by a first analog signal processor and provided to a first input of a control element, e.g., an analog multiplexer control element provided as one of the electronic components 624 on the substrate 620.
  • the analog output signal from the second image sensor 618 is amplified by a second analog signal processor and provided to a second input of the control element.
  • the function of the control element is to select either the first sensor output from the first image sensor 614 or the second sensor output from the second image sensor 618, depending on user input from the keypad 606 as to zoom selection, thereby providing a selected sensor output from the cellular image capture assembly 600 to the image processor 50.
  • the GPS location and the pointing direction of the camera which is provided by the GPS unit 57 and the electronic compass 59 in the camera, are combined with distances and directions from the camera to portions of the scene to determine the GPS locations for portions of the scene.
  • FIG. 18 depicts a flow chart for a method to determine GPS locations for portions of the scene.
  • the GPS location of the camera is determined by the GPS unit 57 in the camera.
  • the pointing direction of the camera is determined in block 752 by the electronic compass 59 in the camera.
  • Distance offsets from the camera to portions of the scene are provided by the range map, which is obtained in block 754 from a procedure such as depicted in FIG. 11.
  • Angular offsets from the pointing direction of the camera are determined in block 756 from the location of a portion of the scene in the field of view in the image.
  • GPS locations for portions of the scene are determined in block 758 by adding the distance offsets and the angular offsets to the GPS location and pointing direction of the camera.
  • the GPS locations for portions of the scene are then stored in block 760 in the metadata for the image or displayed as labels on the image in the form of a GPS location map.
  • GPS location for a portion of the image can be displayed on the electronic camera.
  • GPS locations of portions of the scene can be used for a variety of purposes, including without limitation: determining the identity of an object; providing navigational directions to a location in a scene; identifying the field of view of a camera relative to a map or other geographical database, etc.
  • FIG. 19 depicts a flow diagram illustrating a process according to the invention for initially setting both imaging stages in a dual lens camera system to respective default zoom positions and then allowing the user to select one of the imaging stages in the dual lens camera system as the primary capture unit, while relegating the other imaging stage to certain other functions, such as scene analysis.
  • the flow diagram in FIG. 19 further specifies the selection of one of the imaging stages as a default stage if an operator selection has not been made within a predetermined time, or if the capture function of the camera has been actuated before the time has run out. More specifically, the power to the camera is turned on and the initialization process begins (block 1100).
  • the first and second imaging stages 1 and 2 are set to their default zoom positions (block 1102), which are predetermined initial zoom positions that determine the images that are initially captured and displayed. Thereupon, the first and second imaging stages capture and display first and second preview images (block 1104) on the image display 70. These images could be displayed in several ways, for instance, side by side on the display as thumbnails or the like, one within the other, sequentially, etc.
  • the camera operator is asked to decide whether the first or second preview image should be the primary image (block 1106), where the primary image will be the one that is captured and stored. The operator is given a display time interval x in which to make this decision (block 1110).
  • a predetermined one of the imaging stages (the default imaging stage) is automatically set to be the primary capture unit (block 1 1 12).
  • the default imaging stage could be pre-selected for a variety of reasons, one being to provide a comprehensive, wider angle view of the scene as the initial image.)
  • the selected imaging stage is set to be the primary capture unit (block 1124).
  • the other (non-selected) imaging stage is designated as the scene analysis capture unit (block 1114). Thereupon, the primary capture unit is operated in the preview mode, as subsequently explained in connection with FIGS. 20-22.
  • FIGS. 20-22 and 24-26 depict flow diagrams illustrating the usage or analysis of the image from one imaging stage in a dual (or more) lens camera system to modify the image produced by another imaging stage, that is, to influence, analyze, augment or otherwise change the image produced by the other imaging stage.
  • the scene analysis capture unit analyzes the scene (block 1200) and the image processor 50 sets the primary capture unit parameters utilizing the scene analysis data obtained by the scene analysis capture unit (block 1202).
  • Such scene analysis data could include without limitation exposure data, dynamic range data, depth of field data, color balance, identification of different aspects of the scene including faces, grass, sunset, snow, etc, and the capture unit parameters could include without limitation aperture value, exposure time, focus position, white balance, ISO setting, etc.
  • the preview image is then captured by the primary capture unit and displayed on the display 70 (block 1204).
  • the scene analysis capture unit then continues to analyze the scene (block 1206), and if the scene conditions have not changed (no to block 1208) the process loops back to block 1204 and a preview image is again captured by the primary capture unit and displayed on the display 70 (without changing the capture parameters). If the scene conditions have changed (yes to block 1208), the process loops back to block 1202 and the image processor 50 resets the primary capture unit parameters utilizing the scene analysis data obtained by the scene analysis capture unit, and the subsequent process is repeated as before.
  • the primary capture unit parameters are only changed if the change in scene conditions exceed a threshold value x.
  • the scene analysis capture unit captures an image and analyzes the scene (block 1300).
  • the image processor 50 sets the primary capture unit parameters utilizing the scene analysis data obtained by the scene analysis capture unit (block 1302).
  • a preview image is then captured by the primary capture unit and displayed on the display 70 (block 1304).
  • another image is captured from the scene analysis capture unit (block 1306).
  • the process loops back to block 1304 and another preview image is captured by the primary capture unit and displayed on the display 70 (without changing the capture parameters). Then, another image is captured from the scene analysis capture unit (block 1306). Otherwise, when the scene condition change is greater than the threshold x and the situation is considered to be unstable (yes to block 1308), the process loops back to block 1302 and the image processor 50 resets the primary capture unit parameters utilizing the scene analysis data obtained by the scene analysis capture unit. Then, the process is repeated as before. In FIG.
  • the consideration of the threshold scene conditions is enhanced by considering the relative zoom positions of the capture units and utilizing scene information from the images captured by both of the capture units.
  • the zoom position of the scene analysis capture unit is set relative to the zoom position of the primary capture unit (block 1400).
  • the scene analysis capture unit captures an image (block 1402) and the image processor 50 sets the primary capture unit parameters utilizing the scene analysis data obtained by the scene analysis capture unit (block 1404).
  • a preview image is then captured by the primary capture unit (block 1406) and the scene is analyzed utilizing the captured preview image and the scene analysis data (block 1408).
  • the image processor 50 sets the primary capture unit parameters utilizing the results of the preceding scene analysis (block 1410), that is, results obtained by analyzing images from both capture units.
  • the preview image is captured by the primary capture unit and displayed on the display 70 (block 1412) and another image is captured from the scene analysis capture unit (block 1414).
  • the scene is then analyzed by utilizing the captured preview image data and the scene analysis data (block 1416). If the scene conditions fail to change by a threshold equal to x (no to block 1418), that is, the scene conditions are considered to be stable, the process loops back to block 1412 and another preview image and scene analysis image are captured by the primary capture unit and the scene analysis capture unit, respectively (blocks 1412 and 1414) and the aforementioned process continues.
  • the process loops back to block 1410 and the image processor 50 resets the primary capture unit parameters utilizing the results of new scene analysis data (obtained by the scene analysis capture unit in block 1416), and the process is repeated.
  • the preview process is continued as shown unless either the shutter button 42a or the zoom button 42c is pressed. As shown in FIG.
  • FIGS. 24-26 depict flow diagrams illustrating the usage or analysis of the image from one imaging stage in a dual (or more) lens camera system to modify the image produced by another imaging stage, that is, to influence, analyze, augment or otherwise change the image produced by the other imaging stage, specifically during the capture process.
  • the shutter button 42a is pressed half-way (Sl) - otherwise, the process returns to the preview mode.
  • a preview image is captured from the primary capture unit (block 1602).
  • the scene is next analyzed utilizing the captured preview image (block 1604).
  • the image processor 50 sets the primary capture unit parameters utilizing the results of the scene analysis (block 1608). Then, using the set parameters, a preview image is captured and displayed from the primary capture unit (block 1610).
  • the process loops back to the block 1602 and a preview image is captured from the primary capture unit and the scene is analyzed utilizing the captured preview image (block 1604). Then, the process continues as before.
  • the shutter button 42a is pressed (Sl) half-way. Then, a preview image is captured from the primary capture unit (block 1702). Next, an image is captured by the scene analysis capture unit (block 1704), and the scene is analyzed utilizing both the captured preview image and the captured scene analysis image (block 1706). The image processor 50 next sets the primary capture unit parameters utilizing the results of the combined scene analysis (block 1708). Then, using the set parameters, a preview image is captured and displayed from the primary capture unit (block 1710). Next, and as shown before in FIG.
  • the process loops back to block 1702, and a preview image is captured from the primary capture unit (block 1702) and the scene is analyzed utilizing the captured preview image (block 1704). Then, the process continues as before.
  • the shutter button 42a is pressed (Sl) half-way (block 1800). Then, a preview image is captured from the primary capture unit (block 1802) and an image is captured from the scene analysis capture unit (block 1804). The scene is next analyzed utilizing both the captured preview image and the captured scene analysis image (block 1806), and the image processor 50 sets the primary capture unit parameters utilizing the results of the combined scene analysis (block 1808). The scene analysis capture unit is next designated as a secondary capture unit (block 1810), and the image processor 50 sets the secondary capture unit parameters utilizing the results of the combined scene analysis (block 1812).
  • an image is captured from the primary capture unit at one focus position and another image is captured from the scene analysis capture unit (the secondary image capture unit) at another focus position. Then, the two images are combined into a modified image with a broadened depth of field.
  • the advantage is that this can be done without having to stop down the aperture of the primary lens to obtain the greater depth of field, which is particularly useful for low light capture where a large aperture is preferred.
  • the image could be examined by the image processor 50 for its dynamic range. When the dynamic range exceeds a predetermined threshold, the scene analysis capture unit is used to produce a secondary image with different dynamic range than the primary image.
  • the primary image could be captured at a normal exposure, while the secondary image could be captured at more of an extreme exposure, i.e., either under- or over-exposed. Then, if for example, highlights are blown out in the primary exposure, the secondary exposure could be underexposed to capture detail in the highlights. Alternatively, if shadows are darkened in the primary exposure, the secondary exposure could be overexposed to capture detail in the shadows. Then, a modified image is created with a broadened dynamic range by replacing portions of the primary image with portions of the secondary image.
  • the images captured by the primary and secondary capture units could be a still image or a video image, and in the case of a video image could be a series of images.
  • the still image or the series of images constituting the video signal may be modified in accordance with the particular form of augmentation described.
  • an electronic camera incorporating the capture units may produce a video image signal and the secondary output image is used to modify at least, e.g., the depth of field and/or the dynamic range of the series of images constituting the video image signal.
  • the flow diagram shown in FIG. 14 can be modified to obtain a primary image that is enhanced specifically for dynamic range. For instance, referring to FIG.
  • a primary still image is captured (similar to blocks 510 or 530) using the first (or second) image capture stage set to a primary exposure level. Then, (similar to blocks 512 or 532) a secondary still image is captured using the second (or first image) capture stage set to a secondary exposure level. Then, (similar to blocks 514 or 534) the secondary still image is used to enhance the dynamic range of the primary image, for instance, where the secondary exposure, which is underexposed to capture detail in the highlights, is combined with the primary image to obtain an extended dynamic range primary image.
  • a sliding scale is used to create the modified image in which the pixel values are determined by considering the pixel values of both the primary and secondary images, as described in commonly-assigned, copending United States Patent Application Serial Number 1 1/460,364 (which was filed July 27, 2006 in the names of John Border and Efrain Morales, and entitled "Producing an Extended Dynamic Range Digital Image"), which is incorporated herein by reference. It should be noted that having two exposure levels is to enable a correction for dynamic range is particularly important when using a flash, wherein overexposure conditions are often produced. Consequently, the exposure time of one of the capture stages will be set to a very short exposure or the timing of the exposure will be shifted to be either just before the flash or just after the flash.
  • the augmentation process can also be applied in connection with image pairs having different resolutions.
  • a first wide angle digital image of a scene and a second telephoto digital image of a portion of substantially the same scene are captured by two capture stages.
  • a composite image is then formed by combining a portion of the first wide angle digital image and a portion of the telephoto digital image, to produce a digital image with improved resolution during digital zooming. More specifically, digital zooming of the composite image produces a zoomed image with high resolution throughout the zoom range with improved image quality.
  • the primary capture stage and secondary capture stage are set for different exposure times so that different levels of noise and motion blur are present in the respective images.
  • the primary capture stage is set for a relatively long exposure so that the digital noise in the image is low, but any motion present either from movement of the camera or from movement of objects in the scene results in motion blur.
  • the secondary capture stage is set for a relatively fast exposure so that digital noise in the image is higher, but the motion blur is less.
  • the primary image and the secondary image are then compared to one another to identify motion blur.
  • the gain of the secondary image is increased so the average pixel values in the secondary image match those of the primary image.
  • a modified image is then created by replacing portions of the primary image with portions of the secondary image. In effect, portions of the primary image (areas of lower noise but with some motion blur) are replaced with corresponding portions of the secondary image (areas of higher noise but little or no motion blur) to obtain a modified image with relatively low noise and good sharpness.
  • focus change check block 480 correlate images block, determining pixel offsets

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Studio Devices (AREA)
  • Cameras In General (AREA)

Abstract

La présente invention concerne une caméra électronique qui comprend les première et seconde étapes d'imagerie pour capturer des images distinctes d'une scène, l'une des étapes étant indiquée comme étape d'imagerie par défaut. Un processeur permet de capturer et d'afficher les images distinctes et répond en outre à une sélection par l'opérateur de l'une des étapes d'imagerie comme unité de capture principale qui doit être surtout utilisée pour capturer une image de la scène qui est stockée par la caméra numérique. Si la sélection par l'opérateur ne se produit pas pendant une période de temps prédéterminée, ou si la caméra est actionnée avant l'expiration du temps, le processeur sélectionne automatiquement l'étape d'imagerie par défaut comme unité de capture principale.
EP08725358A 2007-03-09 2008-02-08 Caméra multiobjectif pouvant fonctionner dans divers modes Not-in-force EP2135442B1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US11/684,050 US7729602B2 (en) 2007-03-09 2007-03-09 Camera using multiple lenses and image sensors operable in a default imaging mode
PCT/US2008/001713 WO2008112055A1 (fr) 2007-03-09 2008-02-08 Caméra multiobjectif pouvant fonctionner dans divers modes

Publications (2)

Publication Number Publication Date
EP2135442A1 true EP2135442A1 (fr) 2009-12-23
EP2135442B1 EP2135442B1 (fr) 2012-02-08

Family

ID=39493264

Family Applications (1)

Application Number Title Priority Date Filing Date
EP08725358A Not-in-force EP2135442B1 (fr) 2007-03-09 2008-02-08 Caméra multiobjectif pouvant fonctionner dans divers modes

Country Status (6)

Country Link
US (1) US7729602B2 (fr)
EP (1) EP2135442B1 (fr)
JP (1) JP5150651B2 (fr)
CN (1) CN101632295A (fr)
AT (1) ATE545272T1 (fr)
WO (1) WO2008112055A1 (fr)

Families Citing this family (188)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8139130B2 (en) 2005-07-28 2012-03-20 Omnivision Technologies, Inc. Image sensor with improved light sensitivity
US8274715B2 (en) 2005-07-28 2012-09-25 Omnivision Technologies, Inc. Processing color and panchromatic pixels
US7916362B2 (en) 2006-05-22 2011-03-29 Eastman Kodak Company Image sensor with improved light sensitivity
US8031258B2 (en) 2006-10-04 2011-10-04 Omnivision Technologies, Inc. Providing multiple video signals from single sensor
US7859588B2 (en) * 2007-03-09 2010-12-28 Eastman Kodak Company Method and apparatus for operating a dual lens camera to augment an image
JP4959535B2 (ja) * 2007-12-13 2012-06-27 株式会社日立製作所 撮像装置
WO2009097552A1 (fr) * 2008-02-01 2009-08-06 Omnivision Cdm Optics, Inc. Systemes et procedes de fusion de donnees d’image
US11792538B2 (en) 2008-05-20 2023-10-17 Adeia Imaging Llc Capturing and processing of images including occlusions focused on an image sensor by a lens stack array
KR101733443B1 (ko) 2008-05-20 2017-05-10 펠리칸 이매징 코포레이션 이종 이미저를 구비한 모놀리식 카메라 어레이를 이용한 이미지의 캡처링 및 처리
US8866920B2 (en) 2008-05-20 2014-10-21 Pelican Imaging Corporation Capturing and processing of images using monolithic camera array with heterogeneous imagers
US8164682B2 (en) * 2008-10-14 2012-04-24 Omnivision Technologies, Inc. Dithered focus evaluation
US20100226448A1 (en) * 2009-03-05 2010-09-09 Paul Wilkinson Dent Channel extrapolation from one frequency and time to another
US20100271533A1 (en) * 2009-04-28 2010-10-28 Crispin Porter & Bogusky Image Recordation Device with a Plurality of Lenses
US9380292B2 (en) 2009-07-31 2016-06-28 3Dmedia Corporation Methods, systems, and computer-readable storage media for generating three-dimensional (3D) images of a scene
US9792012B2 (en) 2009-10-01 2017-10-17 Mobile Imaging In Sweden Ab Method relating to digital images
US20160360121A1 (en) * 2009-11-09 2016-12-08 Yi-Chuan Cheng Portable device with successive extension zooming capability
KR101725888B1 (ko) * 2009-11-13 2017-04-13 삼성전자주식회사 카메라 또는 원격 제어 장치에서의 이미지 제공 방법 및 그 장치
EP2502115A4 (fr) 2009-11-20 2013-11-06 Pelican Imaging Corp Capture et traitement d'images au moyen d'un réseau de caméras monolithique équipé d'imageurs hétérogènes
JP2011135490A (ja) * 2009-12-25 2011-07-07 Seiko Epson Corp デジタルカメラおよびその製造方法
JP2011133796A (ja) * 2009-12-25 2011-07-07 Seiko Epson Corp カメラおよびその製造方法
KR101214536B1 (ko) * 2010-01-12 2013-01-10 삼성전자주식회사 뎁스 정보를 이용한 아웃 포커스 수행 방법 및 이를 적용한 카메라
SE534551C2 (sv) 2010-02-15 2011-10-04 Scalado Ab Digital bildmanipulation innefattande identifiering av ett målområde i en målbild och sömlös ersättning av bildinformation utifrån en källbild
WO2011159401A1 (fr) 2010-05-03 2011-12-22 Invisage Technologies, Inc. Dispositifs et procédés de capture d'image et de vidéo haute résolution
US20120012748A1 (en) 2010-05-12 2012-01-19 Pelican Imaging Corporation Architectures for imager arrays and array cameras
US8655893B2 (en) * 2010-07-16 2014-02-18 Shutterfly, Inc. Organizing images captured by multiple image capture devices
US9270838B2 (en) * 2010-07-20 2016-02-23 Kodak Alaris Inc. Verifying accuracy of a scanned document
US9344701B2 (en) * 2010-07-23 2016-05-17 3Dmedia Corporation Methods, systems, and computer-readable storage media for identifying a rough depth map in a scene and for determining a stereo-base distance for three-dimensional (3D) content creation
JP5547854B2 (ja) * 2010-11-29 2014-07-16 デジタルオプティックス・コーポレイション・ヨーロッパ・リミテッド 携帯型装置で捕捉される多重画像からの肖像画像合成
US8878950B2 (en) 2010-12-14 2014-11-04 Pelican Imaging Corporation Systems and methods for synthesizing high resolution images using super-resolution processes
US8836767B2 (en) * 2010-12-17 2014-09-16 Samsung Electronics Co., Ltd Imaging apparatus and imaging method
US8941751B2 (en) 2011-05-06 2015-01-27 Empire Technology Development Llc Higher resolution still image generation from lower resolution video camera frames
US8553129B2 (en) * 2011-05-10 2013-10-08 Htc Corporation Handheld electronic device with two lens modules, dual image capturing method applying for the handheld electronic device, and computer program product for load into the handheld electronic device
KR101973822B1 (ko) 2011-05-11 2019-04-29 포토네이션 케이맨 리미티드 어레이 카메라 이미지 데이터를 송신 및 수신하기 위한 시스템들 및 방법들
SE1150505A1 (sv) 2011-05-31 2012-12-01 Mobile Imaging In Sweden Ab Metod och anordning för tagning av bilder
US20120320237A1 (en) * 2011-06-15 2012-12-20 Wei-Ting Liu Camera with function of removing unwanted object and method thereof
US8868684B2 (en) * 2011-06-17 2014-10-21 At&T Intellectual Property I, L.P. Telepresence simulation with multiple interconnected devices
US8964102B2 (en) * 2011-06-29 2015-02-24 Maxim Integrated Products, Inc. Self-calibrated ringing compensation for an autofocus actuator in a camera module
EP2718896A4 (fr) * 2011-07-15 2015-07-01 Mobile Imaging In Sweden Ab Procédé pour fournir une représentation graphique numérique ajustée d'une vue et appareil approprié
US8754977B2 (en) * 2011-07-28 2014-06-17 Hewlett-Packard Development Company, L.P. Second camera for finding focal target in poorly exposed region of frame taken by first camera
KR101843450B1 (ko) 2011-08-23 2018-03-29 엘지전자 주식회사 이동 단말기 및 이동 단말기의 제어 방법
US20130070060A1 (en) 2011-09-19 2013-03-21 Pelican Imaging Corporation Systems and methods for determining depth from multiple views of a scene that include aliasing using hypothesized fusion
EP2761534B1 (fr) 2011-09-28 2020-11-18 FotoNation Limited Systèmes de codage de fichiers d'image de champ lumineux
US8729653B2 (en) 2011-10-26 2014-05-20 Omnivision Technologies, Inc. Integrated die-level cameras and methods of manufacturing the same
JP5898481B2 (ja) * 2011-12-13 2016-04-06 キヤノン株式会社 撮像装置及び焦点検出方法
US8872944B2 (en) * 2011-12-19 2014-10-28 Pelco, Inc. Method and system for image centering during zooming
US20130162835A1 (en) * 2011-12-23 2013-06-27 Fluke Corporation Thermal imaging camera for infrared rephotography
EP2817955B1 (fr) 2012-02-21 2018-04-11 FotoNation Cayman Limited Systèmes et procédés pour la manipulation de données d'image de champ lumineux capturé
CN103327221B (zh) * 2012-03-20 2016-12-14 华晶科技股份有限公司 摄像装置及其影像预览系统及影像预览方法
US9712738B2 (en) 2012-04-17 2017-07-18 E-Vision Smart Optics, Inc. Systems, devices, and methods for managing camera focus
US9210392B2 (en) 2012-05-01 2015-12-08 Pelican Imaging Coporation Camera modules patterned with pi filter groups
JP2015534734A (ja) 2012-06-28 2015-12-03 ペリカン イメージング コーポレイション 欠陥のあるカメラアレイ、光学アレイ、およびセンサを検出するためのシステムおよび方法
US20140002674A1 (en) 2012-06-30 2014-01-02 Pelican Imaging Corporation Systems and Methods for Manufacturing Camera Modules Using Active Alignment of Lens Stack Arrays and Sensors
EP3869797B1 (fr) 2012-08-21 2023-07-19 Adeia Imaging LLC Procédé pour détection de profondeur dans des images capturées à l'aide de caméras en réseau
US20140055632A1 (en) 2012-08-23 2014-02-27 Pelican Imaging Corporation Feature based high resolution motion estimation from low resolution images captured using an array source
EP2901671A4 (fr) 2012-09-28 2016-08-24 Pelican Imaging Corp Création d'images à partir de champs de lumière en utilisant des points de vue virtuels
US9143711B2 (en) 2012-11-13 2015-09-22 Pelican Imaging Corporation Systems and methods for array camera focal plane control
CN105556944B (zh) 2012-11-28 2019-03-08 核心光电有限公司 多孔径成像系统和方法
US9462164B2 (en) 2013-02-21 2016-10-04 Pelican Imaging Corporation Systems and methods for generating compressed light field representation data using captured light fields, array geometry, and parallax information
US9374512B2 (en) 2013-02-24 2016-06-21 Pelican Imaging Corporation Thin form factor computational array cameras and modular array cameras
US9774789B2 (en) 2013-03-08 2017-09-26 Fotonation Cayman Limited Systems and methods for high dynamic range imaging using array cameras
US8866912B2 (en) 2013-03-10 2014-10-21 Pelican Imaging Corporation System and methods for calibration of an array camera using a single captured image
US9888194B2 (en) 2013-03-13 2018-02-06 Fotonation Cayman Limited Array camera architecture implementing quantum film image sensors
WO2014164550A2 (fr) 2013-03-13 2014-10-09 Pelican Imaging Corporation Systèmes et procédés de calibrage d'une caméra réseau
WO2014159779A1 (fr) 2013-03-14 2014-10-02 Pelican Imaging Corporation Systèmes et procédés de réduction du flou cinétique dans des images ou une vidéo par luminosité ultra faible avec des caméras en réseau
WO2014153098A1 (fr) 2013-03-14 2014-09-25 Pelican Imaging Corporation Normalisation photométrique dans des caméras matricielles
US9497429B2 (en) 2013-03-15 2016-11-15 Pelican Imaging Corporation Extended color processing on pelican array cameras
US10122993B2 (en) 2013-03-15 2018-11-06 Fotonation Limited Autofocus system for a conventional camera that uses depth information from an array camera
US9438888B2 (en) 2013-03-15 2016-09-06 Pelican Imaging Corporation Systems and methods for stereo imaging with camera arrays
US9445003B1 (en) 2013-03-15 2016-09-13 Pelican Imaging Corporation Systems and methods for synthesizing high resolution images using image deconvolution based on motion and depth information
WO2014185970A1 (fr) * 2013-05-14 2014-11-20 Invisage Technologies, Inc. Capture d'images et de vidéos à haute résolution
GB2517674A (en) * 2013-05-17 2015-03-04 Wonga Technology Ltd Image capture using client device
JP2014230179A (ja) * 2013-05-24 2014-12-08 ソニー株式会社 撮像装置及び撮像方法
JP6139713B2 (ja) * 2013-06-13 2017-05-31 コアフォトニクス リミテッド デュアルアパーチャズームデジタルカメラ
US9857568B2 (en) 2013-07-04 2018-01-02 Corephotonics Ltd. Miniature telephoto lens assembly
CN108519655A (zh) 2013-07-04 2018-09-11 核心光电有限公司 小型长焦透镜套件
CN108989649B (zh) 2013-08-01 2021-03-19 核心光电有限公司 具有自动聚焦的纤薄多孔径成像系统及其使用方法
WO2015048694A2 (fr) 2013-09-27 2015-04-02 Pelican Imaging Corporation Systèmes et procédés destinés à la correction de la distorsion de la perspective utilisant la profondeur
US10119808B2 (en) 2013-11-18 2018-11-06 Fotonation Limited Systems and methods for estimating depth from projected texture using camera arrays
WO2015081279A1 (fr) 2013-11-26 2015-06-04 Pelican Imaging Corporation Configurations de caméras en réseau comprenant de multiples caméras en réseau constitutives
CA2837578C (fr) * 2013-11-29 2019-11-05 Avigilon Corporation Commande de systeme de camera pour corriger un decalage de viseur
US10306165B2 (en) 2013-12-06 2019-05-28 Huawei Device Co., Ltd. Image generating method and dual-lens device
KR101916355B1 (ko) 2013-12-06 2018-11-07 후아웨이 디바이스 (둥관) 컴퍼니 리미티드 듀얼-렌즈 장치의 촬영 방법, 및 듀얼-렌즈 장치
KR102126507B1 (ko) * 2013-12-09 2020-06-24 삼성전자주식회사 센서 데이터 스트림을 처리하는 단말기, 시스템 및 방법
WO2015134996A1 (fr) 2014-03-07 2015-09-11 Pelican Imaging Corporation Système et procédés pour une régularisation de profondeur et un matage interactif semi-automatique à l'aide d'images rvb-d
US9300877B2 (en) * 2014-05-05 2016-03-29 Omnivision Technologies, Inc. Optical zoom imaging systems and associated methods
EP2942936B1 (fr) * 2014-05-06 2017-12-20 Nokia Technologies OY Entrée de zoom et sélection d'appareil photo
JP2015216491A (ja) * 2014-05-09 2015-12-03 ソニー株式会社 撮像装置
US20150334309A1 (en) * 2014-05-16 2015-11-19 Htc Corporation Handheld electronic apparatus, image capturing apparatus and image capturing method thereof
CN103986876B (zh) * 2014-05-29 2017-06-06 宇龙计算机通信科技(深圳)有限公司 一种图像获取终端和图像获取方法
US9392188B2 (en) 2014-08-10 2016-07-12 Corephotonics Ltd. Zoom dual-aperture camera with folded lens
US9992483B2 (en) * 2014-09-03 2018-06-05 Intel Corporation Imaging architecture for depth camera mode with mode switching
EP3201877B1 (fr) 2014-09-29 2018-12-19 Fotonation Cayman Limited Systèmes et procédés d'étalonnage dynamique de caméras en réseau
CN104601883B (zh) * 2014-12-30 2018-01-23 广东欧珀移动通信有限公司 一种图像拍摄的方法及装置
WO2016108093A1 (fr) 2015-01-03 2016-07-07 Corephotonics Ltd. Module de lentille de téléobjectif miniature et appareil photographique utilisant un tel module de lentille
CN106034204B (zh) * 2015-03-12 2019-05-31 宏碁股份有限公司 图像获取装置及其自动对焦方法
CN106034207A (zh) * 2015-03-19 2016-10-19 联想(北京)有限公司 图像获得方法以及电子设备
US10142533B2 (en) * 2015-03-27 2018-11-27 Intel Corporation Technologies for controlling user access to image sensors of a camera device
EP3492958B1 (fr) 2015-04-02 2022-03-30 Corephotonics Ltd. Structure de moteur de bobine acoustique double dans une caméra à module optique double
US9927600B2 (en) 2015-04-16 2018-03-27 Corephotonics Ltd Method and system for providing auto focus and optical image stabilization in a compact folded camera
US10036895B2 (en) 2015-05-28 2018-07-31 Corephotonics Ltd. Bi-directional stiffness for optical image stabilization in a dual-aperture digital camera
KR102352682B1 (ko) * 2015-06-23 2022-01-18 삼성전자주식회사 디지털 촬영 장치 및 그 동작 방법
US10291842B2 (en) * 2015-06-23 2019-05-14 Samsung Electronics Co., Ltd. Digital photographing apparatus and method of operating the same
US10148864B2 (en) * 2015-07-02 2018-12-04 Pixart Imaging Inc. Imaging device having phase detection pixels and regular pixels, and operating method thereof
US9769367B2 (en) 2015-08-07 2017-09-19 Google Inc. Speech and computer vision-based control
EP3787281B1 (fr) 2015-08-13 2024-08-21 Corephotonics Ltd. Caméra à zoom à double ouverture avec support de vidéo et commande dynamique de commutation/non-commutation
CN106470309A (zh) * 2015-08-19 2017-03-01 中兴通讯股份有限公司 获取双摄像的焦点距离的方法和装置
KR101993077B1 (ko) 2015-09-06 2019-06-25 코어포토닉스 리미티드 소형의 접이식 카메라의 롤 보정에 의한 자동 초점 및 광학식 손떨림 방지
CN106556961A (zh) * 2015-09-30 2017-04-05 佳能企业股份有限公司 摄像装置及其操作方法
KR102446442B1 (ko) * 2015-11-24 2022-09-23 삼성전자주식회사 디지털 촬영 장치 및 그 동작 방법
CN109889708B (zh) 2015-12-29 2021-07-06 核心光电有限公司 具有自动可调节长焦视场的双孔径变焦数字摄影机
US9836819B1 (en) 2015-12-30 2017-12-05 Google Llc Systems and methods for selective retention and editing of images captured by mobile image capture device
US10225511B1 (en) 2015-12-30 2019-03-05 Google Llc Low power framework for controlling image sensor mode in a mobile image capture device
US9836484B1 (en) 2015-12-30 2017-12-05 Google Llc Systems and methods that leverage deep learning to selectively store images at a mobile image capture device
US9838641B1 (en) 2015-12-30 2017-12-05 Google Llc Low power framework for processing, compressing, and transmitting images at a mobile image capture device
US10732809B2 (en) 2015-12-30 2020-08-04 Google Llc Systems and methods for selective retention and editing of images captured by mobile image capture device
US10194089B2 (en) 2016-02-08 2019-01-29 Qualcomm Incorporated Systems and methods for implementing seamless zoom function using multiple cameras
CN105959553B (zh) * 2016-05-30 2018-12-04 维沃移动通信有限公司 一种摄像头的切换方法及终端
US10488631B2 (en) 2016-05-30 2019-11-26 Corephotonics Ltd. Rotational ball-guided voice coil motor
WO2017221106A1 (fr) 2016-06-19 2017-12-28 Corephotonics Ltd. Synchronisation de trame dans un système de caméra à double ouverture
CN106210512B (zh) * 2016-06-30 2019-06-07 维沃移动通信有限公司 一种摄像头切换方法及移动终端
KR102524498B1 (ko) * 2016-07-06 2023-04-24 삼성전자주식회사 듀얼 카메라를 포함하는 전자 장치 및 듀얼 카메라의 제어 방법
US10706518B2 (en) 2016-07-07 2020-07-07 Corephotonics Ltd. Dual camera system with improved video smooth transition by image blending
KR102390572B1 (ko) 2016-07-07 2022-04-25 코어포토닉스 리미티드 폴디드 옵틱용 선형 볼 가이드 음성 코일 모터
US10290111B2 (en) 2016-07-26 2019-05-14 Qualcomm Incorporated Systems and methods for compositing images
US10297034B2 (en) 2016-09-30 2019-05-21 Qualcomm Incorporated Systems and methods for fusing images
CN106454132A (zh) * 2016-11-29 2017-02-22 广东欧珀移动通信有限公司 控制方法、控制装置及电子装置
TWI626620B (zh) * 2016-12-20 2018-06-11 廣東歐珀移動通訊有限公司 圖像處理方法與裝置、電子裝置及計算機可讀儲存介質
KR102269547B1 (ko) 2016-12-28 2021-06-25 코어포토닉스 리미티드 확장된 광-폴딩-요소 스캐닝 범위를 갖는 폴디드 카메라 구조
CN113791484A (zh) 2017-01-12 2021-12-14 核心光电有限公司 紧凑型折叠式摄影机及其组装方法
IL302577A (en) 2017-02-23 2023-07-01 Corephotonics Ltd Lens designs for a folded camera
CN110582724B (zh) 2017-03-15 2022-01-04 核心光电有限公司 具有全景扫描范围的照相装置
US11218626B2 (en) * 2017-07-28 2022-01-04 Black Sesame International Holding Limited Fast focus using dual cameras
US10482618B2 (en) 2017-08-21 2019-11-19 Fotonation Limited Systems and methods for hybrid depth regularization
US10904512B2 (en) 2017-09-06 2021-01-26 Corephotonics Ltd. Combined stereoscopic and phase detection depth mapping in a dual aperture camera
US10951834B2 (en) 2017-10-03 2021-03-16 Corephotonics Ltd. Synthetically enlarged camera aperture
KR102424791B1 (ko) 2017-11-23 2022-07-22 코어포토닉스 리미티드 컴팩트 폴디드 카메라 구조
US10721419B2 (en) * 2017-11-30 2020-07-21 International Business Machines Corporation Ortho-selfie distortion correction using multiple image sensors to synthesize a virtual image
US10757320B2 (en) 2017-12-28 2020-08-25 Waymo Llc Multiple operating modes to expand dynamic range
CN110268710B (zh) * 2018-01-07 2022-01-04 深圳市大疆创新科技有限公司 图像数据处理方法、设备、平台及存储介质
CN114609746A (zh) 2018-02-05 2022-06-10 核心光电有限公司 折叠摄像装置
EP4191315B1 (fr) 2018-02-12 2024-09-25 Corephotonics Ltd. Caméra pliée avec stabilisation d'image optique
JP7104301B2 (ja) * 2018-03-19 2022-07-21 ミツミ電機株式会社 測距カメラ
US11010646B2 (en) * 2018-04-10 2021-05-18 Facebook Technologies, Llc Object tracking assisted with hand or eye tracking
US10694168B2 (en) 2018-04-22 2020-06-23 Corephotonics Ltd. System and method for mitigating or preventing eye damage from structured light IR/NIR projector systems
CN111936908B (zh) 2018-04-23 2021-12-21 核心光电有限公司 具有扩展的两个自由度旋转范围的光路折叠元件
JP7034828B2 (ja) * 2018-05-21 2022-03-14 シャープ株式会社 電子機器、電子機器の制御装置、制御プログラムおよび制御方法
US11425307B2 (en) * 2018-07-19 2022-08-23 Hangzhou Hikvision Digital Technology Co., Ltd. Image capture device in which the focal length of the image capture device can be expanded without increasing the size of lenses
WO2020031005A1 (fr) 2018-08-04 2020-02-13 Corephotonics Ltd. Caméra d'en haut de système d'information d'affichage continu commutable
WO2020039302A1 (fr) 2018-08-22 2020-02-27 Corephotonics Ltd. Caméra pliée à zoom à deux états
CN108848316A (zh) * 2018-09-14 2018-11-20 高新兴科技集团股份有限公司 摄像机的自动变焦控制方法、自动变焦装置和摄像机
US11287081B2 (en) 2019-01-07 2022-03-29 Corephotonics Ltd. Rotation mechanism with sliding joint
CN109696788B (zh) * 2019-01-08 2021-12-14 武汉精立电子技术有限公司 一种基于显示面板的快速自动对焦方法
US10892287B2 (en) * 2019-01-18 2021-01-12 Cista System Corp. Image sensor with image receiver and automatic image switching
EP4224841A1 (fr) 2019-03-09 2023-08-09 Corephotonics Ltd. Système et procédé d'étalonnage stéréoscopique dynamique
KR20200110906A (ko) * 2019-03-18 2020-09-28 삼성전자주식회사 자동 초점 조절 기능을 제공하기 위한 전자 장치 및 그 동작 방법
CN110095078B (zh) * 2019-05-07 2021-02-23 歌尔光学科技有限公司 基于tof系统的成像方法、设备及计算机可读存储介质
CN110248101B (zh) * 2019-07-19 2021-07-09 Oppo广东移动通信有限公司 对焦方法和装置、电子设备、计算机可读存储介质
KR102365748B1 (ko) 2019-07-31 2022-02-23 코어포토닉스 리미티드 카메라 패닝 또는 모션에서 배경 블러링을 생성하는 시스템 및 방법
MX2022003020A (es) 2019-09-17 2022-06-14 Boston Polarimetrics Inc Sistemas y metodos para modelado de superficie usando se?ales de polarizacion.
KR20230004423A (ko) 2019-10-07 2023-01-06 보스턴 폴라리메트릭스, 인크. 편광을 사용한 표면 법선 감지 시스템 및 방법
US11659135B2 (en) 2019-10-30 2023-05-23 Corephotonics Ltd. Slow or fast motion video using depth information
WO2021108002A1 (fr) 2019-11-30 2021-06-03 Boston Polarimetrics, Inc. Systèmes et procédés de segmentation d'objets transparents au moyen de files d'attentes de polarisation
US11949976B2 (en) 2019-12-09 2024-04-02 Corephotonics Ltd. Systems and methods for obtaining a smart panoramic image
CN114641983A (zh) 2019-12-09 2022-06-17 核心光电有限公司 用于获得智能全景图像的系统及方法
US10944915B1 (en) * 2020-01-05 2021-03-09 Ningbo Sunny Opotech Co., Ltd. Multi-aperture imaging system and application thereof
US11195303B2 (en) 2020-01-29 2021-12-07 Boston Polarimetrics, Inc. Systems and methods for characterizing object pose detection and measurement systems
KR20220133973A (ko) 2020-01-30 2022-10-05 인트린식 이노베이션 엘엘씨 편광된 이미지들을 포함하는 상이한 이미징 양식들에 대해 통계적 모델들을 훈련하기 위해 데이터를 합성하기 위한 시스템들 및 방법들
CN114641805A (zh) 2020-02-22 2022-06-17 核心光电有限公司 用于微距摄影的分屏特征
WO2021220080A1 (fr) 2020-04-26 2021-11-04 Corephotonics Ltd. Régulation de température pour correction de capteur à barre à effet hall
US11509837B2 (en) 2020-05-12 2022-11-22 Qualcomm Incorporated Camera transition blending
KR102495627B1 (ko) 2020-05-17 2023-02-06 코어포토닉스 리미티드 전체 시야 레퍼런스 이미지 존재 하의 이미지 스티칭
WO2021243088A1 (fr) 2020-05-27 2021-12-02 Boston Polarimetrics, Inc. Systèmes optiques de polarisation à ouvertures multiples utilisant des diviseurs de faisceau
EP3966631B1 (fr) 2020-05-30 2023-01-25 Corephotonics Ltd. Systèmes et procédés pour obtenir une super macro-image
US11265462B2 (en) 2020-07-06 2022-03-01 Qualcomm Incorporated Automatic camera selection capturing intended subjects in an image
KR20230004887A (ko) 2020-07-15 2023-01-06 코어포토닉스 리미티드 스캐닝 폴디드 카메라에서의 시점 수차-보정
US11637977B2 (en) 2020-07-15 2023-04-25 Corephotonics Ltd. Image sensors and sensing methods to obtain time-of-flight and phase detection information
EP4065934A4 (fr) 2020-07-31 2023-07-26 Corephotonics Ltd. Géométrie d'aimant de capteur à effet hall de détection de position linéaire de grande course
KR102480820B1 (ko) 2020-08-12 2022-12-22 코어포토닉스 리미티드 스캐닝 폴디드 카메라의 광학 이미지 안정화
KR102629883B1 (ko) 2020-12-26 2024-01-25 코어포토닉스 리미티드 스캐닝 줌 카메라를 갖는 멀티-애퍼처 모바일 카메라에서의 비디오 지원
US12020455B2 (en) 2021-03-10 2024-06-25 Intrinsic Innovation Llc Systems and methods for high dynamic range image reconstruction
US12069227B2 (en) 2021-03-10 2024-08-20 Intrinsic Innovation Llc Multi-modal and multi-spectral stereo camera arrays
CN115868168A (zh) 2021-03-11 2023-03-28 核心光电有限公司 用于弹出式相机的系统
US11954886B2 (en) 2021-04-15 2024-04-09 Intrinsic Innovation Llc Systems and methods for six-degree of freedom pose estimation of deformable objects
US11290658B1 (en) 2021-04-15 2022-03-29 Boston Polarimetrics, Inc. Systems and methods for camera exposure control
US12067746B2 (en) 2021-05-07 2024-08-20 Intrinsic Innovation Llc Systems and methods for using computer vision to pick up small objects
WO2022259154A2 (fr) 2021-06-08 2022-12-15 Corephotonics Ltd. Systèmes et caméras pour incliner un plan focal d'une super macro-image
US11689813B2 (en) 2021-07-01 2023-06-27 Intrinsic Innovation Llc Systems and methods for high dynamic range imaging using crossed polarizers
US20230075701A1 (en) * 2021-09-03 2023-03-09 Motional Ad Llc Location based parameters for an image sensor

Family Cites Families (16)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS6020111A (ja) * 1983-07-14 1985-02-01 Fuji Electric Corp Res & Dev Ltd 距離測定装置
KR940005344Y1 (ko) 1991-07-09 1994-08-10 삼성전자 주식회사 주변배경 서치기능을 갖는 캠코더
US5668597A (en) * 1994-12-30 1997-09-16 Eastman Kodak Company Electronic camera with rapid automatic focus of an image upon a progressive scan image sensor
US5874994A (en) * 1995-06-30 1999-02-23 Eastman Kodak Company Filter employing arithmetic operations for an electronic sychronized digital camera
US6441855B1 (en) * 2000-05-09 2002-08-27 Eastman Kodak Company Focusing device
JP2003037757A (ja) * 2001-07-25 2003-02-07 Fuji Photo Film Co Ltd 画像撮像装置
JP3914060B2 (ja) * 2002-02-08 2007-05-16 富士フイルム株式会社 合焦装置
JP4198449B2 (ja) * 2002-02-22 2008-12-17 富士フイルム株式会社 デジタルカメラ
JP2006033155A (ja) * 2004-07-13 2006-02-02 Make Softwear:Kk 写真プリント提供装置、写真プリント提供方法および写真プリント提供プログラム
US7417674B2 (en) * 2004-08-25 2008-08-26 Micron Technology, Inc. Multi-magnification color image sensor
JP2006080889A (ja) * 2004-09-09 2006-03-23 Funai Electric Co Ltd 電子写真装置
US7206136B2 (en) * 2005-02-18 2007-04-17 Eastman Kodak Company Digital camera using multiple lenses and image sensors to provide an extended zoom range
US7256944B2 (en) * 2005-02-18 2007-08-14 Eastman Kodak Company Compact image capture assembly using multiple lenses and image sensors to provide an extended zoom range
US7512262B2 (en) * 2005-02-25 2009-03-31 Microsoft Corporation Stereo-based image processing
US7756330B2 (en) * 2006-07-27 2010-07-13 Eastman Kodak Company Producing an extended dynamic range digital image
US20080030592A1 (en) * 2006-08-01 2008-02-07 Eastman Kodak Company Producing digital image with different resolution portions

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See references of WO2008112055A1 *

Also Published As

Publication number Publication date
JP2010523015A (ja) 2010-07-08
ATE545272T1 (de) 2012-02-15
US20080218613A1 (en) 2008-09-11
CN101632295A (zh) 2010-01-20
US7729602B2 (en) 2010-06-01
EP2135442B1 (fr) 2012-02-08
JP5150651B2 (ja) 2013-02-20
WO2008112055A1 (fr) 2008-09-18

Similar Documents

Publication Publication Date Title
US7729602B2 (en) Camera using multiple lenses and image sensors operable in a default imaging mode
US7859588B2 (en) Method and apparatus for operating a dual lens camera to augment an image
US7683962B2 (en) Camera using multiple lenses and image sensors in a rangefinder configuration to provide a range map
US7676146B2 (en) Camera using multiple lenses and image sensors to provide improved focusing capability
US7583308B2 (en) Image capturing apparatus
US9210408B2 (en) Stereoscopic panoramic image synthesis device, image capturing device, stereoscopic panoramic image synthesis method, recording medium, and computer program
US8648961B2 (en) Image capturing apparatus and image capturing method
US20050012833A1 (en) Image capturing apparatus
US20140063287A1 (en) Imaging apparatus
US10187564B2 (en) Focus adjustment apparatus, imaging apparatus, focus adjustment method, and recording medium storing a focus adjustment program thereon
CN103888645A (zh) 摄像设备及用于控制摄像设备的方法
JP2003241072A (ja) カメラ
US7292279B2 (en) Solid state image sensing device and photographing apparatus
JPH11211974A (ja) 撮像装置
JP3991448B2 (ja) カメラ
JP2011247979A (ja) 撮影装置及び撮影距離取得方法
JP2003207713A (ja) カメラ

Legal Events

Date Code Title Description
PUAI Public reference made under article 153(3) epc to a published international application that has entered the european phase

Free format text: ORIGINAL CODE: 0009012

17P Request for examination filed

Effective date: 20090806

AK Designated contracting states

Kind code of ref document: A1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

17Q First examination report despatched

Effective date: 20100315

DAX Request for extension of the european patent (deleted)
GRAP Despatch of communication of intention to grant a patent

Free format text: ORIGINAL CODE: EPIDOSNIGR1

GRAS Grant fee paid

Free format text: ORIGINAL CODE: EPIDOSNIGR3

GRAA (expected) grant

Free format text: ORIGINAL CODE: 0009210

AK Designated contracting states

Kind code of ref document: B1

Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HR HU IE IS IT LI LT LU LV MC MT NL NO PL PT RO SE SI SK TR

REG Reference to a national code

Ref country code: GB

Ref legal event code: FG4D

REG Reference to a national code

Ref country code: AT

Ref legal event code: REF

Ref document number: 545272

Country of ref document: AT

Kind code of ref document: T

Effective date: 20120215

Ref country code: CH

Ref legal event code: EP

REG Reference to a national code

Ref country code: NL

Ref legal event code: T3

REG Reference to a national code

Ref country code: DE

Ref legal event code: R096

Ref document number: 602008013255

Country of ref document: DE

Effective date: 20120405

LTIE Lt: invalidation of european patent or patent extension

Effective date: 20120208

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: IS

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120608

Ref country code: HR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: NO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: PT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120608

Ref country code: GR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120509

Ref country code: LV

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: BE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: PL

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: FI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

REG Reference to a national code

Ref country code: AT

Ref legal event code: MK05

Ref document number: 545272

Country of ref document: AT

Kind code of ref document: T

Effective date: 20120208

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MC

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20120229

Ref country code: CY

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

REG Reference to a national code

Ref country code: CH

Ref legal event code: PL

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: CZ

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: DK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: EE

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: CH

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20120229

Ref country code: SI

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: RO

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: LI

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20120229

REG Reference to a national code

Ref country code: IE

Ref legal event code: MM4A

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: SK

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: IT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

PLBE No opposition filed within time limit

Free format text: ORIGINAL CODE: 0009261

STAA Information on the status of an ep patent application or granted ep patent

Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT

REG Reference to a national code

Ref country code: FR

Ref legal event code: ST

Effective date: 20121207

26N No opposition filed

Effective date: 20121109

GBPC Gb: european patent ceased through non-payment of renewal fee

Effective date: 20120508

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: FR

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20120410

Ref country code: IE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20120208

Ref country code: AT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

REG Reference to a national code

Ref country code: DE

Ref legal event code: R097

Ref document number: 602008013255

Country of ref document: DE

Effective date: 20121109

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: GB

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20120508

Ref country code: ES

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120519

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: DE

Payment date: 20130228

Year of fee payment: 6

REG Reference to a national code

Ref country code: DE

Ref legal event code: R082

Ref document number: 602008013255

Country of ref document: DE

Representative=s name: MUELLER-BORE & PARTNER PATENTANWAELTE, EUROPEA, DE

REG Reference to a national code

Ref country code: DE

Ref legal event code: R081

Ref document number: 602008013255

Country of ref document: DE

Owner name: INTELLECTUAL VENTURES FUND 83 LLC (N.D.GES.D.S, US

Free format text: FORMER OWNER: EASTMAN KODAK COMPANY, ROCHESTER, US

Effective date: 20130430

Ref country code: DE

Ref legal event code: R082

Ref document number: 602008013255

Country of ref document: DE

Representative=s name: MUELLER-BORE & PARTNER PATENTANWAELTE, EUROPEA, DE

Effective date: 20130430

Ref country code: DE

Ref legal event code: R081

Ref document number: 602008013255

Country of ref document: DE

Owner name: INTELLECTUAL VENTURES FUND 83 LLC (N.D.GES.D.S, US

Free format text: FORMER OWNER: EASTMAN KODAK COMPANY, ROCHESTER, N.Y., US

Effective date: 20130430

Ref country code: DE

Ref legal event code: R082

Ref document number: 602008013255

Country of ref document: DE

Representative=s name: MUELLER-BORE & PARTNER PATENTANWAELTE PARTG MB, DE

Effective date: 20130430

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: MT

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

Ref country code: BG

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120508

REG Reference to a national code

Ref country code: NL

Ref legal event code: SD

Effective date: 20130913

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: TR

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20120208

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: LU

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20120208

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: HU

Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT

Effective date: 20080208

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602008013255

Country of ref document: DE

REG Reference to a national code

Ref country code: DE

Ref legal event code: R119

Ref document number: 602008013255

Country of ref document: DE

Effective date: 20140902

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: DE

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20140902

PGFP Annual fee paid to national office [announced via postgrant information from national office to epo]

Ref country code: NL

Payment date: 20150210

Year of fee payment: 8

REG Reference to a national code

Ref country code: NL

Ref legal event code: MM

Effective date: 20160301

PG25 Lapsed in a contracting state [announced via postgrant information from national office to epo]

Ref country code: NL

Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES

Effective date: 20160301