US9270902B2 - Image processing apparatus, image capturing apparatus, image processing method, and storage medium for obtaining information on focus control of a subject - Google Patents
Image processing apparatus, image capturing apparatus, image processing method, and storage medium for obtaining information on focus control of a subject Download PDFInfo
- Publication number
- US9270902B2 US9270902B2 US14/195,951 US201414195951A US9270902B2 US 9270902 B2 US9270902 B2 US 9270902B2 US 201414195951 A US201414195951 A US 201414195951A US 9270902 B2 US9270902 B2 US 9270902B2
- Authority
- US
- United States
- Prior art keywords
- image
- subject
- shifted
- focus
- subject areas
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Expired - Fee Related
Links
- 238000012545 processing Methods 0.000 title claims abstract description 167
- 238000003672 processing method Methods 0.000 title claims description 4
- 230000008859 change Effects 0.000 claims description 25
- 238000000605 extraction Methods 0.000 claims description 9
- 230000004044 response Effects 0.000 claims description 6
- 230000003287 optical effect Effects 0.000 description 85
- 238000000034 method Methods 0.000 description 72
- 238000010586 diagram Methods 0.000 description 50
- 210000001747 pupil Anatomy 0.000 description 23
- 230000006870 function Effects 0.000 description 14
- 238000009795 derivation Methods 0.000 description 10
- 238000005070 sampling Methods 0.000 description 5
- 230000009467 reduction Effects 0.000 description 4
- 239000002585 base Substances 0.000 description 3
- 230000008901 benefit Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 2
- 238000004891 communication Methods 0.000 description 2
- 238000001514 detection method Methods 0.000 description 2
- 230000004907 flux Effects 0.000 description 2
- 230000010365 information processing Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000002360 preparation method Methods 0.000 description 2
- WHXSMMKQMYFTQS-UHFFFAOYSA-N Lithium Chemical compound [Li] WHXSMMKQMYFTQS-UHFFFAOYSA-N 0.000 description 1
- 229910005580 NiCd Inorganic materials 0.000 description 1
- 229910005813 NiMH Inorganic materials 0.000 description 1
- 239000003513 alkali Substances 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 230000004888 barrier function Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 230000005674 electromagnetic induction Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 238000001914 filtration Methods 0.000 description 1
- 229910052744 lithium Inorganic materials 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 230000007935 neutral effect Effects 0.000 description 1
- 238000010422 painting Methods 0.000 description 1
- 230000008707 rearrangement Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 230000035945 sensitivity Effects 0.000 description 1
- 238000000926 separation method Methods 0.000 description 1
- 238000010897 surface acoustic wave method Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 230000009466 transformation Effects 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
- 238000002834 transmittance Methods 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/80—Camera processing pipelines; Components thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/272—Means for inserting a foreground image in a background image, i.e. inlay, outlay
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/62—Control of parameters via user interfaces
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/63—Control of cameras or camera modules by using electronic viewfinders
- H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
- H04N23/632—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/63—Control of cameras or camera modules by using electronic viewfinders
- H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
- H04N23/635—Region indicators; Field of view indicators
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/951—Computational photography systems, e.g. light-field imaging systems by using two or more images to influence resolution, frame rate or aspect ratio
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/958—Computational photography systems, e.g. light-field imaging systems for extended depth of field imaging
- H04N23/959—Computational photography systems, e.g. light-field imaging systems for extended depth of field imaging by adjusting depth of field during image capture, e.g. maximising or setting range based on scene characteristics
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/142—Edging; Contouring
-
- H04N5/23212—
-
- H04N5/23232—
-
- H04N5/23293—
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2624—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects for obtaining an image which is composed of whole input images, e.g. splitscreen
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/2628—Alteration of picture size, shape, position or orientation, e.g. zooming, rotation, rolling, perspective, translation
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/222—Studio circuitry; Studio devices; Studio equipment
- H04N5/262—Studio circuits, e.g. for mixing, switching-over, change of character of image, other special effects ; Cameras specially adapted for the electronic generation of special effects
- H04N5/265—Mixing
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
- G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
- G01S17/02—Systems using the reflection of electromagnetic waves other than radio waves
- G01S17/06—Systems determining position data of a target
- G01S17/08—Systems determining position data of a target for measuring distance only
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0081—Depth or disparity estimation from stereoscopic image signals
Definitions
- the present invention relates to an image processing apparatus, an image capturing apparatus, an image processing method, and a storage medium for obtaining information on focus control of a subject.
- an image capturing apparatus such as a digital camera
- an apparatus that makes focus adjustment of a subject included in a captured image by a user specifying the subject by a touch panel operation etc. for example, Japanese Patent Laid-Open No. 2011-39457.
- the technique is also proposed, which makes it possible for a user to easily grasp an object in focus the image of which is to be captured (subject) by displaying a map image for indicating a depth position representing a position of the object the image of which is to be captured in the depth direction and a focus position representing the position of focus (for example, Japanese Patent Laid-Open No. 2010-177741).
- the light field photography technique which acquires information of the orientation and intensity of a light ray (light field data) and makes adjustment of the focus position (refocus) and adjustment of the depth of field by the later image processing (for example, Japanese Patent Laid-Open No. 2008-515110).
- a plenoptic camera for example, Japanese Patent Laid-Open No. 2009-124213
- a microlens array is arranged behind a main lens and a camera array in which compact cameras are arranged side by side are known.
- refocus processing there is known a method in which acquired data of images from a plurality of viewpoints is subjected to projective transformation onto a virtual sensor, and added and averaged (for example, WO2008050904).
- a refocus image As a method for displaying an image having been subjected to refocus processing (hereinafter, a refocus image) while changing the focus position, mention is made of, for example, the method disclosed in United States Patent No. 2008/0131019.
- the user interface (UI) to adjust the focus position is prepared on the screen on which a refocus image is displayed and the focus position is changed via the UI.
- the method is disclosed, which displays a refocus image whose focus position is adjusted to a subject which a user desires to put to a focus position and has given instructions through a screen on which the refocus image is displayed.
- the image processing apparatus has an acquisition unit configured to acquire an image including a plurality of subject areas and distance information corresponding to the plurality of subject areas, and a generation unit configured to generate a shifted image in which positions of the plurality of subject areas are shifted in the image.
- the image processing apparatus has an acquisition unit configured to acquire an image including a plurality of subject areas and distance information corresponding to the plurality of subject areas, a first generation unit configured to generate a rearranged image in which the plurality of subject areas is rearranged in the image based on the distance information, and a second generation unit configured to generate an image in which information on a focus state is reflected in the rearranged image at the time of performing image combination processing to change the focus state of the image including the plurality of subject areas.
- FIG. 1 is a diagram showing an example of an external appearance of a digital camera
- FIG. 2 is a block diagram showing an internal configuration of the digital camera
- FIG. 3 is a diagram showing a relationship between FIGS. 3A to 3C , and FIGS. 3A to 3C are flowcharts showing a flow of image processing performed in the digital camera according to a first embodiment
- FIG. 4 is a diagram showing an example of a two-dimensional image displayed on a display unit of the digital camera
- FIG. 5 is a diagram showing a state where subjects are detected from the two-dimensional image
- FIG. 6 is a diagram showing an example of a distance image
- FIG. 7 is a diagram showing a state where subject areas are extracted
- FIG. 8 is a flowchart showing a flow of processing to determine a rate of amount of shift between the horizontal direction and the vertical direction;
- FIG. 9 is a diagram showing the results of performing shifted image generation processing on the image in which the subject areas are extracted.
- FIG. 10 is a diagram showing the results of performing shear processing on each subject area in the shifted image
- FIG. 11 is a diagram showing the results of superimposing a ground lattice pattern on the image having been subjected to the shear processing
- FIG. 12 is a diagram showing the results of superimposing an index of focus distance and an index of depth of field on the image with which the ground lattice pattern is combined;
- FIG. 13 is a diagram showing an example of an internal configuration of an apparatus that handles light field data
- FIG. 14 is a flowchart showing a flow of image processing performed in an apparatus according to a second embodiment
- FIG. 15 is a block diagram showing an internal configuration of a camera array image capturing apparatus according to a third embodiment
- FIG. 16 is an example of a configuration in which a lens array is arranged on the image side of an image forming optical system
- FIG. 17 is an example of a configuration in which the lens array is arranged on the image side of the image forming optical system
- FIG. 18 is an example of a configuration in which the lens array is arranged on the image side of the image forming optical system
- FIG. 19 is an example of a configuration in which a plurality of image forming optical systems is arrayed (camera array);
- FIG. 20 is a diagram showing a relationship between a lens array and an image sensing element
- FIG. 21 schematically represents a relationship of an expression expressing coefficients ⁇ ⁇ of refocus in the third embodiment
- FIG. 22 is a block diagram showing an internal configuration of an image processing unit according to the third embodiment.
- FIG. 23 is a flowchart showing a flow of processing in the image processing unit according to the third embodiment.
- FIG. 24A is a diagram showing an example of a single-viewpoint image and FIG. 24B is a diagram showing a distance map as distance information derived from the single-viewpoint image;
- FIGS. 25A and 25B are each a diagram showing an example of the results of extracting subject areas from the distance map
- FIGS. 26A to 26D are each a diagram showing an example of a rearranged image in the case where the scene of the single-viewpoint image is looked down upon from directly above;
- FIG. 27 is a diagram showing an optical arrangement in the configuration of an image capturing unit shown in FIG. 16 ;
- FIGS. 28A and 28B are each a diagram showing an example of a focus information display image
- FIGS. 29A to 29C are each a diagram showing a variation of the focus information display image
- FIGS. 30A to 30C are each a diagram showing an example in the case where the focus information display image is displayed together with a single-viewpoint image and/or a combined image;
- FIGS. 31A and 31B are each a diagram obtained by mainly enlarging the portion of the lens array and the image sensing element in the configuration of an image capturing unit shown in FIG. 17 ;
- FIG. 32 schematically represents a relationship of an expression expressing the coefficients ⁇ ⁇ of refocus in a fourth embodiment
- FIG. 33 is a diagram in the case where an image capturing unit having a configuration of a camera array is viewed from the front (object side);
- FIG. 34 is a diagram (section) in the case where an image forming optical system and an image sensing element according to a fifth embodiment are viewed from the side;
- FIG. 35 schematically represents a relationship of an expression expressing the coefficients ⁇ ⁇ of refocus in the fifth embodiment.
- FIG. 1 is a diagram showing an example of an external appearance of a digital camera.
- a display unit 101 for example, a liquid crystal display is used and images and various kinds of information are displayed.
- the display unit 101 has a touch screen function and is capable of detecting a touch on the display unit 101 .
- a touch panel is configured so that the transmittance of light does not affect the display of the display unit 101 and the touch panel is attached to an upper layer of the display surface of the display unit 101 . Then, the input coordinates on the touch panel and the display coordinates on the display unit 101 are associated with each other. Due to this, a GUI with which it seems possible for a user to directly operate a screen displayed on the display unit 101 can be configured.
- a shutter button 102 is an operation unit for giving instructions to perform image capturing.
- a mode dial 103 is an operation unit for switching various kinds of modes.
- a connector 104 is a connector between a connection cable 111 and a digital camera 100 .
- An operation unit 105 includes various kinds of switches, buttons, etc., to receive various kinds of operations from a user.
- a controller wheel 106 is an operating member included in the operation unit 105 and capable of being operated to rotate.
- a power source switch 107 switches between on and off of a power source of the digital camera 100 .
- a storage medium 108 is an information storage medium, such as a memory card and a hard disc, for storing captured image data and light field data.
- a storage medium slot 109 is a slot for housing the storage medium 108 . It is made possible for the storage medium 108 housed in the storage medium slot 109 to communicate with the digital camera 100 .
- a lid 110 is a lid of the storage medium slot 109 .
- FIG. 2 is a block diagram showing an internal configuration of the digital camera 100 .
- An image capturing lens 201 includes a focus lens.
- a shutter includes an aperture function and controls exposure in an interlocking manner with the shutter button 102 .
- An image capturing unit 203 includes a CCD that converts an optical image into an electric signal, a CMOS element, etc.
- An A/D converter 204 converts an analog signal output from the image capturing unit 203 into a digital signal.
- a barrier 205 prevents stain and breakage of the image capturing system by covering the image capturing unit 203 including the image capturing lens 201 and the shutter 202 .
- An image processing unit 206 performs resize processing, such as predetermined pixel interpolation and reduction, and color conversion processing on image data from the A/D conversion unit 204 or image data from a memory control unit 207 .
- predetermined arithmetic operation processing is performed using captured image data and a system control unit 208 performs exposure control and distance measuring control based on the obtained arithmetic operation results. Due to this, TTL (Through The Lens) AF (Auto Focus) processing, AE (Auto Exposure) processing, and EF (pre-flash) processing are performed.
- predetermined arithmetic operation processing is further performed using captured image data and TTL AWB (Auto White Balance) processing is also performed based on the obtained arithmetic operation results.
- the image data from the A/D converter 204 is directly written to a memory 209 via the image processing unit 206 and the memory control unit 207 , or via the memory control unit 207 .
- the memory 209 stores image data obtained by the image capturing unit 203 and converted into digital data by the A/D converter 204 , and image data to be displayed on the display unit 101 .
- the memory 209 includes a storage capacity sufficient to store a predetermined number of still images, moving images for a predetermined period of time, and voice. By sequentially displaying captured image data using the display unit 101 , it is possible to implement an electronic finder function (through-image display). Further, the memory 209 serves also as a memory for displaying images (video memory).
- a D/A converter 210 converts the data for displaying images stored in the memory 209 into analog signals and supplies the signals to the display unit 101 . In this manner, the image data to be displayed written to the memory 209 is displayed on the display unit 101 via the D/A converter 210 .
- the display unit 101 produces a display on a display, such as an LCD, in accordance with the analog signal from the D/A converter 210 .
- a nonvolatile memory 211 is an electrically erasable/programmable memory and, for example, an EEPROM etc. is used.
- the nonvolatile memory 211 stores constants for the operation of the system control unit 208 , programs, etc. Programs referred to here are programs used to perform various kinds of flowcharts, to be described later.
- the system control unit 208 controls the whole of the digital camera 100 .
- the system control unit 208 implements each piece of processing, to be described later, by executing the programs stored in the nonvolatile memory 211 described previously.
- a system memory 212 a RAM is used and on which constants, variables for the operation of the system control unit 208 , programs read from the nonvolatile memory 211 , etc., are developed.
- the system control unit 208 also performs display control by controlling the memory 209 , the D/A converter 210 , the display unit 101 , etc. Furthermore, it is possible for the system control unit 208 to detect operations below to a touch panel.
- touch down Operation to touch the touch panel with a finger or a pen (hereinafter, referred to as touch down)
- touch on a finger or a pen is in touch with the touch panel
- pinch in Operation to touch the touch panel with two fingers and to move the fingers so as to narrow the interval therebetween
- pinch out Operation to touch the touch panel with two fingers and to move the fingers so as to increase the interval therebetween
- touch up Operation to let go a finger or a pen in touch with the touch panel (hereinafter, referred to as touch up)
- touch off State where nothing is in touch with the touch panel (hereinafter, referred to as touch off)
- Flick is the operation to quickly move a finger in touch with the touch panel a certain distance and then let the finger go.
- flick is the operation to quickly slide a finger on the touch panel so as to flick it.
- move of a predetermined distance or more at a predetermined speed or higher is detected and then touch up is detected, it is possible to determine that flick is performed. It is assumed that in the case where move of a predetermined distance or more at less than a predetermined speed is detected, it is determined that drag is performed.
- a touch panel it may also be possible to use any of various kinds of systems, such as the resistance film system, the electrostatic capacitance system, the surface acoustic wave system, the infrared system, the electromagnetic induction system, the image recognition system, and the optical sensor system.
- the mode dial 103 switches the operation mode of the system control unit 208 to any of the image capturing mode capable of recording still images and moving images, the reproduction mode, etc.
- the shutter button 102 includes a first shutter switch and a second shutter switch, not shown schematically.
- the first shutter switch turns on during the operation of the shutter button 102 , i.e. by the so-called half-press operation (instructions to perform the image capturing preparation) and generates a first shutter switch signal SW1.
- the first shutter switch signal SW1 causes the operations, such as AF (Auto Focus) processing, AE (Auto Exposure) processing, AWB (Auto White Balance) processing, and EF (pre-flash) processing (hereinafter, at least one of these operations is referred to as an image capturing preparation operation).
- the second shutter switch turns on at the time of completion of the operation of the shutter button 102 , i.e. by the so-called full-press operation (instructions to perform image capturing) and generates a second shutter switch signal SW2.
- the system control unit 208 starts a series of operations of image capturing processing from reading a signal from the image capturing unit 203 to writing image data to the storage medium 108 .
- a function is allocated appropriately for each scene by performing the operation to select various kinds of function icons displayed on the display unit 101 and each operating member acts as each of the various kinds of function buttons.
- Function buttons include, for example, an end button, a return button, an image forward button, a jump button, a narrow-down button, an attribute change button, etc.
- a menu button is pressed, a menu screen on which various kinds of settings can be performed is displayed on the display unit 101 . It is possible for a user to intuitively perform various kinds of settings using the menu screen displayed on the display unit 101 , a four-direction button, and a SET button.
- the controller wheel 106 is an operating member included in the operation unit 105 and capable of being operated to rotate and is used at the time of specifying a selected item together with the direction button.
- a power source control unit 213 includes a battery detection circuit, a DC-DC converter, a switch circuit that switches blocks to be energized, etc., and detects whether or not a battery is attached, the kind of a battery, and the remaining battery life. Further, the power source control unit 213 controls the DC-DC converter based on the detection results and the instructions of the system control unit 208 and supplies a necessary voltage for a necessary period of time to each unit including the storage medium 108 .
- a power source unit 214 includes a primary battery, such as an alkali battery and a lithium battery, a secondary battery, such as a NiCd battery, a NiMH battery, and a Li battery, an AC adaptor, etc.
- a primary battery such as an alkali battery and a lithium battery
- a secondary battery such as a NiCd battery, a NiMH battery, and a Li battery, an AC adaptor, etc.
- An interface 215 is an interface with the storage medium 108 , such as a memory card and a hard disc.
- FIGS. 3A to 3C are flowcharts showing a flow of image processing performed in the digital camera according to the present embodiment.
- the series of processing is implemented by the system control unit 208 developing programs stored in the nonvolatile memory 211 in the system memory 212 and executing the programs.
- the system control unit 208 starts image capturing under predetermined image capturing conditions (focus, exposure, white balance, etc.) at step 301 .
- the image capturing conditions depend on the specifications of the apparatus, such as that the value before the power source is turned off is recovered and used.
- FIG. 4 is a diagram showing an example of a two-dimensional image displayed on the display unit 101 .
- the two-dimensional image in FIG. 4 is an image captured outdoors and the sky is photographed, and therefore, the distance to the farthest subject is an infinity.
- the forward depth of field thereof becomes equal to 1 ⁇ 2 of the so-called “hyperfocal distance”.
- the value of the hyperfocal distance is obtained by expression (1) below. ( f ⁇ 2)/( ⁇ F ) expression (1)
- f denotes the focal length (mm) of the lens
- F denotes the f-stop of the lens
- ⁇ denotes the diameter of the permissible circle of confusion (mm).
- the hyperfocal distance is about 26.8 m and the forward depth of field is about 13.4 m.
- an area whose distance is greater than about 13.4 m is the “background area”.
- the farthest subject is brought into focus with full-aperture and a certain subject in the forward depth of field is brought into focus, all the subjects in this range are brought into focus as a result. Because of this, by displaying the areas of these subjects as one “background area”, it is possible to clearly demonstrate to a user that the subjects in this area cannot be classified into those which are brought into focus and those which are not brought into focus.
- the system control unit 208 performs the setting.
- the system control unit 208 determines whether there are instructions to change the display mode from a user. In the case where there are instructions to change the display mode, the procedure proceeds to step 305 . On the other hand, in the case where there are not instructions to change the display mode, the procedure returns to step 302 .
- the system control unit 208 detects subjects included in a captured image.
- the method for detecting subjects within a captured image it may be possible to apply various kinds of methods, such as, for example, the method disclosed in Japanese Patent Laid-Open No. 1997-186931.
- FIG. 5 is a diagram showing a state where a person, an automobile in profile, a traffic sign, and a building are detected, respectively, as subjects from the two-dimensional image shown in FIG. 4 .
- FIG. 5 is a diagram showing a state where a person, an automobile in profile, a traffic sign, and a building are detected, respectively, as subjects from the two-dimensional image shown in FIG. 4 .
- each subject of a clump of trees, a mailbox, and an automobile facing the front is regarded to be at a distance of 1 ⁇ 2 of the hyperfocal distance at the time of the aperture being fully open or farther, and therefore, is included in the above-described “background area” together with the “sky”, which is the farthest subject.
- the system control unit 208 acquires distance information (information of the distance to each subject detected from the two-dimensional image) of the captured two-dimensional image.
- the distance information is information of the distance to each subject in the two-dimensional image recorded in association with the coordinates of the two-dimensional image, and for example, corresponds to a “distance image (distance map)” in which the magnitude of the depth of the coordinates (the distance from the camera) are represented by a shade of gray.
- the method for obtaining distance information of a captured image it may be possible to apply the various kinds of methods, such as, for example, the method disclosed in Japanese Patent Laid-Open No. 2001-169308.
- FIG. 6 shows an example of a distance image generated for the two-dimensional image shown in FIG.
- the areas of the four subjects of the person, the automobile in profile, the traffic sign, and the building are expressed by images in a gray scale different in density according to each distance (the person whose distance is the shortest is expressed in the darkest shade of gray and the building in the lightest shade of gray). Those such as a clump of trees farther than the building are included in part of the background as described above, and therefore, they are not treated as “subjects” here.
- the system control unit 208 performs processing to extract subject areas to be subjected to shift processing at the next step 308 based on the subjects detected at step 305 and the distance information acquired at step 306 .
- the subject area extraction processing can also be said as processing to separate subject areas for each fixed distance. For example, among the subjects detected at step 305 but not included in the background area, the portion within a range of a fixed distance (for example, 1 m ⁇ 50 cm) is extracted as one of units of subject areas.
- FIG. 7 is a diagram showing a state where the subject areas are extracted from the two-dimensional image shown in FIG. 4 and the portion (each area of the person, the automobile in profile, the traffic sign, and the building) indicated by the broken line represents each extracted subject area.
- the system control unit 208 generates a “shifted image” in which each subject area is shifted in a fixed direction by performing processing to change (shift) the position (coordinates) of each subject area extracted at step 307 in accordance with the distance information acquired at step 306 .
- a “shifted image” in which each subject area is shifted in a fixed direction by performing processing to change (shift) the position (coordinates) of each subject area extracted at step 307 in accordance with the distance information acquired at step 306 .
- the distances to the four subject areas shown in FIG. 7 correspond to 1 m, 3 m, 5 m, and m, respectively
- an image is generated in which the coordinates of each subject area are shifted in the horizontal direction and in the vertical direction by predetermined amounts, respectively, (for example, 100 pixels, 33 pixels, 20 pixels, and 13 pixels, respectively).
- the amount of shift in the “shifted image generation” does not necessarily needs to be in inverse proportion to the distance and they may be in a logarithmic ratio or a correspondence table of the distance and the amount of shift is prepared in advance and the amount of shift may be determined by applying the table. At this time, in order to make the same the shift direction of each area, the rate between the amount of shift in the horizontal direction and the amount of shift in the vertical direction is made constant. Alternatively, it may also be possible to determine the rate of amount of shift between the horizontal direction and the vertical direction by processing as shown below.
- FIG. 8 is a flowchart showing a flow of processing to determine a rate of amount of shift between the horizontal direction and the vertical direction.
- the system control unit 208 generates an image in which a shift is caused at a predetermined rate.
- a shifted image in which a shift is caused by a predetermined initial value (for example at a rate of 1:1) is generated.
- the system control unit 208 determines whether there is an overlap of subject areas within the shifted image obtained at step 801 . In the case where there is an overlap of subject areas, the procedure proceeds to step 803 . On the other hand, in the case where there is no overlap of subject areas, the procedure proceeds to step 805 .
- the system control unit 208 determines whether the ratio between the area of the hidden portion of a subject area to be displayed behind another subject area overlapping thereon and the area of the subject area to be displayed behind the other subject area exceeds a predetermined threshold value (for example, 50%).
- a predetermined threshold value for example, 50%.
- the predetermined threshold value is determined in advance in view of this point.
- the procedure proceeds to step 804 .
- the procedure proceeds to step 805 .
- the system control unit 208 changes the rate at which the shift is caused in the horizontal and vertical directions and the procedure returns to step 801 . That is, the rate is changed so that the ratio between the area of the hidden portion of the subject area to be displayed behind the other subject area overlapping thereon and the area of the subject area to be displayed behind becomes smaller and then a shifted image in which a shift is caused in accordance with the changed rate is generated.
- the system control unit 208 determines the rate in the case where the ratio of the hidden area of the subject area to be displayed behind becomes lower than the threshold value as the formal rate for display.
- the amount of shift in accordance with the rate determined as described above is determined as the amount of shift in shifted image generation.
- the shifted image generation processing it is desirable to shift subject areas in the direction in which the subject area corresponding to the subject at the short distance (first subject area) appears lower than the subject area corresponding to the subject at the great distance (second subject area).
- first subject area the subject area corresponding to the subject at the short distance
- second subject area the subject area corresponding to the subject at the great distance
- FIG. 9 is a diagram showing the results of performing the shifted image generation processing on the image in which the four subject areas are extracted (see FIG. 7 ) and it is known that the position (coordinates) of each subject area is shifted in accordance with its distance.
- the rate of amount of shift between the horizontal direction and the vertical direction is made constant.
- the system control unit 208 performs shear processing (oblique deformation processing) on each subject area in the “shifted image” generated by the shifted image generation processing at step 308 so that the horizontal direction is inclined.
- shear processing oblique deformation processing
- To which degree the oblique deformation is applied is determined by a shear angle determined in advance.
- FIG. 10 is a diagram showing the results of performing the shear processing on each subject area in the shifted image (see FIG. 9 ).
- the shear angle may be a given angle determined in advance or it may also be possible to enable a user to specify the shear angle arbitrarily.
- the system control unit 208 performs processing to superimpose (combine) a lattice pattern based on the shift direction and the shear angle. For example, a ground lattice pattern including segments in the same direction as the shift direction and segments in the same direction as the shear (oblique deformation) direction is generated and superimposed on the image.
- FIG. 11 is a diagram showing the results of superimposing a lattice pattern on the image in FIG. 10 having been subjected to the shear processing. Due to this, it is made possible for a user to grasp the distance interval between subjects still more easily. In the example in FIG. 11 , two kinds of lattice patterns in total, i.e.
- a ground lattice pattern 1101 and a wall surface lattice pattern 1102 are superimposed and displayed. Due to this, it is made possible to produce a display still more suitable to the feeling that the user views the scene from the oblique front. It may also be possible to display the “background area” as a “wall” and not to display the lattice pattern corresponding to the greater distances. Due to this, it is made possible for a user to intuitively grasp the limit of the change in the depth of field etc. (that is, the fact that the focus distance or the depth of field cannot be changed for the portion displayed as the “wall”).
- the system control unit 208 derives the forward depth of field and the backward depth of field from the image capturing conditions (focus, exposure, white balance, etc.) of a captured image. For example, by arithmetic operations using expression (2) and expression (3) below, the forward depth of field and the backward depth of field are derived, respectively.
- the diameter of the permissible circle of confusion is a characteristic value derived from the diagonal length of the image sensing element and for example, for the image sensing element having a width of 36 mm, a height of 24 mm, and a diagonal length of 43.27 mm, a value of 0.03328 mm is used.
- the system control unit 208 performs processing to superimpose (combine) indexes indicative of the distance to the subject in focus (hereinafter, referred to as a “focus distance”) and indicating the forward depth of field and the backward depth of field derived at step 311 on the captured image. That is, in the positions corresponding to these distances on the ground lattice pattern superimposed at step 310 , the images that serve as indexes are superimposed.
- FIG. 12 is a diagram showing the results of superimposing the index of the focus distance and the index of depth of field (the index of the forward depth of field and the index of the backward depth of field) on the image in FIG. 11 with which the lattice pattern is combined.
- FIG. 12 is a diagram showing the results of superimposing the index of the focus distance and the index of depth of field (the index of the forward depth of field and the index of the backward depth of field) on the image in FIG. 11 with which the lattice pattern is combined.
- an index 1201 indicates the distance (focus distance) in the depth direction of the position where the person (subject P) is located along the ground lattice pattern, representing that the person (subject P) is in focus.
- Indexes 1202 and 1203 indicate the forward depth of field and the backward depth of field, respectively, along the ground lattice pattern, representing that the automobile (subject C) in profile is in the depth of field (it is possible to recognize that the subject C is in focus in the captured image).
- the system control unit 208 displays the captured image with which each index is combined at step 312 on the display unit 101 . Then, as will be described later, by a user giving instructions and performing operations on the combined indexes, it is possible to control the image capturing lens 201 , the aperture of the shutter 202 , etc., to change the focus distance and the depth of field. At this time, it may also be possible to produce a highlighted display of the lattice line corresponding to the position of the index or to display the area of the subject corresponding to the position differently from the display of the areas of the other subjects in accordance with the user's operation to move the index. Due to this, it is possible for a user to grasp which subject is brought into focus or which subject is in the depth of field more clearly.
- the system control unit 208 determines whether there is a touch operation on the display unit 101 by a user. In the case where there is a touch operation by a user, the procedure proceeds to step 315 . On the other hand, in the case where there is no touch operation by a user, the procedure returns to step 305 .
- the system control unit 208 determines whether the touch operation by the user is the operation “to perform touch down in the vicinity of the index of the focus distance and then perform move”. In the case where the touch operation by the user is the operation “to perform touch down in the vicinity of the index of the focus distance and then perform move”, the procedure proceeds to step 316 . On the other hand, in the case where the touch operation by the user is the operation other than the operation “to perform touch down in the vicinity of the index of the focus distance and then perform move”, the procedure proceeds to step 318 .
- the system control unit 208 adjusts the focus by controlling the image capturing lens 201 so that the subject at the distance corresponding to the position touched by a user is brought into focus.
- the system control unit 208 moves the display position of the index indicative of the focus distance to the position touched by the user.
- the system control unit 208 determines whether the touch operation by the user is the operation “to perform touch down in the vicinity of the index of the depth of field and then perform pinch in or pinch out”. In the case where the touch operation by the user is the operation “to perform touch down in the vicinity of the index of the depth of field and then perform pinch in or pinch out”, the procedure proceeds to step 319 . On the other hand, in the case where the touch operation by the user is the operation other than the operation “to perform touch down in the vicinity of the index of the depth of field and then perform pinch in or pinch out”, the procedure proceeds to step 321 .
- the system control unit 208 adjusts the depth of field by controlling the image capturing lens 201 and the shutter 202 so that the subject between the distances corresponding to the user's two fingers (fingers in touch), respectively, is brought into focus. It is known that the depth of field increases by reducing the aperture in the case where the conditions, such as the focal length and the diameter of the permissible circle of confusion, are fixed. It is possible to adjust an increase or a reduction in the quantity of light due to the change in the f-stop by using the ND (Neutral Density) filter or by increasing or reducing the sensitivity of the image sensing element.
- ND Neutral Density
- the system control unit 208 moves the display positions of the two indexes indicative of the depth of field to the positions of the user's two fingers in touch, respectively.
- the system control unit 208 performs processing corresponding to the contents of the touch operation of the user. For example, in the case where any of the areas extracted from the captured image is touched by the user, the system control unit 208 controls the image capturing lens 201 so that the distance corresponding to the area is brought into focus and moves the display position of the index indicative of the focus distance to the corresponding position.
- the system control unit 208 exits the present processing in accordance with a predetermined end operation, such as turning off of the power source by the user.
- the digital camera to which the image processing apparatus according to the present embodiment is applied it is made possible for a user to intuitively grasp which subject is brought into focus on the UI thereof in the state where the image is close to the actual image.
- Light field data is data recording the quantity of light for a light ray path.
- a quantity of light L of the light ray is expressed by L (u, v, x, y) as a function of u, v, x, and y.
- a two-dimensional image is expressed by the quantity of light of a light ray at the coordinates (x, y) on the single plane.
- That which records the quantity of light at the coordinates (x, y) of the image sensing element is the two-dimensional image of a captured image.
- that in which the two-dimensional images are bundled in the number corresponding to the number of coordinates on the first plane is light filed data.
- a method for acquiring light field data by an image capturing apparatus mentions is made of a method in which a microlens array is arranged in front of the image sensing element, a method in which the opening position of the aperture is controlled, etc.
- it is possible to obtain light field data by recording the quantity of light at the coordinates of the image sensing element in correspondence to the coordinates on a plane parallel to this (for example, the coordinates of the position of each microlens).
- a two-dimensional image that can be displayed on a two-dimensional display is obtained. That is, it is made possible to display a plurality of two-dimensional images corresponding to the coordinates in a fixed range on the above-described first plane configuring the light field data on a two-dimensional display by superimposing the plurality of two-dimensional images.
- the two-dimensional images configuring light field data By superimposing two-dimensional images configuring light field data so that the position of a subject at a specific distance agrees with itself at the time of superimposing the two-dimensional images, it is possible to obtain a two-dimensional image in which the subject at the distance is in focus.
- the two-dimensional images configuring light field data each have a parallax corresponding to the distance of the separation between the coordinates on the first plane described above, and therefore, subjects at distances other than the specific distance are superimposed with positions shifted and their images are blurred on the obtained two-dimensional image as a result.
- the parallax of each two-dimensional image is limited, and therefore, it is possible to obtain a two-dimensional image with a large depth of field. That is, by changing the range on the above-described first plane corresponding to the two-dimensional images to be superimposed, it is made possible to change the depth of field of the two-dimensional image obtained by superimposition.
- a distance image from light field data as below by applying the “image correlation method”. That is, two coordinates are selected from the above-described first plane and the two two-dimensional images corresponding thereto are compared. For a small area around an arbitrary pixel in one of the two two-dimensional images, a small area having a similar pattern is specified in the other two-dimensional image, and an amount of movement of the two corresponding small areas between the two two-dimensional images is calculated. This amount of movement is the so-called “parallax” and is in inverse proportion to the distance of the subject displayed at the coordinates. Consequently, by finding the distance to the subject from the parallax for each coordinate and reflecting the distance in the color of the pixel at the coordinate (for example, a shade of gray), a distance image is obtained.
- FIG. 13 is a diagram showing an example of an internal configuration of an apparatus that performs information processing, such as, for example, a general personal computer (hereinafter, PC) and a tablet terminal, which handles light field data.
- PC general personal computer
- tablet terminal which handles light field data.
- an apparatus 1300 includes a CPU 1301 , a hard disk (HD) 1302 , a memory 1303 , a display control unit 1304 , an input unit 1305 , a drive device 1306 , and a communication I/F 1307 and these units are connected to one another by an internal bus 1308 . It is possible for each unit connected to the internal bus 1308 to transmit and receive data to and from one another via the internal bus 1308 .
- a CPU 1301 a hard disk (HD) 1302 , a memory 1303 , a display control unit 1304 , an input unit 1305 , a drive device 1306 , and a communication I/F 1307 and these units are connected to one another by an internal bus 1308 . It is possible for each unit connected to the internal bus 1308 to transmit and receive data to and from one another via the internal bus 1308 .
- the memory 1303 includes, for example, a RAM.
- the CPU 1301 as a display control unit, a change unit, and a calculation unit controls each unit of the apparatus 1300 using the memory 1303 as a work memory in accordance with programs, for example, stored in the hard disk 1302 .
- the programs for the CPU 1301 to operate are not limited to those stored in the hard disk 1302 and it may also be possible to store the programs in advance, for example, in a ROM, not shown schematically.
- the input unit 1305 receives a user's operation and generates a control signal in accordance with the operation, and then supplies the signal to the CPU 1301 .
- the input unit 1305 has a character information input device, such as a keyboard, a pointing device, such as a mouse and a touch panel, etc., as an input device configured to receive a user's operation.
- the touch panel is an input device by which coordinate information in accordance with a touched position on the input unit 1305 configured, for example, into a planar shape is output.
- the CPU 1301 controls each unit of the apparatus 1300 in accordance with programs based on the control signal generated and supplied by the input unit 1305 in response to a user's operation performed to the input device. Due to this, it is possible to cause the apparatus 1300 to perform the operation in accordance with the user's operation.
- the display control unit 1304 outputs a display signal for causing a display 1310 to display an image. For example, to the display control unit 1304 , a display control signal generated by the CPU 1301 in accordance with programs is supplied. The display control unit 1304 generates a display signal based on the display control signal and outputs the signal to the display 1310 . For example, the display control unit 1304 causes the display 1310 to display a GUI screen configuring a GUI (Graphical User Interface) based on the display control signal generated by the CPU 1301 .
- GUI Graphic User Interface
- the input unit 1305 it may also be possible to integrally configure the input unit 1305 and the display 1310 .
- an external storage medium 1320 such as a CD and a DVD
- the drive device 1306 reads data from the external storage medium 1320 mounted thereon and writes data to the external storage medium 1320 based on the control of the CPU 1301 .
- the external storage medium 1320 that can be mounted on the drive device 1306 is not limited to the disc storage medium, such as a CD and a DVD, and a nonvolatile semiconductor memory, such as, for example, a memory card, may be mounted on the drive device 1306 .
- the communication interface (I/F) 1307 communicates with a network 1330 , such as a LAN and the Internet, based on the control of the CPU 1301 .
- FIG. 14 is a flowchart showing a flow of image processing performed in the apparatus that handles light field data according to the present embodiment.
- the series of processing is implemented by the CPU 1301 developing programs recorded in the hard disk 1302 or a ROM, not shown schematically, onto the memory 1303 and executing the programs.
- a two-dimensional image is generated (at step 1401 ) and the generated two-dimensional image is displayed on the display 1310 by the display control unit 1304 (step 1402 ).
- step 1404 the procedure returns to step 1402 (step 1403 ).
- Each piece of processing at step 1404 to step 1412 that follow corresponds to that at step 305 to step 313 in the first embodiment. That is, each piece of processing below is performed.
- step 1406 Processing to extract subject areas based on the subjects detected at step 1404 and the distance information acquired at step 1405 (step 1406 )
- step 1412 Processing to display the two-dimensional image with which the indexes are combined on the display 1310 (step 1412 )
- step 1413 whether there are user's instructions/operation for each index within the image displayed on the display 1310 is determined.
- the user's instructions/operation for the index are input using a mouse in the case of a general PC and are input by the same touch operation as that in the first embodiment in the case of a tablet terminal having a touch panel.
- the procedure proceeds to step 1414 and in the case where there are not user's instructions/operation, the procedure returns to step 1404 .
- processing to change the focus distance and/or the depth of field is performed using the previously-described light field data (step 1414 ).
- the display position of the index indicative of the focus distance or the depth of field is moved to the new position specified by the user (step 1415 ).
- the present processing is exited (step 1416 ).
- the “shifted image” in which the position of each subject area is shifted as in FIG. 9 is obtained. That is, different from an image in which the viewpoint is only changed based on light field data (in this case, each subject is represented in a stereoscopic manner), an image in which each subject area is represented as a portable painting used as part of the backdrop (stage setting) (each subject area has no depth and is represented in a planar manner) is obtained. Due to this, it is made easier for a user to grasp the feeling of distance to each subject.
- the apparatus according to the present embodiment also, it is made possible for a user to intuitively grasp which subject is brought into focus on the display of a PC etc. in the state where the image is close to the actual image.
- the first and second embodiments it is made possible to intuitively grasp the range in which refocus can be performed by the image (shifted image) in which the position of each subject area is shifted in accordance with the distance of each subject area and which is close to the actual image.
- FIG. 15 is a block diagram showing an internal configuration of a camera array image capturing apparatus (also referred to simply as a “camera array”, as known as a camera array system, multiple lens camera, and the like) according to the present embodiment.
- An image capturing unit 1500 acquires image data (digital data) by receiving light information of a subject by an image sensing element and by A/D-converting the received signal.
- the image data is saved in a storage medium, such as an SD card, as captured image data upon receipt of instructions to perform image capturing from a user.
- the image data acquired by the image capturing unit 1500 is data of an image in which a subject space is captured from a plurality of viewpoints and which has parallax (hereinafter, referred to as “parallax image data”).
- the image data acquired by the image capturing unit 1500 is also utilized in the so-called live view function to sequentially display images in real time on a display unit 1506 provided at the rear side of the image capturing apparatus.
- a live view image an image saved in a storage medium in response to instructions to perform image capturing from a user.
- a central processing unit (CPU) 1501 comprehensively controls each unit described below.
- a RAM 1502 functions as a maim memory, a work area, etc.
- a ROM 1503 stores control programs etc. executed in the CPU 1501 .
- a bus 1504 is a transfer path of various kinds of data and for example, digital data acquired by the image capturing unit 1500 is sent to a predetermined processing unit via the bus 1504 .
- An operation unit 1505 configured to receive instructions of a user includes buttons, a mode dial, etc.
- the display unit 1506 configured to display images and characters, for example, a liquid crystal display is used.
- the display unit 1506 may have the touch screen function and in such a case, it is also possible to handle user's instructions using the touch screen as inputs to the operation unit 1505 .
- the range in which refocus can be performed is displayed and the focus position at the time of refocus is specified, such a position is specified by an input of a user via such a touch screen.
- a display control unit 1507 performs display control of images and characters displayed on the display unit 1506 .
- An image capturing unit control unit 1508 performs control of an image capturing system based on instructions from the CPU 1501 , such as focusing, shutter opening/closing, and aperture stop adjustment, etc.
- a digital signal processing unit 1509 performs various kinds of processing, such as white balance processing, gamma processing, and noise reduction processing, on digital data received via the bus 1504 .
- An encoder unit 1510 performs processing to convert digital data into a file format, such as JPEG and MPEG.
- An external memory control unit 1511 is an interface configured to connect to a PC and other media (for example, hard disk, memory card, CF card, SD card, USB memory).
- An image processing unit 1512 performs image processing, such as refocus processing, to be described later, on image data acquired by the image capturing unit 1500 or image data output from the digital signal processing unit 1509 . Details of the image processing unit 1512 will be described later.
- An exposure state prediction unit 1513 predicts an exposure state of the image capturing unit 1500 at the time of image capturing. There exist components of the image capturing apparatus other than those described above, however, they are not the main purpose of the present embodiment, and therefore, explanation is omitted.
- the image capturing unit 1500 acquires parallax image data, i.e. light field data.
- a configuration example of the image capturing unit 1500 is shown in FIG. 16 to FIG. 19 .
- FIG. 16 to FIG. 18 each show a configuration in which a lens array is arranged on the image side of an image forming optical system
- FIG. 19 shows a configuration in which a plurality of image forming optical systems is arranged (camera array).
- a camera array configuration capable of obtaining images from a plurality of viewpoints at the same time as shown in FIG. 16 to FIG. 19 is desirable.
- a lens array 1603 is arranged on an image side conjugate surface of an image forming optical system 1602 with respect to a subject surface 1601 . Further, the lens array 1603 is configured so that an exit pupil of the image forming optical system 1602 and an image sensing element 1604 establish substantially a conjugate relationship. Light rays from the subject surface 1601 enter different pixels of the image sensing element 1604 in accordance with the position and angle of the light ray on the subject surface 1601 via the image forming optical system 1602 and the lens array 1603 . Due to this, parallax image data (light field data) is acquired.
- the lens array 1603 has a role to prevent light rays having passed through different positions on the subject surface 1601 from entering the same pixel. As a result of that, an image in which a group of pixels of the same area on the subject surface 1601 captured from a plurality of viewpoints is arranged side by side is acquired in the image sensing element 1604 .
- An object such as a person and building, does not necessarily need to exist on the subject surface 1601 in FIG. 16 to FIG. 19 . This is because it is possible to focus a person or building that exists behind or in front of the subject surface 1601 by refocus after image capturing.
- FIG. 16 the pupil of the image forming optical system is divided two-dimensionally into nine pupils (three in one dimension), and therefore, images from nine viewpoints are acquired as a result.
- an image corresponding to a certain divided pupil is called a single-viewpoint image.
- the nine single-viewpoint images have parallax with respect to one another, and therefore, a relative positional relationship of the subjects on the image changes in accordance with the subject distance.
- single-viewpoint images are combined so that a certain subject overlaps itself, subjects located at different subject distances are combined in the shifted state. Because of this shift, subjects located at different subject distances are blurred.
- the blurring at this time is determined by the pupil corresponding to the single-viewpoint image used for combination and in the case where all the nine single-viewpoint images are combined, it is possible to reproduce the blurring of the image acquired by the image forming optical system 1602 .
- the subject overlapped on itself in combination of the single-viewpoint images is arbitrary, and therefore, it is possible to reproduce an image in which an arbitrary subject is brought into focus in the image forming optical system 1602 .
- FIG. 20 is a diagram depicting a relationship between the lens array 1603 and the image sensing element 1604 in FIG. 16 .
- a broken line circle 2001 represents an area of a pixel that a light ray having passed through one lens (microlens) configuring the lens array 1603 enters.
- FIG. 20 corresponds to the case where a plurality of microlenses is arranged in the form of a lattice, however, the arrangement of microlenses is not limited to this. For example, the arrangement may have hexagonal symmetry (honeycomb structure). It may also be possible to slightly shift each microlens from a regular arrangement.
- Refocus is performed by superimposing single-viewpoint images, and therefore, it is not possible to refocus a subject blurred within each single-viewpoint image. This is because even by superimposing blurred images, a high frequency component is not obtained and the images remain blurred.
- the focus control range depends on the divided pupils of the image forming optical system 1602 .
- the depth of field in the single-viewpoint image does not necessarily agree with the focus control range. This is because the focus control range changes depending on the resolution ratio between the single-viewpoint images and the combined image obtained by combining them. For example, in the case where the resolution of the combined image is lower than the resolution of the single-viewpoint image from each viewpoint, the sampling pitch of the spatial component in the combined image becomes large relative to that in the single-viewpoint image.
- the depth of field of the combined image becomes deeper than that of the single-viewpoint image and accompanying that, the focus control range is extended. Conversely, in the case where the resolution of the combined image is higher than that of the single-viewpoint image, the focus control range becomes narrower than the depth of field of the single-viewpoint image.
- a range (range conjugate with respect to the image forming optical system 1602 ) conjugate with a range ⁇ +s 2 to ⁇ _s 2 in which refocus can be performed on the image side expressed using the expression (4) is the focus control range, which is a range in which refocus can be performed on the object side.
- FIG. 21 schematically shows the relationship of the expression (4) and the center position of the range in which refocus can be performed is the focus position of the image forming optical system 1602 (in this case, the position of the lens array 1603 is the center position of the range in which refocus can be performed).
- s 2 is an interval between the image side main plane of the image forming optical system 1602 and the image side conjugate surface of the image forming optical system 1602 with respect to the subject surface 1601 .
- the image side focus control range refers to a range conjugate with the focus control range via the image forming optical system 1602 .
- ⁇ y is the sampling pitch of the two-dimensional intensity distribution of light and in the configuration in FIG. 16 , it is equal to a pitch ⁇ LA of the lens array 1603 .
- the relationship of the expression (4) holds also in each of the configurations in FIG. 16 to FIG. 19 .
- the exit pupil distance P of the image forming optical system 1602 is an interval between an exit pupil surface of the image forming optical system 1602 and the image side conjugate surface of the image forming optical system 1602 with respect to the subject surface 1601 .
- N is the number of one-dimensionally divided pupils of the image forming optical system 1602 and F is the F-number of the image forming optical system 1602 . It is possible to derive the F-number by expression (6) below from a focal length f and a lens effective diameter D of the image forming optical system 1602 .
- F f/D expression (6)
- the range in which refocus can be performed is determined according to the number of one-dimensionally divided pupils of the image forming optical system and the range of the field angle defined by the focal length and the lens effective diameter of the image forming optical system. Further, according to the resolution of the combined image data, to be described later, the range in which refocus can be performed is determined.
- FIG. 22 is a block diagram showing an internal configuration of the image processing unit 1512 according to the present embodiment.
- a parallax image data acquisition unit 2201 acquires parallax image data supplied from the image capturing unit 1500 etc.
- An optical parameter acquisition unit 2202 acquires optical parameters of the image capturing unit 1500 necessary for a distance derivation unit 2203 and a focus control range derivation unit 2207 .
- optical parameters refer to a configuration of the image capturing unit 1500 , the exposure state of an aperture etc., the focus position, the focal length in a zoom lens, etc.
- the distance derivation unit 2203 drives information of the distance to the subject in the scene.
- the distance to the subject is derived using the parallax image data supplied from the parallax image data acquisition unit 2201 and the optical parameters supplied from the optical parameter acquisition unit 2202 . Details of distance derivation will be described later.
- a subject area extraction unit 2204 performs processing to extract a subject area to be subjected to processing in a rearranged image generation unit 2205 based on the subject in the scene and the distance information supplied from the distance derivation unit 2203 .
- the subject area extraction processing can also be said as processing to separate subject areas for each fixed distance.
- the rearranged image generation unit 2205 performs processing to change the position (coordinates) of each subject area extracted in the subject area extraction unit 2204 in accordance with the distance information supplied from the distance derivation unit 2203 .
- this processing an image is generated, in which each subject area is rearranged in accordance with the distance from the image capturing apparatus (hereinafter, referred to as a “rearranged image”) and by which it is made easy to grasp the feeling of distance for each subject area.
- An image combination parameter acquisition unit 2206 acquires parameters (resolution after combination, viewpoint, focus position, depth of field, etc.) necessary for image combination.
- the focus control range derivation unit 2207 derives a focus control range using optical parameters supplied from the optical parameter acquisition unit 2202 and image combination parameters supplied from the image combination parameter acquisition unit 2206 .
- a focus position acquisition unit 2208 acquires a focus position at the time of image combination based on a user's input via the operation unit 1505 .
- the distance specified by a user via the touch screen as the operation unit 1505 based on the rearranged image generated by the rearranged image generation unit 2205 is specified as the focus position.
- Information on the specified focus position is sent to a display image generation unit 2209 .
- the display image generation unit 2209 generates an image in which the information on the focus state (focus position, focus control range, depth of field, etc.) is reflected (hereinafter, referred to as a “focus information display image”) for the rearranged image generated in the rearranged image generation unit 2205 .
- the optical parameters, the image combination parameters, the focus control range, and the focus position supplied from each unit described above are used.
- the generation of the focus information display image may be performed on a live view image before image capturing or on a record image after image capturing. Further, the focus information display image may be based on an arbitrary single-viewpoint image included in the parallax image or may be based on a combined image obtained by combining a plurality of single-viewpoint images.
- the data of the generated focus information display image is output to the display unit 1506 .
- the display image generation unit 2209 according to the present embodiment also performs image combination processing in accordance with the image combination parameters using the parallax image data, however, it may also be possible to provide a processing unit configured to perform image combination processing separately from the display image generation unit 2209 .
- FIG. 23 is a flowchart showing a flow of processing in the image processing unit 1512 according to the present embodiment. The series of processing is performed by the CPU 1501 executing a computer executable program in which a procedure shown below is described after reading the program from the ROM 1503 onto the RAM 1502 .
- the parallax image data acquisition unit 2201 acquires parallax image data acquired by the image capturing unit 1500 .
- the parallax image data acquired here may be parallax image data as a live view image or may be parallax image data as a record image as described above.
- the optical parameter acquisition unit 2202 and the image combination parameter acquisition unit 2206 acquire the optical parameters and the image combination parameters, respectively.
- the distance derivation unit 2203 derives distance information in the whole of the scene. Specifically, the distance derivation unit 2203 derives the distance of the scene by a method or the like of performing stereo matching between the single-viewpoint images included in the parallax image. For example, in the stereo matching method, first, a single-viewpoint image that serves as a reference (for example, a single-viewpoint image corresponding to the light ray passing through the vicinity of the center of the pupil) is divided into a plurality of blocks and among the other single-viewpoint images, a block position nearest to the target block is detected.
- a single-viewpoint image that serves as a reference for example, a single-viewpoint image corresponding to the light ray passing through the vicinity of the center of the pupil
- the distance of the target block is derived by the triangulation.
- the single-viewpoint image that serves as a reference it may also be possible to select a single-viewpoint image the viewpoint position of which is other than the center.
- a method for acquiring distance information there are a method, such as DFD (Depth From Defocus), and a method of performing measurement using a range finding unit that utilizes infrared rays etc., besides the stereo matching method described above.
- FIG. 24A is a diagram showing an example of a single-viewpoint image and FIG. 24B shows a distance map as distance information derived for the single-viewpoint image in FIG. 24A .
- three kinds of subjects person, building, mountain
- the distance map shown in FIG. 24B is displayed by the shade in accordance with the distance from the image capturing apparatus, and the subject “person” in the vicinity of the image capturing apparatus is displayed by the color in the darkest shade, the subject “mountain” far from the image capturing apparatus by the color in the lightest shade, and the subject “building” located between the person and the mountain by the color in an intermediate shade.
- the distance information of the subjects in the scene is derived.
- the subject area extraction unit 2204 performs processing to extract subject areas based on the distance map of the whole of the scene derived at step 2303 .
- the subject area extraction processing can also be said as processing to separate subject areas for each fixed distance. Image areas having substantially the same distance are grouped and the grouped image area is extracted as a subject area.
- the method for extracting subject areas mention is made of, for example, methods as below besides the method that uses the distance information of subjects.
- Areas in which a person, animal, plant, etc., exist are identified by applying a technique, such as face recognition, to the parallax image data and the identified person etc. is extracted as a subject area.
- a technique such as face recognition
- Data of images of representative objects of interest such as a person, animal, and plant, is prepared in advance as a template and a subject area is extracted by pattern matching with the template.
- Prior learning is performed as in a neural network and a subject area is extracted by recognizing main objects using the learning result.
- Extraction of a subject area in the present embodiment can be implemented by applying a variety of publicly-known methods as mentioned in 1) to 3) described above besides the method that uses the distance information of subjects.
- FIGS. 25A and 25B are each a diagram showing an example of the results of extracting subject areas from the distance map shown in FIG. 24B
- FIG. 25A shows the diagram in the case where the subject areas are extracted along the contour of each subject
- FIG. 25B shows the diagram in the case where rectangular areas circumscribing each subject are extracted as subject areas.
- each area surrounded by the dotted line is the subject area corresponding to “person”, “building”, and “mountain”, respectively.
- the method for extracting subject areas other methods can also be applied as long as areas are set so as to include main subjects.
- the rearranged image generation unit 2205 generates a rearranged image of the scene using the distance information in the whole of the scene derived at step 2303 and the subject areas in the scene extracted at step 2304 .
- FIGS. 26A to 26D are each a diagram showing an example of a rearranged image in the case where the scene of the single-viewpoint image shown in FIG. 24A is looked down upon from directly above.
- the “subject distance” in FIGS. 24A to 24D does not necessarily need to be the scaled distance to the actual subject and, for example, the subject distance may be represented in a reciprocal ratio or a logarithmic ratio, or it may also be possible to apply a correspondence table by preparing in advance the table in which actual distances and corresponding values are associated with each other.
- the distant view such as “mountain”
- the actual distance of which may reach several kilometers or more
- “person” and “building” the actual distances of which are several meters to tens of meters are displayed by rearrangement with the actual distances being scaled
- subjects in the close-range view are displayed densely in a small area compared to the subjects in the distant view.
- the “subject distance” serves as a guide for a user to grasp the focus state of a subject or to change the focus state of a subject, and therefore, the case where the subjects in the close-range view whose focus states are generally different are displayed densely is not appropriate because it becomes difficult for a user to grasp or change the focus state of a subject.
- the difference between the focus states due to the distances of the subjects in the distant view is very slight compared to that of the subjects at short distances, and therefore, there will not arise any problem for a user to grasp the focus states of the subjects in the distant view even in the case where the subjects in the distant view are displayed densely.
- the number of pixels of one side that can be displayed on the display unit of the image capturing apparatus is 640 and the subjects, such as “mountain”, “building”, and “person”, are located, for example, at distances of 10 km, 8 m, and 2 m, respectively, from the image capturing apparatus, by producing a display in which the subjects are rearranged in positions 0 pixels, 80 pixels, and 320 pixels, respectively, moved from the end part of the display area, the shorter distances the subjects are located at, the more sparsely the subjects are displayed on the display unit, and therefore, this display is convenient for a user to grasp or change the focus state.
- FIGS. 26A to 26D the position nearer to the image capturing apparatus is displayed at the lower part and the position farther from the image capturing apparatus is displayed at the upper part, and therefore, the “person” is arranged at the lower part because of being in the nearest position to the image capturing apparatus and the “mountain” is arranged at the upper part because of being in the farthest position from the image capturing apparatus.
- FIG. 26A is a rearranged image in which the image areas cut out from the above-described single-viewpoint image along the subject areas extracted at step 2304 (see FIG. 25A described above) are rearranged in the positions corresponding to each subject distance.
- FIG. 26B is a rearranged image in which the image areas extracted along the above-described subject areas from the distance map derived at step 2303 (see FIG. 24B ) are rearranged in the positions corresponding to each subject distance.
- each image area is displayed in a shade corresponding to the distance, and therefore, in this rearranged image, the subject distances are enhanced.
- FIGS. 26C and 26D are each a rearranged image in which the frames of subject areas extracted at step 2304 (see FIGS. 25A and 25B described above) are rearranged in the positions corresponding to each subject distance. In the case of these rearranged images, the rearranged image are more simplified.
- the focus control range derivation unit 2207 derives a focus control range in the case where image combination of the parallax image acquired at step 2301 is performed by using the optical parameters and the image combination parameters acquired at step 2302 . Detailed explanation is given below.
- the focus control range changes depending on the resolution ratio between each single-viewpoint image and the combined image, however, here, for simplification of explanation, a case where the resolution ratio is 1 is considered.
- the image side focus control range is within the range expressed by the above-described expression (5), it can be concluded that an area in which focus control can be performed is already acquired. Consequently, the image side conjugate surface of the image forming optical system 1602 with respect to the subject surface 1601 and an image side focus control range d refocus are required only to satisfy expression (7) below.
- FIG. 27 is a diagram showing an optical arrangement in the configuration of the image capturing unit 1500 shown in FIG. 16 , and ⁇ is an interval between the image side main plane of the lens array 1603 and the image sensing element 1604 .
- two parallel broken lines extending from both ends of the microlens at the center represent an area of the image sensing element 1604 corresponding to the microlens and pixels with slashes within the image sensing element 1604 represent a dead zone that no light rays enter.
- the configuration is not limited to this and there may exist a dead zone. From FIG.
- R mono denotes the resolution of the single-viewpoint image used for combination and R comb denotes the resolution of the combined image.
- the ratio of ⁇ y is obtained by calculating the square root of the ratio between R mono and R comb . It is known that the resolution R mono of the single-viewpoint image is represented by expression (10) below from FIG. 27 .
- R mono ( ⁇ ⁇ LA ) 2 ⁇ R total expression ⁇ ⁇ ( 10 )
- R total denotes the number of effective pixels of the image sensing element 1604 .
- the range is set between ⁇ 10.0, however, by setting the range between ⁇ 6.0, it is possible to obtain a clearer combined image. More desirably, by setting the range between ⁇ 3.0, it is possible to obtain a still clearer combined image.
- Pixel pitch ⁇ of the image sensing element 1604 0.0043 (mm)
- Focal length f of the image forming optical system 1602 14.0 (mm)
- the range of the conditional expression (11), the resolution R comb of the combined image corresponding to each range of the conditional expression (11), and d refocus corresponding to each resolution of the combined image are, for example, as shown in Table (1) below.
- the resolution R comb of the combined image is selected, for example, from among the above-mentioned three kinds.
- each resolution R comb of the combined image is an example and the number of kinds is not limited to three. It is only necessary to appropriately determine R comb and d refocus that satisfy the above-mentioned conditional expression (11).
- the information of the focus control range acquired in this manner may be attached to the parallax image data or the combined image data as tag information and stored in the ROM 1503 .
- a method for acquiring a focus control range As another method for acquiring a focus control range, a method can be thought of, which evaluates the contrast etc. of a subject existing in the focus position by actually generating a combined image for which refocus is performed.
- this method it is necessary to generate a combined image while moving the focus position and to sequentially determine whether refocus can be performed successfully, and therefore, it takes time to perform the processing.
- evaluation cannot be carried out, and therefore, it is not possible to acquire an accurate focus control range. Consequently, it is desirable to use the previously-described method in order to acquire a focus control range.
- the display image generation unit 2209 generates a focus information display image, which is the rearranged image generated at step 2305 and in which information on the focus state (focus position, focus control range, depth of field, etc.) is reflected.
- the optical parameters and the image combination parameters acquired at step 2302 and the focus control range derived at step 2306 are used.
- FIGS. 28A and 28B are each an example of the focus information display image generated based on the rearranged image shown in FIG. 26A and each piece of information of a focus position 2801 , a focus control range 2802 , and a depth of field 2803 is displayed.
- the focus position 2801 indicated by a rectangular black mark on a slider bar indicates the focus position of the lens at the time of image capturing included in the optical parameters, or the focus position specified by a user's input, to be described later.
- the focus position 2801 is set in a position at a subject distance d0 (position of the person), and in the focus information display image in FIG.
- the focus position 2801 is set in a position at a subject distance d0′ (position of the building), respectively.
- the focus control range 2802 indicated by slashes on the slider bar indicates a range from a subject distance d1 (d1′) to a subject distance d2 (d2′) and in the present embodiment, it is known that the “person” and the “building” are located at distances in this focus control range. In the following, it is assumed that in the case where the “focus control range” is simply referred to, it refers to this object side focus control range.
- the depth of field 2803 indicated by slashes on the rearranged image is a range from a subject distance d3 (d3′) to a subject distance d4 (d4′) and in the focus information display image in FIG. 28A , the depth of field in the focus position d0 where the person is in focus is shown, and in the focus information display image in FIG. 28B , the depth of field in the focus position d0′ where the building is in focus is shown.
- FIGS. 28A and 28B for convenience of explanation, a schematic diagram of the image capturing apparatus and a viewing field range (field angle) 2804 where image capturing can be performed by the image capturing apparatus are also shown, however, it is not necessarily required to include these pieces of information in the focus information display image generated at this step.
- the display image generation unit 2209 outputs the data of the focus information display image generated at step 2307 to the display unit 1506 .
- the CPU 1501 determines whether a user's input relating to the focus position at the time of refocus processing is newly done.
- the user's input in this case is done via the touch screen or the operation unit 1505 of the image capturing apparatus.
- a method for directly specifying a subject desired to be newly brought into focus on the rearranged image a method for directly specifying the subject distance to a new focus position, or a method for specifying by operating a mark indicative of the focus position 2801 on the slider bar.
- the procedure returns to step 2307 and the series of processing at step 2307 to step 2309 is repeated.
- the focus information display image shown in FIG. 28B described previously is newly displayed as a result.
- the depth of field 2803 has also changed in accordance with the newly set focus position (the subject distance d0′). That is, in the case where a new focus position at the time of refocus processing is input, the focus control range 2802 does not change but the focus position 2801 and the depth of field 2803 in the focus position change mainly as a result.
- FIGS. 29A to 29C each show an example of the variation of the focus information display image.
- FIG. 29A is a focus information display image based on a rearranged image in which the scene is looked down upon from the lateral direction.
- FIG. 29B is a focus information display image based on a rearranged image in which the scene is looked down upon from the obliquely above direction.
- FIG. 29A is a focus information display image based on a rearranged image in which the scene is looked down upon from the lateral direction.
- FIG. 29B is a focus information display image based on a rearranged image in which the scene is looked down upon from the obliquely above direction.
- 29C is a focus information display image based on a rearranged image in which the scene is looked down upon from the obliquely lateral direction.
- the rearranged image that serves as a base of the focus information display image may be any image in which subjects are sequentially arranged in order in the depth direction based on distance information.
- the focus information display image is displayed, and at this time, it may also be possible to simultaneously display the single-viewpoint image acquired by the image capturing unit 1500 and/or the combined image combined in the display image generation unit 2209 .
- FIGS. 30A to 30C each show an example of the case where a single-viewpoint image and/or a combined image is displayed together in the focus information display image.
- FIG. 30A shows a case where a single-viewpoint and two kinds of combined images are displayed in addition to the focus information display image (see FIG. 29A described previously) based on the rearranged image in which the scene is looked down upon from the lateral direction.
- a combined image 1 in FIG. 30A is a combined image in the state where the focus position is set by taking the “person” to be a subject of interest (the “building” and the “mountain” that are subjects other than the subject of interest are in the blurred state because of being outside the range of the depth of field).
- the rearranged image in which the scene is looked down upon from the lateral direction is used as a base, however, it is needless to say that any type of rearranged image may be used as a base.
- two kinds of combined images are displayed, however, only one kind of combined image, or three or more kinds of combined images may be displayed.
- FIG. 30B shows a case where a combined image is displayed in the case where the focus position is set by taking the “person” to be the subject of interest, in addition to the focus information display image based on the rearranged image in which the scene is looked down upon from directly above.
- the focus information display image in FIG. 30B is based on the rearranged image shown in FIG. 26A in which the scene is looked down upon from directly above, from which information in the lateral direction is omitted, and in which only information in the depth direction is left.
- FIG. 30C shows a case where a combined image is displayed in addition to the focus information display image based on the rearranged image in which the scene is looked down upon from directly above.
- the focus information display image in FIG. 30C is based on the rearranged image shown in FIG. 26D in which the scene is looked down upon from directly above, in which the subject areas are surrounded by rectangles, from which information in the lateral direction is omitted, and in which only information in the depth direction is left.
- information on the focus state such as the focus position, the focus control range, and the depth of field at the time of image combination, is displayed based on the rearranged image in which subjects are arranged in order in the depth direction. Due to this, it is made possible for a user to intuitively grasp the focus state of a subject at the time of image capturing and/or at the time of editing the image.
- the lens array 1603 is arranged between the image forming optical system and the image side conjugate surface (left side of the image side conjugate surface).
- the lens array 1603 is arranged outside (right side of the image side conjugate surface), not between the image forming optical system and the image side conjugate surface. Because of this, in the case of the configuration in FIG. 17 and FIG. 18 , data of a parallax image in which a plurality of small images from different image capturing viewpoints and with different image capturing ranges is arranged in order is acquired by the image sensing element 1604 .
- the lens array 1603 is arranged between the image forming optical system and the image side conjugate surface (left side of the image side conjugate surface).
- the lens array 1603 is arranged outside (right side of the image side conjugate surface), not between the image forming optical system and the image side conjugate surface. Because of this, in the case of the configuration in FIG.
- the lens array 1603 regards the image formed by the image forming optical system 1602 as a real object and forms an image again on the image sensing element 1604 .
- the lens array 1603 regards the image formed by the image forming optical system 1602 as an object and forms an image of the object on the image sensing element 1604 , and therefore, both the configurations are the same essentially.
- the configuration in FIG. 17 is explained as an example, however, the contents of the explanation also apply to the configuration in FIG. 18 .
- the processing is similar to that in the third embodiment and in the processing, images of divided pupils of the image forming optical system 1602 are superimposed on one another with the amount of shift corresponding to the distance of the subject desired to be brought into focus.
- FIG. 31 is an enlarged view of the portion of the lens array 1603 and the image sensing element 1604 as a main portion in the configuration in FIG. 17 .
- the lens array 1603 is configured by microlenses whose surface on the object side is flat and whose surface on the image side is convex.
- the shape of the lens array 1603 is not limited to this.
- the alternate long and short dash line represents the field angle of each microlens.
- the virtual image forming surface refers to a surface conjugate with the surface on the side of the object desired to be brought into focus by image combination (surface conjugate via the image forming optical system 1602 ).
- a virtual image forming surface is set to an image side conjugate surface 1701 .
- the pixels projected at the time of combined image generation are represented by the broken line being shifted in units of field angle of each microlens.
- the amount of translation of the pixels is the same. That is, the operation of the pixel at the time of combined image generation in FIG. 17 and FIG. 18 is determined in accordance with the area of the lens array 1603 through which the luminous flux incident on pixels has passed.
- the focus control range in the present embodiment is also described by the same expression (4) as in the third embodiment. That is, the range (range conjugate with respect to the image forming optical system 1602 ) conjugate with the refocus range ⁇ +s 2 to ⁇ s 2 on the image side expressed using the expression (4) described previously is the focus control range, which is the refocus range on the object side.
- FIG. 32 is a diagram corresponding to FIG. 21 according to the third embodiment. ⁇ y in FIG.
- the lens array 1603 regards the image formed by the image forming optical system 1602 as a virtual object, and therefore, forms a reduced image on the image sensing element 1604 with a magnification of ⁇ 2 / ⁇ 1 .
- the expression (4) can be approximated as the expression (5) because ⁇ P (P: exit pupil distance of the image forming optical system 1602 ).
- the flow of the processing in the image processing unit 1512 according to the present embodiment is the same as that in the flowchart in FIG. 23 according to the third embodiment, however, the configuration of the image capturing unit 1500 is different, and therefore, the focus control range is derived as follows (step 2306 ).
- R mono ( ⁇ 2 ⁇ 1 ) 2 ⁇ R total expression ⁇ ⁇ ( 12 )
- conditional expression (13) that the image side focus control range d refocus in the present embodiment should satisfy is found.
- the range of the conditional expression (13), the resolution R comd of the combined image corresponding to each range expressed by the conditional expression (13), and d refocus corresponding to each resolution of the combined image are, for example, as shown in Table (2) below.
- the resolution R comd of the combined image is selected, for example, from among the above-mentioned three kinds.
- the resolution R mono per single-viewpoint image is 6.0 ⁇ 10 6 pix, and therefore, it is known that it is necessary to increase the resolution by the super-resolution by pixel shifting etc.
- FIG. 33 is a diagram of the image capturing unit 1500 having the configuration of the camera array in FIG. 19 according to the present embodiment in the case where the image capturing unit 1500 is viewed from the front (object side) and an arrangement of image forming optical systems 1602 a to 1602 g and image sensing elements 1604 a to 1604 g in each of seven image capturing modules is shown.
- the camera array has hexagonal symmetry with an optical axis of the image forming optical system 1602 b as an axis of rotation.
- the configuration of the camera array is not limited to this and the number and array of the image forming optical systems are arbitrary.
- each of the image sensing elements 1604 a to 1604 g is arrayed in correspondence to each of the image forming optical systems 1602 a to 1602 g , however, in the case where it is possible to acquire the image formed by each of the image forming optical systems 1602 a to 1602 g , the number of image sensing elements may be one.
- the light ray refracted by each of the image forming optical systems 1602 a to 1602 g is received by each of the image sensing elements 1604 a to 1604 g corresponding thereto.
- a plurality of images acquired by the image sensing elements 1604 a to 1604 g are parallax images of a subject space observed from different viewpoints. By performing image combination using the plurality of parallax images, it is possible to obtain the light field of the subject space.
- FIG. 34 is a diagram (sectional view) of the image forming optical system 1602 a and the image sensing element 1604 a viewed from the side.
- the other image forming optical systems 1602 b to 1602 g and image sensing elements 1604 b to 1604 g are the same, however, the configuration of each image forming optical system may differ from one another.
- the image forming optical system 1602 a is a single focus lens and performs refocus by changing the interval between the image forming optical system 1602 a and the image sensing element 1604 a.
- F mono represents the F-number in each of the image forming optical systems 1602 a to 1602 g and P mono represents the exit pupil distance corresponding to each image forming optical system.
- ⁇ P mono and therefore, the expression (4) can be approximated as expression (14) below.
- ⁇ ⁇ s 2 s 2 ⁇ F mono
- ⁇ y s 2 ⁇ F mono ⁇ expression (14)
- the flow of the processing in the image processing unit 1512 according to the present embodiment is the same as that in the flowchart in FIG. 23 according to the third embodiment, however, the configuration of the image capturing unit 1500 is different, and therefore, the focus control range is derived as follows (step 2302 and step 2306 ).
- each of the image forming optical systems 1602 a to 1602 g configuring the image capturing unit 1500 has an aperture the f-stop of which is variable. Because of this, at step 2302 , the f-stop of each image forming optical system at the time of image capturing is acquired as an optical parameter.
- the F-number of each of the image forming optical systems 1602 a to 1602 g at the time of image capturing obtained at step 2302 is substituted for F mono and the focus control range is derived.
- conditional expression (15) that d refocus should satisfy is found.
- the resolution R comb of the combined image is selected, for example, from among the above-mentioned three kinds.
- d refocus is determined so as to satisfy the conditional expression (15).
- the image combination processing explained in the third to fifth embodiments is implemented by using the parallax images acquired by the image capturing apparatus having the image capturing unit shown in FIG. 16 to FIG. 19 and obtained by performing image capturing from a plurality of different viewpoints.
- the image combination processing is not limited to the image combination processing using parallax images. It may also be possible to adjust the focus position and the depth of field of the data of a captured image from a single viewpoint later by image processing.
- the method for performing filter processing on captured image data in accordance with the relative distance from the focus position is known.
- data of a captured image with a deep depth of field and distance information of the scene are acquired.
- the distance information is acquired by providing a distance sensor separately from the image capturing apparatus or by analyzing the captured image data of the image capturing apparatus.
- the original captured image data is used without performing filter processing, and for the areas other than the subject desired to be brought into focus, the relative distance from the subject is found from the distance information.
- filter processing to reduce the amount of blur is performed (for example, the setting to relatively reduce the filter size).
- filter processing to increase the amount of blur is performed (for example, the setting to relatively increase the filter size).
- the focus control range is the depth of field at the time of acquisition of the single-viewpoint image.
- an image in which the focus position etc. is changed is generated by performing image combination processing on an image from a plurality of viewpoints or an image from a single viewpoint.
- image combination processing it is possible to obtain an image in which the focus position etc. is changed without the need to perform image combination processing.
- the method for acquiring a plurality of pieces of captured image data which acquires a plurality of images by bracket image capturing while changing the image capturing parameter little by little.
- data of captured images in which a focus adjustment ring and an aperture adjustment ring included in the image capturing apparatus are changed stepwise is acquired. It is assumed that in the captured image data by bracket image capturing, the parameters at the time of image capturing are held integrally with the captured images. That is, in the third to fifth embodiments, from one piece of captured image data including images from a plurality of viewpoints, a combined image in which the focus position etc. is changed is generated by an arithmetic operation, however, in the present embodiment, from a plurality of pieces of captured image data obtained by bracket image capturing, a desired captured image is selected and displayed.
- bracket image capturing in which the focus position is changed stepwise (focus bracket image capturing)
- image capturing is performed a plurality of times by changing the focus position stepwise while using the operation unit configured to make focus adjustment included in the image capturing apparatus.
- the three kinds of subjects person, building, mountain
- the “person” is located in the nearest position from the image capturing apparatus, the “building” in the intermediately distant position, and the “mountain” in the farthest position, respectively.
- captured image data in each focus position is acquired by continuously changing the focus position so that the subjects located from the nearest position to the farthest position are included.
- the bracket image capturing data of captured images in which the focus position is changed stepwise is displayed without performing image combination processing.
- the focus control range is from the nearest focus position to the image capturing apparatus to the farthest focus position among the focus positions changed at the time of bracket image capturing. Further, it is possible to handle the depth of field substantially the same as the depth of field determined by the aperture adjustment ring of the image capturing apparatus at the time of bracket image capturing.
- a desired captured image is selected and displayed in accordance with the focus position supplied from the focus position acquisition unit 2208 .
- Embodiments of the present invention can also be realized by a computer of a system or apparatus that reads out and executes computer executable instructions recorded on a storage medium (e.g., non-transitory computer-readable storage medium) to perform the functions of one or more of the above-described embodiment(s) of the present invention, and by a method performed by the computer of the system or apparatus by, for example, reading out and executing the computer executable instructions from the storage medium to perform the functions of one or more of the above-described embodiment(s).
- the computer may comprise one or more of a central processing unit (CPU), micro processing unit (MPU), or other circuitry, and may include a network of separate computers or separate computer processors.
- the computer executable instructions may be provided to the computer, for example, from a network or the storage medium.
- the storage medium may include, for example, one or more of a hard disk, a random-access memory (RAM), a read only memory (ROM), a storage of distributed computing systems, an optical disk (such as a compact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)TM), a flash memory device, a memory card, and the like.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computing Systems (AREA)
- Theoretical Computer Science (AREA)
- Human Computer Interaction (AREA)
- Studio Devices (AREA)
- Image Processing (AREA)
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
US14/984,071 US9521320B2 (en) | 2013-03-05 | 2015-12-30 | Image processing apparatus, image capturing apparatus, image processing method, and storage medium |
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2013-043264 | 2013-03-05 | ||
JP2013043264 | 2013-03-05 | ||
JP2013-134111 | 2013-06-26 | ||
JP2013134111A JP6214236B2 (ja) | 2013-03-05 | 2013-06-26 | 画像処理装置、撮像装置、画像処理方法、及びプログラム |
Related Child Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/984,071 Division US9521320B2 (en) | 2013-03-05 | 2015-12-30 | Image processing apparatus, image capturing apparatus, image processing method, and storage medium |
Publications (2)
Publication Number | Publication Date |
---|---|
US20140253760A1 US20140253760A1 (en) | 2014-09-11 |
US9270902B2 true US9270902B2 (en) | 2016-02-23 |
Family
ID=51469264
Family Applications (2)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/195,951 Expired - Fee Related US9270902B2 (en) | 2013-03-05 | 2014-03-04 | Image processing apparatus, image capturing apparatus, image processing method, and storage medium for obtaining information on focus control of a subject |
US14/984,071 Active US9521320B2 (en) | 2013-03-05 | 2015-12-30 | Image processing apparatus, image capturing apparatus, image processing method, and storage medium |
Family Applications After (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US14/984,071 Active US9521320B2 (en) | 2013-03-05 | 2015-12-30 | Image processing apparatus, image capturing apparatus, image processing method, and storage medium |
Country Status (3)
Country | Link |
---|---|
US (2) | US9270902B2 (fr) |
JP (1) | JP6214236B2 (fr) |
CN (2) | CN104038690B (fr) |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US9797716B2 (en) | 2015-01-09 | 2017-10-24 | Ricoh Company, Ltd. | Estimating surface properties using a plenoptic camera |
US10009540B2 (en) | 2012-08-14 | 2018-06-26 | Canon Kabushiki Kaisha | Image processing device, image capturing device, and image processing method for setting a combination parameter for combining a plurality of image data |
US10200590B2 (en) | 2015-07-31 | 2019-02-05 | Canon Kabushiki Kaisha | Image pickup apparatus and method of controlling the same |
US10269130B2 (en) | 2014-11-06 | 2019-04-23 | Beijing Zhigu Tech Co., Ltd. | Methods and apparatus for control of light field capture object distance adjustment range via adjusting bending degree of sensor imaging zone |
US11138684B2 (en) * | 2019-04-25 | 2021-10-05 | Fanuc Corporation | Image processing apparatus, image processing method, and robot system |
Families Citing this family (54)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2013254432A (ja) * | 2012-06-08 | 2013-12-19 | Canon Inc | 画像処理装置及び画像処理方法 |
JP2014126612A (ja) * | 2012-12-25 | 2014-07-07 | Canon Inc | 撮像装置およびその制御方法 |
JP6305053B2 (ja) * | 2013-01-15 | 2018-04-04 | キヤノン株式会社 | 画像処理装置、撮像装置、画像処理方法、及びプログラム |
US9621794B2 (en) * | 2013-02-21 | 2017-04-11 | Nec Corporation | Image processing device, image processing method and permanent computer-readable medium |
WO2015137635A1 (fr) * | 2014-03-13 | 2015-09-17 | Samsung Electronics Co., Ltd. | Appareil de capture d'image et procédé de génération d'image comportant des informations de profondeur |
JP6432038B2 (ja) * | 2014-03-19 | 2018-12-05 | パナソニックIpマネジメント株式会社 | 撮像装置 |
CN104243823B (zh) | 2014-09-15 | 2018-02-13 | 北京智谷技术服务有限公司 | 光场采集控制方法和装置、光场采集设备 |
CN104243825B (zh) * | 2014-09-22 | 2017-11-14 | 广东欧珀移动通信有限公司 | 一种移动终端自动对焦方法及系统 |
CN105574818B (zh) * | 2014-10-17 | 2020-07-17 | 中兴通讯股份有限公司 | 一种景深渲染方法和装置 |
CN104469147B (zh) * | 2014-11-20 | 2018-09-04 | 北京智谷技术服务有限公司 | 光场采集控制方法和装置、光场采集设备 |
JP6704700B2 (ja) * | 2014-12-15 | 2020-06-03 | キヤノン株式会社 | 画像処理装置およびその制御方法ならびにプログラム |
US11340602B2 (en) | 2014-12-19 | 2022-05-24 | Raytheon Technologies Corporation | Sensor data fusion for prognostics and health monitoring |
JP6504693B2 (ja) * | 2015-01-06 | 2019-04-24 | オリンパス株式会社 | 撮像装置、操作支援方法及び操作支援プログラム |
JP6529267B2 (ja) * | 2015-01-23 | 2019-06-12 | キヤノン株式会社 | 情報処理装置及びその制御方法、プログラム、並びに記憶媒体 |
US10605916B2 (en) | 2015-03-17 | 2020-03-31 | Cornell University | Depth field imaging apparatus, methods, and applications |
KR102354989B1 (ko) * | 2015-04-14 | 2022-01-24 | 삼성전자주식회사 | 경로 렌더링을 위한 타일 비닝을 수행하는 방법 및 장치. |
JP6521715B2 (ja) * | 2015-04-22 | 2019-05-29 | キヤノン株式会社 | 撮像装置、その制御方法、および制御プログラム |
JP6646361B2 (ja) | 2015-04-27 | 2020-02-14 | ソニーセミコンダクタソリューションズ株式会社 | 画像処理装置、撮像装置、画像処理方法およびプログラム |
CN105827936B (zh) * | 2015-08-28 | 2019-06-07 | 维沃移动通信有限公司 | 一种图像处理方法及移动终端 |
WO2017038158A1 (fr) * | 2015-08-31 | 2017-03-09 | 富士フイルム株式会社 | Dispositif de mesure de distance, procédé de commande pour une mesure de distance, et programme de commande pour une mesure de distance |
JP6611531B2 (ja) * | 2015-09-16 | 2019-11-27 | キヤノン株式会社 | 画像処理装置、画像処理装置の制御方法、およびプログラム |
JP6592330B2 (ja) * | 2015-11-02 | 2019-10-16 | キヤノン株式会社 | 撮像装置及び撮像方法 |
CN106888352B (zh) * | 2015-12-16 | 2020-12-18 | 中兴通讯股份有限公司 | 一种推焦位置确定方法及装置 |
CN106254804B (zh) * | 2016-01-18 | 2019-03-29 | 北京智谷睿拓技术服务有限公司 | 光场显示控制方法和装置、光场显示设备 |
JP6628617B2 (ja) * | 2016-01-25 | 2020-01-15 | キヤノン株式会社 | 画像処理装置及び画像処理方法 |
JP6702738B2 (ja) * | 2016-01-27 | 2020-06-03 | キヤノン株式会社 | 照明装置、照明システム及び外部電源装置 |
JP2017224970A (ja) * | 2016-06-15 | 2017-12-21 | ソニー株式会社 | 画像処理装置、画像処理方法、および撮像装置 |
US9953432B2 (en) * | 2016-08-17 | 2018-04-24 | David M. Smith | Systems and methods of detecting motion |
JP6752681B2 (ja) * | 2016-10-19 | 2020-09-09 | キヤノン株式会社 | 表示制御装置、表示制御装置の制御方法及びプログラム並びに記憶媒体 |
JP7014175B2 (ja) * | 2016-11-08 | 2022-02-15 | ソニーグループ株式会社 | 画像処理装置、画像処理方法、及び、プログラム |
CN106657782B (zh) * | 2016-12-21 | 2020-02-18 | 努比亚技术有限公司 | 一种图片处理方法和终端 |
JP6729428B2 (ja) * | 2017-02-01 | 2020-07-22 | オムロン株式会社 | 画像処理システム、光学センサ、及び学習装置 |
CN107037683A (zh) * | 2017-05-25 | 2017-08-11 | 歌尔股份有限公司 | 一种投影仪及自动对焦的方法 |
CN107091800A (zh) * | 2017-06-06 | 2017-08-25 | 深圳小孚医疗科技有限公司 | 用于显微成像粒子分析的聚焦系统和聚焦方法 |
JP7033865B2 (ja) * | 2017-08-10 | 2022-03-11 | キヤノン株式会社 | 画像生成装置、画像生成方法、及びプログラム |
JP2019053644A (ja) * | 2017-09-19 | 2019-04-04 | セイコーエプソン株式会社 | 頭部装着型表示装置、及び頭部装着型表示装置の制御方法 |
JP7158841B2 (ja) * | 2017-11-08 | 2022-10-24 | キヤノン株式会社 | 撮像装置、撮像方法、プログラム、記録媒体および画像処理装置 |
DE102018218986B4 (de) * | 2017-11-10 | 2023-02-16 | Denso Corporation | Kameramodul |
DE102018103092A1 (de) * | 2018-02-12 | 2019-08-14 | Arnold & Richter Cine Technik Gmbh & Co. Betriebs Kg | Fokuseinstellungs-Anzeigeeinheit |
US10863103B2 (en) * | 2018-02-21 | 2020-12-08 | Canon Kabushiki Kaisha | Setting apparatus, setting method, and storage medium |
CN108600623B (zh) * | 2018-04-20 | 2020-07-28 | 维沃移动通信有限公司 | 重聚焦显示方法以及终端设备 |
JP7472795B2 (ja) * | 2018-11-13 | 2024-04-23 | ソニーグループ株式会社 | 情報処理装置、情報処理方法及びプログラム |
JP7173841B2 (ja) | 2018-11-14 | 2022-11-16 | キヤノン株式会社 | 画像処理装置およびその制御方法ならびにプログラム |
JP7198055B2 (ja) * | 2018-11-16 | 2022-12-28 | キヤノン株式会社 | 画像処理装置およびその制御方法ならびにプログラム |
CN111510694A (zh) | 2019-01-30 | 2020-08-07 | 中强光电股份有限公司 | 投影装置及调校格线产生方法 |
JP7327964B2 (ja) * | 2019-03-20 | 2023-08-16 | キヤノン株式会社 | 制御装置、撮像システム、プログラム、および、記憶媒体 |
WO2020196378A1 (fr) * | 2019-03-26 | 2020-10-01 | パナソニックIpマネジメント株式会社 | Procédé d'acquisition d'image de distance et dispositif de détection de distance |
JP2020167517A (ja) * | 2019-03-29 | 2020-10-08 | ソニー株式会社 | 画像処理装置と画像処理方法とプログラムおよび撮像装置 |
US10853959B2 (en) * | 2019-04-01 | 2020-12-01 | Sandisk Technologies Llc | Optical inspection tool and method |
JP7373297B2 (ja) * | 2019-04-25 | 2023-11-02 | キヤノン株式会社 | 画像処理装置、画像処理方法及びプログラム |
WO2021060397A1 (fr) * | 2019-09-26 | 2021-04-01 | 株式会社小糸製作所 | Caméra de déclenchement, automobile, feu de véhicule, dispositif de traitement d'image et procédé de traitement d'image |
US20230005213A1 (en) * | 2020-01-23 | 2023-01-05 | Sony Group Corporation | Imaging apparatus, imaging method, and program |
CN111524062B (zh) * | 2020-04-22 | 2023-11-24 | 北京百度网讯科技有限公司 | 图像生成方法和装置 |
CN113747067B (zh) * | 2021-09-07 | 2024-02-02 | 维沃移动通信有限公司 | 拍照方法、装置、电子设备及存储介质 |
Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH09186931A (ja) | 1995-12-28 | 1997-07-15 | Canon Inc | 画像入力装置および画像処理装置 |
JP2001169308A (ja) | 1999-12-06 | 2001-06-22 | Mixed Reality Systems Laboratory Inc | 奥行き情報計測装置及び複合現実感提示システム |
US20070025074A1 (en) | 2004-05-19 | 2007-02-01 | Masahiro Sone | Disk array device |
US20080089557A1 (en) * | 2005-05-10 | 2008-04-17 | Olympus Corporation | Image processing apparatus, image processing method, and computer program product |
WO2008050904A1 (fr) | 2006-10-25 | 2008-05-02 | Tokyo Institute Of Technology | Procédé de génération d'image dans un plan de focalisation virtuel haute résolution |
JP2008515110A (ja) | 2004-10-01 | 2008-05-08 | ザ ボード オブ トラスティーズ オブ ザ レランド スタンフォード ジュニア ユニバーシティー | 撮像の装置と方法 |
US20080131019A1 (en) | 2006-12-01 | 2008-06-05 | Yi-Ren Ng | Interactive Refocusing of Electronic Images |
JP2009124213A (ja) | 2007-11-12 | 2009-06-04 | Sony Corp | 撮像装置 |
JP2010177741A (ja) | 2009-01-27 | 2010-08-12 | Olympus Corp | 撮像装置 |
JP2011039457A (ja) | 2009-08-18 | 2011-02-24 | Canon Inc | 撮像装置及び撮像装置の制御方法 |
US20130194385A1 (en) * | 2011-03-31 | 2013-08-01 | JVC Kenwood Corporation | Stereoscopic image generation apparatus and stereoscopic image generation method |
Family Cites Families (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP4637942B2 (ja) * | 2008-09-30 | 2011-02-23 | 富士フイルム株式会社 | 3次元表示装置および方法並びにプログラム |
JP5478935B2 (ja) * | 2009-05-12 | 2014-04-23 | キヤノン株式会社 | 撮像装置 |
JP5495683B2 (ja) * | 2009-09-10 | 2014-05-21 | キヤノン株式会社 | 撮像装置および測距方法 |
US20120200673A1 (en) * | 2010-06-15 | 2012-08-09 | Junichi Tagawa | Imaging apparatus and imaging method |
JP5871113B2 (ja) * | 2011-03-31 | 2016-03-01 | 株式会社Jvcケンウッド | 立体画像生成装置、立体画像生成方法及び立体画像生成プログラム |
JP2013009274A (ja) * | 2011-06-27 | 2013-01-10 | Canon Inc | 画像処理装置および画像処理方法、プログラム |
JP6080417B2 (ja) * | 2011-08-19 | 2017-02-15 | キヤノン株式会社 | 画像処理装置、及び画像処理方法 |
JP6029380B2 (ja) | 2012-08-14 | 2016-11-24 | キヤノン株式会社 | 画像処理装置、画像処理装置を具備する撮像装置、画像処理方法、及びプログラム |
JP6305053B2 (ja) | 2013-01-15 | 2018-04-04 | キヤノン株式会社 | 画像処理装置、撮像装置、画像処理方法、及びプログラム |
-
2013
- 2013-06-26 JP JP2013134111A patent/JP6214236B2/ja not_active Expired - Fee Related
-
2014
- 2014-03-04 US US14/195,951 patent/US9270902B2/en not_active Expired - Fee Related
- 2014-03-05 CN CN201410078684.7A patent/CN104038690B/zh active Active
- 2014-03-05 CN CN201610015181.4A patent/CN105491294B/zh active Active
-
2015
- 2015-12-30 US US14/984,071 patent/US9521320B2/en active Active
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH09186931A (ja) | 1995-12-28 | 1997-07-15 | Canon Inc | 画像入力装置および画像処理装置 |
JP2001169308A (ja) | 1999-12-06 | 2001-06-22 | Mixed Reality Systems Laboratory Inc | 奥行き情報計測装置及び複合現実感提示システム |
US20070025074A1 (en) | 2004-05-19 | 2007-02-01 | Masahiro Sone | Disk array device |
JP2008515110A (ja) | 2004-10-01 | 2008-05-08 | ザ ボード オブ トラスティーズ オブ ザ レランド スタンフォード ジュニア ユニバーシティー | 撮像の装置と方法 |
US20080089557A1 (en) * | 2005-05-10 | 2008-04-17 | Olympus Corporation | Image processing apparatus, image processing method, and computer program product |
WO2008050904A1 (fr) | 2006-10-25 | 2008-05-02 | Tokyo Institute Of Technology | Procédé de génération d'image dans un plan de focalisation virtuel haute résolution |
US20080131019A1 (en) | 2006-12-01 | 2008-06-05 | Yi-Ren Ng | Interactive Refocusing of Electronic Images |
JP2009124213A (ja) | 2007-11-12 | 2009-06-04 | Sony Corp | 撮像装置 |
JP2010177741A (ja) | 2009-01-27 | 2010-08-12 | Olympus Corp | 撮像装置 |
JP2011039457A (ja) | 2009-08-18 | 2011-02-24 | Canon Inc | 撮像装置及び撮像装置の制御方法 |
US20130194385A1 (en) * | 2011-03-31 | 2013-08-01 | JVC Kenwood Corporation | Stereoscopic image generation apparatus and stereoscopic image generation method |
Non-Patent Citations (1)
Title |
---|
Ren Ng, "Fourier Slice Photography", ACM Trans. Graph. 24, p. 735 to 744 (2005). |
Cited By (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US10009540B2 (en) | 2012-08-14 | 2018-06-26 | Canon Kabushiki Kaisha | Image processing device, image capturing device, and image processing method for setting a combination parameter for combining a plurality of image data |
US10269130B2 (en) | 2014-11-06 | 2019-04-23 | Beijing Zhigu Tech Co., Ltd. | Methods and apparatus for control of light field capture object distance adjustment range via adjusting bending degree of sensor imaging zone |
US9797716B2 (en) | 2015-01-09 | 2017-10-24 | Ricoh Company, Ltd. | Estimating surface properties using a plenoptic camera |
US10200590B2 (en) | 2015-07-31 | 2019-02-05 | Canon Kabushiki Kaisha | Image pickup apparatus and method of controlling the same |
US11138684B2 (en) * | 2019-04-25 | 2021-10-05 | Fanuc Corporation | Image processing apparatus, image processing method, and robot system |
Also Published As
Publication number | Publication date |
---|---|
CN104038690B (zh) | 2017-06-20 |
US20160134807A1 (en) | 2016-05-12 |
CN105491294B (zh) | 2018-11-09 |
US20140253760A1 (en) | 2014-09-11 |
JP6214236B2 (ja) | 2017-10-18 |
CN104038690A (zh) | 2014-09-10 |
JP2014197824A (ja) | 2014-10-16 |
CN105491294A (zh) | 2016-04-13 |
US9521320B2 (en) | 2016-12-13 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US9521320B2 (en) | Image processing apparatus, image capturing apparatus, image processing method, and storage medium | |
US9503633B2 (en) | Image processing apparatus, image capturing apparatus, image processing method, and storage medium | |
US9456141B2 (en) | Light-field based autofocus | |
US10264174B2 (en) | Photographing apparatus and focus detection method using the same | |
JP6548367B2 (ja) | 画像処理装置、撮像装置、画像処理方法及びプログラム | |
US20120105590A1 (en) | Electronic equipment | |
JP2014197824A5 (fr) | ||
US20110025830A1 (en) | Methods, systems, and computer-readable storage media for generating stereoscopic content via depth map creation | |
US20150222810A1 (en) | Image processing apparatus and image processing method | |
JP6202910B2 (ja) | 映像処理装置およびその制御方法、プログラム | |
JP2010093422A (ja) | 撮像装置 | |
CN102959943A (zh) | 立体全景图像合成装置、图像捕获装置、立体全景图像合成方法、记录介质、以及计算机程序 | |
US9774782B2 (en) | Image pickup apparatus and image pickup method | |
US10109036B2 (en) | Image processing apparatus, control method for same, and program that performs image processing for image data having a focus state that is changeable | |
CN103370943A (zh) | 成像装置和成像方法 | |
JP6234401B2 (ja) | 画像処理装置、撮像装置、画像処理方法、及びプログラム | |
US20130155204A1 (en) | Imaging apparatus and movement controlling method thereof | |
JP6611531B2 (ja) | 画像処理装置、画像処理装置の制御方法、およびプログラム | |
JP2015005925A (ja) | 画像処理装置および画像処理方法 | |
US9172860B2 (en) | Computational camera and method for setting multiple focus planes in a captured image | |
US11323689B2 (en) | Image processing device, imaging device, image processing method, and recording medium | |
JP2013090241A (ja) | 表示制御装置、撮像装置および表示制御プログラム | |
EP2911383B1 (fr) | Appareil de traitement d'images, procédé de contrôle, programme et support d'enregistrement | |
JP2013175821A (ja) | 画像処理装置、画像処理方法およびプログラム | |
EP2839339A1 (fr) | Mise au point d'image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: CANON KABUSHIKI KAISHA, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:WATANABE, KAZUHIRO;SAKATA, OSAMU;TAKATANI, KEITA;AND OTHERS;SIGNING DATES FROM 20140215 TO 20140218;REEL/FRAME:033124/0367 |
|
ZAAA | Notice of allowance and fees due |
Free format text: ORIGINAL CODE: NOA |
|
ZAAB | Notice of allowance mailed |
Free format text: ORIGINAL CODE: MN/=. |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |
|
MAFP | Maintenance fee payment |
Free format text: PAYMENT OF MAINTENANCE FEE, 4TH YEAR, LARGE ENTITY (ORIGINAL EVENT CODE: M1551); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY Year of fee payment: 4 |
|
FEPP | Fee payment procedure |
Free format text: MAINTENANCE FEE REMINDER MAILED (ORIGINAL EVENT CODE: REM.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
LAPS | Lapse for failure to pay maintenance fees |
Free format text: PATENT EXPIRED FOR FAILURE TO PAY MAINTENANCE FEES (ORIGINAL EVENT CODE: EXP.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STCH | Information on status: patent discontinuation |
Free format text: PATENT EXPIRED DUE TO NONPAYMENT OF MAINTENANCE FEES UNDER 37 CFR 1.362 |
|
FP | Lapsed due to failure to pay maintenance fee |
Effective date: 20240223 |