JP2019110365A - Imaging system, lens device, and image processing tool - Google Patents

Abstract

To provide an imaging system capable of performing depth control with a small amount of data, a lens device, and an image processing tool.SOLUTION: An imaging system includes: means for detecting a defocus state of a plurality of points during photographing; means for storing information of the curvature radius, surface separation and reflective index of an optical system; means for calculating and generating an optical filter required for depth control on the basis of the information; and means for processing a photographed image from the optical filter.SELECTED DRAWING: Figure 2A

Description

  The present invention relates to an imaging system capable of optical correction, a lens apparatus, and an image processing tool, and is particularly suitable for an interchangeable lens imaging system and an imaging apparatus such as a video.

  2. Description of the Related Art Conventionally, various means for obtaining a good image and various processing methods have been reported in an imaging apparatus by correcting optical imaging deterioration using correction information specific to the optical system. Patent Document 1 reports an image restoration process for restoring optical image deterioration such as spherical aberration and coma aberration. Further, Patent Document 2 introduces a method of performing depth expansion by defocus detection and a point spread function (PSF) according to defocus.

  On the other hand, in the imaging apparatus, an imaging surface phase difference which can detect the state of the pupil of the optical system and process it to obtain a defocused state of the optical system on the imaging surface on which the object is imaged by the optical system There is known an imaging device having a defocus detection means of the type. Patent Document 3 describes an imaging surface phase difference type defocus detection means and an arithmetic processing method thereof.

  Further, Patent Document 4 introduces a refocusing technology in which the techniques of Patent Documents 1 to 3 are combined to finely adjust the focus position of an out-of-focus image. According to this refocusing technique, it is possible to make an out-of-focus photograph into an image that does not have a sense of discomfort in which the subject is in focus by image processing.

JP, 2011-217087, A JP, 2012-065114, A Unexamined-Japanese-Patent No. 2010-025997 JP, 2014-153890, A

  It is desirable for an imaging device to focus an object at high speed and with high accuracy. In order to realize this, high-speed and high-precision has been achieved by enhancing the accuracy of the phase difference type defocus detection means and increasing the focusing speed of the optical system. Even so, it is not uncommon for a camera to move out of focus due to a lack of control due to a subject moving at a very high speed or a photographer's operation failing to catch up, etc. Therefore, there is a demand for a depth control technique including refocusing technique as in Patent Document 4 and addition of blur to the depth as in Patent Document 2.

  However, in these depth control techniques, it is an issue to have a huge amount of data. In particular, it is more preferable that the processing is performed not immediately after moving the image to a PC or the like, but immediately after shooting on the imaging device, in which case weight reduction of data becomes essential.

  An object of the present invention is to provide an imaging system, a lens apparatus, and an image processing tool capable of performing depth control with a small amount of data.

  In order to achieve the above object, an imaging system according to the present invention comprises: means for detecting defocusing states of a plurality of points at the time of shooting; and means for storing information of curvature radius, surface distance, and refractive index of optical system; It is characterized by having a means for computing and generating an optical filter necessary for depth control based on the information, and a means for processing a photographed image from the optical filter.

  Further, the lens apparatus according to the present invention comprises means for detecting the defocus state of a plurality of points at the time of photographing, and an optical filter necessary for depth control based on information of the radius of curvature, surface distance and refractive index of the optical system. The image processing apparatus is characterized in that it is detachable from an imaging apparatus having means for generating operations and means for processing a photographed image from the optical filter, and means for storing the information.

  In addition, the image processing tool according to the present invention comprises: means for detecting defocusing states of a plurality of points at the time of shooting; means for storing information of curvature radius, surface distance, and refractive index of optical system; The image processing apparatus is characterized by comprising: means for computing and generating an optical filter necessary for depth control from a photographed image obtained by an imaging device having means for writing in; and means for processing the photographed image from the optical filter.

  According to the present invention, depth control can be performed with a small amount of data.

Figure showing subject with depth and shooting optics A diagram showing a planar projection object A picture showing a photographed image of a subject with depth Flowchart for measuring depth by imaging surface phase difference ranging Flowchart for measuring by performing contrast operation while focusing drive the depth Flow chart for performing focus adjustment based on RDN data from captured image and defocus information (depth map) A diagram showing a depth map obtained by imaging surface phase difference ranging Diagram showing the process of obtaining a depth map from contrast autofocus

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In the imaging optical system, the original image on the object side is f (x, y), the point spread function (PSF) representing the performance of the optical system is h (x, y), and the image plane side after passing through the optical system Assuming that the degraded image of is g (x, y), the following relationship is established. Here, * is convolution (convolution), and (x, y) are coordinates on the image.

g (x, y) = h (x, y) * f (x, y) ... (Equation 1)
Next, a method of performing image restoration processing based on this relational expression will be described. The Fourier transform of Equation 1 results in the form of a product for each frequency as follows: Here, H is OTF, and (u, v) are coordinates in a two-dimensional frequency plane, that is, frequency.

G (u, v) = H (u, v) × F (u, v) (Equation 2)
The original image f (x, y) is obtained as a restored image by dividing both sides of Equation 2 by H and performing inverse Fourier transform on F (u, v) to return to the real surface.

  Here, if the inverse Fourier transform of H-1 is R, the following equation holds.

f (x, y) = g (x, y) * R (x, y) ... (Equation 3)
R (x, y) is an image restoration filter and can be created based on optical information of the lens, so an original image can be similarly obtained by performing convolution on the image on the real side. The above method is the basic idea of image restoration.

  However, this concept is limited to the relationship between the image plane and the object plane, which is in an imaging relationship in the optical system. In an actual imaging apparatus, although an imaging plane corresponding to an image plane is a plane, a subject corresponding to an object plane is not necessarily a plane, so many subjects deviate in a depth direction (z direction) from the object plane.

  At this time, a subject deviating in the depth direction from the object plane forms an image at a position deviating in the z direction also in the imaging plane, and thus becomes a blurred image on the imaging plane. That is, when an object deviated from the object plane of the optical system is matched with an object which is originally intended to be in focus, the image is blurred, and the photographer recognizes that the image is out of focus. This is expressed by a point spread function.

  The point spread function h (x, y) is a three-dimensional h (x, y) because the image is blurred or sharpened depending on the depth direction of the subject when viewed from the imaging plane. , Z).

  That is, when photographing a three-dimensional object, it is mathematically expressed as follows.

g '(x, y) = f' (x, y) * h (x, y, z) ... (Equation 4)
Here, f ′ (x, y) is a plane projection original image obtained by projecting a subject f (x, y, z) that is a three-dimensional object onto an arbitrary plane.

  The planar projection original image will be described with reference to FIG. FIG. 1A is a diagram showing a subject with depth and a photographing optical system. When a line is drawn radially from the principal point H of the imaging optical system toward each subject ABC, the point at which they are in tolerance with an arbitrary plane parallel to the image plane is the projection point of the subject onto the object plane.

  What performed these for all the subjects is a plane projection original image, which is as shown in FIG. 1B. Although the size of the subject ABC is actually the same, it can be seen that the closer the distance from the main point, the more the image is enlarged. An image obtained by multiplying this plane projection original image f ′ (x, y) by the respective coordinates and the point spread function h (x, y, z) in the out-focus state is g ′ (x, y) It is.

  Since the subject A and the subject C deviate from the object plane, the blurred point image intensity function is applied, and the subject B relatively close to the object plane is subjected to the sharp point image intensity function, and the resultant g is obtained '(X, y) becomes like FIG. 1C.

Next, a method of obtaining an object plane projection original image f ′ (x, y) from the deteriorated image g ′ (x, y) obtained in this manner will be considered. From the relationship between Equation 1 and Equation 3, it is obvious that the degraded image can be expressed as the following equation by applying an image restoration filter with R (x, y, z) and also a function in the z direction f '(x , Y) = g '(x, y) * R (x, y, z) (5)
Here, the problem is that the degraded image g '(x, y) is plane information, so the information of z is missing, and the z term of R (x, y, z) can not be identified, and the degraded image It is impossible to perform recovery processing to a plane projection original image only from the point.

  Therefore, in the present embodiment, the defocus state is obtained according to the coordinates of the sensor surface at the time of shooting, and z is specified from the information. Specifically, a method of obtaining the defocus amount by the imaging surface phase difference method or a method of reading the contrast value while focusing and calculating the defocus amount from the contrast waveform obtained thereby is used. As a result, since z can be specified for each (x, y) coordinate point, R (x, y, z) is determined, and it has been found that an object surface projection original image f '(x, y) can be obtained.

  Next, a method of obtaining an image out of focus by a desired defocus amount will be considered. In FIG. 1A, since the object plane is at the position of the subject B, the subject B is sharp, and the subjects A and C are in a blurred state. The focus shift amount at this time is Δz.

  First, a plane projection original image f '(x, y) is obtained by Expression 5. After the plane projection original image is obtained, the point image intensity distribution function in a state to be originally focused is reproduced from the optical data, and a desired image can be obtained by multiplication.

  Here, the point image intensity distribution function reproduced from the optical data is defined as k in order to distinguish it from the point image intensity distribution function h of the real lens.

  In addition, the point spread function and the image restoration filter will be expressed in more detail. Image restoration filter R and point spread function k change according to zoom position z, focus position fo, f-number fno, image height (x, y) and defocus state z of optical system R (z, fo, fno, x, y, z) and k (z, fo, fno, x, y, z).

  Now, assuming that the point spread function with focus on subject B is k (z, fo, fno, x, y, z), then focus on subject C defocused from subject B by Δz The point spread function of is k (z, fo + Δfo, fno, x, y, z + Δz).

    Here, Δfo can be calculated by giving a focus advancing amount for defocusing by Δz, and hence k (z, fo + Δfo, fno, x, y, z + Δz) can be obtained from only Δz. Therefore, the optical information of Δz and k can be expressed as the following equation.

gd (x, y) = f '(x, y) * k (z, fo + Δfo, fno, x, y, z + Δz)
... (Equation 6)
gd (x, y) is a desired image in which the subject C is in focus, the subject B is blurred by Δz, and the subject A is further out of focus.

  Further, from Equation 5 and Equation 6, the following equation is obtained.

gd (x, y) = g '(x, y) * R (z, fo, fno, x, y) * k (z, fo +? fo, fno, x, y, z +? z) ... (Equation 7) )
Then, as shown in the following formula, Rk (z, fo, fno, x, y, z, Δfo, Δz) is a focus shift filter, and a desired focus position z at each coordinate position of the captured image and By providing the focus shift amount Δz, it is possible to perform focus adjustment of a desired amount from the original image.

gd (x, y) = g '(x, y) * Rk (z, fo, fno, x, y, z, Δfo, Δz)
... (Equation 8)
However, the problem here is that the image restoration filter R necessary for depth control and the point spread function k are two-dimensional map information, which further zoom, focus, image height, f-number, and more. Is a multi-dimensional function including the depth direction. Because the amount of data becomes enormous, interchangeable lenses and imaging devices hold data and perform processing on the imaging device, as well as moving images to a PC for processing Holding information is an issue.

  Therefore, the present inventor has changed the idea that if the data required to cope with all the shooting patterns is held, it is overwhelmingly less system load to calculate the necessary data for each shot image each time. It came to

  Specifically, only the RDN information of the optical system is held, the F value and the depth information acquired by the imaging device are added thereto, and the point spread function k and the image restoration filter R are calculated and generated. .

  The point image intensity distribution function represents the intensity distribution as a result of imaging each surface of an optical system on a single image according to Snell's law, one light from each of the object points. Because of the relatively simple calculation loop, it is possible.

  Since the image restoration filter is also a Fourier transform of the point spread function, it can also be calculated.

  It has been found that depth control is possible with very light data.

  Hereinafter, specific configurations for specifically implementing the present invention will be described. FIG. 2A shows a method of measuring z by imaging surface phase difference ranging. FIG. 2B shows a method of measuring the defocus position by reading the contrast value while focusing z. FIG. 2C shows an example in which focus adjustment is performed based on RDN data from the photographed image and defocus information (depth map) obtained as shown in FIGS. 2A and 2B. FIG. 2C may be performed on the imaging device which has performed FIGS. 2A and 2B, or may be performed by external software.

  FIG. 3A shows a depth map obtained by imaging surface phase difference ranging. FIG. 3B shows a process of performing autofocus and obtaining a depth map by reading a contrast value while focusing.

  First, a method of measuring z by imaging surface phase difference ranging will be described with reference to FIG. 2A. When a photographing execution instruction is given in step 101, the process moves to distance measurement point phase difference information acquisition in step 102. The defocus state is measured at a focus detection point selected by the user in advance and desired to be in focus. It may be so-called external phase difference ranging in which the pupil is separated by entering the autofocusing optical system using a mirror, or imaging plane phase difference ranging disposed on the imaging surface.

  Next, parameter acquisition of the optical system in step 103 is performed. Specifically, the zoom position, focus position, etc. of the optical system are read.

  In step 104, optical information specific to the optical position is used to determine the amount of focus drive necessary to focus the optical system and drive is performed. In step 105, distance measurement after driving is performed again.

  In step 106, it is judged whether or not the distance measurement result in step 105 is the in-focus state, and in the case of out-of-focus, the process returns to acquiring the parameters of the optical system 103 again and enters the focusing loop.

  In the case of focus determination, acquisition of imaging surface phase difference information for creating a depth map is performed in step 109 (step 107).

  Next, in step 110, image signal acquisition (step 108) for creating a photographed image is performed.

  Also, in step 109, the RDN of the optical system is acquired.

  The obtained depth map, photographed image and RDN information are stored at step 112.

  Here, a flow from acquisition of imaging surface phase difference information in step 107 to creation of a depth map in step 109 will be described with reference to FIG. 3A. Innumerable rectangles shown in FIG. 3A are points at which imaging surface distance measurement is performed. In practice, the pupil separation means is disposed at the pixel located at the center of the rectangular area, but the detailed description is as shown in Reference 3.

  The depth map is obtained by mapping the defocus amount obtained from each as shown in FIG. 3B. Now, the defocus of subject B in the in-focus state is 0, and the subject A in the front pin state is a negative defocus amount, and the subject C in the back pin state is deeper than the positive defocus amount and subject C The background D in the image is an even larger positive defocus amount. With this depth map, information of (x, y, z) is obtained, and it is possible to create a focus adjustment filter.

  Next, a method of obtaining the contrast map by reading the contrast value while focusing driving the depth map will be described with reference to FIGS. 2B and 3B. The current focus position is the position of s in FIG. 3B, and the subject to be focused is the subject B.

  As soon as the photographing command in step 201 is lowered, acquisition of the contrast value in step 202 is started. Specifically, the sharpness of the edge portion of the subject is calculated.

  In the initial focus drive of step 203, the focus drive is moved to an infinite side or close side in an arbitrary direction, and the process proceeds to the drive direction determination of step 204.

  If the contrast value goes up in step 204, the initial drive direction is the correct drive direction, and if the contrast value goes down, the drive direction is opposite, so it needs to be reversed. In FIG. 3B, when the initial driving direction is negative, the contrast value of the subject B is lowered, so that the image is reversed in the positive direction.

  In step 205, focus drive is performed in the direction determined in step 204. In FIG. 3B, drive in the positive direction from s.

  Next, in step 206, overrun determination is performed.

  In FIG. 3B, it is determined that the peak is passed at a position t where the contrast peak position u of the subject B is passed and it is determined that the contrast is clearly lowered.

  Next, at step 207, a drive amount for returning the overrun is determined. Specifically, the drive amount of ut is obtained.

  Next, in step 208, reverse drive is performed to stop. Here, the acquisition of the contrast value is ended (step 209).

  Next, in step 212, image signal acquisition (step 210) for creating a captured image is performed.

  Next, parameter acquisition of the optical system in step 211 is performed. Specifically, the zoom position, focus position, etc. of the optical system are read.

  Next, in step 213, based on the contrast value and the parameters of the optical system in step 211, a depth map is created.

  Further, in step 214, the RDN of the optical system is acquired. Finally, the obtained depth map, photographed image and RDN are saved in step 215.

  Here, the depth map generation method in step 213 will be specifically described. After starting the overrun in step 205, the contrast in each subject is acquired from s to t in FIG. Therefore, for example, in the subject A and the subject C, since the contrast peaks exist between s and t, the contrast peak positions can be determined as w and v, respectively.

  From the position of the optical system in step 211, the image plane movement amount S per focus pulse can be obtained, and the defocus amount of each subject can be calculated. In the case of the subject A, S × (w−u), and in the case of the subject C, S × (v−u). Thus, as in the case of imaging surface phase difference ranging, a depth map can be created for the entire imaging surface.

  Next, a flow of performing focus adjustment from the depth map and the photographed image obtained by the imaging surface phase difference distance measurement or the method of acquiring the contrast value while focusing will be described with reference to FIG. 2C. Although this processing may be performed by a photographing device or external calculation software, it is possible to read the previously obtained image and depth map, and the point spread function k (z, fo of the photographing optical system) , Fno, x, y, z) and the image restoration filter R (z, fo, fno, x, y, z).

  First, when a display instruction of a photographed image is given by the user in step 301, the image is displayed in step 302. Here, if it is determined that the subject is out of focus by the user in step 303, the process proceeds to adjustment value input in step 304. When an adjustment value is input from the user, a focus adjustment filter is created in step 305, and focus adjustment processing is performed in step 306. At step 307, the focus adjustment result is displayed, and the process returns to step 303.

  If it is determined in step 303 that focus adjustment is unnecessary, the process ends in step 308.

  Next, a method for obtaining a better image in the present embodiment will be described. In Expression 5, when the plane projection original image is obtained, an image in which the aberration of the optical system is satisfactorily corrected is obtained by the recovery filter R (z, fo, fno, x, y, z).

  In equation 7, the point spread function k (z, fo + Δfo, fno, x, y, z + Δz) of the optical system in a state in which the focus position and the defocus are shifted is given.

  At this time, even at the best focus position after Δz adjustment, the aberration of the optical system is given again, and some image deterioration occurs.

  Therefore, in the present embodiment, a point projection intensity distribution function is not given in the vicinity of the best focus after the Δz adjustment, and the plane projection original image itself is obtained, so that the focus adjustment image in which the vicinity of the best focus is sharp succeeds. .

  Further, in the present embodiment, not only the focus shift but also the f-number of the optical system can be given from the user, whereby the depth can be adjusted. In Equation 7, the f-number of the optical system at the time of shooting is given to fno of the image restoration filter R (z, fo, fno, x, y, z), and the point spread function k (z, fo + Δfo) of the optical system , Fno, x, y, z + Δz) give the desired F-number to which depth adjustment is desired.

  As mentioned above, although the preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

A, B, C, ... A subject with depth

Claims (12)

A means for detecting the defocus state of a plurality of points at the time of shooting;
A means for storing information on the radius of curvature, surface spacing, and refractive index of the optical system;
A unit for calculating and generating an optical filter necessary for depth control based on the information;
Means for processing a captured image from the optical filter;
An imaging system comprising:   The imaging system according to claim 1, wherein the optical filter is generated based on an externally input focus shift amount.   The imaging system according to claim 1, wherein the optical filter is generated based on an externally input blur amount.   The imaging system according to any one of claims 1 to 3, wherein effective diameter information of the optical system is also stored. A means for detecting the defocus state of a plurality of points at the time of shooting;
A means for computing and generating an optical filter necessary for depth control based on information on the radius of curvature, surface spacing, and refractive index of the optical system;
Means for processing a captured image from the optical filter;
Removable to an imaging device having
A lens apparatus characterized by comprising means for storing the information.   The lens apparatus according to claim 5, wherein the optical filter is generated based on an externally input focus shift amount.   The lens apparatus according to claim 5, wherein the optical filter is generated based on an externally input blur amount.   The lens apparatus according to any one of claims 5 to 7, wherein effective diameter information of the optical system is also stored. A means for detecting the defocus state of a plurality of points at the time of shooting;
A means for storing information on the radius of curvature, surface spacing, and refractive index of the optical system;
A unit for computing and generating an optical filter necessary for depth control from a captured image obtained by an imaging apparatus having a unit for writing the information in the captured image;
Means for processing a captured image from the optical filter;
An image processing tool characterized by having.   The image processing tool according to claim 9, wherein the optical filter is generated based on an externally input focus shift amount.   The image processing tool according to claim 9, wherein the optical filter is generated based on an externally input blur amount.   The image processing tool according to any one of claims 9 to 11, wherein the effective diameter information of the optical system is also stored.