JP6590561B2 - Image processing apparatus, imaging apparatus, and image processing method - Google Patents

Description

  The present invention relates to an image processing device, an imaging device, and an image processing method.

  Conventionally, in a shooting scene such as a movie, focusing is performed manually by an expert called a focus man. Since Focusman performs focusing while watching the video displayed on a dedicated monitor, it is possible to shoot with detailed expressions. In recent years, video signals have been adapted to high resolution (number of pixels) such as 4K and 8K. However, it is difficult to confirm a high-definition resolution image on a monitor for photographing. Therefore, the Nyquist frequency differs greatly between the recorded video signal and the video signal of the photographing monitor.

  Further, the lens has spherical aberration, and the image plane position where the contrast is highest differs for each frequency. Therefore, if focus adjustment is performed on a monitor with a low Nyquist frequency, a recorded video signal with a high Nyquist frequency will be out of focus.

  Therefore, Patent Document 1 discloses a method of performing high-frequency contrast detection called peaking detection using a recording resolution signal and displaying the detection result graphically superimposed on an image having a display resolution. Patent Document 2 discloses a technique for reconstructing an image by performing a refocus process based on a plurality of parallax images obtained by arranging a plurality of pixels with respect to one microlens.

JP 2013-201703 A JP 2014-64213 A

  However, in Patent Document 1, since the peaking image is superimposed on the display resolution image, the visibility of the image is deteriorated. In Patent Document 2, the amount of data to be recorded becomes enormous, and focus adjustment must be performed after recording.

  In view of such a problem, the present invention provides an image processing apparatus, an imaging apparatus, and an image processing method capable of performing appropriate focus confirmation using a monitor having a resolution lower than the resolution of an image to be recorded. is there.

An imaging apparatus according to one aspect of the present invention includes a plurality of pixels each including a plurality of photoelectric conversion units, and a plurality of images having different resolutions using signals obtained from an imaging element that receives subject light from an imaging optical system. an image processing apparatus for generating a video signal, a recording image generating means for generating an image to be recorded in the recording unit using the signal, have rows refocus processing on the image obtained by using the signal, the display Display image generating means for generating an image to be displayed on the unit , wherein the display image generating means is configured to record an image to be recorded on the recording unit and an image to be displayed on the display unit based on a state of the imaging optical system. The refocus processing is performed with a parameter that reduces a shift in focus between an image recorded in the recording unit and an image displayed on the display unit due to a difference in resolution .

An image processing method according to another aspect of the present invention includes a plurality of pixels each including a plurality of photoelectric conversion units, and uses a signal obtained from an image sensor that receives subject light from an imaging optical system, An image processing method for generating video signals of a plurality of different images, comprising: generating an image to be stored in a recording unit using the signal; performing refocus processing on an image obtained using the signal; Generating an image to be displayed on the display unit, and generating the image to be displayed on the display unit includes: an image to be recorded on the recording unit and a display unit based on a state of the imaging optical system. Patent that the parameter so as to reduce the deviation of the focus of an image to be displayed on the image and the display unit to be recorded in the recording unit due to the difference in the resolution of the image to be displayed performing the refocus processing To.

  According to the present invention, it is possible to provide an image processing apparatus, an imaging apparatus, and an image processing method capable of performing appropriate focus confirmation using a monitor having a resolution lower than the resolution of a recorded image.

1 is a block diagram of an imaging apparatus according to Embodiment 1. FIG. It is a figure which shows the pixel structure of an image pick-up element. It is explanatory drawing of a refocus process. It is a figure which shows the influence of spherical aberration. It is a figure which shows lens characteristic information. It is a figure which shows an imaging device and a display apparatus. 6 is a block diagram of an image pickup apparatus according to Embodiment 2. FIG. It is a flowchart of a display part frequency characteristic detection. 6 is a block diagram of an image pickup apparatus according to Embodiment 3. FIG. It is a flowchart of an amount-of-focus detection.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings. In each figure, the same members are denoted by the same reference numerals, and redundant description is omitted.

  FIG. 1 is a block diagram of the imaging apparatus of the present embodiment. The imaging optical system 101 includes a focus lens and guides subject light to the imaging element. Each pixel of the image sensor 102 is provided with a microlens and a plurality of photoelectric conversion units that convert subject light into electrical signals. The mixing unit 103 adds signals output from the plurality of photoelectric conversion units in units of microlenses. The signal processing unit 104 converts the signal output from the mixing unit 103 into a video signal. The encode recording unit 105 encodes and records the video signal output from the signal processing unit 104. The reconstruction unit 106 performs refocus processing for virtually changing the imaging plane (imaging plane, in-focus plane, focal plane) by rearranging the arrangement of signals from the imaging element 102. The mixing unit 107 adds the signals output from the reconstruction unit 106 in units of microlenses. The resizing unit 108 converts the signal output from the mixing unit 107 into a reduced image signal. The signal processing unit 109 converts the signal output from the resizing unit 108 into a video signal. The display unit 110 displays the video signal output from the signal processing unit 109 as an image. The lens characteristic information unit 112 stores lens characteristic information. The refocus parameter determination unit 111 acquires the state of the imaging optical system 101 and determines a parameter for refocus processing based on the lens characteristic information in the lens characteristic information unit 112. The refocus parameter determination unit 111 and the lens characteristic information unit 112 function as an image processing apparatus according to this embodiment.

  FIG. 2 is a diagram illustrating the structure of one pixel of the image sensor 102. The micro lens 205 is provided in each pixel of the image sensor 102. Four photoelectric conversion units 201 to 204 corresponding to the microlens 205 receive light that has passed through the microlens 205. With such a structure, light incident on the microlens 205 is photoelectrically converted for each of four incident angles.

  Next, the principle of the refocus process will be described with reference to FIG. FIG. 3 shows some pixels of the image sensor. Each pixel is provided with microlenses P1 to P4. On the imaging surface 301, the photoelectric conversion units A1 and B1 are arranged in the pixel provided with the microlens P1. Photoelectric conversion units A2 and B2 are arranged in the pixel provided with the microlens P2. Photoelectric conversion units A3 and B3 are arranged in the pixel provided with the microlens P3. Photoelectric conversion units A4 and B4 are arranged in the pixel provided with the microlens P4. The signal added with the output from each pixel is converted into a video signal.

  Here, a description will be given of the image plane 301 and light rays incident on the refocus plane 303 that is separated from the image plane 301 by a focal plane distance 302. On the imaging surface 301, the light beam 304 incident on the pixel 307 enters the photoelectric conversion unit B2, and the light beam 305 enters the photoelectric conversion unit A2. On the refocus plane 303, the light ray 304 enters the pixel 308. Further, a light beam 306 incident on the photoelectric conversion unit A4 on the imaging surface 301 enters the pixel 308. That is, the light rays 304 and 305 are incident on the same pixel on the imaging surface 301, but the light rays 304 and 306 are incident on the same pixel on the refocus surface 303. Therefore, the same effect as moving the imaging surface back and forth can be obtained by configuring a plurality of photoelectric conversion units for one microlens and changing the combination thereof. In this embodiment, the reconstruction unit 106 performs such reconstruction.

  FIG. 4 is a diagram illustrating the influence of spherical aberration when a subject is photographed. The horizontal axis represents the position of the focus lens, and the vertical axis represents the contrast value of the subject image projected on the image sensor. The position of the focus lens with the highest contrast value is the position in focus. FIG. 4 shows a case where the video signal 402 output to the monitor is full HD (High Definition), but the monitor image signal 403 does not have full HD resolution (number of pixels) due to the number of pixels and the like. . At this time, the focus lens position at which the contrast value of the monitor image signal 403 peaks differs from the focus lens position at which the contrast value of the 4K video recording signal 401 reaches a peak by an amount of focus deviation 404. .

  Next, a method for eliminating the shift amount of the focus surface position between the monitor image signal and the recording signal by refocusing will be described.

  FIG. 5A is a graph obtained by converting the characteristics shown in FIG. 4 into a curve 501 showing the change of the focus surface position for each frequency. The horizontal axis represents the spatial frequency, and the vertical axis represents the focus plane position. In FIG. 5A, the focus surface position 502 in the 4K video recording signal 401 is set as a reference position. The focus plane position 503 in the monitor image signal 403 is different from the focus plane position 502 by a difference 504. A difference 504 is a shift amount of the focus surface position that must be corrected by refocusing.

  FIG. 5B shows the axis on which the curve of FIG. 5A depends, and shows the lens characteristic information output from the lens characteristic information unit 112. FIG. 5A shows the relationship between the spatial frequency and the amount of focus shift, but FIG. 5B further changes with the image height and the lens state. The state of the lens includes an aperture, a focus lens position, a zoom lens position, and the like. When these change, the curve in FIG. 5A also changes accordingly. In the present embodiment, the refocus parameter determination unit 111 acquires the state of the imaging optical system 101, determines a refocus processing parameter based on information from the lens characteristic information unit 112, and outputs the parameter to the reconstruction unit 106.

  As described above, in the present embodiment, the output of the image sensor 102 is output to the mixing unit 103 and the reconstruction unit 106. The signal output to the mixing unit 103 is converted into a 4K video signal by the signal processing unit 104 and recorded in the encode recording unit 105. Since the signal from the photoelectric conversion unit is added for each pixel in the mixing unit 103 as the image to be recorded, the image has exactly the same quality as the image obtained by the normal pixel that is not the pixel having the structure of FIG. is there.

  The signal output to the reconstruction unit 106 is reconstructed in accordance with the refocus parameter determined by the refocus parameter determination unit 111, and the focus plane shift caused by spherical aberration is corrected. Then, it is added for each microlens by the mixing unit 107 and converted into an HD size image by the resizing unit 108. Then, it is converted into a video signal as an HD signal by the signal processing unit 109 and displayed on the display unit 110.

  In this way, it is possible to perform correct focus adjustment by performing focusing while viewing the display based on the video signal whose focus surface position is corrected. In other words, it is possible to reduce a focus shift in a monitor having a low Nyquist frequency and a focus difference in a video signal to be recorded having a high Nyquist frequency.

  In the first embodiment, the description has been made on the assumption that the resolution of the monitor is known. In the present embodiment, a function for measuring the resolution of the monitor will be described.

  FIG. 6 is a diagram illustrating a state in which a video signal output from the imaging apparatus according to the present exemplary embodiment is displayed on a monitor, and is captured and calibrated by the imaging apparatus. FIG. 7 is a block diagram of the imaging apparatus of the present embodiment. Unlike the imaging apparatus of the first embodiment, the imaging apparatus of the present embodiment includes an amplitude detection unit (detection unit) 701, a display unit frequency characteristic detection unit (acquisition unit) 702, and a video signal generation unit (output unit) 703. It has a calibration means. Other configurations are the same as those of the image pickup apparatus according to the first embodiment, and thus detailed description thereof is omitted. The display unit frequency characteristic detection unit 702 generates a vertical line image using the video signal generation unit 703 and outputs the vertical line image to the display unit 110. The image displayed on the display unit 110 is captured by the image sensor 102, and the amplitude of the video signal output from the image sensor 102 is detected by the amplitude detector 701.

  FIG. 8 is a flowchart of detection of the display unit frequency characteristics. The display unit frequency characteristic detection is performed by a microcomputer inside the display unit frequency characteristic detection unit 702. When the display unit frequency characteristic detection starts in step S801, an initial frequency is set in step S802. The initial frequency is a sufficiently low frequency. In step S <b> 803, a vertical line video signal that is a sine wave having a set frequency is displayed on the display unit 110 using the video signal generation unit 703. In step S804, the amplitude of the video signal output from the image sensor 102 from the amplitude detector 701 is acquired. In step S805, it is determined whether the set frequency has reached the Nyquist frequency. If not reached, the process proceeds to step S806, and if reached, the process proceeds to step S807. In step S806, the frequency is changed, and the amplitude of the video signal at each frequency is detected in step S804 until the changed frequency reaches the Nyquist frequency. In step S807, what frequency is displayed from the acquired amplitude is analyzed, and then the display unit frequency characteristic detection is terminated in step S808.

  In the present embodiment, the frequency characteristic of the display device is acquired by the above method, and the refocus parameter determination unit 111 determines the refocus parameter based on the frequency characteristic.

  In the present embodiment, calibration is performed by a method different from that in the second embodiment. FIG. 9 is a block diagram of the imaging apparatus of the present embodiment. Unlike the imaging apparatus of the first embodiment, the imaging apparatus of the present embodiment includes a calibration unit including an amplitude detection unit (detection unit) 901 and a focus deviation amount detection unit (calculation unit) 902. Other configurations are the same as those of the image pickup apparatus according to the first embodiment, and thus detailed description thereof is omitted. In this embodiment, the photographer performs focusing based on the displayed image, and the calibration function is started by the operation member when the focus adjustment can be performed (the focus surface position is determined).

  FIG. 10 is a flowchart of focus amount detection. The focus shift amount detection is performed by a microcomputer inside the focus shift amount detection unit 902. When focus detection starts in step S1001, a parameter for refocus processing is set on the front pin side of the refocus limit in step S1002. Based on the set parameters, a signal refocused by the reconstruction unit 106 and added by the mixing unit 107 is input to the amplitude detection unit 901. In step S1003, the signal output from the amplitude detector 901 is acquired. In step S1004, it is determined whether the refocus position is on the rear pin side of the refocus limit. If it is not the rear pin side, the process proceeds to step S1005, and if it is the rear pin side, the process proceeds to step S1006. In step S1005, the refocus position is changed, and the amplitude of the signal at each refocus position is detected until the changed refocus position is on the rear pin side. In step S1006, the refocus position that is in focus is detected from the acquired amplitude. In step S1007, the difference between the focus surface position adjusted by the photographer and the refocus position detected in step S1006 is calculated (the lens information is reversed), which is equivalent to the frequency characteristic of the display device. Information can be obtained. As a result, the focus adjustment result of the photographer and the 4K resolution focus surface position coincide with each other.

  In addition, this invention is implement | achieved also by performing the following processes. Processing for supplying software (program) for realizing the functions of each embodiment to a system or apparatus via a network or various recording media, and reading and executing the program by a computer (or CPU, MPU, etc.) of the system or apparatus It is.

  As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

DESCRIPTION OF SYMBOLS 101 Image pick-up optical system 102 Image pick-up element 106 Reconstruction part (processing means)
111 Refocus parameter determination unit (determination means)

Claims (11)

An image processing apparatus that includes a plurality of pixels each including a plurality of photoelectric conversion units and generates video signals of a plurality of images having different resolutions using signals obtained from an image sensor that receives subject light from an imaging optical system. And
Recorded image generating means for generating an image to be recorded in the recording unit using the signal;
There line refocus processing on the image obtained by using the signal, has a display image generating means for generating an image to be displayed on the display unit,
The display image generation unit includes: an image to be recorded on the recording unit due to a difference in resolution between an image to be recorded on the recording unit and an image to be displayed on the display unit based on a state of the imaging optical system; An image processing apparatus that performs the refocus processing with a parameter that reduces a focus shift of an image displayed on the screen .   The image processing apparatus according to claim 1, wherein the state of the imaging optical system is at least one of a diaphragm, a position of a focus lens, and a position of a zoom lens. It further has an acquisition means for acquiring the characteristics of the display device,
Based on the characteristics of the prior Symbol display device, an image processing apparatus according to claim 1 or 2 wherein the parameters of the refocus process is determined, characterized in Rukoto. Output means for outputting a first video signal to the display device;
Detecting means for detecting an amplitude of a second video signal acquired from the first video signal displayed on the display device;
The image processing apparatus according to claim 3, further comprising an acquisition unit configured to acquire a characteristic of the display device based on a detection result of the detection unit. Detecting means for detecting the amplitude of the video signal refocused by the display image generating means at different refocus positions;
The image processing apparatus according to claim 3, further comprising a calculation unit that calculates a difference between a focus surface position adjusted by a photographer and a focus surface position based on a detection result of the detection unit. Based on the previous SL difference, the image processing apparatus according to claim 5 wherein the parameters of the refocus process is determined, characterized in Rukoto.   7. The image processing apparatus according to claim 1, wherein the image processing device reduces a shift in focus between a monitor with a low Nyquist frequency and a recorded video signal with a high Nyquist frequency. Image processing apparatus.   An imaging apparatus comprising the image processing apparatus according to claim 1. An image processing method that includes a plurality of pixels each including a plurality of photoelectric conversion units and generates video signals of a plurality of images having different resolutions using signals obtained from an image sensor that receives subject light from an imaging optical system. And
Generating an image to be stored in the recording unit using the signal;
Performs refocus processing on the image obtained by using the previous SL signal, comprising the steps of generating an image to be displayed on the display unit, and
The step of generating an image to be displayed on the display unit is recorded on the recording unit due to a difference in resolution between the image recorded on the recording unit and the image displayed on the display unit based on the state of the imaging optical system. An image processing method , wherein the refocus processing is performed with a parameter that reduces a shift in focus between an image to be displayed and an image to be displayed on the display unit .   A program for causing a computer to function as each unit of the image processing apparatus according to any one of claims 1 to 7.   The computer-readable recording medium which recorded the program of Claim 10.