WO2022024671A1 - 撮像装置、ピント位置設定方法、プログラム - Google Patents

撮像装置、ピント位置設定方法、プログラム Download PDF

Info

Publication number
WO2022024671A1
WO2022024671A1 PCT/JP2021/025190 JP2021025190W WO2022024671A1 WO 2022024671 A1 WO2022024671 A1 WO 2022024671A1 JP 2021025190 W JP2021025190 W JP 2021025190W WO 2022024671 A1 WO2022024671 A1 WO 2022024671A1
Authority
WO
WIPO (PCT)
Prior art keywords
focus
subject
focus position
target
user
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2021/025190
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
誠一 猪俣
娜 許
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Group Corp
Original Assignee
Sony Group Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Group Corp filed Critical Sony Group Corp
Priority to US18/005,637 priority Critical patent/US12401896B2/en
Priority to JP2022540102A priority patent/JP7687337B2/ja
Priority to EP21850127.8A priority patent/EP4191331B1/en
Publication of WO2022024671A1 publication Critical patent/WO2022024671A1/ja
Anticipated expiration legal-status Critical
Priority to JP2025085626A priority patent/JP2025116032A/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/61—Control of cameras or camera modules based on recognised objects
    • H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
    • G02B7/02—Mountings, adjusting means, or light-tight connections, for optical elements for lenses
    • G02B7/04—Mountings, adjusting means, or light-tight connections, for optical elements for lenses with mechanism for focusing or varying magnification
    • G02B7/08—Mountings, adjusting means, or light-tight connections, for optical elements for lenses with mechanism for focusing or varying magnification adapted to co-operate with a remote control mechanism
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
    • G02B7/28—Systems for automatic generation of focusing signals
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B13/00—Viewfinders; Focusing aids for cameras; Means for focusing for cameras; Autofocus systems for cameras
    • G03B13/32—Means for focusing
    • G03B13/34—Power focusing
    • G03B13/36—Autofocus systems
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B15/00—Special procedures for taking photographs; Apparatus therefor
    • G03B15/02—Illuminating scene
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B17/00—Details of cameras or camera bodies; Accessories therefor
    • G03B17/18—Signals indicating condition of a camera member or suitability of light
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
    • H04N23/632—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters for displaying or modifying preview images prior to image capturing, e.g. variety of image resolutions or capturing parameters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/63—Control of cameras or camera modules by using electronic viewfinders
    • H04N23/633—Control of cameras or camera modules by using electronic viewfinders for displaying additional information relating to control or operation of the camera
    • H04N23/635—Region indicators; Field of view indicators
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/672—Focus control based on electronic image sensor signals based on the phase difference signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/675—Focus control based on electronic image sensor signals comprising setting of focusing regions
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B27/00—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00
    • G02B27/0093—Optical systems or apparatus not provided for by any of the groups G02B1/00 - G02B26/00, G02B30/00 with means for monitoring data relating to the user, e.g. head-tracking, eye-tracking
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G06T2207/20221—Image fusion; Image merging

Definitions

  • This technology relates to an image pickup device, a focus position setting method, and a program related to automatic focus control and manual focus control.
  • Patent Document 1 proposes a method of determining a main subject with high accuracy in order to keep focusing on a subject intended by a user.
  • the image pickup apparatus includes a user operation detection unit that detects a user's operation for focus control, a focus position movement detection unit that detects a movement of the focus position based on the user's operation detected by the user operation detection unit, and a focus position movement detection unit. It includes a depth information detection unit that detects depth information in a focus control target area, and a focus control unit that sets a target focus position based on the movement of the focus position and the depth information. That is, the change in the focus position based on the user operation is detected, and after the user operation is completed, the target focus position and the focus control that reflect the user operation are performed.
  • the "focus position” referred to in the present disclosure is a position in the optical axis direction that is in focus from the image pickup device, and is the position of the subject with respect to the image pickup device when there is a subject that is in focus.
  • the "target focus position” is a target position that changes the focus position, and in many cases, refers to a position in the depth direction of the subject that is the target of focus control.
  • the depth direction refers to the depth direction from the image pickup device to the subject side.
  • the focus position movement detection unit in the above-mentioned image pickup apparatus may detect the movement direction of the focus position. By detecting the moving direction of the focus position, the user's intention can be estimated more accurately.
  • the focus position movement detection unit in the above-mentioned image pickup apparatus may detect the movement direction at predetermined time intervals. By detecting the moving direction at predetermined time intervals, the operation direction of the user is appropriately detected.
  • the depth information detection unit in the image pickup apparatus described above detects the depth information of the entire focus control target area, and the focus control unit determines the depth information of the entire focus control target area, the movement direction of the focus position, and the like.
  • the target focus position may be set based on the end focus position when it is detected that the user's operation regarding the movement of the focus position is completed. By setting the target focus position based on the focus position at the end, the user's intention is reflected in the focus control.
  • the image pickup apparatus described above includes an in-screen target position setting unit that sets an in-screen target position based on the depth information, and the focus control unit sets a focus position corresponding to the in-screen target position to the target focus position. It may be set as.
  • the target position in the screen is an area set on the captured image, and is an area composed of a group of pixels having similar depth information. For example, the pixel area in which a certain subject is captured is set as a target position in the screen because the depth information for each pixel is similar.
  • the focus control unit in the above-mentioned image pickup apparatus sets the focus position at the end of the focus positions corresponding to the in-screen target positions.
  • the closest focus position may be set as the target focus position.
  • the image pickup apparatus described above includes a subject recognition unit that recognizes a subject, and the focus control unit is an in-screen subject that is a target position in the screen and a position in the screen of the subject recognized by the subject recognition unit.
  • the focus position corresponding to the subject position in the screen may be set as the target focus position.
  • a subject as a focus control target is selected by using image recognition processing or the like.
  • the focus control unit in the image pickup apparatus describes the focus position closest to the end focus position among the focus positions corresponding to the subject positions in the screen. It may be set as a target focus position. As a result, an appropriate subject can be selected as the focus control target, and a subject for which the user's intention is determined can be selected as the focus control target.
  • the focus control unit in the image pickup apparatus may perform tracking focus control on the subject focused according to the target focus position set after the movement of the focus position is completed. As a result, the user does not have to manually keep focusing on the focused subject.
  • the image pickup apparatus described above may include an area setting unit that sets the focus control target area according to the operation of the user. This makes it possible to select the focus control target from the subject by reflecting the user's intention more.
  • a process of superimposing a first frame image on a subject focused by the focus control unit among the subjects recognized by the subject recognition unit and a second frame image on other subjects are displayed. It may be provided with a display control unit which performs a process of superimposing and displaying. As a result, the user can confirm the focus control target by confirming the display unit.
  • the focus control unit in the above-mentioned image pickup apparatus may perform focus control according to a set focus movement speed.
  • the speed (time) until focusing in focus control can be specified simply by changing the operation mode.
  • the user operation detection unit in the image pickup apparatus may detect the user's operation on the focus ring, and the focus position movement detection unit may detect the movement direction of the focus position based on the operation direction of the focus ring. good. Further, the user operation detection unit in the above-mentioned imaging device detects the user's focus operation by receiving the operation information transmitted from the external operation device, and the focus position movement detection unit detects the operation direction of the focus operation. The moving direction of the focus position may be detected based on the above. As a result, the moving direction of the focus position is appropriately detected.
  • the focus control unit in the above-mentioned image pickup apparatus can switch between a first mode in which autofocus control is performed on a subject and a second mode in which manual focus control is performed.
  • the user operation detection unit is said to be capable of switching.
  • the mode may be switched to the second mode. As a result, the user only has to operate the focus ring when switching to the second mode.
  • the user operation detection unit in the above-mentioned image pickup apparatus detects the end of the movement of the focus position when the operation of the user changes from the state where the operation for the focus ring is detected to the state where the operation for the focus ring is not detected. It may be determined that it has been done. This makes it possible to detect the end of the user operation on the focus ring using, for example, a sensor or the like.
  • the focus control unit in the image pickup apparatus described is the first control for setting the focus position closest to the focus position at the end of the focus positions corresponding to the target positions in the screen as the target focus position, and the end of the movement.
  • the second control for setting the target focus position according to the current focus position and the moving direction is enabled, and the deviation between the closest focus position and the end focus position is equal to or greater than a predetermined threshold value.
  • the first control may be switched to the second control. As a result, an appropriate selection is made according to the user's operation mode and the position of the subject.
  • the focus position setting method detects a user's operation for focus control, detects a movement of the focus position based on the detected user's operation, detects depth information in a focus control target area, and detects the focus.
  • the target focus position is set based on the movement of the position and the depth information.
  • the program according to the present technology includes a process of detecting a user's operation on focus control, a process of detecting a movement of a focus position based on the detected user's operation, and a process of detecting depth information in a focus control target area.
  • This is a program for causing an image pickup apparatus to execute a process of setting a target focus position based on the movement of the focus position and the depth information.
  • Configuration of image pickup device The appearance of the image pickup apparatus 1 according to this embodiment is shown in FIGS. 1 and 2.
  • the proximity or distance of the subject or the object of focus control may be described, but “close” means closer to the image pickup device 1, and “far” means from the image pickup device 1. Indicates farther. That is, the "close subject” means a subject closer to the image pickup apparatus 1.
  • the image pickup device 1 is an example provided with an interchangeable lens, but the present invention is not limited to this, and can be widely applied to, for example, various image pickup devices incorporated in a still camera, a video camera, or another device. can.
  • the image pickup apparatus 1 includes a camera housing 2 in which required parts are arranged inside and outside, and a lens barrel 3 attached to the front surface portion 2a of the camera housing 2.
  • a rear monitor 4 is arranged on the rear surface portion 2b of the camera housing 2.
  • a through image, a recorded image, or the like is displayed on the rear monitor 4.
  • the rear monitor 4 is a display device such as a liquid crystal display (LCD) or an organic EL (Electro-Luminescence) display, for example.
  • LCD liquid crystal display
  • organic EL Electro-Luminescence
  • the EVF (Electric ViewFinder) 5 is arranged on the upper surface portion 2c of the camera housing 2.
  • the EVF 5 includes an EVF monitor 5a and a frame-shaped enclosure 5b projecting rearward so as to surround the upper side and the left and right sides of the EVF monitor 5a.
  • the EVF monitor 5a is formed by using an LCD, an organic EL display, or the like.
  • An optical finder ODF: Optical ViewFinder
  • OVF Optical ViewFinder
  • Various controls 6 are provided on the rear surface portion 2b and the upper surface portion 2c.
  • Examples of the operator 6 include a playback menu start button, an enter button, a cross key, a cancel button, a zoom key, a slide key, a shutter button 6S (release button), and the like.
  • Various controls 6 include various modes such as buttons, dials, pressing and rotatable compound controls.
  • the controls 6 in various modes enable, for example, menu operation, playback operation, mode selection / switching operation, focus operation, zoom operation, and parameter selection / setting such as shutter speed and F-number. ..
  • the lens barrel 3 has various lenses arranged inside, and is configured to include a ring-shaped focus ring 7 and a ring-shaped zoom ring 8.
  • the focus ring 7 is rotatable in the circumferential direction, and the focus position can be moved in the optical axis direction by moving various lenses in the optical axis direction according to the rotation direction.
  • the "focus position" is a position in the optical axis direction in which the image pickup device 1 is in focus. This is, for example, the position of the subject with respect to the image pickup apparatus 1 when there is a subject in focus.
  • the focus position changes depending on the focus control. By rotating the focus ring 7, the focus position with respect to the image pickup apparatus 1 can be made closer or farther. Further, by rotating the focus ring 7, it is possible to realize manual focus control for manually adjusting the in-focus state.
  • the zoom ring 8 is rotatable in the circumferential direction, and various lenses move in the optical axis direction according to the rotation direction, so that zooming control can be performed manually.
  • FIG. 3 is a block diagram of the image pickup apparatus 1. Inside and outside the camera housing 2 and lens barrel 3 of the image pickup device 1, the lens system 9, the image sensor unit 10, the signal processing unit 11, the recording control unit 12, the display unit 13, the output unit 14, the operation unit 15, and the camera. A control unit 16, a memory unit 17, a driver unit 18, a sensor unit 19, and the like are provided. In addition to this, a power supply unit and the like are appropriately provided.
  • the lens system 9 includes various lenses such as an incident end lens, a zoom lens, a focus lens, and a condenser lens, and a lens or an iris (so that sensing is performed while the signal charge is not saturated and is within the dynamic range. It is equipped with a diaphragm mechanism that controls exposure by adjusting the aperture amount by the diaphragm) and a shutter unit such as a focal plane shutter. A part of the optical system components such as the lens system 9 may be provided in the camera housing 2.
  • the image pickup device unit 10 is, for example, a CCD (Charge Coupled Device) type or a CMOS (Complementary Metal-Oxide Semiconductor) type, and is configured to have a sensing element in which a plurality of pixels are two-dimensionally arranged to form a lens system 9. The exposure is controlled for the light from the subject incident through the CMOS.
  • CCD Charge Coupled Device
  • CMOS Complementary Metal-Oxide Semiconductor
  • the image sensor unit 10 includes a processing unit that performs, for example, CDS (Correlated Double Sampling) processing, AGC (Automatic Gain Control) processing, and A / D (Analog / Digital) conversion processing on the electric signal photoelectrically converted by the pixels. Has been done. Therefore, the image sensor unit 10 outputs the captured image signal as digital data to the signal processing unit 11 and the camera control unit 16.
  • CDS Correlated Double Sampling
  • AGC Automatic Gain Control
  • a / D Analog / Digital
  • the signal processing unit 11 is composed of, for example, a microcomputer specialized in digital signal processing such as a DSP (Digital Signal Processor), a microcomputer, or the like.
  • a microcomputer specialized in digital signal processing such as a DSP (Digital Signal Processor), a microcomputer, or the like.
  • the signal processing unit 11 includes various units for performing various signal processing on the digital signal (image pickup image signal) sent from the image pickup element unit 10.
  • processing such as correction processing between R, G, and B color channels, white balance correction, aberration correction, and shading correction is performed. Further, the signal processing unit 11 generates (separates) a luminance (Y) signal and a color (C) signal from the image data of R, G, and B, a YC generation process, a process of adjusting the luminance and the color, and a knee correction. And each process such as gamma correction.
  • the signal processing unit 11 performs conversion to the final output format by performing resolution conversion processing, codec processing for coding for recording and communication, and the like.
  • the image data converted into the final output format is stored in the memory unit 17. Further, by outputting the image data to the display unit 13, the image is displayed on the rear monitor 4 and the EVF monitor 5a. Further, by outputting from the external output terminal, it is displayed on a device such as a monitor provided outside the image pickup apparatus 1.
  • the recording control unit 12 is composed of, for example, a non-volatile memory, and functions as a storage means for storing an image file (content file) such as still image data or moving image data, attribute information of the image file, a thumbnail image, or the like.
  • the image file is stored in a format such as JPEG (Joint Photographic Experts Group), TIFF (Tagged Image File Format), GIF (Graphics Interchange Format), or the like.
  • the actual form of the recording control unit 12 can be considered in various ways.
  • the recording control unit 12 may be configured as a flash memory built in the image pickup device 1, or may be stored in a memory card (for example, a portable flash memory) that can be attached to and detached from the image pickup device 1 and the memory card. It may be composed of an access unit that performs access for reading. Further, it may be realized as an HDD (Hard Disk Drive) or the like as a form built in the image pickup apparatus 1.
  • HDD Hard Disk Drive
  • the display unit 13 executes processing for performing various displays on the imager.
  • the display unit 13 is, for example, a rear monitor 4 or an EVF monitor 5a.
  • the display unit 13 performs a process of displaying image data converted to an appropriate resolution input from the signal processing unit 11.
  • a so-called through image which is a captured image during the standby of the release, is displayed.
  • the display unit 13 realizes on the screen the display of various operation menus, icons, messages, etc. as a GUI (Graphical User Interface) based on the instruction from the camera control unit 16.
  • the display unit 13 can display a reproduced image of the image data read from the recording medium by the recording control unit 12.
  • both the EVF monitor 5a and the rear monitor 4 are provided, but the implementation of the present technology is not limited to such a configuration, and only one of the EVF monitor 5a and the rear monitor 4 is provided. May be provided, or either one or both of the EVF monitor 5a and the rear monitor 4 may be detachable.
  • the output unit 14 performs data communication and network communication with an external device by wire or wirelessly. For example, captured image data (still image file or moving image file) is transmitted to an external display device, recording device, playback device, or the like. Further, the output unit 14 may function as a network communication unit. For example, communication may be performed by various networks such as the Internet, a home network, and a LAN (Local Area Network), and various data may be transmitted and received to and from a server, a terminal, or the like on the network.
  • networks such as the Internet, a home network, and a LAN (Local Area Network)
  • the operation unit 15 includes not only the various controls 6 described above but also a rear monitor 4 that employs a touch panel method, and camera controls operation information according to various operations such as tap operation and swipe operation of the imager. Output to unit 16.
  • the operation unit 15 may function as a reception unit of an external operation device such as a remote controller that is separate from the image pickup device 1. Possible external operation devices include smartphones, tablets, Bluetooth (registered trademark) remote controllers, wired remote controllers, and wireless operation devices for focus operation.
  • the focus ring 7 for detecting the operation for manual focus control and the zoom ring 8 for detecting the operation for zooming control are one aspect of the operation unit 15.
  • the camera control unit 16 is composed of a microcomputer (arithmetic processing device) provided with a CPU (Central Processing Unit), and controls the image pickup device 1 in an integrated manner. For example, the shutter speed is controlled according to the operation of the imager, the signal processing unit 11 instructs various signal processing, the imaging operation and the recording operation are performed according to the user's operation, and the recorded image file is reproduced.
  • the camera control unit 16 switches various shooting modes and the like.
  • the various shooting modes include, for example, a still image shooting mode, a moving image shooting mode, and a continuous shooting mode for continuously acquiring still images.
  • the camera control unit 16 controls the user interface to enable the user to operate these functions.
  • the UI (User Interface) control performs a process of detecting an operation on each operator 6 provided in the image pickup apparatus 1, a display process on the rear monitor 4, an operation detection process, and the like.
  • the camera control unit 16 gives an instruction to the driver unit 18 in order to control various lenses included in the lens system 9. For example, a process of designating an aperture value in order to secure a necessary amount of light for AF (Auto Focus) control, an operation instruction of an aperture mechanism according to the aperture value, and the like are performed.
  • AF Auto Focus
  • the memory unit 17 stores information and the like used for processing executed by the camera control unit 16.
  • a ROM Read Only Memory
  • RAM Random Access Memory
  • flash memory and the like are comprehensively shown.
  • the memory unit 17 may be a memory area built in the microcomputer chip as the camera control unit 16, or may be configured by a separate memory chip.
  • Programs and the like used by the camera control unit 16 are stored in the ROM, flash memory, and the like of the memory unit 17.
  • ROM Read Only Memory
  • flash memory etc.
  • content files such as an OS (Operating System) for the CPU to control each part and an image file
  • application programs and firmware for various operations are stored.
  • the camera control unit 16 controls the entire image pickup apparatus 1 including the lens barrel 3 by executing the program.
  • the RAM of the memory unit 17 is used as a work area of the camera control unit 16 by temporarily storing data, programs, and the like used in various data processing executed by the CPU of the camera control unit 16.
  • the driver unit 18 is provided with, for example, a motor driver for the zoom lens drive motor, a motor driver for the focus lens drive motor, an aperture mechanism driver for the motor that drives the aperture mechanism, and the like. Each driver supplies a drive current to the corresponding drive motor in response to an instruction from the camera control unit 16.
  • the sensor unit 19 comprehensively shows various sensors mounted on the image pickup apparatus 1.
  • a position information sensor, an illuminance sensor, an acceleration sensor, and the like are mounted.
  • a sensor provided on the focus ring 7 or the zoom ring 8 for detecting the rotation direction of the focus ring 7 or the zoom ring 8 and detecting the amount of operation is one aspect of the sensor unit 19.
  • the camera control unit 16 executes various functions by executing the program stored in the memory unit 17. Each function of the camera control unit 16 will be described with reference to FIG.
  • the signal processing unit 11 may have a part of each function. Further, a part of each function may be realized by the cooperation of the camera control unit 16 and the signal processing unit 11.
  • the camera control unit 16 includes a user operation detection unit 31, a focus position movement detection unit 32, a depth information detection unit 33, an in-screen target position setting unit 34, a subject recognition unit 35, an area setting unit 36, a mode switching unit 37, and focus control. It has a function as a unit 38 and a display control unit 39.
  • the user operation detection unit 31 performs a process of detecting a user operation on the operation unit 15. Specifically, a process of detecting an operation of pressing the shutter button 6S, a process of rotating the focus ring 7, a process of detecting an operation of rotating the zoom ring 8, and the like are performed.
  • the user operation detection unit 31 also detects the operation amount and the operation mode. For example, the operation mode in which the shutter button 6S is half-pressed or the operation mode in which the shutter button 6S is fully pressed is distinguished and detected. Further, the user operation detection unit 31 detects the rotation direction and the rotation amount when the operation of rotating the focus ring 7 or the zoom ring 8 is detected.
  • the user operation detection unit 31 detects the start and end of the user operation for rotating the focus ring 7.
  • Various types of detection of the start of the user operation can be considered, for example, the start of the user operation is determined by detecting that the user touches the focus ring 7 by using a sensor or the like provided for the focus ring 7. ..
  • the detection of the end of the user operation can be considered in various ways, for example, when the operation on the focus ring 7 changes from the detected state to the non-detected state, it is determined that the user operation is completed.
  • the end of the user operation may be detected according to the output of the sensor that detects the contact of the human finger with the focus ring 7, or the end of the user operation may be detected when the operation is not performed for a certain period of time. It may be detected.
  • the user operation detection unit 31 may detect an operation by an external operation device. For example, when the operation unit 15 receives an operation signal from an external operation device such as a smartphone, a tablet, a Bluetooth remote controller, a wired remote controller, or a wireless operation device for focus operation, the user operation detection unit 31 operates those operations. May be recognized as a focus operation. In the following description, the focus operation on the focus ring 7 will be given as an example, but this technique can also be applied to these remote-controlled focus operations.
  • the focus position movement detection unit 32 has a focus position movement direction (focus position movement direction) and a movement distance (focus position movement amount) based on the operation direction and operation amount of the focus ring 7 detected by the user operation detection unit 31. Is detected (calculated).
  • Various modes of detecting the moving direction of the focus position can be considered.
  • the minute operation amount (or the difference in the rotation position) of the focus ring 7 detected for each frame (1/60 sec, 1/30 sec, etc.) is acquired, and the focus is based on a plurality of minute operation amounts.
  • the user's operation direction with respect to the ring 7, that is, the rotation direction of the focus ring 7 is determined.
  • the moving distance of the focus position can be obtained by integrating a minute amount of manipulation.
  • the operation amount may be acquired not every frame but every predetermined time.
  • the depth information detection unit 33 detects one depth information for each pixel region composed of, for example, a plurality of pixels.
  • the depth information is the depth information of the subject, and may be detected based on the phase difference signal output from the image plane phase difference pixel included in the image sensor unit 10, for example, or a depth sensor may be provided separately from the image sensor unit 10. It may be detected by receiving the reflected light of the reference light such as near infrared rays or by acquiring the distortion or intensity of the distribution of the reflected light. Depth information may be detected for each pixel. When the focus control target area FA described later is set, the depth information may be detected only for the focus control target area FA.
  • the in-screen target position setting unit 34 sets the in-screen target position PS based on the depth information.
  • the target position PS in the screen is an area set on the captured image, and is an area composed of a group of pixels having similar depth information. That is, the in-screen target position PS set by the in-screen target position setting unit 34 is an area that can be set for each subject, and is an area in which a subject that is a target candidate for focus control is captured.
  • Figure 5 shows an example of the target position PS on the screen. This is an example in which the in-screen target position PS1 having a depth value DP1 and the in-screen target position PS2 having a depth value DP2 are set on the captured image.
  • the subject recognition unit 35 performs a process of recognizing a subject in the captured image by performing image processing on the captured image.
  • the subject recognition unit 35 may be capable of executing a process of determining the type of the subject. This makes it possible to perform appropriate focus control that reflects the user's desire to perform focus control on a person and the user's desire to perform focus control on a cat.
  • the subject recognition unit 35 sets the position of the recognized subject on the captured image as the subject position PO in the screen.
  • the in-screen subject position PO is set to indicate the display position of the subject recognized by image recognition, unlike the in-screen target position PS set based on the depth information.
  • the area setting unit 36 sets the focus control target area FA.
  • the focus control target area FA is an area where autofocus control is performed. For example, when the central portion of the captured image is selected as the focus control target region FA, the focus control target is selected from the subjects captured in the central portion.
  • the focus control target area FA can be set for the entire captured image or a part of the captured image.
  • the focus control target area FA may be set according to the user's operation, or may be automatically set by automatically determining the situation by image recognition. Further, a plurality of focus control target area FAs may be set. Then, a priority may be given to those plurality of focus control target area FAs.
  • the mode switching unit 37 switches between various modes. For example, the mode switching unit 37 switches between an AF mode for performing autofocus control and an MF (Manual Focus) mode for performing manual focus control.
  • the switching process is executed, for example, when the user operation detection unit 31 detects that the user has operated the operator 6 for switching between the AF mode and the MF mode. Alternatively, the switching process may be executed by detecting the menu operation of the user.
  • the mode switching unit 37 switches between a plurality of types of AF modes.
  • the AF mode includes a tracking focus mode in which focus control is performed according to the movement of a subject set as a focus control target, and a single focus mode in which focus control is performed only once when the shutter button 6S is pressed halfway. These switchings are performed according to the user's menu operation.
  • the mode switching unit 37 detects a specific user operation during the execution of the AF mode and temporarily switches to the MF mode. Further, in the temporary MF mode, the end of a specific user operation is detected and the AF mode is switched again. That is, in the AF mode, when the user performs a specific operation, the transition to the MF mode is performed, and when the specific operation is completed, the transition to the AF mode is performed again.
  • the specific user operation is, for example, a rotation operation of the focus ring 7.
  • the mode By turning the focus ring 7 in the circumferential direction during the AF mode, the mode temporarily transitions to the MF mode.
  • the focus control unit 38 performs autofocus control according to each mode. For example, in the single focus mode, the lens drive control is performed so as to focus on the subject in the focus control target area. Further, in the tracking focus mode, lens drive control is performed to keep the focus control target in focus according to the movement of the focus control target (particularly, the movement in the optical axis direction). Further, when the AF mode is switched to the temporary MF mode by the above-mentioned specific user operation, the focus control unit 38 is changed according to the operation mode of the manual focus ring 7 by the user. Performs processing to estimate (subject). Then, when the specific user operation is completed and the AF mode is set, the control is performed to focus on the estimated new focus control target.
  • the tracking focus control is performed before the temporary MF mode
  • the tracking focus control is performed for the new focus control target when the temporary MF mode is returned to the AF mode again.
  • the process of estimating the new focus control target that has been changed may be executed after returning from the temporary MF mode to the AF mode.
  • the focus control unit 38 sets the target focus position.
  • the target focus position is set to maintain the in-focus state of the focus control target by executing a process of finely adjusting the target focus position according to the movement of the focus control target.
  • the fine adjustment of the target focus position is performed, for example, based on the depth information.
  • the focus control unit 38 controls the focus based on the operation direction (rotation direction of the focus ring 7) and the operation amount in the specific operation of the user detected by the user operation detection unit 31. I do. Specifically, when the focus control unit 38 returns to the AF mode again, the focus control unit 38 calculates the movement direction and the movement distance of the focus position based on the operation direction and the operation amount in the specific operation, and estimates a new focus control target. .. Then, the estimated focus control target is selected and the target focus position is set again. A specific example of the process of estimating and selecting a new focus control target will be described later.
  • the display control unit 39 uses the display unit 13 to perform display processing for displaying a menu screen to the user, display processing for displaying a captured image, display processing for displaying a warning, and the like.
  • the display control unit 39 executes a process of displaying a through image.
  • the display control unit 39 superimposes and displays the frame image on the subject recognized by the subject recognition unit 35.
  • the frame image may be changed according to the focus state of the subject.
  • the frame image superimposed on the subject selected as the focus control target is the first frame image FP1 which is a solid line frame.
  • the frame image superimposed on the subject not selected as the focus control target is the second frame image FP2 having a broken line frame.
  • FIG. 6 shows an example of the first frame image FP1 and the second frame image FP2.
  • a state in which two people are recognized as subjects by the subject recognition unit 35 is shown. Of the two people, one is a close subject and the other is a distant subject. Further, the focus control unit 38 is in a state where a close subject is selected as the focus control target.
  • the focus control unit 38 performs focus control on the focus control target to focus on a close subject. In addition, it is said that a distant subject is out of focus.
  • the display control unit 39 superimposes and displays the first frame image FP1 on the subject as the focus control target selected by the focus control unit 38. Further, the display control unit 39 superimposes and displays the second frame image FP2 on the other subjects.
  • the display control unit 39 may superimpose and display on the image on the display unit 13.
  • a circular or elliptical frame may be superimposed and displayed instead of a rectangular frame.
  • the area in which the subject is displayed may be indicated by superimposing a rectangular frame on each of the four corners of the subject being recognized.
  • the inside of the frame of the first frame image FP1 or the second frame image FP2 may be hatched and displayed.
  • FIG. 6 an example in which the display mode of the subject as the focus control target and the other subjects is changed depending on the line type is shown, but they may be distinguished by the line color or the line thickness, and they may be distinguished. It may be distinguished by combining.
  • the display of the subject as the focus control target and the display of other subjects may be separated without using the frame.
  • a depth map display that displays the subject by changing the display color according to the depth amount may be performed, or a peaking display that displays the subject by changing the display color according to the contrast amount in the screen may be performed. You may go.
  • Focus control target selection example> As described above, in the image pickup apparatus 1, a new focus control target is selected when the AF mode is changed again based on the user operation when the AF mode is temporarily changed to the MF mode. Here, some examples of selection of a new focus control target will be described.
  • the first selection example is new according to the rotation position of the focus ring 7 at the end of the user operation when the user performs an operation to rotate the focus ring 7 when transitioning to the temporary MF mode. This is an example of selecting a focus control target.
  • FIG. 7 shows an example of a through image displayed on the rear monitor 4 as the display unit 13.
  • the focus control target area FA is displayed on the display unit 13 with a thick frame line.
  • the in-screen target positions PS3, PS4, and PS5 for each of the three subjects recognized by the in-screen target position setting unit 34 based on the depth information in the frame of the focus control target area FA are frame images. It is shown.
  • the subject located at the target position PS3 in the screen is the subject A with a smile mark
  • the subject located at the target position PS4 in the screen is the subject B of the soccer ball
  • the subject located at the target position PS5 in the screen is the subject C of the plant. It is said that.
  • the subjects A, B, and C are located at different positions in the depth direction. That is, the distances from the image pickup apparatus 1 to each subject are different as shown in FIG. Then, as shown in FIG. 7, the depth value of the in-screen target position PS3 is the depth value DP3, the depth value of the in-screen target position PS4 is the depth value DP4, and the depth value of the in-screen target position PS5 is the depth value. It is said to be DP5. Further, the magnitude relationship between the depth value DP3, the depth value DP4, and the depth value DP5 is that the depth value DP3 ⁇ depth value DP5 ⁇ depth value DP4.
  • the subject A is the closest subject to the image pickup device 1
  • the subject C is the closest subject to the image pickup device 1
  • the subject B is the subject farthest from the image pickup device 1 (see FIG. 8).
  • the depth information is calculated for each small area 41 composed of a plurality of pixels.
  • the subject A is selected as the focus control target, and the first frame image FP1 which is a solid line frame image is superimposed and displayed at the position.
  • the area of the target position PS3 in the screen is shown as the focus control target in the screen.
  • the subject B is not selected as the focus control target, and the second frame image FP2, which is the frame image of the broken line, is superimposed on the position.
  • the area of the target position PS4 in the screen is shown as a subject that is not the focus control target in the screen.
  • the subject C is not selected as the focus control target, and the second frame image FP2, which is the frame image of the broken line, is superimposed on the position.
  • the area of the target position PS5 in the screen is shown as a subject that is not the focus control target in the screen.
  • the subject A is selected as the focus control target before the user operates the focus ring 7 (see FIGS. 7 and 8). That is, the focus position before the user operation is the position of the subject A.
  • the image pickup apparatus 1 transitions from the tracking focus mode to the temporary MF mode. Then, it is assumed that the user operates the focus ring 7 so that the focus position is between the subject C and the subject B and the focus position is moved to a position closer to the subject C (see FIG. 8).
  • the image pickup apparatus 1 estimates the user's intention and selects the subject as the focus control target.
  • the subject C which is the subject closest to the focus position after the user operation, is selected as the focus control target.
  • the in-screen target position PS5 for the subject C is indicated by the first frame image FP1 which is a solid frame image, and the in-screen target positions PS3 and PS4 for the subjects A and B are broken lines. It is shown by the second frame image FP2 which is a frame image (see FIG. 10).
  • the in-screen target position setting unit 34 sets the in-screen target position PS for each subject detected based on the depth information, and selects the focus control target from the in-screen target position PS.
  • the in-screen target position PS set for each subject detected by the in-screen target position setting unit 34 based on the depth information and the in-screen subject set for each subject recognized by the subject recognition unit 35 by performing image processing.
  • the position PO matches it may be shown on the display unit 13 as a candidate for the focus control target.
  • the display unit 13 may show candidates for focus control based only on the subject position PO in the screen without using the target position PS in the screen.
  • the subject closest to the focus position after the user operation is selected as the focus control target from the subjects detected as the subjects in both the detection process by the depth information and the detection process by the image recognition process. This prevents focus control from being performed on an inappropriate subject.
  • FIG. 11 is a diagram showing the positional relationship between the image pickup apparatus 1 and the subjects A, B, and C. Further, FIG. 11 shows the focus position before the user's operation and the focus position after the operation.
  • the image pickup apparatus 1 transitions from the tracking focus mode to the temporary MF mode, and the user Shows a state in which the focus position is a position between the subject C and the subject B and is moved to a position close to the subject C by operating the focus ring 7. That is, the focus position at the end of the user operation (focus position at the end) is the position between the subject C and the subject B.
  • the subject C that is closest to the focus position after the user operation is the subject C, but the subject C is not simply selected as the focus control target, but the focus ring 7 is used.
  • the focus control target is selected in consideration of the rotation direction. This is synonymous with selecting the focus control target in consideration of the moving direction of the focus position.
  • the image pickup apparatus 1 determines whether or not the deviation between the focus position and the position of the subject C after the user operation is equal to or greater than a predetermined threshold value. Then, when the image pickup apparatus 1 determines that the deviation is equal to or greater than a predetermined threshold value, it is estimated that the focus control target intended by the user is not the subject C but the subject B ahead of the moving direction of the focus position.
  • the image pickup apparatus 1 selects the subject B as the focus control target.
  • the subject B is indicated by the first frame image FP1 on the display unit 13.
  • the focus control target may be selected by further using the in-screen subject position PO recognized by the subject recognition unit 35 by the image processing.
  • a new focus control target is selected in consideration of the movement direction of the focus position and the focus control target area FA.
  • FIG. 13 shows the relationship between the focus control target area FA set in the tracking focus mode and each subject. That is, the subject A and the subject B are located in the focus control target area FA, and the subject C is located outside the focus control target area FA. Further, the subject A is selected as the focus control target.
  • FIG. 14 shows a state in which the user operates the focus ring 7 in this state to change the focus position and the focus position.
  • the subject C and the subject B are located ahead of the moving direction of the focus position when viewed from the focus position at the end.
  • the subject C is closer to the focus position than the subject B.
  • the subject C is located ahead of the moving direction, but the subject C is a subject located outside the focus control target area FA as shown in FIG.
  • the subject as the focus control target is selected without considering the focus control target area FA. Therefore, depending on the conditions, a subject outside the focus control target area FA may be selected as the focus control target.
  • the focus control target is selected in consideration of the focus control target area FA, and the result is as follows.
  • the subject C is located ahead of the moving direction of the focus position, but is located outside the focus control target area FA, so the subject B is selected as the focus control target.
  • FIG. 13 shows an example in which the in-screen target position PS5 is set for the subject C outside the focus control target area FA
  • the subject in which the in-screen target position PS is set is in the focus control target area FA. It may be limited to the subject of.
  • the focus control target may be selected by further using the in-screen subject position PO recognized by the subject recognition unit 35 by the image processing.
  • the focus position is the position between the subject A and the subject C.
  • the focus control target at this time is determined to correspond to the specific operation of the user. For example, as in the first selection example, the subject C closest to the focus position at the end may be selected as the focus control target.
  • the subject B closest to the focus position at the end is not selected as the focus control target, but the subject B is selected as the focus control target in consideration of the rotation direction of the focus ring 7. You may.
  • the focus position at the end, the rotation direction of the focus ring 7, and the focus control target area FA are taken into consideration, and the subject is in the focus control target area FA and the focus position is moved.
  • a subject located ahead of the direction may be selected as the focus control target.
  • tracking focus control is performed according to the movement of the selected subject.
  • Example of display mode in the display unit> Here, an example of the display mode in the display unit 13 will be described.
  • FIG. 16 shows a scene in which tracking focus is performed as the AF mode.
  • the display unit 13 shows the target position PS6 in the screen corresponding to the subject H1 in the first frame image FP1 with respect to the through image in which the subject H1 in the foreground and the subject H2 in the back are captured.
  • the target position PS7 in the screen corresponding to the subject H2 is shown by the second frame image FP2.
  • the image pickup apparatus 1 transitions to the temporary MF mode during the operation of the focus ring 7, and then returns to the AF mode (in this case, the tracking focus mode) again.
  • the image pickup apparatus 1 After returning to the tracking focus mode, the image pickup apparatus 1 is set as a new focus control target according to the relationship between the focus position at the end of the rotation operation of the focus ring 7 and the position of the subject, the rotation direction of the focus ring 7, and the like. Select subject H2.
  • the in-screen target position PS7 for the subject H2 selected as the new focus control target is shown by the solid first frame image FP1.
  • the in-screen target position PS6 for the subject H1 that is not the focus control target is indicated by the broken line second frame image FP2.
  • the focus control in the tracking focus mode is executed, so that the focus position is changed according to the change in the position of the subject H2. That is, even if the subject moves, the image pickup apparatus 1 continues to focus on the subject to be tracked (see FIG. 18).
  • the focus ring 7 By sandwiching the user operation on the focus ring 7 during the tracking focus mode, it is possible to keep focusing on the subject while changing the subject to be tracked to the subject intended by the user.
  • FIG. 19 is a diagram showing a through image in the tracking focus mode.
  • the focus control target is the subject H1
  • the third frame image FP3 composed of the four frame images is superimposed and displayed on the pupil region, indicating that the subject H1 is in focus.
  • the image pickup apparatus 1 has already recognized the subject H1 and the subject H2 based on the depth information and the result of the image processing.
  • FIG. 20 shows a state in which the focus position is moved to the back side by the user's operation, that is, a state in which the focus position is located between the subject H1 and the subject H2.
  • the focus control target is the subject H1, but the focus position is deviated from the pupil position and the focused state to the pupil cannot be maintained. Therefore, the first frame image FP1 is superimposed on the subject H1. It is said to be in a state. That is, the user can grasp that the focused state on the pupil cannot be maintained due to the change in the display mode of the frame image.
  • the image pickup apparatus 1 transitions from the temporary MF mode to the tracking focus mode again.
  • a subject to be a new focus control target tilt target
  • the subject H2 is selected as a new focus control target, and the state thereof is shown in FIG.
  • the first frame image FP1 is superimposed on the subject H2 which is the focus control target.
  • This state indicates that the subject H2 is the focus control target, and that the pupil position is not recognized and the focus control on the pupil is not performed.
  • the image pickup apparatus 1 When the subject H2 moves closer to the image pickup apparatus 1, a through image as shown in FIG. 22 is displayed on the display unit 13. In this state, the pupil position of the subject H2 can be recognized by the subject H2 approaching the image pickup apparatus 1, and the subject H2 is appropriately focused on the pupil. Therefore, the third frame image FP3 composed of four frame images is superimposed and displayed on the pupil portion of the subject H2. Although none of the frame images is superimposed and displayed on the subject H1, it is said that the image pickup apparatus 1 has already recognized the subject H1 based on the depth information and the result of image processing.
  • the camera control unit 16 determines whether or not a specific operation has been detected in step S101 of FIG.
  • the specific operation is, for example, the above-mentioned rotation operation of the focus ring 7, when it is desired to manually change the focus control target in the AF mode, when it is desired to change the focus control target in the MF mode, or when it is desired to change the focus control target in the MF mode. This operation is performed when you want to smoothly shift to the AF mode while changing the target.
  • the camera control unit 16 proceeds to step S102 and executes control according to the current mode.
  • the current mode is the tracking focus mode
  • the focus position is moved according to the movement of the subject subject to focus control, and the subject is continuously focused.
  • the pupil AF mode that automatically focuses on the pupil
  • the pupil of the subject is detected to control the focus on the pupil.
  • the camera control unit 16 proceeds to step S103 and changes the mode to the temporary MF mode. Further, the camera control unit 16 acquires the moving direction of the focus position (rotation direction of the focus ring 7) and the moving distance of the focus position (operation amount of the focus ring 7) in step S104.
  • the camera control unit 16 determines in step S105 whether or not the end of the specific operation is detected.
  • the camera control unit 16 repeats the process of step S104 until the end of the specific operation is detected.
  • the image pickup apparatus 1 drives the lens system 9 according to the user's specific operation until the end of the user's specific operation is detected, so that the user manually adjusts the focus position. Is realized.
  • the camera control unit 16 proceeds to step S106 and performs a process of transitioning to the AF mode before the transition to the MF mode.
  • the case where the end of the specific operation is detected is, for example, a case where the state in which the user's finger touches the focus ring 7 is detected is changed to a state in which the state is not detected.
  • step S107 the camera control unit 16 acquires the information in the focus control target area FA.
  • Two examples of the information acquisition process in the focus control target area FA will be described. The first example is shown in FIG.
  • step S151 the camera control unit 16 acquires the depth information in the focus control target area FA.
  • step S152 the camera control unit 16 identifies a subject as a candidate for focus control target based on the acquired depth information.
  • the second example is shown in FIG.
  • the camera control unit 16 acquires depth information in step S151. Subsequently, the camera control unit 16 acquires the information of the subject recognized by the image processing in step S153. Next, the camera control unit 16 selects the subject recognized in step S153 among the subjects recognized based on the depth information in step S154 as a candidate for the focus control target. That is, in the example shown in FIG. 25, the in-screen target position PS set for each subject detected by the in-screen target position setting unit 34 based on the depth information and the subject recognized by the subject recognition unit 35 by performing image processing. This is an example of being selected as a candidate for focus control target when the subject position PO in the screen set for each matches.
  • the camera control unit 16 After selecting or specifying the focus control target candidate, the camera control unit 16 has, in step S108, the movement direction and movement distance of the focus position acquired in step S104 in the MF mode, and the focus control target area FA acquired in step S107.
  • a new focus control target is set based on the depth information in. As a result based on this information, it may be determined that the focus control target is not changed. For example, if no other suitable subject is detected, the focus control target is not changed.
  • step S108 After executing the process of step S108, the camera control unit 16 returns to step S101, determines whether or not a specific operation has been detected again, and if not, controls according to the current mode in step S102. Will be executed. That is, when a new focus control target is selected in step S108, the target target distance is set according to the subject selected as the new focus control target in step S102 until the next specific operation is detected. As a result, tracking focus control is executed.
  • step S201 the camera control unit 16 identifies the subject closest to the focus position after the user operation, that is, the focus position at the end, from the subjects extracted as candidates.
  • the camera control unit 16 selects the specified subject as the focus control target in step S202.
  • the selection mode described in the first selection example is realized.
  • Second example of focus control target selection process> A second example of the process of selecting a new focus control target will be described with reference to FIG. 27. The second example corresponds to the above-mentioned second selection example of the focus control target.
  • step S211 the camera control unit 16 identifies the closest subject among the subjects located ahead of the moving direction of the focus position from the end focus position.
  • the camera control unit 16 selects the specified subject as the focus control target in step S202.
  • the selection mode described in the second selection example is realized.
  • the camera control unit 16 performs each process from step S101 to step S106 in FIG. 23 to detect a user's specific operation and perform a process of transitioning to a temporary MF mode, and then detects the end of the specific operation. Then, the process of transitioning to the original mode is performed again. Since each process is the same as each process with the same reference numerals described in the first example, detailed description thereof will be omitted.
  • the camera control unit 16 acquires information about the entire area regardless of the focus control target area FA instead of step S107. Specifically, the subject is specified based on the depth information and the subject is specified by image processing for the entire area.
  • step S108 the camera control unit 16 performs a process of selecting a new focus control target from the subjects specified in the focus control target area FA. However, if the candidate subject does not exist in the focus control target area FA, a new focus control target is selected from the subjects outside the focus control target area FA. This can be realized by acquiring information for the entire area in step S107.
  • the subject may be temporarily located outside the focus control target area FA. Even in such a case, by acquiring information about the outside of the focus control target area FA, it is possible to select an appropriate subject as the focus control target.
  • step S107 of FIG. 23 information about the focus control target area FA may be acquired. Also in this case, the subject C (see FIG. 13) located outside the focus control target area FA is not focused, and the subject B in the focus control target area FA is appropriately focused. Further, by limiting the acquired information to the focus control target area FA, it is possible to reduce the load of the processing executed by the image pickup apparatus 1.
  • the camera control unit 16 identifies the subject closest to the focus position after the user operation in step S201 in order to select a new focus control target.
  • the specified subject is a subject that can be selected as a focus control target depending on the conditions.
  • step S222 the camera control unit 16 determines whether or not the distance (dissociation) between the focus position after the user operation and the position of the specified subject is equal to or greater than a predetermined threshold value.
  • the camera control unit 16 selects the specified subject as a new focus control target in step S202. ..
  • the camera control unit 16 in step S211 is ahead of the focus position in the moving direction from the end focus position. Identify the closest subject among the subjects located in. Then, in step S202, the camera control unit 16 selects the specified subject as a new focus control target.
  • Focus movement speed control> it is conceivable to adjust the time and speed (focus movement speed) from the selection of the subject as the focus control target to the actual focusing according to the user's operation mode.
  • the image pickup apparatus 1 can immediately focus on the subject H2 after estimating that the subject suitable for the focus control target is the subject H2.
  • the lens may be slowly driven to focus on the subject H2 in consideration of the operation mode in which the user slowly moves the focus ring 7 to move the focus position.
  • the user can specify the speed (time) until focusing in the focus control only by changing the operation mode, so that the desired autofocus control can be realized by a simple operation. This improves user convenience.
  • FIG. 29 shows an example of the processing executed by the camera control unit 16 in order to adjust the time and speed (focus movement speed) until focusing to the user's operation mode.
  • the same processing as in FIG. 23 is designated by the same reference numerals and description thereof will be omitted as appropriate.
  • the camera control unit 16 determines in step S101 whether or not a specific operation has been detected, and if it determines that a specific operation has been detected, the camera control unit 16 shifts to a temporary MF mode in step S103. Processing is performed, and in step S111, information about the user's specific operation is acquired.
  • the information acquired in step S111 acquires not only the moving direction and moving distance of the focus position but also the moving speed of the focus position according to the operation of moving the focus position in the optical axis direction as a specific operation.
  • the camera control unit 16 determines in step S105 whether or not the end of the specific operation is detected.
  • the camera control unit 16 repeats the process of step S111 until the end of the specific operation is detected.
  • the camera control unit 16 proceeds to step S106 and performs a process of transitioning to the mode before the transition to the MF mode.
  • the camera control unit 16 acquires the information in the focus control target area FA in step S107, and selects a new focus control target in step S108.
  • step S101 determines whether or not the specific operation has been detected again, and if not detected, the control according to the current mode is executed in step S112.
  • the camera control unit 16 performs focus control according to the mode in step S112 while taking the moving speed into consideration.
  • the subject as the focus control target newly selected in step S108 is controlled to be focused at a speed according to the operation mode of the user.
  • the focus movement speed can be set in advance by a menu operation in addition to being changed according to the user's specific operation as described above.
  • it may be configured so that a setting for increasing the focus speed, a setting for decreasing the focus speed, and a setting that can be changed according to a specific operation of the user can be selected.
  • the technique according to the present disclosure can be applied to a medical imaging system.
  • the medical imaging system is a medical system using imaging technology, for example, an endoscopic system or a microscope system.
  • FIG. 30 is a diagram showing an example of a schematic configuration of an endoscope system 5000 to which the technique according to the present disclosure can be applied.
  • FIG. 31 is a diagram showing an example of the configuration of the endoscope 5001 and the CCU (Camera Control Unit) 5039.
  • FIG. 30 illustrates how a surgeon (eg, a doctor) 5067, who is a participant in surgery, is performing surgery on patient 5071 on patient bed 5069 using the endoscopic system 5000. As shown in FIG.
  • the endoscope system 5000 supports an endoscope 5001, a CCU 5039, a light source device 5043, a recording device 5053, an output device 5055, and an endoscope 5001, which are medical imaging devices. It is composed of a support device 5027 and a support device 5027.
  • Trocca 5025 an insertion aid called Trocca 5025 is punctured by patient 5071. Then, the scope 5003 and the surgical instrument 5021 connected to the endoscope 5001 are inserted into the body of the patient 5071 via the trocca 5025.
  • the surgical tool 5021 is, for example, an energy device such as an electric knife, forceps, or the like.
  • a surgical image which is a medical image of the inside of the patient 5071 taken by the endoscope 5001, is displayed on the display device 5041.
  • the surgeon 5067 treats the surgical target using the surgical tool 5021 while looking at the surgical image displayed on the display device 5041.
  • the medical image is not limited to the surgical image, and may be a diagnostic image captured during the diagnosis.
  • the endoscope 5001 is a camera that images the inside of the patient 5071.
  • the endoscope 5001 has a condensing optical system 50051 that collects incident light, and an optical zoom by changing the focal distance of the camera.
  • It is a camera head including a zoom optical system 50052 that enables the above, a focus optical system 50053 that enables focus adjustment by changing the focal distance of the camera, and a light receiving element 50054.
  • the endoscope 5001 generates a pixel signal by condensing light on a light receiving element 50054 via a connected scope 5003, and outputs the pixel signal to the CCU 5039 through a transmission system.
  • the scope 5003 is an insertion unit having an objective lens at the tip and guiding light from the connected light source device 5043 into the body of the patient 5071.
  • the scope 5003 is, for example, a rigid scope for a rigid mirror and a flexible scope for a flexible mirror.
  • the pixel signal may be a signal based on the signal output from the pixel, and is, for example, a RAW signal or an image signal.
  • a memory may be mounted in a transmission system connecting the endoscope 5001 and the CCU 5039, and parameters related to the endoscope 5001 and the CCU 5039 may be stored in the memory.
  • the memory may be arranged, for example, on a connection portion of a transmission system or a cable.
  • the light receiving element 50054 is a sensor that converts the received light into a pixel signal, and is, for example, a CMOS (Complementary Metal Oxide Semiconductor) type image pickup device.
  • the light receiving element 50054 is preferably an image pickup device having a Bayer array and capable of color photographing.
  • the light receiving element 50054 is, for example, 4K (horizontal pixel number 3840 x vertical pixel number 2160), 8K (horizontal pixel number 7680 x vertical pixel number 4320) or square 4K (horizontal pixel number 3840 or more x vertical pixel number 3840 or more). It is preferable that the image pickup element has the number of pixels corresponding to the resolution.
  • the light receiving element 50045 may be a single sensor chip or a plurality of sensor chips. For example, a prism that separates incident light into predetermined wavelength bands may be provided, and each wavelength band may be imaged by a different light receiving element. Further, a plurality of light receiving elements may be provided for stereoscopic viewing.
  • the light receiving element 50054 may be a sensor including an arithmetic processing circuit for image processing in the chip structure, or may be a sensor for ToF (Time of Flight).
  • the transmission system is, for example, an optical fiber cable or wireless transmission.
  • the wireless transmission may be performed as long as the pixel signal generated by the endoscope 5001 can be transmitted.
  • the endoscope 5001 and the CCU 5039 may be wirelessly connected, or the endoscope may be wirelessly connected to the endoscope via a base station in the operating room.
  • the mirror 5001 and the CCU 5039 may be connected.
  • the endoscope 5001 may simultaneously transmit not only the pixel signal but also information related to the pixel signal (for example, processing priority of the pixel signal, synchronization signal, etc.).
  • the endoscope may be integrated with a scope and a camera head, or a light receiving element may be provided at the tip of the scope.
  • the CCU 5039 is a control device that collectively controls the connected endoscope 5001 and the light source device 5043.
  • the CCU 5039 may collectively control the connected display device 5041, recording device 5053, and output device 5055.
  • the CCU 5039 controls the irradiation timing, irradiation intensity, and type of the irradiation light source of the light source device 5043.
  • the CCU 5039 performs image processing such as development processing (for example, demosaic processing) and correction processing on the pixel signal output from the endoscope 5001, and the processed pixel signal (for example, an image) is applied to an external device such as a display device 5041. ) Is output. Further, the CCU 5039 transmits a control signal to the endoscope 5001 to control the driving of the endoscope 5001.
  • the control signal is information regarding imaging conditions such as the magnification and focal length of the camera, for example.
  • the CCU 5039 has an image down-conversion function, and may be configured to be capable of simultaneously outputting a high resolution (for example, 4K) image to the display device 5041 and a low resolution (for example, HD) image to the recording device 5053.
  • the CCU 5039 may be connected to an external device via an IP converter that converts a signal into a predetermined communication protocol (for example, IP (Internet Protocol)).
  • IP Internet Protocol
  • the connection between the IP converter and the external device may be configured by a wired network, or a part or all of the network may be constructed by a wireless network.
  • the IP converter on the CCU5039 side has a wireless communication function, and the received video is output to an IP switcher or output via a wireless communication network such as a 5th generation mobile communication system (5G) or a 6th generation mobile communication system (6G). It may be transmitted to the side IP converter.
  • 5G 5th generation mobile communication system
  • 6G 6th generation mobile communication system
  • the light source device 5043 is a device capable of irradiating light in a predetermined wavelength band, and includes, for example, a plurality of light sources and a light source optical system that guides the light of the plurality of light sources.
  • the light source is, for example, a xenon lamp, an LED light source, or an LD light source.
  • the light source device 5043 has, for example, LED light sources corresponding to each of the three primary colors R, G, and B, and emits white light by controlling the output intensity and output timing of each light source.
  • the light source device 5043 may have a light source capable of irradiating special light used for special light observation, in addition to the light source for irradiating normal light used for normal light observation.
  • the special light is light having a predetermined wavelength band different from that of normal light for observing normal light, and is, for example, near-infrared light (light having a wavelength of 760 nm or more), infrared light, blue light, and ultraviolet light.
  • Normal light is, for example, white light or green light.
  • narrow-band light observation which is a type of special light observation, by alternately irradiating blue light and green light, a predetermined tissue such as a blood vessel on the surface layer of the mucous membrane is utilized by utilizing the wavelength dependence of light absorption in the body tissue. Can be taken with high contrast.
  • fluorescence observation which is a type of special light observation
  • the excitation light that excites the reagent injected into the body tissue is irradiated, and the fluorescence emitted by the body tissue or reagent is received to obtain a fluorescence image.
  • a fluorescence image It is possible to make it easier for the surgeon to visually recognize body tissues and the like that are difficult for the surgeon to see.
  • near-infrared light is irradiated as excitation light to excite a reagent such as indocyanine green (ICG) injected into the body tissue, thereby irradiating the structure at the back of the body tissue.
  • ICG indocyanine green
  • a reagent for example, PDD or 5-ALA
  • the light source device 5043 sets the type of irradiation light under the control of the CCU 5039.
  • the CCU 5039 may have a mode in which normal light observation and special light observation are alternately performed by controlling the light source device 5043 and the endoscope 5001. At this time, it is preferable to superimpose the information based on the pixel signal obtained by the special light observation on the pixel signal obtained by the normal light observation.
  • the recording device 5053 is a device for recording pixels acquired from the CCU 5039, and is, for example, a recorder.
  • the recording device 5053 records the image acquired from the CCU 5039 on the HDD, SDD, or optical disk.
  • the recording device 5053 may be connected to a network within the hospital and accessible from equipment outside the operating room. Further, the storage device 5053 may have an image down-conversion function or an image down-conversion function.
  • the display device 5041 is a device capable of displaying an image, for example, a display monitor.
  • the display device 5041 displays a display image based on the pixel signal image-processed by the CCU 5039 under the control of the CCU 5039.
  • the display device 5041 may also function as an input device that enables line-of-sight recognition, voice recognition, and instruction input by gestures by including a camera and a microphone.
  • the output device 5055 is a device that outputs information acquired from the CCU 5039, and is, for example, a printer.
  • the output device 5055 prints, for example, a printed image based on the pixel signal acquired from the CCU 5039 on paper.
  • the support device 5027 is an articulated arm including a base portion 5029 having an arm control device 5045, an arm portion 5031 extending from the base portion 5029, and a holding portion 5032 attached to the tip of the arm portion 5031.
  • the arm control device 5045 is composed of a processor such as a CPU, and controls the drive of the arm unit 5031 by operating according to a predetermined program.
  • the support device 5027 controls parameters such as the length of each link 5035 constituting the arm portion 5031, the rotation angle of each joint 5033, and the torque by the arm control device 5045, for example, the endoscope 5001 held by the holding portion 5032. Control the position and posture of.
  • the support device 5027 functions as an endoscope support arm that supports the endoscope 5001 during the operation. Thereby, the support device 5027 can take the place of the assistant scoopist who has the endoscope 5001. Further, the support device 5027 may be a device that supports the microscope device 5301 described later, and may also be called a medical support arm.
  • the control of the support device 5027 may be an autonomous control method by the arm control device 5045, or may be a control method controlled by the arm control device 5045 based on the input of the user.
  • control method may be a master-slave method in which the support device 5027 as a slave device is controlled based on the movement of the master device at the user's hand. Further, the control of the support device 5027 may be remotely controlled from outside the operating room.
  • the above is an example of the endoscope system 5000 to which the technique according to the present disclosure can be applied.
  • the techniques according to the present disclosure may be applied to a microscope system.
  • FIG. 32 is a diagram showing an example of a schematic configuration of a microsurgery system to which the techniques according to the present disclosure can be applied.
  • the same components as those of the endoscope system 5000 are designated by the same reference numerals, and the overlapping description thereof will be omitted.
  • FIG. 32 schematically shows how the surgeon 5067 is performing surgery on patient 5071 on patient bed 5069 using the microsurgery system 5300.
  • the illustration of the cart 5037 in the configuration of the microscope surgery system 5300 is omitted, and the microscope device 5301 instead of the endoscope 5001 is shown in a simplified manner.
  • the microscope device 5301 in the present description may refer to the microscope unit 5303 provided at the tip of the link 5035, or may refer to the entire configuration including the microscope unit 5303 and the support device 5027.
  • the image of the surgical site taken by the microscope device 5301 is enlarged and displayed on the display device 5041 installed in the operating room by using the microscope surgery system 5300.
  • the display device 5041 is installed at a position facing the operator 5067, and the operator 5067 observes the state of the surgical site by the image projected on the display device 5041, for example, excision of the affected area, etc.
  • Various measures are taken for.
  • the support device 5027 may support another observation device or other surgical instrument at its tip in place of the endoscope 5001 or the microscope unit 5303.
  • the other observation device for example, forceps, a forceps, a pneumoperitoneum tube for pneumoperitoneum, an energy treatment tool for incising a tissue or sealing a blood vessel by abdomen, or the like can be applied.
  • the technique according to the present disclosure may be applied to a support device that supports a configuration other than such a microscope unit.
  • the technique according to the present disclosure can be suitably applied to the endoscope system 5000, the microscope device 5301, and the like among the configurations described above.
  • the subject intended by the operator is estimated and selected as the focus control target according to the moving direction and the moving distance of the focus position.
  • the focus position can be adjusted quickly and accurately compared to the case where the focus position is manually adjusted to the focus control target, so that the operation time can be shortened and the efficiency can be improved, and the burden on the patient can be reduced. Can be done.
  • the image pickup apparatus 1 has a user operation detection unit 31 that detects a user operation for focus control (for example, a rotation operation with respect to the focus ring 7) and a focus based on the user operation detected by the user operation detection unit 31.
  • the focus position movement detection unit 32 that detects the movement (change) of the position (the position in the optical axis direction of the in-focus subject), the depth information detection unit 33 that detects the depth information in the focus control target area FA, and the depth information detection unit 33. It includes a focus control unit 38 that sets a target focus position based on the movement of the focus position and the depth information.
  • the change in the focus position due to the user operation is detected, and after the user operation, the target focus position and the focus control that reflect the user operation are performed.
  • the movement of the focus position by the user is detected by temporarily transitioning to the MF mode.
  • This detection process is, so to speak, a process for estimating the focus control target (subject) intended by the user, and is also a process for collecting information for selecting the subject as the focus control target in the subsequent AF control. ..
  • the camera control unit 16 of the image pickup apparatus 1 detects the manual operation of the focus ring 7 by the user, and detects the operation direction (movement direction of the focus position) and the operation amount (movement amount of the focus position) of the user.
  • the optimum focus control using the MF mode and the AF mode can be performed. Further, when only the AF mode is used, there are many scenes in which a subject different from the subject intended by the user is in focus.
  • the image pickup apparatus 1 provided with this configuration, it is possible to specify the subject intended by the user via the temporary MF mode while the AF mode is being executed, so that the desired AF control can be realized. be able to. In particular, by seamlessly switching between the AF mode and the MF mode, and by using the information acquired in one mode in another mode of the migration destination, such highly convenient control can be realized.
  • the operation of rotating the focus ring 7 has been described as the focus operation, but the same applies even when the focus operation is performed by remote control by an external operating device as described above. As a result, the same effect can be obtained. In this case, usability in remote control can be improved. In particular, it is suitable for focusing on the subject intended by the user by remote control with respect to the fixedly arranged image pickup device 1, and since it is not necessary to touch the image pickup device 1, the occurrence of blurring during the focus operation is eliminated. can do.
  • such a configuration of the image pickup apparatus 1 includes a user operation detection unit 31 that detects a user operation for focusing control, and a focus position that detects movement of the focus position by a user operation detected by the user operation detection unit 31.
  • the movement detection unit 32, the depth information detection unit 33 that detects the depth information in the focus control target area FA of the focus control, and the user operation detection unit 31 detect the end of the movement of the focus position the depth information is used. It is provided with a focus control unit 38 for setting a target focus position based on the above.
  • the focus position movement detection unit 32 in the image pickup apparatus 1 may detect the movement direction of the focus position. good. By detecting the moving direction of the focus position, the user's intention can be estimated more accurately. Therefore, easy-to-use autofocus control can be realized.
  • the focus position movement detection unit 32 of the image pickup apparatus 1 may detect the movement direction every predetermined time (for example, one frame). By detecting the moving direction at predetermined time intervals, the operation direction of the user is appropriately detected. Therefore, it is possible to appropriately estimate the focus control target intended by the user.
  • the depth information detection unit 33 of the image pickup apparatus 1 detects the depth information of the entire focus control target area FA and focuses on it.
  • the control unit 38 detects the depth information of the entire focus control target area FA, the movement direction of the focus position, and the end of the user's operation regarding the movement of the focus position, and the focus position at the end (at the time of the end of movement).
  • the target focus position may be set based on the focus position). By setting the target focus position based on the focus position at the end, the user's intention is reflected in the focus control.
  • the user operation information acquired in the MF mode is utilized in the tracking focus mode. By doing so, it is possible to perform tracking focus control according to the user's intention.
  • the image pickup device 1 includes an in-screen target position setting unit 34 for setting the in-screen target position PS based on the depth information, and focus control is provided.
  • the unit 38 may set the focus position corresponding to the target position PS in the screen as the target focus position.
  • the target position PS in the screen is an area set on the captured image, and is an area composed of a group of pixels having similar depth information. For example, since the pixel area in which a certain subject is captured has similar depth information for each pixel, it is set as the target position PS in the screen. As a result, it is possible to perform appropriate focus control on the recognized subject based on the depth information.
  • the configuration of such an image pickup apparatus 1 is an area setting unit (in the screen) that sets an area in the image where the in-focus candidate is located in the image as an in-focus candidate area (target position in the screen) based on the depth information.
  • the target position setting unit 34 is provided, and the focus control unit 38 sets the target focus position so as to focus on the in-focus candidate of the in-focus candidate area set by the area setting unit.
  • the in-screen target position setting unit 34 sets a plurality of in-screen target position PS.
  • the focus position closest to the end focus position among the focus positions corresponding to the target position PS in the screen may be set as the target focus position.
  • the focus control unit 38 has such a configuration when the area setting unit (target position setting unit 34 in the screen) sets a plurality of focusing candidate areas (target positions in the screen). , The target focus position is set so as to focus on the focusing candidate at the position closest to the focus position at the end in the optical axis direction.
  • the image pickup apparatus 1 includes a subject recognition unit 35 for recognizing a subject.
  • the focus control unit 38 determines that the target position PS in the screen and the subject position in the screen, which is the position of the subject recognized by the subject recognition unit 35 in the screen, match, the focus control unit 38 corresponds to the subject position in the screen.
  • the focus position may be set as the target focus position.
  • a subject as a focus control target is selected by using image recognition processing or the like. Therefore, it is possible to increase the possibility that an appropriate subject is selected as the focus control target, and it is possible to execute appropriate focus control.
  • the configuration of the image pickup device 1 includes a subject recognition unit 35 that recognizes the subject based on the captured image, and the focus control unit 38 is a focusing candidate region (focusing candidate region that matches the position of the subject in the image).
  • the target focus position is set so as to focus on the focusing candidate (target position in the screen).
  • the focus control unit 38 of the image pickup apparatus 1 focuses at the end of the focus positions corresponding to the subject positions in the screen.
  • the focus position closest to the position may be set as the target focus position.
  • an appropriate subject can be selected as the focus control target, and a subject for which the user's intention is determined can be selected as the focus control target.
  • the focus control unit 38 matches the subject at the position closest to the end focus position in the optical axis direction when the subject recognition unit 35 recognizes a plurality of subjects.
  • the target focus position is set so as to be impatient.
  • the focus control unit 38 of the image pickup apparatus 1 is set to the target focus position set after the movement of the focus position is completed. Tracking focus control may be performed on the focused subject accordingly. As a result, the user does not have to manually keep focusing on the focused subject. This can be realized by selecting the focus control target in the tracking focus mode in consideration of the content of the user's operation on the focus ring 7. By performing such control, convenience can be improved.
  • the unit 36 may be provided. This makes it possible to select the focus control target from the subject by reflecting the user's intention more. Therefore, it is possible to execute focus control appropriate for the user.
  • the subject recognition unit 35 in the image pickup apparatus 1 A display in which the first frame image FP1 is superimposed and displayed on the subject focused by the focus control unit 38 among the subjects recognized by the above-mentioned subject, and the second frame image FP2 is superimposed and displayed on the other subjects.
  • the control unit 39 may be provided. As a result, the user can confirm the focus control target by confirming the display unit 13. Therefore, the convenience can be improved.
  • the focus control unit 38 of the image pickup apparatus 1 may perform focus control according to the set focus movement speed. This makes it possible to specify the speed (time) until focusing in focus control simply by changing the operation mode. Therefore, the desired autofocus control can be realized by a simple operation, and the convenience of the user can be improved.
  • the user operation detection unit 31 of the image pickup device 1 detects the user's operation on the focus ring 7, and the focus position movement detection unit 32 detects the focus ring.
  • the moving direction of the focus position may be detected based on the operation direction of 7. As a result, the moving direction of the focus position is appropriately detected. Therefore, it is possible to perform appropriate focus control based on the user's operation on the focus ring 7.
  • the focus control unit 38 of the image pickup device 1 performs a first mode (AF mode) in which autofocus control is performed on the subject and manual focus control.
  • the second mode (MF mode) to be performed can be switched, and when the user operation detection unit 31 detects an operation on the focus ring 7 in the first mode, the mode may be switched to the second mode.
  • the user only has to operate the focus ring 7 when switching to the MF mode. That is, since the mode can be switched by a simple operation, the convenience can be improved.
  • the user operation detection unit 31 of the image pickup device 1 has not detected the operation of the focus ring 7 as a user operation.
  • This makes it possible to detect the end of the user operation on the focus ring 7 by using, for example, a sensor or the like. Therefore, it is possible to appropriately determine the end time point of the user's operation, and it is possible to appropriately perform subsequent focus control.
  • the focus control unit 38 of the image pickup apparatus 1 is the closest to the end focus position among the focus positions corresponding to the in-screen target position PS.
  • the first control selection mode shown in the first selection example
  • the second control second control
  • the first control is switched to the second control. May be good.
  • the subject to be focused controlled is selected according to the user's operation mode and the position of the subject.
  • the configuration of such an image pickup apparatus 1 is an area setting unit (in the screen) that sets an area in the image where the in-focus candidate is located as an in-focus candidate area (target position in the screen) based on the depth of field information.
  • a target position setting unit 34 is provided, the focus position movement detection unit 32 detects the movement direction of the focus position, and the focus control unit 38 sets the focus position (focus position at the end) at the end of the movement in the optical axis direction.
  • the first control that sets the target focus position so as to focus on the focus candidate at the closest position
  • the second control that sets the target focus position according to the focus position and the movement direction at the end of the movement. It is feasible, and when the difference between the target focus position in the first control and the focus position at the end of the movement is equal to or greater than a predetermined threshold value, the first control is switched to the second control.
  • the program to be executed by the image pickup apparatus 1 is a program to execute each process shown in each of FIGS. 23 to 29 by, for example, a CPU, a DSP, or a device including these. That is, this program includes a process of detecting the user's operation on the focus control, a process of detecting the movement of the focus position based on the detected user's operation, and a process of detecting the depth information in the focus control target area FA.
  • This is a program for causing the image pickup apparatus 1 or the like to execute a process of setting a target focus position based on the movement of the focus position and the depth information. With such a program, the above-mentioned image pickup apparatus 1 can be realized.
  • a program that realizes such an image pickup device 1 can be recorded in advance in an HDD as a recording medium built in a device such as the image pickup device 1, a ROM in a microcomputer having a CPU, or the like.
  • flexible discs CD-ROMs (Compact Disc Read Only Memory), MO (Magnet optical) discs, DVDs (Digital Versatile Discs), Blu-ray discs (Blu-ray Disc (registered trademark)), magnetic discs, semiconductor memories, It can be temporarily or permanently stored (recorded) on a removable recording medium such as a memory card.
  • Such removable recording media can be provided as so-called package software.
  • a program from a removable recording medium on a personal computer or the like it can also be downloaded from a download site via a network such as a LAN (Local Area Network) or the Internet.
  • LAN Local Area Network
  • a program it is suitable for a wide range of provision of the image pickup apparatus 1 of the embodiment.
  • a mobile terminal device such as a smartphone or tablet equipped with a camera function, a mobile phone, a personal computer, a game device, a video device, a PDA (Personal Digital Assistant), etc.
  • these devices are disclosed in the present disclosure. It can function as an image pickup device 1.
  • This technology> can also adopt the following configurations.
  • a user operation detection unit that detects the user's operation on focus control
  • the focus position movement detection unit that detects the movement of the focus position based on the user's operation detected by the user operation detection unit, and the focus position movement detection unit.
  • Depth information detection unit that detects depth information in the focus control target area
  • An image pickup apparatus including a focus control unit for setting a target focus position based on the movement of the focus position and the depth information.
  • the depth information detection unit detects the depth information of the entire focus control target area, and detects the depth information.
  • the focus control unit includes depth information of the entire focus control target area, the movement direction of the focus position, and the end focus position when the user's operation regarding the movement of the focus position is completed.
  • the image pickup apparatus according to any one of (2) to (3) above, which sets the target focus position based on the above. (5) It is provided with an in-screen target position setting unit that sets an in-screen target position based on the depth information.
  • the focus control unit sets the focus position closest to the end focus position among the focus positions corresponding to the in-screen target positions.
  • the image pickup apparatus according to (5) above which is set as the target focus position.
  • (7) Equipped with a subject recognition unit that recognizes the subject
  • the focus control unit determines that the target position in the screen and the subject position in the screen, which is the position of the subject recognized by the subject recognition unit in the screen, match, the focus control unit sets the subject position in the screen.
  • the image pickup apparatus according to any one of (5) to (6) above, wherein the corresponding focus position is set as the target focus position.
  • the focus control unit sets the focus position closest to the end focus position among the focus positions corresponding to the subject positions in the screen as the target focus position.
  • the image pickup apparatus according to (7) above.
  • the focus control unit performs tracking focus control on the subject focused according to the target focus position set after the movement of the focus position is completed. Described in any of the above (7) to (8).
  • Imaging device (10) The image pickup apparatus according to any one of (1) to (9) above, which includes an area setting unit for setting the focus control target area according to a user operation.
  • the image pickup apparatus according to any one of (7) to (9) above, which is provided with a display control unit for performing the above. (12) The image pickup apparatus according to any one of (1) to (11) above, wherein the focus control unit performs focus control according to a set focus movement speed. (13) The user operation detection unit detects the user's operation on the focus ring and detects the user's operation. The image pickup apparatus according to any one of (2) to (9) above, wherein the focus position movement detection unit detects the movement direction of the focus position based on the operation direction of the focus ring.
  • the user operation detection unit detects the user's focus operation by receiving the operation information transmitted from the external operation device.
  • the image pickup apparatus according to any one of (2) to (9) above, wherein the focus position movement detection unit detects the movement direction of the focus position based on the operation direction of the focus operation.
  • the focus control unit is It is possible to switch between the first mode, which controls the autofocus of the subject, and the second mode, which controls the focus manually.
  • the image pickup apparatus according to any one of (13) to (14) above, which switches to the second mode when the user operation detection unit detects an operation on the focus ring in the first mode.
  • the user operation detection unit determines that the end of the movement of the focus position has been detected when the operation of the user changes from the state of detecting the operation to the focus ring to the state of not detecting the operation.
  • the imaging device according to any one of (13) to (15) above.
  • the focus control unit is The first control of setting the focus position closest to the end focus position among the focus positions corresponding to the in-screen target positions as the target focus position, and according to the focus position and the movement direction at the end of the movement.
  • the second control for setting the target focus position can be executed.
  • the imaging according to any one of (5) to (9) above, which switches from the first control to the second control when the deviation between the closest focus position and the focus position at the end is equal to or greater than a predetermined threshold value.
  • Image pickup device 7 Focus ring 31 User operation detection unit 32 Focus position movement detection unit 33 Depth information detection unit 34 In-screen target position setting unit 35 Subject recognition unit 36 Area setting unit 38 Focus control unit PS, PS1, PS2, PS3, PS4 , PS5, PS6, PS7 Target position in screen PO Subject position in screen FP1 1st frame image FP2 2nd frame image

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Optics & Photonics (AREA)
  • Human Computer Interaction (AREA)
  • Studio Devices (AREA)
  • Automatic Focus Adjustment (AREA)
PCT/JP2021/025190 2020-07-27 2021-07-02 撮像装置、ピント位置設定方法、プログラム Ceased WO2022024671A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
US18/005,637 US12401896B2 (en) 2020-07-27 2021-07-02 Imaging device and focus position setting method
JP2022540102A JP7687337B2 (ja) 2020-07-27 2021-07-02 撮像装置、ピント位置設定方法、プログラム
EP21850127.8A EP4191331B1 (en) 2020-07-27 2021-07-02 Imaging device for setting focus position
JP2025085626A JP2025116032A (ja) 2020-07-27 2025-05-22 撮像装置、ピント位置設定方法、プログラム

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-126618 2020-07-27
JP2020126618 2020-07-27

Publications (1)

Publication Number Publication Date
WO2022024671A1 true WO2022024671A1 (ja) 2022-02-03

Family

ID=80035535

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/025190 Ceased WO2022024671A1 (ja) 2020-07-27 2021-07-02 撮像装置、ピント位置設定方法、プログラム

Country Status (4)

Country Link
US (1) US12401896B2 (https=)
EP (1) EP4191331B1 (https=)
JP (2) JP7687337B2 (https=)
WO (1) WO2022024671A1 (https=)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009044682A (ja) * 2007-08-10 2009-02-26 Canon Inc 撮像装置及びその制御方法
JP2009103912A (ja) * 2007-10-23 2009-05-14 Olympus Corp 撮像装置
JP2014021328A (ja) * 2012-07-19 2014-02-03 Canon Inc 立体映像撮影システムに用いる光学装置
JP2015001717A (ja) * 2013-06-18 2015-01-05 オリンパス株式会社 撮影機器及びその制御方法
JP2016206306A (ja) * 2015-04-17 2016-12-08 キヤノン株式会社 制御装置、その方法、および制御プログラム
JP2018125536A (ja) 2009-09-04 2018-08-09 株式会社半導体エネルギー研究所 表示装置

Family Cites Families (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US9509913B2 (en) * 2013-06-18 2016-11-29 Olympus Corporation Image pickup apparatus, method of controlling image pickup apparatus, image pickup apparatus system, and image pickup control program stored in storage medium of image pickup apparatus
JP6335396B2 (ja) * 2015-09-30 2018-05-30 富士フイルム株式会社 撮影装置及びその制御方法
JP6659148B2 (ja) * 2016-02-03 2020-03-04 キヤノン株式会社 表示制御装置及びその制御方法、プログラム、並びに記憶媒体
JP2018155904A (ja) * 2017-03-17 2018-10-04 キヤノン株式会社 表示制御装置及びその制御方法
JP7079161B2 (ja) * 2018-06-29 2022-06-01 キヤノン株式会社 撮像装置及びその制御方法
JP7121612B2 (ja) * 2018-09-14 2022-08-18 キヤノン株式会社 撮像制御装置およびその制御方法、プログラム、並びに記憶媒体
JP7204622B2 (ja) * 2018-09-21 2023-01-16 キヤノン株式会社 ズーム制御装置およびその制御方法
JP6703164B2 (ja) * 2019-03-29 2020-06-03 キヤノン株式会社 表示制御装置及びその制御方法
JP6818814B2 (ja) * 2019-06-26 2021-01-20 キヤノン株式会社 焦点検出装置およびその制御方法
JP2024082603A (ja) * 2022-12-08 2024-06-20 キヤノン株式会社 焦点調節制御装置及び方法、撮像装置、プログラム及び記録媒体

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2009044682A (ja) * 2007-08-10 2009-02-26 Canon Inc 撮像装置及びその制御方法
JP2009103912A (ja) * 2007-10-23 2009-05-14 Olympus Corp 撮像装置
JP2018125536A (ja) 2009-09-04 2018-08-09 株式会社半導体エネルギー研究所 表示装置
JP2014021328A (ja) * 2012-07-19 2014-02-03 Canon Inc 立体映像撮影システムに用いる光学装置
JP2015001717A (ja) * 2013-06-18 2015-01-05 オリンパス株式会社 撮影機器及びその制御方法
JP2016206306A (ja) * 2015-04-17 2016-12-08 キヤノン株式会社 制御装置、その方法、および制御プログラム

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4191331A4

Also Published As

Publication number Publication date
US20230276126A1 (en) 2023-08-31
JPWO2022024671A1 (https=) 2022-02-03
JP2025116032A (ja) 2025-08-07
US12401896B2 (en) 2025-08-26
JP7687337B2 (ja) 2025-06-03
EP4191331A1 (en) 2023-06-07
EP4191331A4 (en) 2023-12-27
EP4191331B1 (en) 2025-08-27

Similar Documents

Publication Publication Date Title
US20210019921A1 (en) Image processing device, image processing method, and program
JP2015195845A (ja) 内視鏡システム、内視鏡システムの作動方法、プロセッサ装置、プロセッサ装置の作動方法
US20250016443A1 (en) Imaging device, imaging control device, and imaging method
JPWO2017010157A1 (ja) 医療用観察装置及び医療用観察方法
CN110637458B (zh) 信息处理装置、信息处理方法以及计算机可读存储介质
WO2018221068A1 (ja) 情報処理装置、情報処理方法および情報処理プログラム
US10542866B2 (en) Medical imaging device
JP7230807B2 (ja) 信号処理装置、撮像装置、信号処理方法及びプログラム
WO2018088237A1 (ja) 画像処理装置および設定方法、並びにプログラム
JP7687337B2 (ja) 撮像装置、ピント位置設定方法、プログラム
JP7136093B2 (ja) 情報処理装置、情報処理方法および情報処理プログラム
JP7207296B2 (ja) 撮像装置とフォーカス制御方法およびフォーカス判定方法
JPWO2018088236A1 (ja) 画像処理装置および方法、並びにプログラム
JPWO2018088238A1 (ja) 画像処理装置および制御方法、並びにプログラム
US20220256096A1 (en) Image processing device, image processing method, and program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21850127

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022540102

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2021850127

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2021850127

Country of ref document: EP

Effective date: 20230227

WWG Wipo information: grant in national office

Ref document number: 18005637

Country of ref document: US

WWG Wipo information: grant in national office

Ref document number: 2021850127

Country of ref document: EP