Classifications

• • G—PHYSICS
  • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
  • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
  • G03B13/00—Viewfinders; Focusing aids for cameras; Means for focusing for cameras; Autofocus systems for cameras
  • G03B13/32—Means for focusing
  • G03B13/34—Power focusing
  • G03B13/36—Autofocus systems
• • G—PHYSICS
  • G02—OPTICS
  • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
  • G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
  • G02B7/28—Systems for automatic generation of focusing signals
• • G—PHYSICS
  • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
  • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
  • G03B15/00—Special procedures for taking photographs; Apparatus therefor
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/61—Control of cameras or camera modules based on recognised objects
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/61—Control of cameras or camera modules based on recognised objects
  • H04N23/611—Control of cameras or camera modules based on recognised objects where the recognised objects include parts of the human body
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/67—Focus control based on electronic image sensor signals
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
  • H04N23/67—Focus control based on electronic image sensor signals
  • H04N23/672—Focus control based on electronic image sensor signals based on the phase difference signals

Definitions

• the technology of the present disclosure relates to an imaging method, an imaging device, and a program.
• Japanese Patent Application Laid-Open No. 2009-098317 discloses an imaging device that suppresses the occurrence of misfocus caused by a background image included in the autofocus target area when performing autofocus using the autofocus target area determined based on the face area obtained by face detection.
• the face detection unit performs face detection to identify a face region containing a person's face image.
• the AF target area determination unit determines an AF target area from the face area.
• the AF target area determining section can change the area ratio of the AF target area to the face area.
• the AF evaluation value calculation unit, control unit, and lens driving unit adjust the imaging position of the subject image formed by the imaging optical system based on the contrast of the captured image data corresponding to the AF target area determined by the AF target area determination unit.
• Japanese Patent Application Laid-Open No. 2021-132362 discloses a subject tracking device capable of reducing erroneous tracking of a subject.
• the subject tracking device described in Japanese Patent Application Laid-Open No. 2021-132362 includes image acquiring means for successively acquiring images, tracking means for tracking a subject detected from the images acquired by the image acquiring means by comparing the images over a plurality of images successively acquired by the image acquiring means, and switching means for switching the duration of tracking by the tracking means according to the type of the subject detected from the images.
• An embodiment according to the technology of the present disclosure provides an imaging method, an imaging device, and a program capable of improving the accuracy of focusing on a subject to be focused.
• the imaging method of the present disclosure includes an imaging step of generating image data with an imaging device, a detecting step of detecting a first range including a subject to be focused from the image data, a determining step of determining an attribute of the subject, and a determining step of determining whether the size of a second range for obtaining distance information of the subject is less than the first range or greater than the first range based on the attribute.
• the detection process and determination process are preferably performed using machine-learned models.
• the determining step it is preferable to determine which of the two or more types of objects the attribute of the subject corresponds to, or to determine which of the two or more types of parts of the object it corresponds to.
• the object is preferably a person, animal, bird, train, car, motorcycle, ship, or airplane.
• the size of the second range is different between when the attribute is determined to be the first part of the first object and when the attribute is determined to be the first part of the second object in the determination step.
• the focusing step can selectively execute a continuous focusing mode in which the focusing operation is performed continuously as the focusing mode, and the determining step preferably varies the size of the second range depending on whether the focusing mode is the continuous focusing mode.
• the determining step preferably includes a correcting step of correcting the size of the second range.
• the correction step preferably corrects the size of the second range based on the state of the subject, whether the subject is a moving body, or the reliability of attribute determination.
• the correction step preferably reduces the second range when the size of the second range exceeds the first threshold, and expands the second range when the size of the second range falls below a second threshold that is smaller than the first threshold.
• the imaging device of the present disclosure includes an imaging device that generates image data, and a processor.
• the processor executes detection processing for detecting a first range including a subject to be focused from the image data, determination processing for determining the attribute of the subject, and determination processing for determining whether the size of the second range for acquiring distance information of the subject is less than the first range or greater than the first range based on the attribute.
• the program of the present disclosure causes a computer to execute a detection process of detecting a first range including a subject to be focused from image data, a determination process of determining the attribute of the subject, and a determination process of determining whether the size of the second range for acquiring distance information of the subject is less than the first range or greater than the first range based on the attribute.
• FIG. 3 is a block diagram showing an example of a functional configuration of a processor;
• FIG. 4 is a diagram conceptually showing an example of processing by a machine-learned model;
• FIG. 4 is a diagram conceptually showing an example of processing by a subject area detection unit;
• An example of a table is conceptually shown.
• FIG. 5 is a diagram conceptually showing an example of processing by an AF area determination unit; It is a figure explaining the 1st threshold value which a magnification correction
• FIG. 11 is a diagram showing magnification correction processing according to a second modification
• FIG. 14 is a diagram showing magnification correction processing according to a third modification
• AF is an abbreviation for "Auto Focus”.
• MF is an abbreviation for "Manual Focus”.
• IC is an abbreviation for "Integrated Circuit”.
• CPU is an abbreviation for "Central Processing Unit”.
• ROM is an abbreviation for "Read Only Memory”.
• RAM is an abbreviation for "Random Access Memory”.
• CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor.”
• FPGA is an abbreviation for "Field Programmable Gate Array”.
• PLD is an abbreviation for "Programmable Logic Device”.
• ASIC is an abbreviation for "Application Specific Integrated Circuit”.
• OPF is an abbreviation for "Optical View Finder”.
• EMF is an abbreviation for "Electronic View Finder”.
• the technology of the present disclosure will be described by taking a lens-interchangeable digital camera as an example.
• the technique of the present disclosure is not limited to interchangeable-lens type digital cameras, and can be applied to lens-integrated digital cameras.
• FIG. 1 shows an example of the configuration of the imaging device 10.
• the imaging device 10 is a lens-interchangeable digital camera.
• the imaging device 10 is composed of a main body 11 and an imaging lens 12 that is exchangeably attached to the main body 11 and includes a focus lens 31 .
• the imaging lens 12 is attached to the front side of the main body 11 via a camera side mount 11A and a lens side mount 12A.
• the main body 11 is provided with an operation unit 13 including dials, a release button, and the like.
• the operation modes of the imaging device 10 include, for example, a still image imaging mode, a moving image imaging mode, and an image display mode.
• the operation unit 13 is operated by the user when setting the operation mode. Further, the operation unit 13 is operated by the user when starting execution of still image capturing or moving image capturing.
• Focus modes include AF mode and MF mode.
• the AF mode is a mode in which a subject area selected by the user or a subject area automatically detected by the imaging device 10 is set as a focus detection area (hereinafter referred to as an AF area) and focus control is performed.
• the MF mode is a mode in which the user manually performs focus control by operating a focus ring (not shown). In this embodiment, each of the subject area and the AF area is rectangular.
• AF modes include continuous AF mode (hereinafter referred to as AF-C mode) and single AF mode (hereinafter referred to as AF-S mode).
• the AF-C mode is a mode in which focus control is continued (that is, position control of the focus lens 31 is continued) while the release button is half-pressed.
• the AF-C mode corresponds to the "continuous focusing mode in which the focusing operation is performed continuously" according to the technology of the present disclosure.
• “continuously” means that the focus control for a specific subject is automatically repeated over a plurality of frame periods, and a frame period in which the focus control is not performed may be part of the plurality of frame periods.
• the AF-S mode is a mode in which focus control is performed once in response to the release button being half-pressed, and the position of the focus lens 31 is fixed while the release button is half-pressed.
• AF-C mode and AF-S mode can be switched using the operation unit 13 .
• a settable focus target subject is an object or a part of an object.
• Objects to be focused include, for example, people, animals (dogs, cats, etc.), birds, trains, cars, motorcycles (motorcycles), ships, and airplanes.
• the part to be focused is, for example, the face of a person, the pupil of a person, the pupil of an animal, or the pupil of a bird.
• the pupil is set as the part to be focused, it is possible to set which of the right eye and the left eye is prioritized as the subject to be focused.
• the main body 11 is provided with a finder 14 .
• the finder 14 is a hybrid finder (registered trademark).
• a hybrid viewfinder is a viewfinder in which, for example, an optical viewfinder (hereinafter referred to as "OVF") and an electronic viewfinder (hereinafter referred to as "EVF”) are selectively used.
• OVF optical viewfinder
• EMF electronic viewfinder
• a user can observe an optical image or a live view image of a subject projected through the viewfinder 14 through a viewfinder eyepiece (not shown).
• a display 15 is provided on the back side of the main body 11 .
• the display 15 displays an image based on an imaging signal obtained by imaging, various menu screens, and the like. The user can also observe a live view image projected on the display 15 instead of the viewfinder 14 .
• the body 11 and the imaging lens 12 are electrically connected by contact between an electrical contact 11B provided on the camera side mount 11A and an electrical contact 12B provided on the lens side mount 12A.
• the imaging lens 12 includes an objective lens 30, a focus lens 31, a rear end lens 32, and an aperture 33. Each member is arranged along the optical axis A of the imaging lens 12 in the order of the objective lens 30, the diaphragm 33, the focus lens 31, and the rear end lens 32 from the objective side.
• the objective lens 30, focus lens 31, and rear end lens 32 constitute an imaging optical system.
• the type, number, and order of arrangement of lenses that constitute the imaging optical system are not limited to the example shown in FIG.
• the imaging lens 12 also has a lens drive control section 34 .
• the lens drive control unit 34 is composed of, for example, a CPU, a RAM, a ROM, and the like.
• the lens drive control section 34 is electrically connected to the processor 40 in the main body 11 via the electrical contacts 12B and 11B.
• the lens drive control unit 34 drives the focus lens 31 and the diaphragm 33 based on control signals sent from the processor 40 . In order to adjust the position of the focus lens 31 , the lens drive control unit 34 performs drive control of the focus lens 31 based on a control signal for focus control transmitted from the processor 40 .
• the diaphragm 33 has an aperture whose aperture diameter is variable around the optical axis A.
• the lens drive control unit 34 performs drive control of the diaphragm 33 based on the control signal for diaphragm adjustment transmitted from the processor 40.
• an imaging sensor 20 a processor 40, and a memory 42 are provided inside the main body 11.
• the operations of the imaging sensor 20 , the memory 42 , the operation unit 13 , the viewfinder 14 and the display 15 are controlled by the processor 40 .
• the processor 40 is composed of, for example, a CPU, RAM, and ROM. In this case, processor 40 executes various processes based on program 43 stored in memory 42 . Note that the processor 40 may be configured by an assembly of a plurality of IC chips. In addition, the memory 42 stores a machine-learned model LM that has undergone machine learning for object detection.
• the imaging sensor 20 is, for example, a CMOS image sensor.
• the imaging sensor 20 is arranged such that the optical axis A is perpendicular to the light receiving surface 20A and the optical axis A is positioned at the center of the light receiving surface 20A.
• Light (subject image) that has passed through the imaging lens 12 is incident on the light receiving surface 20A.
• a plurality of pixels that generate imaging signals by performing photoelectric conversion are formed on the light receiving surface 20A.
• the imaging sensor 20 photoelectrically converts light incident on each pixel to generate and output image data PD including an imaging signal.
• the imaging sensor 20 is an example of an “imaging device” according to the technology of the present disclosure.
• a Bayer array color filter array is arranged on the light receiving surface 20A of the image sensor 20, and a color filter of R (red), G (green), or B (blue) is arranged to face each pixel. Note that some of the plurality of pixels arranged on the light receiving surface of the image sensor 20 are phase difference detection pixels that output phase difference detection signals for performing focus control.
• FIG. 2 shows an example of the light receiving surface 20A of the imaging sensor 20.
• FIG. A plurality of imaging pixels 21 and a plurality of phase difference detection pixels 22 are arranged on the light receiving surface 20A.
• the imaging pixels 21 are pixels in which the color filters described above are arranged.
• the imaging pixels 21 receive light beams passing through the entire exit pupil of the imaging optical system.
• the phase difference detection pixel 22 receives a light flux passing through a half area of the exit pupil of the imaging optical system.
• some of the G pixels arranged diagonally in the Bayer array are replaced with the phase difference detection pixels 22 .
• the phase difference detection pixels 22 are arranged at regular intervals in the vertical and horizontal directions on the light receiving surface 20A.
• the phase difference detection pixels 22 are divided into first phase difference detection pixels that receive the light flux passing through the half area of the exit pupil and second phase difference detection pixels that receive the light flux passing through the other half area of the exit pupil.
• a plurality of imaging pixels 21 output imaging signals for generating an image of a subject.
• the multiple phase difference detection pixels 22 output phase difference detection signals.
• the image data PD output from the imaging sensor 20 includes an imaging signal and a phase difference detection signal.
• FIG. 3 shows an example of the functional configuration of the processor 40.
• the processor 40 implements various functional units by executing processes according to programs 43 stored in the memory 42 .
• the processor 40 implements a main control unit 50, an imaging control unit 51, an image processing unit 52, a display control unit 53, an image recording unit 54, a subject detection unit 55, an AF area determination unit 56, and a distance information acquisition unit 57.
• the main control unit 50 comprehensively controls the operation of the imaging device 10 based on instruction signals input from the operation unit 13 .
• the imaging control unit 51 controls the imaging sensor 20 to perform an imaging process for causing the imaging sensor 20 to perform an imaging operation.
• the imaging control unit 51 drives the imaging sensor 20 in still image imaging mode or moving image imaging mode.
• the imaging sensor 20 outputs image data PD generated by imaging through the imaging lens 12 .
• the image data PD output from the imaging sensor 20 is supplied to the image processing section 52 , subject detection section 55 and distance information acquisition section 57 .
• the image processing unit 52 acquires the image data PD output from the imaging sensor 20, and performs image processing including white balance correction, gamma correction processing, etc. on the image data PD.
• the display control unit 53 causes the display 15 to display a live view image based on the image data PD subjected to image processing by the image processing unit 52 .
• the image recording unit 54 records the image data PD subjected to the image processing by the image processing unit 52 in the memory 42 as the recorded image PR when the release button is fully pressed.
• the subject detection unit 55 reads the machine-learned model LM stored in the memory 42 .
• the subject detection unit 55 performs detection processing for detecting a subject area including a subject to be focused from the image data PD using the machine-learned model LM, and determination processing for determining attributes of the subject using the machine-learned model LM.
• the subject detection unit 55 includes a subject area detection unit 55A that performs detection processing and an attribute determination unit 55B that performs determination processing.
• the subject area is an example of the "first range" according to the technology of the present disclosure.
• the attribute is, for example, a category for classifying the type of subject.
• the machine-learned model LM is composed of, for example, a convolutional neural network, detects an object appearing in the image data PD, and outputs detection information of the object together with the attribute and detection score of the detected object.
• a machine-learned model LM enables detection of more than one type of object.
• the objects detected by the machine-learned model LM are, for example, two or more kinds of objects selected from people, animals, birds, trains, cars, motorcycles, ships, and airplanes.
• the machine-learned model LM detects the parts of the object and outputs the detection information of the parts of the object together with the attributes and detection scores of the detected parts of the object.
• a machine-learned model LM enables detection of two or more types of object parts.
• the parts of the object detected by the machine-learned model LM are, for example, two or more kinds of objects selected from human face, human pupil, animal pupil, and bird pupil.
• the subject area detection unit 55A Based on the detection information output from the machine-learned model LM, the subject area detection unit 55A detects the area including the subject to be focused as the subject area from the object and the part of the object included in the detection information.
• the subject area detection unit 55A detects an object or an area including a part of the object that matches the type of the subject to be focused set using the operation unit 13 from the object and the part of the object included in the detection information as the subject area. For example, when "person's right eye” is set as the type of subject to be focused, the subject area detection unit 55A sets the area including the person's right eye as the subject area.
• the subject area detection unit 55A sets the center of the image represented by the image data PD or the area including the object or part of the object closest to the initially set AF area as the subject area.
• the attribute determination unit 55B determines attributes of the subject included in the subject area detected by the subject area detection unit 55A. Specifically, the attribute determination unit 55B determines to which of two or more types of objects the attribute of the subject corresponds, or to which part of two or more types of object parts. For example, when the subject included in the subject area detected by the subject area detection unit 55A is a pupil, it is determined whether the pupil is the pupil of a person, an animal, or a bird.
• the AF area determination unit 56 determines the AF area based on the subject area detected by the subject area detection unit 55A and the attribute determined by the attribute determination unit 55B.
• the AF area is an area for acquiring subject distance information. Note that the AF area is an example of the "second range" according to the technology of the present disclosure.
• the AF area determination unit 56 basically sets the subject area detected by the subject area detection unit 55A as the AF area, but reduces or expands the AF area based on the attribute determined by the attribute determination unit 55B. That is, the AF area determination unit 56 determines whether the size of the AF area should be smaller or larger than the subject area (that is, whether the second range should be less than the first range or greater than the first range) based on the attribute. Note that the AF area determination unit 56 may determine the AF area to have the same size as the subject area (that is, the second range to have the same size as the first range).
• the AF area determination unit 56 includes a magnification acquisition unit 56A and a magnification correction unit 56B.
• the magnification acquisition unit 56A acquires the magnification corresponding to the attribute of the subject determined by the attribute determination unit 55B by referring to the table TB stored in the memory 42.
• the magnification correction unit 56B corrects the magnification acquired by the magnification acquisition unit 56A. That is, the magnification corrector 56B corrects the size of the AF area. In this embodiment, the magnification correction unit 56B corrects the magnification using the first threshold and the second threshold. Here, the second threshold is smaller than the first threshold. When the size of the AF area multiplied by the magnification acquired by the magnification acquisition section 56A exceeds the first threshold, the magnification correction section 56B corrects the magnification so as to reduce the AF area. Further, when the size of the AF area multiplied by the magnification acquired by the magnification acquisition unit 56A is below the second threshold, the magnification is corrected so as to enlarge the AF area.
• the AF area determination unit 56 determines the size of the AF area with respect to the subject area according to the magnification acquired by the magnification acquisition unit 56A and corrected by the magnification correction unit 56B.
• the distance information acquisition section 57 performs acquisition processing for acquiring distance information of the subject in the AF area determined by the AF area determination section 56 . Specifically, the distance information acquisition unit 57 acquires a phase difference detection signal from a portion corresponding to the AF area of the image data PD output from the imaging sensor 20, and calculates the defocus amount as distance information based on the acquired phase difference detection signal.
• the defocus amount represents the amount of deviation from the in-focus position of the focus lens 31 .
• the main control unit 50 moves the position of the focus lens 31 via the lens drive control unit 34 based on the distance information calculated by the distance information acquisition unit 57, thereby performing focusing processing to bring the subject included in the AF area into focus. In this way, in the present embodiment, the focusing control of the phase difference detection method is performed.
• Exposure control is control for calculating an exposure evaluation value from image data PD and adjusting exposure (shutter speed and aperture value) based on the exposure evaluation value.
• FIG. 4 conceptually shows an example of processing by the machine-learned model LM.
• Image data PD is input to the machine-learned model LM.
• the machine-learned model LM detects an area including an object in the image data PD and an area including the parts of the object, and outputs them together with attributes and detection scores.
• the detection score represents the likelihood of the attribute of the detected object or part of the object.
• "person” and “bird” are detected as objects, and "person's face”, “person's pupil (right eye)”, “person's pupil (left eye)", and “bird's eye” are detected as parts of the object.
• the detection score is expressed as a percentage, and the closer to 100%, the more reliable the attribute determination.
• the detection score is an example of “attribute determination reliability” according to the technology of the present disclosure. Note that the detection score does not have to be displayed on the screen. Also, a detection frame whose color or shape changes based on the value of the detection score may be displayed on the screen.
• FIG. 5 conceptually shows an example of processing by the subject area detection unit 55A.
• the subject area detection unit 55A detects a subject area including a subject to be focused from a plurality of objects and parts of the objects detected by the machine-learned model LM.
• the example shown in FIG. 5 shows a case where "person's right eye” is set as the type of subject to be focused.
• the subject area detection unit 55A detects an area including the human pupil (right eye) as the subject area SR.
• the attribute determination unit 55B determines attributes of the subject included in the subject area SR. In this example, the attribute determined by the attribute determination unit 55B is "person's eyes".
• a machine-learned model LM is generated by machine-learning a machine-learning model using a large amount of teacher data in the learning phase.
• a machine learning model subjected to machine learning in the learning phase is stored in the memory 42 as a machine-learned model LM. Note that the learning process of the machine learning model is performed, for example, by an external device.
• the machine-learned model LM is not limited to being configured as software, and may be configured as hardware such as an IC chip. Also, the machine-learned model LM may be configured by an aggregate of a plurality of IC chips.
• FIG. 6 conceptually shows an example of the table TB.
• magnifications are set for attributes of various objects and parts of the objects.
• a magnification acquisition unit 56A of the AF area determination unit 56 acquires a magnification corresponding to the attribute determined by the attribute determination unit 55B from the table TB.
• the magnification acquisition unit 56A acquires the magnification "3.0" corresponding to the person's pupil.
• the magnification is determined in advance based on the difficulty of predicting the motion of the object and the size of the object or part.
• a larger magnification is basically associated with an object whose motion is more difficult to predict.
• the smaller the size of the object or the part of the object the larger the magnification.
• Objects whose movements are difficult to predict are highly likely to move out of the AF area after the next frame period if the subject area is set as the AF area. Therefore, by increasing the magnification and expanding the AF area for an object whose movement is difficult to predict, the possibility that the object will be included in the AF area even if it moves increases. Also, since the parts of the object such as the pupil are minute and the subject area is small, similarly, by increasing the magnification to enlarge the AF area, the possibility of being included in the AF area increases.
• Airplanes, trains, etc. are moving bodies that move at high speed, but in most cases they are captured from a distance and their movements are easy to predict, so the magnification is set to "1.0" so that the subject area can be used as the AF area.
• the magnification for the bird's pupil is set larger than the magnification for the person's pupil. This is because the movements of birds are more difficult to predict than those of humans, and the pupils of birds are smaller than those of humans.
• the first object is a "person" and the second object is a "bird".
• the first part is the "pupil" for both the first object and the second object.
• FIG. 7 conceptually shows an example of processing by the AF area determination unit 56.
• the AF area determination unit 56 determines the size of the AF area AR according to the magnification acquired by the magnification acquisition unit 56A and corrected by the magnification correction unit 56B. In the example shown in FIG. 7, the AF area determination unit 56 enlarges the AF area AR to three times the size of the subject area SR.
• FIG. 8 explains the first threshold used for correction processing by the magnification correction unit 56B.
• the magnification correction unit 56B compares the horizontal length LH of the AF area AR multiplied by the magnification acquired by the magnification acquisition unit 56A with a first threshold value T1H, and corrects the horizontal magnification so that LH ⁇ T1H when LH>T1H.
• the magnification correction unit 56B compares the vertical length LV of the AF area AR multiplied by the magnification acquired by the magnification acquisition unit 56A with the first threshold value T1V, and if LV>T1V, corrects the magnification in the vertical direction so that LV ⁇ T1V.
• the first threshold is set for each of the horizontal and vertical directions of the image data PD.
• the first threshold T1H is defined based on the horizontal length FH of the image data PD.
• the first threshold T1H is 70% of the length FH.
• the first threshold T1V is defined based on the vertical length FV of the image data PD.
• the first threshold T1V is 70% of the length FV.
• the processing time required for focus control becomes long. Therefore, if the AF area is larger than the first threshold, the AF area is reduced so as to shorten the processing time. Also, if the AF area is too large, there is a high possibility that an object or the like other than the subject to be focused will be included in the AF area. As described above, if an object or the like other than the subject to be focused is included in the AF area, the focusing accuracy is lowered.
• FIG. 9 conceptually explains the second threshold used by the magnification correction unit 56B for correction processing.
• the second threshold T2H in the horizontal direction and the second threshold T2V in the vertical direction are determined based on the number of phase difference detection pixels 22 in the horizontal and vertical directions. This is because if the AF area is too small, the number of the phase difference detection pixels 22 included in the AF area is reduced, thereby reducing the accuracy of distance information calculation and the focusing accuracy.
• the second threshold value T2H has a length that includes four phase difference detection pixels 22 arranged in the horizontal direction. Also, the second threshold value T2V is set to a length including two phase difference detection pixels 22 arranged in the vertical direction. Since the phase difference detection pixel 22 detects a phase difference in the horizontal direction, it is preferable that T2H>T2V.
• the magnification correction unit 56B compares the horizontal length LH of the AF area AR multiplied by the magnification acquired by the magnification acquisition unit 56A with the second threshold T2H, and corrects the horizontal magnification so that LH ⁇ T2H when LH ⁇ T2H. Similarly, the magnification correction unit 56B compares the vertical length LV of the AF area AR multiplied by the magnification acquired by the magnification acquisition unit 56A with a second threshold value T2V, and if LV ⁇ T2V, corrects the magnification in the vertical direction so that LV ⁇ T2V.
• FIG. 10 shows an example of processing by the magnification correction unit 56B.
• the horizontal length LH and the vertical length LV of the AF area AR obtained by multiplying the size of the subject area SR by the magnification obtained by the magnification obtaining unit 56A exceed the first threshold T1H and the first threshold T1V, respectively.
• the magnification correction unit 56B corrects the magnification so that LH ⁇ T1H and LV ⁇ T1V.
• magnification correction unit 56B does not perform correction processing.
• the second threshold T2H and the second threshold T2V are determined based on the number of phase difference detection pixels 22, but when the AF area is displayed on the display 15 or viewfinder 14, it may be determined based on the minimum size that the user can recognize as a rectangular area.
• FIG. 11 is a flowchart showing an example of an imaging operation by the imaging device 10.
• FIG. FIG. 11 shows a case where the AF-C mode is selected as the focusing mode and the mode in which the imaging device 10 automatically detects the subject area is selected.
• the main control unit 50 determines whether the release button has been half-pressed by the user (step S10).
• the main control unit 50 controls the imaging control unit 51 to cause the imaging sensor 20 to perform an imaging operation (step S11).
• the image data PD output from the imaging sensor 20 is input to the subject detection section 55 .
• the subject area detection unit 55A of the subject detection unit 55 uses the machine-learned model LM to perform detection processing for detecting the subject area, which is the first range including the subject to be focused, from the image data PD (step S12).
• the attribute determination unit 55B performs determination processing for determining the attribute of the subject included in the subject area detected in step S12 (step S13).
• the magnification acquisition unit 56A of the AF area determination unit 56 acquires the magnification corresponding to the attribute determined in step S13 by referring to the table TB (step S14).
• the magnification correction unit 56B performs correction processing for correcting the magnification acquired in step S14 (step S15). It should be noted that the magnification correction unit 56B does not perform correction processing when there is no need to correct the magnification.
• the AF area which is the second range, is determined according to the magnification acquired in step S14 and corrected in step S15 and the size of the first range.
• the distance information acquisition unit 57 performs acquisition processing for acquiring distance information of the subject in the AF area (step S16). Based on the distance information acquired in step S16, the main control unit 50 performs focusing processing to bring the subject included in the AF area into focus (step S17).
• the main control unit 50 determines whether or not the release button has been fully pressed by the user (step S18). If the release button is not fully pressed (that is, if the release button continues to be half-pressed) (step S18: NO), the main control unit 50 returns the process to step S11 and causes the imaging sensor 20 to perform the imaging operation again. The processing of steps S11 to S17 is repeatedly executed until the main control unit 50 determines that the release button has been fully pressed in step S18.
• step S18 When the release button is fully pressed (step S18: YES), the main control unit 50 causes the imaging sensor 20 to perform an imaging operation (step S19).
• the image recording unit 54 records the image data PD output from the imaging sensor 20 and subjected to image processing by the image processing unit 52 in the memory 42 as a recorded image PR (step S20).
• step S11 corresponds to the "imaging step” according to the technology of the present disclosure.
• Step S12 corresponds to the “detection step” according to the technology of the present disclosure.
• Step S13 corresponds to the “determining step” according to the technology of the present disclosure.
• Steps S14 and S15 correspond to the "determining step” according to the technology of the present disclosure.
• Step S15 corresponds to the "correction step” according to the technology of the present disclosure.
• Step S16 corresponds to the "acquisition step” according to the technology of the present disclosure.
• Step S17 corresponds to the "focusing step” according to the technology of the present disclosure.
• the image represented by the image data PD may be displayed on the display 15 or the viewfinder 14 while the release button is half-pressed.
• a frame representing the AF area may be displayed on the image.
• the size of this frame may be different from the size of the AF area. For example, by enlarging the frame indicating the display area, the user can easily determine whether or not there is a subject, so the frame of the display area may be displayed larger than the AF area.
• the imaging device 10 of the present disclosure it is determined whether the size of the AF area is less than the subject area or greater than the subject area based on the attribute of the subject included in the subject area to be focused. Therefore, it is possible to improve the accuracy of focusing on the subject to be focused.
• one table TB is stored in the memory 42, but a plurality of tables may be stored in the memory 42 so that the magnification acquisition unit 56A selects the table to be used to acquire the magnification.
• FIG. 12 shows an example of multiple tables stored in the memory 42 in the first modified example.
• the first table TB1 is a table for AF-C mode.
• the second table TB2 is a table for AF-S mode.
• a larger magnification is set for the same attributes than in the second table TB2. This is because the AF-C mode is generally used in scenes where the subject moves more than the AF-S mode.
• FIG. 13 shows magnification acquisition processing according to the first modified example.
• the magnification acquisition unit 56A determines whether or not the AF-C mode is set (step S140). When the AF-C mode is set (step S140: YES), the magnification acquisition unit 56A selects the first table TB1 (step S141). If the AF-C mode is not set (that is, if the AF-S mode is set) (step S140: NO), the magnification acquisition unit 56A selects the second table TB2 (step S142).
• the magnification acquisition unit 56A reads the magnification corresponding to the attribute determined in step S13 from the first table TB1 selected in step S141 or the second table TB2 selected in step S142 (step S143).
• the imaging device 10 can selectively execute the AF-C mode as the focusing mode, and in the determination processing according to this modification, the size of the AF area is changed depending on whether the focusing mode is the AF-C mode. Thereby, the size of the AF area is optimized according to the focusing mode, and the focusing accuracy is further improved.
• FIG. 14 shows magnification correction processing according to the second modification.
• the magnification correction unit 56B performs scene recognition based on the image data PD (step S150).
• the magnification correction section 56B may perform scene recognition in consideration of the attribute of the subject determined by the attribute determination section 55B. Further, during scene recognition, moving body determination is performed to determine whether or not the subject is a moving body that actually moves.
• the magnification correction unit 56B determines whether or not the subject to be focused is a moving object (step S151). If the subject to be focused is a moving object (step S151: YES), the magnification correction unit 56B corrects the magnification so as to enlarge the AF area (step S152). If the subject to be focused is not a moving object (step S151: NO), the magnification correction unit 56B does not perform correction.
• magnification correction unit 56B may correct the magnification so as to reduce the AF area when the subject to be focused is not a moving object.
• FIG. 15 shows magnification correction processing according to the third modification.
• the magnification correction unit 56B acquires a detection score for the attribute of the subject determined by the attribute determination unit 55B (step S160).
• a detection score represents the reliability of the attribute determination.
• the magnification correction unit 56B determines whether or not the detection score is equal to or less than the threshold (step S161). If the detection score is equal to or less than the threshold (step S161: YES), the magnification correction unit 56B corrects the magnification so as to enlarge the AF area (step S162). If the detection score is not equal to or less than the threshold (step S161: NO), the magnification correction unit 56B does not perform correction.
• magnification correction unit 56B may correct the magnification so as to reduce the AF area when the detection score is not equal to or less than the threshold.
• the magnification correction unit 56B may perform magnification correction processing based on the state of the subject to be focused.
• the state of the subject is the brightness of the subject, the color of the subject, and the like.
• the magnification correction unit 56B corrects the magnification so as to enlarge the AF area, for example, when the brightness of the subject is equal to or less than a certain value. This is because in a scene where the subject is dark, the focusing accuracy decreases when the AF area is small.
• the brightness of the subject can be obtained using the exposure evaluation value calculated by the main control section 50 during exposure control.
• the magnification correction unit 56B performs primary correction processing based on the magnification for determining the size of the AF area, the determination result of whether or not the subject is a moving object, the value of the detection score, or the state of the subject.
• the magnification correction unit 56B may perform secondary correction processing based on the first threshold value or the second threshold value so that the size of the AF area subjected to the primary correction processing is within the range defined by the first threshold value and the second threshold value.
• the subject detection unit 55 performs detection processing and determination processing using the machine-learned model LM.
• the main control unit 50 performs focus control of the phase difference detection method based on the phase difference detection signals output from the plurality of phase difference detection pixels 22, but the contrast detection method based on the contrast of the image data PD may be performed.
• the contrast detection method the distance information acquisition unit 57 acquires the contrast of the portion corresponding to the AF area of the image data PD as distance information.
• the technology of the present disclosure is not limited to digital cameras, and can also be applied to electronic devices such as smartphones and tablet terminals that have imaging functions.
• the following various processors can be used as the hardware structure of the control unit, with the processor 40 being an example.
• the above-mentioned various processors include CPUs, which are general-purpose processors that function by executing software (programs), as well as processors such as FPGAs whose circuit configuration can be changed after manufacture.
• FPGAs include dedicated electric circuits, which are processors with circuitry specifically designed to perform specific processing, such as PLDs or ASICs.
• the control unit may be composed of one of these various processors, or may be composed of a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs or a combination of a CPU and an FPGA). Also, the plurality of control units may be configured by one processor.
• control unit there are multiple possible examples of configuring multiple control units with a single processor.
• first example as typified by computers such as clients and servers, there is a mode in which one or more CPUs and software are combined to form one processor, and this processor functions as a plurality of control units.
• second example is the use of a processor that implements the functions of the entire system including multiple control units with a single IC chip, as typified by System On Chip (SOC).
• SOC System On Chip
• an electric circuit combining circuit elements such as semiconductor elements can be used.

Landscapes

• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Optics & Photonics (AREA)
• Studio Devices (AREA)

Priority Applications (3)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=87348092

Family Applications (1)

Country Status (4)

Families Citing this family (1)

Citations (1)

• 2022
  • 2022-12-06 JP JP2023575114A patent/JPWO2023139954A1/ja active Pending
  • 2022-12-06 WO PCT/JP2022/044973 patent/WO2023139954A1/ja not_active Ceased
  • 2022-12-06 CN CN202280089030.7A patent/CN118633295A/zh active Pending
• 2024
  • 2024-06-30 US US18/759,986 patent/US20240357233A1/en active Pending

Patent Citations (1)

Also Published As

Similar Documents

Legal Events