JP2020053756A - Imaging apparatus and control method therefor - Google Patents

Abstract

To provide an imaging apparatus capable of detecting a motion vector accurately.SOLUTION: A photoelectric sensor 208 included in an imaging apparatus can acquire image data of continuous frames, at the time of finder imaging using an optical view finder. When calculating brilliance of a subject, a camera control section 201 performs photoelectric operation, and flicker detection operation in first control. In second control, the camera control section 201 performs photoelectric operation, and vector detection operation for detecting a motion vector of the subject between images of continuous frames acquired by the photoelectric sensor 208. The camera control section 201 and a lens control section (LPU401) calculate an angular speed of the subject for the imaging apparatus by using detection results of the motion vector, and detection information of panning the angular speed of the imaging apparatus detected by an angular speed sensor 402, and performs image blur correction of the subject at the time of panning by controlling drive of a correction lens in an imaging optical system.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an imaging device for detecting a motion vector of a subject and a control method thereof.

  Panning is a method of shooting at a slow shutter speed while adjusting the movement of the camera to a moving subject. By moving the background while the main subject is stationary, it is possible to take a photograph with a lively feeling. However, in panning, it is difficult to match the moving speed of the subject with the angular velocity at which the camera is moved (hereinafter, also referred to as panning angular velocity). When a beginner actually performs a panning shot, there is a high possibility that the subject does not stop and blur remains.

  Patent Literature 1 discloses an imaging apparatus having a panning assist control unit that assists a user in panning. The panning assist control means corrects the image blur (subject blur) of the main subject based on the difference between the main subject angular velocity and the panning angular velocity by drive control of the shift lens.

JP-A-2006-171541

In the related art disclosed in Patent Literature 1, it is assumed that a sensor that detects a motion vector is an imaging sensor for photographing. For this reason, for example, when an optical viewfinder (hereinafter, referred to as OVF) is photographed through a pentamirror on the front surface of an image sensor, a motion vector cannot be detected, and an image blur correction can be performed by detecting a subject area in an image from the motion vector. Absent. If the detection accuracy of the motion vector decreases due to a change in the state of the imaging optical system or the like, processing and control using the motion vector may be hindered.
An object of the present invention is to provide an imaging device capable of accurately detecting a motion vector.

  An apparatus according to an embodiment of the present invention includes an imaging unit that obtains image data of a continuous frame, a calculation unit that calculates a luminance of a subject from the image data, a first control that detects flicker by the imaging unit, A control unit that switches between a second control for detecting a motion vector of a subject between images of a plurality of frames acquired by the imaging unit, a first detection unit for detecting an angular velocity of panning or tilting of the imaging device, Second detecting means for detecting the motion vector by the second control. In the second control, the control unit performs a photometric operation by the arithmetic unit using the image data acquired by the image capturing unit, and performs the photometric operation under the same conditions as the image data used in the photometric operation. Using the acquired image data, control is performed to detect a motion vector by the second detection means.

  According to the present invention, it is possible to provide an imaging device capable of accurately detecting a motion vector.

1 is a block diagram illustrating a basic configuration of an imaging device according to an embodiment of the present invention. FIG. 2 is a diagram illustrating a configuration example of an imaging device according to the present embodiment. 5 is a flowchart illustrating a process according to the embodiment. 5 is a timing chart of a photometric calculation and a flicker calculation. 6 is a timing chart of a photometric calculation and a motion vector detection calculation. 4 is a flowchart illustrating a subject extraction process in FIG. 3. FIG. 3 is a diagram illustrating calculation of an angular velocity of a subject.

  Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings. In the present embodiment, a digital camera having a panning assist function for assisting panning will be exemplified. The control mode in which the panning assist setting is performed is referred to as “panning assist mode”.

  FIG. 1 is a block diagram illustrating a basic configuration of an imaging device 100 according to the present embodiment. The imaging device 100 is a digital camera, a digital video camera, or a mobile phone or a computer having an imaging function. The present invention is applicable to any electronic device having an imaging function.

  The imaging optical system 101 includes optical members such as a lens, a shutter, and an aperture, and forms light from a subject on the imaging element 102 under the control of a CPU (Central Processing Unit) 104. The imaging device 102 is a CCD (charge coupled device) type image sensor, a CMOS (complementary metal oxide semiconductor) type image sensor, or the like, and photoelectrically converts light formed through the imaging optical system 101 into an image signal. The focus detection circuit 103 performs focus detection using, for example, a phase difference detection method, and outputs a focus detection signal to the CPU 104.

  The CPU 104 controls each unit configuring the imaging apparatus 100 according to an input signal or a program stored in advance. The primary storage device 105 is a volatile storage device such as a RAM (random access memory), stores temporary data, and is used as a working memory of the CPU 104. The data stored in the primary storage device 105 is used for detecting a motion vector and a main subject area, and is recorded on the recording medium 108.

  The angular velocity sensor 106 is a shake detection sensor such as a gyro sensor, and detects a shake, a panning operation, and the like of the imaging device 100. The angular velocity sensor 106 is provided in the main body of the imaging device 100 or a lens device attachable to the main body, and outputs an angular velocity detection signal to the CPU 104.

  The image processing unit 107 processes data of an image captured by the image sensor 102 and outputs a processing result to the primary storage device 105 for storage. The image processing unit 107 performs a noise reduction process, various correction processes, a development process, and the like.

  The recording medium 108 records image data and the like obtained by shooting. The primary storage device 105 stores captured image data, and the captured image data is recorded on the recording medium 108 in response to a user instruction or the like. The recording medium 108 can be attached to and detached from the main body of the imaging device 100, for example, like a semiconductor memory card. The user can attach the recording medium 108 to an information processing device such as a personal computer and read out the recorded photographic image data. That is, the main body of the imaging apparatus 100 includes a mechanism for attaching and detaching the recording medium 108, and the CPU 104 has a function of reading from and writing to the recording medium 108.

  The secondary storage device 109 is, for example, a non-volatile storage device such as an electrically erasable programmable read-only memory (EEPROM). The secondary storage device 109 stores a program (firmware) for controlling the imaging device 100 and various kinds of setting information, and is used by the CPU 104.

  The display unit 110 includes a display device such as a liquid crystal display panel, and performs various displays according to a control command of the CPU 104. The various displays are a viewfinder image display at the time of shooting, a shot image display, a GUI (Graphical User Interface) screen for interactive operation, and the like.

  The operation unit 111 includes an input device that receives a user operation and transmits operation instruction information to the CPU 104. The operation unit 111 includes input devices such as buttons, switches, levers, and touch panels, and input devices using voice, eyes, and the like. For example, the operation unit 111 includes first and second switches that are turned on / off in response to operation of a release button. The user performs a half-press operation of the release button, and turns on a first switch (SW1) of the release switch. Thereby, it is possible to instruct the imaging apparatus to start operations such as AF (autofocus) processing, AE (automatic exposure) processing, AWB (auto white balance) processing, and EF (strobe pre-flash) processing. The user further performs a full-press operation of the release button, turns on the second switch (SW2) of the release switch, and can instruct the start of the photographing operation. The operation unit 111 includes a switch for setting the panning assist mode.

  The motion vector detection unit 112 detects a motion vector between frames by using a plurality of image data obtained at different imaging times obtained by imaging. The detected motion vector is output to the CPU 104. The main subject area detection unit 113 detects a main subject area from an image obtained by imaging, and outputs data of a detection result to the CPU 104.

  FIG. 2 is a block diagram illustrating a configuration example of the imaging device of the present embodiment. 1 illustrates an example of an imaging system including a main body 200 of an imaging device and a lens device 400. First, the configuration inside the main body 200 will be described. In the following, the positional relationship between the components will be described with the subject side defined as the front side. The camera control unit 201 includes a CPU and controls each unit of the imaging device. The memory 202 is a RAM, a ROM (Read Only Memory) connected to the camera control unit 201, or the like.

  The imaging device 203 is an image sensor including an infrared cut filter, an optical low-pass filter, and the like. An image of a subject is formed on the imaging surface by the lens device 400 at the time of shooting. The shutter 204 shields the image sensor 203 when light is not captured, and opens to guide light rays to the image sensor 203 during image capture. The half mirror 205 is an optical member that reflects a part of light incident from the lens device 400 and forms an image on the focus plate 206. The display element 207 is a device that uses a polymer network (PN) liquid crystal or the like to display AF ranging points. The display element 207 indicates at which position the AF process is being performed when the user looks into the OVF.

  The photometric sensor 208 is configured by an imaging device such as a CCD or a CMOS, and performs photometric detection. The pentaprism 209 is an optical member that guides the subject image on the focus plate 206 to the photometric sensor 208 and an OVF (not shown). The photometric sensor 208 sees the subject image formed on the focus plate 206 from an oblique position via the pentaprism 209, and receives the light via the pentaprism 209. In the present embodiment, a pentaprism is illustrated, but a penta roof mirror may be used. In this case, the half mirror 205 reflects a part of the light from the subject during the finder shooting, and guides the light to the photometric sensor 208 via the penta roof mirror.

  The focus detection circuit 210 receives the light reflected from the AF mirror 211 and performs focus detection. The AF mirror 211 is located behind the half mirror 205, and guides a part of the light incident from the lens device 400 and passing through the half mirror 205 to the focus detection circuit 210. The focus detection circuit 210 includes an AF sensor inside and acquires focus detection information.

  A CPU (hereinafter, referred to as an APU) 212 for image processing and calculation is connected to the camera control unit 201 and the photometric sensor 208. The memory 213 is a RAM, a ROM, or the like connected to the APU 212. The APU 212 obtains an output signal of the photometric sensor 208 and performs AE calculation and the like. In the present embodiment, the APU dedicated to the photometric sensor is used, but the processing performed by the APU may be performed by the camera control unit 201. Further, the camera control unit 201 or the APU 212 may include the motion vector detection unit 112 and the main subject region detection unit 113 in FIG.

  Next, the configuration of the lens device 400 will be described. The lens device 400 is an interchangeable lens that can be mounted on the main body 200. A CPU (hereinafter, referred to as an LPU) 401 that controls each unit in the lens device 400 transmits, for example, information on a distance from an imaging device to a subject and information on angular velocity detection to the camera control unit 201. The angular velocity sensor 402 is a gyro sensor or the like, detects the angular velocity of the shake, converts the angular velocity information into an electric signal, and outputs the electric signal to the LPU 401.

  In the lens device 400, a plurality of lenses, an aperture, and the like that constitute the imaging optical system are arranged. The LPU 401 acquires a shake detection signal from the angular velocity sensor 402, and performs image blur correction of a subject by controlling driving of a correction lens (such as a shift lens) in the imaging optical system.

  An imaging sequence in the panning assist mode in the present embodiment will be described with reference to the flowchart in FIG. It is assumed that the subject (moving object) is performing a linear motion at a constant velocity, and the user moves (pans) the imaging apparatus to follow the movement of the subject to perform a panning shot. Further, the flowchart of FIG. 3 shows a photographing sequence from a state where the photographer is observing the subject with OVF.

  In step S301, the camera control unit 201 determines whether or not the photographer performs a half-press operation of the release switch to turn on the first switch SW1. If it is determined that SW1 has been turned on, the process proceeds to S302. If SW1 is off, the determination processing of S301 is repeated.

  In step S302, the camera control unit 201 detects a motion vector using the photometric sensor 208, and extracts a subject region in the image. When the user is observing the subject with the OVF, the light from the subject is not guided to the image sensor 203 in the mirror-down state, so the photometric sensor 208 is used. As for the extraction of the subject area, data of an image captured by the photometric sensor 208 is input to the motion vector detection unit 112 in the camera control unit 201 or the APU 212. A process of extracting a subject region using the calculated motion vector is performed. The detailed processing will be described later with reference to FIG.

  If the subject area cannot be extracted in S302, the subject angular velocity cannot be calculated, so that the panning assist function cannot be realized. In this case, it is possible to switch to normal shooting in which a correction lens (such as a shift lens) is not driven in S309 described later.

  Next, in step S303, the camera control unit 201 acquires an angular velocity signal (a signal corresponding to a panning angular velocity) detected by the angular velocity sensor 402, and proceeds to step S304. In step S304, the camera control unit 201 calculates the subject angular velocity. Details of the subject angular velocity calculation process will be described later.

  In step S305, the camera control unit 201 stores the subject angular velocity calculated in step S304 in the memory 213 as history data. When the subject angular velocity is calculated from the image captured by the image sensor 203, the memory 202 is used. When the subject angular velocity is calculated from the image captured by the photometric sensor 208, the memory 213 is used. Shall be used. Note that the use of such a memory is merely an example, and can be freely changed according to the embodiment.

  In step S306, the camera control unit 201 determines whether or not the photographer performs a full-press operation of the release switch and the second switch SW2 is turned on. If it is determined that the switch SW2 has been turned on, the process advances to step S307 to start an exposure operation. If it is determined that SW2 is off, the process returns to S302.

  In step S307, the camera control unit 201 executes a process of estimating the subject angular velocity at the time of exposure. It is known that, even when the subject is performing a uniform linear motion, if the calculation is performed as the angular velocity of the subject viewed from the imaging apparatus, the subject performs an acceleration motion. Therefore, the subject angular velocity at the time when the second switch SW2 is turned on is different from the subject angular velocity at the time of exposure. In S307, a process of predicting the subject angular velocity at the time of exposure based on the subject angular velocity at the time when the second switch SW2 is turned on, the time lag from the time to the exposure operation, and the historical data of the subject angular velocity stored in S305. Is performed.

  Next, in step S308, the camera control unit 201 controls the shutter 204 after raising the half mirror 205, and starts the shutter travel. In step S309, the camera control unit 201 controls the driving of the correction lens (shift lens) included in the imaging optical system 101 to perform panning assist. By driving the correction lens based on the angular velocity of the subject at the time of exposure, image blurring of the subject during panning is corrected. In S310, the camera control unit 201 determines whether or not the exposure time set based on the AE calculation has elapsed. While the exposure time set in this photographing has not elapsed, drive control of the correction lens is performed in S309. If it is determined that the set exposure time has elapsed, a series of processing ends.

  The subject extraction processing shown in S302 of FIG. 3 will be described with reference to FIGS. First, a normal AE operation will be described with reference to FIG. FIG. 4 is an explanatory diagram of camera control, AE sensor control, and calculation processing, and shows an operation during photometry when panning assist is not performed. In the AE process at the time of normal photographing, the photometric sensor 208 alternates between accumulation for acquiring luminance information of a subject (hereinafter, referred to as AE accumulation) and accumulation for detecting flicker (hereinafter, referred to as flicker detection accumulation). Is repeated. After the AE accumulation is performed after the start of the photometry and the signal is read out, the photometry calculation is executed. The flicker detection is performed to detect the presence of a flicker light source such as a fluorescent lamp. After the flicker detection and accumulation is performed and the signal is read, the flicker calculation is performed. In this case, since the AE accumulation and the flicker detection accumulation have different image sizes and photographing parameters, it is not possible to detect a motion vector from the images respectively acquired by the AE accumulation and the flicker detection accumulation. In other words, in order to perform motion vector detection while driving during normal shooting, it is necessary to perform motion vector detection from images obtained by AE accumulation or flicker detection accumulation. In this case, the frame rate when performing motion vector detection decreases, and the detection accuracy decreases.

  In the present embodiment, the flicker detection and accumulation are not performed in the panning assist mode, and the photometric sensor 208 performs accumulation for detecting a motion vector (hereinafter, referred to as vector detection accumulation). This will be specifically described with reference to FIG.

  FIG. 5 is an explanatory diagram of camera control, AE sensor control, AE calculation processing, and vector detection control, and shows the operation of photometry and motion vector detection when panning assist is performed. After the AE accumulation is performed after the start of the photometry and the signal is read out, the photometry operation and the motion vector detection operation are executed. Next, after the vector detection and accumulation are performed and the signal is read out, the motion vector detection calculation is performed. In the panning assist mode, the photometric calculation is performed at the first frame rate, whereas the motion vector detection calculation is performed at a second frame rate higher than the first frame rate. That is, since the motion vector is detected after the AE accumulation and after the vector detection accumulation except for the first time, it is possible to further improve the vector detection accuracy between the images of the continuous frames.

  The subject extraction process (FIG. 3: S302) will be described in detail with reference to the flowchart in FIG. In step S601, the camera control unit 201 determines whether to perform AE accumulation. The determination of AE accumulation is performed based on an AE accumulation request from the outside, a photometry cycle specified by the user, and the like. In the present embodiment, there is no limitation on the method of determining AE accumulation. If it is determined that AE accumulation is to be performed, the process proceeds to S602. If it is determined that AE accumulation is not to be performed, the process proceeds to S608.

  In step S <b> 602, the camera control unit 201 sets an accumulation parameter of the photometric sensor 208. The accumulation parameter includes an image size, a gain, an accumulation time, and the like, and varies depending on an image sensor used. In the next step S603, a process of storing the parameters set in S602 in the memory 202 (or 213) is performed.

  In step S604, AE accumulation of the photometric sensor 208 starts based on the parameters set in step S602. When the AE accumulation is completed, the process proceeds to S605, where a photometric calculation is performed. There are various methods of photometric calculation, such as a method of averaging the brightness of the entire captured image, and a method of performing calculation by weighting the center of the image. After S605, the process proceeds to S606.

  If it is determined in step S601 that AE accumulation is not to be performed, the process advances to step S608 to read out accumulation parameters. The stored parameters are obtained by reading the parameters stored in the memory 202 (or 213) in S603. That is, when performing AE accumulation, an accumulation parameter equivalent to the parameter set in S602 is used. In the next step S609, vector detection accumulation is performed under the same conditions as AE accumulation based on the accumulation parameters read in S608. The processing of S608 and S609 is executed in the panning assist mode under the control of the camera control unit 201 (or APU 212). After S609, the process proceeds to S606.

  In step S606, the camera control unit 201 performs motion vector detection using the image data accumulated and read in steps S604 and S609. A motion vector of a subject can be detected from a plurality of pieces of image data having different imaging times.

  In step S607, the camera control unit 201 extracts a subject area in the image using the motion vector detected in step S606. Since various methods such as template matching are known as a method of extracting a subject region using a motion vector, a detailed description thereof will be omitted. After the process in S607, the process shifts to the return process.

  Next, the calculation processing of the subject angular velocity shown in S304 of FIG. 3 will be described with reference to FIG. In the present embodiment, since the calculation is performed together with the panning angular velocity, a process of calculating the subject angular velocity centering on the principal point is performed. The method of calculating the angular velocity will be described with the radian method.

FIG. 7 is a diagram schematically illustrating a situation in which the subject moves from point A to point B in t seconds, and the subject image formed on the image plane of the image sensor moves from point C to point D accordingly. It is. Here, the distance between the point C and the point D is d [pixel], the focal length is f [mm], and the pixel pitch of the image sensor is p [μm / pixel]. If the angular velocity of the subject on the image plane is expressed as ω [rad / sec], ω can be expressed by the following equation.

Assuming that the photographer is performing a panning operation of the imaging apparatus, the subject itself of the angular velocity (subject angular velocity) is expressed as omega _s with respect to the imaging device, it referred to the panning velocity and omega _p. The angular velocity ω of the subject on the image plane is obtained by subtracting ω _p from ω _s as in the following equation.

Accordingly, the subject angular velocity ω _s is calculated by adding the panning angular velocity ω _p of the imaging device detected by the angular velocity sensor 402 to the angular velocity ω of the subject on the image plane as in the following equation.

The calculation processing of the subject angular velocity ω _s is performed after the motion vector detection calculation is performed, and the angular velocity of the subject corresponding to the detected motion vector is obtained.

  In the present embodiment, by replacing the flicker detection accumulation with the vector detection accumulation in the panning assist mode (FIG. 5), it is possible to achieve both motion vector detection at a higher frame rate and photometry. According to the present embodiment, the motion vector detection and the photometry are performed by the photometry sensor at the time of the photographing using the optical viewfinder, and the panning assist can be realized.

  Although the example of the panning assist at the time of the user's panning operation has been described, the present invention is also applicable at the time of the tilting operation. Processing using a motion vector calculated between images of continuous frames includes subject tracking processing and auto zoom processing in addition to panning assist. Further, in the above-described embodiment, an example has been described in which flicker detection and accumulation is replaced with vector detection and accumulation in a configuration in which flicker detection and accumulation are performed by the photometric sensor. Is also applicable. For example, in a configuration in which photometry and flicker detection and accumulation are performed by an image sensor, flicker detection and accumulation may be replaced with vector detection and accumulation in the follow shot assist mode. In such a configuration, since a motion vector can be detected using an image sensor that acquires an image for recording, processing and control using a highly accurate motion vector can be performed. Although the preferred embodiment of the present invention has been described above, the present invention is not limited to the above embodiment, and various modifications and changes can be made within the scope of the gist.

201: Camera control unit 203: Image sensor 208: Photometric sensor 212: APU
402: angular velocity sensor

Claims (7)

Imaging means for acquiring image data of continuous frames;
Calculating means for calculating the brightness of the subject from the image data;
Control means for switching between first control for detecting flicker by the imaging means and second control for detecting a motion vector of a subject between images of a plurality of frames acquired by the imaging means;
First detection means for detecting an angular velocity of panning or tilting of the imaging device;
And a second detection means for detecting the motion vector in the second control,
In the second control, the control unit performs a photometric operation by the arithmetic unit using the image data acquired by the image capturing unit, and performs the photometric operation under the same conditions as the image data used in the photometric operation. An imaging apparatus, comprising: performing control to detect a motion vector by the second detection unit using acquired image data. The control means sets, in the second control, parameters used for the first accumulation of the imaging means and stores the parameters in a storage means, and controls the photometric calculation and the motion vector detection after the first accumulation. The second storage of the imaging unit reads out a parameter from the storage unit, and performs the control of the motion vector detection after the second storage using the parameter. Imaging device. The photometric operation is performed at a first frame rate under the second control, and the motion vector is detected at a second frame rate higher than the first frame rate. The imaging device according to claim 1. Correction means for correcting image blur;
Calculating means for calculating an angular velocity of the subject with respect to the imaging device from the angular velocity detected by the first detecting means and the motion vector detected by the second detecting means;
A driving control unit for calculating an angular velocity of the subject at the time of shooting using the history data of the angular velocity of the subject calculated by the calculating unit and driving the correcting unit, thereby correcting an image blur related to the subject at the follow shot The imaging device according to any one of claims 1 to 3, further comprising: Equipped with a mirror that guides light from the subject to the optical viewfinder,
The imaging device according to claim 1, wherein the imaging unit photoelectrically converts the light guided by the mirror to obtain image data. It has setting means to set a mode to support panning,
The imaging apparatus according to claim 4, wherein the control unit switches to the second control when the mode is set. A control method executed by the imaging device,
A step of acquiring image data of a continuous frame by an imaging unit;
Calculating the brightness of the subject from the image data by a calculating means;
A step of switching between a first control for detecting flicker by the imaging unit and a second control for detecting a motion vector of a subject between images of a plurality of frames acquired by the imaging unit;
Detecting the angular velocity of panning or tilting of the imaging device, and detecting the motion vector in the second control,
In the second control, a photometric calculation is performed by the calculation unit using the image data obtained by the imaging unit, and the image data obtained by the imaging unit under the same conditions as the image data used for the photometry calculation. Controlling the motion vector by using the control method.

Images