JP7357596B2 - Image processing device, image processing method, image processing program, and imaging device - Google Patents

Description

The present invention relates to an image processing device, an image processing method, an image processing program, and an imaging device, and particularly relates to an image processing device, an image processing method, an image processing program, and an imaging device that process moving images captured by the imaging device.

2. Description of the Related Art Techniques for estimating a main subject within a screen based on motion vectors between frames in a moving image are known (for example, Patent Documents 1 to 4).

Japanese Patent Application Publication No. 2012-160780 Japanese Patent Application Publication No. 2020-95673 Japanese Patent Application Publication No. 2009-65573 Japanese Patent Application Publication No. 2010-272953

One embodiment of the technology of the present disclosure provides an image processing device, an image processing method, an image processing program, and an imaging device that can estimate a tracking target in a short time.

(1) An image processing device that processes moving images captured by an imaging device, and includes a processor, and the processor calculates motion vectors from the moving images in chronological order within a specific period, and changes in motion vectors. An image processing device that performs processing for calculating amounts in chronological order and processing for estimating a tracking target based on the amount of change in a motion vector.

(2) The image processing device according to (1), wherein the processor estimates the tracking target based on a change tendency of the amount of change in the motion vector.

(3) The image processing device according to (2), wherein the processor estimates a subject whose motion vector has a decreasing tendency to be a subject to be tracked.

(4) The image processing device according to (3), wherein the processor estimates a subject whose motion vector decreases over a plurality of frames as a tracking target.

(5) The image processing device according to any one of (1) to (4), wherein the processor starts the specific period based on the amount of movement of the imaging device.

(6) The processor further performs a process of calculating the average value of the amount of change in the motion vector within a specific period, and the processor determines a subject to be tracked for which the average value of the amount of change in the motion vector exceeds the threshold value in the negative direction. The image processing device according to any one of (1) to (5).

(7) The image processing device according to (6), wherein the processor further performs a process of setting a threshold value based on the amount of movement of the imaging device when capturing the moving image.

(8) The image processing device according to (7), wherein the processor sets the threshold value to be larger in the negative direction as the amount of movement of the imaging device becomes larger.

(9) If the amount of movement of the imaging device exceeds the reference value, the processor sets the threshold to the first threshold, and if the amount of movement of the imaging device is less than or equal to the reference value, the processor sets the threshold to be smaller than the first threshold in the negative direction. The image processing device according to (8), which sets the second threshold value.

(10) The image processing device according to any one of (1) to (9), wherein the processor further performs a process of setting a specific period based on the amount of movement of the imaging device when capturing the moving image.

(11) The image processing device according to (10), wherein the processor sets the specific period to be longer as the amount of motion of the imaging device increases.

(12) If the amount of movement of the imaging device exceeds the reference value, the processor sets the specific period to the first specific period, and if the amount of movement of the imaging device is less than or equal to the reference value, the processor sets the specific period to be less than the first specific period. The image processing device according to (11), which is set to a short second specific period.

(13) The image processing device according to any one of (1) to (12), wherein the processor further performs a process of calculating the amount of motion of the imaging device based on the moving image.

(14) The imaging device includes a sensor that detects the amount of movement of the main body, and the processor further performs a process of acquiring information about the amount of movement of the imaging device from the sensor. Image processing device.

(15) An imaging device comprising the image processing device according to any one of (1) to (14) and a focus adjustment unit that adjusts the focus by moving a focus lens so as to focus on the estimated tracking target. .

(16) An image processing method for processing a moving image captured by an imaging device, which includes the steps of calculating a motion vector from the moving image in a chronological order within a specific period, and calculating an amount of change in the motion vector in a chronological order. and estimating a tracking target based on the amount of change in a motion vector.

(17) An image processing program that processes moving images captured by an imaging device, which has a function of calculating motion vectors from moving images in a chronological order within a specific period and calculating the amount of change in the motion vector in a chronological order. An image processing program that causes a computer to realize a function and a function of estimating a tracking target based on the amount of change in a motion vector.

Block diagram showing an example of the hardware configuration of an imaging device Block showing an example of the functions realized by the CPU when performing tracking AF Block diagram of the functions of the tracking target setting section Diagram showing an example of time-series images obtained by imaging An image displayed by superimposing the two images in chronological order shown in Figure 4 A diagram showing an example of the distribution of motion vectors obtained from the two images in chronological order shown in Figure 4. Graph showing an example of changes in the amount of change in the motion vector of the tracking target Graph showing an example of changes in the amount of change in background motion vectors Flowchart showing the procedure for setting the tracking target Graph showing another example of changes in the amount of change in the motion vector of the tracking target Block diagram of the functions of the tracking target setting section Block diagram of the functions of the tracking target setting section when a shake detection sensor is provided Block diagram of the functions of the tracking target setting unit when automatically starting tracking target estimation processing Flowchart showing the processing procedure when starting tracking target setting processing based on the amount of movement of the imaging device

Hereinafter, preferred embodiments of the present invention will be described in detail with reference to the accompanying drawings.

Here, an example will be described in which a tracking target is automatically set in an imaging apparatus equipped with a tracking AF (Auto Focus) function.

Tracking AF is one of the AF functions of an imaging device, and is a function of tracking a moving subject and keeping it in focus. In tracking AF, it is first necessary to set a subject to be tracked. However, if the subject to be tracked is moving from the beginning, it is difficult for the user to make this setting manually (by button operation, etc.). Therefore, in such a case, it is desirable to be able to automatically set the subject to be tracked on the device side.

In this embodiment, an imaging device equipped with tracking AF and capable of automatically setting a tracking target will be described.

[Hardware configuration of imaging device]
FIG. 1 is a block diagram showing an example of the hardware configuration of an imaging device.

The imaging device 10 of the present embodiment includes an imaging optical system 12, an imaging optical system driving section 14, an imaging section 16, a display section 18, a storage section 20, a connection section 22, an operation section 24, a CPU (Central Processing Unit) 26, It includes a ROM (Read Only Memory) 28, a RAM (Random Access Memory) 30, and the like.

The imaging optical system 12 includes a plurality of lens groups that form a subject image on an image sensor of the imaging unit 16, an aperture 12B that adjusts the amount of light, and the like. The focus of the imaging optical system 12 is adjusted by moving some of the lens groups or the entire lens back and forth along the optical axis. The lens group for focus adjustment constitutes a focus lens 12A in the imaging optical system 12.

The imaging optical system drive unit 14 includes a focus lens drive unit 14A that drives the focus lens 12A, and an aperture drive unit 14B that drives the aperture 12B. The focus lens drive unit 14A includes an actuator that drives the focus lens 12A and a drive circuit thereof. The aperture drive unit 14B includes an actuator that drives the aperture 12B and its drive circuit.

The imaging unit 16 includes an imaging element 16A that converts an optical image into an electrical signal. As the image sensor 16A, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor or a CCD (Charged Coupled Device) image sensor having a color filter in a predetermined array (eg, Bayer array, etc.) is used. Note that it is not necessary to necessarily provide a color filter. In addition, an organic thin film image sensor or the like can also be used as the image sensor 16A.

In the imaging device 10 of this embodiment, a CMOS color image sensor including a drive section, an ADC (Analog to Digital Converter), a signal processing section, etc. is used as the imaging element 16A. In this case, the image sensor 16A is driven by a built-in drive section to operate. Further, the signal of each pixel is converted into a digital signal by a built-in ADC. Further, the signal of each pixel is subjected to processing such as correlated double sampling processing, gain processing, correction processing, etc., as necessary, by a built-in signal processing section. These signal processes may be performed before the signal of each pixel is converted into a digital signal, or may be performed after the signal of each pixel is converted into a digital signal.

Further, in the imaging device 10 of the present embodiment, an imaging element incorporating a phase difference pixel is used as the imaging element 16A. By using an image sensor incorporating phase difference pixels, information on the phase difference of a subject within the screen can be acquired. Furthermore, the direction and amount of focus shift (defocus amount) of the subject within the screen can be detected from the acquired phase difference information. 2. Description of the Related Art An image sensor incorporating a phase difference pixel and a method for detecting a phase difference using the image sensor are well known. Therefore, detailed explanation thereof will be omitted.

The display unit 18 is configured with a display such as an LCD (Liquid Crystal Display) or an organic EL display (Organic Light Emitting Diode display, OLED display). Moreover, a touch panel display equipped with a touch panel can also be used as the display constituting the display section 18. The display section 18 also includes one in the form of an EVF (Electronic View Finder). However, when the display section 18 is configured with an EVF, the display section 18 is not equipped with a touch panel.

The storage unit 20 is used to store captured image data as well as various data. The storage unit 20 includes, for example, EEPROM (Electrically Erasable Programmable Read-only Memory) including Flash Memory, SSD (Solid State Drive), and Flash Memory. It is composed of non-volatile semiconductor memory such as. The storage unit 20 may be integrally provided in the device body (in the form of a so-called built-in memory), or may be removable (in the form of a so-called memory card).

The connection unit 22 includes a terminal for connecting an external device (for example, an external display device, an external recording device, etc.). As the communication standard, for example, HDMI (High-Definition Multimedia Interface) can be adopted (HDMI is a registered trademark).

The operation unit 24 includes various operation devices for operating the imaging device 10. The operation device includes various operation buttons such as a power button, a shutter button, and a recording button. Further, when the display forming the display unit 18 is a touch panel display, the touch panel is also included in the operation device. The imaging apparatus 10 of this embodiment includes a tracking AF switch that turns tracking AF on and off as an operating device. Note that tracking AF can also be turned on and off on a setting screen.

CPU26 is an example of a processor. The CPU 26 functions as a control unit, an image processing unit, etc. of the imaging device 10 by executing a predetermined program. As a control unit, the CPU 26 performs processing such as automatic exposure (AE) control, imaging control such as AF control, display control on the display unit 18, and recording control on the storage unit 20. Further, the CPU 26 acts as an image processing unit and performs processing such as generation of image data for display and generation of image data for recording. Image data for display and image data for recording are generated by performing predetermined image processing (so-called development processing) on RAW data (unprocessed moving image data output from the imaging unit 16). The image processing here includes offset processing, gamma correction processing, demosaic processing, RGB/YCrCb conversion processing, white balance processing, and the like. Since these are known processes, detailed explanation thereof will be omitted. Further, the CPU 26, as an image processing unit, performs tracking target setting processing when performing tracking AF. This point will be discussed later.

The above-mentioned processes performed by the CPU 26 may be partially or entirely performed by hardware provided inside the CPU 26.

The ROM 28 stores programs executed by the CPU 26 and various data necessary for control and the like. Note that the memory constituting the ROM 28 includes an EEPROM including Flash Memory.

The RAM 30 is used as a work area when the CPU 26 performs various processes.

[Tracking AF function]
FIG. 2 is a block diagram showing an example of the functions realized by the CPU when performing tracking AF.

As shown in the figure, when performing tracking AF, the CPU 26 functions as a tracking target setting unit 100, a tracking target detection unit 110, a focus area setting unit 112, a phase difference detection unit 114, a focus lens drive control unit 116, etc. Realize.

The tracking target setting unit 100 sets a tracking target. In the imaging device 10 of this embodiment, a main subject is estimated based on image data (image data of each frame in a moving image) sequentially output from the imaging unit 16, and set as a tracking target. Details of the tracking target setting unit 100 will be described later.

The tracking target detection unit 110 sequentially detects tracking targets from image data (image data of each frame in a moving image) sequentially output from the imaging unit 16. Detection is performed for the subject set by the tracking target setting unit 100. For detection, a known method such as template matching is employed.

The focus area setting unit 112 sets a focus area based on the detection result of the tracking target detection unit 110. Specifically, a rectangular area including the tracking target detected by the tracking target detection unit 110 is set as the focus area.

The phase difference detection unit 114 detects the phase difference of the subject within the focus area set by the focus area setting unit 112. The phase difference detection unit 114 detects the phase difference of the subject within the focus area based on the image data obtained from the imaging unit 16. More specifically, the phase difference of the subject within the focus area is detected based on the signal of the phase difference pixel obtained from the image data.

The focus lens drive control section 116 controls the drive of the focus lens drive section 14A based on the detection result of the phase difference by the phase difference detection section 114. Specifically, the drive amount of the focus lens 12A required to focus on the subject within the focus area is calculated based on the phase difference detection result, and the focus lens drive unit 14A is driven with the calculated drive amount. .

In the imaging device 10 of this embodiment, the tracking target detection unit 110, the focus area setting unit 112, the phase difference detection unit 114, and the focus lens drive control unit 116 move the focus lens so as to focus on the estimated tracking target. It functions as a focus adjustment section that adjusts the focus by adjusting the focus.

[Tracking target setting section]
As described above, the tracking target setting unit 100 estimates the main subject from the image data (image data of each frame in the moving image) sequentially output from the imaging unit 16, and sets it as the tracking target. In the imaging device 10 of this embodiment, the tracking target setting unit 100 is an example of an image processing device.

FIG. 3 is a block diagram of the functions of the tracking target setting section.

As shown in the figure, the tracking target setting unit 100 has the functions of a motion vector calculation unit 100A, a tracking target candidate extraction unit 100B, a change amount calculation unit 100C, and a tracking target estimation unit 100D. The subject estimated by the tracking target estimation unit 100D is set as the tracking target. The CPU 24 implements each function by executing a predetermined program (image processing program).

The motion vector calculation unit 100A sequentially processes image data (image data of each frame in a moving image) sequentially output from the imaging unit 16, and calculates a motion vector within the image. By sequentially processing image data sequentially output from the imaging unit 16, motion vectors are calculated in chronological order. In the imaging device 10 of this embodiment, the screen is divided into M×N blocks (M and N are integers of 2 or more), and a motion vector is calculated for each block. The motion vector calculation method will be outlined below.

FIG. 4 is a diagram showing an example of time-series images obtained by imaging. In the figure, (A) represents the n-1th frame image, and (B) represents the nth frame image.

The figure shows an example in which a person O1 moving toward the right side of the screen is followed and imaged. In addition to the person O1, there are a person O2 and a tree O3 on the screen, and the person O2 is moving toward the left of the screen. The tree O3 conveniently represents the background as a stationary object.

FIG. 5 is an image in which the two images shown in FIG. 4 in chronological order are superimposed and displayed. In the figure, the (n-1)th frame image is represented by a broken line, and the nth frame image is represented by a solid line.

The two images are block matched block by block, and the motion vector of each block is calculated from the correlation of each block. That is, the motion vector of each block is calculated by determining which block in the n-th frame image each block of the n-1th frame image has moved to.

FIG. 6 is a diagram showing an example of the distribution of motion vectors obtained from the two images in chronological order shown in FIG. 4. The figure shows the normalized motion vectors of each block.

In the figure, the area indicated by diagonal lines is the area where the moving objects, the person O1 and the person O2, exist in the n-th frame image. The blocks in the area where the moving objects, the persons O1 and O2, exist exhibit motion vectors different from the blocks in the background. Furthermore, the moving objects, the person O1 and the person O2, exhibit different motion vectors because their movements are different.

The tracking target candidate extraction unit 100B extracts tracking target candidates based on the motion vector calculation results. As shown in FIG. 6, substantially the same motion vector is calculated for blocks to which the same subject belongs. Therefore, tracking target candidates can be extracted by extracting blocks with almost the same motion vector as a block. The tracking target candidate extraction unit 100B extracts tracking target candidates by, for example, combining blocks in which the difference in motion vector between adjacent blocks is equal to or less than a threshold value. Alternatively, clustering processing is performed to extract tracking target candidates. In the example shown in FIG. 6, the person O1, the person O2, and the background including the tree O3 are extracted as tracking target candidates.

The change amount calculation unit 100C calculates the amount of change in the motion vector of each tracking target candidate. The amount of change is calculated as the difference between the previous and subsequent motion vectors. That is, if the motion vector of the n-th frame is Vn and the motion vector of the n-1 frame is Vn-1, the amount of change ΔV is calculated as ΔV=[Vn]-[Vn-1]. Since the motion vectors are calculated sequentially, the amount of change is also calculated sequentially. Note that the motion vector of each tracking target candidate is calculated as, for example, the average of the motion vectors of blocks extracted as tracking target candidates.

The tracking target estimating unit 100D estimates a tracking target from among the tracking target candidates based on the amount of change in the motion vector of each tracking target candidate calculated by the change amount calculating unit 100C. Specifically, the tracking target is estimated based on the change tendency of the amount of change in the motion vector. More specifically, a tracking target candidate whose motion vector change amount tends to decrease is estimated to be a tracking target. In particular, a tracking target candidate whose motion vector has decreased over a plurality of frames is estimated to be a tracking target.

Here, the reason why a subject whose motion vector change amount tends to decrease is set as a tracking target will be explained.

Consider a case where a moving subject is tracked and imaged from the beginning. In this case, the user first performs an action to capture the tracking target within the screen. After capturing the tracking target within the screen, the imaging device is moved in accordance with the movement of the tracking target. When such an operation is performed, the motion vector of the tracking target gradually converges in the captured image. That is, after capturing the tracking target, the imaging device is moved to follow the movement of the tracking target, so there is almost no relative movement. Therefore, the motion vector of the tracking target gradually converges. Since the motion vector gradually converges, the amount of change in the motion vector of the tracking target gradually decreases. Therefore, the tracking target can be estimated by extracting subjects whose motion vectors tend to decrease.

FIG. 7 is a graph showing an example of changes in the amount of change in the motion vector of the tracking target. In the figure, the vertical axis represents the amount of change in the motion vector, and the horizontal axis represents time.

This figure shows the transition of the amount of change in the motion vector from the beginning of shooting when a comparatively fast-moving subject is the tracking target. As shown in the figure, the amount of change in the motion vector of the tracking target tends to gradually decrease from the beginning of shooting. In particular, since the image is captured in pursuit, the amount of change generally remains at a negative value and converges.

FIG. 8 is a graph showing an example of changes in the amount of change in the background motion vector. In the figure, the vertical axis represents the amount of change in the motion vector, and the horizontal axis represents time.

As shown in the figure, the amount of change in the motion vector of the background also gradually converges. However, since the background is a stationary object, when the tracking target is followed and imaged, the amount of change in the motion vector changes to a positive value and converges.

In this way, the variation pattern of the amount of change in the motion vector is different between the tracking target and the other targets. Therefore, the tracking target can be estimated from the variation pattern of the amount of change in the motion vector.

As described above, the amount of change in the motion vector of the tracking target changes to approximately a negative value and converges. In this case, the average value over multiple frames shows a negative value. In the imaging device 10 of the present embodiment, the average amount of change in motion vector during a specific period is calculated for each tracking target candidate, and the tracking target is estimated based on the calculated average value. Specifically, the calculated average value is compared with a threshold value, and a subject that exceeds the threshold value in the negative direction is estimated to be a tracking target. Note that since a subject exceeding the threshold in the negative direction is estimated to be a tracking target, the threshold is set to a negative value. The specific period is a predetermined period. The specific period is set, for example, as an imaging period for a predetermined number of frames.

Therefore, the tracking target estimating unit 100D calculates the average value of the amount of change in the motion vector of each tracking target candidate calculated by the change amount calculating unit 100C, and estimates the tracking target based on the calculated average value. Perform processing. In the process of calculating the average value, the average value in a specific period is calculated. In the process of estimating the tracking target based on the calculated average value, a process of comparing it with a threshold value is performed, and a process of extracting tracking target candidates exceeding the threshold value in the negative direction is performed. The extracted tracking target candidate is estimated to be the tracking target. If the tracking target cannot be extracted from the average value calculation result for the first specific period, the average value calculation process is performed again, and the tracking target extraction process is performed based on the calculation result. That is, the process is repeated until the tracking target is extracted.

FIG. 9 is a flowchart showing the procedure for setting a tracking target.

First, a motion vector is calculated (step S1). The motion vector is calculated by sequentially processing image data sequentially output from the imaging unit 16. Therefore, motion vectors are calculated in chronological order.

Next, a subject to be tracked is extracted based on the calculated motion vector (step S2). Then, the amount of change in the motion vector of each calculated tracking target candidate is calculated (step S3). As described above, the amount of change in the motion vector is calculated as the difference between the previous and subsequent motion vectors.

Next, regarding the calculated amount of change in the motion vector, an average value over a specific period is calculated (step S4). Then, a tracking target is extracted based on the calculated average value (step S5). Specifically, the average value of the amount of change in the motion vector of each tracking target candidate is compared with a threshold value, and tracking target candidates exceeding the threshold value in the negative direction are extracted and set as tracking targets. As a result of this process, it is determined whether the tracking target has been extracted (step S6). If the tracking target has been extracted, the extracted subject is set as the tracking target (step S7). On the other hand, if the tracking target cannot be extracted, the process returns to step S1 and the tracking target is extracted again in the next specific period.

Note that if a plurality of tracking target candidates exceeding the threshold in the negative direction are extracted, it is determined that the tracking target cannot be extracted. That is, if the tracking target could not be narrowed down to one, it is determined that the tracking target could not be extracted. Therefore, in this case, the tracking target is extracted again in the next specific period. That is, the process of estimating the tracking target is repeatedly performed until the number of tracking targets is finally narrowed down to one. Note that in this case, a configuration may be adopted in which the tracking target estimation process is performed only for tracking target candidates extracted as tracking targets in the previous specific period.

In this way, in the imaging device 10 of the present embodiment, when setting a tracking target, the tracking target is estimated based on the amount of change in the motion vector of the tracking target candidate. As a result, even when the tracking target is a moving subject from the beginning, the tracking target can be estimated in a short time from the beginning of shooting. That is, in the conventional method, it is necessary to wait for the motion vector to converge to a certain value in order to estimate the main subject (tracking target), but according to the imaging device 10 of the present embodiment, the motion vector The main subject (tracking target) can be estimated without waiting for convergence.

[Modified example]
[Setting the threshold or specific period according to the characteristics of the subject]
The specific period for calculating the average amount of change in motion vectors and the threshold for estimating the tracking target can be optimized according to the characteristics of the tracking target to improve accuracy. Tracking target can be estimated well. This point will be explained below.

FIG. 10 is a graph showing another example of the transition of the amount of change in the motion vector of the tracking target. In the figure, the vertical axis represents the amount of change in the motion vector, and the horizontal axis represents time.

This figure shows the transition of the amount of change in the motion vector from the beginning of shooting when a comparatively slow-moving subject is the tracking target. If the tracking target moves slowly, the tracking target can be easily captured. Therefore, in this case, as shown in the figure, the amount of change converges relatively quickly. However, in this case as well, the amount of change remains approximately at a negative value and converges.

Therefore, by setting the specific period and the threshold value according to the speed of movement of the subject to be tracked, the subject to be tracked can be estimated with higher accuracy. Specifically, the faster the movement of the subject to be tracked, the longer the specific period is set. Alternatively, the threshold value is set to a larger value in the negative direction.

The speed of movement of the subject to be tracked can be estimated from the movement of the imaging device at the beginning of shooting. That is, a fast-moving subject causes the imaging device to move greatly in a short period of time in accordance with the subject's movement, so the amount of movement of the imaging device at the beginning of shooting becomes large. Therefore, the speed of movement of the subject to be tracked can be estimated from the amount of movement of the imaging device.

Since the speed of movement of the subject to be tracked can be estimated from the amount of movement of the imaging device, the threshold value or specific period can be optimized from the amount of movement of the imaging device. Specifically, the larger the amount of movement of the imaging device, the larger the threshold value is set in the negative direction. Alternatively, the specific period may be set longer as the amount of movement of the imaging device increases. That is, the period for calculating the average value is lengthened.

For example, the threshold value is set in two stages, and the threshold value to be used is switched depending on the amount of movement of the imaging device. Specifically, a first threshold value and a second threshold value are determined as the threshold values. The second threshold is set to a value smaller in the negative direction than the first threshold. In other words, it is set to a value closer to zero. When the amount of movement of the imaging device exceeds the reference value, the threshold is set to the first threshold. The case where the amount of movement of the imaging device exceeds the reference value is the case where the amount of movement of the imaging device is relatively large. The case where the amount of movement of the imaging device is relatively large is the case where the movement of the subject to be tracked is relatively fast. Therefore, in this case, the first threshold value, which is a larger threshold value, is set. On the other hand, if the amount of movement of the imaging device is less than or equal to the reference value, the threshold is set to the second threshold. The case where the amount of movement of the imaging device is equal to or less than the reference value is the case where the amount of movement of the imaging device is relatively small. The case where the amount of movement of the imaging device is relatively small is the case where the movement of the subject to be tracked is relatively slow. Therefore, in this case, the second threshold, which is a smaller threshold, is set. Thereby, the threshold value can be optimized according to the characteristics (speed of movement) of the tracking target.

Regarding the specific period, for example, the value of the specific period is set in two stages, and the specific period to be used is switched depending on the amount of movement of the imaging device. Specifically, a first specific period and a second specific period are defined as specific periods. The second specific period is set to be shorter than the first period. When the amount of movement of the imaging device exceeds the reference value, the specific period is set to the first specific period. As described above, the case where the amount of movement of the imaging device exceeds the reference value is when the amount of movement of the imaging device is relatively large, and the movement of the subject to be tracked is relatively fast. Therefore, in this case, the first specific period, which is a longer period, is set. On the other hand, if the amount of movement of the imaging device is less than or equal to the reference value, the specific period is set to the second specific period. As described above, the case where the amount of movement of the imaging device is equal to or less than the reference value means that the amount of movement of the imaging device is relatively small, and the movement of the subject to be tracked is relatively slow. Therefore, in this case, the second specific period, which is a shorter period, is set. This allows the specific period to be optimized depending on the characteristics (speed of movement) of the tracking target.

[Detection of amount of movement of imaging device]
When switching the threshold value or specific period to be used based on the amount of movement of the imaging device, it is necessary to obtain information on the amount of movement of the imaging device.

The amount of movement of the imaging device can be determined, for example, from image data obtained by imaging. Furthermore, for example, if a sensor that detects the amount of movement of the main body is provided, information output from the sensor can also be used. For example, if the imaging device is equipped with a camera shake correction function, information output from the shake detection sensor can be used.

[Detection from image data]
Here, a case will be described in which the amount of movement of the imaging device is calculated from image data obtained by imaging (image data of each frame in a moving image). When determining the amount of motion of the imaging device from image data obtained by imaging, information on motion vectors determined from the image data can be used.

FIG. 11 is a block diagram of the functions of the tracking target setting section.

This figure shows an example of a case where the amount of motion of the imaging device 10 is determined using a motion vector and the threshold value is optimized according to the determined amount of motion.

As shown in FIG. 11, the tracking target setting section 100 of this example further has the functions of a motion amount calculation section 100E and a threshold value setting section 100F.

The motion amount calculation unit 100E calculates the amount of motion of the imaging device 10 based on the motion vector calculated by the motion vector calculation unit 100A. In this example, the average motion vector of the entire screen for a plurality of frames from the beginning of shooting is calculated, and its magnitude is taken as the amount of motion of the imaging device. For example, the average of the motion vectors of the entire screen for three frames from the beginning of shooting is calculated, and the magnitude of the calculated motion vector is taken as the amount of motion of the imaging device. The motion vector of the entire screen is calculated as the average of the motion vectors of all blocks. For example, the value of the motion vector V1 of the entire screen in one frame is (x1, y1), the value of the motion vector V2 of the entire screen in two frames is (x2, y2), and the value of the motion vector V3 of the entire screen in three frames is (x2, y2). Suppose that it is (x3, y3). Note that x1, x2, and x3 are the x-direction components of the motion vectors V1, V2, and V3, respectively, and y1, y2, and y3 are the y-direction components of the motion vectors V1, V2, and V3, respectively. The x direction is the horizontal direction of the screen, and the y direction is the vertical direction of the screen. In this case, the average size of the motion vector for the entire screen is calculated as [(x1+x2+x3) ² +(y1+y2+y3) ² ] ^1/2 /3. For example, the value of the motion vector V1 of the entire screen in one frame is (0,0), the value of the motion vector V2 of the entire screen in the second frame is (10,0), and the value of the motion vector V3 of the entire screen in the third frame is (0,0). In the case of (15,0), it is calculated as [(0+10+15) ² +(0+0+0) ² ] ^1/2 /3=[(25) ² +(0) ² ] ^1/2 /3≈8.3.

The threshold value setting unit 100F sets a threshold value based on the value of the amount of movement of the imaging device 10 calculated by the amount of movement calculation unit 100E. In this example, when the amount of motion exceeds the reference value, the threshold is set to the first threshold. On the other hand, if the amount of motion is less than or equal to the reference value, the threshold is set to the second threshold. The second threshold is set to a value smaller in the negative direction than the first threshold (a value closer to zero).

In this way, the amount of motion of the imaging device can be determined from the image data obtained by imaging, and the threshold value can be optimized according to the determined amount of motion.

In this example, an example in which the threshold value is optimized has been described, but a specific period can also be optimized using a similar method.

[How to use the sensor]
As described above, if the imaging device is equipped with a sensor that detects the amount of movement of the main body, the sensor can also be used to obtain information about the amount of movement of the imaging device.

FIG. 12 is a block diagram of the functions that the tracking target setting section has when a shake detection sensor is provided.

When the imaging device is equipped with a camera shake correction function, a shake detection sensor is included in order to detect camera shake. When the image capturing device is equipped with this shake detection sensor, the amount of movement (shake amount) of the image capturing device can be detected using the output of the shake detection sensor.

As shown in FIG. 12, the tracking target setting unit 100 of this example includes a shake amount calculation unit 100G that calculates the amount of shake based on the signal output from the shake detection sensor 200, and a shake amount calculation unit 100G that calculates the amount of shake based on the signal output from the shake detection sensor 200. The camera further includes a threshold setting unit 100F that sets a threshold based on the determined amount of blur.

The shake detection sensor 200 is originally used to detect camera shake, but in this example, it is also used to set a tracking target. The shake detection sensor 200 is composed of, for example, a gyro sensor.

The shake amount calculation unit 100G calculates the shake amount based on the signal output from the shake detection sensor 200. For example, when the shake detection sensor 200 is composed of a gyro sensor, the amount of shake (angle signal) is calculated by integrating the angular velocity signal output from the sensor. The calculated amount of blur is taken as the amount of movement of the imaging device.

The threshold value setting unit 100F sets a threshold value based on the amount of blur (amount of movement) calculated at the beginning of shooting. In this example, when the amount of motion exceeds the reference value, the threshold is set to the first threshold. On the other hand, if the amount of motion is less than or equal to the reference value, the threshold is set to the second threshold. The second threshold is set to a value smaller in the negative direction than the first threshold (a value closer to zero).

In this way, when the imaging device is equipped with a sensor that detects the amount of movement of the main body, the sensor can be used to obtain information about the amount of movement of the imaging device and optimize the threshold value.

In this example, an example in which the threshold value is optimized has been described, but a specific period can also be optimized using a similar method.

Further, in this example, the case where a shake detection sensor is used has been described as an example, but the sensor that detects the amount of movement of the main body is not limited to this. In addition, for example, an acceleration sensor can also be used.

Further, in this example, a shake detection sensor for camera shake correction is used, but a dedicated sensor may be provided.

[Start timing of tracking target estimation process]
As described above, when tracking and imaging a moving subject from the beginning, the user first performs an action to capture the tracking target within the screen. When performing this operation, the imaging device is moved significantly. That is, in order to capture the desired subject within the screen, the camera moves relatively widely in a short period of time. Therefore, by monitoring the movement of the imaging device, the timing for starting the tracking target estimation process can be optimized. That is, the tracking target estimation process can be started at an appropriate timing. Specifically, the amount of movement of the imaging device is monitored, and when the amount of movement greater than a predetermined value is detected, the process of estimating the tracking target is started.

FIG. 13 is a block diagram of the functions that the tracking target setting unit has when automatically starting the tracking target estimation process.

As shown in the figure, the function of a start determination section 100H is further provided. The start determination unit 100H determines the timing to start the tracking target estimation process based on the value of the amount of movement of the imaging device 10 calculated by the amount of movement calculation unit 100E. In this example, when the amount of motion exceeds a specified value, it is determined that it is the timing to start the tracking target estimation process. Note that in this example, the amount of movement of the imaging device 10 is detected from the image data.

The tracking target estimating unit 100D starts the estimation process when the start determining unit 100H determines that it is the timing to start the tracking target estimation process. That is, a specific period is started, the average value of the amount of change in the motion vector of each tracking target candidate is calculated, and the tracking target is estimated.

FIG. 14 is a flowchart illustrating a processing procedure when starting tracking target setting processing based on the amount of movement of the imaging device.

First, it is determined whether the tracking AF function is turned on (step S11). If the tracking AF function is turned on (“Yes” in step S11), detection of the amount of movement of the imaging device 10 is started (step S12). Then, it is determined whether the detected amount of motion is greater than or equal to a specified value (step S13). If the amount of motion is equal to or greater than the specified value, tracking target estimation processing is started (step S14). That is, it is determined that the user has started an operation to capture the tracking target, and the tracking target estimation process is started. On the other hand, if the amount of movement is less than the specified value, it is determined whether the tracking AF function is turned off (step S15). If the tracking AF function is not turned off, the process returns to step S12, and the amount of movement of the imaging device 10 is detected again (step S12). On the other hand, if the tracking AF function is turned off, the process ends.

In this way, the amount of movement of the imaging device 10 is detected, and when the amount of movement greater than a predetermined value is detected, the process of estimating the tracking target is started. Thereby, processing load can be reduced. Further, misrecognition can also be prevented.

Note that this example has been described using the case where the amount of movement of the imaging device 10 is detected based on image data obtained by imaging, but if a sensor is provided to detect the amount of movement of the device body, , the sensor may be used to detect the amount of movement of the imaging device 10.

[Instruction to start and cancel manual estimation process]
The tracking target estimation process may be started in response to an instruction from the user. For example, a configuration can be adopted in which the start of tracking target estimation processing is instructed by a button operation. In this case, for example, a button for instructing the start of tracking target estimation processing is provided on the device main body. When the user presses this button, tracking target estimation processing is disclosed.

Furthermore, the tracking target estimation process that has started may be forcibly canceled in response to an instruction from the user. For example, the configuration can be such that a button operation instructs to cancel the tracking target estimation process. In this case, for example, a button for instructing cancellation of tracking target estimation processing is provided on the main body of the apparatus. When the tracking target estimation process is started, if the user presses this button, the tracking target estimation process is canceled.

[Screen display]
When displaying a live view on the display unit 18 and performing tracking AF, it is preferable that the subject set as a tracking target be displayed surrounded by a frame.

Furthermore, when a tracking target estimation process is performed, a configuration may be adopted in which the subject that is a tracking target candidate extracted in the process is displayed surrounded by a frame. In this case, it is preferable that the frame surrounding the subject finally set as the tracking target and the frame surrounding the subject as a tracking target candidate are displayed in different formats (color, brightness, line type, etc.). For example, a configuration may be adopted in which a tracking target candidate is surrounded by a white frame, and after the tracking target is determined, it is surrounded by a yellow frame.

[Extraction of tracking target candidates]
In the embodiment described above, the tracking target candidates are extracted based on the motion vector detection results, but the configuration for extracting the tracking target candidates is not limited to this. A configuration may be adopted in which tracking target candidates are extracted using a known image recognition technique. For example, by using a technique for extracting a moving object through image recognition, the extracted moving object can be used as a tracking target candidate. Further, for example, by using a technique for extracting a person (including a case where only a face is extracted) through image recognition, the extracted person can be used as a tracking target candidate. A trained model can be used for image recognition.

[Motion vector calculation]
In the above embodiment, one screen is divided into a plurality of blocks and a motion vector is calculated for each block, but the method of calculating a motion vector is not limited to this. For example, as described above, when a tracking target candidate is extracted using image recognition or the like, a configuration may be adopted in which the motion vector of the extracted tracking target candidate is directly calculated.

[Focus detection]
In the embodiment described above, focus detection (distance measurement) is performed using an image sensor incorporating a phase difference pixel, but the focus detection means is not limited to this. For example, a configuration may be adopted in which focus is detected using a dedicated phase difference AF sensor. In addition, a configuration may be adopted in which focus is detected using a so-called active type AF sensor (infrared sensor, ultrasonic sensor, etc.). Alternatively, a configuration may be adopted in which focus detection is performed using a contrast method.

[Imaging device]
The imaging device of the present invention includes not only a so-called digital camera, a video camera, etc., but also an imaging device integrated into a device having functions other than imaging. For example, it also includes imaging devices built into smartphones, personal computers, and the like.

[Processor]
The processor that realizes each function of the image processing device (in the above embodiment, the tracking target setting section corresponds to the image processing device) includes a CPU (Central Processing Unit), which is a general-purpose processor, and a FPGA (Field Programmable Gate Array). ), which are processors whose circuit configurations can be changed after manufacturing, and ASICs (Application Specific Integrated Circuits), which are processors with circuit configurations specifically designed to perform specific processes. This includes dedicated electrical circuits such as processors with

Each function of the image processing device may be realized by one of these various processors, or by two or more processors of the same type or different types (for example, multiple FPGAs or a combination of a CPU and FPGA). May be realized.

Further, each function of the image processing device may be realized by one processor. As an example of realizing each function of a signal processing device with one processor, first, one processor is configured by a combination of one or more CPUs and software, and this processor realizes each function of the signal processing device. There is a form of doing so. Second, there is a form of using a processor that realizes the functions of the entire signal processing device with one IC (Integrated Circuit) chip, as typified by a system on chip (SoC).

In this way, each function of the image processing device is realized as a hardware structure using one or more of the various processors described above.

Furthermore, the hardware structure of these various processors is, more specifically, an electric circuit that is a combination of circuit elements such as semiconductor elements.

10 Imaging device 12 Imaging optical system 12A Focus lens 12B Aperture 14 Imaging optical system drive section 14A Focus lens drive section 14B Aperture drive section 16 Imaging section 16A Imaging element 18 Display section 20 Storage section 22 Connection section 24 Operation section 26 CPU
28 ROM
30 RAM
100 Tracking target setting unit 100A Motion vector calculation unit 100B Tracking target candidate extraction unit 100C Change amount calculation unit 100D Tracking target estimation unit 100E Motion amount calculation unit 100F Threshold setting unit 100G Shake amount calculation unit 100H Start determination unit 110 Tracking target detection unit 112 Focus area setting unit 114 Phase difference detection unit 116 Focus lens drive control unit 200 Shake detection sensor O1 Person O2 Person O3 Trees S1 to S7 Tracking target setting processing steps S11 to S15 Setting the tracking target based on the amount of movement of the imaging device Processing steps when starting processing

Claims (14)

An image processing device that processes moving images captured by an imaging device,
Equipped with a processor,
The processor includes:
a process of calculating motion vectors from the video images in chronological order within a specific period;
a process of calculating the amount of change in the motion vector in chronological order;
a process of calculating an average value of the amount of change in the motion vector within the specific period;
A process of estimating a subject whose average value of the amount of change in the motion vector exceeds a threshold value in a negative direction as a tracking target;
I do,
Image processing device. The processor further performs a process of setting the threshold value based on the amount of movement of the imaging device when capturing the moving image.
The image processing device according to claim 1 . The processor sets the threshold value to a larger value in a negative direction as the amount of movement of the imaging device increases.
The image processing device according to claim 2 . The processor sets the threshold to a first threshold when the amount of motion of the imaging device exceeds a reference value, and sets the threshold to be less than the first threshold when the amount of motion of the imaging device is less than or equal to the reference value. Set to a smaller second threshold in the negative direction,
The image processing device according to claim 3 . The processor starts the specific period based on the amount of movement of the imaging device.
The image processing device according to any one of claims 1 to 4. An image processing device that processes moving images captured by an imaging device,
Equipped with a processor,
The processor includes:
a process of calculating motion vectors from the video images in chronological order within a specific period;
a process of calculating the amount of change in the motion vector in chronological order;
a process of estimating a tracking target based on the amount of change in the motion vector;
a process of setting the specific period based on the amount of movement of the imaging device when capturing the moving image, the process of setting the specific period to be longer as the amount of movement of the imaging device increases;
I do,
Image processing device. The processor sets the specific period to the first specific period when the amount of motion of the imaging device exceeds the reference value, and sets the specific period to the first specific period when the amount of motion of the imaging device is equal to or less than the reference value. Set to a second specific period that is shorter than the specific period,
The image processing device according to claim 6 . The processor further performs a process of calculating a motion amount of the imaging device based on the moving image.
The image processing device according to any one of claims 1 to 7 . The imaging device includes a sensor that detects the amount of movement of the main body,
The processor further performs a process of acquiring information on the amount of movement of the imaging device from the sensor.
The image processing device according to any one of claims 1 to 8 . An image processing device according to any one of claims 1 to 9 ,
a focus adjustment unit that adjusts the focus by moving a focus lens so as to focus on the estimated tracking target;
An imaging device equipped with An image processing method for processing a moving image captured by an imaging device, the method comprising:
calculating motion vectors from the video images in chronological order within a specific period;
calculating the amount of change in the motion vector in chronological order;
calculating an average value of the amount of change in the motion vector within the specific period;
estimating a subject whose average value of the amount of change in the motion vector exceeds a threshold in the negative direction as a tracking target;
including,
Image processing method. An image processing method for processing a moving image captured by an imaging device, the method comprising:
calculating motion vectors from the video images in chronological order within a specific period;
calculating the amount of change in the motion vector in chronological order;
estimating a tracking target based on the amount of change in the motion vector;
setting the specific period based on the amount of movement of the imaging device when capturing the moving image, and setting the specific period to be longer as the amount of movement of the imaging device increases;
including,
Image processing method. An image processing program that processes moving images captured by an imaging device,
a function of calculating motion vectors from the video images in chronological order within a specific period;
a function of calculating the amount of change in the motion vector in chronological order;
a function of calculating an average value of the amount of change in the motion vector within the specific period;
a function of estimating a subject whose average value of the amount of change in the motion vector exceeds a threshold value in a negative direction as a tracking target;
An image processing program that allows computers to realize this. An image processing program that processes moving images captured by an imaging device,
a function of calculating motion vectors from the video images in chronological order within a specific period;
a function of calculating the amount of change in the motion vector in chronological order;
a function of estimating a tracking target based on the amount of change in the motion vector;
a function of setting the specific period based on the amount of movement of the imaging device when capturing the moving image, and a function of setting the specific period to be longer as the amount of movement of the imaging device increases;
An image processing program that allows computers to realize this.

Images