JP2020057302A - Imaging device and control method of the same, and program - Google Patents

Abstract

To provide an imaging device that can always obtain high tracking performance irrespective of the tracking status by determining a size of a feature amount area based on image information.SOLUTION: An imaging device can obtain a first image and a second image in a time-series manner. The imaging device comprises: subject detection means for detecting a subject serving as a tracking subject from the first image; feature amount area registration means for registering a feature amount area including the subject based on the first image; feature amount area update means for updating the size of the feature amount area registered by the feature amount area registration means based on the information related to the first and second images; and subject tracking means for tracking the subject by searching an image area corresponding to the feature amount area updated by the feature amount area update means from the second image.SELECTED DRAWING: Figure 2

Description

  The present invention relates to an imaging device, a control method thereof, and a program.

  In a digital camera, a subject is tracked based on a feature amount of a specific region (feature amount region) of an image based on obtained image data, and shooting conditions such as focus, brightness, and color for the subject are preferably adjusted. It is common to take pictures according to the state. At this time, if the size of the feature amount region is set to be large as in Patent Document 1, information used for searching for an image region corresponding to the feature amount region in tracking the subject increases, so that tracking performance is improved. Be expected.

JP-A-8-329110

  However, the technique of Patent Document 1 does not change the size of the feature amount area based on image information when searching for an image. Therefore, if the technology according to Patent Literature 1 is applied to search for a feature amount region in tracking a subject, and the size of the feature amount region is increased in order to improve tracking performance, conversely, tracking will fail. Sometimes.

  For example, when the feature amount region is larger than the subject region, the feature amount region also includes the background of the subject. In this case, if, for example, a situation occurs in which the camera largely moves or the subject is originally a moving object during the tracking of the subject, the background information in the feature amount area changes over time. That is, the positional relationship between the subject (tracking target) and the background greatly changes, and the same feature amount does not exist. As a result, tracking fails.

  An object of the present invention is to provide an imaging apparatus that can always obtain high tracking performance regardless of a tracking state by determining a size of a feature amount area based on image information.

  An imaging device according to an example of the present invention is an imaging device capable of acquiring a first image and a second image in a time series, and a subject detection unit that detects a subject as a tracking target from the first image. A feature amount region registration unit that registers a feature amount region including the subject based on the first image, and the feature amount region registration unit that is registered in the feature amount region registration unit based on information related to the first and second images. A feature amount region updating unit that updates the size of the feature amount region; and, from the second image, a search for an image region corresponding to the feature amount region updated by the feature amount region updating unit. Subject tracking means for tracking the subject.

  According to the present invention, it is possible to provide an imaging apparatus that can always obtain high tracking performance regardless of a tracking state by determining the size of a feature amount area based on image information.

FIG. 2 is a block diagram illustrating a configuration example of an imaging device. FIG. 3 is a block diagram illustrating a configuration example of a subject recognition unit. 6 is a flowchart illustrating an operation example of the imaging device. 9 is a flowchart illustrating an example of a subject recognition process. 9 is a flowchart illustrating an example of subject detection processing. FIG. 9 is a diagram illustrating an example of an evaluation pixel and an estimation area in subject detection processing. FIG. 9 is a diagram illustrating an example of an evaluation area in subject detection processing. 13 is a flowchart illustrating an example of a background motion amount acquisition process. 9 is a flowchart illustrating an example of a feature amount area update process. FIG. 7 is a diagram illustrating a relationship between a size of a feature amount region and a difference in a motion amount. FIG. 9 is a diagram illustrating an example of changing the size of a feature amount area. FIG. 9 is a diagram illustrating an example of template matching.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
The present invention provides a function of setting a feature amount region for a subject, tracking the subject based on the feature amount region, and adjusting shooting conditions such as focus, brightness, and color to the subject in a suitable state. The present invention relates to an imaging device having: The imaging device described below is applied to an optical device such as a video camera, a digital still camera, and an interchangeable lens that can be used for them.

<Imaging device>
FIG. 1 illustrates a configuration example of an imaging device.
The imaging device 100 can capture and record images such as moving images and still images. The functional blocks in the imaging device 100 are connected by the bus 160 so that they can communicate with each other. Each function of the imaging device 100 is realized by the main control unit 151 executing a program and controlling each functional block.

  The imaging device 100 includes a subject recognition unit 161 that analyzes an image and estimates a specific subject area from the analysis result. One of the features of the present embodiment is that the size of the feature amount area can be changed based on image information (information of an image obtained at the time of shooting) using the subject recognition unit 161.

  Here, the image information refers to information relating to a plurality of images (a plurality of frames) that can be acquired in time series by the imaging lens (lens unit) 101 and the image sensor 141. The image information includes, for example, average luminance of the screen, WB (white balance), position information by a GPS function, and the like. Further, the image information is information obtained from the plurality of images, for example, the amount of movement of the subject, the amount of movement of the background, and the position of the subject (tracking target) and the background obtained from them, as described later. It also includes the amount of change in the relationship.

  As described above, in the present embodiment, by making the size of the feature amount area changeable by the subject recognizing unit 161, an imaging apparatus capable of always obtaining high tracking performance regardless of the tracking situation is realized. . Therefore, the subject recognizing unit 161 has a subject detection function, a feature amount area registration function, a background motion amount acquisition function, a feature amount area update function, and a subject tracking function. Details of these functions will be described later.

  The taking lens 101 includes a fixed first group lens 102, a zoom lens 111, an aperture 103, a fixed third group lens 121, a focus lens 131, a zoom motor 112, an aperture motor 104, and a focus motor 132. . The fixed first group lens 102, the zoom lens 111, the aperture 103, the fixed third group lens 121, and the focus lens 131 constitute a photographic optical system.

  Although each of the lenses 102, 111, 121, and 131 is shown as a single lens for convenience in the drawing, the lenses 102, 111, 121, and 131 may be configured by combining a plurality of lenses. When the imaging device 100 forms a camera system, the imaging lens 101 may be integrated with the camera body or may be configured to be detachable from the camera body.

  The aperture control unit 105 controls the operation of an aperture motor 104 that drives the aperture 103. That is, the aperture control unit 105 changes the aperture diameter of the aperture 103. The zoom control unit 113 controls the operation of a zoom motor 112 that drives the zoom lens 111. That is, the zoom control unit 113 changes the focal length (angle of view) of the photographing lens 101.

  The focus control unit 133 acquires a defocus amount and a defocus direction of the imaging lens 101 based on a phase difference between a pair of focus detection signals (A image and B image) obtained from the image sensor 141. Then, the focus control unit 133 converts the defocus amount and the defocus direction into the drive amount and the drive direction of the focus motor 132. Based on the driving amount and the driving direction, the focus control unit 133 controls the operation of the focus motor 132 and drives the focus lens 131 to control the focus state of the photographing lens 101.

  As described above, the focus control unit 133 performs the automatic focus detection (AF) based on the phase difference detection method. However, the focus control unit 133 may execute the AF using a contrast detection method based on a contrast evaluation value obtained from an image signal, instead of the AF using the phase difference detection method.

  A subject image formed on the imaging surface of the image sensor 141 by the imaging lens 101 is converted into an electric signal (image signal) by a photoelectric conversion element of each of a plurality of pixels arranged in the image sensor 141. The imaging element 141 is, for example, a CCD or CMOS image sensor, and includes a pixel array of m pixels in the horizontal direction and n pixels in the vertical direction. Here, n and m are each a natural number of 2 or more. Each pixel has two photoelectric conversion elements (photoelectric conversion regions). The imaging control unit 143 controls reading of an image signal from the imaging element 141 based on an instruction from the main control unit 151.

  The image signal read from the image sensor 141 is supplied to the image signal processing unit 142. The image signal processing unit 142 applies signal processing such as noise reduction processing, A / D conversion processing, and automatic gain control processing to the image signal, and outputs the image signal to which the signal processing has been applied to the imaging control unit 143. . The imaging control unit 143 stores the image signal received from the image signal processing unit 142 in a RAM (random access memory) 154.

  The image processing unit 152 applies a predetermined image processing to the image signal stored in the RAM 154. The image processing applied by the image processing unit 152 includes, for example, so-called development processing such as white balance adjustment processing, color interpolation (demosaic) processing, and gamma correction processing. The image processing may include a signal format conversion process, a scaling process, and the like. Examples of the image processing are not limited to these, and processing other than the above may be performed.

  The image processing unit 152 can also generate information on subject brightness and the like for use in automatic exposure control (AE). Information about a specific subject area is supplied from the subject recognition unit 161 to the image processing unit 152, and is used for, for example, white balance adjustment processing. When performing the AF by the contrast detection method, the image processing unit 152 can also generate an AF evaluation value. The image processing unit 152 stores the processed image signal in the RAM 154.

  The main control unit 151 generates a data file of a predetermined recording format by adding a predetermined header to the image signal temporarily stored in the RAM 154. At this time, the image compression / decompression unit 153 may encode the data file based on an instruction from the main control unit 151 and compress the information amount regarding the image signal. Then, the main control unit 151 records the generated data file on a recording medium 157 such as a memory card.

  When displaying the image signal stored in the RAM 154, the main control unit 151 instructs the image processing unit 152 to perform a scaling process on the image signal so as to match the display size of the display unit 150 such as a monitor display. I do. Thereafter, the main control unit 151 writes the scaled image signal in the video memory area (VRAM area) of the RAM 154 as display image data. The display unit 150 reads display image data from the VRAM area of the RAM 154 and displays the read image data on a display device such as an LCD and an organic EL display.

  The image capturing apparatus 100 according to the present embodiment displays the captured moving image on the display unit 150 immediately when capturing a moving image (in a shooting standby state or during recording of a moving image), so that the display unit 150 is used as an electronic viewfinder (EVF). Let it work. A moving image and its frame image displayed when the display unit 150 functions as an EVF are called a live view image or a through image. Further, the imaging device 100 displays the photographed still image on the display unit 150 for a certain period of time so that the user can confirm the photographing result when the photographing is performed. These display operations are also realized by the control of the main control unit 151.

  The operation unit 156 includes switches, buttons, keys, a touch panel, and the like for a user to input an instruction to the imaging device 100. The input from the operation unit 156 is transferred to the main control unit 151 via the bus 160. The main control unit 151 controls each unit to realize an operation according to an input from the operation unit 156.

  The main control unit 151 has one or more programmable processors such as a CPU and an MPU. The main control unit 151 reads out a program stored in a storage unit 155 such as a flash memory into the RAM 154 and executes the program to control each unit and realize each function or each process of the imaging device 100. In addition, the main control unit 151 executes an AE process for automatically determining an exposure condition (shutter speed, accumulation time, aperture value, sensitivity, and the like) based on information on a subject luminance. The information on the subject brightness can be acquired from the image processing unit 152, for example. The main control section 151 can also determine the exposure condition based on a subject area such as a person's face.

  The main control unit 151 determines the electronic shutter speed (accumulation time) and the magnitude of the gain, for example, with the aperture fixed at the time of capturing a moving image. The main control unit 151 notifies the imaging control unit 143 of the determined exposure condition (the storage time and the magnitude of the gain). The imaging control unit 143 controls the operation of the imaging element 141 so that imaging is performed according to the exposure condition notified from the main control unit 151.

  The result of the object recognition unit 161 can be used, for example, for automatically setting a focus detection area. In this case, a tracking AF function for a specific subject area can be realized. In addition, AE processing can be performed based on luminance information of the focus detection area, and image processing (eg, gamma correction processing, white balance adjustment processing, and the like) can be performed based on pixel values of the focus detection area. Note that the main control unit 151 may superimpose an index indicating the current position of the subject area (for example, a rectangular frame surrounding the subject area) on the display image.

  The battery 159 is managed by the power management unit 158, and supplies power to the entire imaging device 100. The storage unit 155 stores a program to be executed by the main control unit 151, setting values required for executing the program, GUI data, user setting values, and the like. For example, when a transition from the power-off state to the power-on state is instructed by operating the operation unit 156, a program stored in the storage unit 155 is read into the RAM 154, and the main control unit 151 executes the program.

<Subject recognition unit>
FIG. 2 illustrates a configuration example of the subject recognition unit.
The subject recognition unit 161 includes a subject detection unit 201, a feature amount area registration unit 202, a background motion amount acquisition unit 203, a feature amount area update unit 204, and a subject tracking unit 205.

  The subject detection unit 201 detects the position and size of a subject (tracking target) based on the image acquired by the imaging element 141 and processed by the image processing unit 152. For example, the subject detection unit 201 acquires images (frames) in time series from the image processing unit 152, and detects a subject as a tracking target included in each image. In addition, the subject detection unit 201 obtains a region occupied by the subject in each image, that is, a subject region, from the position and size of the subject, and generates information on the reliability of the subject region.

  The feature amount area registration unit 202 determines a feature amount area including a subject based on the subject area detected by the subject detection unit 201, and registers the feature amount area. Here, the feature amount region refers to an image region including a subject set in n frames (first image) when tracking a subject as a tracking target. Here, n is a natural number (1, 2, 3,...), And the larger the value of n, the later the time. The tracking of the subject is performed by searching for an image area corresponding to the feature amount area in a frame later than the n-th frame, for example, an (n + 1) -th frame (second image), as described later.

  When registering a feature amount region based on the subject region detected from the first image by the subject detection unit 201, the feature amount region registration unit 202 sets the size of the feature region to a default value (initial value). The default value is predetermined. However, the default value may be configured so that the user can freely change it via the operation unit 156 or the like, or may be configured so that the program can automatically set the optimum value.

  Note that, when the subject tracking unit 205 described later finds an image region corresponding to the feature amount region and the tracking is successful, the feature amount region registration unit 202 replaces the already registered feature amount region with The image area is registered as a new feature amount area. On the other hand, when the subject tracking unit 205 described later cannot find an image region corresponding to the feature amount region and the tracking is unsuccessful, the feature amount region registration unit 202 continues to register the already registered feature amount region. I do.

  The background motion amount acquisition unit 203 acquires a background motion amount based on a plurality of images sequentially supplied in time series. For example, the background motion amount acquisition unit 203 estimates a background from the plurality of images, and detects a motion vector between the plurality of images based on the estimated background. Thereby, the background motion amount acquisition unit 203 can acquire the background motion amount based on the motion vector.

  Further, the background motion amount acquisition unit 203 detects the motion of the imaging device 100 using at least one of the gyro sensor and the acceleration sensor, and acquires the background motion amount based on the motion of the imaging device 100. You can also. The operation of the background motion amount acquisition unit 203 will be described later in detail.

  The feature amount area update unit 204 updates the size of the feature amount area registered in the feature amount area registration unit 202 based on information about a plurality of images sequentially supplied in time series. For example, the feature amount area update unit 204 compares the background of the subject with the background based on the background motion acquired by the background motion acquisition unit 203 and the subject motion acquired by the subject tracking unit 205 described below. Acquires the amount of change in the positional relationship. Then, the feature amount area update unit 204 updates the size of the feature amount area registered in the feature amount area registration unit 202 based on the change amount.

  Here, the size (default value) of the feature amount area registered first by the feature amount area registration unit 202 is such that when the amount of change in the positional relationship between the subject and the background that occurred most frequently in the past is the reference value Δref. It is desirable to set the size to the reference value Δref. The relationship between the reference value Δref and the default value will be described later.

  Then, when the amount of change between the subject and the background is greater than the reference value Δref, the feature amount area update unit 204 reduces the size of the feature amount area to a value smaller than the default value. In addition, when the change amount is smaller than the reference value Δref, the feature amount area update unit 204 sets the size of the feature amount area to be larger than the default value. Further, the feature amount area updating unit 204 reduces the size of the feature amount area as the change amount increases, and increases the size of the feature amount area as the change amount decreases. However, when the change amount is larger than the threshold value, the feature amount area update unit 204 keeps the size of the feature amount area constant.

  The subject tracking unit 205 performs tracking of a subject by searching for an image area corresponding to the feature amount area updated by the feature amount area updating unit 204 from a plurality of images sequentially supplied in time series. For example, the subject tracking unit 205 sets an image region having the highest similarity to the feature amount region updated by the feature amount region updating unit 204 as an image region corresponding to the feature amount region, and performs tracking of the subject based on the image region. Do.

  Here, the similarity is determined based on, for example, an evaluation value defined by a predetermined function, and the subject tracking unit 205 determines a region with the lowest evaluation value as a region with the highest similarity. When the minimum value of the evaluation value is larger than the threshold value, the subject tracking unit 205 can also conclude that the image area corresponding to the feature amount area cannot be found. The search result includes information such as a subject area in the image, reliability, and the amount of movement of the subject, and is used for various processes in the main control unit 151.

<Operation example of imaging device>
FIG. 3 shows an operation example of the imaging apparatus.
In the present embodiment, a moving image shooting operation including a subject detection process, a subject tracking process, and a motion amount detection process for detecting a motion amount of a subject and a background will be described. The moving image shooting operation is executed at the time of shooting standby or at the time of moving image recording.

  Although details such as resolution of an image (frame) to be handled are different between the shooting standby mode and the moving image recording mode, the processing is performed in the subject detection processing, the subject tracking processing, and the motion detection processing for detecting the motion of the subject and the background. Such contents are basically the same. Therefore, the following description will be made without distinguishing between the shooting standby mode and the moving image recording mode.

  First, in S301, the main control unit 151 determines whether the power switch of the imaging device 100 is on. If the power switch is off, the present flow ends, and if the power switch is on, the present flow proceeds to S302.

  Next, in S302, the main control unit 151 controls each unit to execute the imaging process for one frame, and then proceeds to S303. In the present embodiment, a pair of parallax images and a captured image for one screen are acquired as an imaging process, and are stored in the RAM 154. In S303, the main control unit 151 issues an instruction to the subject recognition unit 161 to execute the subject recognition processing. Here, the subject recognition process includes a subject detection process, a subject tracking process, and a motion amount detection process for detecting a motion amount of the subject and the background. Details of the subject recognition processing in the subject recognition unit 161 will be described later.

  Thereafter, the position, size, and amount of movement of the subject area are notified from the subject recognition unit 161 to the main control unit 151, and are stored in the RAM 154. The main controller 151 sets a focus detection area based on the notified subject area.

  Next, in S304, the main control unit 151 causes the focus control unit 133 to execute focus detection processing. The focus control unit 133 generates an A image by connecting a plurality of A signals obtained from a plurality of pixels arranged in the same row among a plurality of pixels included in a focus detection region of a pair of parallax images, In addition, a B image is generated by connecting a plurality of B signals. Then, the focus control unit 133 calculates the amount of correlation between the A image and the B image while shifting the relative position between the A image and the B image, and determines the relative position at which the similarity between the A image and the B image is highest. It is obtained as a phase difference (shift amount) between the A image and the B image. Further, the focus control unit 133 converts the phase difference into a defocus amount and a defocus direction.

  Next, in S305, the focus control unit 133 drives the focus motor 132 in accordance with the defocus amount and the lens driving amount and driving direction corresponding to the defocus direction obtained in S304, and moves the focus lens 131. When the lens driving process ends, the main control unit 151 returns the flow to S301.

  Thereafter, the processing of S302 to S305 is repeatedly executed until it is determined in S301 that the power switch is turned off. Thus, a plurality of images (frames) sequentially supplied in time series are searched for a subject as a tracking target, and a subject tracking function is realized. In FIG. 3, the subject tracking process is performed for each sequentially supplied frame. However, the subject tracking process may be performed for every several frames for the purpose of reducing the processing load and power consumption of the main control unit 151.

<Subject recognition processing>
FIG. 4 shows an example of subject recognition processing.
This flowchart is a subroutine of S303 in FIG.

  First, in S401, the subject recognition unit 161 acquires an image (frame) from the imaging control unit 143. Thereafter, in S402, the subject detection unit 201 sets a plurality of evaluation areas having different center positions and sizes for the image from the imaging control unit 143, and detects a subject from each evaluation area. The details of the process of detecting a subject will be described later.

  Next, in S403, the feature amount area registration unit 202 determines the feature amount based on the subject area detected by the subject detection unit 201 or the image area corresponding to the feature amount area of the previous frame detected by the subject tracking unit 206. Register an area. In the present embodiment, the registration of the feature amount area is performed based on the result of the subject detection unit 201 in the first one frame, and is performed based on the result of the subject tracking unit 206 in the subsequent frames.

  Next, in S404, the background motion amount acquisition unit 203 acquires the background motion amount based on a plurality of images sequentially supplied in time series. The details of the process of acquiring the background motion amount will be described later. Thereafter, in S405, the feature amount area update unit 204 updates the size of the feature amount area based on the background motion amount acquired in S404 and the subject motion amount of the previous frame acquired in S406 described later. I do. Details of the process of updating the size of the feature amount area will be described later.

  Next, in step S <b> 406, the subject tracking unit 205 searches for an image area corresponding to the feature amount area updated in step S <b> 405 from a plurality of images sequentially supplied in time series. For example, the subject tracking unit 205 sets a region having the highest similarity to the feature amount region as an image region corresponding to the feature amount region, and performs tracking of the subject based on the image region. The details of the processing for tracking the subject will be described later.

<Subject detection processing>
FIG. 5 shows an example of the subject detection processing.
This flowchart is a subroutine of S402 in FIG.

  First, in step S501, the subject detection unit 201 determines an evaluation area in which an image is evaluated. For example, in FIG. 6, 601 is an input image, 602 is an evaluation pixel, and 603 is an evaluation area candidate. As shown in FIGS. 6A, 6B, and 6C, the evaluation region candidates 603 are set to different sizes. The evaluation pixels 602 are sequentially selected from all the pixels of the input image 601 by raster processing.

  Next, in S502, the subject detection unit 201 determines an outer evaluation area with respect to the evaluation area. For example, in FIG. 7, reference numeral 701 denotes an input image, 702 denotes an evaluation pixel, 703 denotes an evaluation area, and 704 denotes an outer evaluation area. As shown in FIG. 7, the center position of the outer evaluation area 704 is the same as the center position of the evaluation area 703. Further, the size from the center position of the outer evaluation area 704 to the outer frame is larger than the size from the center position of the evaluation area 703 to the outer frame. That is, the outer evaluation area 704 is a ring-shaped area excluding the area corresponding to the evaluation area 703. Note that the difference between the size from the center position of the outer evaluation area 704 to the outer frame and the size from the center position of the evaluation area 703 to the outer frame is determined based on the size of the input image 701. 10% of the horizontal size.

  Next, in step S <b> 503, the subject detection unit 201 calculates the difference between the image signals corresponding to the evaluation area and the outer evaluation area as an evaluation value.

The evaluation value D is calculated by the following equation.

Here, d _H is the difference calculated from the hue information of the evaluation area and the outer evaluation area, p _Hi represents the number of pixels whose hue information of the evaluation area is i, and q _Hi is the outer evaluation area. Represents the number of pixels whose hue information is i. d _S , p _Si , and q _Si are values relating to chroma information, respectively, and d _V , p _Vi , and q _Vi are values relating to luminance information. M is the maximum value that can be taken by hue information, saturation information, and luminance information.

  Next, in S504, the subject detection unit 201 determines whether there is an unprocessed evaluation area. If there is an unprocessed evaluation area, the subject detection unit 201 returns the flow to S501, and repeats a series of processing. If there is no unprocessed evaluation area, the subject detection unit 201 advances the flow to S505.

Next, in S505, the subject detection unit 201 determines a subject. That is, the subject detection unit 201 determines the region having the highest evaluation value among the evaluation values calculated in S503 as the subject.
Note that the method described here is an example, and the subject can be determined by a method other than the flow in FIG. For example, in the flow of FIG. 5, it is also possible to determine a subject in consideration of depth information of an image.

<Process of acquiring the amount of background motion>
FIG. 8 illustrates an example of a process of acquiring a background motion amount.
This flowchart is a subroutine of S404 in FIG.

FIG. 8A is an example in which the amount of motion of the background is obtained from the estimated background.
First, in step S8011, the background motion amount acquisition unit 202 estimates a background based on a plurality of images sequentially acquired in step S401. For example, when a subject is detected by the subject detection processing in S402, the background motion amount acquisition unit 202 can estimate an image area other than the subject as the background.

  Next, in step S8012, the background motion amount acquisition unit 202 uses the image related to the current frame and the image related to the frame one frame earlier than the plurality of images sequentially acquired in step S401, A motion vector of the background is calculated from the estimated background (image).

  Finally, in step S8013, the background motion amount acquisition unit 202 acquires the background motion amount based on the background motion vector.

FIG. 8B is an example in which a plurality of motion vectors are calculated by feature point matching, and a background motion vector is selected from these to obtain a background motion amount.
First, in S8021, the background motion amount acquisition unit 202 uniformly distributes a plurality of feature points in the image acquired in S401, and calculates a motion vector for each feature point from the plurality of uniformly distributed feature points. Is calculated.

  Next, in S8022, the background motion amount acquisition unit 202 selects a background motion vector from the plurality of calculated motion vectors. As a method of selecting a background motion vector, a known method can be used. For example, the background motion amount acquisition unit 202 converts the plurality of motion vectors calculated in S8021 into a histogram, and determines the motion vector related to the maximum number of bins as the background motion vector.

  Finally, in S8023, the background motion amount acquisition unit 202 acquires the background motion amount based on the background motion vector.

  In calculating the background motion vector, the start point of the motion vector is detected from the image of the frame immediately before the image of the current frame, and the end point of the motion vector is detected from the image of the current frame. It is desirable. However, if the direction of the amount of motion of the background obtained from the motion vector of the background with respect to time is the same as the direction of the amount of motion of the subject described later with respect to time, the start point and the end point of the motion vector may be reversed.

  Further, in the present embodiment, the motion vector calculated from the image is used to obtain the amount of movement of the background. Alternatively, the motion of the imaging device is detected by a gyro sensor, an acceleration sensor, or the like, and The amount of movement of the background may be determined based on the movement of the device.

<Feature amount area update processing>
FIG. 9 illustrates an example of a process of updating a feature amount area.
This flowchart is a subroutine of S405 in FIG.

  First, in step S901, the feature amount area update unit 204 calculates a difference between the background motion amount acquired in step S8013 or S8023 and the subject motion amount. This difference corresponds to the amount of change in the positional relationship between the subject and the background. Here, the movement amount of the subject has already been acquired by the subject tracking unit 205 from an image related to a frame earlier than the current frame, and is stored in the RAM 154, for example.

  Regarding the difference between the amount of movement of the subject and the amount of movement of the background, the feature amount area update unit 204 calculates the absolute value of the difference in the horizontal direction of the image and the absolute value of the difference in the vertical direction of the image, One of them or a value obtained by adding them is defined as a difference between the motion amount of the subject and the motion amount of the background. For example, the feature amount area update unit 204 determines the one having the larger value among the absolute value of the difference in the horizontal direction and the absolute value of the difference in the vertical direction as the difference between the motion amount of the subject and the motion amount of the background.

  Next, in step S902, the feature amount area update unit 204 updates the size of the feature amount area registered in the feature amount area registration unit 202 based on the difference between the movement amount of the subject and the movement amount of the background. End the flow.

  Here, in S902, when updating the size of the feature amount area, the feature amount area update unit 204 determines the difference between the movement amount of the subject and the movement amount of the background, that is, the change amount of the positional relationship between the subject and the background. Becomes larger, the size of the feature amount area is reduced. In addition, the feature amount area updating unit 204 increases the size of the feature amount area as the difference between the movement amount of the subject and the movement amount of the background decreases.

FIG. 10 shows the relationship between the size of the feature amount area and the difference in the amount of movement between the subject and the background.
The default value (initial value) is the size of the feature amount area registered first in the feature amount area registration unit 202. The difference in the amount of motion corresponding to the default value is the reference value Δref.

  According to the figure, when the difference in the amount of motion is larger than the reference value Δref (ΔH), the size of the feature amount area changes from the default value to a value smaller than that (arrow A). When the difference between the motion amounts is smaller than the reference value Δref (ΔL), the size of the feature amount region changes from the default value to a larger value (arrow B). That is, as the difference in the amount of motion increases, the size of the feature amount region decreases, and as the difference in the amount of motion decreases, the size of the feature amount region increases. However, when the difference between the motion amounts is larger than the threshold value Δth, the size of the feature amount region is constant.

FIG. 11 schematically shows the relationship between the size of the feature amount area and the difference in the amount of motion.
FIG. 7A shows an image related to n frames, and shows the feature amount area 1101 registered in S403. The size of the feature amount area 1101 has, for example, a default value. FIG. 7B is an image relating to the (n + 1) frame, and is a case where there is no or almost no difference in the amount of motion from the image of FIG. FIG. 11D shows an image relating to the (n + 1) frame, and shows a case where the difference in the amount of motion is larger than the image shown in FIG.

  In the case of FIG. 11B, in S405, the feature amount area update unit 204 determines that the difference in the amount of motion is smaller than the reference value Δref. Therefore, as shown in FIG. 10C, the feature amount area updating unit 204 updates the size of the feature amount area 1102 to a value larger than the size of the feature amount area 1101 shown in FIG.

  In this case, the feature amount area 1102 is larger than the subject area and includes information on the subject and the background. That is, when searching for an image area corresponding to the feature amount area 1102 in an image related to the (n + 2) frame, information used for the search increases, and tracking performance can be improved. In addition, since the background information included in the feature amount area 1102 does not change or hardly changes, a situation in which tracking fails due to a change in the background information does not occur.

  In the case of FIG. 11D, in S405, the feature amount area update unit 204 determines that the difference in the amount of motion is larger than the reference value Δref. Accordingly, as shown in FIG. 10E, the feature amount area updating unit 204 updates the size of the feature amount area 1103 to a value smaller than the size of the feature amount area 1101 shown in FIG.

  In this case, the feature amount area 1103 has substantially the same size as the subject area, and does not include or hardly includes the background. That is, when searching for an image region corresponding to the feature amount region 1103 in an image related to the (n + 2) frame, a situation in which tracking fails due to a change in background information of the feature amount region 1103 does not occur. Absent.

<Subject tracking processing>
Next, the subject tracking processing in S405 will be described with reference to FIG.
The subject tracking unit 205 searches for an image area corresponding to the feature amount area updated in S404. The search result is output as output information of the subject tracking unit 205.

FIG. 12 shows an example of template matching.
The search for the image area corresponding to the feature amount area can be performed by template matching. The template matching is a technique of setting a pixel pattern as a template and searching for an area having the highest similarity with the template in the image. As the similarity between the template and the image area, a correlation amount such as a sum of absolute differences between corresponding pixels can be used.

  FIG. 12A schematically shows a template 1201 and a configuration example 1202 thereof. When performing template matching, the pixel pattern of the template 1201 is set in advance. In the present embodiment, the template 1201 has a size of the number of horizontal pixels W and the number of vertical pixels H. Then, the template matching is performed using the luminance values of the pixels included in the template 1201.

The characteristic amount T (i, j) of the template 1201 is represented by the following equation when a coordinate system as shown in FIG.

  FIG. 12B schematically shows an image area 1204 in the search area 1203 and a configuration example 1205 thereof. A search area 1203 indicates a range in which pattern matching is performed in an image of a subsequent frame in which tracking of a subject is performed. The search area 1203 may be the whole or a part of the image.

  The coordinates in the search area 1203 are represented by (x, y). The image area 1204 is for searching the search area 1203 for an image area having the highest similarity to the template 1201, and shifts within the search area 1203. For this reason, the size of the search area 1203 is set to be the same as the size of the template 1201 (the number of horizontal pixels W, the number of vertical pixels H).

Every time the image area 1204 is shifted, the subject tracking unit 205 calculates the similarity between the luminance value of each pixel included in the image area 1204 and the luminance value of each pixel included in the template 1201 (pattern matching). Here, the characteristic amount S (i, j) of the image area 1204 is represented by the following equation when a coordinate system as shown in FIG. 12B is set.

但し、Ｖ（ｘ，ｙ）は、画像領域１２０４の左上頂点の座標（ｘ，ｙ）における評価値として表す。 Also, assuming that an evaluation value indicating the similarity between the template 1201 and the image area 1204 is V (x, y), V (x, y) is a sum of absolute difference (SAD) represented by the following equation. It is represented by

Here, V (x, y) is represented as an evaluation value at the coordinates (x, y) of the upper left vertex of the image area 1204.

  Then, the subject tracking unit 205 shifts the image area 1204 one pixel at a time from the upper left (x = 0, y = 0) of the search area 1203 to the right (x direction), and evaluates the evaluation value V (x, y) is calculated. Further, when the image area 1204 reaches the right end (x = X−W) of the search area 1203, the subject tracking unit 205 returns the image area 1204 to the left end (x = 0) and then moves down (y). 1) (y = 1). Then, the subject tracking unit 205 shifts one pixel at a time from the left end (x = 0, y = 1) of the search area 1203 to the right (x direction) again, and changes the evaluation value V (x, y) at each position. calculate.

  The subject tracking unit 205 repeats the above operation until the search area 1203 reaches the lower right of the search area 1203 (x = X−W, y = Y−H). Here, X is the number of horizontal pixels in the search area 1203 (the number of pixels in the x direction), and Y is the number of vertical pixels in the search area 1203 (the number of pixels in the y direction). Further, the subject tracking unit 205 determines that the image area 1204 at the position V (x, y) having the lowest evaluation value among the evaluation values V (x, y) calculated at each position is different from the template 1201. It is determined that the image area has the highest similarity.

  As described above, the subject tracking unit 205 determines the position of the pixel region 1204 where the evaluation value V (x, y) indicates the minimum value, the position of the pixel region corresponding to the feature amount region, that is, the position of the subject (tracking target). To be determined. When the reliability of the search result is low, for example, when the minimum value of the evaluation value V (x, y) is larger than the threshold, the subject tracking unit 205 cannot find an image region corresponding to the feature amount region. May be determined.

  In the present embodiment, an example in which a luminance value is used as a feature amount in pattern matching has been described. However, elements (lightness, hue, saturation, and the like) other than the luminance value may be used. Further, the evaluation value V (x, y) in pattern matching may be determined in consideration of a plurality of elements (such as luminance value, lightness, hue, and saturation). Further, the evaluation value V (x, y) can be represented by a normalized cross-correlation (NCC), a ZNCC, or the like instead of the SAD.

<Other embodiments>
A program for realizing each function or each process of the above-described embodiment can be installed in advance in the flash memory 155 in the imaging device 100 or in a ROM (not shown) instead of the flash memory 155. In addition, a program that implements one or more functions or processes of the above-described embodiment can be supplied to the imaging device 100 via a network or a storage medium. In this case, the main control unit 151 in the imaging device 100 or at least one processor (not shown) instead of the main control unit 151 executes the supplied program, so that each function or each process of the above-described embodiment is performed. Is achieved.

  Further, as described above, each function or each process of the above-described embodiment can be realized by a program (software) installed in a computer and also by hardware (circuit). For example, one or more functions or processes of the above-described embodiment can be realized by a circuit such as an ASIC (Application Specific IC) or an FPGA (Field Programmable Gate Array).

<Conclusion>
As described above, according to the present invention, it is possible to provide an imaging apparatus that can always obtain high tracking performance regardless of a tracking situation by determining the size of a feature amount area based on image information. For example, in tracking under conditions in which the positional relationship between the subject and the background changes significantly, the feature amount region is set to be approximately the same size as the subject region, and in tracking under conditions in which the positional relationship between the subject and the background hardly changes, The feature amount area is made larger than the subject area. This makes it possible to always obtain high tracking performance regardless of the tracking situation.

161: subject recognition unit 201: subject detection unit 202: feature amount region registration unit 203: background motion amount acquisition unit 204: feature amount region update unit 205: subject tracking unit

Claims (15)

An imaging device capable of acquiring a first image and a second image in time series,
Subject detection means for detecting a subject as a tracking target from the first image;
Feature amount region registration means for registering a feature amount region including the subject based on the first image;
A feature amount area update unit that updates a size of the feature amount area registered in the feature amount area registration unit based on information related to the first and second images;
Subject tracking means for tracking the subject by performing a search for an image area corresponding to the feature quantity area updated by the feature quantity area updating means from the second image.
An imaging device characterized by the above-mentioned. The feature amount area registration unit sets a size of the feature amount area to be registered based on the first image as an initial value.
The imaging device according to claim 1, wherein: The feature amount area update unit, when the amount of change in the positional relationship between the subject and its background is larger than a reference value, makes the size of the feature amount area smaller than the initial value, and the amount of change is smaller than the initial value. When smaller than a reference value, the size of the feature amount region is larger than the initial value,
The imaging device according to claim 2, wherein: The feature amount area update unit reduces the size of the feature amount area as the change amount increases, and increases the size of the feature amount area as the change amount decreases.
The imaging device according to claim 3, wherein: The feature amount area update unit, when the change amount is larger than a threshold, makes the size of the feature amount area constant,
The imaging device according to claim 4, wherein: When the search by the subject tracking unit is performed and the tracking is successful, the feature amount region registration unit replaces the already registered feature amount region with the feature amount region used in the search. Register a new,
The imaging device according to claim 1, wherein: The information includes at least one of an average brightness of a screen, a white balance, and positional information based on a GPS function.
The imaging device according to claim 1, wherein: The information includes a motion amount of the subject and a motion amount of a background of the subject.
The imaging device according to claim 1, wherein: The image processing apparatus further includes a background motion amount acquisition unit that acquires the background motion amount from the first and second images,
The subject tracking means acquires a motion amount of the subject from the first and second images,
The feature amount region update unit updates the size of the feature amount region based on the amount of movement of the subject and the amount of movement of the background.
The imaging device according to claim 8, wherein: The background motion amount obtaining means includes:
Estimating the background from the first and second images;
Detecting a motion vector between the first and second images based on the estimated background;
Obtaining the amount of motion of the background based on the motion vector;
The imaging device according to claim 9, wherein: The background motion amount obtaining means includes:
Detecting the movement of the imaging device by at least one of a gyro sensor and an acceleration sensor,
Acquiring the amount of movement of the background based on the movement of the imaging device;
The imaging device according to claim 9, wherein: The apparatus further includes control means for controlling a shooting condition for the subject based on tracking of the subject.
The imaging device according to claim 1, wherein: An image sensor that captures the first image and the second image in time series;
At least one processor for processing the first image and the second image,
The at least one processor comprises:
Subject detection means for detecting a subject as a tracking target from the first image;
Feature amount region registration means for registering a feature amount region including the subject based on the first image;
A feature amount area update unit that updates a size of the feature amount area registered in the feature amount area registration unit based on information related to the first and second images;
Subject tracking means for tracking the subject by performing a search for an image area corresponding to the feature quantity area updated by the feature quantity area updating means from the second image.
An imaging device characterized by the above-mentioned. A control method of an imaging device capable of acquiring a first image and a second image in time series,
A subject detection step of detecting a subject as a tracking target from the first image;
A feature amount region registration step of registering a feature amount region including the subject based on the first image;
A feature amount region updating step of updating the size of the feature amount region registered in the feature amount region registration unit based on information on the first and second images;
A subject tracking step of performing tracking of the subject by performing a search for an image area corresponding to the feature amount area updated by the feature amount area updating unit from the second image.
A control method characterized in that:   A program for causing a computer to function as each unit of the imaging apparatus according to claim 1.

Images