JP6346484B2 - Image processing apparatus and control method thereof - Google Patents

Description

  The present invention relates to an image processing apparatus and a control method thereof, and more particularly to a subject tracking technique.

  Conventionally, in a camera equipped with a solid-state image sensor, a function for tracking the main subject by continuously detecting an arbitrary subject (main subject) extracted from the captured image from the captured image is practical. It has become. In such a camera, it is possible to perform automatic focus adjustment and automatic exposure calculation appropriate for the main subject by detecting the position of the main subject in the image.

  Some cameras having such subject tracking technology have a function of displaying the position of the main subject so that the user can recognize the main subject being tracked. For example, Patent Document 1 discloses a camera having a function of highlighting an AF frame mark that overlaps the position of a main subject detected by template matching on a display unit of an optical viewfinder. Thereby, the user can observe the subject optical image projected on the focusing screen and the image in which the image of the AF frame is superimposed with the optical viewfinder.

JP 2011-002814 A

  However, the optical viewfinder image in the camera of Patent Document 1 is a composite image of the subject optical image projected on the focusing screen and the AF frame image displayed on the display unit of the optical viewfinder. For this reason, when subject tracking is performed using an image obtained by photographing an optical viewfinder image with an AE sensor (imaging device), an AF frame that is not included in the subject optical image but is included in the optical viewfinder image. However, this image has a problem that the tracking accuracy deteriorates.

  The present invention has been made in view of the above-described problems of the prior art, and provides an image processing apparatus and a control method thereof that can suppress a decrease in accuracy of subject tracking processing using an image in which an image of additional information is superimposed on a subject image. For the purpose of provision.

  The purpose described above is to acquire an image obtained by superimposing an additional information image on a subject optical image, and to detect a region corresponding to a reference image including a subject to be tracked from a plurality of images acquired by the acquisition unit. Thus, the tracking means for realizing tracking of the subject and the updating means for updating the reference image, the updating means converts the pixels of the additional information image included in the original image of the new reference image to the pixels before the update. This is achieved by an image processing apparatus characterized in that the updated reference image is generated by complementing with pixels of the reference image.

  With such a configuration, according to the present invention, it is possible to provide an image processing apparatus and a control method thereof that can suppress a decrease in accuracy of subject tracking processing using an image in which additional information is superimposed on a subject image.

A flowchart for explaining the operation of the digital single-lens reflex camera according to the first embodiment. 1 is a side sectional view of a digital single-lens reflex camera as an example of an imaging apparatus according to an embodiment. 1 is a block diagram showing a functional configuration example of a digital single-lens reflex camera according to an embodiment. The figure which shows the example of the ranging point arrangement | positioning and imaging range of the digital single-lens reflex camera which concerns on embodiment The figure which shows typically the reference | standard image generation process in the digital single-lens reflex camera which concerns on 1st Embodiment. The figure which shows typically the reference | standard image update process in the digital single-lens reflex camera which concerns on 1st Embodiment. The figure which shows typically the template matching process which a digital single-lens reflex camera performs to 2nd Embodiment. Flowchart for explaining the operation of the digital single-lens reflex camera according to the third embodiment Flowchart for explaining the operation of the digital single-lens reflex camera according to the fourth embodiment

● (first embodiment)
Hereinafter, the present invention will be described in detail based on exemplary embodiments with reference to the accompanying drawings. In the following embodiment, an example in which the present invention is applied to an imaging apparatus that is an example of an image processing apparatus will be described. However, an image in which an additional information image used for subject tracking in the present invention is superimposed on a subject optical image is described below. It may be recorded in advance. In addition, information on the shape and position of the additional information image and information on the distance measuring point used at the time of photographing can be acquired as additional information of the image data. Therefore, the photographing function is not essential to the present invention, and can be implemented in a general electronic device. The imaging device means an electronic device having an imaging function, and includes any electronic device that has or can use a camera, such as a mobile phone, a game machine, and a personal computer.

  FIG. 2 is a side sectional view showing an example of the arrangement of main components of a digital single lens reflex camera (DSLR) as an example of an imaging apparatus according to an embodiment of the present invention. The DSLR includes a main body 201 and a photographing lens 202 attached to the front surface of the main body 201. The taking lens 202 can be exchanged, and the main body 201 and the taking lens 202 are mechanically and electrically connected by a lens mount (not shown). The taking lens 202 includes a focusing lens 213 and a diaphragm 214. The focusing lens 213 and the diaphragm 214 operate according to electrical control by the main body 201 via the lens mount contact group 215, and adjust the focus of the photographing lens 202 and the amount of light incident from the photographing lens 202.

  The main mirror 203 is a half mirror, and a sub mirror 204 is provided on the back surface. In the state where the mirror is not raised, a part of the light beam from the photographing lens 202 is reflected and incident on the upper finder optical system, and the transmitted light is reflected by the sub mirror 204 and incident on the lower AF unit 205. . In a mirror-up state, such as during exposure, it is retracted from the optical path.

  The AF unit 205 is a phase difference detection type AF sensor. By forming the secondary imaging plane of the photographic lens 202 on the focus detection line sensor of the AF unit 205, the defocus amount and the focal direction of the photographic lens 202 are detected, and the focusing lens 213 is based on the detection result. To adjust the focus automatically. Since focus control by the phase difference detection method is known and a known configuration can be applied to the AF unit 205, a detailed configuration of the AF unit 205 and description of focus control are omitted. In the AF unit 205 of the present embodiment, 61 ranging points (also called focus detection areas or AF frames) are laid out as shown in FIG.

  The focus plate 206 is a translucent plate-like member disposed on the planned image plane of the photographic lens 202 as a finder optical system. In addition, a finder display unit 220 is further disposed on or near the planned imaging plane of the photographic lens 202. The finder display unit 220 is a transmissive liquid crystal display, for example, and displays an additional information image such as a mark indicating the frame of the distance measuring point shown in FIG. The user observes the focusing screen 206 from the eyepiece 209 through the finder optical path changing pentaprism 207 to view an image in which the additional information image displayed on the finder display unit 220 is superimposed on the optical image of the field of view to be photographed. be able to. The finder display unit 220 may use other types of display devices such as a light emitting diode (LED), an organic light emitting diode (OLED), and an organic EL display, or a plurality of display devices. Different display devices may be used depending on display contents. The finder display unit 220 may also serve as the focus plate 206.

  The AE unit 208 converts an optical viewfinder image into photometric image data with an AE sensor having a plurality of pixels in order to observe the brightness of the subject. Here, it is assumed that the AE sensor captures the range indicated as the photometric sensor range 402 in FIG. On this optical viewfinder image, information such as a frame of a distance measuring point displayed on the viewfinder display unit 220 is superimposed. The AE sensor may have the same configuration as that of the image sensor 211 except for the number of pixels. However, since the AE sensor is mainly used for detection of subject luminance, it is called an AE sensor. The photometric image data has R, G, and B components. In the present embodiment, the AE unit 208 performs subject brightness detection, exposure calculation, subject detection, subject tracking, and the like using the generated photometric image data. Note that the dynamic range of the photometric image data may be expanded by adding a plurality of photometric image data as necessary.

  The imaging element 211 is a CMOS image sensor, a CCD image sensor, or the like, and a plurality of pixels having photoelectric conversion elements are arranged. By retracting the main mirror 203 and the sub mirror 204 from the photographing optical path and opening the focal plane shutter 210, the image sensor 211 is exposed and a subject optical image is photographed.

  The display 212 is a liquid crystal display (LCD), for example, and displays a live view image and uses it as an electronic viewfinder, or displays various setting values, shooting information, device information, GUI screens, shot images, and the like. .

  FIG. 3 is a block diagram illustrating a functional configuration example of the DSLR according to the present embodiment. In addition, the same code | symbol is attached | subjected about the same component as FIG.

  The operation unit 301 detects an operation performed by a user via a button, a switch, a dial, a connected device, or the like attached to the main body 201 or the photographing lens 202, and outputs a signal corresponding to the operation content to the system control unit 306. . Further, the operation unit 301 transmits a SW1 signal and a SW2 signal according to the amount of pressing the release button. Specifically, the SW1 signal is output to the system control unit 306 when the release button is pressed about half, and the SW2 signal is output when the release button is further pressed by a predetermined amount. The pressed state of the release button that outputs the SW1 signal is called “half-pressed”, and the pressed state of the release button that outputs the SW2 signal is called “fully pressed”. In addition, a state where the half-pressed state of the release button is maintained is referred to as “SW1 holding state”, and a state where the fully pressed state is maintained is referred to as “SW2 holding state”. In addition, the operation unit 301 displays the SW1 release signal when the release button is pressed less than the SW1 signal is transmitted, and the SW2 release signal when the release button is less than the SW2 signal is transmitted. Output to.

  The system control unit 306 may be a processor that can execute a program, such as a CPU or MPU, and controls the DSLR 100 by executing a program recorded in a non-volatile memory or storage device. Note that at least some of the functions of the AE control unit 302 and the AF control unit 304 may be executed by the system control unit 306.

  As described above, the AF unit 205 performs automatic focus detection by the phase difference detection method and drives the focusing lens 213 to the in-focus position (automatic focus adjustment) based on the focus detection result. The AF unit 205 includes an AF control unit 304 and a distance measuring sensor 305. The distance measuring sensor 305 is composed of, for example, a plurality of line sensors corresponding to the 61 distance measuring points shown in FIG. 4, and converts the light incident through the sub mirror 204 into an electric signal, for each distance measuring point. The pair of image signals are output to the AF control unit 304. The AF control unit 304 performs a focus adjustment calculation based on a focus detection result of a distance measuring point near the object position detected by the object tracking process or a distance measuring point with the smallest defocus amount, and the lens mount contact group 215. Through, the position of the focusing lens 213 is controlled.

  The AE unit 208 performs automatic exposure calculation and subject tracking. The AE unit 208 includes an AE control unit 302 and a photometric sensor 303. The AE control unit 302 performs automatic exposure calculation based on the photometric image data read from the photometric sensor 303 and outputs the result to the system control unit 306. The system control unit 306 controls the value (aperture size) of the aperture 214 based on the result of the automatic exposure calculation output from the AE control unit 302 and adjusts the amount of light incident on the main body 201.

  In the SW1 holding state and during continuous shooting, the AE control unit 302 tracks the subject by sequentially detecting the subject using photometric image data sequentially obtained by continuous imaging by the photometric sensor 303, and detects the detected subject. The position information is output to the system control unit 306. Details of the subject detection processing and subject tracking processing will be described later. The system control unit 306 outputs the subject position information output from the AE control unit 302 to the AF control unit 304.

  The AF control unit 304 calculates the defocus amount of the distance measuring point at or near the position of the subject output from the system control unit 306, and compares the calculation result with a preset threshold value. If the defocus amount is equal to or smaller than the threshold value as a result of the comparison, the selected ranging point position data output to the system control unit 306 is selected from the ranging points selected by the AF control unit 304 based on the defocus amount at each ranging point. Update to a distance measuring point near the subject.

  When the SW2 signal is output from the operation unit 301, the system control unit 306 raises the main mirror 203 (and the sub mirror 204) and retracts it from the optical path. Then, the focal plane shutter 210 is driven to expose the image sensor 211 for a time based on the result of the automatic exposure calculation. When the exposure time ends, the system control unit 306 lowers the main mirror 203 (and the sub mirror 204).

  The image sensor 211 converts light incident through the photographing lens 202 during the exposure time into an electrical signal at each pixel, generates image data, and outputs the image data to the system control unit 306. The system control unit 306 applies predetermined image processing or the like to the image data output from the image sensor 211 and displays the data on the display 212 or writes it in the image storage device 307.

  FIG. 4 shows an example of the relationship between the imaging ranges of the image sensor 211 and the photometric sensor 303 in this embodiment and the arrangement of distance measuring point marks that can be displayed by the finder display unit 220. The imaging range of the image sensor 211 is 401, the imaging range of the photometric sensor 303 is indicated by 402, and the distance measuring point of the distance measuring sensor 305 is indicated by a distance measuring point mark 403 which is a frame-shaped fixed pattern.

  When the main mirror 203 is in the down state shown in FIG. 2, the subject optical image is projected onto the focus plate 206, so that the user can view the subject optical image by looking into the eyepiece 209. Also, among the distance measuring point marks 403 in FIG. 4, the one corresponding to the distance measuring point used for focus adjustment at that time is displayed on the finder display unit 220, so that the user can display the distance measuring point on the subject optical image. The distance measuring point can be confirmed from the optical viewfinder image on which the mark is superimposed.

Since the photometric sensor 303 takes an optical viewfinder image and generates image data used for photometry and subject tracking, a distance measuring point mark 403 is reflected in the subject optical image in the image data used for subject tracking.
The position in the image of the distance measuring point mark 403 reflected in the image data obtained by the photometric sensor 303 is the distance measuring point arrangement information stored in advance in the non-volatile memory in the main body 201 and the time of shooting. It can be discriminated from the information of the distance measuring points used.

FIG. 1 is a flowchart showing the operation of the DSLR 100 in a state where SW1 is pressed.
In step S101, the system control unit 306 sets the reference image flag to FALSE. The reference image flag is a flag that is TRUE if the reference image has been generated even once after the tracking process is started.

  In S102, if the value of the reference image flag is TRUE, the system control unit 306 advances the process to S103 in order to perform the tracking process because the tracking reference image has been generated. If the value of the reference image flag is FALSE, the system control unit 306 advances the process to S109 in order to generate a reference image because no reference image has been generated.

  S109 to S113 are processes executed when there is no reference image. In S109, the system control unit 306 instructs the AF control unit 304 to start the operation, and the AF control unit 304 performs charge accumulation and distance measurement in the distance measurement sensor 305. Perform a read.

  In S110, the AF control unit 304 calculates the defocus amount for all the distance measurement points based on the image signal read from the distance measurement sensor 305, and selects the distance measurement point having the minimum defocus amount. Then, the AF control unit 304 outputs information on the selected distance measuring point to the system control unit 306. The system control unit 306 causes the finder display unit 220 to display a ranging point mark corresponding to the selected ranging point.

  In step S <b> 111, the system control unit 306 instructs the AE control unit 302 to start operation in order to newly generate a reference image, and the AE control unit 302 executes charge accumulation and readout in the photometric sensor 303. In addition, the system control unit 306 notifies the AE control unit 302 of information on the selected distance measuring point.

  In step S112, the AE control unit 302 cuts out an area having a predetermined size centered on the position of the distance measuring point selected in step S110 from the image read from the photometric sensor 303 in step S111, and generates a reference image for tracking processing. To do.

  In S110, a distance measuring point mark corresponding to the selected distance measuring point is displayed on the finder display unit 220. Therefore, the distance measuring point mark is reflected in the optical viewfinder image acquired by the photometric sensor 303 in S111. Since the reference image is generated around the selected distance measuring point, the distance measuring point mark is also reflected in the reference image.

  FIG. 5 schematically shows an example of the relationship between the image 501 captured by the photometric sensor 303, the distance measuring point mark 502, and the reference image 504. Here, a distance measuring point corresponding to a dog face existing in the center of the photographing range is selected, and a corresponding distance measuring point mark 502 is displayed. A reference image 504 is generated by cutting out a predetermined region 503 centered on the selected distance measurement point. Therefore, a distance measurement point mark 502 is reflected in the reference image 504.

  Returning to FIG. 1, since the system control unit 306 newly generates a reference image in S113, the reference image flag is set to TRUE, and the process proceeds to S114.

  On the other hand, if the reference image flag is True in S102, the system control unit 306 instructs the AE control unit 302 to start operation in S103, and the AE control unit 302 executes charge accumulation and readout in the photometric sensor 303.

  In S104, the AE control unit 302 performs subject tracking processing. The AE control unit 302 uses the reference image as a tracking target image (template), and executes template matching processing on the image read in S102. Then, the AE control unit 302 detects the position of the reference image in the read image as a subject tracking result and notifies the system control unit 306 of it.

  Although there is no restriction | limiting in particular about the specific method of a template matching process, For example, it can carry out as follows. First, the AE control unit 302 cuts out an arbitrary region in the image so as to have the same size as the reference image from the image read from the photometric sensor 303 and sets it as a tracking input image. Then, the AE control unit 302 calculates the sum of the difference values of the luminance signal and the color difference signal for all corresponding pixels of the tracking input image and the reference image, and uses the result as a matching evaluation value. The smaller the matching evaluation value, the higher the degree of correlation.

  The AE control unit 302 calculates the matching evaluation value for the entire image while shifting the position where the tracking input image is cut out by a predetermined value, and the tracking input image where the matching evaluation value with the highest correlation is obtained is tracked. The target detection position (subject position) is used. The subject position may be one or more of the vertex coordinates and the center coordinates of the cutout range. Since the size of the tracking input image is known to the system control unit 306, the system control unit 306 can specify the range of the tracking input image corresponding to the subject position. Information on the subject position obtained by subject tracking processing in the AE control unit 302 is notified to the AF control unit 304 through the system control unit 306.

  In step S <b> 105, the system control unit 306 instructs the AF control unit 304 to start operation, and the AF control unit 304 performs charge accumulation and readout in the distance measuring sensor 305.

  In step S106, the AF control unit 304 calculates the defocus amount for all the distance measuring points based on the image signal read from the distance measuring sensor 305, and typically the distance measuring point at the position in consideration of the subject position, typically Select the closest focus point. Then, the AF control unit 304 outputs information on the selected distance measuring point to the system control unit 306. The system control unit 306 causes the finder display unit 220 to display a ranging point mark corresponding to the selected ranging point. At this time, if there are a plurality of distance measuring points close to the subject position, a plurality of distance measuring point marks are displayed. Here, the distance measuring point close to the subject position is, for example, a distance measuring point included in the tracking input image region corresponding to the subject position and a distance measuring point closest to the tracking input image region corresponding to the subject position. It may be.

  In step S107, the AE control unit 302 determines whether the reference image needs to be updated. Here, it is determined that the reference image is to be updated when the reliability of the subject position obtained in the subject tracking process of S104 is high, and the process proceeds to S108. This is because if the reliability is low, the accuracy of the complementary processing of the reference image described later is lowered. On the other hand, if it is determined that the reliability of the subject position obtained by the tracking process is not high, the AE control unit 302 determines that the reference image is not updated, and the process proceeds to S114.

  Although there is no particular limitation on the reliability index of the subject position, for example, a matching evaluation value having the highest correlation (ie, corresponding to the tracking input image corresponding to the subject position) obtained in the template matching process can be used. . It can be determined that the reference image is updated when the matching evaluation value having the highest correlation value is equal to or less than a first predetermined value.

  In S108, the AE control unit 302 replaces the reference image with the tracking input image that has obtained the matching evaluation value having the highest degree of correlation in the tracking process, and complements the pixels of the distance measuring point mark portion that is reflected, thereby correcting the reference image. Update.

  The reference image update process in S108 will be described in detail with reference to FIG. FIG. 6 schematically illustrates a case where subject tracking using the reference image 504 is performed and a matching evaluation value with the highest degree of correlation is obtained for the tracking input image 602 obtained by cutting out the area 606. Further, the image 601 obtained by photographing the finder optical image with the photometric sensor 303 includes a plurality of distance measuring point marks 605 displayed on the finder display unit 220 and close to the subject position. Further, here, the distance measuring point mark is also reflected in the reference image 504 before the update.

  Here, in updating the reference image, as shown in FIG. 6, the pixel of the distance measuring point mark reflected in the tracking input image 602 as a new reference image is replaced with the pixel at the corresponding position in the current reference image. By replacing with, the reflected distance measuring point mark is removed. A tracking input image 602 after the replacement processing is shown as a complementary image 603.

  Usually, the distance measuring point mark can be removed by this replacement process. However, when a distance measuring point mark is reflected in the reference image as in the first generated reference image 504, the distance measuring point mark reflected in the same position in both the reference image 504 and the tracking input image 602. Also remains in the complementary image 603. Therefore, the AE control unit 302 complements such pixels with values obtained by interpolating surrounding pixel values in the complementary image 603, and generates a new reference image 604 from which the distance measuring point mark is removed. The new reference image 604 is used for the next tracking process.

  Note that the AE control unit 302 determines the position of the distance measuring point mark reflected in the current reference pixel and the position of the distance measuring point mark reflected in the tracking input image that is the basis of the new reference image, respectively. The selected distance measuring point information and the distance measuring point mark information can be discriminated. More specifically, information corresponding to the image coordinates of the corresponding distance measuring point mark is stored in advance for each distance measuring point position, and the image coordinates of the corresponding distance measuring point mark corresponding to the selected distance measuring point are stored. Can be obtained and clipped within the cut-out image range. Alternatively, the image coordinates of the distance measuring point mark are specified by extracting pixels of a specific color used for displaying the distance measuring point mark and specifying pixels that match the pattern (shape and size) of the distance measuring point mark. May be. These are merely examples, and other arbitrary methods can be used.

The AE control unit 302 specifies the pixel of the distance measuring point mark reflected at the same position in the tracking input image 602 that is the basis of the current reference image 504 and the new reference image, and the values of the surrounding pixels in the complementary image 603. Complement from Complementation can be performed by an arbitrary method such as a method used when the pixel value of the defective pixel position is obtained from the peripheral pixel value.
By updating the reference image as described above, the updated reference image does not include the distance measurement point mark, so the influence of the distance measurement point mark on the accuracy of the tracking process based on the degree of correlation with the reference image is reduced. Can be suppressed.

  In step S <b> 114, the AE control unit 302 performs automatic exposure calculation based on the image data read from the photometric sensor 303 in step S <b> 103 and outputs an exposure calculation result to the system control unit 306. In addition, when performing the automatic exposure calculation based on the main subject region that is the tracking target, the automatic exposure calculation may be performed based on the data of the reference image.

  In S115, the system control unit 306 drives the focusing lens 213 based on the focus detection result for the selected distance measuring point obtained in S106 or S110, and performs automatic focus adjustment. Further, based on the automatic exposure calculation result of S114, an aperture value and a shutter speed (and further photographing sensitivity as necessary) at the time of actual photographing are determined.

  In S116, the system control unit 306 determines whether or not the SW2 signal has been received from the operation unit 301. If the SW2 signal has been received, the system control unit 306 advances the processing to S117, and if not, returns the processing to S102.

In step S117, the system control unit 306 raises the main mirror 203 (and the sub mirror 204) and retracts it from the optical path. In addition, the diaphragm 214 and the focal plane shutter 210 are controlled according to the result of the automatic exposure calculation performed immediately before, and the main photographing is executed. Thereafter, the system control unit 306 lowers the main mirror 203 (and the sub mirror 204), and returns the process to S102.
The above processing is repeated while SW1 is ON (while the SW1 signal is received). When the SW1 signal is no longer received, the processing in FIG.

  As described above, according to the present embodiment, when subject tracking processing is performed using an optical viewfinder image in which a subject optical image and an additional information image are superimposed, additional information is updated when a reference image used for tracking processing is updated. By removing the image, the influence on the accuracy of the tracking process can be suppressed.

In the present embodiment, the distance measuring point mark included in the reference image generated first is not removed. However, the pixels of the distance measuring point mark may be complemented from the peripheral pixels.
Further, although the distance measuring point mark has been described as an example of the additional information image, other additional information images can be removed by the same processing.

● (Second embodiment)
Next, a second embodiment of the present invention will be described. In the first embodiment, when the reference image is updated, the distance measuring point marks reflected in the tracking input image as a new reference image are removed by complementing them based on the pixels of the current reference image or peripheral pixels. It was a thing. However, when interpolation from surrounding pixels is used, if the spatial frequency near the pixel to be complemented is high, proper interpolation may be difficult, and as a result, the accuracy of subject tracking processing may be reduced. The present embodiment is characterized in that such a decrease in accuracy of the subject tracking process is suppressed by changing the matching process in the subject tracking process.

  Since the present embodiment may be the same as the first embodiment except for the matching processing method in the subject tracking processing in S104 of FIG. 1, only the matching processing will be described. The matching process according to the present embodiment is characterized in that the pixels of the distance measuring point mark included in the reference image or the tracking input image are excluded from the correlation calculation targets.

  FIG. 7 is a diagram schematically showing matching processing (correlation calculation) between the reference image 701 of 8 × 8 pixels and the tracking input image 702. In FIG. 7, black pixels indicate distance measurement point pixels, and white pixels indicate subject optical image pixels. The matching area 703 indicates the logical sum of the reference image 701 and the tracking input image 702 for the pixels of the distance measuring point mark.

In the present embodiment, the total sum of the difference values between the pixels in the white pixel position in the matching region 703, that is, the position that is not the distance measuring point mark in either the reference image 701 or the tracking input image 702 is obtained. Then, a value normalized by dividing the sum by the number of white pixels in the matching region 703 (the number of pixel pairs for which the difference is obtained) is set as a matching evaluation value. This calculation is expressed as follows.
Here, Pb (x, y) in Expression (1) represents the pixel value of the (x, y) coordinate of the reference image 701, and Pi (x, y) represents the (x, y) coordinate of the tracking input image 702. Represents a pixel value. The function W (x, y) is 0 if the pixel of the distance measuring point mark (black pixel) in the coordinate (x, y) of the matching area 703, and 1 if the pixel of the subject optical image (white pixel). return it. Therefore, the difference value for the coordinates at which at least one of the reference image 701 and the tracking input image 702 becomes a pixel of the distance measuring point mark is always 0.

  By detecting the subject position based on the matching evaluation value obtained in this way, even when the distance measuring point mark is reflected in at least one of the reference image and the tracking input image, the object is affected by the distance measuring point mark. The degree of correlation can be calculated without any problem. That is, since the distance measurement point mark pixels are excluded and the degree of correlation can be calculated only for the subject optical image, a decrease in tracking accuracy due to the reflection of the distance measurement point mark can be reduced.

  As another method for obtaining the same effect, the reference image 701 and the tracking input image 702 may be subjected to the matching process after reflecting the pixel of the distance measuring point mark reflected in the other image. That is, the matching process is performed after both the reference image 701 and the tracking input image 702 have pixels of distance measuring point marks as shown in the matching area 703. As described above, after the pixels of the distance measuring point mark are shared by the reference image 701 and the tracking input image 602, the matching evaluation value according to the equation (1) is obtained.

  For the pixels of the distance measuring point mark, the difference value becomes 0 at the time of the matching processing, and thus is not counted in the sum of the difference values. Then, since the sum of the difference values is normalized by the number of pixels in the portion without the distance measuring point mark, a matching evaluation value excluding the pixels of the distance measuring point mark can be obtained.

  According to the present embodiment, when subject tracking processing is performed using an optical viewfinder image in which a subject optical image and an additional information image are superimposed, pixels of the additional information image are excluded and only pixels of the subject optical image are excluded. The subject position is detected based on the evaluation value based on the evaluation value. Therefore, even if the spatial frequency of the subject optical image in the vicinity of the distance measuring point mark is high, it is possible to obtain an accurate tracking process result.

● (Third embodiment)
Next, a third embodiment of the present invention will be described. The present embodiment is characterized in that the necessity of updating the reference image is determined in consideration of the number of pixels of the distance measuring point mark included in the current reference image. When the number of AF points included in the reference image is small, it is less likely to be updated than when there are many, thereby reducing the possibility that the AF points will enter the reference image. This suppresses a decrease in accuracy of the tracking process.

  This embodiment may be the same as the first embodiment except for the reference image update necessity determination process corresponding to S107 in FIG. Therefore, in the following description, reference image update necessity determination processing characteristic of the present embodiment will be described.

  FIG. 8 is a flowchart showing the operation of the DSLR 100 of this embodiment in a state where SW1 is pressed. Steps for performing the same processing as in the first embodiment are denoted by the same reference numerals as in FIG. Is omitted.

  In step S801, the AE control unit 302 advances the process to step S802 if the number of pixels of the distance measurement point mark included in the current reference image is less than the predetermined value, and advances the process to step S803 if the number is greater than the predetermined value.

  In step S802, the AE control unit 302 compares whether the highest correlation obtained in the subject tracking process in step S104 is higher than a second determination value that is higher than the first determination value. If the degree of correlation is higher than the second determination value, the AE control unit 302 advances the process to S108, and performs the above-described reference image update process. On the other hand, when the correlation degree is equal to or lower than the second determination value, the AE control unit 302 advances the process to S114 without updating the reference image. As described above, when the degree of correlation is represented by a matching evaluation value, the process of S802 is a determination as to whether or not it is less than a second determination value that is lower than the first determination value.

  In step S803, the AE control unit 302 compares whether the highest correlation obtained in the subject tracking process in step S104 is higher than the first determination value that is lower than the second determination value. If the degree of correlation is higher than the first determination value, the AE control unit 302 advances the process to S108, and performs the above-described reference image update process. On the other hand, when the correlation degree is equal to or less than the first determination value, the AE control unit 302 advances the process to S114 without updating the reference image.

  As described above, when the number of pixels of the additional information image included in the reference image is smaller than the predetermined value, the reference image is less likely to be updated than when the additional information image is greater than or equal to the predetermined value. The probability that pixels are mixed can be reduced.

  In the first and second embodiments, when the reliability of the subject tracking processing result is high, the reference image is updated regardless of the number of pixels of the additional information image included in the tracking input image corresponding to the highest correlation. It was something to do. However, if the number of pixels of the additional information image included in the reference image increases, the influence of the pixels of the additional information image in the correlation calculation increases or the number of pixels of the subject optical image used in the correlation calculation decreases. Reliability may be reduced. According to this embodiment, since it becomes difficult to update the reference image having a small number of pixels of the additional information image, occurrence of such a problem can be suppressed.

● (Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. In the present embodiment, in the case of having a subject detection function for detecting a specific subject such as a human face from an image, considering whether or not the reference image is generated at the position of the specific subject detected by the subject detection function, It is characterized by determining whether or not the reference image needs to be updated. If the reference image is generated at the position of the specific subject, there is a high possibility that correct tracking will be performed. It is.

  The DSLR according to the present embodiment has a function in which the AE control unit 302 or the system control unit 306 detects a specific subject area in an image, for example, a face detection function. Here, as a typical example, a person's face is described as being a specific subject, but it may be any subject that can be detected based on a feature amount of an image. The face detection function is a function for analyzing an input image to detect all faces existing in the image and specifying the coordinate position and size of each face.

  FIG. 9 is a flowchart showing the operation of the DSLR 100 of this embodiment in a state where SW1 is pressed, and the same reference numerals as in FIG. 1 or FIG. 8 are used for the steps for performing the same processing as in the first or third embodiment. A duplicate description will be omitted.

  In step S <b> 901, the system control unit 306 instructs the AE control unit 302 to start operation, and the AE control unit 302 executes charge accumulation and readout in the photometric sensor 303.

  In step S <b> 902, the AE control unit 302 performs face detection processing on the image read from the photometric sensor 303 and detects the position and size of the face present in the image. When a plurality of faces are detected, the AE control unit 302 selects one of them as the main face. The method of selecting the main face is arbitrary, but examples include the largest one, the one closest to the center of the screen, the one closest to the distance measuring point selected in advance by the user, and the like. If only one face is detected, that face becomes the main face. The AE control unit 302 notifies the system control unit 306 of information on the size and position of the main face.

In step S903, the system control unit 306 advances the process to step S904 if a face is detected in step S902, and advances the process to step S109 if no face is detected.
In S904 and S109, the system control unit 306 instructs the AF control unit 304 to start operation, and the AF control unit 304 executes charge accumulation and readout in the distance measuring sensor 305. Further, the system control unit 306 notifies the AF control unit 304 of the main face information notified from the AE control unit 302.

  In step S905, the AF control unit 304 calculates the defocus amount for all distance measuring points in the same manner as in step S110, and sets the distance measuring point closest to the main face as the selected distance measuring point. Alternatively, the AF control unit 304 may calculate the defocus amount only for the distance measuring point closest to the main face.

  In step S906, the AE control unit 302 cuts out an image from the image read from the photometric sensor 303 in step S901 according to the detection position and size of the main face, and sets it as a new reference image.

  On the other hand, if the reference image flag is True, after the ranging point is selected in S106, the AE control unit 302 generates the current reference image for the main face in S907 (S906). Generated). Then, the AE control unit 302 advances the process to S802 if the reference image is generated for the main face, and advances to S803 if not (generated in S112).

  The processing of S802 and S803 is the same as that of the third embodiment. However, the first and second determination values may or may not be the same as in the third embodiment (the second determination value> the first determination value does not change).

  When the reference image is generated for the main face detected by the face detection function, it is considered that the reference image is generated at an appropriate position and size at that time, and is more reliable than the reference image generated at a position other than the main face. It is considered possible. For this reason, the reference image generated for the main face is controlled to maintain high tracking accuracy by making it less likely to be updated than a reference image that is not.

(Other embodiments)
As mentioned above, although preferable embodiment of this invention was described, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary. For example, as a method for recognizing the position of a subject, tracking using color information as well as subject tracking by template matching may be used.

  In addition, when the additional information image is included in the current reference image, even when not updated, a tracking input image that is determined to have a high correlation enough to be used for complementing the additional information image is obtained. May supplement the pixels of the additional information image of the current reference image with the pixels of the tracking input image. As a result, the accuracy of the subject tracking process that is subsequently performed can be improved.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

DESCRIPTION OF SYMBOLS 201 ... Main body, 202 ... Shooting lens, 203 ... Main mirror, 204 ... Sub mirror, 206 ... Focus plate, 211 ... Imaging element, 212 ... Display, 213 ... Focusing lens, 214 ... Aperture

Claims (8)

Acquisition means for acquiring an image in which the additional information image is superimposed on the subject optical image;
For a plurality of images acquired by the acquisition means, tracking means for realizing tracking of the subject by detecting a region corresponding to a reference image including the subject to be tracked;
Updating means for updating the reference image,
The update means complements the pixels of the additional information image included in the original image of the new reference image with the pixels of the reference image before update to generate an updated reference image. An image processing apparatus.   The image processing apparatus according to claim 1, wherein the complement is replacement of a pixel of the additional information image with a pixel at the same position in the reference image before update.   The image processing apparatus according to claim 1, wherein the update unit supplements pixels of the additional information image that cannot be complemented using the reference image before the update using peripheral pixels.   The updating means updates the reference image when the number of pixels of the additional information image included in the reference image before the update is smaller than a predetermined value than when the number of pixels is equal to or larger than the predetermined value. The image processing apparatus according to claim 1, wherein the image processing apparatus is made difficult. A detecting unit for detecting a specific subject from the image acquired by the acquiring unit;
When the reference image before the update is generated for the specific subject detected by the detection unit, the update unit is configured for the specific subject detected by the detection unit. The image processing apparatus according to claim 1, wherein the reference image is less likely to be updated than when the reference image is not generated. It has an image processing device given in any 1 paragraph of Claims 1-5 ,
A photometric sensor that captures an optical viewfinder image is used as the acquisition unit. An acquisition step of acquiring an image in which the additional information image is superimposed on the subject optical image;
A tracking step that realizes tracking of the subject by detecting a region corresponding to a reference image including the subject to be tracked for the plurality of images acquired in the acquisition step;
An update means for updating the reference image,
In the updating step, the updating means supplements the pixels of the additional information image included in the image that is the basis of the new reference image with the pixels of the reference image before the update, and generates a reference image after the update. And a control method for the image processing apparatus. The program for functioning a computer as each means of the image processing apparatus of any one of Claim 1 to 5 .