JP2016081095A - Subject tracking device, control method thereof, image-capturing device, display device, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve the performance of tracking a subject.SOLUTION: The present invention is a subject tracking device for tracking a targeted subject from successively supplied images, the subject tracking device having: histogram matching means for collating a histogram expressing an object to be tracked and the histogram of the subregion of a supplied image, changing the subregion to be collated, and outputting a plurality of high-ranking similarities as found by collation and subregion position information; and similar object region registration means for registering, as a similarity object region, a subregion indicating a second rank or following similarity higher than a prescribed threshold that exists at a position away from the position of a subregion indicating a first rank similarity as collated by the histogram matching means. The histogram matching means does not include the region registered by the similar object region registration means in the outputted plurality of high-ranking similarities and subregion position information.SELECTED DRAWING: Figure 2

Description

  The present invention relates to a subject tracking device, a control method thereof, an imaging device, a display device, and a program.

  A technique for extracting a specific subject from an image supplied in time series and tracking the extracted subject is very useful. For example, it is used for specifying a human face region or a human body region in a moving image. . Such techniques can be used in many areas, for example, teleconferencing, man-machine interface, security, monitor systems for tracking any subject, image compression, etc.

  For digital still cameras and digital video cameras, a technique for extracting and tracking an arbitrary subject included in a captured image specified by using a touch panel or the like and optimizing a focus state and an exposure state for the subject is proposed. (See Patent Documents 1 and 2).

  Patent Document 1 discloses an imaging apparatus that detects (extracts) and tracks the position of a face included in a captured image, focuses the face, and captures an image with an optimal exposure. Patent Document 2 discloses a technique for automatically tracking a specific subject using template matching. Here, with template matching, a partial image obtained by cutting out an image area including a specific subject (tracking target) is registered as a template image, and an area having the highest similarity or the lowest dissimilarity with the template image is estimated in the image. In this method, a specific subject is tracked.

  In template matching, an arbitrary area included in the image is designated using an input interface such as a touch panel, and the template image is registered with the area as a reference.

JP 2005-318554 A JP 2001-60269 A

  In template matching, image data is used as an evaluation scale. There is a problem that a correct region cannot be estimated when there is a large change in the appearance of the subject such as a change in posture between the template image and the image to be searched. To solve this problem, there is a method of using a histogram instead of image data itself as a matching evaluation scale. By converting the information indicating the subject from image data into a histogram, subject tracking that is ambiguous and robust to changes in posture of the subject can be realized.

  On the other hand, in the matching based on the histogram, since the feature amount is made ambiguous, the similar object cannot be identified and the similar object may be tracked erroneously. In particular, in a scene where a similar target exists, there is a high possibility that an error will occur at the timing when the tracking target is concealed.

  The present invention has been made in view of the above-described problems, and an object of the present invention is to provide a subject tracking device that can reduce erroneous tracking of a similar target in a subject tracking method based on similarity of histograms. To do.

  In order to achieve the above object, the present invention provides a subject tracking device that tracks a target subject from sequentially supplied images, and includes a histogram of pixel values of a partial region indicating the target subject from the image. Using the histogram registration means to be registered and the histogram registered by the histogram registration means, the histogram of the pixel value of the partial area of the supplied image is collated, the partial area to be collated is changed, and the similarity by collation is high Histogram matching means for outputting a plurality of upper similarities and position information of the partial area, and a position separated from the position of the partial area indicating the first degree of similarity by the histogram matching means at the second and subsequent positions above a predetermined threshold When there is a partial region indicating the similarity, the similar target region registration means for registering the region as a similar target region, Strike grams matching means, the position information of the upper plurality of similarity and the partial area to output, is characterized by not including the area registered by the similar target area registering means.

  According to the subject tracking device of the present invention, it is possible to reduce the frequency of erroneous tracking when a similar target exists, and to improve subject tracking performance.

It is a block diagram which shows the structure of the imaging device which concerns on 1st Embodiment. It is a block diagram which shows the structure of the subject tracking which concerns on 1st Embodiment. It is a figure explaining the histogram matching which concerns on 1st Embodiment. It is a figure explaining the state of the tracking state discrimination | determination concerning 1st Embodiment, and its transition. It is a figure explaining registration of the similar object field concerning a 1st embodiment. It is a flowchart which shows the procedure of the subject tracking process which concerns on 1st Embodiment.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. However, the present invention is not limited to the following embodiments. The embodiments of the present invention show preferred modes of the invention and do not limit the scope of the invention.

[Example 1]
FIG. 1 is a schematic block diagram illustrating the configuration of the imaging apparatus 100. In this embodiment, the imaging apparatus 100 is embodied as a digital camera that captures an image of a subject. The imaging device 100 also functions as a subject tracking device that tracks subjects included in images that are sequentially supplied in time series. Imaging optical system 101, imaging device 102, analog signal processing unit 103, A / D conversion unit 104, control unit 105, image processing unit 106, display unit 107, recording unit 108, subject specifying unit 109 and a subject tracking unit 110.

  Referring to FIG. 1, light representing an image of a subject is collected by an imaging optical system 101 and is incident on an imaging device 102 configured by a CCD image sensor, a CMOS image sensor, or the like. The image sensor 102 outputs an electrical signal corresponding to the intensity of incident light in units of pixels (that is, photoelectrically converts an image of a subject formed by the imaging optical system 101). Here, the electrical signal output from the image sensor 102 is an analog image signal indicating an image of a subject imaged by the image sensor 102.

  The video signal output from the image sensor 102 is subjected to analog signal processing such as correlated double sampling (CDS) in the analog signal processing unit 103. The video signal output from the analog signal processing unit 103 is converted into a digital data format by the A / D conversion unit 104 and input to the control unit 105 and the image processing unit 106. The control unit 105 is a CPU, a microcontroller, or the like, and centrally controls the operation of the imaging apparatus 100. Specifically, the control unit 105 controls each unit of the imaging apparatus 100 by developing the program code stored in the ROM in the work area of the RAM and sequentially executing the program code.

  The image processing unit 106 performs image processing such as gamma correction and white balance processing on the input digital video signal. In addition to the normal image processing, the image processing unit 106 also has a function of performing image processing using information related to a subject area in an image supplied from a subject tracking unit 110 described later.

  The video signal output from the image processing unit 106 is sent to the display unit 107. The display unit 107 is, for example, an LCD or an organic EL display, and displays a video signal. The imaging apparatus 100 can cause the display unit 107 to function as an electronic viewfinder (EVF) by sequentially displaying images sequentially captured in time series by the imaging element 102 on the display unit 107. Further, the subject area including the subject tracked by the subject tracking unit 110 is displayed in a rectangle or the like.

  The video signal output from the image processing unit 106 is recorded on a recording medium 108, for example, a removable memory card. Note that the recording destination of the video signal may be the built-in memory of the imaging apparatus 100 or an external apparatus (both not shown) that are communicably connected via a communication interface. The subject designation unit 109 is an input interface including, for example, a touch panel and buttons. A user (imager) can designate an arbitrary subject included in the image as a tracking target via the subject designation unit 109.

  The subject tracking unit 110 tracks subjects included in images (image signals) that are sequentially supplied from the image processing unit 106 in time series (that is, taken at different times). The subject tracking unit 110 tracks a subject based on the similarity of histograms. It has means to alleviate the problem of tracking similar objects in error. Details of the subject tracking unit 110 will be described later.

  The control unit 105 controls image capturing conditions such as a focus state and an exposure state when the image sensor 102 captures an image. Specifically, the control unit 105 controls a focus control mechanism and an exposure control mechanism (neither shown) of the imaging lens 101 based on the video signal output from the A / D conversion unit 104. For example, the focus control mechanism is an actuator that drives the imaging lens 101 in the optical axis direction, and the exposure control mechanism is an actuator that drives an aperture or a shutter.

  The control unit 105 can use the subject area information supplied from the subject tracking unit 110 for the control of the focus control mechanism and the exposure control mechanism. Specifically, focus control using the contrast value of the subject area and exposure control using the brightness value of the subject area are performed. Therefore, the imaging apparatus 100 can perform imaging processing in consideration of a specific subject area in the captured image. The control unit 105 also performs readout control of the image sensor 102 such as output timing and output pixels of the image sensor 102.

  Details of the subject tracking unit 110 will be described. FIG. 2 is a block diagram showing the configuration of the subject tracking 110. The histogram registration unit 201 estimates a partial region indicating the target subject from the image data supplied from the image processing unit 106 based on the position information designated by the subject designation unit 109, and a histogram of pixel values of the partial region. Register. The registered histogram is a subject model that expresses the tracking target. Further, the configuration is not limited to the designation by the subject designation unit 109, and a specific subject detection unit (not shown) such as face detection may be provided, and a histogram may be registered based on the detected subject area.

  The histogram matching unit 202 collates the histogram of the partial area obtained from the image data supplied from the image processing unit 106 with the histogram registered by the histogram registration unit 201. The partial area to be collated is changed, and a plurality of similarities with high similarity by collation and position information of the partial area are output. Details of the histogram matching will be described with reference to FIG. FIG. 3A shows an example of a subject model in histogram matching. Reference numeral 301 denotes an image showing a target region to be tracked, and a histogram generated from pixel data of this image is handled as a feature amount. 302 represents the feature quantity 301. The feature amount is expressed by the following expression when the luminance signal is an M-gradation histogram.

  FIG. 3B shows image information for searching for a tracking target. Reference numeral 303 denotes an image in a range where matching processing is performed. The coordinates in the search image are expressed by (x, y). Reference numeral 304 denotes a partial area for acquiring a matching evaluation value. A feature amount generated from the partial region 304 is indicated by 305. The feature amount is expressed by the following expression when the luminance signal is an M-gradation histogram of the luminance signal as in the subject model.

  A Bhattacharyya coefficient is used as a calculation method for evaluating the histogram similarity between the subject model and the partial region. The Bhattacharyya coefficient is calculated by the following formula.

  The Bhattacharyya coefficient D (x, y) is calculated while shifting the partial area 304 one pixel at a time from the upper left of the search range 303. The coordinate (x, y) where the calculated D (x, y) has the maximum value indicates the most similar position. That is, the position showing the maximum value is a position where there is a high possibility that the target tracking target exists in the search image.

  In the above description, an example in which one-dimensional information of a luminance signal is used as a feature amount has been described, but three-dimensional information such as a lightness / hue / saturation signal may be treated as a feature amount. Further, although the Bhattacharyya coefficient has been described as a method of calculating the matching evaluation value, a different calculation method such as a histogram intersection may be used.

  Based on the maximum value of the similarity output from the histogram matching unit 202, the tracking state determination unit 203 performs tracking state determination. The tracking state and state transition in the tracking state determination unit 203 will be described with reference to FIG. 4, 401, 402, and 403 indicate the tracking state, 401 is a state in which the tracking target is found, 402 is a state in which the tracking target is temporarily lost due to occlusion or frame out of the tracking target, and 403 ends the tracking Indicates the state to be performed. When tracking starts, the state is 401. If the maximum value of similarity by the histogram matching unit 202 is lower than a predetermined threshold, the state transitions to state 402.

  In the state 402, if the maximum value of the similarity by the histogram matching unit 202 is higher than a predetermined threshold value, the state transitions to the state 401. In the state 402, if the elapsed time 402 is equal to or longer than a predetermined time, the state transitions to the state 403 and the tracking is finished. A partial area indicating the maximum value of similarity output from the histogram matching unit 202 together with the tracking state is supplied to the control unit 105 and the image processing unit 106. The control unit 105 and the image processing unit 106 perform control and processing using information on the subject area if the tracking state is a state where 401 is found.

  Information output from the histogram matching unit 202 is also supplied to the similar target region registration unit 204. The similar target area registration unit 204 acquires not only the maximum similarity value but also a plurality of similarities and position information of partial areas. In the histogram matching, since the evaluation scale is the similarity of the histogram as shown in Equation 3, the feature is more ambiguous than the similarity of the pixel pattern like the template matching, and it is robust to the change in the appearance of the subject. , Prone to errors with similar objects.

  Therefore, in the similar target area registration unit 204, when there is a partial area indicating the second or higher similarity that is equal to or greater than the predetermined threshold at a position away from the position of the partial area indicating the first similarity, this area Are registered as similar target areas. The region registered as a similar target is excluded from the output information of the histogram matching unit 202 from the next frame onward. Thereby, it is possible to reduce the probability of occurrence of a problem of tracking an erroneous area.

  Registration of the similar target area will be described with reference to FIG. The horizontal axis in FIG. 5 indicates the position of the partial region in the similarity calculation, and the vertical axis indicates the calculated similarity. The waveform 501 is the calculation result of the histogram matching unit 202. In subject tracking, since a two-dimensional image is a processing target, the position is expressed in two dimensions, but in order to simplify the explanation, FIG. 5 shows the position information as one dimension. Reference numeral 502 denotes the maximum value of similarity. Reference numeral 503 denotes a threshold value for determining that the similarity is high. 503 may be a fixed value or a value determined by a relative value of the maximum value 502.

  A range indicated by reference numeral 504 having a high degree of similarity is registered as a similar target region at a position farther from the maximum value. In the range of 505, since the similarity is high but close to the position showing the maximum value, it is not registered as a similar target area.

  The information registered by the similar target area registration unit 204 is retained for a predetermined period. In other words, it is deleted after a predetermined period. However, this does not apply when the conditions described later are met. In the tracking state determination unit 203, when the tracking target is temporarily lost, the registration information of the similar target region registration unit 204 is not updated. That is, no additional registration or registration information is deleted. This prevents registration of an incorrect region (region indicating a tracking target) as a similar target region, and estimation of an erroneous region (region different from the tracking target) due to deletion of registration information. Because.

  If the position of the tracking target in subject tracking is close to the position of the area registered as the similar target area, the registration information is deleted. This is because the possibility that the area determined to be a similar target is a tracking target has increased. In addition, a motion determination unit (not shown) for determining whether or not the entire image has moved is included. When the motion determination unit determines that the entire image has moved, information registered as a similar target region is deleted. To do.

  This is because when the entire image moves, the reliability of the similar target area that is held decreases. As the motion determination, there are various methods such as a method based on a difference between frames of the entire image, a method based on a gyro signal of an imaging device, and a combination thereof.

  The processing flow of the subject tracking unit 110 will be described with reference to the flowchart of FIG. First, an image is supplied from the image processing unit 106 as an input image of the subject tracking unit 110 (S601). A tracking target area is determined based on the input image and information of the image subject specifying unit 109 (S602). Based on the region, the histogram registration unit 201 registers a histogram which is a subject model for histogram matching (S603).

  Next, an image is supplied from the image processing unit 106 as an input image of the subject tracking unit 110 (S604). The input image of S604 indicates a search image of the subject tracking unit 110, and the images of S601 and S604 are images having different times. Based on the image of S604, the histogram matching unit 202 calculates the similarity (S605). In the calculated similarity, a position belonging to an area registered in the similarity target area registration unit 204 is excluded from the output information of the histogram matching unit 202 (S606). That is, the position registered in the similar target area registration unit 204 is not the position where the similarity is the maximum value.

  Next, the tracking state determination unit 203 determines the tracking state from the current tracking state and the maximum value of the similarity that is the output information of the strut matching unit 202 (S607). The tracking state and state transition conditions are the same as those described with reference to FIG. If the tracking target is found in the tracking state (the state 401 in FIG. 4) (YES in S608), the similar target region registration unit 204 determines registration of the similar target region (S609). If there is a partial area that shows a second or higher similarity that is equal to or greater than a predetermined threshold at a position that is distant from the position of the partial area that shows the first similarity, this area is registered as a similarity target area.

  Next, a policy determination based on the elapsed time of the registered similar target area is performed (S610). At the time of registration, the current time is stored together with the area information, and if the elapsed time exceeds a predetermined time, the registered area is deleted. Further, deletion determination based on the tracking position of the similar target area is performed (S611). When the position of the tracking target in the subject tracking is close to the position of the area registered as the similar target area, the registration information is deleted. The position of the tracking target in the subject tracking indicates a position where the similarity output by the histogram matching unit 202 is maximized. In addition, deletion determination based on movement determination of the entire image of the similar target region is performed (S612).

  If a motion equal to or greater than a predetermined threshold is detected, all the registration information of the similar target area is deleted. Next, the process returns to S604, and the iterative process is executed based on the sequentially supplied images. If the tracking state is a state in which the tracking target is temporarily lost (state 402 in FIG. 4) (NO in S608 and YES in S613), it is based on the motion determination of the entire image of the similar target region. Deletion determination is performed (S612). In other words, if the tracking target is found (the state 401 in FIG. 4), the similar target region update process from S609 to S612 is performed, and the tracking target is temporarily lost (see 402 in FIG. 4). State), only the update processing of the similar target area in S612 is performed.

  When the similar target region update process is completed, the process returns to S604, and the iterative process is executed based on the sequentially supplied images. If the tracking state is a state in which the tracking process is finished (state of 403 in FIG. 4) (NO in S608, NO in S613), the tracking process is terminated. When the tracking process is finished, the registration information of the similar target area is deleted.

  As described above, the frequency of erroneous tracking when there is a similar target can be reduced, and the performance of subject tracking can be improved.

[Other Embodiments]
In the above-described embodiment, the case where the subject tracking device is applied to the imaging device is illustrated, but the device to which the subject tracking device is applied is not limited to the imaging device. For example, a subject tracking device that performs subject tracking may be applied to a display device that displays an image (reproduction data) supplied from an external device or a recording medium. In this display device, the reproduction data is used as subject tracking processing data, and subject tracking processing is performed. A control unit such as a microcontroller in the display device controls display conditions for displaying an image based on subject information (position, size, etc. of the subject in the image) extracted by subject tracking processing.

  Specifically, information indicating a subject such as a frame is superimposed on the position of the subject in the image, and the brightness and color of the display image are controlled according to the brightness and color information of the subject portion. Moreover, the description in embodiment mentioned above shows an example, and is not limited to this. The configuration and operation in the embodiment described above can be changed as appropriate.

  The above-described embodiment can also be realized in software by a computer of a system or apparatus (or CPU, MPU, etc.). Therefore, the computer program itself supplied to the computer in order to implement the above-described embodiment by the computer also realizes the present invention. That is, the computer program itself for realizing the functions of the above-described embodiments is also one aspect of the present invention.

  The computer program for realizing the above-described embodiment may be in any form as long as it can be read by a computer. For example, it can be composed of object code, a program executed by an interpreter, script data supplied to the OS, but is not limited thereto. A computer program for realizing the above-described embodiment is supplied to a computer via a storage medium or wired / wireless communication. Examples of the storage medium for supplying the program include a magnetic storage medium, an optical / magneto-optical storage medium, and a nonvolatile semiconductor memory.

  As a computer program supply method using wired / wireless communication, there is a method of using a server on a computer network. In this case, a data file (program file) that can be a computer program forming the present invention is stored in the server. The program file may be an executable format or a source code. Then, the program file is supplied by downloading to a client computer that has accessed the server. In this case, the program file can be divided into a plurality of segment files, and the segment files can be distributed and arranged on different servers.

  That is, a server apparatus that provides a client computer with a program file for realizing the above-described embodiment is also one aspect of the present invention. In addition, a storage medium in which the computer program for realizing the above-described embodiment is encrypted and distributed is distributed, and key information for decrypting is supplied to a user who satisfies a predetermined condition, and the user's computer Installation may be allowed. The key information can be supplied by being downloaded from a homepage via the Internet, for example. Further, the computer program for realizing the above-described embodiment may use an OS function already running on the computer.

  Further, a part of the computer program for realizing the above-described embodiment may be configured by firmware such as an expansion board attached to the computer, or may be executed by a CPU provided in the expansion board. Good.

100 imaging device, 101 imaging lens, 102 imaging device,
103 analog signal processing unit, 104 A / D conversion unit, 105 control unit,
106 image processing unit, 107 display unit, 108 storage medium, 109 subject designation unit,
110 subject tracking unit, 201 histogram registration unit,
202 Histogram matching unit, 203 tracking state determination unit,
204 Similar target area registration section

Claims (9)

A subject tracking device that tracks a target subject from sequentially supplied images,
A histogram registration means for registering a histogram of pixel values of a partial area indicating a target subject from the image;
The histogram registered by the histogram registration means is used to collate with a histogram of pixel values of the partial area of the supplied image, change the partial area to be collated, and a plurality of similarities and parts with high similarity by collation Histogram matching means for outputting position information of the area;
When there is a partial region indicating the second and subsequent similarities equal to or higher than a predetermined threshold at a position apart from the position of the partial region indicating the first similarities by the histogram matching means, this region is set as the similarity target region. It has similar target area registration means to register,
The histogram matching means outputs a plurality of upper similarities and partial area position information to be output.
An object tracking apparatus, wherein an area registered by the similar target area registration means is not included.   The subject tracking device according to claim 1, wherein the information registered by the similar target region registration unit is held for a predetermined period.   In tracking the subject, it further comprises a tracking state determining means for determining whether the tracking target is found and a state in which the tracking target is temporarily lost, and when the tracking target is temporarily lost in the tracking state determining means, The subject tracking device according to claim 2, wherein the information of the similar target area registration unit is not updated.   4. The subject according to claim 2, wherein the registration information is deleted when the location of the subject to be tracked in subject tracking is close to the location of the region registered by the similar target region registering unit. 5. Tracking device.   In the sequentially supplied images, the image processing apparatus further includes a motion determination unit that determines whether or not the entire image has moved. When the motion determination unit determines that the entire image has moved, registration is performed by the similarity target region registration unit. 5. The subject tracking apparatus according to claim 2, wherein the recorded information is deleted. 6. Imaging means for sequentially supplying captured images;
The subject tracking device according to any one of claims 1 to 5,
Control means for controlling imaging conditions when the imaging means captures images in accordance with subject information output by the subject tracking device;
An imaging apparatus comprising: Display means for displaying sequentially supplied images;
The subject tracking device according to any one of claims 1 to 5,
Control means for controlling display conditions displayed by the display means in accordance with subject information output by the subject tracking device;
A display device comprising: A method of controlling a subject tracking device that tracks a target subject from sequentially supplied images,
A histogram registration step of registering a histogram of pixel values of a partial region indicating a target subject from the image;
Using the histogram registered in the histogram registration step, the pixel value histogram of the partial area of the supplied image is collated, the partial area to be collated is changed, and a plurality of similarities and parts having high similarity by collation are high A histogram matching step for outputting region position information;
When there is a partial area indicating the second and subsequent similarities equal to or higher than a predetermined threshold at a position away from the position of the partial area indicating the first similarities in the histogram matching step, this area is set as a similar target area. It has a similar target area registration process to register,
In the histogram matching step, the plurality of similarities to be output and the position information of the partial areas are
A method for controlling a subject tracking apparatus, wherein the region registered in the similar target region registration step is not included. The program for making a computer perform each process of the control method of the to-be-photographed object tracking apparatus of Claim 8.