JP5800559B2 - Subject tracking device, imaging device, subject tracking method and program - Google Patents

Description

  The present invention relates to a technique for tracking a subject in a tracking target area in image data.

  Japanese Patent Application Laid-Open No. 2004-133867 discloses a technique for tracking an arbitrary subject in image data by extracting a feature color of the subject in the image data and detecting a region having the feature color from different image data. It is disclosed. In the technique disclosed in Patent Document 1, the feature colors of the image data in the tracking target area and the entire image data are extracted. Then, tracking is performed using binary image data in which the characteristic color region that is unique to the image data in the tracking target region and can be stably extracted is 1 and the other region is 0, thereby improving the tracking accuracy. It is improving. Similarly, in the case of luminance, characteristic luminance between the image data in the tracking target area and the entire image data is extracted. Then, tracking is performed by using binary image data in which a characteristic luminance area that is unique to image data in the tracking target area and can be stably extracted is 1 and other areas are 0. The accuracy is improved.

Japanese Patent Laid-Open No. 9-181953

  In the technique disclosed in Patent Document 1, only one feature color used for tracking is extracted, and tracking is performed by binarizing a search area in image data based on the feature color. Therefore, in a scene in which the main subject is composed of a plurality of colors, there are few portions that can be extracted as characteristic colors, and therefore, the main subject is easily affected by noise and disturbance. In addition, since matching is performed using binarized data, there is a problem that accuracy is rough and tracking accuracy cannot be ensured in a scene where a change in gradation and tone is small. Even in the case of using luminance instead of color, the same problem occurs because binarization is performed using characteristic luminance.

  Accordingly, an object of the present invention is to accurately track a subject.

The subject tracking device according to the present invention classifies hues in a predetermined classification unit based on color information obtained from a tracking target area in the first image data, and the center of the tracking target area in the first image data. A histogram classified by the classification unit is generated based on the distance from the pixel to each pixel and the number of pixels corresponding to each of the classification units, and a plurality of characteristic colors and a plurality of the plurality of the characteristic colors are generated based on the histogram. third image data having a determining means for determining a respective characteristic of the characteristic color, from the tracking target area in said first image data, the gradation corresponding to each feature of the plurality of the characteristic color First image data is generated from second image data in a frame different from the first image data to generate fourth image data having gradations corresponding to the respective characteristic degrees of the plurality of characteristic colors. Generating means, and A first correlation amount that is a correlation amount with the third image data is obtained for each region of the image data, and a tracking target region in the second image data is set based on the first correlation amount. And setting means.

  According to the present invention, it is possible to accurately track a subject.

It is a figure which shows the main structures of the imaging device which concerns on embodiment of this invention. It is a flowchart which shows the tracking process of the target object in a subject tracking part. It is a figure which shows hue space and a histogram. It is a figure which shows concretely the process of step S202 of FIG. 2, and S205. It is a figure which shows the process which produces | generates 1st brightness | luminance image data from template image data, and the process which produces | generates 1st brightness | luminance image data from search area image data. It is a figure for demonstrating the search process in step S207 of FIG. It is a figure which shows the process which produces | generates 2nd luminance image data from template image data, and the process which produces | generates 2nd luminance image data from search area image data. It is a flowchart which shows the detail of the characteristic color determination process in step S203 of FIG. It is a figure which shows the correspondence of the partial hue space of Fig.3 (a), and the UV signal of a pixel (x, y). It is a figure shown about the correction value applied to the pixel which belongs to each partial hue space in creation of the histogram shown in Drawing 3 (b). It is a figure for demonstrating the process in step S814 of FIG. It is a flowchart which shows the determination process of the position of the tracking target in step S210 of FIG.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments to which the invention is applied will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram illustrating a main configuration of an imaging apparatus according to an embodiment of the present invention. As illustrated in FIG. 1, the imaging device 112 includes a subject tracking unit 100, an imaging element 108, an imaging optical system operation unit 109, an operation unit 110, and a display unit 111. The subject tracking unit 100 includes an image acquisition unit 101, an image storage unit 102, a system control unit 103, an image conversion processing unit 104, a correlation calculation unit 105, and a tracking information storage unit 106. Note that the subject tracking unit 100 is an application example of the subject tracking device.

  The image acquisition unit 101 acquires an image signal from the image sensor 108, performs A / D conversion, converts the image signal into YUV format image data, and outputs the image data to the image storage unit 102. Here, the YUV format image data used in this embodiment has a Y signal (having a value of 0 to 255) that is a luminance signal and a UV signal (having a value of -128 to 127) that is a color difference signal. . The image storage unit 102 stores the image data output from the image acquisition unit 101. The system control unit 103 performs overall control of the imaging device 112. The image conversion processing unit 104 reads out image data from the image storage unit 102 and performs image processing to generate two pieces of luminance image data. The correlation calculation unit 105 uses one of the two luminance image data generated by the image conversion processing unit 104 as template image data, and is the most similar to the one of the luminance image data in the other luminance image data. A region having a high correlation amount is searched. The tracking information storage unit 106 stores tracking processing data such as a subject position and a correlation amount. The image acquisition unit 101, the system control unit 103, the image conversion processing unit 104, and the correlation calculation unit 105 have a functional configuration that is realized when the microcomputer executes a computer program stored in the ROM. The image storage unit 102 and the tracking information storage unit 106 are configured to correspond to a partial storage area of the RAM.

  The image sensor 108 is a photoelectric conversion element that converts an optical signal based on an image formed on the light receiving surface into an electric signal for each light receiving pixel at a corresponding position. The imaging optical system operation unit 109 changes the aperture diameter of the diaphragm or performs movement control of the focus lens. The operation unit 110 has a release button and accepts an operation from the user. The display unit 111 displays the setting status of the imaging device 112 or thumbnail image data generated after shooting. Note that the imaging apparatus 112 according to the present embodiment can capture both still image data and moving image data.

  Hereinafter, an example of processing for tracking an arbitrary object in the subject tracking unit 100 described above will be described. FIG. 2 is a flowchart showing the object tracking processing in the subject tracking unit 100. 2 is started when the release button on the operation unit 110 is half-pressed.

  When the tracking processing of the object is started, in step S201, the image acquisition unit 101 acquires an image signal from the image sensor 108, performs A / D conversion, and then converts the image data into YUV format image data. The data is output to the storage unit 102 and stored.

  In step S <b> 202, the system control unit 103 reads m-th (m = 1, 2,...) Image data from the image storage unit 102. Then, the system control unit 103 extracts a portion corresponding to the tracking target area from the m-th image data (first image data), and generates template image data. Here, the tracking start position is the center of the image data, and the size of the tracking target area is a numerical value (LTx, LTy) set in the tracking information storage unit 106. When the extraction of the image data corresponding to the tracking target area is completed, the system control unit 103 outputs the image data as template image data to the tracking information storage unit 106 and stores it.

  In step S203, the system control unit 103 determines a feature color in the template image data. For the determination of the characteristic color, the hue space and the histogram shown in FIG. 3 are used. A hue space 301 shown in FIG. 3A is a space constituted by a hue circle, and a partial hue space 302 is a space obtained by dividing the hue space 301 into 12 equal parts. Each of the partial hue spaces 302 has regions of hues of Red, Red-Yellow, Yellow, Yellow-Green, Green, Green-Cyan, Cyan, Cyan-Blue, Blue, Blue-Magenta, Magenta, and Magenta-Red. Is a candidate for a feature color. In the present embodiment, the partial hue space 302 is obtained by dividing the hue space 301 into 12 equal parts, but the size of each partial hue space 302 may not be equal, and the number of the partial hue spaces 302 is also the template image. The number may be changed according to the color distribution in the data.

  The histogram 303 shown in FIG. 3B classifies each pixel of the template image data into the partial hue space 302 that is a classification unit based on the color difference component (color information), and the horizontal axis indicates each partial hue space 302 and the vertical axis. The axis is created by multiplying the number of pixels classified into each partial hue space 302 by a correction value described later. Here, the numerical value on the vertical axis of the histogram 303 is the characteristic level of the color corresponding to the partial hue space 302. However, if the color difference component of the pixel is less than or equal to the threshold value, the pixel is classified into an achromatic color space (MT: Monotone) and handled as a pixel that does not belong to the partial hue space 302. The histogram generation process will be described in detail with reference to FIG. The process of classifying each pixel into the partial hue space 302 based on the color difference component is a process example of the hue determination unit.

  The system control unit 103 outputs the characteristic color and the histogram to the tracking information storage unit 106 and stores them. Here, the feature colors are the top three colors in the histogram, and the first to third feature colors are determined in descending order of the feature degree. As for the top color of the histogram, the fixed color as described above is used, all hues having a characteristic value equal to or higher than a predetermined value are set as characteristic colors, or the total value of the characteristic values is set to a predetermined value. The feature color may be selected until it is reached. In this way, the condition for determining the characteristic color is arbitrary.

  In step S204, the system control unit 103 waits for the (m + 1) th image data (second image data) to be stored in the image storage unit 102, and when stored, reads the (m + 1) th image data. . In step S205, the system control unit 103 generates search region image data by extracting a portion corresponding to the search region from the (m + 1) th image data. When the search area image data is generated, the system control unit 103 outputs the search area image data to the tracking information storage unit 106 and stores it.

  FIG. 4 is a diagram specifically showing the processing of steps S202 and S205 in FIG. In FIG. 4, 401 is m-th image data, and 403 is m + 1-th image data. In step S202, template image data 405 is generated by extracting a portion corresponding to the tracking target area 402 from the m-th image data 401. In step S205, search area image data 406 is generated by extracting a portion corresponding to the search area 404 from the (m + 1) th image data. The m-th image data 401 and the (m + 1) -th image data 403 are both color image data. The notations of RY, Y, R, G, C, and MT in FIG. 4 represent the red and yellow intermediate colors (Red-Yellow), Yellow, Red, Green, Cyan, and Monotone, respectively, and the corresponding pattern portions are Indicates that the color.

  In the example of FIG. 4, the vicinity of the torso of the person located near the center of the m-th image data 401 is set as the tracking target area 402. The method for specifying the tracking target area 402 may be that the user specifies an arbitrary position with an operating device such as a touch panel, or may be a position in the image data that has been focused immediately before the start of the subject tracking process. Note that the position of the search area 404 is set to be concentric with the tracking target area 402 in the m-th image data 401, and the size of the search area 404 is a numerical value (LSx) set in the tracking information storage unit 106. , LSy).

  In step S206, the system control unit 103 reads the template image data 405 generated in step S202 or step S212 described later and the search area image data 406 generated in step S205 from the tracking information storage unit 106. Then, the system control unit 103 outputs the template image data 405 and the search area image data 406 to the image conversion processing unit 104. Step S206 is a processing example of the first generation unit.

  The image conversion processing unit 104 extracts a luminance signal (Y signal) of each pixel from each of the template image data 405 and the search area image data 406. Then, the image conversion processing unit 104 generates first luminance image data including only the extracted luminance signal for each of the template image data 405 and the search area image data 406.

  FIG. 5 is a diagram showing a process for generating the first luminance image data 501 from the template image data 405 and a process for generating the first luminance image data 502 from the search area image data 406. Note that the template image data 405 and the search area image data 406 are YUV format image data. Therefore, the first luminance image data 501 and 502 can be generated by cutting the color difference signal of the original color image data (template image data 405, search area image data 406). After generating the first luminance image data 501 and 502, the image conversion processing unit 104 outputs the first luminance image data 501 and 502 to the correlation calculation unit 105.

  In step S207, the correlation calculation unit 105 searches the region having the highest correlation with the template image data 405 in the search region image data 406 by matching the first luminance image data 501 and 502 with the luminance value. To do. Step S207 is a processing example of the first calculation means.

FIG. 6 is a diagram for explaining the search process in step S207 of FIG. In FIG. 6, the first luminance image data 602 of the search area image data 406 is set to the origin (0, 0) at the upper left, and the x-axis value is incremented by 1 as it proceeds to the right by one pixel, and as it proceeds downward by one pixel. The two-dimensional coordinates (x, y) for adding 1 to the y-axis value are considered. The correlation calculation unit 105 performs matching on the first luminance image data 602 while shifting the first luminance image data 601 pixel by pixel from (0, 0), and calculates the correlation amount at each position. The correlation amount is calculated from the SAD value that represents the sum of the luminance differences of pixels at the same position between regions. The correlation amount when the first luminance image data 601 is at an arbitrary position (i, j) in the first luminance image data 602 is calculated by the following equation 1.
Correlation amount = {1- (SAD value of (i, j) / maximum SAD value)} × 100 (1)
Here, the maximum SAD value is the maximum value that the SAD value can take. Therefore, the amount of correlation takes a numerical value of 0 to 100 according to Equation 1.

  When matching is completed at all positions, the correlation calculation unit 105 sets the position (i, j) having the highest correlation as the tracking target position in the next image data, and the correlation amount of the tracking target position (i, j). At the same time, it is output to the system control unit 103. The system control unit 103 sends the tracking target position (i, j) and the correlation amount of the tracking target position (i, j) to the tracking information storage unit 106 as a tracking result based on the first luminance image data. Output and memorize.

In step S208, the system control unit 103 reads out the feature color and the histogram from the tracking information storage unit 106, and based on the feature degree of the feature color indicated by the histogram, the second luminance image data corresponding to the feature color. Calculate the luminance value. Thereafter, the system control unit 103 outputs the calculation result to the image conversion processing unit 104. Here, the luminance value of the second luminance image data corresponding to the kth characteristic color (2 ≦ k ≦ n) is obtained by the following equation 2.
Luminance value of the second luminance image data corresponding to the kth feature color = (feature of the kth feature color / feature of the first feature color) × maximum luminance value.
However, the system control unit 103 sets a luminance value of 0 for hues that are not employed as characteristic colors. Here, the maximum luminance value shown in Expression 2 is the maximum luminance value that can be taken by the YUV format image data handled in the present embodiment, and the value is Y = 255.

  Further, the system control unit 103 reads the template image data 405 generated in step S202 or S212 and the search area image data 406 generated in step S205 from the tracking information storage unit 106. Then, the system control unit 103 outputs the template image data 405 and the search area image data 406 to the image conversion processing unit 104. The image conversion processing unit 104 extracts a color difference signal (UV signal) of each pixel from each of the template image data 405 and the search area image data 406.

  Then, based on the extracted color difference signal, the image conversion processing unit 104 sets the luminance value obtained by Expression 2 for the area having the color difference signal that matches the characteristic color, and the template image data 405 and the search area image data 406 are set. Second luminance image data is generated from each of the above. Thereafter, the image conversion processing unit 104 outputs the second luminance image data generated from the template image data 405 and the second luminance image data generated from the search area image data 406 to the correlation calculation unit 105.

  FIG. 7 is a diagram showing a process for generating the second luminance image data 701 from the template image data 405 and a process for generating the second luminance image data 702 from the search area image data 406. In FIG. 7, the second luminance image data 701 is generated by setting an area having a color difference signal that matches the characteristic color in the template image data 405 to a high luminance and setting other areas having the color difference signal to a luminance value of 0. It is shown that. Similarly, for the second luminance image data 702, an area having a color difference signal matching the characteristic color in the search area image data 406 is set to high luminance, and an area having other color difference signals is set to a luminance value of 0. Is generated.

  In step S209, the correlation calculation unit 105 performs a correlation calculation based on matching on the second luminance image data 701 and 702 in the same manner as in step S207. Then, the correlation calculation unit 105 outputs the position (i ′, j ′) with the highest correlation and the correlation amount at the position (i ′, j ′) to the system control unit 103. The system control unit 103 determines the position (i ′, j ′) having the highest correlation and the correlation amount between the position (i ′, j ′) and the position (i ′, j ′) of the tracking target and the tracking target. The correlation information of the position (i ′, j ′) is output to the tracking information storage unit 106 and stored. The correlation amount between the tracking target position (i ′, j ′) and the tracking target position (i ′, j ′) is a tracking result based on the second luminance image data. Step S208 is a processing example of the second generation unit. Step S209 is a processing example of the second calculation means.

  In step S210, the system control unit 103 reads from the tracking information storage unit 106 the tracking result based on the first luminance image data in step S207 and the tracking result based on the second luminance image data in step S209. Then, the system control unit 103 selects any tracking result from the correlation amount in the tracking result based on the first luminance image data and the correlation amount in the tracking result based on the second luminance image data. Next, the system control unit 103 adopts the position (tx, ty) of the tracking target in the tracking result and stores it in the tracking information storage unit 106. Then, the system control unit 103 outputs the adopted tracking target position (tx, ty) to the imaging optical system operation unit 109.

  However, when both the correlation amount in the tracking result based on the first luminance image data and the correlation amount in the tracking result of the second luminance image data are smaller than the predetermined value, the system control unit 103 has lost track of the tracking target. Judgment is made and a tracking target loss signal is output. The details of step S210 will be described with reference to the flowchart shown in FIG.

  In step S211, the imaging optical system operation unit 109 changes the aperture diameter of the diaphragm or controls the movement of the focus lens based on the input tracking target position (tx, ty). However, the imaging optical system operation unit 109 does not change the aperture diameter of the aperture and move the focus lens when receiving the tracking target loss signal. In step S212, the system control unit 103 reads the correlation amount at the tracking target position (tx, ty) from the tracking information storage unit 106, and determines whether the correlation amount is higher than a preset threshold value. When the correlation amount is higher than the threshold value, the system control unit 103 reads the search region image data 406 and the set value (LTx, LTy) of the tracking target region 402 from the tracking information storage unit 106. Then, the system control unit 103 extracts an area having a size (LTx, LTy) corresponding to the tracking target position (tx, ty) in the search area image data 406, and the next image data (m + 1th image data). ) Is output to the tracking information storage unit 106 as template image data 405 for the tracking process in FIG. On the other hand, when the correlation amount is equal to or smaller than a preset threshold value, the system control unit 103 extracts an area corresponding to the m-th template image data 405 from the search area image data 406, and the next image data (m + 1). Output to the tracking information storage unit 106 as template image data 405 in the first image data). Step S211 is a processing example of the first extraction unit, the second extraction unit, and the selection unit.

  In step S213, the system control unit 103 determines whether or not the release button has been pressed. When the release button is not pressed, the system control unit 103 ends the subject tracking process. On the other hand, when the release button is pressed halfway or fully, the system control unit 103 continues the subject tracking process. Thereby, the process returns to step S203. Here, the position of the tracking target area 402 in the next image data (m + 1th image data) is the position (tx, ty) of the tracking target in the previous image data (mth image data).

  Next, the characteristic color determination process in step S203 of FIG. 2 will be described in detail with reference to the flowchart shown in FIG. In step S <b> 801, the system control unit 103 generates a partial hue space 302 by dividing a hue space 301 including a hue circle into 12 as illustrated in FIG. 3A. In step S802, the system control unit 103 repeats the processes in steps S803 to S806 as many times as the number of vertical pixels of the template image data. At this time, the system control unit 103 defines y as a count value of the number of repetitions, and repeats the processing of steps S803 to S806 as long as the value of y is less than the number of vertical pixels of the template image data. In step S803, the system control unit 103 repeats the processes in steps S804 to S806 as many times as the number of horizontal pixels of the template image data. At this time, the system control unit 103 defines x as a count value of the number of repetitions, and repeats the processes of steps S804 to S806 as long as the value of x is less than the number of horizontal pixels of the template image data.

  In step S804, the system control unit 103 calculates a distance Dist (x, y) from the pixel (x, y) to the center point of the template image data. In step S805, the system control unit 103 sets the importance to the pixel (x, y) based on Dist (x, y). At this time, the system control unit 103 sets so that the importance becomes larger as the value of Dist (x, y) is smaller. In step S806, the system control unit 103 determines the partial hue space 302 to which the pixel (x, y) belongs based on the UV signal of the pixel (x, y).

  FIG. 9 is a diagram illustrating a correspondence relationship between the partial hue space 302 in FIG. 3A and the UV signal of the pixel (x, y). That is, in FIG. 9, a two-dimensional coordinate area in which the vertical axis represents the value of V and the horizontal axis represents the value of U corresponds to the hue space 301, and represents which partial hue space 302 belongs to the UV signal. . Here, the characters in each area in FIG. 9 indicate the color of the area. R is Red, RY is Red-Yellow, Y is Yellow, YG is Yellow-Green, G is Green, and GC is Green. -Cyan, C is Cyan, CB is Cyan-Blue, B is Blue, BM is Blue-Magenta, M is Magenta, and MR is Magenta-Red.

  In step S807, the system control unit 103 calculates the sum of the importance of each pixel classified in each partial hue space 302 in which each pixel is classified in step S806, and calculates the total value as a feature of the partial hue space 302. Degree. FIG. 10 is a diagram illustrating correction values to be applied to pixels belonging to each partial hue space 302 when the histogram 303 illustrated in FIG. 3B is created. Here, this correction value is referred to as importance, and the importance is set to a high numerical value for a pixel near the center of the template image data 405 and a low numerical value for a far pixel. As a result, the characteristic feature of the hue that is closer to the center of the template image data 405 and exists more in the template image data 405 is increased. The advantage of setting the importance of the pixel is that, when shooting with a camera, in many cases, the tracking target is caught at the center of the tracking target area, so the color feature of the tracking target at the center of the tracking target area is high. The tracking target color can be easily extracted.

  In step S <b> 808, the system control unit 103 creates a histogram (303 in FIG. 3B) based on the feature level of each partial hue space 302 and outputs the histogram to the tracking information storage unit 106. The histogram of the current image data (mth image data from the start of tracking) is defined as H (m). Step S808 is a processing example of the histogram generation unit.

  In step S809, the system control unit 103 repeats steps S810 to S814 for the number of shots (m) from the start of the tracking process in order to refer to all histograms in the past image data. At this time, the system control unit 103 defines i as a count value of the number of repetitions, and repeatedly executes steps S810 to S814 as long as the value of mi is 0 or more.

  In step S <b> 810, the system control unit 103 reads from the tracking information storage unit 106 the histogram H (mi) generated before the i-th image from the current image data (the m-th image data). In step S811, the system control unit 103 applies correction based on the numerical value of i to the feature degree of each partial hue space 302 of the histogram H (m−i). This correction is such that the larger i is, the smaller the characteristic degree is. In step S812, the system control unit 103 reads the correlation amount in the i-th previous tracking result from the tracking information storage unit 106. The correlation amount read here is COR (m−i).

  In step S813, the system control unit 103 applies a correction value based on the correlation amount COR (m−i) to the feature degree of each partial hue space 302 of the histogram H (m−i). This correction is such that the higher the correlation amount COR (m−i) is, the larger the characteristic degree is. In step S814, the system control unit 103 adds the feature of each partial hue space 302 of the histogram H (m−i) to the feature of each partial hue space 302 of the histogram H (m) of the current image data. Note that the feature degree addition processing of each partial hue space 302 of the histogram is a processing example of the synthesis unit.

  FIG. 11 is a diagram for explaining the processing in step S814 of FIG. In the present embodiment, in order to increase the characteristic degree of the color to be tracked, the histogram of the current image data is added to the histogram generated from the past image data. The reason for adding histograms to increase the color feature of the tracking target is that when the continuous shooting is performed using an imaging device, the tracking target is used to track each image data against the background color, which tends to change from moment to moment. This is because there is a high possibility that they exist in common in the target area. In other words, when the histograms are added, the color of the tracking target with a relatively small change has a higher characteristic than the background color in which the characteristic varies by changing to various colors. The extraction accuracy can be expected to improve. In the example of FIG. 11, in the upper image data, (1) → (2) → (3) represent frame numbers, indicating that time has passed in the order of the frame numbers. A broken line frame in the image data represents an area of the template image data. The middle histogram is a histogram generated from each template image data. The lower histogram represents a histogram generated by adding the current image data and the histograms of all past image data in the image data of (1), (2), and (3). From FIG. 11, it can be seen that the histogram obtained by adding the past image data has a higher characteristic level of the tracking target color Red than the histogram generated with the current image data.

  The processing in steps S810 to S814 contributes to alleviating a decrease in tracking accuracy when, for example, the tracking target is small in some image data and the background range is large. When determining the feature color using only the histogram generated from the current image data, if the background becomes dominant in the template image data, the background color feature will increase, and the background will be easily tracked. Become. As a countermeasure, a method of continuing to use the characteristic color of the first image data at the start of tracking can be considered. However, in this method, when the color of the tracking target changes, the tracking target is lost. Therefore, by using a histogram generated by a weighted average with past image data, the color feature that has been recognized as a tracking target in the past becomes relatively large. In addition, since the specific gravity of the histogram of relatively new image data becomes heavy, it is possible to flexibly cope with color changes.

  In step S815, the system control unit 103 determines the top three colors having a high feature level as the feature color among the partial hue spaces 302 of the histogram H (m) generated in steps S809 to S814. Step S815 is a processing example of the characteristic color determining unit.

  Next, the tracking target position determination process in step S210 of FIG. 2 will be described in detail with reference to FIG. In step S1216, the system control unit 103 calculates the tracking target position and the correlation amount calculated in step S207 (correlation calculation based on the first luminance image data) and step S209 (correlation calculation based on the second luminance image data). Read from the tracking information storage unit 106. Then, the system control unit 103 compares the correlation amount of the correlation calculation based on the first luminance image data with the threshold 1, and determines whether the correlation amount of the correlation calculation based on the first luminance image data is larger than the threshold 1. Determine. If the correlation amount of the correlation calculation using the first luminance image data is larger than the threshold 1, the process proceeds to step S1217. On the other hand, when the correlation amount of the correlation calculation using the first luminance image data is equal to or less than the threshold value 1, the process proceeds to step S1218.

  In step S1217, the system control unit 103 outputs a search result using the first luminance image data as a tracking target position. In step S <b> 1218, the system control unit 103 compares the correlation amount of the correlation calculation based on the second luminance image data with the threshold value 2, and whether the correlation amount of the correlation calculation based on the second luminance image data is larger than the threshold value 2. Determine whether or not. When the correlation amount of the correlation calculation using the second luminance image data is larger than the threshold value 2, the process proceeds to step S1219. On the other hand, when the correlation amount of the correlation calculation using the second luminance image data is equal to or smaller than the threshold value 2, the process proceeds to step S1220.

  In step S1219, the system control unit 103 outputs the search result using the second luminance image data as the tracking target position. In step S1220, the system control unit 103 determines that no tracking target has been found in the search using the first luminance image data and the second luminance image data, and outputs the tracking target loss as a tracking result.

  Steps S1216 to S1220 are selection processing for obtaining an appropriate tracking result depending on the situation. The advantages of using such a selection process are as follows. First, since the first luminance image data uses the luminance signals of the mth image data and the m + 1th image data as they are, the gradation is finer than the second luminance image data. . Therefore, tracking with relatively fine accuracy is possible, but tracking accuracy is reduced due to changes in luminance and noise.

  On the other hand, the second luminance image data sets a predetermined luminance value when the color difference signals of the m-th image data and the (m + 1) -th image data match the characteristic color. The gradation is rough compared to the image data, and all the regions belonging to the same hue have uniform luminance. Therefore, for example, when the tracking target is composed of a single color gradation, the tracking accuracy may be rough, but a decrease in tracking accuracy due to a change in luminance or noise can be suppressed more than the first luminance image data. it can.

  As described above, in this embodiment, the correlation calculation unit 105 for matching is used for both color and luminance, and luminance and luminance are reduced in a subject composed of a plurality of colors or in a scene with small changes in gradation and tone. It is possible to track the subject accurately by properly using colors. That is, in the present embodiment, the tracking accuracy of the subject is improved by properly using tracking using luminance to emphasize accuracy and tracking using color to emphasize robustness against noise and disturbance. Yes.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

  100: Subject tracking unit, 101: Image acquisition unit, 102: Image storage unit, 103: System control unit, 104: Image conversion processing unit, 105: Correlation calculation unit, 106: Tracking information storage unit, 108: Image sensor, 109 : Imaging optical system, 110: operation unit, 111: display unit, 112: imaging device

Claims (13)

Based on the color information obtained from the tracking target area in the first image data , the hue is classified in a predetermined classification unit, the distance from the center of the tracking target area in the first image data to each pixel, Generating a histogram classified by the classification unit based on the number of pixels corresponding to each of the classification units, and based on the histogram, a plurality of feature colors and respective feature degrees of the plurality of feature colors; A determination means for determining
From the tracking target area in the first image data , third image data having gradations corresponding to the respective characteristic degrees of the plurality of characteristic colors is generated, and a frame different from the first image data is generated. First generation means for generating, from the second image data, fourth image data having a gradation corresponding to the respective characteristic degrees of the plurality of characteristic colors;
A first correlation amount that is a correlation amount with the third image data is obtained for each region of the fourth image data, and a tracking target region in the second image data is obtained based on the first correlation amount. A setting means for setting
A subject tracking device characterized by comprising: It said determining means, based on the numerical value of the ordinate of the histogram, the subject tracking apparatus according to claim 1, characterized in Rukoto seek each feature of the plurality of the characteristic color. It said determining means, subject tracking apparatus according to claim 1 or 2, wherein the feature degree is characterized that you determine the hue corresponding to the predetermined condition is satisfied classification units as the characteristic color. It said determining means combines the plurality of histograms generated from the image data of a plurality of frames, any one of claims 1 to 3, wherein that you determine the characteristic color based on the combined histogram The subject tracking device according to the item. The subject tracking device according to claim 4 , wherein the determination unit synthesizes the plurality of histograms so that a specific gravity with respect to the histogram obtained from a relatively newly obtained frame is increased . Determining means for determining a characteristic color in the tracking target area in the first image data;
From the tracking target area in the first image data, third image data having gradations corresponding to the respective characteristic degrees of the plurality of characteristic colors is generated, and a frame different from the first image data is generated. First generation means for generating, from the second image data, fourth image data having a gradation corresponding to the respective characteristic degrees of the plurality of characteristic colors;
Fifth image data having a gradation corresponding to the luminance is generated from the tracking target area in the first image data, and a sixth image having a gradation corresponding to the luminance is generated from the second image data. Second generating means for generating data;
Correlation between a first correlation amount that is a correlation amount with the third image data for each region of the fourth image data and the fifth image data obtained for each region of the sixth image data. Setting means for setting a tracking target area in the second image data based on a second correlation amount that is a quantity;
A subject tracking device characterized by comprising: It said setting means setting, when the second predetermined threshold value greater than the amount of correlation is present, the region extracted based on the second correlation amount as the tracking target region in the second image data When the second correlation amount does not have a value larger than a predetermined threshold, the region extracted based on the first correlation amount is set as a tracking target region in the second image data. The subject tracking device according to claim 6. The first generation means sets a feature color having the highest feature degree among the plurality of feature colors as a first feature color, and a feature degree of a feature color different from the first feature color; by comparing 1 and feature of the characteristic color of the said third values for each pixel of the image data, and, claims, characterized in Rukoto determined the value of each pixel of the fourth image data The subject tracking device according to any one of 1 to 7. A subject tracking device according to any one of claims 1 to 8,
For obtaining an image signal to be converted into the first image data and the second image data;
An image sensor;
An imaging device comprising: A subject tracking method executed by the subject tracking device,
Based on the color information obtained from the tracking target area in the first image data, the hue is classified in a predetermined classification unit, the distance from the center of the tracking target area in the first image data to each pixel, Generating a histogram classified by the classification unit based on the number of pixels corresponding to each of the classification units, and based on the histogram, a plurality of feature colors and respective feature degrees of the plurality of feature colors; A decision step to determine,
From the tracking target area in the first image data, third image data having gradations corresponding to the respective characteristic degrees of the plurality of characteristic colors is generated, and a frame different from the first image data is generated. A first generation step of generating, from the second image data, fourth image data having a gradation corresponding to the respective characteristic degrees of the plurality of characteristic colors;
A first correlation amount that is a correlation amount with the third image data is obtained for each region of the fourth image data, and a tracking target region in the second image data is obtained based on the first correlation amount. A setting step to set
A method for tracking an object, comprising: A subject tracking method executed by the subject tracking device,
A determining step for determining a characteristic color in the tracking target area in the first image data;
From the tracking target area in the first image data, third image data having gradations corresponding to the respective characteristic degrees of the plurality of characteristic colors is generated, and a frame different from the first image data is generated. A first generation step of generating, from the second image data, fourth image data having a gradation corresponding to the respective characteristic degrees of the plurality of characteristic colors;
Fifth image data having a gradation corresponding to the luminance is generated from the tracking target area in the first image data, and a sixth image having a gradation corresponding to the luminance is generated from the second image data. A second generation step for generating data;
Correlation between a first correlation amount that is a correlation amount with the third image data for each region of the fourth image data and the fifth image data obtained for each region of the sixth image data. A setting step for setting a tracking target area in the second image data based on a second correlation amount that is a quantity ;
A method for tracking an object, comprising: On the computer,
Based on the color information obtained from the tracking target area in the first image data , the hue is classified in a predetermined classification unit, the distance from the center of the tracking target area in the first image data to each pixel, Generating a histogram classified by the classification unit based on the number of pixels corresponding to each of the classification units, and based on the histogram, a plurality of feature colors and respective feature degrees of the plurality of feature colors; A decision step to determine,
From the tracking target area in the first image data, third image data having gradations corresponding to the respective characteristic degrees of the plurality of characteristic colors is generated, and a frame different from the first image data is generated. A first generation step of generating, from the second image data, fourth image data having a gradation corresponding to the respective characteristic degrees of the plurality of characteristic colors;
A first correlation amount that is a correlation amount with the third image data is obtained for each region of the fourth image data, and a tracking target region in the second image data is obtained based on the first correlation amount. A setting step to set
A program for running On the computer,
A determining step for determining a characteristic color in the tracking target area in the first image data;
From the tracking target area in the first image data, third image data having gradations corresponding to the respective characteristic degrees of the plurality of characteristic colors is generated, and a frame different from the first image data is generated. A first generation step of generating, from the second image data, fourth image data having a gradation corresponding to the respective characteristic degrees of the plurality of characteristic colors;
Fifth image data having a gradation corresponding to the luminance is generated from the tracking target area in the first image data, and a sixth image having a gradation corresponding to the luminance is generated from the second image data. A second generation step for generating data;
Correlation between a first correlation amount that is a correlation amount with the third image data for each region of the fourth image data and the fifth image data obtained for each region of the sixth image data. A setting step for setting a tracking target area in the second image data based on a second correlation amount that is a quantity;
A program for running

Images