JP2011146826A - Unit and method for processing image, and program - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To more stably track a subject. <P>SOLUTION: A subject map generation unit 71 generates a subject map indicating region-likelihood of a subject in the current frame, based on a weighting factor for each feature quantity from a feature quantity map indicating the feature quantity in a predetermined region of the current frame of an input image, for each feature of the input image. A candidate rectangular subject region determining unit 72 determines a rectangular region including a candidate subject region in the subject map. The subject region selection unit 73 selects a rectangular subject region including a subject of interest, from the rectangular region, based on region information on the rectangular region. A weighting factor calculation unit 74 calculates a weighting factor for weighting the feature quantity map of the next frame corresponding to a relatively large feature quantity in feature quantities in a region corresponding to a subject region on the feature quantity map for each feature quantity of the current frame. <P>COPYRIGHT: (C)2011,JPO&INPIT

Description

  The present invention relates to an image processing device and method, and a program, and more particularly, to an image processing device and method, and a program for tracking a subject more stably.

  In recent years, in an image captured by an imaging device, a subject selected by a user is tracked, and parameters (focus position, brightness, etc.) relating to imaging are optimally adjusted according to the position of the subject. .

  For example, in a predetermined frame of the input image, a feature amount such as luminance information or color information is extracted from the subject first selected by the user, and a region having a feature amount that matches the feature amount in the next frame There is a method of tracking a subject by searching from (see, for example, Patent Document 1).

JP 2006-72332 A

  However, in the above-described method, the subject is tracked based on the feature amount of the partial region of the subject first selected by the user, so that only one coordinate or partial region of the entire subject region can be identified. Therefore, the entire subject cannot be tracked stably.

  Further, in the input image, it is first selected by the user due to fluctuations in the state of the subject, such as illumination light (for example, color temperature, illuminance, etc.) to the subject, posture of the subject, and size (distance between the imaging device and the subject). If the feature amount of a part of the subject changes, tracking of the subject fails. For example, when the color information of a part of the subject initially selected by the user is extracted as the feature amount, the region having the color information is tracked, but the subject is selected by rotating, If a part of the subject is hidden, the area having the color information does not exist from the input image, and tracking of the subject fails. This can occur even under low illuminance where luminance information and color information as feature quantities are difficult to be generated.

  The present invention has been made in view of such circumstances, and in particular, is intended to track a subject more stably.

  The image processing apparatus according to one aspect of the present invention provides a feature amount map indicating a feature amount in a predetermined region of the current frame of the input image for each feature included in the input image, based on the weight coefficient for each feature amount. Subject map generation means for generating a subject map indicating the likelihood of the subject area in the current frame; subject candidate area rectangularization means for obtaining a rectangular area including the candidate area in the subject map; Subject area selection means for selecting a subject area, which is the rectangular area including the subject, from the rectangular area based on area information about the rectangular area, and the feature amount map for each feature amount of the current frame Of the current frame corresponding to the relatively large feature amount of the feature amount in the region corresponding to the subject region. And a weight coefficient calculating means for calculating the weighting factor for weighting the feature amount map frame after the post-time.

  The subject map generation means includes a feature quantity map generation means for generating the feature quantity map for each feature quantity from the current frame, and a band indicating a plurality of feature quantities in a predetermined band from the feature quantity map. Band feature quantity map generating means for generating a feature quantity map, band feature quantity map synthesizing means for synthesizing the band feature quantity map for each feature quantity, and for each feature quantity obtained by synthesizing the band feature quantity map. And a combined feature amount map combining unit that generates the subject map by combining the combined feature amount map, and the band feature amount map combining unit includes the band feature amount map for each feature amount. And weighting and combining based on the weighting factor for each band, and the weighting factor calculation means causes the band for each feature amount for the predetermined band of the current frame. Calculating the weighting factor for weighting the band feature map of the subsequent frame corresponding to the relatively large feature quantity among the feature quantities in the area corresponding to the subject area on the collection map. Can do.

  The subject map generation means includes a feature quantity map generation means for generating the feature quantity map for each feature quantity from the current frame, and a band indicating a plurality of feature quantities in a predetermined band from the feature quantity map. Band feature quantity map generating means for generating a feature quantity map, band feature quantity map synthesizing means for synthesizing the band feature quantity map for each feature quantity, and for each feature quantity obtained by synthesizing the band feature quantity map. A combined feature amount map combining unit that generates the subject map by combining the combined feature amount map, and the combined feature amount map combining unit stores the combined feature amount map for each feature amount. An area corresponding to the subject area on the combined feature value map for each feature value of the current frame is weighted and synthesized based on a weighting factor. Among definitive the feature amount, the weighting factor for weighting synthesis feature amount map of the rear frame corresponding to the relatively large the feature amount can be calculated.

  The area information is a center coordinate or a size of the rectangular area on the subject map, and the subject area selecting means selects the rectangular area from the rectangular area in the previous frame temporally before the current frame. The rectangular region having the center coordinate or size closest to the center coordinate or size of the subject region can be selected as the subject region.

  The region information is an integral value or a peak value of the feature amount in the rectangular region on the subject map, and the subject region selecting means sends the previous frame temporally before the current frame from the rectangular region. The rectangular region having the integral value or peak value of the feature value closest to the integral value or peak value of the feature value in the subject region selected in (2) can be selected as the subject region.

  The image processing method according to one aspect of the present invention is based on a weighting factor for each feature amount from a feature amount map indicating a feature amount in a predetermined region of the current frame of the input image for each feature of the input image. A subject map generation step for generating a subject map indicating the likelihood of the region of the subject in the current frame, a subject candidate region rectangularization step for obtaining a rectangular region including a region that is a candidate for the subject in the subject map, A subject area selection step of selecting a subject area, which is the rectangular area including the subject, from the rectangular area based on area information about the rectangular area; and on the feature quantity map for each feature quantity of the current frame Corresponding to the relatively large feature amount among the feature amounts in the region corresponding to the subject region, And a weighting coefficient calculation step for calculating the weighting factor for weighting the feature amount map frame after temporally subsequent from the frame.

  The program according to one aspect of the present invention is based on a weight map for each feature amount from a feature amount map indicating a feature amount in a predetermined area of the current frame of the input image for each feature of the input image. A subject map generating step for generating a subject map indicating the region likeness of the subject in the subject, a subject candidate region rectangularizing step for obtaining a rectangular region including a candidate region in the subject map, and the subject to be noted A subject region selection step of selecting a subject region that is the included rectangular region from the rectangular region based on region information about the rectangular region; and on the feature amount map for each feature amount of the current frame, The current amount corresponding to the relatively large feature amount among the feature amounts in the region corresponding to the subject region. To execute processing including the weighting coefficient calculating step of calculating the weighting factor for weighting the feature amount map frame after the post-frame temporally than the computer.

  In one aspect of the present invention, a region of a subject in a current frame is calculated based on a weighting factor for each feature amount from a feature amount map indicating a feature amount in a predetermined region of the current frame of the input image for each feature of the input image. A subject map indicating the likelihood is generated, a rectangular area including a candidate area is obtained in the subject map, and a subject area that is a rectangular area including a subject to be noticed is based on area information about the rectangular area. The feature amount map of the next frame corresponding to a relatively large feature amount among the feature amounts in the region corresponding to the subject region on the feature amount map for each feature amount of the current frame is selected from the rectangular region. A weighting factor for weighting is calculated.

  According to one aspect of the present invention, it is possible to track a subject more stably.

It is a block diagram which shows the structural example of one Embodiment of the image processing apparatus to which this invention is applied. It is a block diagram which shows the structural example of a subject tracking part. It is a block diagram which shows the structural example of a to-be-photographed map production | generation part. It is a block diagram which shows the structural example of a to-be-photographed object candidate area | region rectangle part. It is a block diagram which shows the structural example of a to-be-photographed region selection part. It is a flowchart explaining a subject tracking process. It is a flowchart explaining a subject map generation process. It is a figure which shows the specific example of a to-be-photographed map production | generation process. It is a flowchart explaining a to-be-photographed object candidate area | region rectangle process. It is a figure which shows the specific example of a to-be-photographed object area | region rectangle process. It is a flowchart explaining a subject area selection process. It is a figure explaining the object area | region feature-value sum of a zone | band feature-value map. It is a figure explaining a weighting coefficient. It is a block diagram which shows the structural example of the hardware of a computer.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Configuration example of image processing apparatus]
FIG. 1 shows a configuration example of an embodiment of an image processing apparatus to which the present invention is applied.

  The image processing apparatus 11 of FIG. 1 is provided in a photographing apparatus such as a digital camera that photographs a moving subject.

  The image processing apparatus 11 includes an optical system 31, an imager 32, a digital signal processing unit 33, a control unit 34, a lens driving unit 35, an interface control unit 36, and a user interface 37.

  The optical system 31 is configured as an optical system including an imaging lens (not shown). The light incident on the optical system 31 is photoelectrically converted by an imager 32 configured by an image sensor such as a CCD (Charge Coupled Device). The electrical signal (analog signal) photoelectrically converted by the imager 32 is converted into image data of a digital signal by an A / D (Analog to Digital) converter (not shown) and supplied to the digital signal processor 33.

  The digital signal processing unit 33 performs predetermined signal processing on the digital signal (image data) from the imager 32. The digital signal processing unit 33 includes a preprocessing unit 51, a demosaic processing unit 52, a YC generation unit 53, a resolution conversion unit 54, and a subject tracking unit 55.

  The pre-processing unit 51 performs clamp processing for clamping the R, G, B black levels to a predetermined level, correction processing between the R, G, B color channels, and the like on the image data from the imager 32. . The demosaic processing unit 52 performs demosaic processing so that the image data for each pixel has all the R, G, and B color components. The YC generation unit 53 generates (separates) a luminance (Y) signal and a color (C) signal from R, G, and B image data. The resolution conversion unit 54 performs resolution conversion processing on the image data that has been subjected to various types of signal processing, and supplies the image data to the control unit 34 and an encoding processing unit (not shown).

  The subject tracking unit 55 detects a subject in the input image (captured image) corresponding to the image data based on the image data including the luminance signal and the color signal generated by the YC generation unit 53, and performs subject tracking processing for tracking. Execute.

Here, the detection of the subject is performed on the assumption that the object on the input image that is estimated to be noticed by the user when the user glances at the input image, that is, the object that is estimated to be looked at by the user is the subject. Therefore, the subject is not necessarily limited to a person.

  The subject tracking unit 55 supplies the control unit 34 with data about a subject frame representing a region including the subject in the input image obtained as a result of the subject tracking process. Details of the subject tracking unit 55 will be described later with reference to FIG.

  The control unit 34 controls each unit of the image processing apparatus 11 based on the control signal supplied from the interface control unit 36.

  For example, the control unit 34 supplies parameters and the like used for various signal processing to the digital signal processing unit 33, and data (image data obtained from the digital signal processing unit 33 as a result of various signal processings). Including) and supplying the information to the interface control unit 36. In addition, the control unit 34 supplies the lens driving unit 35 with a control signal for driving the imaging lens included in the optical system 31 and adjusting the diaphragm and the like.

  The user interface 37 is an input device such as a button or a keyboard operated by the user to input an instruction to the image processing apparatus 11, an LCD (Liquid Crystal Display), a microphone, or the like that provides (displays) information to the user. Output device. By operating the user interface 37 as a button, a control signal indicating an instruction to the image processing apparatus 11 is supplied to the control unit 34 via the interface control unit 36. Further, information corresponding to a control signal (data) supplied from the control unit 34 via the interface control unit 36 is displayed on a user interface 37 as an LCD. On the user interface 37 as an LCD, for example, an input image corresponding to the image data and a subject frame that is a result of subject tracking processing on the subject in the input image are displayed.

[Configuration example of subject tracking unit]
Next, a configuration example of the subject tracking unit 55 in FIG. 1 will be described with reference to FIG.

  The subject tracking unit 55 in FIG. 2 includes a subject map generation unit 71, a subject candidate region rectangularization unit 72, a subject region selection unit 73, and a weight coefficient calculation unit 74.

  The subject map generation unit 71 generates a feature amount map indicating a feature amount in a predetermined region of a predetermined frame of the input image for each feature such as brightness and color of the input image, and supplies the feature amount map to the weight coefficient calculation unit 74. In addition, the subject map generation unit 71 generates a subject map indicating the likelihood of the subject area in the input image based on the generated feature amount map and the weighting factor for each feature amount supplied from the weighting factor calculation unit 74. .

  More specifically, the subject map generation unit 71 generates a subject map by weighting and adding information (feature amount) of each region of the feature amount map for each feature for each region at the same position. The subject map generation unit 71 supplies the generated subject map to the subject candidate area rectangularization unit 72.

  In each feature amount map, a region having a larger amount of information, that is, a region on the input image corresponding to a region having a larger feature amount is a region having a higher possibility of including a subject. A region including a subject in the input image can be specified.

  The subject candidate area rectangularizing unit 72 obtains a candidate area in the subject map from the subject map generating unit 71, that is, a rectangular area including a large amount of information in the subject map, and obtains the coordinates of the rectangular area. The coordinate information to be expressed is supplied to the subject area selection unit 73. In addition, the subject candidate area rectangularization unit 72 calculates information related to the rectangular area represented by the coordinate information on the subject map (hereinafter referred to as area information) and supplies the information to the subject area selection unit 73 in association with the coordinate information.

  The subject area selection unit 73 selects a subject area, which is a rectangular area including a subject to be noticed, as a tracking target from the rectangular areas based on the area information from the subject candidate area rectangularization unit 72, and The coordinate information of the subject area is supplied to the control unit 34 (FIG. 1) and the weight coefficient calculation unit 74.

  The weighting coefficient calculation unit 74 is the feature amount of the next frame corresponding to a relatively large feature amount among the feature amounts in the region corresponding to the subject region on each feature amount map of the predetermined frame from the subject map generation unit 71. A weighting factor for weighting the map is calculated and supplied to the subject map generation unit 71.

  With such a configuration, the subject tracking unit 55 can obtain a subject frame representing a subject region for each frame of the input image.

[Configuration example of subject map generator]
Next, a configuration example of the subject map generation unit 71 in FIG. 2 will be described with reference to FIG.

  3 includes a feature map generation unit 111, a band feature map generation unit 112, a band feature map synthesis unit 113, and a synthesis feature map synthesis unit 114.

  The feature amount map generation unit 111 generates a feature amount map indicating information (feature amount) related to features such as luminance and color from a predetermined frame of the input image for each feature amount, and supplies the feature amount map to the band feature amount map generation unit 112.

  The band feature amount map generation unit 112 extracts a feature amount of a predetermined band component from the feature amount in each feature amount map from the feature amount map generation unit 111 a predetermined number of times, and indicates the extracted feature amount A feature amount map is generated and supplied to the weighting factor calculation unit 74 and the band feature amount map synthesis unit 113.

  The band feature quantity map synthesis unit 113 synthesizes the band feature quantity map from the band feature quantity map generation unit 112 for each feature quantity based on the weighting coefficient from the weighting coefficient calculation unit 74, thereby generating a synthesized feature quantity map. Generated and supplied to the weighting factor calculation unit 74 and the combined feature amount map combining unit 114.

  The composite feature amount map combining unit 114 generates a subject map by combining the combined feature amount map from the band feature amount map combining unit 113 based on the weighting factor from the weighting factor calculating unit 74, and generates a subject candidate area. It supplies to the rectangularization part 72 (FIG. 2).

  Here, in the following, the band feature amount map and the combined feature amount map described above are also simply referred to as a feature amount map.

[Configuration Example of Subject Candidate Area Rectification Unit]
Next, a configuration example of the subject candidate area rectangularization unit 72 in FIG. 2 will be described with reference to FIG.

  4 includes a binarization processing unit 131, a labeling processing unit 132, a rectangular region coordinate calculation unit 133, and a region information calculation unit 134.

  The binarization processing unit 131 binarizes information corresponding to each pixel of the input image in the subject map supplied from the subject map generation unit 71 to either 0 or 1 based on a predetermined threshold. To the labeling processing unit 132. Hereinafter, in the subject map, information corresponding to each pixel of the input image is also simply referred to as a pixel.

  In the binarized subject map from the binarization processing unit 131, the labeling processing unit 132 performs labeling on a region adjacent to a pixel having a value of 1 (hereinafter referred to as a connected region), and rectangular region coordinates. It supplies to the calculation part 133.

  The rectangular area coordinate calculation unit 133 calculates the coordinates of the rectangular area including (surrounding) the connected area in the object map labeled with the connected area from the labeling processing unit 132, and uses the coordinate information representing the coordinates as the object map. At the same time, it is supplied to the area information calculation unit 134.

  The area information calculation unit 134 calculates area information that is information related to the rectangular area represented by the coordinate information on the object map from the rectangular area coordinate calculation unit 133, and associates with the coordinate information to the object area selection unit 73 (FIG. To 1).

[Configuration example of subject area selection unit]
Next, a configuration example of the subject area selection unit 73 will be described with reference to FIG.

  The subject area selection unit 73 in FIG. 5 includes an area information comparison unit 151 and a subject area determination unit 152.

  The region information comparison unit 151 compares the region information of each rectangular region from the subject candidate region rectangularization unit 72 with the region information of the subject region of the previous frame stored in the region information storage unit 153, and the comparison result Is supplied to the subject region determination unit 252.

  Based on the comparison result from the region information comparison unit 151, the subject region determination unit 152 sets the rectangular region represented by the coordinate information associated with the region information closest to the region information of the subject region one frame before the subject. This is an area. The subject area determination unit 152 supplies the coordinate information of the determined subject area to the control unit 34 (FIG. 1) and the weighting coefficient calculation unit 74 (FIG. 2), and the area information of the subject area is stored in the area information storage unit 153. Supply.

  The area information storage unit 153 stores the area information of the subject area from the subject area determination unit 152. The area information of the subject area stored in the area information storage unit 153 is read by the area information comparison unit 151 one frame later.

[Subject tracking processing]
In the following, the subject tracking process of the image processing apparatus 11 will be described.

  FIG. 6 is a flowchart for describing subject tracking processing of the image processing apparatus 11. In the subject tracking process, for example, when the user operates a user interface 37 as a button, the operation mode of the image processing apparatus 11 is changed to a subject tracking process mode in which the subject tracking process is executed, and the user interface 37 as an LCD is displayed. Is started when a predetermined area of the subject as a tracking target is selected by the user.

  In step S <b> 11, the subject map generation unit 71 of the subject tracking unit 55 performs a subject map generation process, generates a subject map, and supplies the subject map to the subject candidate area rectangularization unit 72.

[Subject map generation processing]
Here, the details of the subject map generation process will be described with reference to FIGS. FIG. 7 is a flowchart for describing subject map generation processing, and FIG. 8 is a diagram illustrating a specific example of subject map generation processing.

  In step S31 of the flowchart of FIG. 7, the feature amount map generation unit 111 of the subject map generation unit 71 generates a feature amount map for features (for each feature amount) such as luminance and color from a predetermined frame of the input image, and the bandwidth This is supplied to the feature amount map generation unit 112.

Specifically, as shown in FIG. 8, from the input image 200, a luminance information map F ₁ indicating information relating to luminance, color information maps F _{2 to} F _K indicating information relating to color, and edge information indicating information relating to edges. M types of feature amount maps of maps F _{(K + 1) to} F _M are generated.

In the luminance information map F ₁ , the luminance component (luminance signal) Y obtained from each pixel of the input image is information corresponding to each pixel of the input image. In the color information maps F _{2 to} F _K , the input image Color components (color signals) R, G, and B obtained from each pixel become information corresponding to each pixel of the input image. In the edge information maps F _{(K + 1) to} F _M , for example, edge strengths in directions of 0 degrees, 45 degrees, 90 degrees, and 135 degrees in each pixel of the input image are applied to each pixel of the input image. Corresponding information.

In the feature amount map described above, the average value of the R, G, and B components of the pixel may be used as the information (feature amount) of the luminance information map F ₁ , or the color difference components Cr, Cb, Lab color The a * coordinate component and b * coordinate component in the space may be used as information of the color information maps F _{2 to} F _K. Further, 0 degrees, 45 degrees, 90 degrees, and the edge information in the direction of edge strength other than 135 degrees map F _{(K + 1)} to be as the information of the F _M.

  In step S32, the band feature quantity map generation unit 112 extracts a feature quantity of a predetermined band component from the feature quantities in each feature quantity map N times, and generates a band feature quantity map indicating each extracted feature quantity. And supplied to the weight coefficient calculation unit 74 and the band feature amount map synthesis unit 113.

Specifically, as shown in FIG. 8, the luminance information of a predetermined band 1 to N is extracted from the luminance information in the luminance information map F ₁ , and the band luminance information map R indicating the luminance information of each band. _{11 to} R _1N are generated. Further, color information of predetermined bands 1 to N is extracted from the color information in the color information maps F _{2 to} F _K , and band color information maps R _{21 to} R _{2N,. K1 to RKN} are generated. Further, edge information of predetermined bands 1 to N is extracted from edge information in the edge information maps F _{(K + 1) to} F _M , and a band edge information map R _{(K + 1)} indicating edge information of each band. _{) 1 to} R _{(K + 1) N} ,..., R _{M1 to} R _MN are generated. In this manner, the band feature amount map generation unit 112 generates (M × N) types of band feature amount maps.

  Here, an example of processing of the band feature amount map generation unit 112 will be described.

  For example, the band feature quantity map generation unit 112 uses each feature quantity map to generate a plurality of feature quantity maps having different resolutions, and sets these feature quantity maps as pyramid images of the feature quantities. For example, it is assumed that pyramid images of eight resolution layers from level L1 to level L8 are generated, the pyramid image of level L1 has the highest resolution, and the resolution of the pyramid image sequentially decreases from level L1 to level L8. .

  In this case, the feature amount map generated by the feature amount map generation unit 111 is a pyramid image of level L1. In addition, in the pyramid image of level Li (where 1 ≦ i ≦ 7), one pixel of the pyramid image of level L (i + 1) in which the average value of the pixel values of four pixels adjacent to each other corresponds to those pixels Pixel value. Accordingly, the pyramid image at the level L (i + 1) is an image of half the length and breadth (discarded if not divisible) with respect to the pyramid image at the level Li.

  Further, the band feature amount map generation unit 112 selects two pyramid images having different hierarchies from among a plurality of pyramid images, obtains a difference between the selected pyramid images, and generates N difference images of each feature amount. . Since the pyramid images in each layer have different sizes (number of pixels), the smaller pyramid image is up-converted in accordance with the larger pyramid image when generating the difference image.

  For example, the band feature map generation unit 112 includes the level L6 and level L3, the level L7 and level L3, the level L7 and level L4, the level L8 and level L4, and the level L8 and The difference of the pyramid image of the combination of each hierarchy of level L5 is calculated | required. As a result, difference images of a total of five feature amounts are obtained.

  Specifically, for example, when a differential image of a combination of level L6 and level L3 is generated, the pyramid image at level L6 is up-converted according to the size of the pyramid image at level L3. That is, the pixel value of one pixel of the pyramid image of level L6 before up-conversion is the pixel value of several pixels adjacent to each other of the pyramid image of level L6 after up-conversion corresponding to that pixel. Then, the difference between the pixel value of the pixel of the level L6 pyramid image and the pixel value of the pixel of the level L3 pyramid image at the same position as the pixel is obtained, and the difference is set as the pixel value of the pixel of the difference image. The

  In this way, by generating the difference image, it is possible to extract the feature amount of a predetermined band component from the feature amount map so that the feature amount map is subjected to filter processing using a bandpass filter.

  In the above description, the width of the band extracted from the feature map is determined by the combination of each layer of the pyramid image when obtaining the difference image, but this combination is arbitrarily determined.

  Further, the extraction of the feature amount of the predetermined band component is not limited to the above-described method using the difference image, and other methods may be used.

Returning to the flowchart of FIG. 7, in step S <b> 33, the band feature amount map synthesis unit 113 uses the band feature amount map from the band feature amount map generation unit 112 based on the weight coefficient group W _R from the weight coefficient calculation unit 74. Combining for each feature. The band feature amount map combining unit 113 supplies the combined band feature amount map (combined feature amount map) to the weighting coefficient calculating unit 74 and the combined feature amount map combining unit 114.

Specifically, as shown in FIG. 8, the band luminance information maps R _{11 to} R _1N are weighted by weight coefficients w _{11 to} w _1N that are weights for each band luminance information map from the weight coefficient calculation unit 74. The combined feature value map C ₁ is obtained by addition. Further, the map-band color information R ₂₁ to R _2N, ..., R _K1 to R _KN is the weighting factor w ₂₁ through w _2N is the weight of the band color information for each map from the weight coefficient calculation unit 74, ..., w _K1 to Weighted addition is performed by w _KN to obtain composite feature amount maps C _{2 to} C _K. Further, the band edge information maps R _{(K + 1) 1 to} R _{(K + 1) N} ,..., R _{M1 to} R _MN are weight coefficients w that are weights for each band edge information map from the weight coefficient calculation unit 74. _{(K + 1) 1} to _{w (K + 1) N,} ..., are summed weighted by w _M1 to w _MN, synthetic feature amount map C _{K + 1} to C _M are determined. In this manner, the band feature quantity map synthesis unit 113 generates M types of synthesized feature quantity maps. As will be described later in detail weight coefficient group W _R, the weight coefficients of the weight coefficient group W _R has a value of 0 to 1. However, the first subject map generation processing, each weighting factor of the weighting coefficient group W _R is all 1, band feature amount maps are added without weighting.

In step S34, the combined feature value map combining unit 114 combines the combined feature amount map from the band feature amount map combining unit 113 based on the weighting coefficient group W _C from the weighting coefficient calculating unit 74, so that the subject map Is generated and supplied to the subject candidate area rectangularization unit 72.

Specifically, as shown in FIG. 8, the combined feature amount maps C _{1 to} C _M use weight coefficients w _{1 to} w _M that are weights for each band luminance information map from the weight coefficient calculation unit 74. Linearly combined. Furthermore, the subject map 201 is obtained by multiplying the pixel value of the map obtained as a result of the linear combination by the subject weight, which is a weight obtained in advance, and normalizing it. Although details of the weighting coefficient group W _C will be described later, each weighting coefficient of the weighting coefficient group W _C has a value of 0 to 1. However, in the first subject map generation process, all the weighting factors of the weighting factor group W _C are set to 1, and the combined feature amount map is linearly combined without weighting.

  In other words, if the target position (pixel) on the subject map to be obtained is the target position, the weight coefficient for each composite feature map is set to the pixel value at the same position (pixel) as the target position of each composite feature map. And the sum of the pixel values multiplied by the weighting coefficient is used as the pixel value at the target position. Further, the pixel value at each position of the subject map obtained in this way is multiplied by a subject weight obtained in advance with respect to the subject map and normalized to obtain a final subject map. For example, the normalization is performed so that the pixel value of each pixel of the subject map becomes a value between 0 and 255.

  As described above, the subject map generation unit 71 generates the subject map by generating the band feature amount map and the combined feature amount map from the feature amount map.

  Returning to the flowchart of FIG. 6, in step S <b> 12, the subject candidate region rectangularization unit 72 performs subject candidate region rectangularization processing, and in the subject map from the subject map generation unit 71, a rectangular region that includes regions that are subject candidates. Ask for.

[Subject candidate area rectangle processing]
Here, with reference to FIG. 9 and FIG. 10, details of the subject candidate area rectangularization process will be described. FIG. 9 is a flowchart for describing the subject candidate area rectangularization process, and FIG. 10 is a diagram illustrating a specific example of the subject candidate area rectangularization process.

  In step S51 of the flowchart of FIG. 9, the binarization processing unit 131 of the subject candidate area rectangularization unit 72 sets the information in the subject map supplied from the subject map generation unit 71 to 0 or 1 based on a predetermined threshold. It binarizes to either value and supplies it to the labeling processing unit 132.

  More specifically, the binarization processing unit 131 performs, for example, a pixel value of each pixel of the subject map 201 that is a value between 0 and 255 shown first from the top in FIG. A pixel value having a value smaller than the threshold 127 is set to 0, and a pixel value having a value greater than 127 is set to 1. As a result, a binarized map 202 as shown second from the top in FIG. 10 is obtained. In the binarization map 202 shown in FIG. 10, a portion (pixel) indicated by white has a pixel value of 1, and a portion (pixel) indicated by black has a pixel value of 0. Although the threshold value is 127 here, other values may be used.

  In step S52, the labeling processing unit 132 obtains a pixel having a pixel value of 1 obtained from the binarization processing unit 131 by a morphological operation or the like in the binarization map 202 (binarized subject map). The adjacent connected areas are labeled and supplied to the rectangular area coordinate calculation unit 133.

  More specifically, for example, as shown third from the top in FIG. 10, in the binarization map 202, the connected area 211 is labeled with the label “1”, and the connected area 212 is labeled “2”. ".

  In step S <b> 53, the rectangular area coordinate calculation unit 133 calculates the coordinates of the rectangular area including (enclosed) the connected area in the binarized map 202 from the labeling processing unit 132, and converts the coordinate information representing the coordinates into binary. It is supplied to the region information calculation unit 134 together with the conversion map 202.

  More specifically, as shown in the fourth from the top in FIG. 10, in the binarization map 202, a rectangular frame (circumscribed frame) 221 that surrounds the connected region 211 labeled with the label “1” from the outside is detected. Then, for example, the coordinates of the vertices of the upper left corner and the lower right corner of the rectangular frame are obtained. Further, a rectangular frame 222 surrounding the connection area 212 labeled with the label “2” is detected from the outside, and the coordinates of the vertexes of the rectangular frame, for example, the upper left and lower right in the figure are obtained.

  In step S <b> 54, the region information calculation unit 134 is a region for a rectangular region surrounded by a rectangular frame on the subject map based on the coordinate information from the rectangular region coordinate calculation unit 133 and the subject map from the subject map generation unit 71. Calculate information.

  More specifically, the area information calculation unit 134 represents the rectangular frame size and center position coordinates based on the coordinate information from the rectangular area coordinate calculation unit 133 representing the rectangular frames 221 and 222 in the binarization map 202. Is calculated as region information for the rectangular region. The area information calculation unit 134 supplies the calculated area information to the subject area selection unit 73 in association with the coordinate information from the rectangular area coordinate calculation unit 133.

  As described above, the subject candidate region rectangularization unit 72 determines the characteristics of the rectangular frame surrounding each region that is a candidate for the subject to be noted in the subject map and the region surrounded by the rectangular frame on the subject map. Find the region information to represent.

  Returning to the flowchart of FIG. 6, in step S <b> 13, the subject region selection unit 73 performs subject region selection processing, and uses the subject region, which is a rectangular region including the subject to be noted, as region information from the subject region selection unit 73. Based on this, select from the rectangular area.

[Subject area selection processing]
Here, the details of the subject area selection processing will be described with reference to the flowchart of FIG.

  In step S <b> 71, the region information comparison unit 151 compares the region information of each rectangular region from the subject candidate region rectangularization unit 72 with the region information of the subject region of the previous frame stored in the region information storage unit 153. Then, the comparison result is supplied to the subject region determination unit 152.

  More specifically, for example, the region information comparison unit 151 stores the size of the rectangular frame surrounding each rectangular region on the subject map from the subject candidate region rectangularization unit 72 and the region information storage unit 153. The size of the rectangular frame (subject frame) surrounding the subject area one frame before is compared. Further, for example, the region information comparison unit 151 is stored in the region information storage unit 153 and the coordinates of the center position of a rectangular frame surrounding each rectangular region on the subject map from the subject candidate region rectangularization unit 72. The coordinates of the center position of a rectangular frame (subject frame) surrounding the subject area one frame before are compared.

  In step S72, the subject area determination unit 152, based on the comparison result from the area information comparison unit 151, has the rectangular frame closest to the coordinates of the size or center position of the rectangular frame (subject frame) surrounding the subject area one frame before. A rectangular area having a size or a center position is defined as a subject area. The subject area determination unit 152 supplies the coordinate information of the determined subject area to the control unit 34 and the weighting coefficient calculation unit 74, and the area information storage unit 153 stores the area information (subject frame size or center position) of the subject area. To supply.

  However, in the first subject region selection process, the region information storage unit 153 does not store the region information of the subject region one frame before, so the predetermined region of the subject selected by the user at the start of the subject tracking process A rectangular area including (hereinafter referred to as an initial selection area) is a subject area.

  As described above, the subject area selection unit 73 selects the subject area of the subject to be noted from the rectangular areas that are subject candidates.

[Calculation of weighting factor]
Returning to the flowchart of FIG. 6, in step S <b> 14, the weight coefficient calculation unit 74 includes a band feature amount map and a combined feature amount map from the subject map generation unit 71, and coordinate information representing the subject region from the subject region selection unit 73. Based on the above, the weight coefficient groups W _R and W _C shown in FIG. 8 are calculated.

More specifically, as shown in FIG. 12, a rectangular area corresponding to a subject frame 231 representing a subject area on a predetermined band feature amount map R _mn (1 ≦ m ≦ M, 1 ≦ n ≦ N). when the characteristic amounts of the inner sum of (information amount) was subject region feature amount sum r _mn, weight coefficient group W _R as shown on the upper side of FIG. 13 is calculated.

Each coefficient in the weight coefficient group W _R of FIG. 13 corresponds to each of the weight coefficients w ₁₁ to w _MN shown in FIG. In FIG. 13, Max [a,..., Z] represents the maximum value among the values a to z.

For example, each coefficient of the first row from the top in the weight coefficient group W _R of FIG. 13, shown in FIG. 8, "band 1" is the feature quantity for each of the band feature amount maps R ₁₁ to R _M1 of Weighting factors w _{11 to} w _M1 are shown. As shown in FIG. 13, the weighting factors w _{11 to} w _M1 are denominators that are the maximum values of the subject area feature amount sums r _{11 to} r _{M1 for} the band feature amount maps R _{11 to} R _M1, respectively. Are coefficients for the subject area feature amount sums r _{11 to} r _M1 for the band feature amount maps R _{11 to} R _M1, respectively, and take values of 0 to 1.

Similarly, the coefficients in the Nth row from the top in the weighting coefficient group W _R in FIG. 13 are the band feature amount maps R _{1N to} R _MN for each feature amount that is “band N” shown in FIG. The weighting factors w _{1N to} w _MN are shown. As shown in FIG. 13, the weighting factors w _{1N to} w _MN have the denominator as the maximum value of the subject area feature amount sums r _{1N to} r _{MN for} the band feature amount maps R _{1N to} R _MN, respectively. _Are coefficients for subject area feature amount sums r _{1N to} r _MN for the band feature amount maps R _{1N to} R _MN, respectively, and take values of 0 to 1.

That is, according to the weighting factors w _{1n to} w _Mn , the band feature quantity map of the feature quantity that maximizes the subject area feature quantity sum in the band feature quantity maps R _{1n to} R _{Mn for} each feature quantity that is “band n”. The other band feature amount maps are weighted according to the subject region feature amount sum.

Further, the sum of the feature amounts (information amounts) in the rectangular area corresponding to the rectangular frame 221 representing the subject area on the predetermined composite feature quantity map C _m (1 ≦ m ≦ M) is the subject area feature quantity sum c _m. In this case, a weight coefficient group W _C as shown in the lower side of FIG. 13 is calculated.

Each of the coefficients in the weight coefficient group W _C in FIG. 13 corresponds to each of the weight coefficients w _{1 to} w _M shown in FIG.

That is, each coefficient in the weight coefficient group W _C in FIG. 13 indicates the weight coefficients w _{1 to} w _M for the combined feature amount maps C _{1 to} C _M for each feature amount shown in FIG. As shown in FIG. 13, the weighting factors w _{1 to} w _M have the denominator as the maximum value of the subject area feature amount sums c _{1 to} c _{M for} the combined feature amount maps C _{1 to} C _M, respectively. Are coefficients that are subject area feature amount sums c _{1 to} c _M for each of the combined feature amount maps C _{1 to} C _M , and take values of 0 to 1.

In other words, according to the weighting factors w _{1 to} w _m , the combined feature value map having the maximum subject area feature value in the combined feature value maps C _{1 to} C _{M for} each feature value has a maximum value of 1. The other band feature quantity maps are weighted according to the subject area feature quantity sum.

The weighting factor calculation unit 74 supplies the calculated weighting factor group W _R to the band feature amount map synthesis unit 113 of the subject map generation unit 71 and also uses the weighting factor group W _C as the synthesis feature amount of the subject map generation unit 71. This is supplied to the map composition unit 114. In the flowchart of FIG. 6, after step S14, the process returns to step S11, subject tracking processing for the next frame is executed, and this processing is repeated for each frame.

  According to the above processing, in the feature amount map for each feature amount for a predetermined frame of the input image, the following is performed according to the relative size of the feature amount of the region corresponding to the subject region selected in the frame. A weighting factor for the feature amount map for each feature amount of the frame is determined. Therefore, even when the feature amount varies between frames, a feature map of the feature amount that best represents the subject among the plurality of feature amounts is generated with the most weighted subject map. Even in an environment where the state of the subject fluctuates, the subject can be tracked more stably.

  In addition, since the subject area is determined so as to include the entire subject, the subject can be tracked more stably even in an environment in which the state of a partial area of the subject fluctuates.

  In particular, when any of the coordinates in the subject area (or a partial area including the coordinates) is identified in the conventional subject tracking method, the entire subject cannot be tracked, and AF (Auto The detection frame for Focus, AE (Auto Exposure), and ACC (Auto Color Control) could not be set correctly. In addition, in the case where the same feature amount region having the same feature amount in the subject region is identified, the accuracy of setting the detection frame can be improved as compared with the above case. In many cases, it was only a small part of the area, and sufficient detection accuracy could not be obtained.

  On the other hand, according to the subject tracking process of the present invention, the subject region including the entire subject can be identified, so that the detection accuracy can be improved, and the tracking result can be applied to various applications.

  In addition, conventional subject tracking methods include, for example, a method for detecting and tracking a human by registering an entire image of a human in a dictionary by learning, but a non-human subject that is not registered in the dictionary can be detected. It cannot be tracked. Furthermore, since the amount of information (images) registered in the dictionary is enormous, the apparatus scale becomes large.

  On the other hand, according to the subject tracking process of the present invention, an arbitrary subject can be detected and tracked, and it is not necessary to register a huge amount of information in a dictionary or the like. it can.

  In the above description, the luminance component, the color component, and the edge direction are used as the feature amount. However, the present invention is not limited to this, and for example, motion information or the like may be added. In addition, the feature amount used is preferably a complementary relationship such as a luminance component and a color component, and may be appropriately selected.

In the above description, M × (N + 1) types of weighting coefficients are calculated corresponding to M × (N + 1) types of feature amount maps. However, only weighting factors corresponding to some feature amount maps are calculated. Is appropriately calculated, the amount of calculation in the image processing apparatus 11 can be suppressed. For example, only the weighting factors w _{1 to} w _M corresponding to the M types of feature amount maps of the combined feature amount maps C _{1 to} C _M may be calculated.

  Further, in the above, the area information calculation unit 134 calculates the size of the rectangular frame and the coordinates of the center position as the area information of the rectangular area. However, the integrated value or peak value ( (Maximum value) may be calculated. In this case, in the subject region selection process (FIG. 11), a rectangular region having an integral value or peak value of the pixel value in the region closest to the integrated value or peak value of the pixel value in the subject region one frame before is the subject. It is considered as an area.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 14 is a block diagram illustrating a hardware configuration example of a computer that executes the above-described series of processing by a program.

  In a computer, a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, and a RAM (Random Access Memory) 903 are connected to each other by a bus 904.

  An input / output interface 905 is further connected to the bus 904. The input / output interface 905 includes an input unit 906 made up of a keyboard, mouse, microphone, etc., an output unit 907 made up of a display, a speaker, etc., a storage unit 908 made up of a hard disk, nonvolatile memory, etc., and a communication unit 909 made up of a network interface, etc. A drive 910 for driving a removable medium 911 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory is connected.

  In the computer configured as described above, the CPU 901 loads the program stored in the storage unit 908 to the RAM 903 via the input / output interface 905 and the bus 904 and executes the program, for example. Is performed.

  The program executed by the computer (CPU 901) is, for example, a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only Memory), DVD (Digital Versatile Disc), etc.), a magneto-optical disk, or a semiconductor. The program is recorded on a removable medium 911 which is a package medium including a memory or the like, or is provided via a wired or wireless transmission medium such as a local area network, the Internet, or digital satellite broadcasting.

  The program can be installed in the storage unit 908 via the input / output interface 905 by attaching the removable medium 911 to the drive 910. The program can be received by the communication unit 909 via a wired or wireless transmission medium and installed in the storage unit 908. In addition, the program can be installed in the ROM 902 or the storage unit 908 in advance.

  The program executed by the computer may be a program that is processed in time series in the order described in this specification, or in parallel or at a necessary timing such as when a call is made. It may be a program for processing.

  The embodiments of the present invention are not limited to the above-described embodiments, and various modifications can be made without departing from the scope of the present invention.

  DESCRIPTION OF SYMBOLS 11 Image processing apparatus, 55 Subject tracking part, 71 Subject map generation part, 72 Subject candidate area | region rectangularization part, 73 Subject area selection part, 74 Weight coefficient calculation part, 111 Feature value map generation part, 112 Band feature value map generation part , 113 band feature map synthesis unit, 114 synthesized feature map synthesis unit, 131 binarization processing unit, 132 labeling processing unit, 133 rectangular area coordinate calculation unit, 134 area information calculation unit, 151 area information comparison unit, 152 subject Area determination unit, 200 input image, 201 subject map, 221 and 222 rectangular area, 231 subject frame

Claims (7)

A subject map indicating the likelihood of a subject area in the current frame based on a weighting factor for each feature amount from a feature amount map indicating a feature amount in a predetermined region of the current frame of the input image for each feature of the input image Subject map generating means for generating
In the subject map, subject candidate region rectangularization means for obtaining a rectangular region including a candidate region for the subject;
Subject region selection means for selecting a subject region that is the rectangular region including the subject to be noted from the rectangular region based on region information about the rectangular region;
Of the feature amounts in the region corresponding to the subject region on the feature amount map for each feature amount of the current frame, the feature amount corresponding to the relatively large feature amount is temporally later than the current frame. An image processing apparatus comprising: a weighting factor calculating unit that calculates the weighting factor for weighting a feature amount map of a subsequent frame. The subject map generation means includes:
Feature quantity map generating means for generating the feature quantity map for each feature quantity from the current frame;
Band feature quantity map generating means for generating a plurality of band feature quantity maps indicating the feature quantities of a predetermined band from the feature quantity map;
Band feature quantity map synthesis means for synthesizing the band feature quantity map for each feature quantity;
A combined feature amount map combining unit that generates the subject map by combining the combined feature amount map for each of the feature amounts combined with the band feature amount map;
The band feature amount map combining unit weights and combines the band feature amount map based on the weighting factor for each feature amount and for each band,
The weighting factor calculating unit is configured to calculate a relatively large amount of the feature amount in the region corresponding to the subject region on the band feature amount map for each feature amount for the predetermined band of the current frame. The image processing apparatus according to claim 1, wherein the weighting coefficient for weighting the band feature quantity map of the subsequent frame corresponding to the feature quantity is calculated. The subject map generation means includes:
Feature quantity map generating means for generating the feature quantity map for each feature quantity from the current frame;
Band feature quantity map generating means for generating a plurality of band feature quantity maps indicating the feature quantities of a predetermined band from the feature quantity map;
Band feature quantity map synthesis means for synthesizing the band feature quantity map for each feature quantity;
A combined feature amount map combining unit that generates the subject map by combining the combined feature amount map for each of the feature amounts combined with the band feature amount map;
The synthesized feature map synthesis means weights and synthesizes the synthesized feature map based on the weighting coefficient for each feature.
The weighting factor calculation means corresponds to the relatively large feature amount among the feature amounts in the region corresponding to the subject region on the composite feature amount map for each feature amount of the current frame. The image processing apparatus according to claim 1, wherein the weighting coefficient for weighting the composite feature amount map of the subsequent frame is calculated. The area information is center coordinates or size of the rectangular area on the subject map,
The subject area selecting means is configured to select the rectangular area having a center coordinate or size closest to a center coordinate or size of the subject area selected in the previous frame temporally prior to the current frame from the rectangular area. The image processing device according to claim 1, wherein the image processing device is selected as a subject area. The area information is an integral value or a peak value of the feature amount in the rectangular area on the subject map,
The subject area selection means is an integrated value of the feature quantity closest to the integral value or peak value of the feature quantity in the subject area selected in the previous frame temporally prior to the current frame from the rectangular area, or The image processing apparatus according to claim 1, wherein the rectangular area having a peak value is selected as the subject area. A subject map indicating the likelihood of a subject area in the current frame based on a weighting factor for each feature amount from a feature amount map indicating a feature amount in a predetermined region of the current frame of the input image for each feature of the input image A subject map generation step for generating
In the subject map, a subject candidate region rectangularization step for obtaining a rectangular region including a candidate region for the subject;
A subject region selection step of selecting a subject region that is the rectangular region including the subject to be noted from the rectangular region based on region information about the rectangular region;
Of the feature amounts in the region corresponding to the subject region on the feature amount map for each feature amount of the current frame, the feature amount corresponding to the relatively large feature amount is temporally later than the current frame. A weighting factor calculating step of calculating the weighting factor for weighting the feature amount map of the subsequent frame. A subject map indicating the likelihood of a subject area in the current frame based on a weighting factor for each feature amount from a feature amount map indicating a feature amount in a predetermined region of the current frame of the input image for each feature of the input image A subject map generation step for generating
In the subject map, a subject candidate region rectangularization step for obtaining a rectangular region including a candidate region for the subject;
A subject region selection step of selecting a subject region that is the rectangular region including the subject to be noted from the rectangular region based on region information about the rectangular region;
Of the feature amounts in the region corresponding to the subject region on the feature amount map for each feature amount of the current frame, the feature amount corresponding to the relatively large feature amount is temporally later than the current frame. A program that causes a computer to execute processing including: a weighting factor calculating step of calculating the weighting factor that weights a feature amount map of a subsequent frame.

Images