JP5677080B2 - Image processing apparatus and control method thereof - Google Patents

Description

  The present invention relates to an image processing apparatus. In particular, the present invention relates to an image processing apparatus having a main subject detection function for automatically detecting a subject in an image and a subject tracking function for continuously capturing the movement of the subject.

  In an image processing apparatus typified by a digital camera, it is very effective to improve the quality of a captured image to perform appropriate imaging control and image processing on a subject that is of high interest to the user. In order to perform the above-described appropriate processing on a moving subject, a digital camera having a subject tracking function that keeps capturing a subject by a pattern matching method or the like has been developed. In tracking a subject, it is necessary to specify a subject to be tracked, for example, a subject that is highly interested by the user (hereinafter referred to as a main subject). However, in the configuration in which the main subject area is designated by a user's manual operation such as a touch panel or button operation, there is a problem that the user's operation becomes complicated.

  In recent years, as disclosed in Patent Document 1, a digital camera having a main subject detection function for automatically detecting a main subject and separating it from the background has been developed. Various proposals have also been made regarding image processing techniques related to the main subject detection function.

  In Patent Document 2, it is proposed that a neural network is configured based on the extracted subject area, thereby realizing subject recognition that is resistant to misalignment and has high accuracy. Further, in Patent Document 3, in a moving image, each frame is divided into pixel blocks of a certain size, an average value within the block of pixel data is obtained for each pixel block, and an average within the block between the current frame and the previous frame is obtained. A value difference is calculated, and an area in which blocks having a large difference appear continuously is regarded as a movement of the subject, and an image display area is determined from the amount of movement of the subject.

JP 2007-129310 A JP 05-282275 A JP 2000-295517 A

  In a digital camera having a main subject detection function and a subject tracking function, a method of continuously capturing a moving subject by applying a subject tracking function to a subject area detected by the main subject detection function is generally used. It is used. However, the main subject detection algorithm related to subject extraction is generally different from the subject tracking algorithm for tracking a moving subject by a pattern matching method or the like. Therefore, when the main subject area detected as in Patent Literature 2 and Patent Literature 3 is applied to subject detection as it is, there is a problem that the tracking performance may be deteriorated.

  The present invention has been made in view of the above problems, and an object thereof is to improve the tracking performance in tracking the automatically detected main subject region.

In order to achieve the above object, an image processing apparatus according to an aspect of the present invention has the following arrangement. That is,
Detecting means for detecting a subject area from at least one frame of the moving image;
Setting means for setting an area for extracting image information used in the tracking process from an area set based on the position of the subject area detected by the detecting means, using a method different from the detecting means. When,
Tracking means for executing the tracking process based on the image information of the area set by the setting means ,
The setting means divides one of the frames into a plurality of blocks, sets a predetermined number of blocks including a block including the position of the subject area and the surrounding blocks from the plurality of blocks as a reference range. from blocks included in the scope, one or more blocks, select a region for extracting the image information.

  According to the present invention, it is possible to automatically detect a main subject without a complicated input operation by a user, accurately track the subject even in a moving subject, and perform appropriate imaging control and image processing on the subject. An image processing apparatus that can be provided can be provided.

1 is a block diagram of a digital camera according to an embodiment. 6 is a flowchart illustrating main subject detection processing according to the embodiment. The figure explaining the histogram division process by the main subject detection process. Explanatory drawing regarding calculation of the object evaluation value in the main object detection process. Explanatory drawing related to calculation of subject evaluation value in main subject detection processing 5 is a flowchart showing overall processing of the digital camera according to the embodiment. The figure explaining operation | movement of a positional information update part. 6 is a flowchart for explaining subject tracking processing. Explanatory drawing in connection with template selection. The figure explaining the template selection of a subject tracking part. The figure which shows the example of distribution of the similar color pixel in a subject tracking part. The figure explaining the template shape in a subject tracking part. The figure which shows the example of the weight table design curve in embodiment. The figure which shows the example of the weight table in embodiment. The figure which shows the example of the conversion function of an evaluation value.

  Hereinafter, a preferred embodiment of the present invention will be described in detail with reference to the accompanying drawings.

  FIG. 1 is a diagram illustrating a configuration example of a digital camera 100 as an image processing apparatus of the present invention. In FIG. 1, 10 is a photographing lens, 12 is a mechanical shutter, and 14 is an aperture. Reference numeral 16 denotes an image sensor that converts an optical image into an electrical signal. In this embodiment, a CCD is used. Reference numeral 18 denotes a gain circuit that applies a gain to the output signal from the image sensor 16, and reference numeral 22 denotes an A / D converter that converts an analog signal output into a digital signal. In addition to the mechanical shutter 12, the accumulation time can be controlled as an electronic shutter by controlling the reset timing of the image sensor 16, and the digital camera 100 can be used for moving image shooting and the like.

  An image processing circuit 50 performs predetermined pixel interpolation processing and color conversion processing on the data from the A / D converter 22. The image processing circuit 50 performs predetermined calculation processing using the captured image data, and provides the obtained calculation result to the system control circuit 60. The system control circuit 60 controls the exposure control unit and the lens driving circuit 42 based on the calculation result provided from the image processing circuit 50, and performs AE processing (automatic exposure function) / AF processing (automatic focus function). It is carried out. The exposure control unit includes an aperture driving circuit 26, a mechanical shutter driving circuit 28, and a flash light emitting circuit 46. The image processing circuit 50 performs predetermined arithmetic processing using the captured image data, and also performs AWB (auto white balance) processing based on the obtained arithmetic result. The system control circuit 60 implements various processes by executing a program stored in a memory (not shown) by a CPU (not shown). A main subject detection process, which will be described later, and a subject tracking process using the detection result are executed by the system control circuit 60.

  The image data digitally converted by the A / D converter 22 is converted into desired image data by the image processing circuit 50. The compressed image data is written to the image storage medium 82 via the image storage medium I / F 80 by the compression / decompression circuit 32 that compresses and expands the image data.

  An image display unit 108 displays image data for display written in the VRAM 34. By sequentially displaying image data captured by the image sensor 16 using the image display unit 108, an electronic viewfinder function (EVF: Electric Viewfinder) can be realized. At this time, it is also possible to superimpose and display the photometry area information and distance measurement area information in AE and AF on the EVF live image. Further, in order to indicate the recognition status of the subject, it is possible to display a frame superimposed on the EVF live image in the subject area that is the recognition result of the human face or the main subject. Further, if a touch panel is attached to the image display unit 108, it can also be used as an input device. In addition to the image display unit 108 provided in the digital camera 100, the image data processed by the image processing circuit 50 is displayed on an external device 86 such as a TV display connected via the external device I / F 84. Is also possible.

  Reference numeral 30 denotes a temporary storage memory that stores captured still images and moving images. The temporary storage memory 30 has a storage capacity sufficient to store a predetermined number of still images and a moving image for a predetermined time, and can also be used as a work area for the system control circuit 60.

  The mechanical shutter drive circuit 28 controls the mechanical shutter 12 and, in conjunction with the flash light emission circuit 46, realizes a flash light control function. The lens driving circuit 42 controls the focusing of the taking lens 10. A system control circuit 60 controls the entire digital camera 100. Reference numeral 70 denotes an operation unit made up of operation members of the digital camera 100, which is used by the user to instruct the system control circuit 60 to turn on / off the power, start shooting, switch shooting modes, and change various settings.

  When shooting is instructed by the shutter switch of the operation unit 70, the image sensor 16 is exposed for the exposure time determined by the AE process. Thereafter, a signal is read from the image sensor 16, and the read signal is converted into a digital value by the A / D converter 22 and written to the temporary storage memory 30. Development processing in the image processing circuit 50 and compression in the compression / decompression circuit 32 are executed, and recording processing for writing image data to the image storage medium 82 via the image storage medium I / F 80 is performed. The image storage medium 82 is a recording medium such as a memory card or a hard disk.

  The image processing circuit 50 includes various signal processing circuits for performing main subject discrimination and subject tracking, which will be described later, and can execute processing such as image block division, histogram calculation, and difference calculation at high speed. it can.

  Next, in the digital camera having the above-described configuration, the main subject is automatically selected in order to shoot the main subject that is the subject to be photographed under optimum conditions without focusing on the main subject. A detection method (main subject detection process) will be described. Note that the main subject detection process described below is merely an example, and any known main subject detection process may be used.

  With reference to FIG. 2, the main subject detection processing procedure according to the present embodiment will be described in detail below. Here, as an example, a method for extracting a subject region by extracting one or a plurality of subject candidate regions from an image and calculating an evaluation value of the subjectness of the region based on a feature amount will be described.

  FIG. 2 is a flowchart illustrating main subject detection processing according to the embodiment. First, in step S101, the system control circuit 60 divides the image into a plurality of blocks having a certain size. Next, in step S102, the system control circuit 60 calculates the average value of the hues of the pixels in each divided block and sets it as the representative value of each block. In step S103, the system control circuit 60 creates a hue histogram with the hue representative value obtained in step S102 as an element. Note that since blocks with low saturation have low hue value reliability, only blocks with saturation equal to or higher than a predetermined threshold may be used for histogram creation. In step S104, the system control circuit 60 divides the hue histogram into a plurality of groups that are regarded as the same color region.

  Hereinafter, a method of dividing the hue histogram into n (n> 1) identical color groups will be described with reference to FIGS. First, the system control circuit 60 scans all sections of the histogram and detects the highest peak HighestPeak1. Next, the system control circuit 60 scans from the peak toward the left and right valleys, and the same color region is used in the interval until the frequency of the histogram becomes equal to or less than TH_Freq or the distance from the HighestPeak1 reaches TH_HueRange. And Then, the system control circuit 60 groups the blocks in which the representative values of the blocks are included in this range as the same color area 1 and records this section as having been grouped. Next, the system control circuit 60 scans again all the sections of the histogram excluding the section subjected to the grouping process, and detects the highest peak HighestPeak2 (FIG. 3B). Thereafter, the same process is repeated until all the sections in the histogram are grouped, whereby the hue histogram can be divided into n identical color groups. When scanning from the peak to the valley direction, if you arrive at a grouped section before reaching the section where the histogram frequency is less than TH_Freq, the section up to that is the same color area range. And

  Next, in step S105, the system control circuit 60 scans all blocks, and performs a labeling process in which adjacent consecutive identical color areas are set as the same group, and non-adjacent identical color areas are set as another group. By performing this labeling process for a plurality of colors, it is possible to recognize another object that is in the same color area and is separated as another area.

Next, in step S106 of FIG. 2, the system control circuit 60 calculates a subject evaluation value P for each of the regions classified by the labeling process in step S105. As a method for calculating the subject evaluation value P, the following method can be used.
A method of adding an evaluation value (center distance evaluation value) according to the distance R from the screen center. For example, as shown in FIG. 4, the evaluation value is set so that the subject evaluation value decreases as the distance from the center of the screen increases. In FIG. 4, when the distance from the screen center exceeds R, the subject evaluation value becomes zero.
A method of obtaining the area S in the area by counting the total number of blocks in the same group area and adding a high evaluation value (subject area evaluation value) according to the area of the area. For example, as shown in FIG. 5A, a higher subject evaluation value is given as the area increases. In the case of an image as shown in FIG. 5B, since the area of the subject 200 is larger than the area of the subject 201, a higher subject evaluation value is given to the subject 200.
A method of obtaining saturation in each region and adding a high evaluation value (subject saturation evaluation value) according to the saturation.
A method of calculating the subject evaluation value P by weighted addition of at least one of the plurality of evaluation values, for example, using the following formula.
p = w ₁ × p ₁ + w ₂ × p ₂ + w ₃ × p ₃
w ₁ , w ₂ , w ₃ : weight counting,
p ₁ : center distance evaluation value,
p ₂ : subject area evaluation value,
p ₃ : Subject saturation evaluation value

  Finally, in step S107 of FIG. 2, the system control circuit 60 determines the main subject area based on the subject evaluation value P calculated in step S106, and extracts it from the image. The region determined as the main subject region may be, for example, a region having the highest evaluation value calculated in step S106, or may be a region combining a plurality of regions whose evaluation values are equal to or greater than a predetermined threshold value.

  Subsequently, a process for transmitting information on the main subject area detected by the main subject detection process described above to the subject tracking process will be described with reference to FIGS.

  FIG. 6 is a flowchart for explaining the main subject detection processing to subject tracking processing according to the present embodiment. The system control circuit 60 performs the above-described main subject detection process, position information update process, and tracking process described later. The position information update process (S302) controls the update of subject information to be passed to the subject tracking process (S303) based on the main subject information detected by the main subject detection process (S301). For example, when subject tracking is not performed and a subject (main subject) having a sufficiently high subject evaluation value P described above is detected, position information update processing is performed at a specific position within the main subject region in the frame. Is acquired as the main subject position, and this is transmitted to the subject tracking process. For example, the position information update unit transmits position information (x, y) of the center or the center of gravity of the main subject area to the subject tracking process as the main subject position. Note that the main subject area used for acquiring a specific position may be, for example, a detected subject area or a rectangle that includes the detected subject area. In the subject tracking process, based on the received position information (main subject position), the tracking frame optimum for subject tracking is reset according to the subject tracking unit algorithm regardless of the main subject area detected by the main subject detection process, Perform tracking. If there is a main subject that is already being tracked, the position information update process determines whether information should be updated based on the subject evaluation value and position information of the subject newly detected by the main subject detection process.

  Hereinafter, an example of a determination criterion in the position information update process will be described with reference to FIG.

The position information update process (S302) provides the position to the subject tracking process according to the detection of the main subject as described above when the subject tracking is not performed. When a subject with subject tracking processing is being tracked, the main subject position is notified to the subject tracking processing under the following conditions to change the tracking target. For example, as shown in FIG. 7A, it is assumed that Object1 and Object2 are present in the screen, and that Object1 is currently determined as the main subject and is being tracked by subject tracking processing. In this state, when any of the following conditions is satisfied, the position information update unit transmits the position information (x2, y2) of Object2 as a new tracking target to the subject tracking unit.
・ When Object1 is out of frame as shown in FIG. The detection of frame-out can be obtained from the tracking result of the subject tracking process.
As shown in FIG. 7C, when Object1 moves to the edge of the screen, Object2 exists at a position close to the center of the screen, and the object evaluation value P2 of Object2 exceeds a predetermined threshold value P_Th (P2> P_Th). (When the tracking position is far from the center of the screen and object2 exists in the center of the screen)
When the evaluation value P2 of Object2 is higher than the evaluation value P1 of Object1 (P1 <P2). However, in this case, the evaluation is performed by increasing the weight at the Object1 position (tracking position) during tracking.

  Next, the subject tracking process will be described in detail below. When the position information of the main subject is updated by the above-described position information update process (S302), the subject tracking process (S303) tracks the subject according to the subject tracking algorithm and continues to capture the moving object. FIG. 8 is a flowchart showing the subject tracking algorithm according to this embodiment. FIG. 9 shows an input image including the main subject and an image after the block division as an example of the subject tracking scene.

  The subject tracking algorithm according to the present embodiment can be roughly classified into a template setting unit that determines a template (feature pixel) and a tracking unit that detects an image pattern similar to the template from the target image. Note that subject tracking in the present embodiment tracks a subject by template matching based on color information, and appropriately selects a template that is resistant to minute color changes and shape changes of the subject while suppressing erroneous tracking. Is important in improving the tracking performance.

  The processing flow of the template setting unit described above will be described in detail below. The template setting unit sets a reference range for generating a template used in the tracking process regardless of the main subject area detected by the main subject detection process based on the main subject position acquired by the main subject detection process. To do. Then, the template setting unit generates a template for tracking the main subject position based on the image information of the reference range. In this embodiment, the frame is divided into a plurality of blocks, and a template is generated by setting a range of a predetermined number of blocks including a block including the main subject position and the surrounding blocks from the plurality of blocks as a reference range. Do.

  First, in step S401, the system control circuit 60 divides the image into a plurality of blocks, calculates the pixel information average value of the pixels in each divided block, and sets it as the representative value of each block. Through the above operation, the input image 401 as shown in FIG. 9A is converted into an image 403 with a coarse resolution as shown in FIG. 9B. Note that the block division (S401) of the subject tracking process and the block division (S101) of the main subject detection process may or may not be the same.

  Here, it is assumed that the position of the subject (main subject position) detected by the above-described main subject detection process is given by coordinates 402. In step S402, based on the coordinates 402, a block 404 including subject position coordinates on the image 403 and an area 405 including eight blocks in the vicinity thereof are set as feature candidate colors (reference ranges) to be tracked. As illustrated in FIG. 10, the area 405 is a 9-block area including blocks A to I (the block 404 corresponds to the block E at the center of the area 405). Then, an operation for extracting a pixel that can be a feature to be tracked from these nine feature candidate blocks is performed. In the present embodiment, as an example, the entire screen is divided into 32 × 24 blocks as shown in FIG. From these nine feature candidate blocks, a block that can be a feature to be tracked, that is, a block having specificity with respect to the entire image is extracted. First, it is investigated where and how much blocks similar to each block of feature candidates are distributed in the screen. In the present embodiment, whether or not the block is similar to the feature candidate block is determined, for example, by determining whether or not the block is the same color as the feature candidate block.

FIG. 11 shows a distribution image as a result of extracting similar colors of the feature candidate pixels A to I from the screen. 11 indicates that blocks indicated by shading in FIG. 11 have similar colors to the candidate blocks. Here, as shown in the following (Equation 1), all of the difference sums of the R, G, and B components of the feature candidate block are within a predetermined signal level ThRGB, so that they are similar colors. Judging. The difference sum of the R, G, B components of the feature candidate block is set to a value of 0-255.
However, ThRGB = 30, n = feature color (A ... I), x = horizontal coordinates (0 ... 31), y = vertical coordinates (0 ... 23).

  When trying to track the ball in the example of the scene shown in FIG. 9, it can be seen from the distribution shown in FIG. 11 that the blocks B, C, and E have a small distribution of similar colors in the periphery, and the specificity is large. It is speculated that it can be a color that characterizes the ball. On the other hand, in block G and the like, it can be seen that a lot of similar color pixels are distributed in the vicinity, and the color of the floor surface is likely to be extracted. Therefore, the block G is excluded from the feature candidates. Here, a block in which the ratio of the number of similar color pixels extracted from the entire screen to the number of similar color pixels extracted from the feature candidate pixels A to I is not less than a threshold is excluded from the feature candidates. Thus, after excluding blocks that are not suitable as tracking targets from the nine feature candidate blocks, the finally remaining block (s) is used as a template unique to the subject. In other words, the block to be used for tracking is selected based on the specificity of each of the predetermined number of blocks as the reference range in the frame of the image information.

In addition to the above criteria,
Exclude feature candidates whose area of similar color pixels is too large (FIG. 11A),
A feature candidate having a part of a similar color pixel on the screen edge is determined to be highly likely to be a background image, and is excluded from the candidate (FIG. 11 (I)).
Feature candidates in which similar colors are widely distributed in the screen are likely to be erroneously tracked by mistake, and are excluded from the candidates (FIG. 11D).
The feature candidate blocks may be extracted by combining at least one of the above criteria. With the feature block extracted by the above method as the center, the 5 blocks including the upper (U), lower (D), left (L), and right (R) blocks as shown in FIG. The tracking of the subject is realized by the template matching set as). Templates are generated by the number of blocks extracted from the candidate blocks as being suitable for tracking. If a plurality of feature pixels remain, the most appropriate template is selected using information on the distribution, area, and saturation of similar colors. Further, in the tracking unit described later, pattern matching may be performed for each template using a plurality of templates at the time of the first tracking, and the subject position supplement accuracy may be improved.

  Next, the operation of the tracking unit in the subject tracking process will be described in detail. The basic tracking concept is to search for a pixel similar to the feature color RGB stored as a feature block (template) in the template setting unit from the target screen (in the subsequent frame), and search for a pixel position with a high similarity Position. Also, if another tracking target with a color similar to the tracking target exists in the screen, or if a color similar to the tracking target exists in the background, the closer the tracking target detection position to the previous tracking target, the more similar The similarity is calculated in consideration of the previous detection position coordinates so that the degree becomes higher. This is to prevent hunting of the tracking target detection result and erroneous tracking to similar color backgrounds.

First, in step S403 in FIG. 8, the system control circuit 60 generates a weight table so that the closer to the previous tracking target detection position, the higher the similarity and the priority tracking position. The weight table uses the previous tracking target detection position as a base point, and the weight amount is set according to the distance from the base point at each position in the screen. The amount of change in the weight according to the distance varies. It shall be adaptively changed according to conditions. FIG. 13 shows an example of a design curve of the weight table corresponding to the distance r obtained by (Equation 2).
(X _t-1 , y _t-1 ): Previous tracking target detection position

  FIG. 14 shows an example of a weight table developed into 32 × 24 blocks based on the design curve of FIG. If a weight table for each distance is set with the previous detection position in FIG. 14 as the center, the weight is 0 for a certain distance or more, so there is no need to perform a substantial search. Thus, depending on the design of the weight table, the possibility of erroneous tracking can be reduced, the substantial search area can be narrowed, and the load of matching calculation can be reduced. However, if the search area is too narrow, problems such as inability to track fast-moving objects occur. Therefore, it is necessary to design with great care.

Next, in step S404 in FIG. 8, the system control circuit 60 divides the processing target image into blocks in the same manner as when generating the template, scans the search area defined in step S403, and has a tracking target having a high similarity to the template. Search for a location. As shown in FIG. 12, the RGB values of the central block (C) and the upper (U), lower (D), left (L), and right (R) blocks of the template are respectively represented by C (Rc, Gc, Bc), U (Ru, Gu, Bu), D (Rd, Gd, Bd), L (Rl, Gl, Bl), R (Rr, Gr, Br). Similarly, the RGB values of the matching target block and its upper, lower, left, and right blocks are set to C ′ (Rc ′, Gc ′, Bc ′), U ′ (Ru ′, Gu ′, Bu ′), D ′ (Rd ′, Gd ′, Bd). '), L' (Rl ', Gl', Bl '), R' (Rr ', Gr', Br '). Then, the evaluation value P1 can be expressed as a difference sum of RGB values from the template in each pixel as shown in the following (Equation 3).

Here, α, β, and γ are weights for each RGB, and by setting α = β = γ, RGB can be handled equivalently. The smaller the evaluation value P1, the smaller the difference between the feature color and the template at the coordinates (x, y), that is, the higher the similarity. Therefore, in this embodiment, the evaluation value P1 obtained by Expression 3 is used as the input value of the conversion table as shown in FIG. 15 to obtain the output evaluation value P2. Next, the system control circuit 60 multiplies the output evaluation value P2 by the weight value w _{(x, y)} given by the weight table obtained in step S403, as shown in the following Equation 4, to thereby display the coordinates (x, y The similarity Ps with the template in) is obtained.

  Since the degree of similarity Ps increases as the degree of coincidence with the template increases, the degree of similarity is obtained for all pixels in the search region, and the coordinate with the highest degree of similarity is determined as the subject position.

  Finally, in step S405, the system control circuit 60 updates various information such as exposure and liquid crystal display based on the tracking result obtained in step S404, and newly sets the upper, lower, left, and right pixels around the new subject coordinates. Save as a template. Similarly, subject tracking can be executed sequentially for each frame based on the updated template, and the subject can be continuously captured. The processing of the tracking unit (steps S403 to S405) described above is performed until new subject information is provided by the position information update processing (S302), or until tracking becomes impossible due to the tracking target being out of frame or the like. Repeated (step S406). When new subject information is provided from the position information update process, the subject tracking process executes the process from the top of the process shown in FIG. 8, that is, the process from the template setting unit.

  As described above, according to the configuration of the above embodiment, the main subject is automatically detected, the subject tracking template is reset based on the position information of the main subject, and the optimal template for subject tracking is used. Subject tracking is performed. With such a configuration, it is possible to continue tracking with high accuracy even when a subject with high user interest is moving.

  Moreover, although preferable embodiment of this invention was described above, this invention is not limited to these embodiment, A various deformation | transformation and change are possible within the range of the summary.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various storage media, and a computer (or CPU, MPU, or the like) of the system or apparatus reads the program. It is a process to be executed.

Claims (13)

Detecting means for detecting a subject area from at least one frame of the moving image;
Setting means for setting an area for extracting image information used in the tracking process from an area set based on the position of the subject area detected by the detecting means, using a method different from the detecting means. When,
Tracking means for executing the tracking process based on the image information of the area set by the setting means ,
The setting means divides one of the frames into a plurality of blocks, sets a predetermined number of blocks including a block including the position of the subject area and the surrounding blocks from the plurality of blocks as a reference range. from blocks included in the scope, the image processing apparatus according to claim you to select one or more blocks, as a region for extracting the image information. The setting means selects from the blocks included in the reference range, the one or more blocks as an area for extracting the image information based on the specificity of each block in the frame. The image processing apparatus according to claim 1 , wherein: It said tracking means, using the template generated based on the image information set by the setting unit area, the image processing apparatus according to claim 1 or 2, characterized in that to perform the tracking process. It said tracking means, using the template generated from a block adjacent to the selected block and the selected block as a region for extracting the image information, 1 to claim, characterized in that to perform the tracking process 4. The image processing apparatus according to any one of items 3 . The image processing apparatus according to claim 3, wherein the tracking unit executes the tracking process by pattern matching using the template. The detection unit recognizes a continuous region of similar colors as the same subject region, and sets one of the recognized subject regions as a main subject region, and the image information based on the position of the main subject region. the image processing apparatus according to any one of claims 1 to 5, characterized in that to set the region for extraction. And a setting unit configured to extract the image information based on a position of a main subject region that is one of the plurality of subject regions when the detection unit detects a plurality of subject regions. It is for setting a region, in the case where the main subject region is switched to another subject, to any one of claims 1 to 6, characterized in that to set a region for extracting the image information The image processing apparatus described. Detecting means for detecting a subject area from at least one frame of the moving image;
Setting means for setting an area for extracting image information used in the tracking process from an area set based on the position of the subject area detected by the detecting means, using a method different from the detecting means. When,
Tracking means for executing the tracking process based on the image information of the area set by the setting means,
The detection unit recognizes a continuous region of similar colors as the same subject region, and sets one of the recognized subject regions as a main subject region, and the image information based on the position of the main subject region. An image processing apparatus characterized in that an area for extraction is set. Detecting means for detecting a subject area from at least one frame of the moving image;
Setting means for setting an area for extracting image information used in the tracking process from an area set based on the position of the subject area detected by the detecting means, using a method different from the detecting means. When,
Tracking means for executing the tracking process based on the image information of the area set by the setting means,
And a setting unit configured to extract the image information based on a position of a main subject region that is one of the plurality of subject regions when the detection unit detects a plurality of subject regions. An image processing apparatus for setting an area, wherein an area for extracting the image information is set when the main subject area is switched to another subject. A control method for an image processing apparatus, comprising:
A detecting step for detecting a subject region from at least one frame of the moving image;
The setting means extracts an area for extracting image information used in the tracking process from an area set based on the position of the subject area output in the detection process, using a method different from the detection process. A setting process to set;
Tracking means, have a, a tracking step of executing the tracking process based on the image information of the set by the setting step region,
In the setting step, one of the frames is divided into a plurality of blocks, and a predetermined number of blocks including a block including the position of the subject area and the surrounding blocks are set as a reference range from the plurality of blocks. A control method for an image processing apparatus , wherein one or a plurality of blocks are selected as an area for extracting the image information from blocks included in a range . A control method for an image processing apparatus, comprising:
A detecting step for detecting a subject region from at least one frame of the moving image;
The setting means extracts an area for extracting image information used in the tracking process from an area set based on the position of the subject area output in the detection process, using a method different from the detection process. A setting process to set;
A tracking unit that performs the tracking process based on the image information of the region set in the setting step;
  In the detecting step, a continuous region of similar colors is recognized as the same subject region, and one of the recognized subject regions is set as a main subject region, and the image information is obtained based on the position of the main subject region. A control method for an image processing apparatus, characterized in that an area for extraction is set.   A control method for an image processing apparatus, comprising:
  A detecting step for detecting a subject region from at least one frame of the moving image;
  The setting means extracts an area for extracting image information used in the tracking process from an area set based on the position of the subject area output in the detection process, using a method different from the detection process. A setting process to set;
  A tracking unit that performs the tracking process based on the image information of the region set in the setting step;
  In the setting step, when a plurality of subject areas are detected by the detection unit, the image information is extracted based on a position of a main subject area which is one of the plurality of subject areas. A method for controlling an image processing apparatus, wherein an area is set, and an area for extracting the image information is set when the main subject area is switched to another subject.   The program for functioning a computer as each means of the image processing apparatus described in any one of Claims 1 thru | or 9.