JP2014165801A - Color tracking device, control method thereof and control program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce storage regions for feature color patterns and further to prevent a matching processing time from being prolonged.SOLUTION: A tracking processing section 120 acquires positional information of a subject in one of images and extracts a first feature pattern of the subject at the position. The tracking processing section discriminates a shape of the subject in accordance with the first feature color pattern and disposes at least one other feature color pattern different from the first feature color pattern in an image different from the one image on the basis of the shape of the subject. While using these feature color patterns, feature color pattern matching processing is performed on the other image and the subject is detected.

Description

  The present invention relates to a color tracking processing device that tracks a subject to be tracked by color tracking processing, a control method thereof, and a control program, and more particularly, to efficiently arrange a plurality of tracking frames on a subject, The present invention relates to a color tracking processing apparatus that uses position information to improve the accuracy of color tracking processing.

  In general, in an imaging apparatus such as a digital camera, in order to automatically adjust exposure and focal length, a specific subject is detected from an image and the subject is tracked. As a method for detecting a specific subject from an image, for example, there is a method for obtaining a movement vector by comparing different images in the time direction. Furthermore, a technique is known in which a subject or a part thereof is converted into a characteristic color pattern, and an image captured at different times is scanned and tracked by characteristic color pattern matching as a detection target.

  On the other hand, in order to avoid a decrease in detection accuracy due to the change in size of the subject due to movement of the subject to be tracked between images taken at different times, the scene is divided into a plurality of regions Thus, there is one that performs photometry (see Patent Document 1). Here, a subject to be tracked in the object scene is set, and an area having photometric data similar to the subject is set as a tracking frame. Then, a region having photometric data similar to the photometric data of the tracking frame is sequentially extracted to detect the subject from the image. At this time, when a desired initial tracking area is set, an appropriate tracking frame is automatically set and tracking is performed.

JP-A-7-110429

  However, with the camera described in Patent Document 1, it is necessary to expand the characteristic color pattern for matching when tracking, even if the subject to be tracked can be automatically tracked according to the photometric data. For this reason, it is necessary to set the storage area for storing the characteristic color pattern to the maximum size that can be followed in the image with respect to the size and shape of the subject that changes over time due to movement or the like.

  Furthermore, in feature color pattern matching, depending on the size of the subject, there are a plurality of overlapping color information, so that not only is the feature color pattern storage area used unnecessarily, but the processing time required for matching processing is also increased. It will be long.

  Accordingly, an object of the present invention is to provide a color tracking processing device, a control method thereof, and a control program that reduce the storage area of the characteristic color pattern and do not increase the matching processing time.

  In order to achieve the above object, a color tracking processing device according to the present invention is a color tracking processing device that tracks a specific subject according to a characteristic color pattern in a plurality of images taken at different times, A position acquisition unit configured to acquire position information indicating a position of the specific subject, and a feature color pattern indicating a characteristic color of the subject at the position of the specific subject indicated by the position information as a first characteristic color pattern Based on the shape of the subject determined by the shape determining means, the shape determining means for determining the shape of the subject according to the first characteristic color pattern, and the first feature At least one other characteristic color pattern different from the color pattern and an arrangement means for arranging the first characteristic color pattern in an image different from one of the images; A feature color pattern matching process is performed on the other image using the feature color pattern and the other feature color pattern, and the similarity between the first feature color pattern and the other feature color pattern in the other image is determined. Matching means for detecting the specific subject in accordance with a position that is equal to or greater than a predetermined threshold and positional relationship information that indicates a position where the first characteristic color pattern and the other characteristic color pattern are arranged. It is characterized by.

  The control method according to the present invention is a control method for a color tracking processing device that tracks a specific subject in accordance with a characteristic color pattern in a plurality of images taken at different times, wherein the specific subject in one of the images A position acquisition step of acquiring position information indicating the position of the object, and an extraction step of extracting a feature color pattern indicating the feature color of the subject at the position of the specific subject indicated by the position information as a first feature color pattern; A shape determining step for determining the shape of the subject according to the first characteristic color pattern; and at least one different from the first characteristic color pattern based on the shape of the subject determined in the shape determining step An arrangement step of arranging another feature color pattern and the first feature color pattern in an image different from one of the images; A characteristic color pattern matching process is performed on the other image using the characteristic color pattern and the other characteristic color pattern, and the similarity between the first characteristic color pattern and the other characteristic color pattern is obtained in the other image. A matching step of detecting the specific subject in accordance with a position that is equal to or greater than a predetermined threshold and positional relationship information indicating a position where the first characteristic color pattern and the other characteristic color pattern are arranged. It is characterized by.

  A control program according to the present invention is a control program used in a color tracking processing device that tracks a specific subject according to a characteristic color pattern in a plurality of images taken at different times, and the computer provided in the color tracking processing device A position acquisition step of acquiring position information indicating the position of the specific subject in one of the images, and a feature color pattern indicating the characteristic color of the subject at the position of the specific subject indicated by the position information. Based on the extraction step of extracting as one feature color pattern, the shape determination step of determining the shape of the subject according to the first feature color pattern, and the shape of the subject determined in the shape determination step, At least one other feature color pattern different from the first feature color pattern and the first feature color pattern An arrangement step of arranging in another image from one of the images, and a feature color pattern matching process for the other image using the first feature color pattern and the other feature color pattern, and the other image The position indicating the position where the similarity between the first characteristic color pattern and the other characteristic color pattern is equal to or greater than a predetermined threshold, and the position where the first characteristic color pattern and the other characteristic color pattern are arranged. And a matching step of detecting the specific subject in accordance with the relationship information.

  According to the present invention, the time required for the matching process can be shortened by reducing the storage area of the characteristic color pattern.

It is a block diagram which shows the structure about an example of an imaging device provided with the color tracking processing apparatus by embodiment of this invention. It is a block diagram for demonstrating operation | movement of the image development part with which the digital signal processing part shown in FIG. 1 was equipped. It is a block diagram for demonstrating operation | movement of the tracking process part with which the digital signal processing part shown in FIG. 1 was equipped. 2 is a flowchart for explaining pattern processing for arranging a plurality of characteristic color patterns in color tracking processing performed at the time of shooting in the imaging apparatus shown in FIG. 1. It is a figure which shows an example of the block integration image produced | generated by the block integration image production | generation process shown in FIG. It is a figure which shows the position of the to-be-tracked subject designated by the user in the block integration image. It is a figure for demonstrating the matching result by the matching process shown in FIG. It is a figure for demonstrating an example of arrangement | positioning of the characteristic color pattern performed by the vector calculating part shown in FIG. It is a figure which expands and shows arrangement | positioning of the characteristic color pattern shown in FIG. It is a figure which shows an example of the state which has arrange | positioned all the characteristic color patterns. It is a figure which shows an example of the data structure of characteristic color pattern information. It is a figure for demonstrating addition of the weight information by the weight information addition part shown in FIG. 4 is a flowchart for explaining color tracking processing performed by the imaging apparatus shown in FIG. 1. It is a figure for demonstrating the characteristic color pattern matching process performed by the matching process part shown in FIG. It is a figure which expands and shows a part of block integration image about the process in the characteristic color pattern matching process. It is a figure which shows an example of one characteristic color pattern. It is a figure which shows the state which extracted the block integral image by 3x3 pixel centering on one coordinate. It is a figure which shows an example of the evaluation result which compared the characteristic color pattern shown in FIG. 16, and the 3 * 3 pixel block integration pixel shown in FIG. It is a figure which shows the evaluation result binarized on the basis of the predetermined threshold value about the pixel except the center pixel of a characteristic color pattern about the evaluation result shown in FIG. FIG. 16 is a diagram illustrating a state in which matching processing is performed while being shifted by one pixel in the horizontal direction from the state illustrated in FIG. 15. FIG. 22 is a diagram illustrating a state in which matching processing is performed while being shifted by one pixel in the horizontal direction from the state illustrated in FIG. 21. It is a figure which shows an example of the state at the time of the to-be-tracked object moving to the direction of a dotted line arrow from the characteristic color pattern information obtained by the pattern arrangement process described in FIG. It is a figure which shows the other example of the state at the time of the to-be-tracked object moved to the direction of a dotted line arrow from the characteristic color pattern information obtained by the pattern arrangement process described in FIG. It is a figure for demonstrating the average vector obtained by weighting about the movement amount of each characteristic color pattern shown in FIG. It is a figure which shows the movement vector which concerns on the characteristic color pattern used as a reference | standard. It is a figure shown in the state which accumulated the movement vector and average vector which concern on a reference | standard feature color pattern.

  Hereinafter, an example of a color tracking processing apparatus according to an embodiment of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration of an example of an imaging apparatus including a color tracking processing apparatus according to an embodiment of the present invention.

  The illustrated imaging apparatus includes an optical system 100 such as a photographing lens unit, and a subject image (optical image) is formed on the imaging element 101 by the optical system 100. The image sensor 101 outputs an electrical signal (analog signal) corresponding to the subject image. An analog signal that is an output of the image sensor 101 is converted into a digital signal (image data) by the A / D converter 102.

  The digital signal processing unit 104 includes a developing unit 119 and a tracking processing unit 120, and the developing unit 119 performs image processing (development processing) on the captured image data to generate development data (image data). Note that the developing unit 119 performs, for example, a Bayer image synchronization process based on the imaging data and performs conversion from RGB to YcrCb to obtain development data. The tracking processing unit 120 will be described later.

  The illustrated operation unit 103 includes a shooting mode changeover switch and a shutter button, and the CPU 108 determines the setting state of the operation unit 103 and sets the shooting mode (for example, the still image shooting mode and the moving image shooting mode). Switch. Further, the CPU 108 switches parameters used in the digital signal processing unit 104 according to the shooting mode.

  An imaging mode setting state (hereinafter referred to as a setting imaging mode) is given from the CPU 108 to the optical axis adjustment control unit 109, and the optical axis adjustment control unit 109 controls the optical axis adjustment unit 105 according to the set imaging mode. For example, the optical axis adjustment control unit 109 controls the optical axis adjustment unit 105 according to the set shooting mode, and adjusts the optical system (100) and the image sensor 101 by the optical axis adjustment unit 105. Thereby, focus adjustment, zoom-in, zoom-out, aperture adjustment, and the like are performed.

  The timing generator 107 determines the drive timing of the image sensor 101 via a driver 106 under the control of the CPU 108. Accordingly, horizontal driving and vertical driving are performed for each pixel of the image sensor 101, and the image sensor 101 outputs an analog signal corresponding to the subject image.

  The development data output from the digital signal processing unit 104 is temporarily recorded in the memory 114 via the data (DATA) bus 112 and the memory control unit 113. The data compression / decompression unit 110 compresses the development data recorded in the memory 114 to generate compressed image data. The data compression / decompression unit 110 records the compressed image data on the recording medium (118) via the DATA bus 112 and the recording control unit 117 under the control of the CPU. The data compression / decompression unit 110 compresses the development data by, for example, JPEG compression processing.

  The digital signal processing unit 104 reads the development data recorded in the memory 114, performs image reduction processing, and records the reduced image data in the memory 114 again. The digital signal processing unit 104 reads the reduced image data from the memory 114 and displays it as an image on the LCD 116 via the LCD (liquid crystal display) control unit 115. Thus, the development data recorded in the memory 114 at the time of shooting can be displayed on the LCD 116 as a live image.

  The LCD 116 may include a touch panel. In this case, although not shown, an operation signal corresponding to the operation of the touch panel is sent to the CPU 108, and the CPU 108 performs control according to the operation signal.

  FIG. 2 is a block diagram for explaining the operation of the developing unit 119 provided in the digital signal processing unit 104 shown in FIG.

  The developing unit 119 includes a sensor correction unit 300 and a development processing unit 301, and imaging data is input from the A / D conversion unit 102 to the sensor correction unit 300. The sensor correction unit 300 corrects chromatic aberration generated in the optical system 100 with respect to imaging data under the control of the CPU 108. Further, the sensor correction unit 300 performs gain variation of the image sensor 101 with respect to the image data, and determines the defective pixel to correct the defective pixel. Then, the sensor correction unit 300 outputs corrected imaging data. The corrected imaging data may be recorded in the memory 114 via the DATA bus 112.

  The corrected imaging data is sent from the sensor correction unit 300 to the development processing unit 301. The development processing unit 301 develops the corrected development data under the control of the CPU 108 to obtain development data. Here, in the development process, for example, a synchronization process and a gamma correction process are performed on the Bayer image to generate development data after YCrCb conversion from RGB (Bayer image). As described above, the development processing unit 301 records the development data in the memory 114 via the DATA bus 112.

  FIG. 3 is a block diagram for explaining the operation of the tracking processing unit 120 provided in the digital signal processing unit 104 shown in FIG.

  The tracking processing unit 120 performs block integration processing that performs horizontal and vertical integration (block integration) of the image every several pixels. Further, the tracking processing unit 120 performs a feature color pattern matching process for performing feature color pattern matching based on the image obtained as a result of block integration and the feature color pattern information indicating the feature color pattern stored in the register 207. .

  In the block integration process, the development data recorded in the memory 114 is read via the DATA bus 112, and the horizontal integration unit 201 performs horizontal integration of the pixel data every several pixels under the control of the CPU 108, and the horizontal integration result is obtained. obtain. Subsequently, the vertical integration unit 202 performs vertical integration every several lines on the horizontal integration result under the control of the CPU 108 to generate block integration image data. The vertical integration unit 201 records block integration image data (hereinafter also simply referred to as a block integration image) in the memory 114 via the DATA bus 112.

  In the feature color pattern matching process, block integration image data is read from the memory 114 via the DATA bus 112, passed through the horizontal integration unit 201 and the vertical integration unit 202, and blocked in the LINE memory 204 in units of LINE. Integral image data is recorded as block integration information.

  Alternatively, the development data may be read from the memory 114 and block integration information may be recorded in the LINE memory 204 in units of LINE while performing block integration by the horizontal integration unit 201 and the vertical integration unit 202. In this way, after acquiring the characteristic color pattern information, it is possible to perform the characteristic color pattern matching process while performing block integration on the development data continuously captured and recording the block integrated image in the memory 114.

  Further, the LINE memory 204 may be shared as a vertical integration memory in the vertical integration unit 202, for example. In the first embodiment, as will be described later, since one type of characteristic color pattern information is 3 × 3 pixel information in the block integration image, the LINE memory 204 is for 3 LINE for matching and for 1 LINE for vertical integration. It is assumed that it is composed of about SRAM.

  The YUV2RGB conversion unit 206 reads block integration information from the LINE memory 204 and converts the YUV data into RGB data.

  Now, if the user touches the touch panel provided on the LCD 116 and designates the subject to be tracked, the feature color pattern extraction unit 209 displays the position information (that is, coordinate information) of the designation operation and the block integrated image. According to the correlation with the coordinate information, 3 × 3 block integration information is extracted as feature color pattern information. Then, the characteristic color pattern extraction unit 209 stores the characteristic color pattern information in the register 207.

  Here, a case where nine types of 3 × 3 feature color pattern information (feature color pattern A to feature color pattern I) are provided will be described. However, each feature color pattern information is NxM (N and M are each 1). It may not be an integer above, and may not be a rectangle. In addition, there may be a plurality of types of characteristic color pattern information. However, the circuit configuration is provided with a finite number of patterns so that the circuit scale does not increase or decrease depending on the shape of the subject.

  The shape determination unit 210 obtains color shape information indicating the color shape of the subject to be tracked using the characteristic color pattern information in the position information designated by the user. In the first embodiment, as will be described later, a plurality of characteristic color pattern information is arranged according to the color shape information.

  Next, the matching processing unit 208 compares the plurality of characteristic color pattern information recorded in the register 207 with the block integration information (here, RGB data) sent from the YUV2RGB conversion unit 206, and matches the matching result as a comparison result. Is output. A matching process using a plurality of characteristic color pattern information will be described later.

  The vector calculation unit 212 obtains the movement amount of the pattern as a vector based on the pattern matching result. Further, the vector calculation unit 212 obtains pattern positional relationship information indicating the positional relationship for a plurality of feature color patterns.

  The weight information addition unit 211 uses the pattern positional relationship information obtained by the vector calculation unit 212 and the matching result obtained by the matching processing unit 208 to add weight information to each feature color pattern. The addition of weight information will be described later.

  Based on the matching result, the CPU 108 obtains movement distance information indicating the movement distance of the subject to be tracked as described later. Further, the CPU 108 controls the optical axis adjustment control unit 109 based on the movement distance information and adjusts the optical system 100 by the optical axis adjustment unit 105. As a result, the focus is adjusted in accordance with the movement of the subject that is the tracking target. Further, the CPU 108 updates the tracking frame display indicating the tracking target displayed on the LCD 116 according to the moving distance information, and clearly indicates the subject that is the tracking target to the user.

  FIG. 4 is a flowchart for explaining pattern processing for arranging a plurality of characteristic color patterns in color tracking processing performed at the time of shooting in the imaging apparatus shown in FIG.

  Here, the color tracking process will be described. However, when shooting without performing the color tracking process, the color tracking process is naturally not performed. In this case, an image captured when the user presses the shooting button is recorded.

  Now, in order to start still image or moving image shooting, when the shooting apparatus is turned on and the shooting mode is set, an image is displayed on the LCD 116 as described above. The CPU 108 determines whether or not the user has selected a subject to be tracked by touching the touch panel provided on the LCD 116 (step S101). If no subject is selected (NO in step S101), CPU 108 waits.

  When the subject is selected (YES in step S101), the CPU 108 develops the image data captured by the developing unit 119 and records the development data (that is, YUV data) in the memory 114 (step S102).

  Subsequently, the tracking processing unit 120 reads the development data recorded in the memory 114 under the control of the CPU 108. Then, the horizontal integration unit 201 and the vertical integration unit 202 generate a block integration image of about n × m (n and m are each an integer of 1 or more) (step S103). Here, n = 32 and m = 24.

  FIG. 5 is a diagram illustrating an example of a block integral image generated by the block integral image generation process illustrated in FIG.

  In FIG. 5, if the upper left corner is the origin (x = 0, y = 0), the block integrated image has 32 pixels in the x-axis direction and 24 pixels in the y-axis direction.

  If the user touches the touch panel provided on the LCD 116 and designates a tracking target subject (specific subject), the feature color pattern extraction unit 209 causes the location information (that is, coordinate information) and block of the designation operation to be performed. 3 × 3 block integration information is extracted as feature color pattern information in accordance with the correlation with the coordinate information of the integrated image. The feature color pattern extraction unit 209 stores the feature color pattern information in the register 207 (step S104).

  Here, for example, if the image size of the development data is about 640 × 480 pixels, the position information of the tracking target specified by the user is 1/20 horizontal and 1/20 vertical block integrated image (32 × 24 pixel) position information.

  FIG. 6 is a diagram showing the position of the tracking target subject specified by the user in the block integration image.

  In FIG. 6, if the position designated by the user is designated coordinates a (17, 14), a block integrated image of 3 × 3 pixels centering on the designated coordinates is defined as a feature color pattern A, and Feature color pattern information indicating the feature color pattern A is recorded in the register 207. In the following, this characteristic color pann information is also given the symbol “A”. This characteristic color pattern A is the first characteristic color pattern.

  Subsequently, the matching processing unit 208 uses the feature color pattern information A to perform a matching process with a block integrated image (here, RGB data) (step S105). Since the matching process performed here is a matching process for obtaining the color outline (color shape) of the subject to be tracked, the matching process is performed on the pixel at the center of the characteristic color pattern A (referred to as a central block integration pixel). And use the result of the matching process.

  FIG. 7 is a diagram for explaining a matching result obtained by the matching process in step S105 shown in FIG.

  Here, when the block integrated image has color information closest to the characteristic color pattern A, the matching result is set to “100”, and the block integrated image is different from the characteristic color pattern A as the value of the matching result decreases. Have

  That is, in step S105, as shown in FIG. 7, the matching extraction unit 208 evaluates the block integration image having color information similar to the central block integration pixel of the characteristic color pattern A, and outputs it as a matching result. .

  Subsequently, the shape determining unit 210 compares a preset threshold value (for example, 60) with the matching result, and examines a pixel whose matching result value is equal to or greater than the threshold value. Then, the shape determination unit 210 extracts the color shape of the subject using pixels whose values are equal to or greater than the threshold value, and obtains color shape information (step S106).

  Next, the vector calculation unit 212 obtains a center position (positional coordinates) for arranging the plurality of characteristic color patterns B to I based on the color shape information (step S107).

  FIG. 8 is a diagram for explaining an example of the arrangement of characteristic color patterns performed by the vector calculation unit 212 shown in FIG. FIG. 9 is an enlarged view showing the arrangement of the characteristic color pattern shown in FIG.

  As shown in the figure, when a plurality of characteristic color patterns are arranged, the characteristic color patterns B to I may be arranged adjacent to the characteristic color pattern A, but the plurality of characteristic color patterns are the same. Since it is a feature color pattern, if the color shape information related to the subject to be tracked occupies a large area with respect to the block integration image, pattern matching is repeated for the same feature color pattern.

  For this reason, in step SS107, pixels that are equal to or greater than a preset threshold value (for example, 60) are obtained in each of the above-described designated coordinates a (17, 14) in the vertical and horizontal directions.

  Referring to FIG. 7, in the upward direction from the designated coordinate a (17, 14), only the pixel at the coordinate (17, 11) having the same horizontal position is equal to or greater than the threshold value. The center position candidate.

  Further, in the pixel on the 11th line, the coordinate (15, 11) is the leftmost pixel indicating the value equal to or greater than the threshold value, and therefore the coordinate (15, 11) is set as the center position candidate. In addition, since the coordinates (19.11) are located on the rightmost side in the pixel indicating the value equal to or greater than the threshold, the coordinates (19, 11) are set as the center position candidate.

  Similarly, in the lower direction than the designated coordinate a, the coordinates (17, 16) that are at the same horizontal position and are at the lowermost side for the pixels that are equal to or greater than the threshold value are the center position candidates. In the pixels on the 16th line, the coordinates (10, 16) are located on the leftmost side and the coordinates (25, 16) are located on the rightmost side in the pixels equal to or greater than the threshold value. Each of (25, 16) is also a center position candidate.

  Further, with respect to the right direction from the designated coordinate a, for the pixels that are equal to or greater than the threshold value, the rightmost coordinate (25, 14) at the same vertical position as the designated coordinate a is set as the center position candidate. Then, in the 25th column located on the rightmost side from the designated coordinate a, the coordinates (25, 14) are located on the uppermost side and the coordinates (25, 16) are located on the lowermost side in pixels that are equal to or greater than the threshold value. Therefore, the coordinates (25, 16) are set as the center position candidate.

  In the left direction from the designated coordinate a, the coordinates (11, 14) located at the same vertical position as the designated coordinate a and the leftmost coordinate (11, 14) are set as candidates for the center position. Then, in the eleventh column located on the leftmost side from the designated coordinate a, the coordinates (10, 16) are located on the lowermost side of the pixels that are equal to or greater than the threshold value, so that the coordinates (10, 16) are the center position candidates. In the illustrated example, there is no center position candidate located above the designated coordinate a in the column located on the leftmost side from the designated coordinate a.

  When overlapping coordinates are removed from the center position candidates obtained in this way, the following eight pixels (block integration pixels) are obtained as center position candidates of the characteristic color pattern.

Coordinates (17, 11): Matching result (70)
Coordinate (15, 11): Matching result (68)
Coordinates (19, 11): Matching result (73)
Coordinates (17, 16): Matching result (91)
Coordinate (10, 16): Matching result (66)
Coordinates (25, 16): Matching result (87)
Coordinates (25, 14): Matching result (80)
Coordinate (11, 14): Matching result (81)
Here, since eight types of feature color patterns B to I can be provided as feature color patterns, all the above center position candidates are set as center position coordinates, and 3 × 3 pixels centering on the pixel of the center position coordinates. A feature color pattern is obtained.

  Information indicating the characteristic color patterns B to I is extracted as characteristic color pattern information from the center position coordinates of the characteristic color pattern thus obtained.

  FIG. 10 is a diagram illustrating a state in which all characteristic color patterns are arranged. FIG. 11 is a diagram illustrating an example of a data configuration of characteristic color pattern information.

  As described above, since each of the characteristic color patterns is a 3 × 3 pixel block integration pixel, each of the characteristic color patterns has R, G, and B values (for example, 8 bits) as color information of the block integration pixel. Data). Therefore, for example, in the characteristic color pattern information A, each of the 8-bit data A0 to A8 has R, G, and B values.

  Here, in order to obtain the center position coordinates of the feature color pattern, similar block integration pixels in the up, down, left, and right directions of the feature color pattern A, and further, the up, down, left, and right directions respectively in the block integration image located on the outermost side in each direction The block integration pixel that is farthest from the center is a candidate for the center position. If the obtained feature color pattern center position coordinates are larger than the number of feature color patterns because the feature color pattern center position coordinates overlap or there is no block integration pixel that satisfies the condition, the matching result And the characteristic color pattern A may be selected according to the distance relationship.

  The feature color pattern may be arranged so that the color shape information of the subject obtained in step S105 can be characteristically captured. For example, the feature color pattern A is located on the outermost side in the vertical and horizontal directions of the feature color pattern A. Obtain four block integration pixels. Then, four block integration pixels may be newly obtained using a similar block integration pixel at a position where interpolation is performed between the four block integration pixels as a center candidate, and eight characteristic color patterns may be obtained.

  Referring to FIG. 4 again, the weight information adding unit 211 adds weight information obtained according to the relative distance between the feature color patterns B to I and the feature color pattern A to the feature color pattern information (step S108). .

  FIG. 12 is a diagram for explaining the addition of weight information by the weight information addition unit 211 shown in FIG.

  In the feature color pattern B, the coordinates (15, 11) are the center position coordinates, and the relative position viewed from the center position coordinates (17, 14) of the feature color pattern A is (−2, −3). Here, the reciprocal of the magnitude of the vector corresponding to the relative position information is used as weight information. With respect to the characteristic color pattern B, the root calculation of the sum of the square of 2 and the square of 3 is about 3.6. Therefore, the reciprocal of this value (3.6) is used as the weight information.

  Since the size of the block integrated image is 32 × 24 pixels, the weight information may be normalized by setting the maximum value of the relative position information that can be taken as 1.

  As a result, the following information is added to the characteristic color pattern information in addition to the R, G, and B 8-bit data.

  In the feature color pattern A, coordinates (17, 14), no matching result, and no weight information.

  In the characteristic color pattern B, coordinates (15, 11), matching result: 68, weight information: 1 / 3.6.

  In the characteristic color pattern C, coordinates (17, 11), matching result: 70, weight information: 1 / 3.0.

  In the characteristic color pattern D, coordinates (19, 11), matching result: 73, weight information: 1 / 3.6.

  In the characteristic color pattern E, coordinates (11, 14), matching result: 81, weight information: 1 / 6.0.

  In the characteristic color pattern F, coordinates (25, 14), matching result: 80, weight information: 1 / 8.0.

  In the characteristic color pattern G, coordinates (10, 16), matching result: 66, weight information: 1 / 7.2.

  In the characteristic color pattern H, coordinates (17, 16), matching result: 91, weight information: 1 / 2.0.

  In the characteristic color pattern I, coordinates (25, 16), matching result: 87, weight information: 1 / 8.2.

  Further, in step S108, the weight information adding unit 211 resembles the central block integration pixel of the feature color pattern A for the feature color patterns B to I in addition to the weight information corresponding to the distance information as the weight information of the feature color pattern. The number of pixels is added as weight information. In particular, regarding the arrangement of the characteristic color pattern, since it is necessary to characteristically capture the color outline (color shape) of the subject, one or more block integration pixels having a high matching result with the characteristic color pattern A are adjacent to each other. It is also possible to preferentially use the feature color pattern center position coordinates.

  As a result, even if there is a block integration pixel similar to the characteristic color pattern A that suddenly occurs in the block integration image, it is possible to follow the color shape of the same subject by setting the weight information low.

  Next, the feature color pattern extraction unit 209 extracts 8-bit data of 3 × 3 pick cells for the feature color patterns B to I from the block integration image, and records them as feature color pattern information in the register 207 (step S109).

  Subsequently, the CPU 108 determines whether or not the user has instructed the end of shooting (step S110). When the end of shooting is instructed (YES in step S110), CPU 108 ends shooting. On the other hand, if the end of shooting is not instructed (NO in step S110), the CPU 108 proceeds to a tracking process flow to be described later.

  In this way, the color shape information is acquired for the tracking target subject specified by the user, and a plurality of characteristic color patterns are arranged according to the color shape of the tracking subject. Thus, it is possible to improve the accuracy of subject tracking. Further, even if the shape and size of the subject to be tracked change between temporally continuous or different frames, tracking can be performed satisfactorily by changing the arrangement of the plurality of characteristic color patterns.

  FIG. 13 is a flowchart for explaining the color tracking processing performed by the imaging apparatus shown in FIG.

  Now, as described with reference to FIG. 4, it is assumed that the characteristic color pattern information relating to the subject to be tracked is recorded. When the tracking process is started, the CPU 108 develops image data (capture data) temporally different from the image data obtained when the characteristic color pattern information is obtained by the development unit 119, and develops the development data (ie, YUV image). It records in the memory 114 (step S201).

  Subsequently, under the control of the CPU 108, the tracking processing unit 120 reads the development data recorded in the memory 114, and generates a block integration image of about n × m pixels in the horizontal integration unit 201 and the vertical integration unit 202 (step S202). ). Here, n = 32 and m = 24. Then, the matching processing unit 208 performs a feature color pattern matching process with the feature color pattern information A to I for the block integrated image converted by the YUV2RGB conversion unit 205 (step S203).

  FIG. 14 is a diagram for explaining the characteristic color pattern matching processing performed by the matching processing unit 208 shown in FIG.

  In the feature color pattern matching process, the matching processing unit 208 first performs a feature color pattern matching process on the feature color pattern A while shifting the feature color pattern to the right from the upper left in the horizontal direction as indicated by a broken line. Then, the matching processing unit 208 performs the feature color pattern matching process on the feature color patterns B to I in the same manner as the feature color pattern A.

  At the time of the feature color pattern matching process, the matching line for nine types of feature color patterns is output for each coordinate using the LINE memory 204.

  Here, in the feature color pattern of 3 × 3 pixels, the center block integrated pixel is positioned at the coordinate (1, 1) and the feature color pattern matching process is started. However, in the feature color pattern of 3 × 3 pixels, the center block The integration pixel may be positioned at the origin, and the block integration pixel having the protruding characteristic color pattern may not be evaluated.

  Here, the output of the evaluation result (matching result) by the characteristic color pattern matching processing will be described.

  FIG. 15 is an enlarged view showing a part of the block integrated image in the process of the feature color pattern matching process. FIG. 16 is a diagram illustrating an example of the characteristic color pattern B.

  Assume that the characteristic color pattern B is the pattern shown in FIG. Then, as shown in FIG. 14, it is assumed that the matching process related to the characteristic color pattern B has reached the position of the coordinates (16, 13) of the block integral image.

  FIG. 17 is a diagram illustrating a state in which the block integration image is extracted with 3 × 3 pixels with the coordinates (16, 13) as the center.

  Here, the values of the R, G, and B color information are compared for the 3 × 3 pixel block integration pixel X and the characteristic color pattern B, and the case where they match is evaluated as 100, and the similarity is evaluated to be low. Decrease the result from 100.

  18 is a diagram illustrating an example of an evaluation result obtained by comparing the characteristic color pattern illustrated in FIG. 16 and the 3 × 3 pixel block integration pixel illustrated in FIG. 17.

  FIG. 19 is a diagram showing an evaluation result obtained by binarizing the evaluation result shown in FIG. 18 with respect to pixels excluding the central pixel of the characteristic color pattern with a predetermined threshold as a reference.

  Assume that the evaluation result shown in FIG. 18 is obtained by the characteristic color pattern matching process. At this time, in order to reduce the amount of information when recording the evaluation result, the matching processing unit 208 binarizes the evaluation result shown in FIG. 18, for example. Here, assuming that the threshold value is 60 and the evaluation result shown in FIG. 18 is equal to or greater than the threshold value for other pixels except the central pixel of the characteristic color pattern, the evaluation result is “1”, and if the evaluation result is less than the threshold value, Binarization is performed with the evaluation result set to “0”.

  FIG. 20 is a diagram illustrating a state in which the matching process is performed while being shifted by one pixel in the horizontal direction from the state illustrated in FIG. 15.

  In the state shown in FIG. 20, the 3 × 3 pixel block integration pixel is in a state that substantially coincides with the feature color pattern B. Therefore, the evaluation result in the center pixel of the feature color pattern has a large value and corresponds to the surrounding pixels. The block integration pixel is also equal to or greater than the threshold. As a result, the block integration pixel at the position has a high similarity to the characteristic color pattern B.

  FIG. 21 is a diagram illustrating a state in which the matching process is performed by shifting one pixel in the horizontal direction from the state illustrated in FIG.

  In the state shown in FIG. 21, although the block integration pixel is similar with respect to the central pixel of the characteristic color pattern B, the evaluation result is high, but the peripheral evaluation result is lower than the example shown in FIG. For this reason, with respect to the characteristic color pattern B, the position shown in FIG. 19 is recorded as a position with high similarity.

  Then, the matching processing unit 208 performs the above-described processing for each of the characteristic color patterns A to I.

  Referring to FIG. 13 again, the matching processing unit 208 coordinates the position with a high similarity according to the evaluation result (that is, the detection result) obtained as a result of performing the pattern feature color pattern matching process on the feature color pattern A. It is determined whether or not (position coordinates) has been obtained (step S204). Here, for example, when the evaluation result related to the central pixel in the characteristic color pattern A is equal to or greater than a predetermined threshold (for example, 60) and the evaluation result related to the surrounding pixels is 6 or more and indicates 1, the matching processing unit 208 determines that a position coordinate having a high degree of similarity has been obtained.

  If it is determined that position coordinates having a high degree of similarity cannot be obtained (NO in step S204), the matching processing unit 208 initializes the characteristic color pattern. Then, the CPU 108 displays a message on the LCD 116 in order to prompt the user to respecify the subject to be tracked (step S217). Next, the process returns to the pattern arrangement process of FIG.

  If the condition in step S204 is not satisfied over several frames between step S204 and step S217, a step for proceeding to step S217 may be inserted.

  If it is determined that position coordinates with a high degree of similarity have been obtained (YES in step S204), the matching processing unit 208 determines whether or not there is an evaluation result with a high degree of similarity for all the feature color patterns B to I ( Step S205). Here, the matching processing unit 208 performs determination in the same manner as in the case of the characteristic color pattern A.

  If there is no evaluation result with high similarity for any one of the feature color patterns B to I (NO in step S205), the matching processing unit 208 initializes the feature color pattern information related to the corresponding feature color pattern (step S205). S215).

  If the condition of step S205 is not satisfied over several frames between step S205 and step S215, a step for proceeding to step S215 may be inserted. In this case, the number of times that the condition is not satisfied in step S205 needs to be stored for each feature color pattern.

  Subsequently, the matching processing unit 208 determines whether or not the number of characteristic color patterns initialized in step S215 is a predetermined number N (N is an integer of 2 or more and 7 or less) (step S216). If the initialized number of characteristic color patterns is equal to or greater than predetermined number N (YES in step S216), the process returns to the pattern arrangement process shown in FIG.

  If there is an evaluation result with a high degree of similarity for all of the feature color patterns B to I (YES in step S205), the matching processing unit 208 has position coordinates having the highest degree of similarity in the block integrated image for the feature color pattern with a high degree of similarity. Is obtained (step S206). If the number of characteristic color patterns initialized in step S216 is less than the predetermined number N (NO in step S216), the process proceeds to step S206.

  Subsequently, based on the position coordinates obtained in step S206, the matching processing unit 208 performs relative processing that indicates the relative position between the position coordinates and the center coordinates of the feature color pattern A as described in FIG. Find location information. Then, the matching processing unit 208 evaluates whether the relative positional relationship is equal to the relative positional information obtained from the positional information when the characteristic color patterns B to I are extracted, and the characteristic color patterns B to I are evaluated. A position evaluation value is obtained for each time (step S207).

  Next, the matching processing unit 208 determines whether or not the degree of coincidence is greater than or equal to a predetermined threshold for each of the feature color patterns B to I according to the position evaluation value (step S208).

  FIG. 22 is a diagram illustrating an example of a state when the subject to be tracked moves in the direction of the dotted arrow from the characteristic color pattern information obtained by the pattern arrangement processing described with reference to FIG.

  In the illustrated example, regarding the block integrated image, the feature color patterns A to I have high positional information similarities at the positions of the movement destinations A ′ to I ′, respectively. Therefore, the characteristic color pattern matching process can be performed while capturing the color outline (color shape) of the subject to be tracked.

  FIG. 23 is a diagram illustrating another example of the state when the subject to be tracked moves in the direction of the dotted arrow from the characteristic color pattern information obtained by the pattern arrangement processing described with reference to FIG.

  In the example shown in FIG. 23, the feature color patterns A to I have high positional information similarity at the positions of the movement destinations A ″ to I ″, respectively. For example, in the movement destination G ″ of the characteristic color pattern G, the degree of coincidence is less than a predetermined threshold in the above-described step S208, and the relative distance does not coincide.

  When determining the degree of coincidence of the relative distances, for example, based on the position information included in the characteristic color pattern information and the position information having the highest similarity obtained in the block integrated image, the characteristic color pattern A Find the vector difference for the relative distance from. The degree of coincidence of the relative distances may be determined based on whether the difference value is equal to or greater than a predetermined threshold value.

  If the degree of matching is less than the predetermined threshold (NO in step S208), matching processing unit 208 adds a flag to the corresponding feature color pattern information (step S219). This flag is for instructing not to use it as a feature color pattern when obtaining the movement amount in the processing of step S209 described later.

  If it is determined that the degree of coincidence is equal to or greater than a predetermined threshold (YES in step S208), vector operation unit 212 has the position information of the feature color pattern obtained by the feature color pattern matching process and the position of the feature color pattern information. The amount of movement of the tracking target subject is determined according to the information (step S209). Note that, after the process of step S219, the process proceeds to step S209.

  For example, regarding the feature color patterns B to I, the movement amount of each feature color pattern is obtained according to the position information obtained by the feature color pattern matching process and the position information included in the feature color pattern information. Subsequently, an average vector of the movement amounts of the characteristic color patterns B to I is obtained based on each weight information.

  FIG. 24 is a diagram for explaining an average vector obtained by weighting the amount of movement of each feature color pattern shown in FIG.

  In FIG. 24, for the feature color pattern G, it is assumed that a flag that is not used as the feature color pattern for obtaining the movement amount in step S219 is added. Accordingly, the average vector T ″ is obtained using the weighting information of the characteristic color pattern for the movement vector, excluding the movement vector G ″ related to the characteristic color pattern G.

  FIG. 25 is a diagram showing a movement vector related to the reference characteristic color pattern A.

  Now, as shown in FIG. 25, it is assumed that high similarity is obtained at a plurality of positions in the same image with respect to the characteristic color pattern A. For example, assume that high similarity is obtained at the positions A ″, A2 ″, and A3 ″. In this case, the average vector T ″ described in FIG. Any one of the obtained positions A ″, A2 ″, and A3 ″ is a destination of the characteristic color pattern A.

  FIG. 26 is a diagram showing a state in which the movement vector and the average vector related to the reference feature color pattern are overlapped.

  In FIG. 26, the movement vector closest to the average vector T ″ among the movement vectors (vector A ″, vector A2 ″, vector A3 ″) and the average vector T ″ relating to the positions A ″, A2 ″, and A3 ″ is the vector. Since A ″, the absolute value of the vector A ″ is the amount of movement of the subject here.

  Subsequently, the CPU 108 controls the optical axis adjustment control unit 109 in accordance with the movement amount obtained in the vector calculation unit 212 (here, the absolute value of the vector A ″), and the optical axis adjustment unit 105 controls the optical system 100. The focus position is driven and changed in accordance with the movement of the subject to be tracked (step S210), and the CPU 108 updates the size and position of the tracking frame displayed on the LCD 116 in accordance with the amount of movement and the color shape of the subject. (Step S211).

  Next, under the control of the CPU 108, the matching processing unit 208 refreshes the characteristic color pattern information (step S212). For example, in step S212, when the aforementioned flag is continuously added over a plurality of frames in the same feature color pattern, the feature color pattern information is updated.

  Next, the CPU 108 determines whether or not the user has instructed the end of shooting (step S213). When the end of shooting is instructed (YES in step S213), CPU 108 ends the tracking process. On the other hand, if the end of shooting is not instructed (NO in step S213), the process returns to step S201.

  In this way, when subject color tracking is performed with one feature color pattern, a situation in which a plurality of pieces of positional information with similar similarity are obtained by the result of feature color pattern matching and the subject to be tracked cannot be tracked accurately is solved. be able to. That is, color tracking can be accurately performed according to the size and shape of the subject using a plurality of characteristic color pattern information arranged according to the color shape of the subject to be tracked.

  As described above, in the embodiment of the present invention, it is possible to arrange a plurality of characteristic color patterns while capturing the color shape of a subject to be tracked whose size and shape change with time. And, when the characteristic color pattern is arranged according to the shape of the subject to be tracked, the weight information is changed according to the surrounding color information, so the characteristic color pattern according to the shape of the subject and the weight information The weight increases.

  Furthermore, since the number of characteristic color patterns is a predetermined number, it is possible to arrange the characteristic color patterns according to the shape of the subject in a state where the characteristic color pattern storage area is constant.

  As is clear from the above description, in the example shown in FIG. 3, the CPU 108 and the feature color pattern extraction unit 209 function as a position acquisition unit and an extraction unit. Further, the CPU 108 and the shape determination unit 210 function as a shape determination unit, and the CPU 108 and the matching processing unit 208 function as an arrangement unit and a matching unit. Further, the CPU 108, the matching processing unit 208, and the weight information adding unit 211 function as a changing unit.

  In the example shown in FIG. 1, at least the CPU 108 and the digital signal processing unit 104 constitute a color tracking processing device.

  As mentioned above, although this invention was demonstrated based on embodiment, this invention is not limited to these embodiment, Various forms of the range which does not deviate from the summary of this invention are also contained in this invention. .

  For example, the function of the above embodiment may be used as a control method, and this control method may be executed by the color tracking processing device. Further, a program having the functions of the above-described embodiment may be used as a control program, and the control program may be executed by a computer included in the color tracking device. The control program is recorded on a computer-readable recording medium, for example.

  Each of the above control method and control program has at least a position acquisition step, an extraction step, a shape determination step, an arrangement step, and a matching step.

  The present invention can also be realized by executing the following processing. That is, software (program) that realizes the functions of the above-described embodiments is supplied to a system or apparatus via a network or various recording media, and the computer (or CPU, MPU, etc.) of the system or apparatus reads the program. To be executed.

101 Image sensor 104 Digital signal processing unit 108 CPU
109 Optical axis adjustment control unit 201 Horizontal integration unit 202 Vertical integration unit 204 LINE memory 205 YUV2RGB conversion unit 206 Matching processing unit 207 Register

Claims (6)

A color tracking processing device that tracks a specific subject according to a characteristic color pattern in a plurality of images taken at different times,
Position acquisition means for acquiring position information indicating the position of the specific subject in one of the images;
Extraction means for extracting, as a first feature color pattern, a feature color pattern indicating the feature color of the subject at the position of the specific subject indicated by the position information;
Shape discriminating means for discriminating the shape of the subject according to the first characteristic color pattern;
Based on the shape of the subject determined by the shape determining means, at least one other characteristic color pattern different from the first characteristic color pattern and the first characteristic color pattern are different from one of the images. Arrangement means for arranging in the image;
Performing a characteristic color pattern matching process on the other image using the first characteristic color pattern and the other characteristic color pattern, and the first characteristic color pattern and the other characteristic color pattern in the other image Matching means for detecting the specific subject in accordance with a position where the similarity is equal to or greater than a predetermined threshold and positional relationship information indicating a position where the first characteristic color pattern and the other characteristic color pattern are arranged ,
A color tracking processing apparatus comprising: Weight information corresponding to a relative distance from the first feature color pattern is added to the other feature color pattern,
The color tracking processing apparatus according to claim 1, wherein the matching unit performs a matching process using the another characteristic color pattern according to the weight information.   The color tracking processing apparatus according to claim 2, further comprising a changing unit that changes the weight information according to the shape of the specific subject determined by the shape determining unit.   4. The color tracking processing apparatus according to claim 3, wherein the changing unit further updates the another characteristic color pattern, the positional relationship information, and the weight information based on a matching processing result by the matching unit. . A control method for a color tracking processing device that tracks a specific subject according to a characteristic color pattern in a plurality of images taken at different times,
A position acquisition step of acquiring position information indicating a position of the specific subject in one of the images;
An extraction step of extracting, as a first feature color pattern, a feature color pattern indicating the feature color of the subject at the position of the specific subject indicated by the position information;
A shape determining step of determining the shape of the subject according to the first characteristic color pattern;
Based on the shape of the subject determined in the shape determination step, at least one other characteristic color pattern different from the first characteristic color pattern and the first characteristic color pattern are different from one of the images. A placement step to place on the image;
Performing a characteristic color pattern matching process on the other image using the first characteristic color pattern and the other characteristic color pattern, and the first characteristic color pattern and the other characteristic color pattern in the other image A matching step of detecting the specific subject according to a position where the similarity is equal to or greater than a predetermined threshold and positional relation information indicating a position where the first characteristic color pattern and the other characteristic color pattern are arranged; ,
A control method characterized by comprising: A control program used in a color tracking processing device that tracks a specific subject according to a characteristic color pattern in a plurality of images taken at different times,
In the computer provided in the color tracking processing device,
A position acquisition step of acquiring position information indicating a position of the specific subject in one of the images;
An extraction step of extracting, as a first feature color pattern, a feature color pattern indicating the feature color of the subject at the position of the specific subject indicated by the position information;
A shape determining step of determining the shape of the subject according to the first characteristic color pattern;
Based on the shape of the subject determined in the shape determination step, at least one other characteristic color pattern different from the first characteristic color pattern and the first characteristic color pattern are different from one of the images. A placement step to place on the image;
Performing a characteristic color pattern matching process on the other image using the first characteristic color pattern and the other characteristic color pattern, and the first characteristic color pattern and the other characteristic color pattern in the other image A matching step of detecting the specific subject according to a position where the similarity is equal to or greater than a predetermined threshold and positional relation information indicating a position where the first characteristic color pattern and the other characteristic color pattern are arranged; ,
A control program characterized by causing