JP5103665B2 - Object tracking device and object tracking method - Google Patents

Description

  The present invention relates to an object tracking device and an object tracking method.

  Object recognition using moving images can be applied to danger prediction in intelligent transport systems (ITS: Intelligent Transport System) and subject tracking by a monitoring camera, and research has been actively conducted in recent years (Non-patent Document 1). ).

  When performing object recognition using an image, preprocessing for extracting an object region from the image is required, and a method based on background difference and optical flow has been proposed. The background subtraction method is widely used because the target area can be extracted relatively easily. However, it is necessary to prepare a background image, and is not suitable for applications other than applications with little environmental change such as surveillance cameras. .

  On the other hand, the method based on the optical flow can extract a plurality of moving objects at the same time, but has a drawback that it requires a large amount of calculation and is vulnerable to illumination change and rotation.

  In view of this, methods for extracting an object by image division and tracking an extraction region by object matching between frames have been proposed (Non-Patent Documents 2 and 3).

This method has a feature that both a stationary object and a moving object can be extracted simultaneously.
J. Shin et al., “Optical flow-based real-time object tracking using non-prior training active feature model,” ELSEVIER Real-Time Imaging, Vol. 11, pp. 204-218, USA, June 2005. T. Morimoto et al., “Object tracking in video pictures based on image segmentation and pattern matching,” ISCAS, pp. 3215-3218, 2005. T. Morimoto et al., “Efficient Video-Picture Segmentation Algorithm for Cell-Network-Based Digital CMOS Implementation,” IEICE Transaction on Information & Systems, vol. E87-D, No.2, pp. 500-503, Feb. 2004.

  However, in the object tracking methods described in Non-Patent Documents 2 and 3, the object region is divided into a plurality of regions by image division, and tracking is performed for each object. There is a problem that it is difficult to track.

  Accordingly, the present invention has been made to solve such a problem, and an object of the present invention is to provide an object tracking device that can easily track a tracking object composed of a plurality of objects.

  Another object of the present invention is to provide an object tracking method that can easily track a tracking object composed of a plurality of objects.

  According to the present invention, the object tracking device is an object tracking device that tracks an object in a moving image composed of a plurality of frame images, and includes an image dividing unit and a tracking unit. The image segmentation unit generates a segmented image region based on the image feature of the object in the image included in each frame constituting the moving image. The tracking unit tracks the object based on the divided image area generated by the image dividing unit. The tracking unit includes a tracking object recognition unit and a grouping unit. The tracking object recognition unit classifies the plurality of divided image areas generated by the image dividing unit into a plurality of still object image areas indicating a stationary object and a plurality of moving object image areas indicating a moving object. The grouping unit groups the moving object image regions classified by the tracking object recognition unit based on a predetermined similarity.

  Preferably, the tracking object recognition unit converts the plurality of divided image regions into the plurality of still object image regions and the plurality of still image regions based on the motion vectors of the divided image regions in the frame to which the divided image region belongs and the frame before or after the frame. The moving object image area is classified.

  Preferably, the grouping unit generates group information for uniquely specifying the grouped moving object images in consecutive frames.

  Further, according to the present invention, an object tracking method is an object tracking method for tracking an object in a moving image composed of a plurality of frame images, wherein the object tracking method includes an image in an image included in each frame constituting the moving image. A first step of generating a divided image region based on an image feature of the object, a plurality of divided image regions generated in the first step, a plurality of still object image regions indicating a stationary object, and a plurality of moving object indicating A second step of classifying the moving object image regions into moving object image regions; and a third step of grouping the moving object image regions classified in the second step based on a predetermined similarity.

  Preferably, in the second step, the plurality of divided image regions include a plurality of still object image regions and a plurality of divided object regions based on motion vectors of the divided image regions in a frame to which the divided image region belongs and a frame before or after the frame. And the moving object image region.

  Preferably, in the third step, group information for uniquely specifying the grouped moving object images is generated in consecutive frames.

  Preferably, a bounding box is used when grouping, and a margin is added to the bounding box.

  In the present invention, an input image consisting of one frame is divided into images of a plurality of objects included in one frame, and among the plurality of objects based on the divided plurality of divided images, Similar objects are grouped into a group and the tracking object is recognized. That is, even when the input image is divided into a plurality of target images included in one frame and the tracking object is subdivided, the grouped tracking objects are recognized.

  Therefore, according to the present invention, it is possible to easily track a tracking object composed of a plurality of objects.

  Embodiments of the present invention will be described in detail with reference to the drawings. In the drawings, the same or corresponding parts are denoted by the same reference numerals and description thereof will not be repeated.

  FIG. 1 is a schematic block diagram showing a configuration of an object tracking apparatus according to an embodiment of the present invention. Referring to FIG. 1, an object tracking device 10 according to an embodiment of the present invention includes an image segmentation unit 1 and a tracking unit 2.

  The image division unit 1 receives an input image of one frame from the outside, and uses the RGB value, hue, and saturation at each pixel of the received input image to input a plurality of input images included in the input image by a method described later. A division process for dividing the image into the images of the object and outputting the divided images to the tracking unit 2 is executed for each frame.

  When the tracking unit 2 receives a plurality of divided images in each frame from the image dividing unit 1, the tracking unit 2 groups the objects constituting each tracking object by a method to be described later based on the received divided images. Each tracked object is tracked by matching the grouped objects between adjacent frames.

  The image division unit 1 includes a pixel value detection circuit 11, a connection weight determination circuit 12, and an image division circuit 13.

  The pixel value detection circuit 11 receives an input image of one frame, detects an R component, a G component, and a B component of each pixel of the received input image, and converts the detected R component, G component, and B component into RGB values. Is output to the connection weight determination circuit 12.

  The connection weight determination circuit 12 receives RGB values from the pixel value detection circuit 1 and determines connection weights between the pixels in a plurality of pixels constituting one frame based on the received RGB values by a method described later. Then, the determined connection weight is output to the image dividing circuit 13.

  The image dividing circuit 13 receives an input image from the outside and receives a connection weight from the connection weight determination circuit 12. Then, the image dividing circuit 13 divides the input image composed of one frame into the images of the respective objects by the region growing type image dividing method using the received connection weight. In this region growing type image dividing method, for example, a leader cell is determined from 3 × 3 pixels, and a region is grown from the determined leader cell region to a surrounding pixel region having a relatively large coupling weight. This is a method of dividing the image by repeating until the leader cell no longer exists. Then, the image dividing circuit 13 outputs the divided plurality of divided images to the tracking unit 2.

  In the embodiment of the present invention, an object means a broad object including a target object and a tracking object.

  FIG. 2 is a schematic diagram showing the configuration of the connection weight determination circuit 12 shown in FIG. Referring to FIG. 2, the coupling weight determination circuit 12 includes a D latch circuit 121, 122, 124, 126, 127, 129, a FIFO circuit 123, 128, an HS conversion circuit 125, selectors 130, 131, Weight calculation units 132-135.

  The D latch circuit 121 receives the RGB value from the pixel value detection circuit 11, latches the received RGB value, and outputs the latched RGB value to the D latch circuit 122 and the weight calculation units 133 and 135.

  The D latch circuit 122 receives the RGB values from the D latch circuit 121, latches the received RGB values, and outputs the latched RGB values to the FIFO circuit 123, the weight calculation unit 132, and the selectors 130 and 131.

  The FIFO circuit 123 receives the RGB value from the D latch circuit 122, holds the received RGB value for one clock, and then outputs the RGB value to the D latch circuit 124, the weight calculation unit 132, and the selectors 130 and 131. To do.

  The D latch circuit 124 latches the RGB value received from the FIFO circuit 123 and outputs the latched RGB value to the weight calculation units 133 and 134.

  The HS conversion circuit 125 receives the RGB values from the pixel value detection circuit 11, and converts the received RGB values into hue and saturation by a method described later. Then, the HS conversion circuit 125 outputs the converted hue and saturation to the D latch circuit 126.

  The D latch circuit 126 receives the hue and saturation from the HS conversion circuit 125, latches the received hue and saturation, and outputs the latched hue and saturation to the D latch circuit 127 and the weight calculation units 133 and 135. To do.

  The D latch circuit 127 receives the hue and saturation from the D latch circuit 126, latches the received hue and saturation, and uses the latched hue and saturation as the FIFO circuit 128, the weight calculation unit 132, and the selector 130, It outputs to 131.

  The FIFO circuit 128 receives the hue and saturation from the D latch circuit 127, holds the received hue and saturation for one clock, and then stores the hue and saturation in the D latch circuit 129, the weight calculation unit 132, and the selection. Output to the units 130 and 131.

  The D latch circuit 129 receives the hue and saturation from the FIFO circuit 128, latches the received hue and saturation, and outputs the latched hue and saturation to the weight calculation units 133 and 134.

  The selector 130 receives a control signal CTL from the outside, receives RGB values from the D latch circuit 122 and the FIFO circuit 123, and receives hue and saturation from the D latch circuit 127 and the FIFO circuit 128. The control signal CTL includes a control signal CTL0 for selecting the RGB value, hue, and saturation inputted to the 0 terminals of the selectors 130, 131, and an RGB value, hue inputted to the 1 terminal of the selectors 130, 131. And a control signal CTL1 for selecting the saturation. Upon receiving the control signal CTL0, the selector 130 selects the RGB value received from the FIFO circuit 123 and the hue and saturation received from the FIFO circuit 128 and outputs them to the weight calculation unit 134, and the control signal CTL1. When received, the RGB value received from the D latch circuit 122 and the hue and saturation received from the D latch circuit 127 are selected and output to the weight calculation unit 134.

  The selector 131 receives a control signal CTL from the outside, receives RGB values from the D latch circuit 122 and the FIFO circuit 123, and receives hue and saturation from the D latch circuit 127 and the FIFO circuit 128. Upon receiving the control signal CTL0, the selector 131 selects the RGB value received from the D latch circuit 122 and the hue and saturation received from the D latch circuit 127, and outputs them to the weight calculation unit 135 for control. When the signal CTL 1 is received, the RGB value received from the FIFO circuit 123 and the hue and saturation received from the FIFO circuit 128 are selected and output to the weight calculation unit 135.

  The weight calculation unit 132 receives RGB values from the D latch circuit 122 and the FIFO circuit 123, and receives hue and saturation from the D latch circuit 127 and the FIFO circuit 128. Then, the weight calculation unit 132 converts the RGB value received from the D latch circuit 122 and the hue and saturation received from the D latch circuit 127 into RGB values, hues and saturations (RGB3, HS3) corresponding to one pixel GE3. The RGB value received from the FIFO circuit 123 and the hue and saturation received from the FIFO circuit 128 are the RGB value, hue, and saturation (RGB2, HS2) corresponding to another pixel GE2. Then, the weight calculation unit 132 performs pixel GE2 and pixel GE3 on the basis of the RGB value, hue and saturation (RGB3, HS3) and the RGB value, hue and saturation (RGB2, HS2) by a method described later. Is calculated, and the calculated connection weight is output to the image dividing circuit 13.

  The weight calculation unit 133 receives RGB values from the D latch circuits 121 and 124 and receives hue and saturation from the D latch circuits 126 and 129. Then, the weight calculation unit 133 converts the RGB value received from the D latch circuit 121 and the hue and saturation received from the D latch circuit 126 into an RGB value, hue and saturation (RGB4, HS4) corresponding to one pixel GE4. Let the RGB value received from the D latch circuit 124 and the hue and saturation received from the D latch circuit 129 be the RGB value, hue and saturation (RGB1, HS1) corresponding to another pixel GE1. Then, the weight calculation unit 133 performs pixel GE1 and pixel GE4 on the basis of the RGB value, hue, and saturation (RGB4, HS4) and the RGB value, hue, and saturation (RGB1, HS1) by a method described later. Is calculated, and the calculated connection weight is output to the image dividing circuit 13.

  The weight calculation unit 134 receives RGB values from the D latch circuit 124, receives hue and saturation from the D latch circuit 129, and receives RGB values, hue and saturation from the selector 130. Then, the weight calculation unit 134 uses the RGB value received from the D latch circuit 124 and the hue and saturation received from the D latch circuit 129 as the RGB value, hue and saturation (RGB1, HS1) corresponding to one pixel GE1. The RGB value, hue, and saturation received from the selector 130 are the RGB value, hue, and saturation (RGB2, HS2) corresponding to another pixel GE2. Then, the weight calculation unit 134 uses the RGB value, hue, and saturation (RGB1, HS1) and the RGB value, hue, and saturation (RGB2, HS2) to perform the pixel GE1 and the pixel GE2 according to a method that will be described later. Is calculated, and the calculated connection weight is output to the image dividing circuit 13.

  The weight calculation unit 135 receives RGB values from the D latch circuit 121, receives hue and saturation from the D latch circuit 126, and receives RGB values, hue and saturation from the selector 131. Then, the weight calculation unit 135 uses the RGB value received from the D latch circuit 121 and the hue and saturation received from the D latch circuit 126 as the RGB value, hue and saturation (RGB4, HS4) corresponding to one pixel GE4. The RGB value, hue, and saturation received from the selector 131 are the RGB value, hue, and saturation (RGB3, HS3) corresponding to another pixel GE3. Then, the weight calculation unit 135 performs pixel GE3 and pixel GE4 on the basis of the RGB value, hue and saturation (RGB4, HS4) and the RGB value, hue and saturation (RGB3, HS3) by a method described later. Is calculated, and the calculated connection weight is output to the image dividing circuit 13.

  When the weight calculation unit 134 receives the RGB value, hue, and saturation (RGB3, HS3) corresponding to the pixel GE3 from the selector 130, the weight calculation unit 134 calculates the connection weight between the pixel GE1 and the pixel GE3, and calculates the calculation. When the weight calculation unit 135 receives the RGB value, hue, and saturation (RGB2, HS2) corresponding to the pixel GE2 from the selector 131, the weight calculation unit 135 outputs the combined weight to the image dividing circuit 13. The connection weight between them is calculated, and the calculated connection weight is output to the image dividing circuit 13.

  FIG. 3 is a conceptual diagram when the connection weight determination circuit 12 shown in FIG. 2 determines connection weights. FIG. 4 is another conceptual diagram when the connection weight determination circuit 12 shown in FIG. 2 determines connection weights.

  The connection weight determination circuit 12 determines connection weights in the pixels GE1 to GE4 arranged in 2 rows and 2 columns in one cycle. When the RGB values RGB1 to RGB4 in the pixels GE1 to GE4 are sequentially input to the connection weight determination circuit 12, the weight calculation unit 132 receives the RGB value RGB2 of the pixel GE2 from the FIFO circuit 123, and the hue of the pixel GE2 from the FIFO circuit 128. And the saturation HS2 and the RGB value RGB3 of the pixel GE3 from the D latch circuit 122 and the hue and saturation HS3 of the pixel GE3 from the D latch circuit 127.

  The weight calculation unit 133 receives the RGB value RGB1 of the pixel GE1 from the D latch circuit 124, receives the hue and saturation HS1 of the pixel GE1 from the D latch circuit 129, and receives the RGB value RGB4 of the pixel GE4 from the D latch circuit 121. In response, the hue and saturation HS4 of the pixel GE4 are received from the D latch circuit 126.

  Further, the weight calculation unit 134 receives the RGB value RGB1 of the pixel GE1 from the D latch circuit 124, receives the hue and saturation HS1 of the pixel GE1 from the D latch circuit 129, and receives the RGB value RGB2 of the pixel GE2 from the selector 130; It receives the hue and saturation HS2 of the pixel GE2.

  Further, the weight calculation unit 135 receives the RGB value RGB4 of the pixel GE4 from the D latch circuit 121, receives the hue and saturation HS4 of the pixel GE4 from the D latch circuit 126, and receives the RGB value RGB3 of the pixel GE3 from the selector 131, It receives the hue and saturation HS3 of the pixel GE3.

  That is, the weight calculation units 132 to 135 synchronize the RGB values, hues and saturations of the pixels GE2 and GE3, the RGB values, hues and saturations of the pixels GE1 and GE4, and the RGB values of the pixels GE1 and GE2, respectively. Receives hue and saturation and RGB values, hue and saturation of pixels GE3 and GE4. Then, the weight calculation unit 132 determines and outputs the coupling weight between the pixel GE2 and the pixel GE3, and the weight calculation unit 133 determines and outputs the coupling weight between the pixel GE1 and the pixel GE4, and the weight calculation unit 134. Determines and outputs the coupling weight between the pixel GE1 and the pixel GE2, and the weight calculation unit 135 determines and outputs the coupling weight between the pixel GE3 and the pixel GE4.

  That is, the weight calculation units 132 to 135 determine and output the four connection weights shown in FIG. 3 in one cycle.

  When the selector 130 selects the RGB value RGB3 and the hue and saturation HS3 of the pixel GE3, and the selector 131 selects the RGB value RGB2 and the hue and saturation HS2 of the pixel GE2, the weight calculation units 132 to 135 determines and outputs the four connection weights shown in FIG. 4 in one cycle.

  As described above, the connection weight determination circuit 12 uses the RGB values, hues, and saturations of the four pixels arranged in 2 rows and 2 columns, and the 4 pixels arranged in 2 rows and 2 columns in one cycle. Determine the four connection weights in between.

  As described above, the connection weight determination circuit 3 determines the connection weight between two pixels using the RGB value, hue, and saturation of each pixel. The reason will be described below.

  In the present invention, not only RGB values but also hue and saturation are used in order to determine the connection weight between pixels, in addition to the RGB color space, hue (H: Hue), saturation (S: Saturation). ) And brightness (V: Value).

  Here, the hue (H) is an element representing the type of color such as R, G, and B. The saturation (S) is an element representing the vividness of the color, and is “0” for an achromatic color (black-gray-white). Further, brightness (V) is an element representing brightness.

  FIG. 5 is a conceptual diagram of the HSV color space. Referring to FIG. 5, the HSV color space has a conical shape. The hue (H) is drawn on an annular CRC at the bottom of the conical shape, and is represented in the range of 0 to 360 degrees. FIG. 5 shows a hue with red as a reference (0 degree), and red (255, 0, 0), green (0, 255, 0), and blue (0, 0, 0) in the RGB color space. 255) corresponds to a hue of 0 degrees, a hue of 120 degrees, and a hue of 240 degrees, respectively.

  Saturation (S) is expressed as the circular radius of the cone and takes a value in the range of 0-1. The lightness (V) is expressed as a distance from the apex of the cone, and takes a value in the range of 0-1.

  As can be seen from FIG. 5, the achromatic color has no hue (H) and no saturation (S).

  The following formula is used for conversion from RGB values to hue (H).

  In Expression (1), MAX represents the maximum value of the R value, G value, and B value, and MIN represents the minimum value of the R value, G value, and B value.

  When MAX = MIN, the hue (H) is not defined. This is when the R value, the G value, and the B value are all equal, and because it represents an achromatic color and is around the gray straight line at the center of the cone, the Hue value at that time has no meaning. .

  The following formula is used for conversion from RGB values to saturation (S).

  Note that when MAX = 0, the saturation (S) is undefined. This is because the color when MAX = 0 represents perfect black, so there is no hue or saturation.

  The following equation is used for conversion from RGB values to lightness (V).

  Subsequently, image evaluation based on hue (Hue) and saturation (Saturation) will be described. FIG. 6 is a diagram illustrating an image used for image evaluation based on hue and saturation. FIG. 7 is a diagram showing another image used for image evaluation based on hue and saturation.

  FIG. 6 shows an image including an area with low contrast, and FIG. 7 shows an image including disturbance reflection.

  FIG. 8 is a diagram showing a result of dividing the region X shown in FIG. 6 and a result of dividing the region Y shown in FIG. FIG. 9 is a diagram for explaining a problem in the conventional image division.

In the conventional image division, the connection weight W _{ij; kl} between two pixels is calculated by the following equations (4) and (5) using only the RGB value of each pixel.

That is, in the conventional image segmentation method, a similarity between pixels defined as a connection weight is obtained from a difference in RGB values between adjacent pixels, and a region is grown using the obtained similarity as an index. The connection weight W _{ij; kl} is obtained by substituting the three connection weights calculated for red, green, and blue using the equation (4) into the equation (5) to obtain the three weights for red, green, and blue. Among the connection weights, the smallest one is determined as a connection weight W _{ij; kl} between pixels.

  Therefore, as shown in FIG. 9, for example, the connection weights of (R, G, B) = (250, 0, 0) and (R, G, B) = (125, 0, 0) R, G, B) = (250, 0, 0) and (R, G, B) = (125, 125, 125) are connected to each other with an achromatic color and a chromatic color, The combination weights using the RGB color space do not consider the color of the pixel.

  FIG. 9A is a graph showing an area having gradation change in units of pixels and a change in the combination weight of the gradation area when the horizontal axis is the x-axis and the vertical axis is the combination weight. FIG.

  In an area where there is a gradation change, since the RGB value difference between pixels is equal, the connection weights are also equal. If the connection weight of neighboring pixels is equal to or greater than the threshold value φz used for region growth, the region continues to expand in the direction of the arrow.

  However, with respect to the problem relating to the gradation shown in FIG. 9A, how much is treated as the same region is highly dependent on the application, and it is difficult to uniquely determine the threshold value.

  The sample image shown in FIG. 6 has a low contrast, a shadow is generated at the boundary due to the overlap between the desk and the human, a clear edge does not appear between the desk area and the person area, and the color smoothly changes between the two areas. This is a gradation image. For this reason, it is difficult to divide a desk and a person using the conventional image division method using the connection weights in the RGB color space (see FIG. 8).

  Further, in the sample image shown in FIG. 7, the color changes due to reflection, and the body region does not progress, but is divided into a plurality of regions or remains as undivided regions (see FIG. 8).

  From these results, in order to improve the image division accuracy, it is necessary to consider shades such as black, red, and blue in addition to the method using the brightness by the RGB color space. In consideration of the hue, in the sample image shown in FIG. 6, the person is black (achromatic color), the desk is brown, and the two regions can be separated. In the case of the sample image shown in FIG. 7, the car can be extracted as a red (chromatic) region.

  Therefore, image evaluation based on hue (Hue) and saturation (Saturation) will be described.

  First, image evaluation based on hue (Hue) will be described. FIG. 10 is a diagram in which a value (0 to 360 degrees) converted from the RGB value of the sample image shown in FIG. 6 to the Hue value using Expression (1) is normalized to 256 gradations to obtain an 8-bit gray image. . FIG. 11 is a diagram showing an 8-bit gray image obtained by normalizing values (0 to 360 degrees) converted from RGB values of the sample image shown in FIG. 7 into Hue values using Equation (1) to 256 gradations. It is.

  In FIGS. 10 and 11, in the case of the two sample images shown in FIGS. 6 and 7, there is no pixel value with a high Hue value such as green and blue, so the achromatic color is set to the maximum value 255 (white). Represents.

  10 and 11, it can be seen that the boundary between the person and the desk and the boundary between the road and the vehicle can be clearly distinguished, but noise pixels are conspicuous in the background and the human area. Assuming that this Hue image is applied to image division, it is expected that noise pixels will affect the division accuracy.

Therefore, the following method was adopted to reduce noise pixels. Strictly speaking, as can be seen from equation (1), only when MAX-MIN = 0, it is treated as an achromatic color. However, when humans actually recognize a color visually, MAX-MIN = 0 is accurately set. Even if not satisfied, the pixel is recognized as an achromatic color. Therefore, the parameter D _MAX-MIN is set in order to remove noise pixels.

D _MAX-MIN in FIGS. 10 and 11 represents D _MAX-MIN = MAX-MIN, and a pixel having a smaller error than D _MAX-MIN is regarded as an achromatic color (white-gray-black). The results of evaluating the influence on the image when the D _MAX-MIN value is increased by “5” from “0” are the results shown in FIGS.

It was found that as the D _MAX-MIN value is increased, the achromatic color is removed, and the color area (chromatic color area) and the gray area (achromatic color area) can be separated to some extent. This is because the robustness in distinguishing between the gray area and the color area is improved. Also, the color areas can be identified by the difference in Hue values.

  Next, image evaluation based on saturation will be described. FIG. 12 is a diagram in which a value (range 0 to 1) obtained by converting the RGB value of the sample image shown in FIG. is there. FIG. 13 shows an 8-bit gray image obtained by normalizing the value (range 0 to 1) obtained by converting the RGB value of the sample image shown in FIG. 7 into the S value using Expression (2) to 256 gradations. FIG.

  In the case of saturation, since it is treated as an achromatic color only when MAX-MIN = 0 or MAX = 0, the road area has saturation as well.

  Therefore, as can be seen from FIG. 5, attention is paid to the fact that it is very difficult to identify the color by the hue (H) in the region where the saturation (S) is low, and the saturation (S) value of each pixel is used as a reference. Assuming that the achromatic color region and the chromatic color region are binarized.

  12 and 13 show the influence on the image when the saturation (S) as the threshold value is changed from 0 to 0.2 in increments of 0.04. The images shown in FIGS. 12 and 13 are binary with a pixel having a saturation (S) smaller than the threshold value as white (255) and a pixel having a saturation greater than the threshold value as black (0). This is a converted image.

  With reference to FIG. 12 and FIG. 13, by treating a pixel with a saturation (S) value smaller than 0.2 as an achromatic color and a pixel with a saturation (S) value of 0.2 or more as a chromatic color, Good results were obtained regarding the discrimination between achromatic and chromatic areas. Other sample images were also evaluated, but good results were obtained when the saturation threshold was set near 0.2.

  As described above, as a result of examining the HSV color space, it is difficult to divide the image in the RGB color space (for example, separation of a person and a desk shown in FIG. 6 and extraction of a car body part shown in FIG. 7). On the other hand, by using the HSV color space, a chromatic color region and a color close to an achromatic color as seen from the human eye are separated, or a difference in brightness and saturation occurs depending on the degree of light even though the hue is the same. It was found that the area can be divided as the same area.

  However, the HSV color space is not suitable for image division of an achromatic region. This is because, as can be seen from FIG. 5, when the saturation is low, the hue can hardly be identified, and the Hue value has no meaning.

  Therefore, in the present invention, the color weights between the pixels are determined using both RGB value and HSV value color spaces, and the color regions that each color space is not good at complement each other. is there.

  When the connection weight determination circuit 12 determines the connection weight using the RGB value, hue, and saturation, the HS conversion circuit 125 generally uses the expression (1) and the expression (2) to calculate RGB. Convert values to Hue (H) and Saturation (S).

  However, when converting RGB values into hue (H) and saturation (S) using the equations (1) and (2), it is necessary to configure the HS conversion circuit 125 using a divider. Since the method of realizing the HS conversion circuit 125 using the method requires a plurality of cycles for the calculation, it cannot satisfy the processing requirement in one cycle. There is also a problem that the circuit area increases.

  Therefore, a method using table mapping can be considered. In other words, the H value and S value calculated in advance using the equations (1) and (2) are stored in the memory, and the calculation result is read out from the address corresponding to each RGB value. Reduce processing.

However, since each of R, G, and B is 8-bit data, a large-capacity memory of ((2 ⁸ ) ³ × 9) bits is necessary, and the circuit area of the decoder is increased. .

  Therefore, in the present invention, preferably, the calculation result is limited to the number of bins by dividing H (0 to 360 degrees) and S (0 to 1) into a plurality of bins (regions), and then the table is displayed. The hue (H) and the saturation (S) are obtained by referring to them. According to this method, it is possible to realize a conversion circuit from RGB values to HSV values with a reduced circuit area without using any divider.

  FIG. 14 is a configuration diagram of a hue (Hue) bin. Referring to FIG. 14, the hue (Hue) bin configuration includes a three-stage configuration, and a 360-degree Hue value is configured by 90 bins.

  A method for determining the Hue value will be described. First, in the first stage (innermost circle), it is determined based on MAX which category the pixel of interest belongs to red (01), green (10), and blue (11).

  Next, MIN is determined, and whether the fractional part of the expression corresponding to the condition of MAX is positive or negative is determined from Expression (1). When the fractional part is negative, it becomes “0”, and when the fractional part is positive, it becomes “1” (second stage).

  Finally, if it is determined which of the 15 bins corresponds to which bin of the Hue value composed of 90 bins, the pixel of interest corresponds to the third bin.

  In the above-described Hue value determination method, it can be easily determined from the magnitude relationship of the R, G, and B values up to the second stage. Therefore, a method for determining the first to 15th bins will be described.

  The fractional part of equation (1) is taken out and the following relational expression is created.

  In Expression (6), MED represents any value of R, G, and B other than MAX and MIN, that is, an intermediate value.

  When formula (6) is transformed, the following formula is obtained.

  Then, the bin number is determined from the formation result (1,0) of the 14 inequalities generated when the value of X in Expression (7) is changed from 1 to 14.

  FIG. 15 is a schematic diagram showing the configuration of the hue conversion circuit included in the HS conversion circuit 125 shown in FIG. The HS conversion circuit 125 includes a hue conversion circuit 1250 shown in FIG. Referring to FIG. 15, hue conversion circuit 1250 includes a determination circuit 1251, data generation circuits 1252 and 1253, a switch 1254, comparison circuits 1255 to 1268, and an encoder 1269.

  The determination circuit 1251 receives the RGB value of each pixel from the pixel value detection circuit 11, and determines the maximum value MAX, the minimum value MIN, and the intermediate value MED of the received RGB value. Then, the determination circuit 1251 outputs the determined maximum value MAX to the data generation circuit 1252, outputs the intermediate value MED to the data generation circuit 1253, and outputs the minimum value MIN to the data generation circuits 1252 and 1253.

  The data generation circuit 1252 receives the maximum value MAX and the minimum value MIN from the determination circuit 1251, and, based on the received maximum value MAX and minimum value MIN, 14 X × (MAX -MIN) is sequentially calculated, and the 14 calculated X * (MAX-MIN) (X = 1 to 14) are output to the switch 1254. That is, the data generation circuit 1252 calculates B on the right side of Expression (7) and outputs the calculated B to the switch 1254.

  In addition, the data generation circuit 1253 receives the intermediate value MED and the minimum value MIN from the determination circuit 1251, calculates 15 × (MED−MIN) based on the received intermediate value MED and minimum value MIN, and calculates 15 × (MED-MIN) is output to the switch 1254. That is, the data generation circuit 1253 calculates A on the left side of Expression (7), and outputs the calculated A to the switch 1254.

  The switch 1254 receives 14 X × (MAX-MIN) (X = 1 to 14) from the data generation circuit 1252 and 15 × (MED-MIN) from the data generation circuit 1253. The switch 1254 outputs 1 × (MAX-MIN) and 15 × (MED-MIN) to the comparison circuit 1255, and outputs 2 × (MAX-MIN) and 15 × (MED-MIN) to the comparison circuit 1256. Thereafter, 14 × (MAX-MIN) and 15 × (MED-MIN) are output to the comparison circuit 1268 in the same manner.

  Comparison circuit 1255 receives 1 × (MAX-MIN) and 15 × (MED-MIN) from switch 1254 and compares 1 × (MAX-MIN) with 15 × (MED-MIN). Then, the comparison circuit 1255 outputs the comparison result to the encoder 1269. More specifically, the comparison circuit 1255 outputs “1” to the encoder 1269 when 15 × (MED−MIN) <1 × (MAX−MIN) is established, and 15 × (MED−MIN) <1. When x (MAX-MIN) is not established, “0” is output to the encoder 1269.

  Comparison circuit 1256 receives 2 × (MAX-MIN) and 15 × (MED-MIN) from switch 1254 and compares 2 × (MAX-MIN) with 15 × (MED-MIN). Then, the comparison circuit 1256 outputs the comparison result (1 or 0) to the encoder 1269.

  Similarly, the comparison circuit 1268 receives 14 × (MAX-MIN) and 15 × (MED-MIN) from the switch 1254, and compares 14 × (MAX-MIN) with 15 × (MED-MIN). . Then, the comparison circuit 1268 outputs the comparison result (1 or 0) to the encoder 1269.

  Note that the comparison circuits 1255 to 1268 compare X × (MAX−MIN) with 15 × (MED−MIN) in parallel, and output the comparison result to the encoder 1269. Therefore, the encoder 1269 receives 14-bit data indicating the comparison result from the comparison circuits 1255 to 1268.

  The encoder 1269 receives the 14-bit comparison result from the comparison circuits 1255 to 1268, and determines the bin number based on the received 14-bit comparison result. More specifically, when the encoder 1269 receives the comparison result of [01000000000000] from the comparison circuits 1255 to 1268, the encoder 1269 determines the bin number as “2” and the comparison result of [000000000000] from the comparison circuits 1255 to 1268. When received, the bin number is determined to be “10”. The encoder 1269 determines the bin number as “15” when receiving the comparison result of “00000000000000” consisting of all “0” from the comparison circuits 1255 to 1268.

  Then, the encoder 1269 outputs the determined bin number to the D latch circuit 126 as a hue bin number.

  As described above, since the data generation circuit 1252 calculates X × (MAX−MIN) and the data generation circuit 1253 calculates 15 × (MED−MIN), each of the data generation circuits 1252 and 1253 is added. This can be realized by a simple circuit using a shifter and a shifter.

  Next, conversion from RGB values to saturation (S) will be described. FIG. 16 is a block diagram of a chroma (S) bin. Referring to FIG. 16, saturation (S) in the range of 0-0.1 is assigned to 1 bin, and saturation (S) in the range of 0.1-0.15 is assigned to 2 bins. Saturation (S) in the range 0.15 to 0.1625 is assigned to 3 bins, and Saturation (S) in the range 0.1625 to 0.1750 is assigned to 4 bins. Saturation (S) in the range of 0.1750 to 0.1875 is assigned to 5 bins, and saturation (S) in the range of 0.1875 to 0.2 is assigned to 6 bins. Also, saturation (S) in the range of 0.2 to 0.2125 is assigned to 7 bins, saturation (S) in the range of 0.2125 to 0.2250 is assigned to 8 bins, Saturation (S) in the range of 0.2250 to 0.2375 is assigned to 9 bins, and saturation (S) in the range of 0.2375 to 0.25 is assigned to 10 bins. Saturation (S) in the range of 25-0.30 is assigned to 11 bins, and saturation (S) in the range of 0.3-1.0 is assigned to 12 bins.

  As described above, the saturation (S) is assigned to the bins that are subdivided as the saturation boundary is near 0.2.

  The conversion from the RGB value to the saturation (S) uses the same method as the conversion from the RGB value to the hue (H). Saturation (S) needs to be calculated when one of the two pixels belongs to the achromatic color area and the other pixel belongs to the chromatic color area. That is, two pixels are in a relationship with a saturation boundary (S = 0.2) sandwiched therebetween.

  Therefore, as shown in FIG. 16, the bins are divided into fine bins limited to the vicinity of the saturation boundary (S = 0.2). Saturation bins are assigned to 12 bins based on the results of the following two expressions (8) and (9).

  If the denominators of Equation (8) and Equation (9) are paid, Equation (10) and Equation (11) are obtained, respectively.

  FIG. 17 is a schematic diagram illustrating a configuration of a saturation conversion circuit included in the HS conversion circuit 125 illustrated in FIG. The HS conversion circuit 125 includes a saturation conversion circuit 1270 shown in FIG. Referring to FIG. 17, saturation conversion circuit 1270 includes a determination circuit 1271, data generation circuits 1272 to 1275, a switch 1276, comparison circuits 1277 to 1288, and an encoder 1289.

  The determination circuit 1271 receives the RGB value from the pixel value detection circuit 11, and determines the maximum value MAX and the minimum value MIN of the received RGB value. Then, the determination circuit 1271 outputs the determined maximum value MAX to the data generation circuits 1272 to 1275, and outputs the determined minimum value MIN to the data generation circuits 1274 and 1275.

  The data generation circuit 1272 receives the maximum value MAX from the decision circuit 1271, and sequentially calculates Y × MAX in which Y is changed to 2, 3, and 6 using the received maximum value MAX, and the calculated three values Y × MAX (Y = 2, 3, 6) is output to the switch 1276. That is, the data generation circuit 1272 calculates D on the right side of Expression (10) and outputs the calculated D to the switch 1276.

  The data generation circuit 1273 receives the maximum value MAX from the decision circuit 1271, and sequentially calculates Z × MAX in which Z is changed from 12 to 20 using the received maximum value MAX, and the calculated nine Z XMAX (Z = 12 to 20) is output to the switch 1276. That is, the data generation circuit 1253 calculates F on the right side of Expression (11), and outputs the calculated F to the switch 1276.

  The data generation circuit 1274 receives the maximum value MAX and the minimum value MIN from the determination circuit 1271, calculates 20 × (MAX−MIN) using the received maximum value MAX and minimum value MIN, and calculates the calculated 20 × ( MAX-MIN) is output to the switch 1276. That is, the data generation circuit 1274 calculates C on the left side of Expression (10), and outputs the calculated C to the switch 1276.

  The data generation circuit 1275 receives the maximum value MAX and the minimum value MIN from the determination circuit 1271, calculates 80 × (MAX−MIN) using the received maximum value MAX and minimum value MIN, and calculates the calculated 80 × (MAX MAX-MIN) is output to the switch 1276. That is, the data generation circuit 1275 calculates E on the left side of Expression (11), and outputs the calculated E to the switch 1276.

  The switch 1276 receives three Y × MAX (Y = 2, 3, 6) from the data generation circuit 1272 and receives nine Z × MAX (Z = 12 to 20) from the data generation circuit 1273 to generate data. 20 × (MAX−MIN) is received from circuit 1274 and 80 × (MAX−MIN) is received from data generation circuit 1275.

  Then, the switch 1276 outputs 2 × MAX and 20 × (MAX−MIN) to the comparison circuit 1277, outputs 3 × MAX and 20 × (MAX−MIN) to the comparison circuit 1278, and 6 × MAX and 20 ×. (MAX-MIN) is output to the comparison circuit 1279.

  Further, the switch 1276 outputs 12 × MAX and 80 × (MAX−MIN) to the comparison circuit 1280, and outputs 13 × MAX and 80 × (MAX−MIN) to the comparison circuit 1281, and so on. 20 × MAX and 80 × (MAX-MIN) are output to the comparison circuit 1288.

  Comparison circuit 1277 receives 2 × MAX and 20 × (MAX−MIN) from switch 1276, compares 2 × MAX with 20 × (MAX−MIN), and outputs the comparison result to encoder 1289. More specifically, the comparison circuit 1277 outputs “1” to the encoder 1289 when 20 × (MAX−MIN) <2 × MAX is satisfied, and 20 × (MAX−MIN) <2 × MAX is satisfied. When not, “0” is output to the encoder 1289.

  The comparison circuit 1278 receives 3 × MAX and 20 × (MAX−MIN) from the switch 1276, compares 3 × MAX with 20 × (MAX−MIN), and 20 × (MAX−MIN) <3 × MAX. When “1” is satisfied, “1” is output to the encoder 1289, and when 20 × (MAX−MIN) <3 × MAX is not satisfied, “0” is output to the encoder 1289.

  Similarly, the comparison circuit 1279 receives 6 × MAX and 20 × (MAX−MIN) from the switch 1276, compares 6 × MAX with 20 × (MAX−MIN), and 20 × (MAX−MIN) <6. When × MAX is satisfied, “1” is output to the encoder 1289, and when 20 × (MAX−MIN) <6 × MAX is not satisfied, “0” is output to the encoder 1289.

  The comparison circuit 1280 receives 12 × MAX and 80 × (MAX−MIN) from the switch 1276, compares 12 × AMX with 80 × (MAX−MIN), and outputs the comparison result to the encoder 1289. More specifically, the comparison circuit 1280 outputs “1” to the encoder 1289 when 80 × (MAX−MIN) <12 × MAX is established, and 80 × (MAX−MIN) <12 × MAX is established. When not, “0” is output to the encoder 1289.

  The comparison circuit 1281 receives 13 × MAX and 80 × (MAX−MIN) from the switch 1276, compares 13 × AMX with 80 × (MAX−MIN), and 80 × (MAX−MIN) <13 × MAX is established. In this case, “1” is output to the encoder 1289, and when 80 × (MAX−MIN) <13 × MAX is not established, “0” is output to the encoder 1289.

  Similarly, the comparison circuit 1288 receives 20 × MAX and 80 × (MAX−MIN) from the switch 1276, compares 20 × AMX with 80 × (MAX−MIN), and 80 × (MAX−MIN). When <20 × MAX is satisfied, “1” is output to the encoder 1289, and when 80 × (MAX−MIN) <20 × MAX is not satisfied, “0” is output to the encoder 1289.

  Note that the comparison circuits 1277 to 1288 perform the above-described comparison in parallel, and output the comparison result to the encoder 1289 in parallel. Therefore, the encoder 1289 receives 12-bit data from the comparison circuits 1277 to 1288.

  The encoder 1289 receives a comparison result of 12 bits from the comparison circuits 1277 to 1288, and determines a bin number based on the received comparison result of 12 bits. More specifically, when the encoder 1289 receives the comparison result [100000000000000] from the comparison circuits 1277 to 1288, the encoder 1289 determines the bin number as “1” and the comparison result of [000000100000] from the comparison circuits 1277 to 1288. When received, the bin number is determined to be “7”. The encoder 1289 determines the bin number in the same manner when receiving the other 12-bit comparison results.

  The encoder 1289 outputs the determined bin number to the D latch circuit 126 as the S bin number.

  As described above, the data generation circuit 1272 calculates Y × MAX, the data generation circuit 1273 calculates Z × MAX, and the data generation circuit 1274 calculates 20 × (MAX−MIN) to generate data. Since the circuit 1275 calculates 80 × (MAX−MIN), each of the data generation circuits 1272 to 1275 can be realized by a simple circuit using an adder and a shifter.

  A method for determining the connection weight will be described. FIG. 18 is a diagram for explaining a method of determining the connection weight. Referring to FIG. 18, the outer circle represents a hue (Hue), and takes saturation from the center in the circumferential direction.

  Since it has been experimentally found that a threshold value Sth of about 0.2 is suitable for separating a color region and a color close to an achromatic color whose hue can hardly be identified, 0.2 is defined as a saturation boundary. To do. The inner circle represents the boundary where the saturation is 0.2, the region where the saturation is lower than 0.2 is a color almost achromatic, and the region where the saturation is higher than 0.2 is Treat as a chromatic color. Further, x in the figure represents a pixel.

  The saturations S1 and S2 are calculated for the two pixels, respectively, and the calculated two saturations S1 and S2 are compared with a threshold value Sth (= 0.2).

  When both of the two saturations S1 and S2 are equal to or smaller than the threshold value Sth, it is determined that the two pixels belong to the achromatic color region (in the case of FIG. 18A), and based on only the RGB value. Thus, the connection weight between the two pixels is determined using equations (4) and (5).

  Further, when one of the two saturations S1 and S2 is equal to or less than the threshold value Sth, it is determined that one of the two pixels belongs to the achromatic region and the other belongs to the chromatic region ( In the case of (b) in FIG. 18, the connection weight between the two pixels is determined using the RGB value and the saturation (S).

  Further, when both of the two saturations S1 and S2 are larger than the threshold value Sth, it is determined that both the two pixels belong to the chromatic color region (in the case of (c) in FIG. 18), and the RGB value and the hue are determined. (H) is used to determine the connection weight between the two pixels.

  As described above, in the present invention, the threshold value Sth (= 0.2) obtained experimentally is used to determine whether the two pixels belong to the achromatic color region or the chromatic color region. This makes it possible to accurately determine which region the two pixels belong to.

  Further, the threshold value Sth (= 0.2) was experimentally obtained in order to separate a color region from a color close to an achromatic color in which a human can hardly distinguish the hue by visual observation. Along with the identification, it can be accurately determined which region the two pixels belong to.

  In the case shown in FIG. 18 (a), even if the coupling weight is calculated using the hue (H) of the HSV color space, the hue is meaningless in the achromatic color region, and therefore, the improvement of the division accuracy of the region is expected. Can not. Although it is possible to use the lightness component (V) of the HSV color space, since the accuracy is improved by using RGB values by simulation, only the RGB values are used in the region shown in FIG. Determine the connection weight.

  Further, in the case shown in FIG. 18B, the leader cell is found in the person area of the sample image shown in FIG. 6, the area has grown in the desk area, and the area has expanded from the achromatic area to the color area. The division error that occurred. On the contrary, the color region may grow from the achromatic region. As in the sample image, when the color gradually changes due to the influence of the shadow generated by the overlap between the person and the desk and no clear edge appears at the boundary of the region, the division is difficult only with the RGB values.

  In order to solve this problem, a color region (desk region) and an achromatic region (person region) are set with a saturation boundary of S = 0.2 obtained experimentally using saturation (S) as a threshold value. And decided to detect the boundary.

  When two pixels belong to the achromatic color region and the chromatic color region, the boundary between the achromatic color region and the chromatic color region is determined using the saturation (S), and the connection between the two pixels belonging to the boundary surface By setting the weight to be smaller than the value of the combination weight obtained by calculating the RGB value, the combination weight in the direction of suppressing the growth of the region from the chromatic color region to the achromatic color region or from the achromatic color region to the chromatic color region. To decide. Thereby, in the image division, the growth of the region from the chromatic color region to the achromatic color region or from the achromatic color region to the chromatic color region is suppressed, and the image can be accurately divided even for an input image with low contrast.

  Further, in the case shown in FIG. 18C, when the lightness and saturation change due to light reflection or the like in the connection weight between the pixels determined based on the RGB values, the difference in each component between the pixels is as follows. Since it becomes large, the connection weight becomes a small value. Therefore, in the sample image shown in FIG. 7, it is difficult to extract the same region collectively.

  Therefore, paying attention to the fact that the hue itself does not change much even if the brightness and saturation change, the hue determination using the hue (H) is taken in. In determining the connection weight between pixels belonging to the chromatic color region, first, the hue comparison between two pixels is performed, and whether or not the difference in hue is within a certain threshold value, that is, the two pixels have the same hue. Judgment is made as to whether or not the range is acceptable. Then, when it is determined that the hues are the same, when the combination weight determined from the RGB values is small (that is, when the difference is large in the RGB color space due to the influence of reflection or the like), the larger combination is performed in the direction of promoting the region. Give weight. Thereby, the pixels of the chromatic color region belonging to the similar hue are easily grown, and the region where the brightness and saturation change due to reflection or the like can be extracted as one region.

  A specific method for determining the connection weight in each of the cases (a), (b), and (c) of FIG. 18 will be described.

  In the case shown in (a) of FIG. 18, the connection weight is determined using only the RGB values according to the expressions (4) and (5), but the connection weight is calculated using the expressions (4) and (5). When the circuit is implemented as a digital circuit, the divider has a large circuit area and requires a plurality of clock cycles. Therefore, in the present invention, the connection weight is determined by table mapping using a decoder. Table 1 shows a conversion table when determining the connection weight using only RGB values.

The conversion table shown in Table 1 shows the difference between the brightness I _{i, j} and the brightness I _{k, l} calculated using the RGB values of the two pixels of the target pixel (i, j) and the neighboring pixel (k, l). The correspondence relationship between the absolute value | I _{i, j} −I _{k, l} | and the connection weight is shown.

Therefore, in the present invention, in the case shown in FIG. 18A, based on the RGB values of the two pixels (i, j) and (k, l), the two lightness values I _{i, j} , I _{k, l} calculates the two intensity _I i obtained by the calculation, _{j, I k,} the absolute value of the difference between the _{l | I i, j -I k} , l | computes, the calculated absolute value _{| I i, j} - By extracting the connection weight corresponding to I _{k, l} | with reference to Table 1, the connection weight of the two pixels (i, j) and (k, l) is determined.

  Next, in the case shown in (b) of FIG. 18, in order to suppress the area from expanding, a connection weight smaller than the connection weight obtained from the RGB color space using the conversion table of Table 1 is given.

  In this case, when the saturation boundary (S = 0.2) is used as a threshold value and the binary separation is performed between an achromatic color and a chromatic color, the connection weight between pixels near the saturation boundary and the saturation boundary are completely eliminated. The connection weight is determined so that the connection weight between the pixels having completely different saturations is not treated on the same basis.

  FIG. 19 is a diagram illustrating the influence of the reflection on the pixel value. FIG. 20 is a diagram illustrating a change in the achromatic pixel value when the saturation boundary is changed.

  19A shows the input image, and FIG. 19B shows the road image shown in FIG. 19A without the saturation boundary (S = 0.2) as a threshold value. An image binarized into a chromatic color (0) and a chromatic color (1) is shown, and FIG. 19C shows a result of image division.

  As can be seen from (b) of FIG. 19, the road portion of the image uses the experimentally determined saturation boundary (S = 0.2) as a threshold for separating the achromatic region from the chromatic region. The same area is divided into a chromatic color area and an achromatic color area at the center of the image. As a result, as shown in FIG. 19C, the road is suppressed from growing at the saturation boundary, and the accuracy of the divided image is lower than when the RGB color space is used.

  Referring to FIG. 20, a curve k1 indicates a change in the number of achromatic pixels for an image including a color area in the image, and a curve k2 indicates a change in the number of achromatic pixels for a gray image not including the color area in the image. Indicates.

  As can be seen from FIG. 20, in an image including a color region, the number of achromatic pixels is saturated with a saturation boundary near 50 to 60 (S = 0.2). On the other hand, in the case of a gray image, the number of achromatic pixels continues to change linearly.

  From this result, if the connection weight between two pixels across the saturation boundary is drastically lowered to a region where the region does not grow, the achromatic region and the chromatic region cannot be separated well by the saturation boundary. In this case, the division accuracy of the gray area decreases.

  Therefore, for the pixels close to the saturation boundary, the reduction amount of the connection weight is reduced, and the reduction amount of the connection weight is increased as the distance from the saturation boundary is increased. Thereby, the division accuracy of the gray area can be improved.

  Table 2 shows a connection weight conversion table in the case shown in FIG.

As can be seen from Table 2, the absolute value _{| I i, j -I k,} l | is 001x_xxxx, 0000_1xxx, 0000_01xx, 0000_001x, if a 0000_000X, binding weights, chroma difference _{| S i,} j _{-S k, As l} | becomes larger, the amount of reduction from the connection weight (see Table 1) determined using only RGB values is increased.

  Finally, a specific method for determining the connection weight in the case shown in FIG. 18C will be described. It has been found that, in a color image, when there is no influence of reflection or the like, a highly accurate region division result can be obtained by determining the connection weight using the RGB color space. Therefore, it is preferable to optimize the coupling weight using the HSV color space only for the coupling weight between pixels affected by reflection.

  Therefore, it was examined by using a sample image how much each of the R, G, and B components changes between continuous pixels due to the influence of irregular reflection. FIG. 21 is a diagram illustrating the influence of irregular reflection of R, G, and B components.

  21A shows a sample image used for the evaluation, and FIG. 21B shows the R, G, and B values of the pixels on the line segment A-B shown in FIG. FIG. 21C shows changes in the R, G, and B values of the pixels on the line segment CD shown in FIG. 21A. That is, (b) of FIG. 21 shows a change in the pixel value of the vehicle body region affected by light reflection, and (c) of FIG. 21 shows a pixel in the road region not affected by light reflection. Indicates a change in value.

  Referring to FIG. 21, the body region has a larger change in pixel value between consecutive pixels than the road region. Then, it was found from the sample image having the reflection area that the amount of change between the pixels due to the reflection of light is concentrated between 30 and 100. That is, it is determined that an area where the pixel value difference is very small, or conversely, an area where the pixel value difference is very large is not affected by light reflection.

  Therefore, in the present invention, the optimization of the coupling weight using the evaluation result of the hue (H) is limited to 01xxx_xxxx, 001x_xxxx, 0001_xxxx shown in Table 1, thereby maintaining the division accuracy of the color area based on the RGB values. The accuracy can be improved by the hue (H).

  Next, based on the difference between the Hue values of the two pixels, it is examined to what value the coupling weights at 01xxx_xxxx, 001x_xxxx, 0001_xxxx in Table 1 are to be raised. If the hue (H) is too wide, the region will grow to a different color. For example, if red and orange pixel values having similar hues are treated as the same hue, it is difficult to separate the two areas.

  Conversely, if the hue (H) width is too small, the conversion calculation from RGB values to the hue (H) requires division, so that the circuit area increases.

  Therefore, as a result of simulation in consideration of both the mounting area and the division accuracy, it has been found that when the minimum unit of hue (H) is set to 4 degrees, colors having similar hues can be divided.

  FIG. 22 is a diagram showing a histogram of the Hue value of the red car body. FIG. 23 is a diagram showing a histogram of Hue values for the body of a blue car.

  Referring to FIGS. 22 and 23, most pixels are concentrated and extracted within a width of ± 10 degrees around the peak of the histogram. Further, the difference in the Hue value between pixels is very small and is within about 10 degrees in the same region (same color).

Therefore, in consideration of the result of the above-described experiment, 4 degrees or less, 5 degrees or more and 8 degrees or less, and the other three determination conditions with respect to the difference in hue value | H _ij −H _kl | between two pixels As shown in Table 3, the connection weight conversion table in the case shown in FIG.

  Thus, every time the hue difference between two pixels changes by 4 degrees, the coupling weight between the two pixels is changed. And the smaller the hue difference is, the greater the connection weight is given.

  FIG. 24 is a block diagram of the weight calculation unit 132 shown in FIG. Referring to FIG. 24, the weight calculation unit 132 includes a combination weight calculation circuit 1320, a hue difference calculation circuit 1330, a saturation difference calculation circuit 1340, combination weight correction circuits 1350 and 1360, and a selector signal generation circuit 1370. , And selector 1380.

  Based on the two RGB values received from the D latch circuit 122 and the FIFO circuit 123, the connection weight calculation circuit 1320 calculates a connection weight between the two pixels using Table 1, and calculates the calculated connection weight as a connection weight. Output to correction circuits 1350 and 1360 and selector 1380.

  The combination weight calculation circuit 1320 includes absolute value calculators 1321, 1323, 1325, encoders 1322, 1324, 1326, and a minimum value selection circuit 1327.

The absolute value calculator 1321 calculates the lightness values I _A (R) and I _B (R) based on the two R values R _A and R _B received from the D latch circuit 122 and the FIFO circuit 123, and calculates lightness _I a _(R), the absolute value of the difference between _{I B (R) | I a} (R) -I B (R) | computes a. Then, the absolute value calculator 1321 outputs the calculated absolute value | I _A (R) −I _B (R) | to the encoder 1322.

The encoder 1322 holds Table 1, and upon receiving the absolute value | I _A (R) −I _B (R) | from the absolute value calculator 1321, the absolute value | I _A ( R) _-I B (R) | corresponding to the connection weight _{W a,} extract the B (R), and outputs the extracted connection weight _{W a,} B and (R) to the minimum value selection circuit 1327.

The absolute value calculator 1323 calculates the lightness values I _A (G) and I _B (G) based on the two G values G _A and G _B received from the D latch circuit 122 and the FIFO circuit 123, and calculates the calculated values. lightness _I a _(G), the absolute value of the difference between _{I B (G) | I a} (G) -I B (G) | computes a. Then, the absolute value calculator 1323 outputs the calculated absolute value | I _A (G) −I _B (G) | to the encoder 1324.

The encoder 1324 holds Table 1, and upon receiving the absolute value | I _A (G) −I _B (G) | from the absolute value calculator 1323, the absolute value | I _A ( G) _-I B (G) | corresponding to the connection weight _{W a,} extract the B (G), and outputs the extracted connection weight _{W a,} B and (G) to the minimum value selection circuit 1327.

The absolute value calculator 1325 calculates the lightness values I _A (B) and I _B (B) based on the two B values B _A and B _B received from the D latch circuit 122 and the FIFO circuit 123, and calculates the calculated values. lightness _I a _(B), the absolute value of the difference between _{I B (B) | I a} (B) -I B (B) | computes a. Then, the absolute value calculator 1325 outputs the calculated absolute value | I _A (B) −I _B (B) | to the encoder 1326.

The encoder 1326 holds Table 1, and upon receiving the absolute value | I _A (B) −I _B (B) | from the absolute value calculator 1325, the absolute value | I _A ( B) _-I B (B) | corresponding to the connection weight _{W a,} extract the B (B), and outputs the extracted connection weight _{W a,} B and (B) to the minimum value selection circuit 1327.

Minimum value selection circuit 1327 receives connection weights WA _{, B} (R), WA _{, B} (G), WA _{, B} (B) from encoders 1322, 1324, and 1326, respectively, and receives the three combinations received. The minimum connection weight WA _{, B} is selected from the weights WA _{, B} (R), WA _{, B} (G), WA _{, B} (B). Then, the minimum value selection circuit 1327 outputs the selected combination weights WA and _B to the combination weight correction circuits 1350 and 1360 and the selector 1380.

The hue difference calculation circuit 1330 holds the assignment map of the hue bin numbers shown in FIG. 14, and receives the hue bin number _A and the hue bin number _B from the D latch circuit 127 and the FIFO circuit 128, respectively. Then, the hue difference calculation circuit 1330 detects the hue values H _A and H _B corresponding to the received hue bin number _A and hue bin number _B with reference to the assignment diagram of hue bin numbers shown in FIG. Then, the hue difference calculation circuit 1330, the hue difference _| H A -H _B | computes, the calculated hue difference _| outputs to the connection weight correction circuit 1350 | H A -H _B.

The saturation difference calculation circuit 1340 holds the S bin number assignment diagram shown in FIG. 16, and receives the S bin number _A and the S bin number _B from the D latch circuit 127 and the FIFO circuit 128, respectively. Then, the saturation difference calculation circuit 1340 detects the S values S _A and S _B corresponding to the received S bin number _A and S bin number _B with reference to the S bin number assignment diagram shown in FIG. . The detected S values S _A and S _B are values normalized to 256 gradations. Then, chroma difference calculation circuit 1340, a chroma difference _| S A -S _B | computes, the calculated saturation difference _| outputs the to connection weight correction circuit 1360 | S A -S _B.

The combination weight correction circuit 1350 holds Table 3, receives the combination weights WA _{and B} from the combination weight calculation circuit 1320 _, and receives the hue difference | H _A −H _B | from the hue difference calculation circuit 1330. Then, the connection weight correction circuit 1350 refers to Table 3 and extracts an absolute value | I _ij −I _kl | that matches the connection weights WA _{and B.}

After that, when the extracted absolute value | I _ij −I _kl | connection weight _{W a} corresponding to _{0000_000X,} extracting B _H from Table 3, and outputs the extracted connection weight _{W a,} the B _H to the selector 380.

In addition, when the extracted absolute value | I _ij −I _kl | is any one of 01xxx_xxxx, 001x_xxxx, and 0001_xxxx shown in Table 3, the connection weight correction circuit 1350 has a hue difference | H _A −H _B | determining whether a hue difference _| if is ○ or less, hue difference _{| | H a -H B H a} -H B | where is ○ less connection weight _{W a,} B _H (○) Are extracted from Table 3.

More specifically, when the extracted absolute value | I _ij −I _kl | is 01xxx_xxxx shown in Table 3, the coupling weight correction circuit 1350 has a hue difference | H _A −H _B | of “4 degrees” or less. determining whether a hue difference _| H a -H _{B |} if is less than "4 °", the connection weight _{W a} of _16, extracts the B _H from Table 3, binding weights the extracted WA _, B_H (= 16) is output to the selector 380. On the other hand, the connection weight correction circuit 350, the hue difference _| H A -H _{B |} if is not less than "4 °", the connection weight _{W A} consisting of _4, extracts the B _H from Table 3, the extracted connection weight W _A, B_H (= 4) is output to the selector 1380.

Further, when the extracted absolute value | I _ij −I _kl | is 001x_xxxx shown in Table 3, the connection weight correction circuit 1350 determines whether or not the hue difference | H _A −H _B | is “4 degrees” or less. When the hue difference | H _A −H _B | is “4 degrees” or less, a combination weight WA _, B_H consisting of 32 is extracted from Table 3, and the extracted combination weight WA _{, B} _H (= 32) is output to the selector 1380. On the other hand, when the hue difference | H _A −H _B | is not “4 degrees” or less, the connection weight correction circuit 1350 further determines whether or not the hue difference | H _A −H _B | is “8 degrees” or less. If the hue difference | H _A −H _B | is equal to or less than “8 degrees”, the coupling weights WA _, B_H consisting of 16 are extracted from Table 3, and the extracted coupling weights WA _, B_H (= 16) is output to the selector 1380. On the other hand, the connection weight correction circuit 1350, the hue difference _| H A -H _{B |} if is not less than "8 degrees", the connection weight _{W A} of _8, then extracted B _H from Table 3, the extracted connection weight W _A, B_H (= 8) is output to the selector 1380.

Further, when the extracted absolute value | I _ij −I _kl | is 0001_xxxx shown in Table 3, the connection weight correction circuit 1350 determines whether or not the hue difference | H _A −H _B | is “4 degrees” or less. When the hue difference | H _A −H _B | is “4 degrees” or less, a combination weight WA _, B_H consisting of 32 is extracted from Table 3, and the extracted combination weight WA _{, B} _H (= 32) is output to the selector 1380. On the other hand, the connection weight correction circuit 1350, the hue difference _| H A -H _{B |} if is not less than "4 °", the connection weight _{W A} of _16, extracts the B _H from Table 3, the extracted connection weight W _A, B_H (= 16) is output to the selector 1380.

The combination weight correction circuit 1360 holds Table 2 and receives the combination weights WA _{and B} from the combination weight calculation circuit 1320 and the saturation difference | S _A −S _B | from the saturation difference calculation circuit 1340. Then, the connection weight correction circuit 1360 refers to Table 2 and extracts the absolute value | I _ij −I _kl | that matches the connection weights WA _{and B.}

Further, the connection weight correction circuit 1360, a chroma difference _| S A -S _{B |} is equal to or less than "6", the saturation difference _| S A -S _{B |} is less than "6" If connection weight _{W a,} the absolute value matches the _{B |} I ij _{-I kl |} a connection weight _{W a} corresponding to the "if ≦ _6", extracts the B _S from Table 2, binding weights the extracted WA _, B_S is output to the selector 1380.

On the other hand, when the saturation difference | S _A −S _B | is not “6” or less, the connection weight correction circuit 1360 further determines whether or not the saturation difference | S _A −S _B | is “9” or less. If the saturation difference | S _A −S _B | is equal to or smaller than “9”, the absolute value | I _ij −I _kl | corresponding to the coupling weights WA _{and B} corresponds to “else if ≦ 9”. The connection weights WA _, B_S to be extracted are extracted from Table 2, and the extracted connection weights WA _, B_S are output to the selector 1380.

On the other hand, when the saturation difference | S _A −S _B | is not “9” or less, the connection weight correction circuit 1360 further determines whether or not the saturation difference | S _A −S _B | is “12” or less. If the saturation difference | S _A −S _B | is equal to or smaller than “12” _, it corresponds to the absolute value | I _ij −I _kl | that matches the coupling weights WA _{and B} and “else if ≦ 12”. The connection weights WA _, B_S to be extracted are extracted from Table 2, and the extracted connection weights WA _, B_S are output to the selector 1380.

On the other hand, when the saturation difference | S _A −S _B | is not equal to or less than “12”, the connection weight correction circuit 1360 has an absolute value | I _ij −I _kl | that matches the connection weights WA _{and B} , and “else”. connection weight _{W a} which correspond to and _to extract B _S from Table 2, and outputs the extracted connection weight _{W a,} the B _S to the selector 1380.

The selector signal generation circuit 1370 holds the threshold Sth (= 0.2) and the S bin number assignment diagram shown in FIG. 16, and the S bin numbers _A and S from the D latch circuit 127 and the FIFO circuit 128, respectively. Receive bin number _B. Then, selector signal generation circuit 1370 detects S values S _A and S _B corresponding to the received S bin number _A and S bin number _B with reference to the S bin number assignment diagram shown in FIG.

Then, selector signal generation circuit 1370 compares the two detected S values S _A and S _B with threshold value Sth. Then, the selector signal generation circuit 1370 selects the combination weights WA _{and B} received from the combination weight calculation circuit 1320 when both of the two S values S _A and S _B are equal to or less than the threshold value Sth. A signal SEL_RGB is generated, and the generated selector signal SEL_RGB is output to the selector 1380.

The selector signal generation circuit 1370 selects the combination weight WA _, B_S received from the combination weight correction circuit 1360 when either one of the two S values S _A and S _B is equal to or less than the threshold value Sth. Selector signal SEL_S is generated, and the generated selector signal SEL_S is output to selector 1380.

Further, the selector signal generation circuit 1370 selects the combination weights WA _, B_H received from the combination weight correction circuit 1350 when both of the two S values S _A , S _B are larger than the threshold value Sth. A selector signal SEL_H is generated, and the generated selector signal SEL_H is output to the selector 1380.

The selector 1380, the connection weight _{W A} from the connection weight computing circuit 1320 _receives the _B, and from the connection weight correction circuit 1350, 1360 connection weights _W A, B _{_H, W A,} receiving the B _S. The selector 1380, the selector signal generator circuit 1370 receives the selector signal SEL_RGB, select the connection weight _{W A, B,} and outputs the selected connection weight _{W A, B} as connection weight CW. The selector 1380, the selector signal generator circuit 1370 receives the selector signal SEL_S, connection weight _{W A,} select B _S, and outputs the selected connection weight _{W A,} the B _S as connection weight CW. Furthermore, the selector 1380, the selector signal generator circuit 1370 receives the selector signal SEL_H, connection weight _{W A,} select B _H, and outputs the selected connection weight _{W A,} the B _H as connection weight CW.

The weight calculation unit 133 shown in FIG. 2 has the same configuration as the weight calculation unit 132 shown in FIG. 24, and receives the RGB values RGB _A and RGB _B received from the D latch circuits 121 and 124 and the D latch circuits 126 and 129. and Hue bin number _a, Hue bin number _B, and based on the S bin number _a, and S bin number _B, in the same manner as weight calculating unit 132, the connection weight _{W a, B, W a,} B _H, W a, calculates the _B _S, the computed connection weight _{W a, B, W a,} B _H, W a, selects and outputs one of B _S as connection weight CW.

2 has the same configuration as the weight calculation unit 132 shown in FIG. 24, and the RGB value RGB _A received from the D latch circuit 124 and the Hue bin number _A received from the D latch circuit 129. and S based and bin number _a, RGB value _RGB B received from the selector 130, to the Hue bin number _B and S bin number _B, in the same manner as weight calculating unit 132, the connection weight _{W a,} B, _{W a ,} B _{_H, W a,} calculates the B _S, the computed connection weight _{W a, B, W a,} B _H, W a, selects and outputs one of B _S as connection weight CW.

Further, the weight calculation unit 135 shown in FIG. 2 has the same configuration as the weight calculation unit 132 shown in FIG. 24, and the RGB value RGB _A received from the D latch circuit 121 and the Hue bin number _A received from the D latch circuit 126. and S based and bin number _a, RGB value _RGB B received from the selector 131, to the Hue bin number _B and S bin number _B, in the same manner as weight calculating unit 132, the connection weight _{W a,} B, _{W a ,} B _{_H, W a,} calculates the B _S, the computed connection weight _{W a, B, W a,} B _H, W a, selects and outputs one of B _S as connection weight CW.

  An image dividing method in the image dividing circuit 13 will be described. FIG. 25 is a diagram for explaining an image dividing method. The image dividing circuit 13 divides one frame into images of each object by a region growing type image dividing method.

  This region-growing image segmentation method is based on the LEGION model (DL Wang, and D. Terman, “Image segmentation based on oscillator correlation,” Neural Computation, Volume 9 (4), pp. 805-836 (1997).) The behavior of the vibrator for each pixel of the child network is handled in four states: self-ignition, self-ignitable (self-excitable), ignition (excitation), and suppression (inhibition). This region-growing image segmentation method includes four steps of initialization, self-ignition, ignition, and suppression.

  The image dividing circuit 13 receives an input image having a 3 × 3 configuration shown in FIG. Then, the image dividing circuit 13 receives eight coupling weights CW1 to CW8 between the pixels in the nine pixels constituting the input image from the coupling weight determination circuit 12, and receives the received eight coupling weights CW1 to CW8. Each pixel is stored in the memory in association with each other. In FIG. 25B, eight connection weights of the pixel (2, 2) and the surrounding pixels are shown.

Then, the image dividing circuit 13 reads the eight connection weights CW1 to CW8 held in the memory, and calculates the sum SUM (CW) of the read eight connection weights CW1 to CW8. The image dividing circuit 13 determines whether or not the calculated sum SUM (CW) is larger than the threshold value phi _P for determining the leader cell. In this application, SUM means calculating a sum represented by a sigma symbol in mathematics.

Image dividing circuit 13, when the sum SUM (CW) is determined to be greater than the threshold value phi _P, to set the pixels that are the center of the calculation of the coupling weights leader cell, set the reader self-lag p _{ij = 1} To do. The image dividing circuit 3, when the sum SUM (CW) is equal to or less than the threshold value phi _P, without setting the pixels that are the center of the calculation of the coupling weights leader cell, the reader self-lag p _ij = Set to 0.

  The image dividing circuit 13 executes this processing by setting each of the nine pixels of the input image as a pixel that is the center of calculation of the coupling weight, and determines the leader cell (see FIG. 25C).

Thereafter, the image dividing circuit 13 causes one of the leader cells to self-ignite (ignition flag x _ij = 1) (see (d) of FIG. 25). Then, if the cell (k, l) εN _ij corresponding to the eight pixels adjacent to each cell (i, j) has been ignited, the image dividing circuit 3 determines that the cell (i, j) and the cell ( k, l) and SUM _{(k, l) ∈ Nij ， xkl = 1} W _{ij; kl} is calculated and the calculated sum SUM _{(k, l) ∈ Nij∧xkl = 1} W _{ij ; kl} is greater than the threshold value phi _Z, and, yet, the cell (i, j) does not belong to any divided area if (label flag _l ij = 0), automatically firing _{(x ij} = 1) Let it be ignited (ignition / regional growth) (see FIG. 25E). By this ignition process, the area expands, and as shown in (f) of FIG. 25, when there is no cell to be newly ignited, division of one area is completed.

Thereafter, the image dividing circuit 13 writes the label number for identifying the divided area into the ignited cell and stores the ignited cell, as shown in FIG. 25 (g), in order to save the divided area. Is set to a label flag indicating that the cell has already been divided (l _ij = 1), and a fire extinguishing process (x _ij = 0, p _ij = 0) is performed.

  When the extinguishing process is completed, the image dividing circuit 13 returns to the self-igniting process for dividing the next area again. Then, the image dividing circuit 13 repeatedly executes the above-described processing until no leader cell exists, and divides each region. Then, after all the reader cells are ignited, the image dividing circuit 13 labels all the areas and ends the division of the image of one frame (see (h) in FIG. 25).

  The operation in the tracking unit 2 shown in FIG. 1 will be described. FIG. 26 is a conceptual diagram for obtaining a tracking object. Referring to FIG. 26, image division unit 1 divides an input image (see FIG. 26A) into a plurality of divided images by the method described above, and outputs the divided plurality of divided images to tracking unit 2. To do.

  The tracking unit 2 receives a plurality of divided images from the image dividing circuit 13 of the image dividing unit 1, and classifies the plurality of divided objects represented by the received plurality of divided images into a stationary object and a moving object. More specifically, the tracking unit 2 includes a plurality of divided images DG1 (t-1) to (t-1) obtained by dividing the t-1 (t is an integer equal to or larger than 2) th frame F (t-1). When DGm (t-1) (m is an integer of 2 or more) is received from the image dividing circuit 13, a plurality of divisions represented by the received plurality of divided images DG1 (t-1) to DGm (t-1). The feature quantities of the objects OJ1 (t-1) to OJm (t-1) are extracted and stored. In this case, the tracking unit 2 uses the position, size (width and height), color (R, G, B), area, and area of each of the divided objects OJ1 (t-1) to OJm (t-1) as feature quantities. Extract the estimated position.

  Thereafter, when the tracking unit 2 receives a plurality of divided images DG1 (t) to DGm (t) obtained by dividing the t-th frame F (t) from the image dividing circuit 13, the plurality of divided images received. Feature quantities of the plurality of divided objects OJ1 (t) to OJm (t) represented by the images DG1 (t) to DGm (t) are extracted and stored.

  Then, the tracking unit 2 uses the feature quantities of the divided objects OJ1 (t-1) to OJm (t-1) and the feature quantities of the divided objects OJ1 (t) to OJm (t) by the built-in associative memory. Then, object matching is performed between the divided objects OJ1 (t-1) to OJm (t-1) and the divided objects OJ1 (t) to OJm (t). For example, the tracking unit 2 detects the divided objects OJ1 (t) to OJm (t) having the feature quantity that minimizes the Manhattan distance from the feature quantities of the divided objects OJ1 (t-1) to OJm (t-1). By doing this, the object matching between the divided objects OJ1 (t-1) to OJm (t-1) and the divided objects OJ1 (t) to OJm (t) is performed.

  Then, the tracking unit 2 obtains the motion vector MV1 (t) of the divided object OJ1 (t) based on the positions of the divided objects OJ1 (t−1) and OJ1 (t) that have been subjected to object matching. The tracking unit 2 obtains motion vectors MV2 (t) to MVm (t) for each of the divided objects OJ2 (t) to OJm (t).

  Thereafter, the tracking unit 2 sets a divided object having a motion vector having a size of “1” or more as a moving object based on the obtained motion vectors MV1 (t) to MVm (t), and the size is “1”. The divided objects OJ1 (t) to OJm (t) are classified into a moving object and a stationary object, with a divided object having a motion vector smaller than "a stationary object as a stationary object.

  Subsequently, when the tracking unit 2 obtains a moving object (see FIG. 26B), the tracking unit 2 groups the obtained moving objects by a method to be described later (see FIG. 26C). recognize.

  Each time the tracking unit 2 receives a plurality of divided images obtained by dividing a new frame from the image dividing circuit 13 of the image dividing unit 1, the tracking unit 2 recognizes the tracking object by the method described above.

FIG. 27 is a diagram for explaining a method for obtaining a motion vector and an estimated position of a divided object. Referring to FIG. 27, when the tracking unit 2 receives a plurality of divided images DG1 (t−1) to DGm (t−1) from the image dividing circuit 13, the tracking unit 2 receives the plurality of divided images DG1 (t−1). ) To DGm (t−1), a box Bi inscribed in the divided object is generated. The box Bi has four vertices A, B, C, and D. In the xy coordinate system shown in FIG. 27, the vertex A has coordinates (X _xmin , Y _ymin ), and the vertex B is It has coordinates (X _xmin , Y _ymax ), vertex C has coordinates (X _xmax , Y _ymax ), and vertex D has coordinates (X _xmax , Y _ymin ).

  When the tracking unit 2 generates the box Bi, the tracking unit 2 determines the center position of the box Bi based on the coordinates of the four vertices A, B, C, and D of the generated box Bi, and the determined center position is divided into objects. The position is (= Segment i). The tracking unit 2 determines the size (width and height) of the divided object (= Segment i) based on the coordinates of the four vertices A, B, C, and D of the box Bi. Furthermore, the tracking unit 2 detects the RGB value and the area of the divided object (= Segment i) in each of the divided images DG1 (t−1) to DGm (t−1).

  When the tracking unit 2 receives the plurality of divided images DG1 (t) to DGm (t) from the image dividing circuit 13, the tracking unit 2 performs the same for each of the received divided images DG1 (t) to DGm (t). Then, the position, size (width and height), RGB value, and area of the divided object (= Segment i) are obtained.

  After that, the tracking unit 2 performs the object matching described above, and the position of the divided object OJ1 (t-1) in the divided image DG1 (t-1) and the position of the divided object OJ1 (t) in the divided image DG1 (t). Based on the above, the motion vector MV1 (t) of the divided object OJ1 (t) is obtained. Similarly, the tracking unit 2 obtains motion vectors MV2 (t) to MVm (t) of the divided objects OJ2 (t) to OJm (t).

  Then, the tracking unit 2 adds the obtained motion vectors MV1 (t) to MVm (t) to the positions of the divided objects Segment_1 to Segment_m in the divided images DG1 (t) to DGm (t), thereby dividing the segment Objects_1 to 1. The estimated position of Segment_m is obtained. Then, the tracking unit 2 stores the position, size (width and height), RGB value, area, and estimated position of the divided object (= Segment i) as the feature amount of the divided object (= Segment i).

  FIG. 28 is a diagram illustrating a definition of a bounding box necessary for grouping divided objects. Referring to FIG. 28, when the tracking unit 2 generates a box Bi by the above-described method, the tracking unit 2 defines a box obtained by adding a margin margin to the generated box Bi as a bounding box BBi.

This bounding box BBi consists of a vertex P _{max, i} consisting of coordinates (X _xmax + margin, Y _ymin -margin) and coordinates (X _xmin -margin, Y _ymax + margin) in the xy coordinate system shown in FIG. It has a vertex P _{min, i} .

  In this way, the box with the margin margin added to the box Bi is defined as the bounding box BBi because the boundary area of the object becomes rougher than the boundary area of the original object due to image division, This is to compensate for this because it is smaller than the size.

  Since the center position of the box Bi matches the center position of the bounding box BBi, the position of the divided object (= Segment i) matches the center position of the bounding box BBi. Therefore, the motion vector MV is obtained starting from the center position of the bounding box BBi.

  When the tracking unit 2 obtains the bounding boxes BB1 to BBm of the divided objects OJ1 (t) to OJm (t) in the divided images DG1 (t) to DGm (t), the tracking objects OJ1 (t) to OJm (t) are obtained. The object is classified into a moving object and a stationary object, and one of the divided objects classified as the moving object is selected as the attention object SOJ, and a divided object similar to the attention object SOJ is selected as the remaining moving object (moving object MOJ1 (t)). ˜MOJn (t) ≠ SOJ).

  In this case, the tracking unit 2 detects a divided object having a motion vector similar to the motion vector of the target object SOJ and having a bounding box intersecting with the bounding box of the target object SOJ as an object similar to the target object SOJ. To do.

  Then, the tracking unit 2 determines whether the two motion vectors are similar by the following method. That is, the tracking unit 2 calculates the difference between the two motion vectors, and determines that the two motion vectors are similar when the calculated difference is less than or equal to the threshold value, and the calculated difference is less than the threshold value. When it is large, it is determined that the two motion vectors are not similar.

  FIG. 29 is a conceptual diagram showing the intersection of bounding boxes. Referring to FIG. 29, bounding box BB1 has vertices max1 and min1, and bounding box BB2 has vertices max2 and min2.

  The x coordinate max1_x of the vertex max1 is not less than the x coordinate min2_x of the vertex min2 (max1_x ≧ min2_x), the x coordinate min1_x of the vertex min1 is not more than the x coordinate max2_x of the vertex max2 (min1_x ≦ max2_x), and When the y coordinate max1_y of the vertex max1 is not less than the y coordinate min2_y of the vertex min2 (max1_y ≧ min2_y) and the y coordinate min1_y of the vertex min1 is less than or equal to the y coordinate max2_y of the vertex max2 (min1_y ≦ max2_y), two boundings Boxes BB1 and BB2 intersect each other.

  Therefore, the tracking unit 2 determines that the two bounding boxes BB1 and BB2 intersect when the four inequalities of max1_x ≧ min2_x, min1_x ≦ max2_x, max1_y ≧ min2_y, and min1_y ≦ max2_y simultaneously hold, and max1_x ≧ min2_x, When at least one of the four inequalities of min1_x ≦ max2_x, max1_y ≧ min2_y, and min1_y ≦ max2_y does not hold, it is determined that the two bounding boxes BB1 and BB2 do not intersect.

  The tracking unit 2 first determines whether or not the bounding box of the target object SOJ intersects with the bounding boxes of other divided objects other than the target object SOJ by the method described above. When the tracking unit 2 determines that the bounding box of the target object SOJ does not intersect with the bounding boxes of the other divided objects, the tracking unit 2 determines that the other divided objects are not similar to the target object SOJ.

  On the other hand, when the tracking unit 2 determines that the bounding box of the object of interest SOJ intersects the bounding box of another moving object, the tracking unit 2 determines whether or not the motion vector of the object of interest SOJ is similar to the motion vector of the other moving object. Further determine. Then, when the motion vector of the target object SOJ and the motion vector of another moving object are similar, the tracking unit 2 determines that the other moving object is similar to the target object SOJ, and determines that the target object SOJ and the other moving object are the same. The object group SOJ and other moving objects are grouped by giving the same group number.

  On the other hand, the tracking unit 2 determines that the other moving object is not similar to the target object SOJ when the motion vector of the target object SOJ is not similar to the motion vector of the other moving object.

  The tracking unit 2 performs this process on the target object SOJ and all the moving objects other than the target object SOJ, and performs grouping.

  The tracking unit 2 groups, for example, four moving objects including the three windows of the vehicle and the ceiling of the vehicle shown in FIG. c) see).

  FIG. 30 is a flowchart for explaining the object tracking method according to the embodiment of the present invention.

  Referring to FIG. 30, when the object tracking operation is started, the image division unit 1 of the object tracking device 10 reads a frame (step S1), and divides the read frame by the above-described method (step S1). S2). Then, the image division unit 1 outputs the divided image to the tracking unit 2.

  The tracking unit 2 receives the divided image from the image dividing unit 1, extracts the feature amount of each divided object represented in the received divided image (step S3), and stores the extracted feature amount.

  Then, the tracking unit 2 determines whether or not the frame being processed is the first frame (step S4), and if the frame being processed is the first frame, whether or not the frame has been read. It is further determined whether or not (step S5).

  When it is determined in step S5 that no frame is read, the series of operations ends.

  On the other hand, when it is determined in step S5 that the frame is read, the series of operations returns to step S2, and the above-described steps S2 to S5 are repeatedly executed.

  On the other hand, when it is determined in step S4 that the frame being processed is not the first frame, the tracking unit 2 performs object matching of each divided object between frames by the method described above (step S6). Then, the tracking unit 2 obtains the motion vector of the divided object subjected to object matching, and calculates the estimated position of the divided object by adding the obtained motion vector to the position of the divided object (step S7).

  Thereafter, the tracking unit 2 performs grouping using the motion vector and the bounding box (step S8), and outputs group information (step S9).

  Then, the tracking unit 2 determines whether or not there is a frame read (step S10). When it is determined in step S10 that a frame has been read, the series of operations returns to step S2, and the above-described steps S2 to S10 are repeatedly executed until it is determined in step S10 that no frame has been read. Is done.

  When it is determined in step S10 that no frame is read, the series of operations ends.

FIG. 31 is a flowchart for explaining the detailed operation of step S2 of FIG. Referring to FIG. 31, after step S <b> 1 in FIG. 30, the pixel value detection circuit 11 determines each pixel GE _ij (1 ≦ i ≦ n, n × n (n is an integer of 2 or more)). RGB value RGB _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n) of 1 ≦ j ≦ n) is detected (step S 21), and the detected RGB value RGB _ij of each pixel is output to the connection weight determination circuit 12. To do.

The combination weight determination circuit 12 receives the RGB value RGB _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n) of each pixel GE _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n) from the pixel value detection circuit 11. The RGB values RGB _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n) of the received pixels GE _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n) are converted into hues H _ij (1 ≦ 1) by the method described above. i ≦ n, 1 ≦ j ≦ n) and saturation S _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n), RGB values RGB _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n), Based on the hue H _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n) and the saturation S _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n), the connection weight W _ij between the pixels _{; kl} (1 ≦ i ≦ n, 1 ≦ j ≦ n, 1 ≦ k ≦ n, 1 ≦ l ≦ n) are determined (step S22). Then, the connection weight determination circuit 12 divides the determined connection weight W _{ij; kl} (1 ≦ i ≦ n, 1 ≦ j ≦ n, 1 ≦ k ≦ n, 1 ≦ l ≦ n) between the pixels. Output to the circuit 13.

The image dividing circuit 13 receives an input image from the outside, and the connection weight W _ij between the pixels from the connection weight determination circuit 12 _{; kl} (1 ≦ i ≦ n, 1 ≦ j ≦ n, 1 ≦ k ≦ n, 1 ≦ l ≦ n). Then, the image dividing circuit 13 uses the received connection weight W _{ij; kl} (1 ≦ i ≦ n, 1 ≦ j ≦ n, 1 ≦ k ≦ n, 1 ≦ l ≦ n) between the received pixels. The frame image is divided (step S23). And a series of operation | movement transfers to step S3 of FIG.

FIG. 32 is a flowchart for explaining the detailed operation of step S22 shown in FIG. 32, the pixel GE1 shown in FIG. 3 is the pixel GE _ij , the pixel GE2 is the pixel GE _kl , the pixel GE3 is the pixel GE _{i + 1, j} , the pixel GE4 is the pixel GE _{k + 1, l} , and the details of step S2 The operation will be described.

Referring to FIG. 32, after step S <b> 21 shown in FIG. 31, the connection weight determination circuit 12 outputs a plurality of pixels GE _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n) from the pixel value detection circuit 11. The RGB values RGB _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n) are sequentially received. Then, the connection weight determination circuit 12 sets i = 1, j = 1, k = 1, and l = 1 (step S221).

Then, the connection weight determination circuit 12 includes two adjacent pixels GE _ij sequentially input from the pixel value detection circuit 11 among the plurality of pixels GE _ij (1 ≦ i ≦ n, 1 ≦ j ≦ n) of the input image. , GE _kl (1 ≦ k ≦ n, 1 ≦ l ≦ n) is selected (step S222).

Thereafter, the combination weight determination circuit 12 converts the RGB value RGB _ij of the pixel GE _ij into the hue H _ij and the saturation S _ij by the above-described method by the HS conversion circuit 125 (Step S223), and the RGB value of the pixel GE _kl . RGB _kl is converted into hue H _kl and saturation S _kl by the above-described method (step S224).

In the connection weight determination circuit 12, the RGB values of four sets of adjacent pixels _GEij , _GEkl ; GEi _{+ 1, j} , _GEkl ; _GEij , GEk _{+ 1, l} ; GEi _{+ 1, j} , GEk _{+ 1, l} are hues. Then, it is determined whether or not it has been converted to saturation (step S225).

In step S225, the RGB values of four sets of adjacent pixels GE _ij , GE _kl ; GE _{i + 1, j} , GE _kl ; GE _ij , GE _{k + 1, l} ; GE _{i + 1, j} , GE _{k + 1, l} are converted to hue and saturation. When it is determined that it has not been performed, either or both of i and k are changed by “1” (step S226). Thereafter, the series of operations returns to step S222, and in step S225, four sets of adjacent pixels GE _ij , GE _kl ; GE _{i + 1, j} , GE _kl ; GE _ij , GE _{k + 1, l} ; GE _{i + 1, j} , GE _{k + 1 , L} are determined to have been converted into hue and saturation, steps S222 to S226 are repeatedly executed.

In step S225, the RGB values of the four sets of adjacent pixels GE _ij , GE _kl ; GE _{i + 1, j} , GE _kl ; GE _ij , GE _{k + 1, l} ; GE _{i + 1, j} , GE _{k + 1, l} are converted to hue and saturation. If it is determined that the weight is converted to, the four weight calculation units 132 to 135 of the connection weight determination circuit 12 determine connection weights between adjacent pixels in parallel (steps S227 to S230).

In this case, the weight calculation unit 132 determines the connection weights W _{k, l; i + 1, j} between the pixels GE _kl , GE _{i + 1, j} , and the weight calculation unit 133 determines the connection between the pixels GE _ij , GE _{k + 1, l.} The weights W _{ij; k + 1, l} are determined, and the weight calculation unit 134 determines the connection weights W _{ij; kl} between the pixels GE _ij and GE _kl , and the weight calculation unit 135 determines the pixels GE _{i + 1, j} , GE _{k + 1,} determining _{k + 1, l;} connection weight _{W i + 1, j} between _l.

  After step S227 to step S230, the connection weight determination circuit 12 determines whether i = n and j = n (step S231). If i = n and j = n are not satisfied, i , J or both are changed by “1” (step S232), and k = 1 and l = 1 are set (step S233).

  Thereafter, the series of operations returns to step S222, and steps S222 to S233 are repeatedly executed until it is determined in step S231 that i = n and j = n.

  If it is determined in step S231 that i = n and j = n, the series of operations proceeds to step S23 in FIG.

FIG. 33 is a flowchart for explaining the detailed operation of step S227 shown in FIG. Referring to FIG. 33, after “YES” in step S225 of FIG. 32, the combined weight calculation circuit 1320 of the weight calculation unit 132 receives the RGB value RGB _kl received from the FIFO circuit 123 and the D latch circuit 122. Based on the RGB values RGB _{i + 1, j} , a connection weight W _{i + 1, j;} kl_RGB between the pixels GE _kl , GE _{i + 1, j} is determined by the above-described method, and the determined connection weight W _{i + 1, j; kl} _RGB is output to the connection weight correction circuits 1350 and 1360 and the selector 1380.

The hue difference calculation circuit 1330 receives the hue bin number _A of the pixel GE _kl from the FIFO circuit 128 and receives the hue bin number _B of the pixel GE _{i + 1, j} from the D latch circuit 127. Then, the hue difference calculation circuit 1330 extracts the hue H _kl corresponding to the hue bin number _A with reference to the allocation diagram of the hue bin number shown in FIG. 14, and the hue H _{i + 1, j} corresponding to the hue bin number _B. To extract.

Then, the hue difference calculation circuit 1330 calculates the hue difference | H _kl −H _{i + 1, j} | between the hue H _kl and the hue H _{i + 1, j,} and calculates the calculated hue difference | H _kl −H _{i + 1, j} | The result is output to the connection weight correction circuit 1350.

The combination weight correction circuit 1350 receives the hue difference | H _kl −H _{i + 1, j} | from the hue difference calculation circuit 1330 and receives the combination weight W _{i + 1, j;} kl_RGB from the combination weight calculation circuit 1320. The connection weight correction circuit 1350, with reference to Table 3, the connection weight _{W i + 1, j; kl} _RGB and hue difference _{| H kl -H i + 1,} j | in based, coupled by the method described above weight _{W i + 1, j;} kl_H is extracted, and the extracted connection weight W _{i + 1, j;} kl_H is output to the selector 1380.

Further, the saturation difference calculating circuit 1340 receives the S bin number _A of the pixel GE _kl from the FIFO circuit 128 and receives the S bin number _B of the pixel GE _{i + 1, j} from the D latch circuit 127.

Then, the saturation difference calculating circuit 1340 extracts the saturation S _kl corresponding to the S bin number _A with reference to the S bin number assignment diagram shown in FIG. 16 and the saturation S corresponding to the S bin number _{B. i + 1, j} are extracted.

Then, the saturation difference calculation circuit 1340 calculates a saturation difference | S _kl −S _{i + 1, j} | between the saturation S _kl and the saturation S _{i + 1, j,} and the calculated saturation difference | S _kl −S. _{i + 1, j} | is output to the connection weight correction circuit 1360.

The combination weight correction circuit 1360 receives the saturation difference | S _kl −S _{i + 1, j} | from the saturation difference calculation circuit 1340 and receives the combination weight W _{i + 1, j;} kl_RGB from the combination weight calculation circuit 1320. Then, the connection weight correction circuit 1360 refers to Table 2, and uses the above-described method to determine the connection weight W _{i + 1} based on the connection weights W _{i + 1, j;} kl_RGB and the saturation difference | S _kl −S _{i + 1, j} |. _{, J;} kl_S is extracted, and the extracted connection weights W _{i + 1, j;} kl_S are output to the selector 1380.

On the other hand, the selector signal generation circuit 1370 of the weight calculation unit 132 receives the S bin number _A of the pixel GE _kl from the FIFO circuit 128 and the S bin number _B of the pixel GE _{i + 1, j} from the D latch circuit 127.

Then, the selector signal generation circuit 1370 extracts the saturation S _kl corresponding to the S bin number _A with reference to the S bin number assignment diagram shown in FIG. 16, and the saturation S _{i + 1} corresponding to the S bin number _{B. , J} are extracted.

Then, the selector signal generation circuit 1370 has a saturation S _{i + 1, j} that is equal to or less than the threshold value S _th (= 0.2) and a saturation S _{kl that} is equal to or less than the threshold value S _th (= 0.2). It is determined whether or not (step S2271).

In step S2271, it is determined that the saturation S _{i + 1, j} is equal to or less than the threshold value S _th (= 0.2) and the saturation S _kl is equal to or less than the threshold value S _th (= 0.2). At this time, the selector signal generation circuit 1370 generates the selector signal SEL_RGB and outputs the generated selector signal SEL_RGB to the selector 1380.

The selector 1380, the connection weight _{W i + 1,} j from the connection weight computing circuit 1320; undergoing kl _RGB connection weight _{W i + 1, j} from the connection weight correction circuit _1350; undergoing kl _H, coupled from the connection weight correction circuit 1360 weight _{W i + 1 , J;} kl_S. The selector 1380, the selector signal generator circuit 1370 receives the selector signal SEL_RGB, in response to the selector signal SEL_RGB, three connection weight _{W i + 1, j; kl} _RGB, W i + 1, j; kl _H, W i + 1, _j; kl coupled from _S weight _{W i + 1,} j; select kl _RGB, connection weight _{W i + 1} and the _selected, j; combine kl _RGB weight _{W i + 1, j;} determined with _kl. That is, the weight calculation unit 132 determines the combination weight W _{i + 1, j; kl} from the RGB values RGB _{i + 1, j} , RGB _kl (step S2272). Then, the weight calculation unit 132 outputs the determined combination weight W _{i + 1, j; kl} to the image dividing circuit 13.

On the other hand, in step S2271, it is determined that the saturation S _{i + 1, j} is not more than the threshold value S _th (= 0.2) and the saturation S _kl is not less than the threshold value S _th (= 0.2). When this is done, the selector signal generation circuit 1370 further determines whether only _{one of} the saturations S _{i + 1, j} and S _kl is equal to or less than the threshold value S _th (= 0.2) (step S2273).

In step S2273, when it is determined that only _one of the saturations S _{i + 1, j} and S _kl is equal to or less than the threshold value S _th (= 0.2), the selector signal generation circuit 1370 receives the selector signal SEL_S. The generated selector signal SEL_S is output to the selector 1380.

Upon receiving the selector signal SEL_S from the selector signal generation circuit 1370, the selector 1380 has three coupling weights W _{i + 1, j;} kl_RGB, W _{i + 1, j;} kl_H, Wi _{+ 1, j; The} connection weight W _{i + 1, j;} kl_S is selected from kl_S, and the selected connection weight W _{i + 1, j;} kl_S is determined as the connection weight W _{i + 1, j; kl} . That is, the weight calculation unit 132 determines the combined weight W _{i + 1, j; kl} from the RGB values RGB _{i + 1, j} , RGB _kl and the saturations S _{i + 1, j} , S _kl (step S2274). Then, the weight calculation unit 132 outputs the determined combination weight W _{i + 1, j; kl} to the image dividing circuit 13.

On the other hand, when it is determined in step S2273 that only _one of the saturations S _{i + 1, j} and S _kl is not equal to or less than the threshold value S _th (= 0.2), the selector signal generation circuit 1370 generates the selector signal SEL_H. The generated selector signal SEL_H is output to the selector 1380.

Upon receiving the selector signal SEL_H from the selector signal generation circuit 1370, the selector 1380 has three coupling weights W _{i + 1, j;} kl_RGB, W _{i + 1, j;} kl_H, W _{i + 1, j;} connection weight from _{kl _S W i + 1, j} ; select kl _H, the selected connection weight _{W i + 1, j;} determines that _kl; a kl _H connection weight _{W i + 1, j.} That is, the weight calculation unit 132 determines the combination weight W _{i + 1, j; kl} from the RGB values RGB _{i + 1, j} , RGB _kl and the hues H _{i + 1, j} , H _kl (step S2275). Then, the weight calculation unit 132 outputs the determined combination weight W _{i + 1, j; kl} to the image dividing circuit 13.

  Then, after any of steps S2272, S2274, and S2275, the series of operations proceeds to step S231 in FIG.

In step S2274, the combination weights W _{i + 1, j;} kl_RGB are determined based only on the RGB values RGB _{i + 1, j} , RGB _kl , and the determined combination weights W _{i + 1, j;} kl_RGB are used as the saturation S _{i + 1, j} , S _kl saturation step | S _{i + 1, j} −S _kl |

In step S2275, the combination weights W _{i + 1, j;} kl_RGB are determined only by the RGB values RGB _{i + 1, j} , RGB _kl , and the determined combination weights W _{i + 1, j;} kl_RGB are used as the hues H _{i + 1, j.} , H _kl hue difference | H _{i + 1, j} −H _kl |

  Further, detailed operations in steps S228 to 230 shown in FIG. 32 are also executed by steps S2271 to S2275 shown in FIG.

  FIG. 34 is a flowchart for explaining the detailed operation of step S23 shown in FIG. Referring to FIG. 34, after step S22 shown in FIG. 31, the image dividing circuit 13 receives the connection weight between the pixels in a plurality of pixels constituting one frame from the connection weight determination circuit 12, and the method described above. To determine the leader cell (step S231).

  Then, the image dividing circuit 13 determines whether or not a self-ignitable cell has been detected (step S232). When it is determined that a self-ignitable cell has been detected, self-ignition is performed by the method described above (step S232). S233).

  Thereafter, the image dividing circuit 13 determines whether or not a firing cell exists (step S234). When the firing cell exists, the image dividing circuit 13 performs ignition (region growth) to the surroundings by the above-described method (step S235). Then, the series of operations returns to step S234, and steps S234 and S235 are repeatedly executed until there are no ignition cells in step S234.

  If it is determined in step S234 that there is no ignition cell, the image dividing circuit 13 performs ignition end / area determination and labeling (extinguishing) by the above-described method (step S236). Thereafter, the series of operations returns to step S232, and steps S232 to S236 are repeatedly executed until it is determined in step S232 that no self-ignitable cell is detected.

  If it is determined in step S232 that a self-ignitable cell is not detected, the series of operations proceeds to step S3 in FIG.

  The image dividing unit 1 of the object tracking device 10 repeatedly executes the flowchart shown in FIG. 31 (FIGS. 32 to 34), and divides each frame into images of each object.

  FIG. 35 is a flowchart for explaining the detailed operation of step S8 of FIG. Referring to FIG. 35, after step S7 of FIG. 30, the tracking unit 2 calculates the difference in the feature amount between all the objects based on the stored feature amount of each object (= target object). (Step S81), an object without a group number is searched (Step S82).

  Then, the tracking unit 2 determines whether there is an object without a group number (step S83).

  When it is determined in step S83 that there is an object without a group number, the tracking unit 2 determines a target object from among the objects without a group number (step S84). A group number is assigned (step S85).

  After that, the tracking unit 2 detects the object whose difference from the feature amount of the target object is equal to or less than the threshold value based on the feature amount difference between all the objects calculated in step S81, thereby determining the target object. Detect similar objects. Then, the tracking unit 2 attaches the same group number as the target object to the detected object (step S86).

  Subsequently, the tracking unit 2 detects an object similar to the object with the group number by the same method as in step S86, and assigns the same group number to the object of interest to the detected object (step S87).

  Thereafter, the series of operations returns to step S83, and the above-described steps S83 to S87 are repeatedly executed until it is determined in step S83 that there is no object without a group number.

  If it is determined in step S83 that there is no object without a group number, the series of operations proceeds to step S9 in FIG.

  In the flowchart shown in FIG. 35, when the loop consisting of “YES” in step S83 → step S84 → step S85 → step S86 → step S87 → step S83 is executed for the first time, all objects similar to one target object are displayed. Are detected, and all the detected objects and the object of interest are grouped by the same group number.

  When the loop consisting of “YES” in step S83 → step S84 → step S85 → step S86 → step S87 → step S83 is executed for the second time, all objects similar to the object of interest different from the first time are detected. , All the detected objects and the object of interest are grouped by the same group number.

  Therefore, until it is determined in step S83 that there is no object without a group number, a loop consisting of “YES” in step S83 → step S84 → step S85 → step S86 → step S87 → step S83 is repeatedly executed. Then, the same number of grouped objects (= tracking objects) as the number of times the loop consisting of “YES” in step S83 → step S84 → step S85 → step S86 → step S87 → step S83 is executed is generated.

  Then, in step S9 of FIG. 30, the tracking unit 2 outputs the object (= tracked object) to which each group number is assigned as group information.

As described above, the object tracking device 10 uses the image segmentation unit 1 to calculate two images based on only the RGB values of two pixels when both of two saturations of two adjacent pixels are equal to or less than the threshold value S _th . determining the connection weight between pixels (see step S2272), when only one of the two saturation is less than or equal to the threshold S _th, the coupling weight between the two pixels by RGB values and the saturation of the two pixels determined (see step S2274), both of the two chroma of two pixels is greater than the threshold _{S th,} it determines the connection weight by two RGB values and hue (see step S2275), and the determined Each frame is divided using the combination weight. Then, the object tracking device 10 recognizes and tracks the tracking object by grouping a plurality of objects in the plurality of divided images divided by the image dividing unit 1 by the tracking unit 2.

  Therefore, even if one frame is divided into images of a plurality of objects included in one frame, the tracking object can be easily recognized and the tracking object can be easily tracked.

  Further, the object tracking device 10 can accurately determine the connection weight between pixels in a color image with low contrast or a color image affected by light reflection, and an object having pixels when belonging to an achromatic region and a chromatic region. , And an object having pixels belonging to the chromatic color region can be accurately divided, and tracking objects can be accurately grouped based on the accurately divided objects. As a result, the tracking object can be accurately tracked.

  Another tracking method for the tracking object will be described. When this another tracking method is used, the tracking unit 2 uses the result of the grouping of the divided objects in the frame F (t−1) in the grouping of the divided objects in the frame F (t).

  FIG. 36 is a diagram for explaining a problem in grouping. Referring to FIG. 36, when two vehicles overlap (= when two objects overlap), the shape of the bounding box is deformed, affecting the motion vector, and one vehicle and a traffic sign are stationary object groups. Thus, a part of a traffic sign, one vehicle, and two unrecognizable objects may be grouped into a moving object group.

  Therefore, in order to eliminate such grouping failure, in the embodiment of the present invention, the tracking unit 2 preferably performs grouping in the current frame using the grouping result in the previous frame.

  FIG. 37 is a diagram illustrating the result of grouping in the previous frame. Referring to FIG. 37, in the front frame F (t−1), the traffic sign, one vehicle, and the other vehicle are grouped into one group.

  FIG. 38 is a diagram for explaining a grouping method used in another tracking method. 38, grouping in the frame F (t) of the vehicle CAR shown in FIG. 37 will be described.

  Referring to FIG. 38, the bounding box BBi (t) is the bounding box of the target object OJi (t) that is focused on in the frame F (t), and the bounding box BBi + 1 (t) is the frame F (t). The bounding box of the object OJi + 1 (t) similar to the object of interest OJi (t).

  The bounding box BBi (t−1) is a bounding box of the target object OJi (t−1) in the frame F (t−1) matched with the target object OJi (t), and the bounding box BBi + 1 (t−1) ) Is a bounding box of the object OJi + 1 (t−1) in the frame F (t−1) matched with the object OJi + 1 (t).

  Further, the bounding box BB (t−1) is an aggregate of bounding boxes of objects (vehicle CAR) grouped in the frame F (t−1) (see FIG. 38A).

  The tracking unit 2 performs object matching between the object of interest OJi (t) in the frame F (t) and the object OJi (t−1) in the frame F (t−1), and also the object OJi + 1 in the frame F (t). Object matching between (t) and the object OJi + 1 (t−1) in the frame F (t−1) is performed.

  The tracking unit 2 then uses the two objects OJi (t), OJi (t-1) and OJi + 1 (t), OJi + 1 (t-1) of the frames F (t-1) and F (t) that have been object-matched. Based on the above, a motion vector between the frames F (t−1) and F (t) is obtained.

  Further, the tracking unit 2 detects the bounding box BB (t−1) of the object (= vehicle CAR) obtained by grouping the objects OJi (t−1) and OJi + 1 (t−1) in the frame F (t−1). To do.

  Then, the tracking unit 2 adds the motion vector between the frames F (t−1) and F (t) to the bounding box BB (t−1), and the objects (to be grouped in the frame F (t) ( = A bounding box BB (t) of the vehicle CAR is obtained (see (b) of FIG. 38).

  Thereafter, the tracking unit 2 determines whether or not the obtained bounding box BB (t) and the bounding boxes BBi (t) and BBi + 1 (t) overlap.

  When the bounding box BB (t) and the bounding boxes BBi (t) and BBi + 1 (t) overlap (see FIG. 38C), the tracking unit 2 bounds the boxes BBi (t) and BBi + 1 (t). Are bound to each other, and when the bounding boxes BBi (t) and BBi + 1 (t) are similar to each other, the same group number is assigned to the objects OJi (t) and OJi + 1 (t).

  FIG. 39 is a flowchart for explaining another object tracking method according to the embodiment of the present invention.

  Referring to FIG. 39, when the object tracking operation is started, the image division unit 1 of the object tracking device 10 reads a frame (step S31), and divides the read frame by the above-described method (step S31). S32). Then, the image division unit 1 outputs the divided image to the tracking unit 2.

  The tracking unit 2 receives the divided image from the image dividing unit 1, extracts the feature amount of each divided object represented in the received divided image (step S33), and stores the extracted feature amount.

  Then, the tracking unit 2 determines whether or not the frame being processed is the first frame (step S34), and if the frame being processed is the first frame, whether or not the frame has been read. It is further determined whether or not (step S35).

  In step S35, when it is determined that no frame is read, the series of operations ends.

  On the other hand, when it is determined in step S35 that the frame is read, the series of operations returns to step S32, and the above-described steps S32 to S35 are repeatedly executed.

  On the other hand, when it is determined in step S34 that the frame being processed is not the first frame, the tracking unit 2 performs object matching of each divided object (= each object) between the frames by the above-described method ( Step S36). Then, the tracking unit 2 obtains a motion vector of each object with which the object is matched, and calculates an estimated position of each object using the obtained motion vector (step S37).

  Then, the tracking unit 2 calculates a difference between frames (step S38), and determines a moving object and a stationary object based on the calculated difference between frames (step S39).

  Thereafter, the tracking unit 2 determines whether or not the currently processed frame is the frame (2) (step S40).

  When it is determined in step S40 that the currently processed frame is the frame (2), the tracking unit 2 groups the objects using the information of the frame (2) (step S41).

  On the other hand, when it is determined in step S40 that the currently processed frame is not the frame (2), the tracking unit 2 uses the grouping result of the frame (t-1) and the information of the frame (t). The objects are grouped (step S42).

  Then, after step S41 or step S42, the tracking unit 2 outputs the grouping result (step S43), and determines whether or not the frame is read (step S44).

  When it is determined in step S44 that the frame is read, the series of operations returns to step S32, and the above-described steps S32 to S44 are repeatedly executed until it is determined in step S44 that no frame is read. Is done.

  If it is determined in step S44 that no frame is read, the series of operations ends.

  The detailed operation in step S32 shown in FIG. 39 is executed according to the flowchart shown in FIG. 31 (FIGS. 32 to 34).

  Further, the detailed operation of step S41 shown in FIG. 39 is executed according to the flowchart shown in FIG. The reason why the objects are grouped using the information of frame (2) in step S41 is that the grouping result in the previous frame cannot be used because the grouping is not yet performed.

  Furthermore, in the flowchart shown in FIG. 39, it has been described that the difference between frames is calculated to discriminate between a moving object and a stationary object (steps S38 and S39), but the embodiment of the present invention is not limited to this. In step S38, a motion vector may be calculated, and in step S39, a moving object and a stationary object may be determined using the calculated motion vector.

  FIG. 40 is a flowchart for explaining the detailed operation of step S42 shown in FIG.

  Referring to FIG. 40, after “NO” in step S40 shown in FIG. 39, tracking unit 2 reads the feature amount of the moving object (step S421), and determines whether the grouped flag is “1”. Determination is made (step S422).

  When it is determined in step S422 that the grouped flag is not “1”, the tracking unit 2 assigns a new group number (step S423).

  Then, when it is determined in step S422 that the grouped flag is “1” or after step S423, the tracking unit 2 updates the group number of the object, and the feature amount of the object of interest and the other object The difference from the feature amount is calculated (step S424).

  Thereafter, the tracking unit 2 determines whether or not the bounding box of the object of interest overlaps the bounding box of another object by the method described above (step S425).

  If it is determined in step S425 that the bounding box of the object of interest and the bounding box of another object do not overlap, the series of operations proceeds to step S431.

  On the other hand, when it is determined in step S425 that the bounding box of the object of interest overlaps the bounding box of the other object, the tracking unit 2 uses the above-described method to match the object (= It is further determined whether or not the bounding box of the object group overlaps with the bounding box of the object (= the object of interest and other objects) (step S426).

  When it is determined in step S426 that the bounding box of the group of objects (= objects) matched in the frame (t-1) and the bounding box of the objects (= target object and other objects) overlap, the tracking unit 2 Further determines whether the height and width of the bounding box of the object of interest and the bounding boxes of other objects are similar (step S427). In this case, the tracking unit 2 determines that the difference between the height of the bounding box of the target object and the height of the bounding box of the other object is equal to or less than the threshold, and the width of the bounding box of the target object and the other object When the difference from the width of the bounding box is equal to or smaller than the threshold value, it is determined that the height and width of the bounding box of the object of interest and the bounding box of the other object are similar. In addition, the tracking unit 2 determines that the difference between the height of the bounding box of the target object and the height of the bounding box of the other object is larger than the threshold, or the width of the bounding box of the target object and the other object When the difference between the bounding box and the bounding box is larger than the threshold, it is determined that the height and width of the bounding box of the object of interest and the bounding boxes of other objects are not similar.

  When it is determined in step S427 that the bounding box of the target object and the bounding box of the other object are similar in height and width, the tracking unit 2 determines the motion vector of the target object, the motion vector of the other object, Is calculated, and it is further determined whether or not the calculated motion vector difference is smaller than the threshold value Mvth2 (step S428). This threshold value Mvth2 is, for example, “10”.

  When it is determined in step S428 that the motion vector difference is greater than or equal to the threshold value Mvth2, the series of operations returns to step S424.

  On the other hand, when it is determined in step S426 that the bounding box of the group of objects (= objects) matched in the frame (t-1) and the bounding box of the objects (= target object and other objects) do not overlap, Alternatively, when it is determined in step S427 that the height and width of the bounding box of the target object and the bounding box of the other object are not similar, the tracking unit 2 determines that the motion vector of the target object and the motion of the other object The difference from the vector is calculated, and it is further determined whether or not the calculated motion vector difference is smaller than the threshold value Mvth1 (step S429). This threshold value Mvth1 is, for example, “5”.

  When it is determined in step S429 that the motion vector difference is greater than or equal to the threshold value Mvth1, the series of operations proceeds to step S431.

  On the other hand, when it is determined in step S428 that the motion vector difference is smaller than the threshold value Mvth2, or when it is determined in step S429 that the motion vector difference is smaller than the threshold value Mvth1, the tracking unit 2 Assigns the same group number to the object of interest and other objects, and sets the grouped flag to “1” (step S430). The threshold value Mvth2 is larger than the threshold value Mvth1.

  In step S425, when it is determined that the bounding box of the object of interest and the bounding box of another object do not overlap, or in step S429, it is determined that the difference between the motion vectors is greater than or equal to the threshold value Mvth1. When or after step S430, the tracking unit 2 determines whether or not the object of interest has been compared with all the moving objects (step S431).

  When it is determined in step S431 that the object of interest has not been compared with all moving objects, the series of operations returns to step S424.

  On the other hand, when it is determined in step S431 that the object of interest has been compared with all the moving objects, the tracking unit 2 further determines whether or not all the grouped flags of the moving objects have been examined (step S432).

  When it is determined in step S432 that all the grouped flags of the moving object have not been checked, the series of operations returns to step S422.

  On the other hand, when it is determined in step S432 that all the grouped flags of the moving objects have been checked, the tracking unit 2 outputs grouping information (step S433). And a series of operation | movement transfers to step S43 of FIG.

  Note that when step S424 to step S431 are executed until it is determined in step S431 that the target object has been compared with all the moving objects, one target object and an object similar to the target object are grouped. It becomes.

  If it is determined in step S432 that all the grouped flags of the moving objects have been checked, all the tracking objects are grouped when steps S422 to S432 are executed.

  In the flowchart shown in FIG. 39, when objects are grouped in frame (2), the objects are also grouped using information in frame (2) (see step S41 and FIG. 35). Are grouped using the grouping information in the previous frame and the information in the current frame (see step S42 and FIG. 40).

  Therefore, by tracking the objects according to the flowchart shown in FIG. 39, the objects can be grouped using the grouping information in the previous frame in the grouping of objects in the third and subsequent frames. Can avoid failure. As a result, each tracking object can be accurately recognized.

  FIG. 41 is a diagram showing the result of grouping objects according to the flowcharts shown in FIGS. 39 and 40.

  Referring to FIG. 41, the input image is composed of a frame t, a frame t + 1, and a frame t + 2 (see (a) of FIG. 41). Then, the grouping in the frame t + 1 is performed using the grouping information in the frame t, and the grouping in the frame t + 2 is performed using the grouping information in the frame t + 1 and the information of the current frame (FIG. 39). Step S42 and FIG. 40).

  As a result, the grouping was successful in both the frame t + 1 and the frame t + 2, and it was demonstrated that the grouping failure described in FIG. 36 can be avoided.

  In the above, the image division unit 1 determines the connection weight between the pixels using the RGB value, the hue, and the saturation of each pixel of the input image, and inputs the input image using the determined connection weight. In the embodiment of the present invention, the image division unit 1 determines the connection weight between the pixels using only the RGB value of each pixel of the input image. Then, the input image using the determined connection weight may be divided into the images of the respective objects. Generally, the input image may be divided into the images of the respective objects regardless of the division method.

  In the first embodiment described above, the object tracking device 10 can also be realized by software or hardware. When the object tracking device 10 is realized by software, the functions of the image dividing unit 1 (the pixel value detection circuit 1, the combination weight determining circuit 12, and the image dividing circuit 13) and the tracking unit 2 described above are typically This is realized by a CPU (Central Processing Unit) executing a computer program. The computer program is stored in advance in a ROM (Read Only Memory) in the mobile terminal, or downloaded from the outside and stored in a nonvolatile memory in the mobile terminal.

  When the object tracking device 10 is realized by hardware, the functions of the image division unit 1 (pixel value detection circuit 1, coupling weight determination circuit 12 and image division circuit 13) and the tracking unit 2 described above are CPU, RAM (Randum Access Memory), by combining with hardware resources such as ROM and nonvolatile memory, it is realized by an LSI (Large Scale Integrated circuit) which is an integrated circuit. The functions of the image division unit 1 (pixel value detection circuit 1, coupling weight determination circuit 12 and image division circuit 13) and the tracking unit 2 may be individually made into one chip, or in several unity units. One chip may be used. Further, the integrated circuit is not limited to an LSI, and may be realized by a dedicated circuit or a general-purpose processor. Furthermore, the integrated circuit uses an FPGA (Field Programmable Gate Array) that can store a program after manufacturing the LSI, or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI. Also good. Furthermore, the functions of the image division unit 1 (pixel value detection circuit 1, coupling weight determination circuit 12 and image division circuit 13) and the tracking unit 2 are integrated circuit technologies that are replaced by LSIs due to the advancement of semiconductor technology or other derived technologies. If (biotechnology, organic chemical technology, etc.) appears, it may naturally be integrated using that technology. Note that an integrated circuit may be referred to as an IC, a system LSI, a super LSI, an ultra LSI, or the like depending on the degree of integration.

  Thus, since the object tracking device 10 can also be realized by software or hardware, in the embodiment of the present invention, the pixel value detection circuit 11 described above constitutes a “pixel value detection unit”, The connection weight determination circuit 12 constitutes a “connection weight determination unit”, and the image division circuit 13 constitutes an “image division unit”.

  The tracking unit 2 includes a tracking object recognition unit and a grouping unit. Then, the tracking object recognition unit classifies the plurality of divided image areas generated by the image dividing unit 1 into a plurality of still object image areas indicating stationary objects and a plurality of moving object image areas indicating moving objects. The grouping unit groups the moving object image regions classified by the tracking object recognition unit based on a predetermined similarity.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is shown not by the above description of the embodiments but by the scope of claims for patent, and is intended to include meanings equivalent to the scope of claims for patent and all modifications within the scope.

  The present invention is applied to an object tracking device that can easily track a tracking object composed of a plurality of objects. Further, the present invention is applied to an object tracking method that can easily track a tracking object composed of a plurality of objects.

It is a schematic block diagram which shows the structure of the object tracking apparatus by embodiment of this invention. It is the schematic which shows the structure of the connection weight determination circuit shown in FIG. It is a conceptual diagram when the connection weight determination circuit shown in FIG. 2 determines connection weight. It is another conceptual diagram when the connection weight determination circuit shown in FIG. 2 determines connection weight. It is a conceptual diagram of HSV color space. It is a figure which shows the image used for the image evaluation by a hue and saturation. It is a figure which shows the other image used for the image evaluation by a hue and saturation. It is a figure which shows the division | segmentation result of the area | region X shown in FIG. 6, and the division | segmentation result of the area | region Y shown in FIG. It is a figure for demonstrating the problem in the conventional image division. It is the figure which normalized the value (0-360 degree | times) converted into the Hue value using Formula (1) from the RGB value of the sample image shown in FIG. It is the figure which normalized the value (0-360 degree | times) converted into the Hue value using Formula (1) from the RGB value of the sample image shown in FIG. It is the figure which normalized the value (range of 0-1) converted into S value using Formula (2) from the RGB value of the sample image shown in FIG. It is the figure which normalized the value (range of 0-1) which converted into RGB value from the RGB value of the sample image shown in FIG. 7 using Formula (2) to 256 gradations, and made it an 8-bit gray image. It is a block diagram of the hue (Hue) bin. It is the schematic which shows the structure of the hue conversion circuit contained in the HS conversion circuit shown in FIG. It is a block diagram of the bin of saturation (S). FIG. 3 is a schematic diagram illustrating a configuration of a saturation conversion circuit included in the HS conversion circuit illustrated in FIG. 2. It is a figure for demonstrating the method of determining a connection weight. It is a figure which shows the influence on the pixel value by reflection. It is a figure which shows the change of an achromatic color pixel value when changing a saturation boundary. It is a figure which shows the influence by irregular reflection of each component of R, G, B. It is a figure which shows the histogram of the Hue value of the body of a red car. It is a figure which shows the histogram of the Hue value of the body of a blue car. It is a block diagram of the weight calculation unit shown in FIG. It is a figure for demonstrating the division method of an image. It is a conceptual diagram for obtaining a tracking object. It is a figure for demonstrating the method of calculating | requiring the motion vector and estimated position of a moving object. It is a figure which shows the definition of the bounding box required for grouping of a division | segmentation object. It is a conceptual diagram which shows the intersection of a bounding box. It is a flowchart for demonstrating the object tracking method by embodiment of this invention. It is a flowchart for demonstrating the detailed operation | movement of step S2 of FIG. It is a flowchart for demonstrating the detailed operation | movement of step S22 shown in FIG. FIG. 33 is a flowchart for describing detailed operation of step S227 shown in FIG. 32. FIG. It is a flowchart for demonstrating the detailed operation | movement of step S23 shown in FIG. It is a flowchart for demonstrating the detailed operation | movement of step S8 of FIG. It is a figure for demonstrating the problem in grouping. It is a figure which shows the result of grouping in the previous frame. It is a figure for demonstrating the method of grouping used in another tracking method. It is a flowchart for demonstrating the other object tracking method by embodiment of this invention. It is a flowchart for demonstrating the detailed operation | movement of step S42 shown in FIG. It is a figure which shows the result of having grouped the object according to the flowchart shown to FIG. 39 and FIG.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image division unit, 2 Tracking unit, 11 Pixel value detection circuit, 12 Connection weight determination circuit, 13 Image division circuit

Claims (6)

An object tracking device for tracking an object in a moving image composed of a plurality of frame images,
An image dividing unit that generates a divided image region based on an image feature of an object in an image included in each frame constituting the moving image;
A tracking unit for tracking an object based on the divided image region generated by the image dividing unit;
The tracking unit is
A tracking object recognition unit that classifies the plurality of divided image areas generated by the image dividing unit into a plurality of stationary object image areas indicating stationary objects and a plurality of moving object image areas indicating moving objects;
Look including a grouping unit for grouping based on the moving object image area which has been classified by the tracking object recognition unit to a predetermined similarity,
The image segmentation unit is configured to inter-pixel based on an RGB value composed of an R component, a G component, and a B component of a pixel indicating an object in an image included in the frame, a hue of the pixel, and a saturation of the pixel. An object tracking device that determines a connection weight of the image and generates the divided image region using the determined connection weight .   The tracking object recognizing unit converts a plurality of the divided image regions into the plurality of still object image regions based on motion vectors of the divided image regions in a frame to which the divided image region belongs and a frame before or after the frame. The object tracking device according to claim 1, wherein the object tracking device is classified into a plurality of moving object image regions.   The object tracking device according to claim 2, wherein the grouping unit generates group information for uniquely specifying the grouped moving object images in consecutive frames. An object tracking method for tracking an object in a moving image composed of a plurality of frame images,
A first step of generating a divided image region based on an image feature of an object in an image included in each frame constituting the moving image;
A second step of classifying the plurality of divided image areas generated in the first step into a plurality of stationary object image areas indicating stationary objects and a plurality of moving object image areas indicating moving objects;
A third step of grouping the moving object image regions classified in the second step based on a predetermined similarity ,
The first step is based on an RGB value composed of an R component, a G component, and a B component of a pixel indicating an object in an image included in the frame, a hue of the pixel, and a saturation of the pixel. An object tracking method for determining a connection weight between the two and generating the divided image region using the determined connection weight .   In the second step, the plurality of divided image areas are determined based on motion vectors of the divided image areas in a frame to which the divided image area belongs and a frame before or after the frame. The object tracking method according to claim 4, wherein the object tracking method is classified into an area and the plurality of moving object image areas.   The object tracking method according to claim 5, wherein in the third step, group information that uniquely specifies the grouped moving object images is generated in consecutive frames.

Images