JP2006072763A - Image processing apparatus, image processing integrated circuit, image processing method, image processing program and computer-readable recording medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To recognize the same object between different frames, even when a plurality of objects are present inside an image. <P>SOLUTION: This image processor has an image division cell network 6 dividing the plurality of objects included inside the input image from the input image in each the frame; a position information calculating part 9 calculating an estimation position of each the object, on the basis of a movement vector showing a movement amount about each the object; and a minimum distance retrieval associative memory 10 for recognizing the same object between the different frames on the basis of the estimation position, and a characteristic amount, other than the estimated position. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention detects a stationary object and a moving object from an image by extracting an object in the image using image division processing and matching the objects using association processing.

  In recent years, there has been an increasing demand for image recognition processing technology for realizing intelligent information processing technology. In particular, in realization of intelligent robots that perform operations and judgments similar to humans, and in real-time image recognition processing, it is necessary to process images captured from cameras and the like at high speed, and high-speed processing is required ( Non-Patent Document 3 and Non-Patent Document 4).

  For this purpose, it is necessary to perform an image division process in which a target object is divided and extracted from various objects in the input image. As an image division processing algorithm, a cellular automaton type digital image division architecture based on LEGION (Locally Excitatory Globally Inhibitory Oscillator Network) (see Non-Patent Document 2, Non-Patent Document 6, and Non-Patent Document 7) has been proposed. (See Non-Patent Documents 11 and 13).

  Furthermore, in the research so far, the cellular automaton type digital image segmentation architecture has been implemented as hardware, and a chip (see Non-Patent Document 8) in which the cell network is fully custom designed has been prototyped and its processing time has been measured. Yes. As a result of the measurement, it is known that according to the cellular automaton type digital image segmentation architecture, image segmentation processing can be performed at a very high speed and with high accuracy.

In addition, in order to realize intelligent information processing technology, there is a high demand for technology for detecting the movement of an object in an image. So far, various algorithms as described in, for example, Patent Document 1 and Patent Document 2 have been proposed for object motion detection.
JP-A-8-241414 (published September 17, 1996) JP 2002-312793 A (released on October 25, 2002) JP 2003-346142 A (released on December 5, 2003) JP 2004-13504 A (published January 15, 2004) NHE Weste and K. Eshraghian, “PRINCIPLES OF CMOS VLSI DESIGN: A System Perspective Second Edition,” Addison-Wesley Publising Company, 1993. DL Wang, and D. Terman, “Image segmentation based on oscillator correlation,” Neural Computation, Vol. 9, No. 4, pp. 805-836, 1997. Takashi Matsuyama, Yoshinori Kuno, Satoshi Imiya, “Computer Vision: Technical Review and Future Prospects,” pp.138-146, 1998. JC Russ, “The Image Processing Handbook,” CRC PRESS, 1999. Keiji Taniguchi, “Image Processing Engineering,” Kyoritsu Publishing Co., Ltd., 1999. H. Ando, T. Morie, M. Nagata and A. Iwata, “A nonlinear oscillator network for gray-level image segmentation and PWM / PPM circuits for its VLSI implementation,” IEICE Trans. Fundamentals, Vol. E83-A, No 2, pp. 329-336, 2000. H. Ando, T. Morie, M. Miyake, M. Nagata, and A. Iwata, “Image segmentation / extraction using nonlinear cellular networks and their VLSI implementation using pulse-modulation techniques,” IEICE Trans. Fundamentals, Vol. E85- A, No.2, pp. 381-388, 2002. Hiroaki Harada, Takashi Morimoto, Tetsushi Koide, Jurgen Mataushhans, “Design of All-Parallel Image-Division Cell Network LSI,” 2002 Annual Conference of the Chugoku Branch of Electrical and Information Society, pp.591-592, 2002. HJ Mattausch, T. Gyohten, Y. Soda, and T. Koide, “Compact associative-memory architecture with fully-parallel search capability for the minimum Hamming distance,” IEEE Journal of Solid-State Circuits, Vol37, pp. 218-227 , 2002. H. Kimura, T. Shibata, “Simple-architecture motion-detection analog V-chip based on quasi-two-dimensional processing,” Extended Abstracts of the 2002 International Conference on Solid State Devices and Materials (SSDM2002), pp. 240-241, 2002. T. Morimoto, Y. Harada, T. Koide, and HJ Mattausch, “Low-complexity, highly-parallel color motion-picture segmentation architecture for compact digital CMOS implementation,” Extended Abstracts of the 2002 International Conference on Solid State Devices and Materials (SSDM2002), pp. 242-243, 2002. Y. Yano, T. Koide, HJ Mattausch, “Fully parallel nearest Manhattan-distance search memory with large reference-pattern number,” Extended Abstracts of the 2002 International Conference on Solid State Devices and Materials (SSDM2002), pp. 254-255 , 2002. T. Koide, T. Morimoto, Y. Harada, and HJ Mattausch, “Digital gray-scale / color image-segmentation architecture for cell-network-based real-time applications,” Proceedings of International Technical Conference on Circuits / Systems, Computers and Communications (ITC-CSCC2002), pp. 670-673, 2002. Bernd Jaehne, “Digital Image Processing,” pp. 481-499, 2002.

  However, the conventional algorithm for motion detection removes the background portion from the input image by taking the difference between the input image and the background for each frame, and recognizes the motion of the object from the remaining portion. Processing is relatively simplified. On the other hand, the conventional algorithm has a problem that it is not easy to simultaneously detect a plurality of moving objects and detect a motion (see Non-Patent Document 10).

  That is, according to the algorithms described in Patent Document 1 and Patent Document 2, feature quantities such as “color”, “centroid position”, and “area” are grasped with respect to divided objects, and the same between different frames based on the feature quantities. Recognize the object. However, it is difficult to recognize the same object between frames when there are a plurality of objects in an image only by grasping the feature amount of such an object. This is because if there are multiple objects in the image, there is a possibility that those objects whose features are approximated are included in the same frame, and those objects whose features are approximated are the same. This is because it may be recognized as an object.

  The present invention has been made in view of the above-described conventional problems, and is capable of recognizing the same object between different frames even when a plurality of objects exist in an image, and image processing integration. An object is to provide a circuit, an image processing method, an image processing program, and a recording medium.

  In order to solve the above problems, an image processing apparatus according to the present invention is configured to divide a plurality of objects included in an input image from the input image for each frame of the input image, and to each divided object , A motion vector calculation means for calculating a motion vector indicating the amount of movement of the object from the difference between the position coordinate of the object in the previous frame and the position coordinate of the previous frame, and for each object, the motion vector of the previous frame of the object Based on the position coordinates and the motion vector, an estimated position calculating means for calculating an estimated position that is estimated to be present in each object in the current frame; the estimated position; and an estimated position other than the estimated position for each object in the previous frame Features other than the estimated position of each object in the current frame Based on the symptoms weight, it is characterized in that it comprises a and object recognizing means for recognizing the same object between the current and previous frames.

  That is, the object included in the input image is divided from the input image for each frame using the image dividing means, and the feature amount such as “color”, “centroid position”, and “area” is calculated for the divided object, Recognizing the same object between frames based on the feature amount is important in realizing intelligent information processing technology. Note that “object” means a region in the input image that belongs to the same category in units of pixels.

  However, in such a conventional image processing method for recognizing the same object, it is difficult to recognize the same object between frames when there are a plurality of objects in the image. This is because if there are multiple objects in the image, those objects whose features are approximate may be included in the same frame, and the objects whose features are approximate are the same. This is because it may be recognized as an object.

  Therefore, the image processing apparatus of the present invention particularly includes a motion vector calculation unit, an estimated position calculation unit, and an object recognition unit. That is, the motion vector is calculated between the frames by the motion vector calculation means, and the estimated position of the object is calculated by the estimated position calculation means using the motion vector, thereby grasping the moving state of the object between the different frames. Can do.

  Furthermore, since the object recognition means recognizes the same object based on the estimated position, the same object can be recognized in consideration of the movement state of the object between different frames.

  In particular, for an object divided by the image dividing means, a motion vector is calculated by the motion vector calculating means, and the object recognition means detects the same object between frames using the motion vector. Correspondence between frames becomes possible. Therefore, unlike the method of simply detecting a moving object from difference information of images between frames, the present invention can recognize not only a moving object but also a stationary object. Furthermore, it is possible to detect an object even when all the captured images that the camera pans are moving.

  It should be noted that a situation in which the motion vector of a certain object suddenly changes due to circumstances such as collision between objects can be considered. In this case, an error may occur between the estimated position calculated by the estimated position calculation unit and the actual position of the object, and the object recognition unit may not be able to correctly recognize the same object between different frames.

  Therefore, the object recognition means recognizes the same object between different frames in consideration of feature quantities other than the estimated position. That is, even if the motion vector suddenly changes between frames, the feature quantity other than the motion vector related to a certain object is often the same between the frames. Therefore, according to the object recognition means of the present invention, the motion vector has changed. However, the same object can be recognized between different frames.

  Thus, according to the image processing apparatus of the present invention, even when there are a plurality of objects in an image, the same object can be correctly recognized between different frames.

  Further, the object recognition means calculates a distance index, which is a value based on the first difference value and the second difference value, between the target object of the current frame and each object of the previous frame, and the distance index is the smallest The first difference value is a difference value between coordinates indicating the estimated position of the object of interest in the current frame and coordinates indicating the actual position of each object in the previous frame. The second difference value is preferably a difference value between a feature quantity other than the estimated position related to each object in the previous frame and a feature quantity other than the estimated position related to the object of interest in the current frame. As the distance index, for example, Manhattan distance or Euclidean distance can be used.

  That is, for an object, even if the object moves between frames, the feature quantity of the object does not change so much. In other words, if the difference value of the feature amount between objects is calculated between different frames, it can be said that the difference value regarding the same object is minimized.

  The object recognition means of the present invention calculates a value based on the above difference value as a distance index, and recognizes the object having the smallest distance index as the same object, so that the same object can be appropriately recognized. it can.

  Further, the above-described distance index is calculated in consideration of a first difference value that is a difference value between the coordinates of the estimated position of the object of interest and the coordinates of the actual position of each object in the previous frame.

  In other words, unless the motion vector of the target object changes suddenly, the difference value between the coordinates of the estimated position of the target object and the coordinates of the actual position in the previous frame of the target object is the estimated position coordinate of the target object and the target object. It becomes smaller than the difference value with the position coordinate in the previous frame of the other object. That is, it can be said that the objects having the smallest first difference value between frames are the same object.

  And the object recognition means of this invention calculates the distance parameter | index which is a value based on the said 1st difference value, and recognizes the object from which the distance parameter | index is the minimum as the same object. Therefore, the same object can be recognized more appropriately.

  Further, in order to calculate the first difference value and the second difference value, it is necessary to calculate the difference value of the position coordinates or the difference value of the feature amount. The calculation of the difference value in this way can be realized with a relatively simple circuit configuration such as providing a subtractor. Therefore, according to the present invention, it is possible to recognize the same object with a simple circuit configuration.

  Furthermore, the feature amount other than the estimated position may include at least one of the number of pixels, the width, the height, and the color information.

  That is, the number of pixels, the width, the height, and the color information can be easily calculated as the feature amount of the object. That is, the number of pixels, the width, and the height can be easily calculated by counting the number of pixels included in the object, and the color information can also be easily calculated from the pixel values.

  Therefore, according to the above configuration, the feature amount of the object can be easily calculated, so that the recognition process of the same object by the object recognition unit can be simplified. Therefore, the same object can be recognized by a simpler process.

The feature quantities other than the estimated position are the number of pixels, the width, the height, and the color information, and the second difference value is a difference value for each of the number of pixels, the width, the height, and the color information. The object recognizing means has a distance index D _ij between the object i of the current frame and the object j of the previous frame, the first difference value Ip _ij , the difference value of the number of pixels Ia _ij , and the width The difference value of Iw _ij , the height difference value Ih _ij , and the color information difference value Ic _ij ,
D _ij = αIa _ij + β (Iw _ij + Ih _ij ) + γIp _ij + εIc _ij (in case of Manhattan distance)
Or
D _ij = √k
k = αIa _ij ² + βIw _ij ² + βIh _ij ² + γIp _ij ² + εIc _ij ² (in the case of Euclidean distance)
(Α, β, γ, ε are coefficients determined according to the respective sizes of Ia _ij , Iw _ij , Ih _ij , Ip _ij , and Ic _ij )
And the distance index D _ij is preferably calculated.

According to the above configuration, the distance index D _ij can be calculated by setting the coefficients α, β, γ, and ε as weights for each of the first difference value, the number of pixels, the width, the height, and the color information. . Therefore, it is possible to calculate a distance index by setting a weight function according to which of the first difference value, the number of pixels, the width, the height, and the color information is important.

  For example, if there is a need to recognize the same object with an emphasis on the number of pixels of the object, the coefficient α may be set larger than β, γ, and ε. Thereby, among the changes in the first difference value, the number of pixels, the width, the height, and the color information, the value of the distance index has the greatest influence on the change in the number of pixels. If the distance index with the weight set in this way is used, it is possible to recognize the same object with an emphasis on the difference value of the number of pixels of the object.

  Thus, according to the above configuration, the same object can be recognized by setting weights α, β, γ, and ε according to user needs.

  The coefficients α, β, γ, and ε may be set so as to satisfy α = 2β = 2γ = ε.

That is, among the difference values Ia _ij , Iw _ij , Ih _ij , Ip _ij , and Ic _ij , Iw _ij and Ih _ij are common as an index indicating the “size” of the object. As for Ip _ij , two parameters are included as a difference value of the x coordinate and a difference value of the y coordinate.

In other words, there are two parameters indicating the difference value of the object size and two parameters indicating the difference value of the coordinates, but there is only one parameter for Ia _ij and Ic _ij . Thus, when the number of parameters is different, object recognition biased to the “size” and “position” of the object may be performed.

  Therefore, in the above configuration, α = 2β = 2γ = ε is set. As a result, equal weight functions are set for “size”, “position”, “number of pixels”, and “color information” as the feature amount of the object. As a result, object recognition biased to a specific feature amount can be prevented, and the same object can be recognized by setting a balance equal to the change in each feature amount.

  Further, the object recognition means preferably uses a normalized value as the estimated position or other feature amount.

  According to the above configuration, a normalized value is used as the estimated position as the feature amount or the other feature amount. Therefore, since the number of bits of the feature value can be reduced in the arithmetic processing using the feature value, the same object can be recognized in a simpler process.

  As a normalization method, it is possible to use a method of dividing an actual value by the maximum value that the feature amount can take. According to this normalization method, the feature amount can be normalized to a value of 0 to 1, so that the arithmetic processing using the feature amount can be greatly simplified.

  Alternatively, as a normalization method, the actual value of the feature amount may be divided by a predetermined value based on the size of the area into which the input image is divided. For example, when an image is divided into a plurality of regions so that one region has a size of 16 × 16 pixels, the feature amount may be divided by 16. Also by normalization based on the size of the divided areas as described above, the number of bits of the feature amount can be sufficiently reduced, so that the same object can be recognized in a simple process.

  Furthermore, an image processing integrated circuit according to the present invention is characterized in that the image processing apparatus having the above configuration is integrated. Thereby, since the processing of each means in the image processing apparatus can be performed in parallel at high speed, hardware capable of detecting an object in real time can be provided.

  The image processing method of the present invention includes a first step of dividing a plurality of objects included in an input image from the input image for each frame of the input image, and for each divided object, before the object. A second step of calculating a motion vector indicating the amount of movement of the object from the difference between the position coordinate in the frame and the position coordinate of the previous frame; for each object, the position coordinate of the object in the previous frame; and the motion vector And a third step of calculating an estimated position, which is a position where each object is estimated to exist in the current frame, the estimated position, a feature amount other than the estimated position for each object in the previous frame, and the current frame Based on features other than the estimated position for each object, the current frame and the previous It is characterized in that it comprises a fourth step recognizes the same object with the frame.

  In the image processing method configured as described above, in each of the first step, the second step, the third step, and the fourth step, the image dividing unit, the motion vector calculating unit, the estimated position calculating unit, and the object recognizing unit of the present invention. The same processing is realized. Therefore, according to the image processing method of the present invention, it is possible to obtain the same effect as the image processing apparatus of the present invention.

  In addition, the effect similar to the image processing method of this invention can be obtained using a computer with the image processing program which makes a computer perform each step in the said image processing method. Furthermore, the image processing program can be executed on an arbitrary computer by storing the image processing program in a computer-readable recording medium.

  The image processing apparatus of the present invention includes motion vector calculation means, estimated position calculation means, and object recognition means. That is, the motion vector is calculated between the frames by the motion vector calculation means, and the estimated position of the object is calculated by the estimated position calculation means using the motion vector, thereby grasping the moving state of the object between the different frames. Can do.

  Furthermore, since the object recognition means recognizes the same object based on the estimated position, the same object can be recognized in consideration of the movement state of the object between different frames.

  Therefore, the object recognition means of the present invention can recognize the same object between different frames even if a plurality of objects exist in the image.

  Furthermore, the object recognition means recognizes the same object between different frames in consideration of feature quantities other than the estimated position. That is, even if the motion vector suddenly changes between frames, the feature quantity other than the motion vector related to a certain object is often the same between the frames. Therefore, according to the object recognition means of the present invention, the motion vector has changed. However, the same object can be recognized between different frames.

  Thus, according to the image processing apparatus of the present invention, there is an effect that the same object can be correctly recognized between different frames even when a plurality of objects are present in the image.

  In particular, for an object divided by the image dividing means, a motion vector is calculated by the motion vector calculating means, and the object recognition means detects the same object between frames using the motion vector. Correspondence between frames becomes possible. Therefore, unlike the method of simply detecting a moving object from difference information of images between frames, the present invention can recognize not only a moving object but also a stationary object. Furthermore, it is possible to detect an object even when all the captured images that the camera pans are moving.

[1. Configuration of image processing circuit]
As shown in FIG. 1, the image processing circuit (image processing apparatus) 1 according to the present invention includes an image division / feature amount calculation unit 2 and an object detection unit 3. Further, the image division / feature amount calculation unit 2 includes a combination weight calculation unit (image division unit) 4, a leader cell calculation unit (image division unit) 5, an image division cell network (image division unit) 6, and object information. And a calculator 7. The object detection unit 3 includes a search information storage unit 8, a position information calculation unit (motion vector calculation means, estimated position calculation means) 9, and a minimum distance search associative memory (object recognition means) 10.

  When the processing in the image processing circuit 1 is outlined, input image data is input to the image division cell network 6 of the image division / feature quantity calculation unit 2 and image division processing is performed. In this image division processing, the feature amount of the image of the corresponding object is calculated at the same time as the image division processing of one object is completed, and the value is sent to the object detection unit 3 and matched between frames (details will be described later). Is taken. Finally, the estimated position and motion vector of the next frame are calculated and stored in the memory from the matching result and the position information of the detected object. The functions of these blocks constituting the image processing circuit 1 will be described below.

[1-1. Configuration of image segmentation / feature amount calculation unit 2]
As described above, the image division / feature amount calculation unit 2 includes the connection weight calculation unit 4, the leader cell calculation unit 5, the image division cell network 6, and the object information calculation unit 7.

  When the input image data is a grayscale image, the connection weight calculation unit 4 is based on the luminance value. When the input image is a color image, the connection weight calculation unit 4 is based on the luminance value of each Red, Green, and Blue component. A connection weight that represents the closeness of luminance between pixels is calculated. The leader cell calculation unit 5 determines a leader cell as a starting point of the image division process described later. In order to determine the leader cell, the connection weight is calculated using the connection weight calculator 4 between the target pixel and each of, for example, eight pixels existing around the target pixel. If the sum of the coupling weights is larger than a threshold value designated in advance by the user, the target pixel is determined as a leader cell.

  The image segmentation cell network 6 performs image segmentation processing including an initialization phase, a self-ignition phase, an ignition phase, and a fire suppression phase. Details of the processing contents in these phases will be described later.

More specifically, as shown in FIG. 2, the image division cell network 6 includes a horizontal combination weight register WR (H), a vertical combination weight register WR (V), and an image division cell C _i (i is an integer). (See Patent Document 3 and Non-Patent Document 13).

Horizontal coupling weight register WR (H) and vertical coupling weight register WR (V) is disposed between the image segmentation cell C _i, are arranged so that the horizontal and vertical coupling weight register alternating. The horizontal connection weight register WR (H) holds four connection weights W _ij for the pixels in the diagonal direction and the horizontal direction, and the vertical connection weight register WR (V) stores the connection weight W for the pixels in the diagonal direction and the vertical direction. _It holds 4 _ij . Note that arrows in each connection weight register in FIG. 2 schematically represent the positional relationship of connection weights between stored cells.

The image division cell C _i actually executes the image division processing based on the output of the combination weight register WR (H) / WR (V).

  Furthermore, the image division | segmentation cell network 6 is provided with the global suppression part 11, as shown in FIG. The global suppression unit 11 takes a logical sum (OR) of global suppressors Zi (details will be described later) in adjacent image division cells to determine whether or not there is a newly fired cell for each row of the image division cells. A Zory signal, which is a signal indicating the above, is output. There is always one image-divided object in the range where the Zory signal rises in the image-dividing cell network 6.

  As shown in FIG. 4A, by outputting a Zorx signal that is a signal indicating whether or not there is a newly fired cell for each column of the image division cells, there exists an object in which the image division exists. A range of coordinates and y coordinates may be calculated. FIG. 4B shows a detailed configuration of the global suppression unit 11 in the i-th row.

  As shown in FIG. 5, the object information calculation unit 7 includes a min-max calculation unit 7a, an area / min-max calculation unit 7b, an object information storage unit 7c, and a feature amount calculation unit 7d. The information such as the feature amount of the object divided by the image division cell network 6 is calculated. Further, as shown in FIG. 5, the image division / feature amount calculation unit 2 is provided with a control circuit 12 that comprehensively controls processing in each block in the image division / feature amount calculation unit 2.

  The min-max calculation unit 7a is a circuit that calculates the size in the vertical direction (y-coordinate direction) of the image-divided object using the global suppression signal Zory output from the global suppression unit 11. In order to calculate the size of the divided object, a Zorx signal output by the improved global suppression unit 11 as shown in FIG. 4A may be used.

  As shown in FIG. 6, the min-max calculator 7a includes registers 7e, min-max detectors 7f, and decoders 7g.

  The register 7e stores the column number from which 1 is output as the Zorx signal. Further, as shown in FIG. 7, the min-max detection unit 7f outputs 1 as the minimum and maximum among the column numbers stored in the register 7e, and outputs the other portions as 0. Is. Further, as shown in FIG. 8, the decoder 7g converts the information of “1” output by the min-max detector 7f into information of the column number, and thereby the horizontal direction (x The size of the coordinate direction can be calculated.

Through the above processing, the min-max calculation unit 7a outputs x of the four coordinates x _{i, min} , x _{i, max} , y _{i, min} , y _{i, max} of the object i detected as shown in FIG. _{Find i, min} and x _{i, max} . Further, by using the information of x _{i, min} and x _{i, max} , the calculation range of the subsequent area calculation processing can be narrowed down. The detailed configuration of the min-max calculator 7a is shown in FIG.

The area / min-max calculation unit 7b calculates the area of the detected object and the remaining coordinate information in the range of x _{i, min} , x _{i, max} calculated by the min-max calculation unit 7a. These calculations are performed in parallel with the output of the division result of one object of the image division result.

Specifically, as shown in FIG. 11, the area / min-max calculation unit 7b includes registers 7h, counters 7i, and adders 7j. First, the control circuit 12 shown in FIG. 11 checks in the image division cell network 6 whether the division area is one column at a time from the position of x _{i, max} , and if the division area is one object, 1 is set. Otherwise, 0 is stored in the register 7h. At this time, the control circuit 12 outputs a signal from the register 7h to the counter 7i. Such a check process of the divided area of one object and an output process of “0” or “1” are repeated up to the place of x _{i, min} .

As a result, when the output process of “0” or “1” is completed between (x _{i, min} , x _{i, max} ), the counter 7 i stores the pixel of one object for each row in the image division cell network 6. Since the number is input, the area of one object currently divided can be calculated as the pixel number a _j by adding the values of all the counters 7i... Using the adder 7j.

Further, the area / min-max calculator 7b obtains the values of y _{i, min} , y _{i, max} simultaneously with the area of the object. More specifically, y _{i, min} , y _{i, max} are detected when a line having a value stored in the register 7h of 1 or more for area calculation is detected, and the y coordinate at the top is defined as y _{i. , max} , and the lowest y coordinate as y _{i, min ,} the values of y _{i, min} , y _{i, max} are obtained. The detailed configuration of the area / min-max calculator 7b is shown in FIG.

As shown in FIG. 13, the object information storage unit 7c is composed of registers 7k..., And each register 7k includes the object pixel number a _i calculated as described above, x _{i, min} , x Four coordinates _{i, max} , y _{i, min} , y _{i, max} are stored.

The feature amount calculation unit 7d selects the size (width w _i (t), height h _i (t)), position (x _i (t)) of the object from the coordinate data stored in the object information storage unit 7c. , y _i (t)) and color information c _i (t). Although details of the calculation method of the size and position information will be described later, normalizing the calculation result to a range of 0 to 1 by a divider may increase the cost of hardware.

  Therefore, as a result of simulating the method of reducing the number of bits, it was found that the number of bits of information can be reduced by dividing the input image into blocks based on an area of 16 × 16 pixels in size. . For example, as shown in FIG. 14, if the input image has a size of 240 × 320 pixels, the vertical direction is divided into 15 blocks and the horizontal direction is divided into 20 blocks. By using the method of reducing the number of bits in this way, the number of bits of each information can be reduced as shown in Table 1.

  Note that the color information of the detected object may be shifted in the range of 0 to 8 even when compared with the same object, so the number of bits corresponding to this range is reduced.

  Further, as shown in FIG. 15, the feature amount calculation unit 7d includes subtracters 7l,... For adding coordinates and sizes, and shifters 7n, for reducing the number of bits. Further, the color information is obtained by transferring the coordinate information obtained so far to the input memory 7o in the feature amount calculation unit 7d to receive the color information of four points and averaging them. The value calculated by the feature amount calculation unit 7d is stored in the object information storage unit 7c.

  As described above, the image division / feature amount calculation unit 2 includes the image division cell network 6, the min-max calculation unit 7a, the area / min-max calculation unit 7b, the object information storage unit 7c, and the feature amount calculation unit. 7d. A detailed configuration of the image division / feature amount calculation unit 2 is shown in FIG.

[1-2. Configuration of Object Detection Unit 3]
As shown in FIG. 17, the object detection unit 3 includes a search information storage unit 8, a motion vector calculation unit (motion vector calculation unit) 9a, an estimated position calculation unit (estimated position calculation unit) 9b, and a minimum distance search association. And a memory (see Non-Patent Document 9 and Non-Patent Document 12). Note that the position information calculation unit 9 shown in FIG. 1 includes a motion vector calculation unit 9a and an estimated position calculation unit 9b.

  When the processing in the object detection unit 3 is outlined, the feature quantity of the image at a certain time t calculated by the image segmentation / feature quantity calculation unit 2, that is, the area, size, position, and color information of the image are obtained as a search information storage unit 8. Are stored in the minimum distance search associative memory 10.

  Then, after matching is performed in the minimum distance search associative memory 10, information on the matched object and position information in the search information storage unit 8 are sent to the motion vector calculation unit 9a. The amount of object movement is calculated.

  From the calculated movement amount of the object and the current position of the object, the estimated position at the time of frame t + 1 is calculated by the estimated position calculation unit 9b. Hereinafter, the function of each block constituting the object detection unit 3 will be described.

  The search information storage unit 8 stores the number of pixels, size, position, and color information as the feature quantities of the object divided in the current frame, and outputs these feature quantities to the minimum distance search associative memory 10 as matching data. To do. The circuit configuration of the search information storage unit 8 is shown in FIG.

  The minimum distance search associative memory 10 matches the retained feature quantity of the object of the previous frame t−1 with the object information of the object of the current frame t stored in the search information storage unit 8, and The closest object between the current frame and the current frame is searched.

  FIG. 19 shows the configuration of the minimum distance search associative memory 10. As shown in FIG. 19, the minimum distance search associative memory 10 includes a memory area 10a, a winner lineup amplifier (hereinafter referred to as WLA) 10b that performs a winner-loser distance amplification, and a winner take-all circuit (hereinafter referred to as WTA). 10c.

Further, the memory area 10a has an integrated unit comparator UC _{i, j} (i = 1 to R, j = 1 to W, the ranges of subscripts i and j are the same below) of R rows and W columns. The storage cell SC _{i, j} includes a word comparator WC _i weighted for word comparison.

Further, a row decoder 10d of R rows is connected to the left side of the memory area 10a, and a word search unit 10e for inputting input data (K × W bits) is arranged on the upper side. A column decoder 10f of K × W column is connected to the lower side, and reading / writing is performed for each storage cell. The K × W bit reference word is written in each storage cell SC _{i, j} in advance _, and the minimum distance reference word most similar to the K × W bit input word input to the word search unit 10e is read as a winner. It is. For details of the minimum distance search associative memory 10, refer to Patent Document 4.

  The information stored in the minimum distance search associative memory 10 is as shown in Table 2, and the information actually used for the search is the number of pixels, size, position information, and color information. A motion vector, which is information that is not used, is calculated as a calculation result of a motion vector calculation unit 9a described later.

As shown in FIG. 20, the motion vector calculation unit 9a includes a subtracter, and the position information (x _n (t−1), y _n (t−1)) of the search information storage unit 8 and the minimum A motion vector (mv _x (t), mv _y (t)) is obtained from the position information (x _n (t), y _n (t)) of the object matched by the distance search associative memory 10 It is.

  Specifically, the motion vector calculation unit 9a obtains a motion vector based on the following formula.

mv _x (t) = x _n (t) −x _n (t−1)
mv _y (t) = y _n (t) −y _n (t−1)
The calculation result of the motion vector calculation unit 9a is overwritten on the information in the minimum distance search associative memory 10 for which matching has been taken.

As shown in FIG. 21, the estimated position calculation unit 9b includes an adder, and as shown in the following equation, the motion vector (mv _x (t), mv _y calculated by the motion vector calculation unit 9a (t)) and the position information (x _n (t), y _n (t)) of the current frame, the estimated position (x _n (t + 1), y _n (t + 1) in the next frame is added. )) Is calculated.

x _n (t + 1) = x _n (t) + mv _x (t)
y _n (t + 1) = y _n (t) + mv _y (t)
Thus, the object detection unit 3 includes the search information storage unit 8, the motion vector calculation unit 9a, the estimated position calculation unit 9b, and the minimum distance search associative memory 10. A detailed configuration of the object detection unit 3 is shown in FIG.

[2. Image processing algorithm)
Next, an image processing algorithm (image processing method) by the image processing circuit 1 having the above configuration will be described. As shown in FIG. 23, the image processing algorithm of this embodiment is composed of four blocks: (1) image segmentation, (2) feature extraction, (3) associative matching, and (4) motion detection.

  First, the flow of the image processing algorithm will be outlined. When the image processing algorithm is started (step 0, the following steps are simply referred to as “S”), the control circuit 12 (see FIG. 5) checks whether or not input frame data exists (S1). .

  At this time, if there is no input frame data, the image processing algorithm ends. On the other hand, when input frame data exists, image division processing is performed on the frame. Here, when one region (one object) is image-divided, a feature extraction process described later is executed from the result. A description example of an algorithm for confirming whether or not input frame data exists will be shown below.

  After the feature extraction process is executed, an associative matching process is executed based on the information of one divided object. At the same time as the associative matching process, the next area division process starts for the input image. If there is no newly divided area, the process returns to the input frame data check process in S1.

  In the associative matching process, a distance index of a feature amount (for example, Manhattan distance, Euclidean distance; details will be described later) is calculated for each object between frames based on the result of the feature extraction process, and an object having the smallest distance index is calculated. Recognize as the same object. Thereafter, the movement amount is calculated from the position information of the original object and the searched object, and the motion vector is obtained, and then the process returns to the image division process. If there is no matching object in the associative matching process, the divided object is recognized as a new object, and the process returns to the image dividing process. The details of each process in the image processing algorithm will be specifically described below.

[2-1. (Image segmentation algorithm)
In this embodiment, an image segmentation algorithm based on a cell network (see Patent Document 3) is used for object extraction. Features of this image segmentation algorithm include that image segmentation can be realized with relatively simple processing, that the number of processing cycles is small and high speed, and that parallel processing is possible. Furthermore, this image segmentation algorithm is divided into four stages: an initialization phase, a self-ignition phase, a flash phase, and a fire suppression phase. Details of the processing in each phase will be described below.

  First, in the initialization phase (S2), a variable z (global suppressor) for determining whether or not there is a cell being ignited is set to zero. When z = 1, this means that there is a cell that is firing, that is, the image segmentation process of one region is continuing, and when z = 0, there is no cell that is firing, that is, one This represents the end of the image segmentation process for the area.

Each cell i is provided with a variable z _i indicating whether or not there is a state change, and z _i = 1 only when transitioning from the non-ignition state to the ignition state, and z _i = 0 otherwise. Based on this variable z _i , the value of the global suppressor z (t) in the frame t is defined as z (t) = ∪ _{∀ i} z _i (t) as the logical sum of all z _i . Here, ∪ represents a logical sum (OR) operator.

In the initialization phase, grayscale image data is input to the connection weight calculation unit 4 (see FIG. 5 and the like) as input image data. Grayscale image data is the luminance value of an image, and the value is in the range of 0-255. Then, in the initialization phase of S2, based on the luminance value of the image data, the coupling weight calculation unit 4 calculates a coupling weight W _ik that represents the closeness of luminance between pixels. This connection weight W _ik is expressed by the following equation.

W _ik = I _max / (1+ | I _i −I _k |), k∈N (i)
Note that I _i and I _k are the luminance values of the pixels i and k, I _max is the maximum luminance value of the pixels, and when the luminance value is expressed in 8 bits, I _max = 255.

Further, in the initialization phase in S2, the leader cell calculation unit 5 (see FIG. 5 and the like) determines whether or not a leader cell exists (S3). That is, the leader cell calculation unit 5 calculates the sum Σ _{k∈N (i)} W _ik of the connection weights W _{ik in} the vicinity of the _target pixel, and if the value is larger than the threshold value φ _p , the leader cell exists. to decide.

Further, Σ _{k∈N (i)} W _ik is the larger than phi _p, and sets the variable p _i indicating whether the self-ignition in 1, to determine the leader cell as the self-excitable cell. If Σ _k ∈ _{N (i)} W _ik is equal to or smaller than φ _p (Σ _k ∈ _{N (i)} W _ik ≤ φ _p ), it is initialized to p _i = 0 (self-ignition is impossible) To do.

In the initialization phase in S2, for all the cells, a variable x _i (t) indicating whether or not the cells are firing is set to zero. Note that this variable x _i (t) is set to 1 when the cell is in an ignition state.

  When it is determined in S3 that a leader cell exists, the image processing algorithm proceeds to a self-ignition phase (S4) in which image division processing is actually performed. In the self-ignition phase, one of the leader cells determined in the initialization phase is set in an ignition state.

  Further, in the self-ignition phase in S4, the control circuit 12 determines whether or not there is a cell that can be ignited (S5).

In other words, the control circuit 12 determines the sum of coupling weights S _i (t) = between the cells in the firing state among the eight cells adjacent to the non-firing state cell and the non-firing state cells. Σ _{kεN (i)} (W _ik × x _k (t)) is calculated, and the sum of the coupling weights is compared with a predetermined threshold value φ _Z. If the sum of the combined weights is larger than a predetermined threshold value, the non-ignited cell is determined as a newly ignitable cell and ignited (S6).

  On the other hand, if the sum of the coupling weights is smaller than the predetermined threshold value, it is determined that there is no cell that can be ignited, and the fire extinguishing process of S7 is performed. The processes in S5 to S7 are performed until there are no non-ignited leader cells. When there are no more leader cells, the image division process is terminated.

  FIG. 24A and FIG. 24B show the result of image division of a grayscale image by the above image division processing. FIGS. 24C and 24D show the result of image division performed on the color image by the image division process. As shown in FIGS. 24C and 24D, according to the image processing algorithm described above, a color image can be divided with high accuracy.

  FIG. 25 shows the relationship between the number of pixels and the processing time when image division processing is performed using the above-described image division algorithm. In the simulation of FIG. 25, a plurality of sample images were given as input data, and the square root of the number of pixels (representing the length of one side of the image) and the longest and shortest processing times required for image division were measured. In addition, in order to estimate the worst case processing time using a calculation formula, a regular image in a black and white grid pattern is used as a theoretical worst case in which the number of image segmentation areas is increased, which is very rare for natural images. The expected image division processing time when assumed is also shown in the graph.

  A description example of the algorithm for image division processing is shown below.

[2-2. (Feature extraction processing algorithm)
After the above-described image division processing is completed, the image processing algorithm shifts to feature extraction processing. In the feature extraction processing, first, as the feature amount of the object divided by the image division processing, the min-max calculation unit 7a (see FIG. 5) uses the left end coordinates x _{i, min} of the object i divided at time t. (t) and the right end coordinate x _{i, max} (t) are obtained (S8). FIG. 9 shows the relationship between the object i and the coordinates x _{i, min} (t) and the coordinates x _{i, max} (t).

After S8, the area / min-max calculator 7b (see FIG. 5) uses the values of the area a _i (t) (number of pixels), coordinates y _{i, min} , and y _{i, max} as the feature quantity of the object i. Is obtained (S9). As shown in FIG. 9, the coordinates y _{i, min} are the coordinates of the lower end of the object i, and the coordinates y _{i, max} are the coordinates of the upper end of the object i.

Further, after S9, the feature amount calculator 7d (see FIG. 5) uses the size (width: w _i (t), height: h _i (t)) of the object i, position information (x _i (t), y _i (t)) and color information c _i (t) are obtained (S10).

Of the above-described object characteristics, the area a _i (t) is obtained by counting the number of pixels when outputting the image division result. The size and position information are calculated using the maximum and minimum values of the x and y coordinates of each object in the current frame t.

That is, the lower left coordinates of the divided object i are (x _{i, min} (t), y _{i, min} (t)), and the upper right coordinates are (x _{i, max} (t), y _{i, max} (t) ), The size (w _i (t), h _i (t)) and the position coordinates (x _i (t), y _i (t)) are calculated based on the following equations.

w _i (t) = x _{i, max} (t) −x _{i, min} (t)
h _i (t) = y _{i, max} (t) −y _{i, min} (t)
x _i (t) = {x _{i, max} (t) + x _{i, min} (t)} / 2
y _i (t) = {y _{i, max} (t) + y _{i, min} (t)} / 2
The color information is the color information (R, G, B) of each coordinate in x _{i, max} (t), x _{i, min} (t), y _{i, max} (t), y _{i, min} (t). Is obtained from the input memory.

  The associative matching process described later is performed using the feature amount obtained by stepping on S8 to S10.

  A description example of the algorithm for feature extraction processing is shown below.

[2-3. (Associative matching algorithm)
The associative matching process writes the information calculated in the feature extraction process from the search information storage unit 8 (see FIG. 17) to the minimum distance search associative memory 10 (see FIG. 17) (S11). The distance index is calculated and an object having the smallest distance index is detected (S12).

Here, the distance index D _ij (t) at time t between the object i and the object j is the absolute value of the difference between the number of pixels, the size, the position, and the color information of the object i, j, Ia _ij ( t), Is _ij (t), Ip _ij (t), and Ic _ij (t).

That is, if the Manhattan distance is used as a distance index, D _ij (t) is expressed by the following equation.

D _ij (t) = αIa _ij + βIs _ij + γIp _ij + εIc _ij
Note that α, β, γ, and ε represent weights for information.

On the other hand, if Euclidean distance is used as a distance index, D _ij (t) is expressed by the following equation.

D _ij = √k
k = αIa _ij ² + βIs _ij ² + γIp _ij ² + εIc _ij ²
Further, here, as position information (x _i (t), y _i (t)) of the current frame t, as shown in FIG. 26, position coordinates (x _i (t−1), y _i (t-1)) to the motion vector of that time _{(mvx i (t-1)} , value estimated position plus the _{mvy i (t-1))} (x i (t) ', y i (t ) ′), But since the motion vector cannot be calculated for the first frame (= 0), the position information of the first frame is used as it is as the position information of the second frame.

An expression for obtaining (x _i (t) ′, y _i (t) ′) is shown below.

x _i (t) '= x _i (t−1) + mvx _i (t−1)
y _i (t) '＝ y _i (t−1) + mvy _i (t−1)
The distance index is calculated from these results. For size and position information, there are two parameters such as w _i (t), h _i (t), x _i , and y _i , respectively. And the color information has only one parameter. Therefore, the weight is adjusted by doubling the number of pixels and the color information.

In other words, the absolute value of the difference in the number of pixels is Ia _ij (t), the absolute value of the difference in size information about width and height is Iw _ij (t), Ih _ij (t), x coordinate and y coordinate, respectively. The distance index D _ij is expressed by the following equation, where Ix _ij (t) and Iy _ij (t) are the absolute values of the positional information differences and Ic _ij (t) are the absolute values of the color information differences.

Dij = 2Ia _ij (t) + (Iw _ij (t) + Ih _ij (t)) + (Ix _ij (t) + Iy _ij (t)) + 2 Ic _ij (t)
The absolute value Ic _ij of the color information difference is expressed by the following equation.

Ic _ij = max (│Cr _i -Cr _j │, │Cg _i -Cg _j │, │Cb _i -Cb _j │)
Each of Cr, Cg, and Cb is red, green, and blue color information, and indicates whether the letter i or j attached to the subscript is the color information of object i or object j. ing.

  However, the maximum value of each information is as shown in Table 3 (in the case of QVGA size). If the calculation result is used as it is, the influence of the value of the number of pixels on the Manhattan distance becomes too large. Balance the whole by normalizing to a range of 1.

  For example, normalization can be performed by dividing the calculated value by the maximum value of each value. That is, the Manhattan distance normalized as a distance index between the objects i and j is expressed by the following equation. Then, the minimum distance search associative memory 10 calculates the normalized Manhattan distance for each object in the frame (S12), and recognizes the one with the smallest Manhattan distance as the same object.

D _ij = 2 × Ia _ij / 2 ¹⁷ + (Iw _ij / 2 ⁹ + Ih _ij / 2 ⁸ ) + (Ix _ij / 2 ⁹ + Iy _ij / 2 ⁸ ) + 2 × Ic _ij / 2 ⁸
If normalization is performed by dividing the calculation result in this way, the hardware cost may increase as described above. Therefore, as shown in Table 1, the input image is determined based on a region having a predetermined size. The number of bits of each information may be reduced by dividing.

  If the minimum distance index value is equal to or greater than the predetermined value, the minimum distance search associative memory 10 recognizes the object as a newly appearing object in the image (S13).

  A description example of the associative matching processing algorithm is shown below.

[2-4. (Motion detection processing algorithm)
In the motion detection process, first, a motion vector is calculated by the motion vector calculation unit 9a (see FIG. 17) (S14). That is, the position information of the object to be searched at frame t is (x _n (t), y _n (t)), and the position information of the detected object at frame t−1 is (x _n (t−1), y _n (t−1)), the motion vector (mv _x (t), mv _y (t)) in the frame t is expressed by the following equation.

mv _x (t) = x _n (t) −x _n (t−1)
mv _y (t) = y _n (t) −y _n (t−1)
Using the next motion vector, the estimated position (x _n (t−1), y _n (t−1)) in the next frame t + 1 used for object detection is converted into an estimated position calculation unit 9b (FIG. 17). Reference). The estimated position (x _n (t + 1), y _n (t + 1)) in the next frame t + 1 is expressed by the following equation.

x _n (t + 1) = x _n (t) + mv _x (t)
y _n (t + 1) = y _n (t) + mv _y (t)
Then, the estimated position (x _n (t + 1), y _n (t + 1)) calculated in this way is written in the minimum distance search associative memory 10 from the estimated position calculation unit 9b (S16), thereby moving the motion. The detection process ends.

  A description example of the algorithm of motion detection processing is described below.

[3. Simulation of image processing algorithm and results]
The inventors of the present invention simulated the image processing algorithm of the present invention on an actual image using the image processing circuit 1 having the above configuration, and the results will be described below. This simulation was performed using MATLAB normalized by the values in Table 1 for all information amounts and using Mathworks MATLAB.

[3-1. Motion detection processing for continuous still images)
First, motion detection by the image processing algorithm of the present invention was performed on an image in which three objects move from left to right (see FIG. 27). It should be noted that the image obtained by performing Morphology processing (see Non-Patent Document 14) on the input image is divided into the first frame image shown in FIGS. 28 (a) to 28 (c) and 28 (a). The results are shown in FIGS. 29 (a) to 29 (c).

  In addition, as an example of the feature amount calculated from these images, Table 4 shows the result of calculating the feature amount related to the object 3 in the third frame and the object in the second frame.

Note that the label (t, i) in Table 4 represents the object number i of the t-th frame, and the area, width, height, position, and color information are the feature values obtained from the objects after image division. Normalized by α = 1/16 ² , β = γ = 1/16, ε = 1/8 and rounded to the first decimal place.

  In Table 4, the Manhattan distance and the Euclidean distance as the distance index are also shown, and these distance indices are the distance indices between the respective objects and the object labeled (3, 3). Show.

For example, the Manhattan distance between the object and (3,3) and the object (2, 3) When _{D M,}
D _M = | 6-7 | + | 2-3 | + | 3-3 | + | 13-12 | + | 12-11 | + | 4-5 | + | 0-1 |
= 6
It becomes.

Also, if Euclidean distance between the object (3, 3) and the object (2, 3) is D _E ,
D _E = √k
k = (6-7) ² + (2-3) ² + (3-3) ² + (13-12) ² + (12-11) ² + (4-5) ² + (0-1) ²
And is obtained as D _E = 2.

In calculating _DE , the first decimal place is rounded off.

  In the table described below, the label, the Manhattan distance, and the Euclidean distance are described in the same manner as in Table 4.

  From Table 4, it can be seen that the distance index has the smallest value with the object with the label (2, 3). This indicates that the object of the label (3, 3) and the object of the label (2, 3) are recognized as being the same, and the second frame image and the third frame object are recognized. It can be seen that the same object is correctly recognized between the images.

  In the motion detection process, the estimated position of the current frame is calculated based on the motion vector calculated in the previous frame, and the distance index is calculated based on the difference between the estimated position and the actual position. Therefore, as shown in FIGS. 30A to 30E, motion detection processing using the image processing algorithm of the present invention is performed on an image in which the motion vector actually changes while the object is moving. With this process, it is possible to verify whether the object can be correctly recognized from other feature amounts even if the estimated position changes.

  In FIGS. 30A to 30E, the blue object 1 on the left side and the red object 2 on the right side collide at the third frame (see FIG. 30C), and the object 1 at the fourth frame. FIG. 30 is an image showing a series of flows in which the moving direction of the object 2 changes by 180 ° (see FIG. 30D). Table 5 shows the object characteristics and the differences between the objects 1 and 2 after the collision.

From Table 5, the Manhattan distance regarding the position (x _i ) between the object with the label (4, 1) and the object with the label (3, 1) is 5 (= 13−8). This Manhattan distance is greater than the Manhattan distance 3 (= 13-10) for the position (x _i ) between the object labeled (4, 1) and the object labeled (3, 2). . Similarly, the Euclidean distance is related to the position (x _i ) between the object with the label (4, 1) and the object with the label (3, 1). It is larger than that relating to the position (x _i ) between the attached object and the labeled object (3, 2).

However, the Manhattan distance for color information (C _i ) between the object with label (4,1) and the object with label (3,2) is labeled (4,1). It is greater than the Manhattan distance for color information (C _i ) between the object and the object labeled (3, 1). The same applies to the Euclidean distance related to the color information (C _i ).

  Therefore, the distance index in consideration of all the feature quantities is smaller in the distance index between the object with the label (4, 1) and the object with the label (3, 1). Therefore, according to the image processing algorithm of the present invention, it can be verified that the same object can be recognized even if the motion vector changes during the movement of the object.

[3-2. Motion detection processing for moving images)
Next, the result of applying the image processing algorithm of the present invention to an image cut out frame by frame from the actually captured moving image will be described.

  First, motion detection processing was performed on a moving image in which a plurality of objects are moving randomly (see FIGS. 31A to 31K). FIG. 32 shows an image obtained as a result of dividing the first frame image (see FIG. 31A).

  Among these images, the change of the object 5 in the seventh frame (the object of the label (7, 5), see FIG. 31 (g)) is the largest, so between this object and each of the objects in the sixth frame Table 6 shows the distance index calculated in (5).

  As shown in Table 6, among the distance indices between the object with the label (7, 5) and the object with each of the labels (6, 1) to (6, 6), the label ( The distance index between the object labeled with (7, 5) and the object labeled (6, 5) is the smallest. From this result, it was verified that the same object can be recognized if the motion detection process of the present invention is performed on a moving image in which a plurality of objects are moving randomly.

  Next, with respect to images (FIGS. 33 (a) to 33 (f) and FIGS. 34 (a) to 34 (f)) in which the motion vector changes suddenly when the ball collides with the floor and rebounds. The image processing algorithm of the present invention was applied. Note that the results of image segmentation of the image shown in FIG. 33 (a) are shown in FIGS. 35 (a) to 35 (d), and the results of image segmentation of the image shown in FIG. 34 (a) are shown in FIGS. As shown in FIG.

  In addition, Table 7 shows the result of calculating the distance index for the images obtained by dividing the images of FIGS. 33 (a) to 33 (f), and the images of FIGS. 34 (a) to 33 (f) are divided. Table 8 shows the result of calculating the distance index for the image.

  As shown in Table 7, among the distance indices between the object of label (4, 3) and each of the objects of label (3, 1) to label (3, 4), the object of label (4, 3) And the distance index between the object of the label (3, 3) is the smallest. As shown in Table 8, among the distance indices between the object of label (5, 2) and each of the objects of label (4, 1) to label (4, 4), label (5, 2) And the object of the label (4, 2) has the smallest distance index.

  Therefore, it can be seen from Tables 7 and 8 that the image processing algorithm of the present invention can accurately recognize the same object even when the motion vector changes rapidly.

  Furthermore, the image processing algorithm of the present invention is also applied to a moving image in which small objects are moving randomly (see FIGS. 37A to 37F).

  The results of dividing the image of FIG. 37A are shown in FIGS. 38A to 38D. As can be seen from FIG. 38 (a), the ball existing in the vicinity of the center in the image of FIG. 37 (a) (a ball not hatched in the same figure) could not be detected.

  Among the objects detected in the image of FIG. 37 (a), a distance index is set between the object of label (3, 1) and each of the objects of label (2, 1) to label (2, 4). The calculated results are shown in Table 9.

  As shown in Table 9, among the distance indexes between the object of label (3, 1) and each of the objects of label (2, 1) to label (2, 4), the object of label (3, 1) And the distance index between the object of the label (2, 1) is the smallest.

  Therefore, it can be seen from Table 9 that the same object can be recognized by executing the image processing algorithm of the present invention even on a moving image in which a plurality of small objects are moving randomly.

[3-3. Motion detection processing for natural images)
Next, a simulation result for an image (see FIGS. 39A to 39C) in which a person is actually walking in a frame is shown. Also, the results of dividing the image shown in FIG. 39A are shown in FIGS. 40A to 40D.

  As shown in FIGS. 40 (a) to 40 (d), the image shown in FIG. 39 (a) is the image of the wall shown in FIG. 40 (a), the image of the floor shown in FIG. 40 (b), and FIG. The image is divided into an image of the person a shown in (c) and an image of the person b shown in FIG.

  Of these divided images, the images that are actually moving are only the image of the person a shown in FIG. 40C and the image of the person b shown in FIG. The shape is changed only by the movement of the person a and the person b. Table 10 shows the results of calculating the distance index for the images shown in FIGS. 39 (a) to 39 (c).

  As shown in Table 10, among the distance indices between the object of label (3, 3) and each of the objects of label (2, 1) to label (2, 4), the object of label (3, 3) And the distance index between the object of the label (2, 3) is the smallest.

  Therefore, it can be seen from Table 10 that the same object can be recognized by executing the motion detection process of the present invention even on a moving image in which a person is actually walking in a frame.

  Furthermore, in an image in which it is generally difficult to recognize the same object, a walking person (object) overlaps another person (object) in the frame (see FIGS. 41A to 41H). On the other hand, the motion detection process of the present invention was performed.

  When the image of FIG. 41A is divided, it is divided as shown in FIGS. 42A to 42D. Further, when the image of FIG. 41D is divided, it is divided as shown in FIGS. 43A to 43C. Note that, as shown in FIG. 42 (d), the upper body of the person stopped in the first frame (see FIG. 41 (a)) is not divided because there is no leader cell due to the gradation of the object, or in combination This may be because the weight is small and cannot be recognized as an object.

  Regarding these division results, the simulation results of the motion detection processing for the third object in the third frame, the third object in the fourth frame where the number of objects change, and the third object in the seventh frame are shown in Tables 11 and 11 below. 12 and Table 13.

  From Table 11, it can be seen that the distance index between the object of the label (3, 3) and the object of the label (2, 3) is the smallest. In Table 12, the distance index between the object of the label (4, 3) and the object of the label (3, 3) is the smallest. Furthermore, in Table 13, the distance index between the object of the label (7, 3) and the object of the label (6, 3) is the smallest. From these results, if the image processing algorithm of the present invention is applied to an image in which a moving object overlaps with other objects as shown in FIGS. 41 (a) to 41 (h). It was verified that the same object can be recognized.

  Furthermore, the result of executing the image processing algorithm of the present invention on the image of the minicar shown in FIGS. 44 (a) to 44 (f), which is an example of an image that is difficult to divide, will be described. 45A to 45C show the results of the division processing performed on the first frame image (see FIG. 44A).

  As shown in FIGS. 45 (a) to 45 (c), the object after the image division is greatly different in shape from the object of the actual image. Table 14 shows the result of motion detection processing performed on the images shown in FIGS. 45 (a) to 45 (c).

  Table 14 shows that the distance index between the object of label (4, 2) and the object of label (3, 2) is the smallest. Therefore, although the accuracy of image division with respect to the input image is not so good, it was verified that the same object could be correctly recognized between frames from the feature amount of the object.

  Next, the results of executing the image processing algorithm of the present invention will be described with respect to images (see FIGS. 46 (a) to 46 (f)) taken of a road on which a car is running. In addition, FIG. 47 (a) and FIG. 47 (b) show the results of performing the division processing on the first frame image (see FIG. 46 (a)).

  As can be seen from FIGS. 47 (a) and 47 (b), the image of the running car is the image of the car roof shown in FIG. 47 (a) and the hood part of the car shown in FIG. 47 (b). Divided into images. Table 15 shows the results of applying the image processing algorithm of the present invention based on the information obtained from FIGS. 47 (a) and 47 (b).

  From Table 15, it can be seen that the distance index between the object of the label (4, 1) and the object of the label (3, 1) is the smallest. From these results, it can be confirmed that the same object can be correctly recognized according to the image processing algorithm of the present invention also for the images shown in FIGS.

[4. Supplement)
In each processing step in the image processing algorithm of the present invention, a calculation unit such as a CPU executes a program stored in a storage unit such as a ROM (Read Only Memory) or a RAM, and an input unit such as a keyboard or an output such as a display. It can be realized by controlling a communication means such as an interface circuit or the like.

  Therefore, various processes of the image processing circuit according to the present embodiment can be realized simply by a computer having these means reading the recording medium storing the program and executing the program. In addition, by recording the program on a removable recording medium, the various functions and various processes described above can be realized on an arbitrary computer.

  As the recording medium, a memory (not shown) such as a ROM may be used as a program medium for processing by the microcomputer, and a program reader is provided as an external storage device (not shown). It may be a program medium that can be read by inserting a recording medium therein.

  In any case, the stored program is preferably configured to be accessed and executed by the microprocessor. Furthermore, it is preferable that the program is read out, and the read program is downloaded to the program storage area of the microcomputer and the program is executed. It is assumed that the download program is stored in the main device in advance.

  The program medium is a recording medium configured to be separable from the main body, such as a tape system such as a magnetic tape or a cassette tape, a magnetic disk such as a flexible disk or a hard disk, or a disk such as a CD / MO / MD / DVD. Fixed disk system, card system such as IC card (including memory card), or semiconductor memory such as mask ROM, EPROM (Erasable Programmable Read Only Memory), EEPROM (Electrically Erasable Programmable Read Only Memory), flash ROM, etc. In particular, there are recording media that carry programs.

  In addition, if the system configuration is capable of connecting to a communication network including the Internet, the recording medium is preferably a recording medium that fluidly carries the program so as to download the program from the communication network.

  Further, when the program is downloaded from the communication network as described above, it is preferable that the download program is stored in the main device in advance or installed from another recording medium.

  In addition, this invention is not limited to embodiment mentioned above, A various change is possible in the range shown to the claim.

  The image processing apparatus according to the present invention includes an image recognition system, a moving object detection system, a digital camera, a digital video camera, a robot vision, a face recognition authentication system, a security system, an artificial intelligence system, and the like. It can be applied as an extraction integrated circuit.

It is a block diagram which shows the structure of the image processing circuit which concerns on one Embodiment of this invention. It is a figure which shows the structure of the image division cell network in the image processing circuit of FIG. It is a figure which shows the structure of the global suppression part in the image division cell network of FIG. (A) is a figure which shows the state which outputs a Zorx signal for every column of an image division cell, (b) is a figure which shows the detailed structure of the i row in the global suppression part of FIG. FIG. 2 is a block diagram illustrating in detail a configuration of an object information calculation unit in the image processing circuit of FIG. 1. It is a figure which shows the structure of the min-max calculation part in the object information calculation part of FIG. It is a figure which shows the structure of the min-max detection part in the min-max calculation part of FIG. It is a figure which shows the structure of the decoder in the min-max calculation part of FIG. It is a figure which shows the relationship between the object divided | segmented by the image division | segmentation cell network of FIG. 1, and a feature-value. It is a figure which shows the structure of the min-max calculation part in the object information calculation part of FIG. It is a figure which shows the structure of the area / min-max calculation part in the object information calculation part of FIG. It is a figure which shows the structure of the area / min-max calculation part in the object information calculation part of FIG. It is a figure which shows the structure of the object information storage part in the object information calculation part of FIG. It is a figure which shows the state which divides | segments an input image on the basis of a 16 * 16 pixel area | region. It is a figure which shows the structure of the feature-value calculation part in the object information calculation part of FIG. FIG. 2 is a diagram illustrating a detailed configuration of an image division / feature amount calculation unit in the image processing circuit of FIG. 1. It is a figure which shows the structure of the object detection part in the image processing circuit of FIG. It is a figure which shows the structure of the search information storage part in the object detection part of FIG. It is a figure which shows the structure of the minimum distance search content addressable memory in the object detection part of FIG. It is a figure which shows the structure of the motion vector calculation part in the object detection part of FIG. It is a figure which shows the structure of the estimated position calculation part in the object detection part of FIG. It is a figure which shows the structure of the object detection part in the image processing circuit of FIG. 3 is a flowchart showing an image processing algorithm by the image processing circuit of FIG. 1. (A) and (b) are images showing the result of image division of a grayscale image by image division processing, and (c) and (d) are images showing the result of image division on a color image. It is. It is a graph which shows the relationship between the number of pixels and processing time when image division processing is performed using an image division algorithm. It is a figure which shows the relationship between the position of the front frame of an object, and the estimated position of the present frame. (A)-(c) is an image which three objects which simulated the image processing method of this invention move from the left to the right. (A)-(c) is the image which performed Morphology processing with respect to the image of FIG. (A)-(c) is an image which shows the result of having divided | segmented the image of FIG. (A)-(e) is the image which the motion vector changed during the movement of the object which simulated the image processing method of this invention. (A)-(k) is the image which simulated the image processing method of this invention and several objects are moving at random. (A)-(f) is an image which shows the result of having divided | segmented the image of FIG. (A)-(f) is an image in which a motion vector changes rapidly, when a ball | bowl collides with a floor and bounces off. (A)-(f) is another image from which a motion vector changes rapidly, when a ball | bowl collides with a floor and bounces off. (A)-(d) is an image which shows the result of having divided | segmented the image of FIG. (A)-(d) is an image which shows the result of having divided | segmented the image of FIG. (A) to (f) are images in which small objects are moving randomly. It is an image which shows the result of having divided | segmented the image of FIG. (A) to (c) are images of a person moving in a frame. (A)-(d) is an image which shows the result of having divided | segmented the image of FIG. (A) to (h) are images in which a walking person overlaps another person in the frame. (A)-(d) is an image which shows the result of having divided | segmented the image of FIG. (A)-(c) is an image which shows the result of having divided | segmented the image of FIG. (A)-(f) are the images which the minicar is moving. (A)-(c) is an image which shows the result of having divided | segmented the image of FIG. (A)-(f) is the image which image | photographed the road where the car is running. (A) And (b) is an image which shows the result of having divided | segmented the image of FIG.

Explanation of symbols

1 Image processing circuit (image processing device)
4 Connection weight calculator (image dividing means)
5 Leader cell calculator (image segmentation means)
6 Image segmentation cell network (image segmentation means)
9 Position information calculation unit (motion vector calculation means, estimated position calculation means)
9a Motion vector calculation unit (motion vector calculation means)
9b Estimated position calculation unit (estimated position calculation means)
10 Minimum distance search associative memory (object recognition means)

Claims (16)

Image dividing means for dividing a plurality of objects included in the input image from the input image for each frame of the input image;
For each divided object, a motion vector calculating means for calculating a motion vector indicating the amount of movement of the object from the difference between the position coordinate of the object in the frame immediately before and the position coordinate of the previous frame;
For each object, estimated position calculation means for calculating an estimated position, which is a position where each object is estimated to exist in the current frame, based on the position coordinates of the previous frame of the object and the motion vector;
The same object is recognized between the current frame and the previous frame based on the estimated position, the feature amount other than the estimated position for each object in the previous frame, and the feature amount other than the estimated position for each object in the current frame. An image processing apparatus comprising: an object recognition unit. The object recognition means calculates a distance index, which is a value based on the first difference value and the second difference value, between the target object of the current frame and each object of the previous frame, and the distance index is minimized. Recognizing objects as the same object,
The first difference value is a difference value between coordinates indicating the estimated position of the object of interest in the current frame and coordinates indicating the actual position of each object in the previous frame,
The said 2nd difference value is a difference value of the feature-value other than the presumed position regarding each object of a previous frame, and the feature-value other than the presumed position regarding the attention object of the present frame, The feature value of Claim 1 characterized by the above-mentioned. Image processing device.   The image processing apparatus according to claim 1, wherein the feature amount other than the estimated position includes at least one of the number of pixels, the width, the height, and the color information. The feature amount other than the estimated position is the number of pixels, width, height, and color information,
The second difference value is a difference value for each of the number of pixels, width, height, and color information,
The object recognition means is
The distance index between the object i of the current frame and the object j of the previous frame is D _ij , the first difference value is Ip _ij , the pixel number difference value is Ia _ij , and the width difference value is Iw _ij , When the height difference value is Ih _ij and the color information difference value is Ic _ij ,
D _ij = αIa _ij + β (Iw _ij + Ih _ij ) + γIp _ij + εIc _ij
(Α, β, γ, ε are coefficients determined according to the respective sizes of Ia _ij , Iw _ij , Ih _ij , Ip _ij , and Ic _ij )
The image processing apparatus according to claim 2, wherein the distance index D _ij is calculated. The feature amount other than the estimated position is the number of pixels, width, height, and color information,
The second difference value is a difference value for each of the number of pixels, width, height, and color information,
The object recognition means is
The distance index between the object i of the current frame and the object j of the previous frame is D _ij , the first difference value is Ip _ij , the pixel number difference value is Ia _ij , and the width difference value is Iw _ij , When the height difference value is Ih _ij and the color information difference value is Ic _ij ,
D _ij = √k
k = αIa _ij ² + βIw _ij ² + βIh _ij ² + γIp _ij ² + εIc _ij ²
(Α, β, γ, ε are coefficients determined according to the respective sizes of Ia _ij , Iw _ij , Ih _ij , Ip _ij , and Ic _ij )
The image processing apparatus according to claim 2, wherein the distance index D _ij is calculated.   6. The image processing apparatus according to claim 4, wherein the coefficients α, β, γ, and ε satisfy α = 2β = 2γ = ε.   The image processing apparatus according to claim 1, wherein the object recognition unit uses a normalized value as the estimated position.   The image processing apparatus according to claim 7, wherein the object recognition unit performs the normalization by dividing an actual value of the estimated position by a maximum value that can be taken by the estimated position.   The object recognition means divides the input image into a plurality of regions each having the same size, and divides the actual value of the estimated position by a predetermined value based on the size of the region, thereby The image processing apparatus according to claim 7, wherein:   The image processing apparatus according to claim 1, wherein the object recognition unit uses a normalized value as a feature amount other than the estimated position.   11. The image according to claim 10, wherein the object recognition unit performs the normalization by dividing an actual value of the feature quantity by a maximum value that can be taken by a corresponding value of the feature quantity. Processing equipment.   The object recognition means divides the input image into a plurality of regions each having the same size, and divides the actual value of the feature amount by a predetermined value based on the size of the region, thereby The image processing apparatus according to claim 10, wherein the image processing is performed.   13. An image processing integrated circuit, wherein the image processing apparatus according to claim 1 is integrated. A first step of dividing a plurality of objects included in the input image from the input image for each frame of the input image;
For each divided object, a second step of calculating a motion vector indicating the amount of movement of the object from the difference between the position coordinate of the object in the previous frame and the position coordinate of the previous frame;
For each object, a third step of calculating an estimated position, which is a position where each object is estimated to exist in the current frame, based on the position coordinates of the previous frame of the object and the motion vector;
The same object is recognized between the current frame and the previous frame based on the estimated position, the feature amount other than the estimated position for each object in the previous frame, and the feature amount other than the estimated position for each object in the current frame. An image processing method comprising: a fourth step.   An image processing program for causing a computer to execute each step in the image processing method according to claim 14.   A computer-readable recording medium on which the image processing program according to claim 15 is recorded.

Images