JP4692344B2 - Image recognition device - Google Patents

Description

  The present invention relates to an image recognition apparatus, and more particularly to an image recognition apparatus that identifies a subject by pattern recognition.

There is known a system in which a road situation is photographed with a camera mounted on a vehicle, and a pedestrian or the like in a screen is detected from acquired image information to inform a driver (for example, see Patent Document 1). In Patent Document 1, an infrared image and a visible light image are acquired, a road region is specified from the visible light image, and moving objects (pedestrians, etc.) in the road region are template-matched from the infrared image in the road region. The technique of detecting by is described.
JP 2002-99997 A

  In this technique, a moving body area that seems to be a pedestrian is cut out from an image of an infrared camera, and a pedestrian is detected by a template matching method. However, with this method, a person riding a bicycle or two-wheeled vehicle cannot be distinguished from a pedestrian because, for example, only the occupant's region is cut out as a moving body region separately from the bicycle or two-wheeled vehicle, and treated as a pedestrian candidate. As a result, the template matching processing time also increases, and there is a possibility that these occupants who are not pedestrians may be erroneously recognized as pedestrians.

  Therefore, an object of the present invention is to provide an image recognition apparatus that improves the accuracy of discrimination between bicycles and two-wheeled vehicles and their occupants and pedestrians.

In order to solve the above-described problems, an image recognition apparatus according to the present invention is an image recognition apparatus for recognizing a subject in an image, and includes a dictionary group including a plurality of dictionaries storing templates of the subject, and an optical in the image. By extracting a flow vector and extracting pixel points with similar vectors, a moving body region is extracted, and a dictionary used for subject recognition is selected from the dictionary group based on the moving speed. It is characterized by comprising a dictionary selection means, a recognition means for recognizing a subject by comparing the extracted moving body region and a template stored in the dictionary selected by the dictionary selection means .

  According to the present invention, an appropriate dictionary, that is, a set of templates is used in template matching according to the moving speed of the subject. Here, these templates are stored in a dictionary corresponding to the type of the subject, and the dictionary selection means switches the dictionary to be used based on the moving speed of the subject or uses the dictionary to be used based on the moving speed of the subject. It is good to switch combinations and their priorities. By switching the dictionary to be used or changing the priority order, the number of templates to be matched is effectively reduced.

The dictionary selection means may calculate the moving speed of the subject using a plurality of time-series images whose imaging times are known. For example, the optical flow may be calculated, and the actual moving speed of the subject may be calculated from the moving distance in the image and the distance between the subject and the camera.

  According to the present invention, since an appropriate template dictionary is used according to the moving speed of a subject, it is possible to reduce misrecognition of a subject having a different moving speed, for example, a traveling bicycle or a motorcycle occupant as a pedestrian. . For this reason, the reliability of the alarm and behavior control using the recognition result is improved, and the safety is also improved.

  At this time, it is effective to prepare a plurality of template dictionaries and switch between them. However, if the priority is changed by combining a plurality of dictionaries, the size of each template dictionary can be reduced. In addition, the probability of matching the patterns at an early stage increases, so that there is an advantage that the identification can be speeded up.

  Furthermore, by calculating the moving speed from the time-series image, it is not necessary to use another means (for example, a radar device) to determine the moving speed of the object, and the system can be simplified.

  DESCRIPTION OF EXEMPLARY EMBODIMENTS Hereinafter, preferred embodiments of the invention will be described in detail with reference to the accompanying drawings. In order to facilitate the understanding of the description, the same reference numerals are given to the same components in the drawings as much as possible, and duplicate descriptions are omitted.

  FIG. 1 is a block diagram showing a configuration of an image recognition apparatus according to the present invention. Here, the example employ | adopted as the alarm system which recognizes a pedestrian etc. from the acquired image and alert | reports to a driver | operator is demonstrated. The camera 1 is a camera that continuously acquires peripheral images in front of the vehicle. For example, a camera that is disposed in front of a room mirror in front of the vehicle interior and acquires moving images at a television frame rate is used. The alarm ECU 2 includes a CPU, a ROM, a RAM, and the like, and includes an image recognition unit 20 and a determination unit 21. The image recognition unit 20 and the determination unit 21 may be realized by different hardware, but may share some or all hardware and may be realized by software. In this case, the software may have a completely independent configuration, but may share a part of the program or may not be clearly divided in the shared program.

  An alarm device 4 that is an output device and a storage device 3 are connected to the alarm ECU 2. As the alarm device 4, in addition to a visual alarm device such as a display shared with a display device or a navigation device in an instrument panel, a bell, a beep device, a speaker that performs an alarm by voice, a driver Any of a means for swinging a seat on which the user sits or a combination thereof can be used. The storage device 3 stores a dictionary for template matching, and a recording device using a readable / writable recording medium, for example, a hard disk device or a flash memory may be used. Note that a part of the dictionary data may be recorded on a read-only storage medium, and the learning result or the like may be recorded on a readable / writable recording medium separately. The storage device 3 can be used to store camera images and processing results thereof.

  Next, the operation of this embodiment will be specifically described. FIG. 2 is a flowchart showing image recognition processing in this apparatus. This processing is repeatedly executed at a predetermined timing in the image recognition unit 20 while the alarm system switch (not shown) is on and the vehicle is powered on. This timing may be performed for each frame in synchronism with the frame rate of the camera 1 or may be executed every predetermined frame.

First, the camera image Im (t ₁ ) acquired by the camera ₁ is read (step S1). Following, t ₁ is assumed to represent the current time. Next, the past image Im (t ₂ ) acquired by the camera 1 is read (step S3). This is stored in a temporary storage area in the image recognition unit 20. FIG. 3 is a diagram illustrating an example of an input image (moving image). For example, t ₂ is a point in time that goes back from the current time t _{1 by} a time step Δt of image processing. In step S5, the image Im (t ₁ ) at the current time is stored in the temporary storage area. The stored image Im (t ₁ ) is read in step S3 in the next time step.

  Next, an optical flow is calculated from these two images having different time series (step S7). Various known methods can be adopted as the optical flow calculation method. Here, a calculation method using the gradient method will be described.

が成り立つ。輝度Ｅがｘ、ｙ、ｔに関して滑らかに変化するとすれば、上記（１）式の左辺はテーラー級数に展開でき、ｄｘ／ｄｔ＝ｕ、ｄｙ／ｄｔ＝ｖが成立する。この関係から（１）式を成立すると、以下の（２）式で表させるオプティカルフロー拘束方程式が成立する。

ここで、Ｅｘ、Ｅｙ、Ｅｔは、それぞれｘ、ｙ方向の空間微分値および時間微分値であり、時系列画像から計算される。拘束方程式は、１画素につき、１つの方程式しか得られないため、フローベクトルを一意に算出することはできない。そこで、「ある注目画素の近傍では、動きは滑らかである」という条件を加え、注目画素近傍の拘束方程式から最小自乗法により解を求め、オプティカルフローとする。 The luminance at the time t of the pixel point (x, y) in the image is E (x, y, t). If the x component and y component of the optical flow vector at this pixel point are u (x, y) and v (x, y), the luminance at a point (x + uΔt, y + vΔt) after a sufficiently small time interval Δt is E ( x, y, t). That is,

Holds. If the luminance E changes smoothly with respect to x, y, and t, the left side of the above equation (1) can be expanded into a Taylor series, and dx / dt = u and dy / dt = v are established. When formula (1) is established from this relationship, an optical flow constraint equation expressed by the following formula (2) is established.

Here, Ex, Ey, and Et are a spatial differential value and a temporal differential value in the x and y directions, respectively, and are calculated from the time series image. Since only one equation per constraint pixel can be obtained, a flow vector cannot be uniquely calculated. Therefore, a condition that “the motion is smooth in the vicinity of a certain pixel of interest” is added, and a solution is obtained by a least square method from a constraint equation in the vicinity of the pixel of interest to obtain an optical flow.

  FIG. 4 is an image diagram in which the optical flow vector thus obtained is superimposed on the camera image with an arrow. The figure shows the case where it is calculated from the stopped image. While the vehicle is running, the flow vector for the stationary coordinate system is calculated by subtracting the movement amount (movement vector) of the host vehicle obtained from the vehicle speed sensor or the like. It is good to calculate.

When the optical flow vector is calculated, a moving object is extracted from the image Im (t ₁ ) (step S9). Specifically, the extraction process is performed by paying attention to the optical flow vector. Here, a case where an object moving in the horizontal direction is extracted from an image will be described as an example. Assuming that the optical flow vector component at a certain pixel point is (u, v) as described above, if | u |> 0 and v≈0, the object is in the X-axis direction in the image, that is, in the horizontal direction. It can be regarded as having moved to. By searching the entire image and extracting points (pixels) that satisfy this relationship, an object moving in the horizontal direction can be extracted. Further, θ = tan ⁻¹ (v / u) is obtained from the vector component (u, v) as the flow direction (gradient), and if this θ is within a predetermined range (for example, within ± 15 °), the lateral direction is obtained. It may be regarded as an object that moves. By considering the direction of the flow vector, an object moving in an arbitrary direction can be extracted.

Next, the area of the moving object is estimated (step S11). For example, as shown in FIG. 5, it is assumed that a pedestrian is crossing from the left to the right of the screen on the road that is the course of the vehicle. Here, each pixel points constituting an image S ₁ of the pedestrian has substantially the same size, the flow vector of the orientation, since it is considered that continuous, the flow vector size, orientation substantially identical extracting pixel points are continuous there is, it sets the minimum rectangular area R ₁ surrounding it as moving object region. Depending on the moving object, it has a complicated shape, but handling it as a rectangular area facilitates subsequent processing. Information on the estimated moving body area is stored in the temporary storage area (step S13). As this information, the coordinate positions of the pixels at the upper left corner and the lower right corner of the moving object region may be used. In the example shown in FIG. 6, regions R ₁ and R ₂ are extracted from images of the pedestrian S ₁ and the preceding vehicle S ₂ , respectively.

Next, the lower left corner points N ₁ (x ₁ , y ₁ ) and N ₂ (x ₂ , y ₂ ) are determined as representative points of the estimated rectangular areas R ₁ , R _2, etc., and the flow vector at that point is determined. Let (u, v) be a representative amount (step S15). The representative quantity may be a flow vector at a representative point, or an average value, median value, mode value, or the like of the flow vector in the rectangular area. When performing statistical processing, it is preferable to exclude pixels having a flow vector size of approximately 0 in the rectangular area from the processing. After the representative point and the representative amount are determined, the position and the representative amount are stored in the temporary storage area (step S17).

  Next, the distance between the camera 1 and the object is obtained (step S19). Here, it demonstrates with reference to FIG. When the vertical resolution of the camera 1 is T and the viewing angle is θ, the viewing angle change amount Δθ per pixel can be expressed by θ / T. If the coordinate of the representative point of the object is A (i, j), and the angle between the line Lt connecting A and the camera center O and the field-of-view range bottom line Ld is α, then α = j × Δθ, In addition, the angle formed between the perpendicular line dropped from the camera center O and the visual field range lower end line Ld is β, and the camera installation angle, that is, the angle formed between the camera visual field center line C and the perpendicular line dropped from the camera center O is ρ. Then, β = ρ−1 / 2 × θ holds. When the installation height of the camera 1 is H, the distance L from the camera 1 to the representative point A of the object can be expressed by L = H × tan (α + β) from the above relationship.

  Next, the moving distance of the object in the minute time Δt (time step) is calculated (step S21). Here, in FIG. 8, a case will be described as an example in which the distance of the quantity change is obtained when the object existing at the point P (i1, j11) in the previous time step moves to the point Q (i2, j2). When the horizontal resolution of the camera is W and the viewing angle is φ, the viewing angle change amount Δφ per pixel can be expressed by φ / W. If an angle formed by two straight lines connecting from the camera center O to each of the points P and Q is γ, γ = Δφ × (i2−i1). Assuming that the camera visual field center line C is a perpendicular bisector of the line segment PQ, the distance K that the object has moved in the minute time Δt is equal to the length of the line segment PQ, and from the point O to Cx If the distance to is L, since L has already been calculated in step S19, it can be calculated by K = 2 × Ltan (1 / 2γ). With the same concept, it is possible to calculate not only the lateral direction but also the moving distance in an arbitrary direction. The obtained moving distance K is divided by the time step Δt to calculate the actual moving speed V (= K / Δt) (step S23).

  Next, a recognition pattern dictionary is set based on the velocity V of the object (step S25). For example, as shown in FIG. 9, when the speed V is equal to or lower than TH1, a dictionary in which template images of pedestrians are recorded is used. When the speed V exceeds TH2 and equal to or lower than TH3, bicycles (including passengers) are used. ) Is used, and when the speed V exceeds TH4, a dictionary in which a template image of a car (including a motorcycle and its passenger) is recorded is used. When the speed V exceeds TH1 and is equal to or less than TH2, a dictionary in which template images of both pedestrians and bicycles are recorded is used. When the speed V exceeds TH3 and equal to or less than TH4, bicycles and two-wheeled vehicles are used. A dictionary in which both template images are recorded may be used.

  Alternatively, three types of dictionaries (hereinafter referred to as dictionaries A to C) in which template images of pedestrians, bicycles, and automobiles (including two-wheeled vehicles) are recorded as pattern dictionaries are prepared, and combinations of dictionaries are determined depending on speed. The priority order may be switched. For example, when the speed V is equal to or lower than TH, the dictionary A is the highest priority dictionary, the dictionary B is used as the next priority dictionary, and the dictionary C is used as the lowest priority dictionary. When the speed V exceeds TH2 and is equal to or lower than TH4, the dictionary B is set as the highest priority dictionary, and the dictionary C is used as a dictionary with low priority. In this case, the dictionary A is not used. When the speed V exceeds TH4, only the dictionary C is used, and the dictionary A and the dictionary B are not used.

  As a method of using priority, pattern recognition is performed from a dictionary with high priority, and if the correlation is equal to or higher than a threshold value, other patterns are not matched and weighted according to priority. A coefficient may be set, and matching may be performed using a value obtained by multiplying the degree of correlation by the weighting coefficient as a correlation coefficient.

  After the dictionary is set, the object is recognized by performing pattern recognition based on the set dictionary (step S27). In the present embodiment, since an appropriate dictionary is selected based on the speed information of the object, there is a case where a bicycle or a two-wheeled vehicle occupant having a higher moving speed than a pedestrian is erroneously recognized as a pedestrian. This can be reduced and the discrimination accuracy is improved. In particular, according to the technique of switching dictionary combination patterns and priorities, it is possible to accurately determine a bicycle or a two-wheeled vehicle that is moving at a low speed comparable to a pedestrian. On the other hand, in the case of a low speed, the method using only the pedestrian dictionary can reliably suppress a case where a passenger of a two-wheeled vehicle moving at a relatively high speed is erroneously recognized as a pedestrian. In low-speed areas, there is a possibility that a bicycle or two-wheeled vehicle passenger moving at a low speed comparable to a pedestrian may be misidentified as a pedestrian. Since there is a high possibility of showing, it is considered that there will be no major trouble even if handled in the same way.

  After recognition, the result is output (step S29), and the process is terminated. For example, as shown in FIG. 10, the display device may be turned on according to the distance from an object (pedestrian, bicycle, car, etc.), or the presence of a moving body may be notified by sound from a speaker. Further, the vehicle behavior control device (not shown) may be notified to perform obstacle avoidance, pre-crash control, and the like.

  In the above description, the method for discriminating the type, speed, distance, etc. of an object by image processing from a time-series image acquired by a single camera has been described. However, the speed and distance of an object are determined using a stereo camera. Or may be acquired by a radar device or the like. In addition, various modifications can be made.

It is a block diagram which shows the structure of the image recognition apparatus concerning this invention. It is a flowchart which shows the image recognition process in the apparatus of FIG. It is a figure explaining the example of the input image (moving image) of an image recognition process. It is an image figure of an optical flow vector. It is a figure explaining the extraction process of a moving object. It is a figure explaining the extracted mobile body area | region. It is a figure explaining the distance calculation method of a camera and a target object. It is a figure explaining the moving speed calculation method of a target object. It is a figure explaining the dictionary switching reference | standard by object speed. An example of alarm display is shown.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Camera, 2 ... Alarm ECU, 3 ... Memory | storage device, 4 ... Alarm device, 20 ... Image recognition part, 21 ... Determination part.

Claims (5)

In an image recognition apparatus that recognizes a subject in an image,
A dictionary group having a plurality of dictionaries storing templates of subjects;
An optical flow vector in the image is extracted, and a pixel point similar to the vector is extracted to extract a moving body region, which is used for subject recognition from the dictionary group based on the moving speed. Dictionary selection means for selecting a dictionary;
An image recognition apparatus comprising: a recognition unit that recognizes a subject by comparing the extracted moving body region with a template stored in a dictionary selected by the dictionary selection unit. The image recognition apparatus according to claim 1, wherein the template is stored in a dictionary corresponding to a type of a subject, and the dictionary selection unit switches a dictionary to be used based on a moving speed of the subject. 2. The template according to claim 1, wherein the template is stored in a dictionary corresponding to a type of subject, and the dictionary selection unit switches a combination of dictionaries to be used and a priority order thereof based on a moving speed of the subject. Image recognition device. The image recognition apparatus according to claim 1, wherein the dictionary selection unit calculates a moving speed of the subject using a plurality of time-series images whose imaging times are known. In an image recognition apparatus that recognizes a subject in an image,
  A moving speed detecting means for detecting the moving speed of the subject;
  A dictionary group having a plurality of dictionaries for storing subject templates according to the subject type;
  Dictionary selection means for switching and selecting a combination of dictionaries used for subject recognition from the dictionary group and their priority order based on the movement speed of the subject detected by the movement speed detection means;
  An image recognition apparatus comprising: a recognizing unit for recognizing a subject by comparing a partial image region including the subject in the image with a template stored in the dictionary selected by the selecting unit.

Images