JP5870871B2 - Image processing apparatus and vehicle control system using the image processing apparatus - Google Patents

Description

  The present invention relates to a technique for detecting an object such as a pedestrian from an image of a vehicle-mounted camera.

  Conventionally, a technique for detecting a pedestrian from an image of a vehicle-mounted camera has been proposed for the purpose of avoiding a collision with an object that becomes an obstacle to vehicle travel. For example, a technique for detecting a pedestrian by detecting an object to be discriminated from an image of an in-vehicle camera and applying a neural network object discrimination method using a pedestrian's contour pattern to the object has been proposed. Yes.

  However, in the neural network object discrimination method using the pedestrian contour pattern, an object such as a tree similar to the pedestrian contour pattern may be detected as a pedestrian, and the detection accuracy of the pedestrian decreases. May end up.

  As a technique for solving this problem, a technique has been proposed in which a determination regarding a predetermined part constituting a pedestrian is performed and a pedestrian is determined based on whether or not the sum of determination output values of each predetermined part exceeds a threshold value. (For example, refer to Patent Document 1). For example, a determination regarding a predetermined part such as a head, a torso, a hand, and a leg is performed, and detection is performed based on the sum of determination output values of each predetermined part.

JP 2008-21034 A

  However, the technique described in Patent Document 1 has a problem that a pedestrian behind a pedestrian or a pedestrian hidden behind a pedestrian cannot be accurately detected. That is, when some of the predetermined parts such as the pedestrian's head, torso, hand, and foot are not shown in the image, the determination output value of the predetermined part is reduced, so that the sum of the determination output values is reduced. Therefore, when a pedestrian is detected based on the sum of the determination output values, there is a concern that no pedestrian is detected.

Note that the problems listed here are not limited to pedestrians, but also occur when an object in an image is detected.
The present invention has been made to solve the above-described problems, and an object of the present invention is to provide an image processing apparatus capable of appropriately detecting an object in an image and vehicle control using the image processing apparatus. To provide a system.

Images processing apparatus has been made in order to achieve the above object (10) detects an object based on the image from the imaging device (32) mounted on a vehicle.
In this image processing apparatus, the part model storage unit (61) stores a part model for detecting and scoring parts that are parts of the object. For example, if the object is a pedestrian, the part model is a head part model corresponding to the head, an arm part model corresponding to the arm, a waist part model corresponding to the waist, and a leg part corresponding to the leg. Prepared as a model. For example, if the object is a vehicle, a tire part model corresponding to a tire, a taillight part model corresponding to a taillight, a window part model corresponding to a window, and the like are prepared.

  Here, in particular, the reference area detecting means (51a) detects the reference area of the reference object that is present on the near side of the image and is not hidden by other objects. The shape of the reference area is not particularly limited, but for example, a rectangular area can be considered. The search area setting means (51b) sets a search area that is partially hidden in the reference area detected by the reference area detection means. The search area may be a rectangular area, for example.

  The weight setting means (51c) sets the weight of each part from the relationship with the reference area based on the reference position of each part in the search area set by the search area setting means. Thus, the score calculation means (51d) scores each part in consideration of the weight set by the weight setting means, and adds the scores of each part to calculate the score of the search area. Then, the object detection means (51e) detects the object based on the score of the search area calculated by the score calculation means.

  That is, when scoring a part that is a part of an object, a weight is set depending on whether or not the part is hidden in a reference region detected in an image, and the weight is changed. In this way, the object in the image can be detected appropriately.

It is a block diagram which shows schematic structure of a vehicle control system. It is explanatory drawing which illustrates a collision risk determination table. It is a flowchart which shows a target object detection process. It is explanatory drawing which shows the overlap with the reference | standard area | region and a search rectangle, and the overlap with the parts in the said overlap part.

Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Hereinafter, description will be made assuming a pedestrian as the object.
The vehicle control system of the present embodiment is configured with the image processing device 10 and the vehicle control device 20 as the center. A distance measuring device 31 and an imaging device 32 are connected to the image processing device 10. In addition, a speaker 41, a brake 42 and a steering 43 are connected to the vehicle control device 20.

  The image processing device 10 and the vehicle control device 20 are configured as a so-called computer. As a result, the image processing device 10 detects a pedestrian based on the image from the imaging device 32, and determines the collision risk based on the distance information from the distance measuring device 31. On the other hand, the vehicle control device 20 performs notification by the speaker 41 according to the collision risk output from the image processing device 10, or performs avoidance control by the brake 42 and the steering 43 when the degree of emergency is large. To go.

Next, the configuration of each unit will be specifically described.
The distance measuring device 31 is embodied as a laser radar or the like. Of course, another configuration such as a millimeter wave radar may be used. The distance measuring device 31 can detect the distance to the object in front of the vehicle.

  The imaging device 32 is embodied as a monocular camera or the like. The imaging device 32 images the front of the vehicle. As described above, the image processing apparatus 10 detects a pedestrian based on this image.

  The speaker 41 is configured for audio output. When the risk of collision with a pedestrian increases, sounds such as “There is a pedestrian in front” and “Be careful” are output via the speaker 41.

  The brake 42 includes a brake driving device, and decelerates the vehicle by a signal from the vehicle control device 20. The steering 43 includes an EPS (Electric Power Steering) device, and assists the driver's steering by a signal from the vehicle control device 20. Such vehicle control is performed when the risk of collision with a pedestrian is further increased, and when the urgency is high.

  The image processing apparatus 10 includes a control unit 50 and a storage unit 60. The control unit 50 is embodied as a so-called microcomputer. The control unit 50 includes an object detection unit 51 and a collision risk determination unit 52. The storage unit 60 is embodied as a ROM, for example. Of course, a flash memory or HDD other than the ROM may be used. The storage unit 60 includes a parts model storage unit 61 and a collision risk determination table storage unit 62.

  Here, the object detection unit 51 detects a pedestrian in the image output from the imaging device 32. Specifically, as will be described later, a reference area of a reference object existing in front is specified, and then a pedestrian hidden in the reference area is detected.

  Since the distance to the pedestrian is known by the distance measuring device 31 after the pedestrian is detected, the collision risk level determination unit 52 determines the collision risk level according to the distance from the pedestrian. Moreover, since the probability of the detected pedestrian is calculated as a score, the collision risk is determined according to the score of the pedestrian. The collision risk is output to the vehicle control device 20.

  The part model storage unit 61 stores a part model. The parts model is a model of each part of the pedestrian. In the present embodiment, the head, arms, waist, and legs are prepared as parts. Hereinafter, the part models corresponding to these parts are appropriately distinguished as “head part model”, “arm part model”, “waist part model”, and “leg part model”. The part model is a feature vector that expresses the feature of each part such as contour information.

The collision risk determination table storage unit 62 stores a collision risk determination table. As shown in FIG. 2, the collision risk determination table is a table that associates the distance (D) to the pedestrian and the pedestrian score (S) with the collision risk and the vehicle control information. The collision risk determination unit 52 determines the collision risk by referring to the collision risk determination table. For example, in the example of FIG. 2, the degree of risk is determined based on the satisfaction of at least one of the distance (D) to the pedestrian and the score (S). Specifically, when the distance D is “D > D2 ” or the score S is “S ≦ S1,” the risk level is determined as “level 1”, and the vehicle control is determined as “alarm processing”. The When the distance D is “D1 <D ≦ D2” or the score S is “S1 <S ≦ S2”, the risk level is determined as “level 2”, and the vehicle control is determined as “brake control”. Is done. Furthermore, when the distance D is “D ≦ D1 ” or when the score S is “S> S2,” the risk level is determined as “level 3”, and the vehicle control is determined as “steer control”.

  The vehicle control device 20 includes a notification control unit 70 and a vehicle control unit 80. Both are embodied as so-called microcomputers. The notification control unit 70 is configured to perform notification via the speaker 41. When the risk of collision increases, notification is first performed via the speaker 41. The vehicle control unit 80 is configured to perform vehicle control via the brake 42 and the steering 43, and collision avoidance control is performed when the risk of collision further increases.

  Specifically, the collision risk determination unit 52 outputs a signal to the notification control unit 70 in order to realize vehicle control “alarm processing” when the risk is determined as “level 1” in the collision risk determination table of FIG. I do. Thereby, the notification through the speaker 41 by the notification control unit 70 is performed. Further, the collision risk determination unit 52 outputs a signal to the vehicle control unit 80 in order to realize vehicle control “brake control” when it is determined that the risk is “level 2” in the collision risk determination table of FIG. . Thereby, deceleration control via the brake 42 by the vehicle control unit 80 is performed. Furthermore, the collision risk determination unit 52 outputs a signal output to the vehicle control unit 80 in order to realize the vehicle control “steer control” when it is determined that the risk is “level 3” in the collision risk determination table of FIG. Do. Accordingly, avoidance control is performed by the vehicle control unit 80 via the steering 43.

  The present embodiment is characterized by an object detection process as a function of the object detection unit 51. Therefore, details of the object detection process will be described below. The object detection process is repeatedly executed at predetermined time intervals.

As shown in FIG. 3, in the first S110, image input is performed. In this process, an image output from the imaging device 32 is input.
In subsequent S120, a reference region in the image is detected. This process detects a rectangular range indicating a reference object in the image from the imaging device 32. The reference object is a three-dimensional object shown on the near side of the image. The reference object may include not only a pedestrian but also a vehicle. As a specific detection method, an existing technique is used. For example, as described above, a part model obtained by modeling each part of a pedestrian is stored in the part model storage unit 61, but a pedestrian in the foreground is extracted by scanning an image using the part model. .

  In the next S130, an overlapping rectangle with the search rectangle is calculated. The search rectangle is a rectangle that is taken in an appropriate size so that a part of the search rectangle overlaps the reference region detected in S120. For example, as shown in FIG. 4A, when the reference region R is detected in the image, search rectangles T1 and T2 are set around the reference region R. A plurality of search rectangles are set around the reference region R. Since the same processing is performed for all the search rectangles, the search rectangle T1 will be described as an example. The overlap rectangle calculated here is an overlap rectangle C between the reference region R and the search rectangle T1, as shown by hatching in FIG.

In subsequent S140, the weight of each part is determined. The reference position of each part is set for the search rectangle. This will be described.
When the search rectangle indicates a pedestrian, a head part, an arm part, a waist part, and a leg part are included. Therefore, the “head part model”, “arm part model”, “waist part model”, and “leg part model” stored in the part model storage unit 61 are used and searched according to the degree of coincidence with these part models. Each part inside the rectangle is scored later (S150).

  At this time, the weight of each part is determined based on the reference position of each part inside the search rectangle. This weight is used to weight each part when scoring each part. The reference position of each part is a position set with respect to the search rectangle, and is calculated statistically. For example, as shown in FIG. 4C, it is assumed that six parts P1, P2, P3, P4, P5, and P6 are set for the search rectangle T1.

  Here, among the six parts P1 to P6, at least a part of the parts P1, P3, P4, and P6 is behind the reference region R. That is, there is a portion that overlaps the overlapping rectangle C. Therefore, the parts P1, P3, P4, and P6 that overlap with the overlapping rectangle C are referred to as “parts with hidden”. Other parts, that is, parts P2 and P5 that do not overlap with the overlapping rectangle C are referred to as “non-hidden parts”.

Various methods can be adopted for the weight determination. Here, attention is paid to “area with hiding” which is a hidden area of “part with hiding” shown by hatching in FIG. For example, when the hidden part P6 is viewed, the hidden area is Pa6 and the non-hidden area is Pn6.

At this time, in this embodiment, the weight Wak of the hidden part Pk is calculated by the following equation 1.

Wak = (Pk non-hidden area (Pnk)) / (part Pk area) Equation 1

Specifically, in the case of P6, Wa6 = Pn6 / (Pn6 + Pa6).

Further, the weight Wn of the non-hidden part is calculated by the following formula 2.

Wn = (total area of parts) / (total area of non-hidden parts)

The weight Wak of the hidden part is calculated for each part Pk. Further, the weight Wn of the non-hidden part is uniformly calculated regardless of the part.

  In the next S150, the score of each part is added by taking the weight into consideration. In the example of FIG. 4 (c), the scores S1 to S6 of the parts P1 to P6 are calculated, and the hidden parts P1, P3, P4, and P6 are multiplied by the Wak calculated by the above formula 1, and the parts that are not hidden. P2 and P5 are multiplied by Wn calculated by the above equation 2 and added together.

  In subsequent S160, a pedestrian is detected by threshold processing. In other words, since the score of the search rectangle is calculated as the sum of the scores of the parts with the weight added, it is determined that the person is a pedestrian when the score exceeds the threshold value.

  As described above in detail, in the present embodiment, the part model storage unit 61 stores a part model for detecting and scoring parts that are parts of the object. In this embodiment, the part model corresponds to each part of the pedestrian, and the head part model corresponding to the head, the arm part model corresponding to the arm, the waist part model corresponding to the waist, the leg It is prepared as a leg part model corresponding to the part.

  At this time, in the present embodiment, the reference area detection unit 51a detects the reference area of the reference object that is present on the near side of the image and is not hidden by other objects (S120 in FIG. 3). The search area setting means 51b sets a search area that is partially hidden in the reference area detected by the reference area detection means 51a (S130), and calculates an overlapping rectangle. The weight setting means 51c sets the weight of each part from the relationship with the reference area based on the reference position of each part in the search area set by the search area setting means 51b (S140). Thereby, the score calculation means 51d scores each part in consideration of the weight set by the weight setting means 51c, and adds the scores of each part to calculate the score of the search area (S150). And the pedestrian as a target object is detected by the target object detection means 51e based on the score of the search area calculated by the score calculation means 51d (S160).

  That is, when scoring a part that is a part of an object, a weight is set depending on whether or not the part is hidden in a reference region detected in an image, and the weight is changed. In this way, the object in the image can be detected appropriately.

  Further, in the present embodiment, paying attention to the “area with hiding” that is the hidden area of the “part with hiding”, the weight Wak of the part with hiding Pk is calculated by the above formula 1. Specifically, in the case of P6, Wa6 = Pn6 / (Pn6 + Pa6). Further, the weight Wn of the non-hidden part is calculated by the above formula 2. That is, the weight setting means 51c sets the weight based on the area of the hidden part that is a part that at least partially overlaps the reference region. Here, “based on the area of the part with hiding” includes the case of “based on the area of the part without hiding” since the total area of the part is known. As a result, the larger the area hidden in the reference region, the larger the weight of the part, so that the object in the image can be detected appropriately.

  In particular, the weight Wak of the hidden part is calculated for each part Pk. That is, the weight setting means 51c sets the weight based on the overlapping area in the area of the hidden part. Thereby, since the weight is set with the actual overlap area, it is possible to set a detailed weight.

  Furthermore, in the present embodiment, the collision (R) or score (S) when the search rectangle indicates a pedestrian is used, and the collision is determined based on the collision risk determination table stored in the collision risk determination table storage unit 62. Determine the risk. That is, the collision risk is determined based on at least one of the position, size, score, and distance to the object detected by the object detection means. Thereby, the determination of the collision risk is appropriate.

  In accordance with the collision risk determined by the collision risk determination unit 52, notification to the driver through the speaker 41 by the notification control unit 70 of the vehicle control device 20, and the brake 42 and steering 43 by the vehicle control unit 80. Collision avoidance control is performed via In other words, in the vehicle control system of the present embodiment, the vehicle control device 20 performs the notification control unit 70 and the collision avoidance control for notifying the driver based on the collision risk determined by the collision risk determination unit 52. A vehicle control unit (80) is provided. Thereby, driving assistance with higher safety is realized.

As described above, the present invention is not limited to the embodiment described above, and can be implemented in various forms without departing from the technical scope thereof.
(A) In the above embodiment, the weight is set by paying attention to the “area with hidden” which is the hidden area of the “part with hidden” (S140 in FIG. 3).

On the other hand, the weight may be set by paying attention to the number of “parts with hidden”. Of course, since the total number of parts is known, the weight may be set by paying attention to the number of “non-hidden parts”. As an example, it is conceivable to set the weight according to the following equations 3 and 4, where Za is the weight of the hidden part and Zn is the weight of the non-hidden part.

Za = (number of non-hidden parts) / (total number of parts) Equation 3

Zn = (total number of parts) / (number of non-hidden parts)

In this case, the process of S140 may be corrected.

  That is, the weight setting means 51c may set the weight based on the number of hidden parts that are parts that at least partially overlap the reference region. Here, “based on the number of hidden parts” includes the case of “based on the number of non-hidden parts” since the total number of parts is known. In this way, as in the above embodiment, the larger the number of parts hidden in the reference area, the larger the weight of the parts, so that the object in the image can be detected appropriately.

(B) In the above embodiment, the weight is set by paying attention to the “area with hidden” which is the hidden area of the “part with hidden” (S140 in FIG. 3).
On the other hand, the score of the search area is not unduly reduced by the existence of “parts with hiding”, and the score of the search area is not affected by “parts with hiding”. You may make it calculate. For example, the weight is set to “0” so that the score of the hidden part does not become negative. Also, for example, the score of the hidden part is a predetermined standard value. In this case, the process of S140 may be corrected. That is, the weight setting means 51c may set the weight so that the score of the hidden part that is a part at least partially overlapping the reference area does not affect the score of the search area.
In this way, since the parts hidden in the reference area do not affect the score of the search area, the object in the image can be detected appropriately.

(C) In the above embodiment, the weight of each part in the search rectangle is set. However, the reliability of the reference area may be calculated, and the part may be scored using this reliability. For example, the standard area score normalized from “0” to “1” is set as the reliability C, the weight Wak of the hidden part is corrected, and a new weight Wak ′ is calculated by the following formula 5. .

Wak ′ = max (0, Wak−C) Equation 5

Similarly, the weight Wn of the non-hidden part is corrected, and a new weight Wn ′ is calculated by the following Expression 6.

Wn ′ = Wn + C Expression 6

That is, the reference area detection unit 51a detects the reference area and calculates the reliability of the reference area, and the score calculation unit 51d scores each part and calculates the search area score in consideration of the reliability. It is good as well. In this way, the score of each part becomes more appropriate.

  Here, the new weight is obtained using the reliability, that is, the process of S140 is corrected, but the reliability may be used when adding the score. That is, you may correct the process of S150.

  (D) In the above embodiment, the collision risk is determined based on the distance (D) to the pedestrian output from the distance measuring device 31 or the score (S) of the pedestrian. In addition to such distance (D) and score (S), the risk of collision may be determined using at least one of the position and size of the search area.

  (E) In the above embodiment, the distance to the pedestrian is measured by the distance measuring device 31, but the distance may be measured based on the image information from the imaging device 32 instead of the distance measuring device 31. . That is, the distance measuring device 31 may be omitted. Of course, by providing the distance measuring device 31, highly accurate measurement is possible.

  DESCRIPTION OF SYMBOLS 10 ... Image processing device, 20 ... Vehicle control device, 31 ... Distance measuring device, 32 ... Imaging device, 41 ... Speaker, 42 ... Brake, 43 ... Steering, 50 ... Control part, 51 ... Object detection part, 51a ... Reference | standard Area detection means, 51b ... Search area detection means, 51c ... Weight setting means, 51d ... Score calculation means, 51e ... Object detection means, 52 ... Collision risk determination part, 60 ... Storage part, 61 ... Parts model storage part, 62 ... Collision risk determination table storage unit, 70 ... Notification control unit, 80 ... Vehicle control unit

Claims (9)

An image processing device (10) for detecting an object based on an image from an imaging device (32) mounted on a vehicle,
A part model storage unit (61) for storing a part model for detecting and scoring parts that are parts of the object;
A reference area detecting means (51a) for detecting a reference area of a reference object existing on the near side of the image and not hidden by other objects, and calculating the reliability of the reference area;
Search area setting means (51b) for setting a search area partially hidden in the reference area detected by the reference area detection means;
Based on the reference position of each part in the search area set by the search area setting means, weight setting means (51c) for setting the weight of each part from the relationship with the reference area;
Score calculating means (51d) for scoring each part in consideration of the reliability, together with the weight set by the weight setting means, and calculating the score of the search area by adding the scores of the parts When,
Object detection means (51e) for detecting the object based on the score of the search area calculated by the score calculation means;
An image processing apparatus comprising: The image processing apparatus according to claim 1.
The image processing apparatus, wherein the object is a pedestrian. The image processing apparatus according to claim 1 or 2,
The weight setting means sets the weight so that the score of a hidden part that is a part that at least partially overlaps the reference area does not affect the score of the search area (S140).
An image processing apparatus. The image processing apparatus according to claim 1 or 2,
The weight setting means sets the weight based on the number of hidden parts that are parts at least partially overlapping the reference region (S140).
An image processing apparatus. The image processing apparatus according to claim 1 or 2,
The weight setting means sets the weight based on an area of a hidden part that is a part that at least partially overlaps the reference region (S140).
An image processing apparatus. The image processing apparatus according to claim 5.
The weight setting means sets the weight based on an overlapping area in the area of the hidden part (S140).
An image processing apparatus. In the image processing apparatus according to any one of claims 1 to 6 ,
A collision risk determination unit (52) for determining a collision risk based on at least one of the position, size, score, and distance to the object detected by the object detection means; An image processing apparatus, further comprising: An image processing apparatus according to claim 7 ;
A vehicle control device (20) having a notification control unit (70) for notifying a driver based on the collision risk determined by the collision risk determination unit;
A vehicle control system comprising: An image processing apparatus according to claim 7 ;
A vehicle control device (20) having a vehicle control unit (80) for performing collision avoidance control based on the collision risk determined by the collision risk determination unit ;
A vehicle control system comprising: