JP4590654B2 - Vehicle periphery monitoring device, vehicle, vehicle periphery monitoring program - Google Patents

Description

  The present invention relates to an apparatus for monitoring the periphery of a vehicle using an image obtained through an in-vehicle imaging device.

Technology that measures the real space position of an object such as a pedestrian based on an image captured by a camera mounted on the vehicle, and determines the level of possibility of contact between the vehicle and the object based on the measurement result Has been proposed (see Patent Document 1).
JP 2007-213561 A

  However, it may be preferable to determine the type of the object from the viewpoint of avoiding contact between the vehicle and the object.

  Therefore, an object of the present invention is to provide an apparatus that can determine the type of an object with high accuracy.

A vehicle periphery monitoring device according to a first aspect of the present invention is a device that monitors the periphery of the vehicle using an image representing the surrounding state of the vehicle obtained through one imaging device mounted on the vehicle, An object detection element that recognizes the presence of an object based on the same, and an identity requirement that the object detected by the object detection element based on the image at each of the first time point and the second time point is the same An object tracking element that determines whether or not the above condition is satisfied, a distance measurement element that measures a distance from the vehicle to the object, and an area where the object exists in the image is set as a first object area A first object region setting element that is measured by the distance measuring element with respect to the object that is determined to satisfy the identity requirement by the object tracking element. A second object area setting element for setting the first object area at the second time point as a second object area, enlarged or reduced at a ratio of the distance at the second time point to the distance at the first time point; In the second object area set by the second object area setting element, a plurality of objects arranged in the first object area set at the first time point by the first object area setting element The degree of correlation with each of the first local areas is greater than or equal to a threshold, and the position of the second object area is different from the position of the first local area of the first object area at the first time point. tolerance searched a second local region such that follows, the third object region for setting the searched second local region area containing a plurality of human settlement was the third object region A constant element, based on the shape of the third object region setting element and the third object region set by, characterized in that it includes a type determining object classifier of the object.

  According to the vehicle periphery monitoring device of the first aspect of the present invention, there is an object having the same object set in an image representing the surrounding situation of the vehicle at two different times (= first time point and second time point). The size of the area (= first object area) is aligned based on the distance from the vehicle to the object at each of the two different times. Further, the second object region (= the first object region at the second time point enlarged or reduced) has the same identity as the first local region arranged in the first object region at the first time point. The objects are classified based on the shape of the area including the local area (= third object area). Since the local region group having such an identity has a high probability that an object having a shape corresponding to the shape exists, the type of the object can be determined with high accuracy.

  A vehicle according to a second aspect of the present invention includes the imaging device according to the first aspect, and the vehicle periphery monitoring device according to the first aspect of the present invention that monitors the periphery of the vehicle using an image representing the surrounding state of the vehicle obtained through the imaging device. It is characterized by being.

  According to the vehicle of the second invention, since the type of the object is determined with high accuracy by the vehicle periphery monitoring device, the operation of the on-board equipment is avoided according to the determination result, etc. From the viewpoint, it can be appropriately controlled.

  According to a third aspect of the present invention, there is provided a vehicle periphery monitoring program that uses a computer mounted on a vehicle to monitor the periphery of the vehicle using an image representing the surrounding state of the vehicle obtained through one imaging device mounted on the vehicle. It is made to function as a vehicle periphery monitoring apparatus of the 1st invention to monitor.

  According to the vehicle periphery monitoring program of the third invention, the computer mounted on the vehicle can be caused to function as a vehicle periphery monitoring device that determines the type of the object with high accuracy.

  DESCRIPTION OF EMBODIMENTS Embodiments of a vehicle periphery monitoring device and the like of the present invention will be described with reference to the drawings.

  A vehicle (four-wheeled vehicle) 1 shown in FIG. 1 is equipped with a vehicle periphery monitoring device 10 and a single infrared camera (imaging device) 11. As shown in FIG. 1, the real space is such that the origin O is located on the front side of the vehicle 1, and each of the X axis, the Y axis, and the Z axis extends in the left-right direction, the vertical direction, and the front-rear direction of the vehicle 1. A coordinate system is defined. The infrared camera 11 is attached to the front side of the vehicle 1 in order to image the front of the vehicle 1. As shown in FIG. 2, the vehicle 1 includes various types of sensors such as a yaw rate sensor 13, a speed sensor 14, and a brake sensor 15 that output signals corresponding to the yaw rate, speed, and brake state of the vehicle 1. A sensor is installed. Further, as shown in FIG. 2, the vehicle 1 is equipped with an audio output device 16 and an image output device 17. As the image output device 17, in addition to a HUD (head-up display) that displays an image on the front window of the vehicle 1, a display device that indicates a traveling state of the vehicle 1 or a navigation device may be employed. .

  The vehicle periphery monitoring device 10 is a device that monitors the periphery of the vehicle 1 using an image obtained through the infrared camera 11. The vehicle periphery monitoring device 10 is configured by a computer (configured by an electronic circuit such as a CPU, a ROM, a RAM, an I / O circuit, and an A / D conversion circuit). Analog signals output from the infrared camera 11, the yaw rate sensor 13, the vehicle speed sensor 14, the brake sensor 15, and the like are digitized and input to the vehicle periphery monitoring apparatus 10 through an A / D conversion circuit. Based on the input data, according to the “vehicle periphery monitoring program” stored in the memory, processing for recognizing the presence of an object such as a human or another vehicle, and the possibility of contact with the object recognized as the vehicle 1 is high or low. And a process for outputting sound to the audio output device 16 and outputting an image to the image output device 17 in accordance with the determination result. The program may be distributed or broadcast from the server to the in-vehicle computer via a network or a satellite at an arbitrary timing, and stored in a storage device such as a RAM. The vehicle periphery monitoring device 10 may be configured by a position ECU, but may be configured by a plurality of ECUs that constitute a distributed control system.

  As shown in FIG. 2, the vehicle periphery monitoring apparatus 10 includes an object detection element 102, an object tracking element 104, a distance measurement element 106, a first object area setting element 110, and a second object area. A setting element 120, a third object area setting element 130, and an object classification element 140 are provided.

  The object detection element 102 recognizes the presence of the object based on an image representing the surrounding situation of the vehicle 1 obtained through the infrared camera 11. The object tracking element 104 determines whether or not the “identity requirement” that the objects detected by the object detection element 102 are the same is satisfied based on the images at the first time point and the second time point. . The image or area at a certain time means an image representing the surrounding situation of the vehicle 1 at this time or an area set in this image. The distance measuring element 106 measures the distance from the vehicle 1 to the object. The first object area setting element 110 sets an area where the object exists in the image as a “first object area”. The second object area setting element 120 sets a “second object area” for the object determined by the object tracking element 104 that the identity requirement is satisfied. The second object setting area is a first object area at the second time point that is enlarged or reduced by the ratio of the object distance at the second time point to the object distance at the first time point measured by the distance measuring element 106. is there. The third object area setting element 130 is the second object area set by the second object area setting element 120 and is arranged in the first object area set by the first object area setting element 110. A region including a second local region having the same identity as the one local region is set as a “third object region”. The object classification element 140 determines the type of the object based on the shape of the third object area set by the third object area setting element 130.

  Functions of the vehicle 1 and the vehicle periphery monitoring device 10 having the above-described configuration will be described.

  First, the object detection element 102 detects the object based on the image obtained through the infrared camera 11. Specifically, a grayscale image is acquired by A / D converting an infrared image that is an output signal of the infrared camera 11 (FIG. 3 / S002). Further, a binarized image is acquired by binarizing the grayscale image (S004 in FIG. 3). The binarization process is a process of classifying each pixel constituting the grayscale image into “1” (white) and “0” (black) according to whether or not the luminance is equal to or higher than a threshold value. The gray scale image and the binarized image are stored in separate image memories. Further, a group of pixels grouped into “1” s constituting a high-luminance region of the binarized image has a width of one pixel in the vertical direction (y direction) of the image, and the horizontal direction (x direction). The lines are classified into extending lines, and each line is converted into run-length data composed of the coordinates of the position (two-dimensional position in the image) and the length (number of pixels) (S006 in FIG. 3). Of the lines represented by the run-length data, labels (identifiers) are attached to each line group that overlaps in the vertical direction of the image (FIG. 3 / S008), and the line group is detected as an object. (FIG. 3 / S010). As a result, as shown in FIG. 4A, an object (binarized object) as indicated by diagonal lines in the binarized image is detected. The target object includes living structures such as humans (pedestrians) and artificial structures such as other vehicles. Note that a plurality of local parts of the same object may be recognized as the target object.

  Further, the object tracking element 104 executes object tracking processing, that is, processing for determining whether or not the detected object is the same for each calculation processing cycle of the object detecting element 101 (see FIG. 3 / S012). For example, the shape and size of the object detected in each of the binarized images at times k-1 and k according to the method described in Japanese Patent Laid-Open No. 2001-6096, and the correlation of the luminance distribution in the grayscale image Or the like. If it is determined that these objects are the same, the label of the object at time k is changed to the same label as the label of the object at time k-1.

  Further, the distance measuring element 106 measures the distance from the vehicle 1 to the object detected by the object detecting element 102 based on the image. Specifically, the position of the center of gravity of the object in the binarized image is calculated, and the circumscribed rectangle of the object is set (FIG. 3 / S014). The center of gravity position of the object is obtained by adding the result of multiplying the length of each line to the coordinates of the center position of each line of the run-length data corresponding to this object, and adding the result to the object in the image. It is obtained by dividing by the area of the object. Instead of the center of gravity of the object, the position of the center of gravity (center) of the circumscribed rectangle of the object may be calculated. Also, the rate of change Rate of the size of the object area representing the object labeled with the same label at different times is evaluated (FIG. 3 / S016). Specifically, the size of the object region at time k-1 (represented by the vertical width, width, or area of the circumscribed rectangle) Size (k) at the time k−1, Size (k− The ratio of 1) is determined as the rate of change Rate (k) = Size (k−1) / Size (k) (<1). Furthermore, the speed and yaw rate of the vehicle 1 are measured based on the respective outputs of the yaw rate sensor 13 and the speed sensor 14, and the turning value (azimuth angle) of the vehicle 1 is calculated by integrating the measured value of the yaw rate. (FIG. 3 / S018).

  Further, based on the current speed v (k) and the imaging time interval δT of the infrared camera 11 in addition to the current rate of change Rate (k), the distance Z (k) from the vehicle 1 to the object according to the equation (1). Is measured (FIG. 3 / S020).

  Further, the vehicle periphery monitoring device 10 calculates the actual space position P (k) = (X (k), Y (k), Z (k)) of the object based on the distance from the vehicle 1 to the object. (FIG. 3 / S022). Specifically, the corrected distance Z (k) from the vehicle 1 to each object, the focal length f of the infrared camera 11, and the image coordinates x (k) and y of the region corresponding to the object in the captured image. Based on (k), the X coordinate X (k) and Y coordinate Y (k) in the real space coordinate system are calculated according to the equation (2). The center, right direction, and downward direction of the captured image are defined as the origin o, + x direction, and + y direction in the image coordinate system. Further, based on the turning angle measured based on the output of the yaw rate sensor 15, the actual space position (X (k), Y (k), Z (k)) of each object is corrected.

In addition, the first object area setting element 110 sets the area representing the object in the grayscale image as the “first object area” based on the center of gravity position of the object in the binarized image and the arrangement of the circumscribed rectangle. (FIG. 3 / S024). Specifically, first, a plurality of masks arranged with reference to the object are set in a grayscale image representing the surrounding situation of the vehicle 1 at the first time point. Thereby, for example, a plurality of rectangular masks a _{i +} (i = 1, 2,...) And a _j− arranged side by side in the vertical direction of the object, as indicated by hatching in FIG. (J = 1, 2,...) Is set. The plurality of masks are arranged so as to have a center position on a reference line (one-dot chain line) that passes through the center of gravity or center of the object and extends in the vertical direction in the image. In addition, a search is made for a mask whose degree of correlation with the plurality of masks is greater than or equal to a threshold in a grayscale image representing the surrounding situation of the vehicle 1 at a second time point before the first time point. Thus, for example, in the image at the second time point as shown in FIG. 4 (b), the mask of the correlation is above threshold and _j- plurality of masks a _{i +} and a in the first time point a _{i +} and a _{j -} is searched. Then, an area including the object and a continuous mask having the same or substantially the same positional relationship with the object at the first time point and the second time point among the plurality of masks is set as the first object region. . As a result, for example, as shown in FIG. 4B, a rectangular area including the object and the masks a _1-to a _4- continuously arranged below the object is a first object area. Set as

  Furthermore, the area obtained by enlarging the first object area at the second time point is set as the “second object area” by the second object area setting element 120 (FIG. 3 / S026). Thereby, for example, the first object area at the second time point shown in FIG. 5A is enlarged, and the second object area shown in FIG. 5B is set. The enlargement ratio of the first object region is measured by the distance measuring element 106 and is the same as the ratio of the distance from the vehicle 1 to the object at the second time point to the distance from the vehicle 1 to the object at the first time point. . When the second time point is after the first time point, an area obtained by reducing the first object area at the second time point may be set as the second object area.

  In addition, the third object area setting element 130 may include an area including a “second local area” having the same identity as the “first local area” arranged in the first object area in the second object area. It is set as a “third object region” (FIG. 3 / S028). Specifically, first, a plurality of first local regions arranged in the first object region at the first time point are set. Thereby, for example, as shown in each upper side of FIGS. 6A and 6B, a plurality of first rectangular shapes formed by dividing the first object region at the first time point vertically and horizontally. A local region is set. In addition, the 1st local area | region which covers only one part instead of all the 1st object area | regions may be set, and several 1st local area | regions may mutually separate or may overlap. In addition, the shape and size of the first local region may be changed as appropriate. Then, in the second object area, a second local area having the same identity as the first local area set in the first object area is searched, and the area including the searched second local area is the third object. Set as an object area. Here, the first local region and the second local region have “identity” means that the correlation between the first local region and the second local region is equal to or greater than a threshold, and the first object region (for example, The deviation between the position of the first local region with reference to the upper left corner) and the position of the second local region with reference to the second object region (for example, the upper left corner) is less than an allowable value. means. As a result, for example, as shown on the lower side of each of FIGS. 6A and 6B, a region including a plurality of second local regions arranged in the second object region is “first”. 3 object areas ”.

  Then, the object classification element 140 classifies the object based on the shape of the third object region (FIG. 3 / S030). Specifically, among the plurality of shapes corresponding to each of the object categories stored in the storage device, the object is classified into a category corresponding to a shape that most closely approximates the shape of the third object region. . Thereby, for example, based on the shape of the third object area shown in FIG. 6A, the object is classified into the category “human”. Further, based on the shape of the third object area shown in FIG. 6B, the object is classified into a category of “animals such as deer and dogs”. In addition, in addition to or instead of the classification of humans and other animals, the objects may be classified into categories such as plants such as automobiles, trees, and fixed objects such as buildings.

  Further, based on the real space position P (k) of each object at different times, for example, according to a collision possibility determination method described in Japanese Patent Laid-Open No. 2001-6096, the vehicle 1 can contact with each object. Whether or not the sex is high or low is determined (FIG. 3 / S032). If it is determined that the possibility of contact between the vehicle 1 and the object is high (FIG. 3 / S032... YES), the alert processing is performed in a manner corresponding to the type of the object determined by the object classification element 140. This is executed (FIG. 3 / S034). Specifically, the sound and the image (such as a frame for highlighting the target object) according to the determination result of the contact possibility and the classification result of the target object are respectively the audio output device 16 and the image output device 17. Output through each. For example, when the object is classified as a human, the output mode of sound and image is adjusted so that the driver of the vehicle 1 is more strongly aware of the presence of the object than when the object is classified as a vehicle. . Note that only one of the sound and the image may be output. Further, based on the output of the brake sensor 15, it is confirmed that the brake of the vehicle 1 is not operated by the driver, or the vehicle 1 is reduced based on the output of the speed sensor 14 or the acceleration sensor (not shown). The alerting process may be executed on the condition that it is confirmed that the speed is equal to or less than the threshold value. On the other hand, when it is determined that the possibility of contact between the vehicle 1 and the object is low (FIG. 3 / S032... NO), the alerting process is not executed.

  In addition, when part or all of the steering device, the brake device, and the accelerator device of the vehicle 1 are operated by the actuator and the driving behavior of the vehicle 1 is operated, the vehicle behavior is controlled instead of or in parallel with the alerting process. May be. Specifically, the steering device, the brake device, and the accelerator device of the vehicle 1 are avoided so as to avoid contact with an object that has been determined to have a high possibility of contact with the vehicle 1 or so as to facilitate avoidance. Some or all of the operations may be controlled by a vehicle control unit (not shown). For example, the accelerator device is controlled to make acceleration difficult so that the driver's required pedaling force on the accelerator pedal is greater than in a normal case where it is not necessary to avoid contact with the object. Further, the required rotational force of the steering handle toward the steering direction of the steering device required to avoid contact between the vehicle 1 and the object is lower than the required rotational force of the steering handle toward the opposite side, The steering handle can be easily operated in the steering direction. Furthermore, the increasing speed of the braking force of the vehicle 1 according to the depression amount of the brake pedal of the brake device is made higher than in the normal case. By doing in this way, the driving | operation of the vehicle 1 for avoiding a contact with a target object becomes easy.

  According to the vehicle periphery monitoring device 10 that performs the above function, there is the same object set in the images representing the surroundings of the vehicle 1 at two different times (= first time point and second time point). The size of the target object area (= first target object area) is aligned based on the distance from the vehicle to the target object at each of the two different times (FIG. 3 / S020, S024, S026, FIG. 5). a) (b)). Further, the second object region (= the first object region at the second time point enlarged or reduced) has the same identity as the first local region arranged in the first object region at the first time point. The objects are classified based on the shape of the area including the local area (= third object area) (see FIG. 3 / S028 and FIGS. 6A and 6B). Since the local region group having such an identity has a high probability that an object having a shape corresponding to the shape exists, the type of the object can be determined with high accuracy.

  Then, depending on the classification result of the target object, the operation of the equipment mounted on the vehicle 1 (sound output device 16, image output device 17, steering device, brake device, and accelerator device) avoids contact between the vehicle 1 and the target object. From the viewpoint, it can be appropriately controlled.

  In the above embodiment, the distance from the vehicle 1 to the object is measured using one infrared camera (monocular camera). However, as another embodiment, a millimeter wave radar, a laser radar, an ultrasonic radar, or the like is used. And the distance from the vehicle 1 to the object may be measured. In yet another embodiment, the distance may be calculated using parallax of a pair of infrared cameras arranged symmetrically with respect to the front center of the vehicle. Further, instead of the infrared camera 11, a camera whose sensitivity is adjusted to another wavelength region such as visible light may be employed as the imaging device.

Configuration diagram of the vehicle of the present invention Configuration explanatory diagram of the vehicle periphery monitoring device of the present invention The flowchart which shows the function of the vehicle periphery monitoring apparatus of this invention Explanatory drawing about the setting method of the 1st object field Explanatory drawing about the setting method of the 2nd object field Explanatory drawing about the setting method of the 3rd object field

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Vehicle, 10 ... Vehicle periphery monitoring apparatus, 11 ... Infrared camera (imaging device)

Claims (3)

An apparatus for monitoring the periphery of the vehicle using an image representing the surrounding situation of the vehicle obtained through one imaging device mounted on the vehicle,
An object detection element for recognizing the presence of the object based on the image;
An object tracking element for determining whether the objects detected by the object detection element based on the images at each of the first time point and the second time point satisfy the identity requirement that they are the same;
A distance measuring element for measuring a distance from the vehicle to the object;
A first object area setting element for setting an area where the object exists in the image as a first object area;
A ratio of the distance at the second time point to the distance at the first time point measured by the distance measuring element for the object that is determined by the object tracking element to satisfy the identity requirement. A second object area setting element that sets the first object area at the second time point, which is enlarged or reduced, as a second object area;
In the second object region set by the second object region setting element, a plurality of the disposed in the first object region which is set in the first time point by the first object region setting element No. The degree of correlation with each of the local areas is equal to or greater than a threshold, and the position in the second object area is allowed to deviate from the position of the first local area in the first object area at the first time point. A third object region setting element that searches for a second local region that is less than or equal to a value , and sets a region that includes the searched second local region as a third object region;
A vehicle periphery monitoring device comprising: an object classification element for determining a type of the object based on a shape of the third object area set by the third object area setting element. The vehicle periphery monitoring device according to claim 1, wherein the vehicle periphery monitoring device monitors the periphery of the vehicle by using an image representing the surrounding state of the vehicle obtained through the image pickup device. vehicle. The vehicle periphery monitoring according to claim 1, wherein a computer mounted on the vehicle monitors the periphery of the vehicle using an image representing a surrounding state of the vehicle obtained through one imaging device mounted on the vehicle. A vehicle periphery monitoring program, which functions as a device.

Images