JP6593588B2 - Object detection apparatus and object detection method - Google Patents

Description

  The present disclosure relates to an object detection apparatus and an object detection method, and more specifically, to detect an object that is mounted on a vehicle, a road infrastructure system, or a monitoring facility of a specific facility individually and accurately. The present invention relates to an object detection apparatus and an object detection method capable of performing the above.

  2. Description of the Related Art In recent years, vehicles such as passenger cars are equipped with an in-vehicle radar device or an in-vehicle camera device that detects other vehicles, pedestrians, two-wheeled vehicles, installations on the road, and the like existing around the vehicle. The in-vehicle radar device or the in-vehicle camera device detects a target object approaching from the front or side of the host vehicle, and measures a relative position, a relative speed, and the like with the host vehicle. Then, the in-vehicle radar device determines whether there is a risk of collision between the host vehicle and the target object based on the measurement result. When it is determined that there is a danger, the in-vehicle radar device further avoids a collision by giving a warning to the driver or automatically controlling the own vehicle.

  For example, Patent Document 1 discloses a technique for detecting an object using both a vehicle-mounted radar device and a vehicle-mounted camera device. Specifically, in Patent Document 1, the measurement information of the camera device is used to determine the number of target objects and the range of azimuth angles, and the measurement information of the radar device is determined based on the number of target objects and the range of azimuth angles. It has been corrected.

  Patent Document 2 discloses a technology for monitoring traffic volume using both a camera device and a radar device installed around a road. Specifically, in Patent Document 2, information on the position and speed of a distant vehicle is detected by a radar device, and after the position of the vehicle in the camera image is determined, the situation of both the vehicle far and near from the vehicle is presented from the camera image. Traffic monitoring and traffic management.

  Conventionally, radar devices or camera devices have been installed to monitor specific facilities such as airports, ports, railway stations, or buildings, and detect intruding objects from the ground or in the air (space above the ground). In addition, information is provided to the related security system or display unit to prevent intrusion of suspicious objects (including suspicious persons).

JP 2010-151621 A US Patent Application Publication No. 2013/0300870

R.C.Gonzalez and R.E.Woods, Digital Image Processing, Prentice Hall, 2001.

  However, in the above-described prior art of Patent Document 1, the number of target objects and the range of azimuth angles of each target object must be determined based on measurement information of a camera device mounted on the vehicle. That is, a high level of object detection performance is required for the in-vehicle camera device.

  Moreover, in the prior art of patent document 2, when the radar apparatus has acquired a plurality of detection results from one vehicle, it becomes difficult to determine the position of the vehicle.

  In other words, in the above-described conventional technology, even when the vehicle is mounted on a vehicle, used for a road infrastructure system, or used for a monitoring system for a specific facility, the detection accuracy of an object is the same between a camera device and a radar device. Depends on one of the performance. That is, it is difficult to effectively superimpose the sensing function of the radar device and the sensing function of the camera device to improve the object detection accuracy.

  An object of the present disclosure is to provide an object detection device and an object detection method that can effectively superimpose a sensing function of a radar device and a sensing function of a camera device to improve an object detection accuracy. To do.

  In the object detection device of the present disclosure, the radar signal includes one or more objects for a plurality of cells obtained by dividing the distance from the radar device at predetermined intervals in each transmission direction of the radar signal transmitted by the radar device. A reflection intensity that is a representative value of the power of the received signal received by the radar apparatus is calculated for each reflection signal reflected by the radar apparatus, and power profile information is generated for each of the plurality of cells using the reflection intensity. An information generator and a cell indicating the maximum value of the reflection intensity among the power profile information of the plurality of cells is calculated as a capture point for capturing the one or more objects, and one cell surrounding the capture point A capture region calculation unit that calculates a capture region that is the above cell; an edge extraction unit that extracts an edge of the one or more objects included in an image acquired by the camera device; The capture area is converted into a partial area in the image based on the measurement range of the reader device and the imaging range of the camera apparatus, and the partial area is calculated as a marker that is the area of the image corresponding to the capture area. A marker calculation unit that performs a component region calculation corresponding to a part of the one or more objects by extending the marker using the edge as a boundary, and the component region as a target object region. And a grouping processing unit for grouping, and an object determination unit for determining the one or more objects from the target object region and outputting the determination result.

  In the object detection method of the present disclosure, the radar signal includes one or more objects for a plurality of cells in which the distance from the radar apparatus is divided at predetermined intervals in each transmission direction of the radar signal transmitted by the radar apparatus. A reflection intensity that is a representative value of the power of the received signal received by the radar apparatus is calculated for each reflected signal reflected by the radar apparatus, and power profile information is generated for each of the plurality of cells using the reflection intensity. Calculating a cell indicating the maximum value of the reflection intensity from the power profile information of the plurality of cells as a capture point for capturing the one or more objects, and at least one cell surrounding the capture point. A certain capture area is calculated, an edge of the one or more objects included in an image acquired by the camera device is extracted, and a measurement range of the radar device and an imaging range of the camera device are extracted. The capture region is converted into a partial region in the image, the partial region is calculated as a marker that is a region of the image corresponding to the capture region, and the marker is expanded with the edge as a boundary. To calculate a component area corresponding to a part of the one or more objects, group the component areas as target object areas, determine the one or more objects from the target object areas, and determine the determination result. Is output.

  According to the present disclosure, the sensing function of the radar device and the sensing function of the camera device can be effectively superimposed to improve the object detection accuracy.

Conceptual diagram of a configuration of a sensing unit using the object detection device according to the present disclosure Conceptual diagram of a configuration of a sensing unit using the object detection device according to the present disclosure Conceptual diagram about installation location of object detection device according to the present disclosure Conceptual diagram about installation location of object detection device according to the present disclosure 1 is a block diagram illustrating a main configuration of an object detection device according to a first embodiment of the present disclosure. The figure which shows an example of the power profile information in Embodiment 1 of this indication The figure which shows an example of the calculation result of the capture | acquisition area in Embodiment 1 of this indication Diagram showing an example of the coordinate system of the radar measurement three-dimensional space Diagram showing the relationship between distance, maximum possible height, and ground distance Diagram for explaining the conversion from the coordinates of the camera 3D space to the coordinates of the camera image plane The figure which shows an example of a camera image plane The figure which shows an example of the calculation result of the marker corresponding to the capture area | region shown in FIG. The figure which shows an example in case a component area | region calculation part divides a marker The figure which shows an example of the result of the area expansion by a component calculation part The figure which shows an example of the result of coordinate-transforming the component area into the area on the radar measurement plane The figure which shows an example of the result of grouping by a grouping process part FIG. 3 is a block diagram illustrating a main configuration of an object detection device according to a second embodiment of the present disclosure. FIG. 3 is a block diagram illustrating a main configuration of an object detection device according to a third embodiment of the present disclosure. FIG. 3 is a block diagram illustrating a main configuration of an object detection device according to a fourth embodiment of the present disclosure.

(Background to this disclosure)
First, the process leading to the present disclosure will be described. The present disclosure relates to an on-vehicle radar device and a camera device, a road infrastructure system radar device and a camera device, and an object detection device used in a monitoring system for a specific facility.

  In-vehicle radar devices and camera devices are already being installed in many vehicles, but road infrastructure system radar devices and camera devices are also being introduced into road infrastructure systems. However, either a radar device or a camera device has been used alone, but the use of a radar device and a camera device in combination is also increasing.

  A radar device and a camera device for a road infrastructure system are installed around a road such as an intersection, and detect a road, a vehicle, a pedestrian, a two-wheeled vehicle, and the like, and monitor a traffic situation and manage a traffic.

  The road infrastructure system radar device and the camera device detect traffic volume, vehicle speed violation, signal ignorance, etc. as traffic condition monitoring. In addition, the road infrastructure system radar device and the camera device control traffic lights based on the detected traffic volume as traffic management. Alternatively, the radar device and the camera device for the road infrastructure system detect an object existing in the blind spot of the vehicle and notify the vehicle driver of information on the detected object. Thus, the road infrastructure system radar device and the camera device can realize traffic efficiency and prevention of traffic accidents.

  In both the on-vehicle radar device and the camera device, or the road infrastructure system radar device and the camera device, target objects having different characteristics such as a vehicle, a pedestrian, a bicycle, and a motorcycle need to be accurately detected. Also, in the radar system and the camera device for the monitoring system, it is necessary to accurately detect various vehicles and pedestrians when the ground is a monitoring area, and various flying objects and birds when the ground is a monitoring area. is there.

  By accurately detecting each target object, it is possible to accurately grasp the state in which the object exists in the space and the state of the traffic volume, and accurately predict the possibility of intrusion or collision. If each target object is not detected correctly, detection of the target object and false detection will occur, making it difficult to grasp the state of the object in the space and the traffic volume, and intrusion or collision is possible Gender prediction becomes difficult.

  In general, in a radar apparatus, a plurality of strong reflection points (hereinafter referred to as capture points) are acquired from one target object. Therefore, in order to detect the target object from the measurement result, it is necessary to group the capture points corresponding to the same object.

  In Patent Document 1, the number of target objects and the range of azimuth angles of each target object are determined based on measurement information of a camera device mounted on the vehicle, and the number of target objects and the range of azimuth angles of each target object are grouped. Regroup or ungroup based on By such processing, the technique disclosed in Patent Document 1 avoids erroneous detection and detection omission.

  However, in the technique disclosed in Patent Document 1, the accuracy of object detection varies depending on the number of target objects and the accuracy of the range of azimuth angles of each target object, that is, the accuracy of the sensing function of the camera device.

  Moreover, in patent document 2, it becomes difficult to detect the vehicle which is a target object when acquiring several capture points, and as a result, the technique which patent document 2 discloses is difficult to use.

  In view of such circumstances, the present disclosure has been achieved by paying attention to the fact that these measurement information can be effectively superimposed if the difference between the measurement information of the camera device and the measurement information of the radar device is taken into consideration.

  According to the present disclosure, in an in-vehicle radar device and a camera device, a vehicle, a two-wheeled vehicle, and a pedestrian existing around the own vehicle are accurately detected, a collision risk with the own vehicle is predicted, and a risk is avoided. Can be alerted and controlled. As a result, prevention of traffic accidents is realized.

  In addition, according to the present disclosure, in a radar device and a camera device for a monitoring facility of a specific facility such as an airport, a port, a railway station, or a building, a flying object or a bird from the air, and various vehicles or intruders from the ground. It is accurately detected and linked with an external security system to prevent the intrusion of suspicious persons and ensure the safety of the facility.

  Further, according to the present disclosure, in the radar device and the camera device for the road infrastructure system, the vehicle, the two-wheeled vehicle, and the pedestrian existing around the road including the intersection are accurately detected, and the possibility of the collision is predicted. You can avoid and manage and manage traffic volume. As a result, traffic accidents can be prevented and traffic management can be made more efficient.

(Use image of this disclosure)
Here, the connection method and installation location of the object detection device according to the present disclosure will be described with reference to the drawings.

  1A and 1B are conceptual diagrams of a configuration of a sensing unit using the object detection device according to the present disclosure. 1A and 1B, R and C indicate a radar device and a camera device, respectively. W represents an object detection device according to the present disclosure. FIG. 1A shows a case where the radar apparatus R and the camera apparatus C are provided in the same housing and are connected to the object detection apparatus W. FIG. 1B shows a case where the radar apparatus R and the camera apparatus C are provided in separate housings and are connected to the object detection apparatus W. 1A and 1B is further connected to an external security system or display unit.

  The present disclosure does not limit the installation method, location, and relative positional relationship between the radar apparatus R and the camera apparatus C. Further, the positional relationship between the detection range of the radar device R and the detection range of the camera device C is not limited. However, since the present disclosure is applied to the overlapping range of the detection range of the radar device R and the detection range of the camera device C, the radar device R and the camera device C are preferably installed so that the overlapping range is large. .

  The present disclosure provides an object detection apparatus W that superimposes and processes measurement information of the radar apparatus R and measurement information of the camera apparatus C. The object detection device W according to the present disclosure does not limit the configuration of the radar device R and the configuration of the camera device C. Both the radar device R and the camera device C may be existing commercial products or products composed of known techniques.

  1A and 1B, the object detection device W is provided separately from the radar device R and the camera device C, but may be included in the radar device R or the camera device C.

  In the present disclosure, the radar apparatus R and the camera apparatus C are connected to the object detection apparatus W and transmit measurement information to the object detection apparatus W, but the transmission method is not limited. The transmission method may be a wired communication method or a wireless communication method.

  Next, the installation location of the object detection device W according to the present disclosure will be described with reference to FIGS. 2A and 2B. 2A and 2B are conceptual diagrams of the installation location of the object detection device W according to the present disclosure. FIG. 2A is a conceptual diagram in which the object detection device W is mounted on a vehicle together with the radar device R and the camera device C, and FIG. 2B is used in a road infrastructure system in which the object detection device W is used together with the radar device R and the camera device C. FIG.

  In FIG. 2A, V is the own vehicle, R / C is a measuring device including a radar device R and a camera device C mounted on the own vehicle, and T1 and T2 are two different target objects. In terms of mounting, the object detection device W may be integrated with the measurement device R / C, or may be installed at a different position from the measurement device R / C. It is convenient if it can be detected.

  In FIG. 2B, R / C is a measuring device including a radar device R and a camera device C mounted on a road infrastructure, P is a road surface, L is a support device such as a pole on which the measuring device R / C is installed, and T1 and T2 Indicates two different target objects. FIG. 2B is a perspective image view of the vicinity of the position where the measuring device R / C is installed.

  The road surface P may be a straight road or a part of an intersection. Further, the position where the measuring device R / C is installed may be above the road, on the roadside, above the intersection, or at each corner of the intersection. In addition, this indication does not limit the position which installs measuring apparatus R / C, and the installation method. It would be advantageous if the measuring device R / C could detect vehicles, pedestrians, two-wheeled vehicles, etc. that exist around the pedestrian crossing at the intersection.

  2A and 2B, the target object T1 is an object larger than the target object T2, and corresponds to an object such as a vehicle, for example. The target object T2 corresponds to, for example, a motorcycle, a bicycle, a pedestrian, and the like. In the conceptual diagrams shown in FIGS. 2A and 2B, the target object T2 is present at a position closer to the radar apparatus than the target object T1. The object detection device W according to the present disclosure separates and detects the target objects T1 and T2.

  Moreover, although not shown in figure, the installation place of the object detection apparatus W which concerns on this indication may be a place which can monitor specific facilities, such as an airport, a port, a railway station, or a building. The measurement region of the object detection device W according to the present disclosure is not limited to the ground region, and can be used for monitoring or measurement in the air.

  Next, an embodiment of the present disclosure will be described in detail with reference to the drawings. Each embodiment described below is an example, and the present disclosure is not limited by these embodiments.

(Embodiment 1)
First, the object detection device according to the first embodiment of the present disclosure will be described with reference to the drawings. FIG. 3 is a block diagram illustrating a main configuration of the object detection device 30 according to the first embodiment of the present disclosure.

  The object detection device 30 according to the first embodiment of the present disclosure is connected to the radar device R and the camera device C. The radar apparatus R includes a transmitter that transmits a radar signal while sequentially changing directions at predetermined angular intervals, a receiver that receives a reflected signal reflected from a target object by the radar signal, and converts the reflected signal to baseband. And a signal processing unit that acquires a delay profile (propagation delay characteristic) for each transmission direction of the radar signal. The camera device C captures an image of a subject (target object) and acquires image data.

  The object detection device 30 includes an information generation unit 31, a supplemental region calculation unit 32, a camera image acquisition unit 33, an edge calculation unit 34, a marker calculation unit 35, a component region calculation unit 36, a grouping processing unit 37, and an object determination unit. 38. Each configuration of the object detection device 30 can be realized by hardware such as an LSI circuit. Or each structure of the object detection apparatus 30 is realizable also as a part of electronic control unit (Electronic Control Unit: ECU) which controls a vehicle.

  The information generation unit 31 uses the delay profile output from the signal processing unit of the radar device to determine the received power of the reflected signal for each cell in which the distance from the radar device is divided at predetermined intervals for each transmission direction of the radar signal. A representative value (hereinafter referred to as “reflection intensity”) is measured. Then, the information generation unit 31 generates power profile information indicating the reflection intensity of each cell and outputs it to the capture point calculation unit 32. The reflection intensity is generally a continuous value, but the information generation unit 31 may perform a quantization process in order to simplify the process. Details of the power profile information generated by the information generation unit 31 will be described later.

  The capture region calculation unit 32 first calculates the maximum point of the reflection intensity from the power profile information. The maximum point calculated by the capture region calculation unit 32 is a capture point for capturing the target object. Specifically, the capture region calculation unit 32 handles the power profile information as an image, and calculates the maximum point by a known method. Next, the capture area calculation unit 32 calculates a capture area for the capture point using a known image processing method. The capture region is a local region surrounding the capture point, and is configured by points having a reflection intensity equal to or greater than a predetermined value among points around the capture point. Note that a method of calculating the capture area in the capture area calculation unit 32 will be described later.

  The camera image acquisition unit 33 receives the image data from the camera device C, performs preprocessing such as image quality improvement, and outputs it to the edge calculation unit 34.

  The edge calculation unit 34 calculates the edge (contour) of the target object from the image data output from the camera image acquisition unit 33 using a known edge extraction method.

  The marker calculation unit 35 calculates a marker from the capture region calculated by the capture region calculation unit 32. The marker is a partial area of the camera image corresponding to the capture area. A marker calculation method in the marker calculation unit 35 will be described later.

  The component area calculation unit 36 calculates the component area by extending the marker calculated by the marker calculation unit 35 using the edge of the camera image calculated by the edge calculation unit 34. A component area calculation method in the component area calculation unit 36 will be described later.

  The grouping processing unit 37 groups component regions belonging to the same object among the component regions calculated by the component region calculating unit 36. The grouping processing unit 37 acquires the target object region as a result of grouping. A method of grouping component areas in the grouping processing unit 37 will be described later.

  Based on the target object region that is the result of the grouping process in the grouping processing unit 37, the object determination unit 38 further determines the position, size, and shape of the target object, and the type of the object (for example, large vehicle, small vehicle, Motorcycles, pedestrians, etc.). A method of determining the target object in the object determination unit 38 will be described later. The object determination unit 38 outputs the determination result to an external security system or display unit.

  Next, power profile information generated by the information generation unit 31 will be described. FIG. 4 is a diagram illustrating an example of power profile information according to the first embodiment of the present disclosure. The horizontal axis of FIG. 4 indicates the azimuth angle of the radar apparatus R, and the vertical axis indicates the distance from the radar apparatus R. In the following description, a plane defined by the azimuth angle of the radar apparatus R and the distance from the radar apparatus R is referred to as a radar measurement plane.

  In the example of FIG. 4, cells are configured by dividing the azimuth angle on the horizontal axis every 10 ° and dividing the distance on the vertical axis every 10 m. In the present embodiment, the cell angle range and distance range are not limited to those described above. Each range is preferably smaller in that high resolution can be obtained.

  In FIG. 4, the shading of each cell in the power profile information indicates the reflection intensity, and the darker the color, the stronger the reflection intensity. For simplicity of explanation, the color of cells other than the specific cell is the same white.

  In the present embodiment, the reflection intensity (representative value) of each cell is the maximum value of received power in the cell range. However, the present disclosure is not limited to this, and other values such as an average value of received power in the range of the cell may be used as the reflection intensity (representative value) of each cell.

  In the following description, each cell of the power profile information as shown in FIG. 4 is appropriately handled as one point.

  Next, a method of calculating the capture area in the capture area calculation unit 32 will be described with reference to FIGS. 4 and 5.

  The capture area calculation unit 32 first calculates a capture point from the power profile information shown in FIG. The capture point is the maximum point of the reflection intensity in the power profile information. A known method may be used to calculate the maximum point of the reflection intensity. For example, the reflection intensity of a specific point and a point adjacent to that point are compared, and if the reflection intensity of the specific point is greater than the reflection intensity of the adjacent point by a certain value or more, the specific point It may be a local maximum point.

  In the case of the power profile information shown in FIG. 4, the capture region calculation unit 32 calculates capture points a1, a2, and a3 that are maximum points of reflection intensity.

  Next, the capture area calculation unit 32 handles the power profile information as an image, and calculates a capture area surrounding each of the capture points a1, a2, and a3 using a known image processing method such as a region growing image processing method. To do. Refer to Non-Patent Document 1 for details of the region growing image processing technique.

  FIG. 5 is a diagram illustrating an example of a capture region calculation result according to the first embodiment of the present disclosure. The horizontal direction in FIG. 5 corresponds to the azimuth angle of the radar apparatus R, and the vertical direction corresponds to the distance from the radar apparatus R. Capture regions A1, A2, and A3 shown in FIG. 5 are local regions surrounding the capture points a1, a2, and a3, respectively. The capture areas A1, A2, and A3 are local areas on the radar measurement plane. Usually, the capture region is less susceptible to noise than the capture point.

  Next, a marker calculation method in the marker calculation unit 35 will be described. The marker calculation unit 35 calculates a marker that is a partial region on the plane of the camera image from a capture region that is a local region on the radar measurement plane. In the following description, a plane defined by the horizontal direction and the vertical direction of the camera image is referred to as a camera image plane. Note that the coordinates of the radar measurement plane and the coordinates of the camera image plane do not match. Therefore, the marker calculation unit 35 performs coordinate conversion and calculates a marker from the capture region. Below, the case where the marker calculation part 35 calculates a marker from the capture area | region A corresponding to target object T1 is demonstrated.

  Specifically, the marker calculation unit 35 converts the coordinates of the radar measurement plane to the coordinates of the radar measurement three-dimensional space, converts the coordinates of the radar measurement three-dimensional space to the coordinates of the camera three-dimensional space, and the camera three-dimensional space. Three coordinate transformations of the coordinate of the space to the coordinate of the camera image plane are sequentially performed.

  The radar measurement three-dimensional space is a space scanned by the radar device R, and the camera three-dimensional space is a space where the camera device C performs photographing. If the installation positions of the radar device R and the camera device C are different, the radar measurement three-dimensional space and the camera three-dimensional space may not match.

  Here, it is assumed that the azimuth angle range of the capture area A on the radar measurement plane is θ1 to θ2, and the distance range is d1 to d2. The azimuth angle range is determined from the minimum azimuth angle θ1 and the maximum azimuth angle θ2 of the capture region A, and the distance range is determined from the minimum distance d1 and the maximum distance d2 of the capture region A.

(Conversion from radar measurement plane coordinates to radar measurement three-dimensional space coordinates)
First, conversion from the coordinates of the radar measurement plane to the coordinates of the radar measurement three-dimensional space will be described. This conversion is a conversion for calculating the position and size of the radar measurement three-dimensional space corresponding to the capture area A from the azimuth angle ranges θ1 to θ2 and the distance ranges d1 to d2 of the capture area A.

  FIG. 6 is a diagram illustrating an example of the coordinate system of the radar measurement three-dimensional space. The origin O and Xr-Yr-Zr shown in FIG. 6 indicate the coordinate system of the radar measurement three-dimensional space. The radar device R is installed on the Zr axis. The height Hr corresponds to the height at which the radar device R is installed. Further, the distance d from the radar apparatus R shown in FIG. 7 corresponds to the distance d on the vertical axis in the radar measurement plane. The ground distance L is a distance on the ground (road surface) to the target object T1 on the Xr-Yr plane. The height h is the height of the target object T1. Note that the position and shape of the target object T1 are schematic.

  The radar apparatus R scans the radar measurement three-dimensional space shown in FIG. 6 with the Yr-Zr plane at Xr = 0 as the direction of the azimuth angle θ = 0 ° and the Zr axis as the axis. At this time, the azimuth angle θ on the horizontal axis in the radar measurement plane corresponds to the projection position of the scan surface of the radar apparatus R on the Xr-Yr plane in the radar measurement three-dimensional space. For example, the angle formed between the projection position of the scan plane and the Yr axis corresponds to the azimuth angle θ. FIG. 6 shows a case where the target object T1 exists at a position corresponding to the azimuth angle θ of 0 °.

  Normally, the radar apparatus R measures the reflection intensity corresponding to the azimuth angle θ and the distance d, while the radar apparatus R uses the Zr-axis direction in FIG. 6, more specifically, the elevation angle φ in FIG. Does not detect accurately. That is, the radar apparatus R cannot detect the height h of the target object T1 from the reflection intensity, and as a result, it is difficult to detect the ground distance L to the target object T1.

  Therefore, the marker calculation unit 35 in the present embodiment presets the maximum possible height hp of the target object T1. The maximum possible height hp is a maximum possible value as the height of the target object T1. For example, when the target object T1 is a pedestrian, the maximum possible height hp is 2 m. At this stage, it is not determined what the target object T1 is, but the maximum possible height hp is set based on the size of the capture region corresponding to the target object T1, the reflection intensity, and the like.

  The marker calculation unit 35 calculates the range of the ground distance L from the distance d on the radar measurement plane to the target object T1 in the radar measurement three-dimensional space using the maximum possible height hp.

  FIG. 7 is a diagram illustrating a relationship among the distance d, the maximum possible height hp, and the ground distance L. FIG. 7 shows the case where the signal reflected by the target object T1 is reflected near the ground of the target object T1 (Zr = 0 in FIG. 7), and the signal reflected by the target object T1 is near the maximum possible height hp of the target object T1. Shows the case of reflection.

  As shown in FIG. 7, the ground distance L is the distance between the ground distance L1 when reflecting near the ground and the ground distance L2 when reflecting near the maximum possible height hp with respect to one distance d of reflection intensity. It becomes the range.

  The marker calculation unit 35 calculates the ground distance L1 (L11) with respect to the distance d1 and the ground distance L2 (L12) with respect to the distance d1 for the distance range d1 to d2 of the capture region A, and the ground distance L1 (L21) with respect to the distance d2 and A ground distance L2 (L22) with respect to the distance d2 is calculated. Then, the marker calculation unit 35 determines the minimum value Lmin and the maximum value Lmax among L11, L12, L21, and L22. As a result, the marker calculation unit 35 calculates the ground distance range Lmin to Lmax in the Yr axis direction from the distance range d1 to d2 of the capture region A.

  Further, as described above, the azimuth angle θ on the horizontal axis in the radar measurement plane corresponds to the projection position on the Xr-Yr plane of the scan plane of the radar apparatus R, so that the marker calculation unit 35 has the azimuth angle range θ1 to θ2. To calculate the azimuth angle range θ1 to θ2 of the target object T1 on the Xr-Yr plane.

(Conversion from radar measurement 3D space coordinates to camera 3D space coordinates)
Next, conversion from the coordinates of the radar measurement three-dimensional space to the coordinates of the camera three-dimensional space will be described. Since the installation position of the radar apparatus R and the installation position of the camera apparatus C are known, conversion from the coordinates in the radar measurement three-dimensional space to the coordinates in the camera three-dimensional space is performed using a standard coordinate conversion method.

  By performing this conversion, even when the installation position of the radar apparatus R and the installation position of the camera apparatus C are different from each other, a marker that is a partial area on the plane of the camera image from a capture area that is a local area on the radar measurement plane. Can be calculated.

  In the following, for the sake of simplicity, it is assumed that the camera three-dimensional space is the same as the radar measurement three-dimensional space having the Xr-Yr-Zr coordinate system. That is, the azimuth angle ranges θ1 to θ2 in the radar measurement three-dimensional space and the ground distance ranges Lmin to Lmax in the Yr axis direction will be described as they are in the camera three-dimensional space.

(Conversion from camera 3D space coordinates to camera image plane coordinates)
Next, conversion from the coordinates of the camera three-dimensional space to the coordinates of the camera image plane will be described. This conversion corresponds to each on the camera image plane from the azimuth angle range θ1 to θ2 and the ground distance range Lmin to Lmax in the Yr axis direction in the camera three-dimensional space (in the following description, the same as the radar measurement three-dimensional space). It is conversion which calculates the range to perform. A range on the camera image plane obtained by this conversion, that is, a partial area is a marker for the capture area A.

  Here, a method for calculating a corresponding range on the camera image plane from the ground distance ranges Lmin to Lmax in the Yr-axis direction in the camera three-dimensional space will be described first.

  FIG. 8 is a diagram for explaining the conversion from the coordinates of the camera three-dimensional space to the coordinates of the camera image plane. FIG. 9 is a diagram illustrating an example of a camera image plane. FIG. 9 schematically shows an image taken by the camera apparatus C in the space shown in FIG. Here, for the sake of explanation, the camera image plane of FIG. 9 is shown, and the marker calculation unit 35 uses an actually captured image, that is, an image acquired by the camera image acquisition unit 33.

  The origin O and Xr-Yr-Zr shown in FIG. 8 indicate the coordinate system of the camera three-dimensional space. The camera device C is installed on the Zr axis. The height Hc corresponds to the height at which the camera device C is installed. In the following description, it is assumed that the position of the camera device C, more specifically, the center point at which the camera device C captures an image is a point C, and that the point C is at the height Hc of the Zr axis.

  An angle ∠PCQ shown in FIG. 8 is a vertical field angle range of the camera apparatus C. The points P and Q shown in FIGS. 8 and 9 correspond to the lower limit and the upper limit of the field angle range of the camera device C, respectively. The points P and Q are calculated from the angle of view range of the camera device C.

  Further, the Yr-Zr plane at Xr = 0 in FIG. 8 corresponds to the PQ line segment in FIG. Further, Xr = 0 in FIG. 8 corresponds to the center of the horizontal field angle range of the camera apparatus C.

  As shown in FIG. 9, the vanishing point F shown in FIG. 8 is an infinite point on the road surface p in the camera image plane. The vanishing point F is calculated by a known method.

  The ground distance ranges Lmin to Lmax shown in FIG. 8 are ground distance ranges obtained in conversion from the coordinates of the radar measurement plane to the coordinates of the radar measurement three-dimensional space. Hereinafter, the ground distance range Lmin to Lmax will be described as the range of point K to point J on the Yr axis.

  As shown in FIGS. 8 and 9, the points on the camera image plane corresponding to the points J and K are point V and point U, respectively. Calculation of the corresponding range on the camera image plane from the ground distance ranges Lmin to Lmax in the Yr-axis direction means that the positions of the points U and V on the camera image plane are calculated.

  First, a method for calculating the position of the point U on the camera image plane will be described.

  For vanishing point F, point P, and point Q, the relationship ∠PCF: ∠PCQ = PF: PQ holds. ∠PCF and ∠PCQ are angles in the camera image three-dimensional space shown in FIG. 8, and PF and PQ are lengths in the camera image plane shown in FIG. Here, ∠PCQ is the vertical angle of view range of the camera device C, and PQ is the vertical width of the camera image, so both are known values determined by the specifications of the camera device C. Since vanishing point F is calculated by a known method, PF is also known. ∠PCF is calculated from the above relationship.

  Next, as shown in FIG. 8, ∠OKC is calculated from the height Hc, which is the length of OC, and the ground distance Lmin, which is the length of OK, using a trigonometric function or the like. Since the straight line connecting point C and point F in FIG. 8 is parallel to the Yr axis, the calculated ∠OKC is the same as ∠UCF.

  Next, for the calculated ∠PCF and ∠UCF, the relationship ∠UCF: ∠PCF = UF: PF holds. PF and UF are the lengths in the camera image plane shown in FIG. The length of the UF is calculated from this relationship.

  The position of the point U on the camera image plane shown in FIG. 9 is calculated from the calculated UF. The point V on the camera image plane shown in FIG.

  As described above, the marker calculation unit 35 calculates the positions of the points U and V on the camera image plane from the ground distance ranges Lmin to Lmax in the Yr axis direction.

  Next, a method for calculating the corresponding range on the camera image plane from the azimuth angle ranges θ1 to θ2 in the camera three-dimensional space will be described.

  The azimuth angle in the camera three-dimensional space corresponds to the distance from the horizontal PQ in the camera image plane shown in FIG. The horizontal field angle range of the camera device C is a known range defined by the specifications, and corresponds to the left and right ends of the camera image plane in the horizontal direction. The marker calculation unit 35 calculates a range on the corresponding camera image plane, that is, a distance from the horizontal PQ, based on the azimuth range θ1 to θ2 and the horizontal view angle range of the camera device C.

  The vertical lines θ1 and θ2 shown in FIG. 9 correspond to the azimuth range θ1 and θ2.

  As described above, the marker calculation unit 35 calculates a range corresponding to each on the camera image plane from the azimuth angle ranges θ1 to θ2 in the camera three-dimensional space and the ground distance ranges Lmin to Lmax in the Yr axis direction. . Then, the marker calculation unit 35 uses a rectangular frame surrounding the calculated range as a marker. A marker B in FIG. 9 is a marker corresponding to the capture region A. The marker B is a rectangle surrounded by a horizontal straight line passing through the calculated points U and V, and the lines θ1 and θ2.

  FIG. 10 is a diagram illustrating an example of a calculation result of markers corresponding to the capture region illustrated in FIG. FIG. 10 shows markers B1, B2, and B3 on the camera image plane corresponding to the capture areas A1, A2, and A3 shown in FIG. In FIG. 10, the markers B <b> 1, B <b> 2, and B <b> 3 are superimposed on the edge of the camera image calculated by the edge calculation unit 34. As shown in FIG. 10, the marker on the camera image plane is calculated as a rectangle from each capture region on the radar measurement plane.

  The marker calculation method by each coordinate transformation described above is an example, and the present disclosure is not limited to this. The marker calculation unit 35 converts the capture region based on the range of azimuth angles that can be measured by the radar apparatus R in real space, the range of distance, and the range that can be photographed by the camera apparatus C, and calculates a marker. can do. The range of azimuth angles and the range of distances that can be measured by the radar device R in real space are determined in advance according to the installation position of the radar device R and the specifications of the radar device R. Further, the range in which the camera device C can shoot is determined in advance by the installation position of the camera device C and the specifications of the camera device C.

  Moreover, although the marker demonstrated above was made into the rectangle, this indication is not limited to this. The marker may have a non-rectangular shape.

  Next, a component area calculation method in the component area calculation unit 36 will be described.

  First, the component area calculation unit 36 superimposes the marker and the edge, and divides the marker when one marker overlaps the edge. In the case of FIG. 10, since the marker B2 overlaps the edge, the component area calculation unit 35 divides the marker B2.

  FIG. 11 is a diagram illustrating an example when the component area calculation unit 36 divides the marker. As shown in FIG. 11, in FIG. 10, the marker B2 that overlaps the edge is divided into a marker B21 and a marker B22.

  Next, the component region calculation unit 36 performs region expansion using a known image processing method such as a watershed algorithm with each marker as a region expansion seed and an edge as a region expansion boundary. Calculate the area. The component area is a partial area on the camera image plane corresponding to a part constituting the object.

  FIG. 12 is a diagram illustrating an example of the result of area expansion by the component calculation unit 36. As a result of the area expansion, the component area C1 is calculated from the markers B1 and B22, the component area C2 is calculated from the marker B21, and the component area C3 is calculated from the marker B3.

  Next, a method for grouping component areas in the grouping processing unit 37 will be described.

  The grouping processing unit 37 groups component regions belonging to the same object among the component regions calculated by the component region calculating unit 36. Whether or not the component areas belong to the same object is determined by one or both of information obtained from the camera image and information obtained from the radar measurement.

  The information obtained from the camera image is, for example, the texture of each component area in the camera image. The grouping processing unit 37 compares the textures of adjacent component areas, and groups adjacent component areas when the textures are similar. Whether the textures are similar may be determined by a predetermined threshold value or the like.

  As information obtained from radar measurement, for example, there is Doppler information. Doppler information is information on the speed of each point on the radar measurement plane. Here, the Doppler information is information on the radar measurement plane, and the component area is an area on the camera image plane. Therefore, when it is determined by Doppler information whether or not the component areas belong to the same object, it is necessary to perform coordinate conversion of the component area to an area on the radar measurement plane.

  The method for converting the component area into the area on the radar measurement plane may be performed in the reverse order of the method for calculating the marker from the capture area described above.

  FIG. 13 is a diagram illustrating an example of a result of coordinate conversion of a component area to an area on a radar measurement plane. The horizontal direction in FIG. 13 corresponds to the azimuth angle of the radar apparatus R, the vertical direction corresponds to the distance from the radar apparatus R, and each point (each cell) includes Doppler information. Regions D1, D2, and D3 in FIG. 13 correspond to component regions C1, C2, and C3 shown in FIG. 12, respectively.

  The grouping processing unit 37 compares the Doppler information included in the regions D1, D2, and D3, and groups the adjacent component regions on the camera image plane when the Doppler information is similar. Whether the Doppler information is similar may be determined by a predetermined threshold value or the like.

  FIG. 14 is a diagram illustrating an example of a grouping result by the grouping processing unit 37. As shown in FIG. 14, the component areas C1 and C2 in FIG. 12 are grouped to become the target object area E1, and the component area C3 in FIG. 12 is not grouped with the other, but becomes the target object area E2.

  In the example illustrated in FIG. 14, the grouping processing unit 37 acquires two target object regions E1 and E2 as a result of grouping.

  Next, a method for determining the target object in the object determination unit 38 will be described.

  The object determination unit 38 determines the position, size, shape, and object type of the target object based on the target object region that is the result of the grouping by the grouping processing unit 37. In the first embodiment of the present disclosure, a specific determination method in the object determination unit 38 is not limited. For example, the object determination unit 38 holds in advance a template model of the size and shape of the target object region corresponding to the type of the object, and compares the template model with the target object region that is the result of grouping by the grouping processing unit 37. The determination may be made by Alternatively, the object determination unit 38 may perform the determination by comparing with a template model of the reflection intensity distribution corresponding to the type of the object.

  For example, the case where it discriminate | determines using the template model with respect to the target object area | regions E1 and E2 shown in FIG. 14 is demonstrated. The object determination unit 38 compares the target object region E1 with a plurality of held template models and determines that the target object region E1 matches the template model of the vehicle. Further, the object determination unit 38 compares the target object region E2 with a plurality of held template models, and determines that the target object region E2 matches the template model of the pedestrian.

  According to the present embodiment described above, it is possible to improve the detection accuracy of the target object by converting the capture region on the radar measurement plane into a marker on the camera image plane and superimposing the marker on the camera image. it can. That is, by effectively superimposing the radar measurement information and the camera measurement information, it is possible to improve the detection accuracy of the target object.

(Embodiment 2)
FIG. 15 is a block diagram illustrating a main configuration of the object detection device 150 according to the second embodiment of the present disclosure. In FIG. 15, the same components as those in FIG. 3 are denoted by the same reference numerals as those in FIG. An object detection device 150 shown in FIG. 15 has a configuration in which the information generation unit 31 and the capture region calculation unit 32 of the object detection device 30 shown in FIG. 3 are replaced with an information generation unit 151 and a capture region calculation unit 152, respectively.

  The information generation unit 151 generates power profile information in the same manner as the information generation unit 31 of the first embodiment. Furthermore, the information generation unit 151 generates Doppler profile information indicating the Doppler speed of each cell from the delay profile received from the radar apparatus R. In the Doppler profile information, the horizontal axis indicates the azimuth and the vertical axis indicates the distance.

  The capture area calculation unit 152 calculates the capture area based on the power profile information and the Doppler profile information.

  Specifically, the capture area calculation unit 152 calculates the capture area from the power profile information by the method described in the first embodiment. Then, the capture region calculation unit 152 compares the Doppler velocities of each point (each cell) included in the capture region and determines whether the Doppler velocities match. The capture region calculation unit 152 excludes points (cells) whose Doppler profile values do not match from the capture region.

  The capture area calculation unit 152 outputs the calculated capture area to the marker calculation unit 35. In the marker calculation unit 35 and later, processing similar to that described in the first embodiment is executed.

  According to the present embodiment described above, the reflection intensity reflected from different objects is included in one capture region by excluding some points (cells) from the capture region using the Doppler velocity. Can be avoided.

  In the present embodiment, the capture region calculation unit 152 calculates the capture region based on the power profile information and the Doppler profile information. However, the capture region calculation unit 152 may calculate the capture region based on the Doppler profile information.

(Embodiment 3)
FIG. 16 is a block diagram illustrating a main configuration of the object detection device 160 according to the third embodiment of the present disclosure. In FIG. 16, the same components as those in FIG. 3 are denoted by the same reference numerals as those in FIG. An object detection device 160 illustrated in FIG. 16 has a configuration in which a reference frame determination unit 161 is inserted between the grouping processing unit 37 and the object determination unit 38 of the object detection device 30 illustrated in FIG.

  The normative frame determination unit 161 obtains a normative frame that covers the target object region as a result of grouping in the grouping processing unit 37. The reference frame is a frame reflecting the shape of the target object, and is, for example, a rectangular frame.

  The normative frame determination unit 161 surrounds the target object region with the obtained normative frame, and interpolates the grouping of the target object region that has been difficult to group in the grouping processing unit 37.

  According to the present embodiment described above, by performing grouping interpolation using the reference frame, the shape of the target object region can be made closer to the shape of the object, and the object determination unit 38 can determine the object. Accuracy can be improved.

(Embodiment 4)
FIG. 17 is a block diagram illustrating a main configuration of an object detection device 170 according to Embodiment 4 of the present disclosure. In FIG. 17, the same components as those in FIG. 3 are denoted by the same reference numerals as those in FIG. An object detection apparatus 170 illustrated in FIG. 17 has a configuration in which an area tracking unit 171 is inserted between the grouping processing unit 37 and the object determination unit 38 of the object detection apparatus 30 illustrated in FIG.

  The area tracking unit 171 tracks the position and shape of the target object area as a result of grouping by the grouping processing unit 37 at different times.

  Specifically, the area tracking unit 171 holds a target object area at a certain detection timing t1. The region tracking unit 171 receives the target object region at the next detection timing t2 from the grouping processing unit 37, and links the target object region at the detection timing t1 and the target object region at the detection timing t2. Then, the region tracking unit 171 tracks changes in the shape and position of the linked target object region, and detects the movement of the target object region.

  The area tracking unit 171 outputs information related to the movement of the target object area to the object determination unit 38. When determining the target object from the target object area, the object determination unit 38 may refer to information regarding the movement of the target object area. In addition, after determining the target object, the object determination unit 38 may output information on the movement of the target object together with information on the target object to an external display unit or a security system.

  According to the present embodiment described above, by tracking the position and shape of the target object region at different detection timings and detecting the movement of the target object region, it is possible to improve the accuracy of object discrimination, Furthermore, information regarding the movement of the object can be obtained.

  Note that the embodiments described above may be combined as appropriate. For example, in the object detection device 171 according to the fourth embodiment, the normative frame determination unit 161 described in the third embodiment may be inserted between the grouping processing unit 37 and the area tracking unit 171. In such a configuration, the shape of the target object region can be made closer to the shape of the object, and the detection accuracy of the motion of the target object by the region tracking unit 171 can be improved.

  Note that although cases have been described with the above embodiment as examples where the present disclosure is configured by hardware, the present disclosure can also be realized by software.

  Further, the method of circuit integration is not limited to LSI's, and implementation using dedicated circuitry or general purpose processors is also possible. An FPGA (Field Programmable Gate Array) that can be programmed after manufacturing the LSI or a reconfigurable processor that can reconfigure the connection and setting of circuit cells inside the LSI may be used.

  Furthermore, if integrated circuit technology comes out to replace LSI's as a result of the advancement of semiconductor technology or a derivative other technology, it is naturally also possible to carry out function block integration using this technology. Biotechnology can be applied.

  An object detection device and an object detection method according to the present disclosure are used for an on-vehicle radar device and a camera device, a road infrastructure system radar device and a camera device, and a facility monitoring system radar device and a camera device. Is preferred. When used in in-vehicle radar devices and camera devices, it detects pedestrians, motorcycles, and other vehicles in the vicinity of the host vehicle and alerts the driver of the host vehicle or controls the driving system to avoid the risk of collision Can be realized. When used in infrastructure system radar devices and camera devices, it detects pedestrians, motorcycles, vehicles, etc. on roads and intersections, monitors traffic conditions, controls infrastructure systems, and transmits information to vehicle drivers. In other words, it is possible to manage traffic volume and avoid traffic accidents. When used for radar equipment and camera equipment for surveillance systems in specific facilities, it detects flying objects and birds from the air, or various vehicles and intruders from the ground, and transmits information to the security system to prevent intruders from entering. can do.

30, 150, 160, 170 Object detection device 31, 151 Information generation unit 32, 152 Capture region calculation unit 33 Camera image acquisition unit 34 Edge calculation unit 35 Marker calculation unit 36 Component region calculation unit 37 Grouping processing unit 38 Object determination unit 161 Reference frame determination unit 171 Area tracking unit

Claims (10)

With respect to a plurality of cells obtained by dividing the distance from the radar device at predetermined intervals for each transmission direction of the radar signal transmitted by the radar device, the reflected signal obtained by reflecting the radar signal by one or more objects An information generation unit that calculates a reflection intensity that is a representative value of the power of the received signal received by the radar apparatus, and generates power profile information for each of the plurality of cells using the reflection intensity;
A cell indicating the maximum value of the reflection intensity from the power profile information of the plurality of cells is calculated as a capture point for capturing the one or more objects, and is one or more cells surrounding the capture point. A capture area calculation unit for calculating the capture area;
An edge extraction unit that extracts edges of the one or more objects included in the image acquired by the camera device;
Based on the measurement range of the radar device and the imaging range of the camera device, the capture region is converted into a partial region in the image, and the partial region is calculated as a marker that is a region of the image corresponding to the capture region. A marker calculation unit to perform,
A component area calculating unit that calculates a component area corresponding to a part of the one or more objects by extending the marker with the edge as a boundary;
A grouping processing unit that groups the component areas as target object areas;
An object determination unit that determines the one or more objects from the target object region and outputs a determination result;
An object detection apparatus comprising: The marker calculation unit has height information indicating a maximum possible value as the height of the one or more objects in advance, and uses the height information as the height of the one or more objects. convert the area, it calculates the marker,
The object detection apparatus according to claim 1. The component region calculation unit superimposes the marker on the edge and divides the marker.
The object detection apparatus according to claim 1 or 2. The information generation unit calculates a Doppler velocity for each cell based on a delay profile obtained by the radar device from the received signal, and generates Doppler profile information indicating the Doppler velocity of each cell,
The capture area calculation unit compares the Doppler velocities of the one or more cells included in the capture area, and excludes cells whose Doppler profile information values do not match from the capture area,
The object detection apparatus according to any one of claims 1 to 3. Providing a reference frame including the target object region, further comprising a reference frame determination unit that performs interpolation of grouping of the target object regions using the reference frame;
The object detection apparatus according to any one of claims 1 to 4. Further comprising a region tracking unit that tracks a change in the shape of the target object region over time and detects information related to the movement of the target object region;
The object detection apparatus according to any one of claims 1 to 5. The object detection device according to any one of claims 1 to 6,
A radar device connected to the object detection device;
An on-vehicle radar device. The object detection device according to any one of claims 1 to 6,
A radar device connected to the object detection device;
A radar device for road infrastructure systems. The object detection device according to any one of claims 1 to 6,
A radar device connected to the object detection device;
And a radar device for a monitoring system. With respect to a plurality of cells obtained by dividing the distance from the radar device at predetermined intervals for each transmission direction of the radar signal transmitted by the radar device, the reflected signal obtained by reflecting the radar signal by one or more objects Calculating a reflection intensity that is a representative value of the power of the received signal received by the radar device, and generating power profile information for each of the plurality of cells using the reflection intensity;
A cell indicating the maximum value of the reflection intensity from the power profile information of the plurality of cells is calculated as a capture point for capturing the one or more objects, and is one or more cells surrounding the capture point. Calculate the capture area,
Extracting an edge of the one or more objects included in the image acquired by the camera device;
Based on the measurement range of the radar device and the imaging range of the camera device, the capture region is converted into a partial region in the image, and the partial region is calculated as a marker that is a region of the image corresponding to the capture region. And
Calculating a component region corresponding to a portion constituting the one or more objects by extending the marker with the edge as a boundary;
Grouping the component areas as target object areas,
Determining the one or more objects from the target object region and outputting a determination result;
Object detection method.