WO2013035612A1 - Obstacle sensing device, obstacle sensing method, and obstacle sensing program - Google Patents

Obstacle sensing device, obstacle sensing method, and obstacle sensing program Download PDF

Info

Publication number
WO2013035612A1
WO2013035612A1 PCT/JP2012/071951 JP2012071951W WO2013035612A1 WO 2013035612 A1 WO2013035612 A1 WO 2013035612A1 JP 2012071951 W JP2012071951 W JP 2012071951W WO 2013035612 A1 WO2013035612 A1 WO 2013035612A1
Authority
WO
WIPO (PCT)
Prior art keywords
pixel
road surface
obstacle
obstacle detection
determined
Prior art date
Application number
PCT/JP2012/071951
Other languages
French (fr)
Japanese (ja)
Inventor
高橋　勝彦
Original Assignee
日本電気株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日本電気株式会社 filed Critical 日本電気株式会社
Publication of WO2013035612A1 publication Critical patent/WO2013035612A1/en

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01S—RADIO DIRECTION-FINDING; RADIO NAVIGATION; DETERMINING DISTANCE OR VELOCITY BY USE OF RADIO WAVES; LOCATING OR PRESENCE-DETECTING BY USE OF THE REFLECTION OR RERADIATION OF RADIO WAVES; ANALOGOUS ARRANGEMENTS USING OTHER WAVES
    • G01S17/00—Systems using the reflection or reradiation of electromagnetic waves other than radio waves, e.g. lidar systems
    • G01S17/88—Lidar systems specially adapted for specific applications
    • G01S17/89—Lidar systems specially adapted for specific applications for mapping or imaging
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle
    • G06T2207/30261—Obstacle

Definitions

  • the present invention relates to obstacle detection based on a distance image acquired by a camera or a distance sensor.
  • sensors that can acquire distance images such as cameras that project pattern light and measure distances based on the principle of triangulation, and time-of-flight (TOF) cameras, have appeared.
  • the obstacle refers to an object that exists closer than a predetermined distance.
  • Patent Document 1 and Patent Document 2 As a general technique for such obstacle detection, for example, the techniques described in Patent Document 1 and Patent Document 2 can be cited.
  • Patent Document 1 a three-dimensional coordinate value of a point in the real world corresponding to each pixel included in a distance image is obtained, a road surface plane is detected based on this, and a point having a predetermined height or more is detected from the road surface plane.
  • An object detection apparatus that detects a corresponding pixel as a target object is disclosed.
  • Patent Document 2 discloses a technique for reducing memory consumption when an obstacle is obtained by Hough transformation. Specifically, it is assumed that the surface corresponding to the obstacle is a plane perpendicular to the road surface. An image processing apparatus that performs Hough conversion while ignoring the Z component is disclosed. Further, Patent Document 2 discloses an image processing apparatus that, when detecting a horizontal plane, determines a candidate whose horizontal vector angle is within a predetermined range as a horizontal plane with respect to a plane candidate to be detected. ing.
  • obstacles can be detected by using a general object detection device or image processing device.
  • the general object detection device and the image processing device as described above are mounted on a vehicle and used for detecting an obstacle around the vehicle in a road environment, the following problems arise. This is a problem.
  • Patent Document 1 The biggest problem is that when the object detection device disclosed in Patent Document 1 is applied to a distance image obtained by imaging the road environment, an obstacle with a height within a certain range from the road surface is accurately detected. It is impossible.
  • FIGS. 2-1 and 2-2 examples of specific processing for detecting an obstacle based on the principle of object detection disclosed in Patent Document 1 are shown in FIGS. 2-1 and 2-2.
  • FIG. 2-1 is a diagram using captured images
  • FIG. 2-2 is a diagram that schematically illustrates the image of FIG. 2-1 for convenience of explanation.
  • FIGS. 2-1 and 2-2 (b) are based on the principle of object detection disclosed in Patent Document 1 from distance images taken in the road environment of FIGS. 2-1 and 2-2 (a). An example of the result of detecting an obstacle is shown.
  • FIGS. 2-1 and 2-2 in this example, as a pedestrian, a first person 11 is present in the foreground, and a second person 11 is present behind the left side. Yes.
  • the range of the pedestrian's calf to the toe approximately 30 cm from the road surface.
  • the road surface is mistakenly determined.
  • the threshold value for the height from the road surface which is used for the determination of the road surface and the obstacle, is lowered, the vicinity of the ankle can be detected.
  • the threshold value is lowered, there is a problem that the ratio of erroneously detecting a pixel corresponding to the road surface as an obstacle increases as described above.
  • Patent Document 2 On the other hand, according to the principle of obstacle detection disclosed in Patent Document 2, it is assumed that the surface of the obstacle is a plane perpendicular to the road surface. For this reason, it is not possible to detect a surface facing obliquely. In Patent Document 2, the horizontal plane is detected based on the angle of the vertical vector of the surface. As described above, in the outdoor road environment, noise is easily applied to the distance value of the pixel corresponding to the road surface. It is difficult to accurately determine the line direction. For this reason, it is difficult to correctly determine the road surface.
  • the present invention provides an obstacle detection device, an obstacle detection method, and an obstacle detection program capable of more accurately detecting an obstacle even in a range as close to the road surface as possible from a distance image capturing a road environment.
  • the purpose is to provide.
  • a distance image acquisition unit that acquires a distance image, and a point in the real world corresponding to each of the pixels based on a pixel value of each pixel included in the distance image
  • a three-dimensional coordinate conversion unit that calculates a three-dimensional coordinate, a road surface equation estimation unit that estimates a parameter of an equation representing a road surface based on the three-dimensional coordinate, and an equation that represents a road surface plane estimated by the road surface equation estimation unit
  • a road surface height estimation unit that estimates the height from the road surface plane for each pixel based on the parameters and the three-dimensional coordinates calculated for each pixel, and the height estimated by the road surface height estimation unit
  • a road surface pixel selecting unit that selects pixels having a value less than a predetermined first threshold as a road surface pixel, and for pixels that are not selected as the road surface pixels, an obstacle based on the pixel value of the pixel It is determined that the road An obstacle detection unit that determines whether a pixel is selected as a nearby pixel based on not
  • a distance image is acquired, and based on a pixel value of each pixel included in the distance image, three-dimensional coordinates of a point on the real world corresponding to each pixel are obtained.
  • Calculating, estimating a parameter of an equation representing a road plane based on the three-dimensional coordinates, and calculating each parameter based on the parameter of the equation representing the estimated road plane and the three-dimensional coordinates calculated for each pixel Estimate the height from the road surface every time, select the pixels whose estimated height is less than a predetermined first threshold as pixels near the road surface, and for pixels not selected as the road surface pixels Is determined to be an obstacle based on the pixel value of the pixel, and for pixels selected as pixels near the road surface, not only the pixel value of the pixel but also the pixel determined to be the obstacle Whether it is an obstacle based on the positional relationship Determining, obstacle detection and wherein the is provided.
  • a distance image acquisition unit that acquires a distance image, and a point on the real world corresponding to each of the pixels based on a pixel value of each pixel included in the distance image
  • a three-dimensional coordinate conversion unit that calculates a three-dimensional coordinate, a road surface equation estimation unit that estimates a parameter of an equation representing a road surface based on the three-dimensional coordinate, and an equation that represents a road surface plane estimated by the road surface equation estimation unit
  • a road surface height estimation unit that estimates the height from the road surface plane for each pixel based on the parameters and the three-dimensional coordinates calculated for each pixel, and the height estimated by the road surface height estimation unit
  • a road surface pixel selecting unit that selects pixels having a value less than a predetermined first threshold as a road surface pixel, and for pixels that are not selected as the road surface pixels, an obstacle based on the pixel value of the pixel It is determined that the road An obstacle detection unit that determines whether a pixel is selected as a nearby pixel based on not
  • the distance value of the point corresponding to the road surface and the estimated road surface equation are determined in consideration of the influence of noise, and the range as close to the road surface as possible from the distance image captured in the road environment is determined. It is also possible to detect obstacles more accurately.
  • FIG. 2 is a simplified schematic diagram of FIG. 2-1. It is a flowchart showing an example of operation
  • FIG. 7 is a simplified schematic diagram of FIG. It is a figure showing the example of an obstacle detection result by embodiment of this invention.
  • FIG. 8 is a simplified schematic diagram of FIG. 8-1. It is a figure showing an example of the composition for realizing the embodiment of the present invention.
  • an obstacle detection apparatus 100 includes a distance image acquisition unit 101, a three-dimensional coordinate conversion unit 102, a road surface equation estimation unit 103, a road surface height estimation unit 104, and a local surface direction.
  • An estimation unit 105, a road surface vicinity pixel selection unit 106, and an obstacle detection unit 107 are included.
  • the obstacle detection device 100 relates to an obstacle detection device that detects an obstacle present in a scene including a road surface (plane) based on a distance image.
  • the pixels near the road surface should be determined based on the height from the road surface.
  • the determination is made based on the characteristics of the obstacle itself in which noise is difficult to be superimposed. That is, the obstacle detection unit 107 determines an obstacle that is estimated to exist above a certain level from the road surface as an obstacle based only on the point, and is estimated to be present below a certain distance value from the road surface. Is determined to be an obstacle when the upper pixel has been determined to be an obstacle and the angle formed by the local normal direction of the surface and the normal direction of the road surface is equal to or greater than a threshold value.
  • the distance image acquisition unit 101 acquires a distance image obtained by converting the distance from the camera that has acquired the distance image to the object in the captured scene into a pixel value. Then, the distance image acquisition unit 101 outputs the acquired distance image to the three-dimensional coordinate conversion unit 102.
  • the distance image acquisition unit 101 can be realized by an arbitrary module.
  • the distance image acquisition unit 101 can be realized by a TOF camera or a distance sensor that measures a distance according to the principle of triangulation by projecting a near infrared pattern.
  • the separated image acquisition unit 101 itself does not have a function as a camera or a distance sensor, and may receive a distance image from a camera or a distance sensor connected to the outside.
  • the three-dimensional coordinate conversion unit 102 converts the distance value represented by the pixel value of the distance image output from the distance image acquisition unit 101 into 3D coordinates of a point on the corresponding object in the scene and outputs the converted 3D coordinate.
  • the focal length of the distance image acquisition unit 101 is f (cm), the number of horizontal pixels of the distance image is a (pixel), the number of pixels in the height direction of the distance image is b (pixel), and the horizontal width of the image plane is ⁇ (cm). ), And the vertical length of the image plane is h (cm).
  • the camera center is the origin
  • the optical axis direction of the distance image acquisition unit 101 is Z (axis)
  • the axis parallel to the vertical axis of the image plane is Y (axis)
  • the axis is parallel to the horizontal axis of the image plane.
  • ⁇ represents a constant representing a scale.
  • the pixel value of the distance image can be converted into a three-dimensional position (X, Y, Z) in the camera center coordinate system.
  • the road surface equation estimation unit 103 acquires the three-dimensional coordinates output from the three-dimensional coordinate conversion unit 102, and estimates and outputs an equation representing a road surface plane existing near the lower part of the distance image. It is easy to express the road surface equation in the same coordinate system as the three-dimensional coordinate output from the three-dimensional coordinate conversion unit 102, that is, the camera center coordinate system.
  • FIG. 3 is a flowchart showing the operation when calculating the road surface equation.
  • an area in the distance image used for calculating the road surface plane is determined (step S301). Any method can be used as the method of determining the region. For example, when the approximate elevation angle and depression angle of the distance image acquisition unit 101 are known, the position of the horizontal line in the distance image is roughly determined. Therefore, the pixel below the horizontal line may be determined as an area used for calculating the road surface plane. Further, it may be determined more simply, for example, a fixed area of the distance image may be used.
  • step S302 three points are randomly selected from the pixels included in the region determined to be used for calculating the road surface in step S301 (step S302).
  • a coefficient of an equation representing a plane passing through the three points is calculated based on the three-dimensional coordinate values corresponding to the three points selected in step S302 (step S303).
  • the number of pixels whose distance from the plane represented by the equation calculated in step S303 is equal to or less than a predetermined road surface equation estimation threshold (corresponding to the “third threshold” of the present invention) is counted. Then, the number of pixels as the counting result is stored in association with the equation calculated in step S303 (step S304).
  • a predetermined road surface equation estimation threshold corresponding to the “third threshold” of the present invention
  • N is an arbitrary integer of 1 or more. Setting a larger number of N increases the possibility of obtaining a more appropriate result, but the number of processing increases accordingly, so it is preferable to set an appropriate number according to the situation.
  • step S302 if the number of repetitions of the processing from step S302 to step S304 is less than N (No in step S305), three new random points are selected, and the processing is repeated from step S302 again.
  • step S305 when N times of processing have been completed (Yes in step S305), the equation when the number of pixels is the largest in step S304 is determined and output as an equation representing the true road surface plane (step S306).
  • step S306 the equation when the number of pixels is the largest is output based on the idea that the equation that correctly represents the road surface plane should have a larger number of pixels whose distance from the road surface plane is less than or equal to the threshold value.
  • the random sampling method such as steps S302 to S305 is called RANSAC (RANdom SAmple Consensus).
  • step S304 may be skipped. As a result, the processing time can be slightly reduced.
  • the road surface height estimation unit 104 calculates, for each point in the distance image, the distance between the corresponding point in the scene and the road surface plane output in step S306. Specifically, the road plane equation is
  • the local surface direction estimation unit 105 calculates, for each point in the distance image, a local surface direction (normal direction) near the corresponding point in the scene.
  • the normal direction of the surface is calculated by the following procedure. First, three points are selected from points existing in the vicinity of the point of interest in the distance image. And the equation of the plane passing through 3 points,
  • ⁇ ⁇ There are various ways to select the three points. For example, as shown in FIG. 4A, a pixel 402 that is in a fixed relative positional relationship around the pixel of interest 401 may be selected, or may be selected randomly. Instead of selecting three points, four pixels 402 around the pixel of interest 401 are selected as shown in FIG. 4B, and coefficients (L, M, N) are calculated using the least square method. You may make it ask.
  • the road surface vicinity pixel selecting unit 106 selects pixels whose distance between the road surface plane output in step S306 and a point on the real world is less than a predetermined first threshold. Sort as a pixel near the road surface. Then, the road surface vicinity pixel selection unit 106 outputs to the obstacle detection unit 107 each of the road surface vicinity pixel and the pixel that has not been selected as the road surface vicinity pixel.
  • the obstacle detection unit 107 detects an obstacle based on the outputs of the local surface direction estimation unit 105 and the road surface vicinity pixel selection unit 106. This will be described with reference to FIG. 5 which is a flowchart showing the obstacle detection process.
  • step S501 all the pixels that are not selected as road surface pixels by the road surface pixel selection unit 106 (that is, pixels whose distance between the road surface plane and a point in the real world is equal to or greater than a predetermined first threshold) are used as obstacles. It determines with corresponding (step S501).
  • other pixels in the vicinity of the road surface that is, pixels in which the distance between the road surface plane and a point in the real world is less than a predetermined first threshold
  • step S502 it is confirmed whether or not it is determined that a pixel adjacent immediately above the focused pixel corresponds to an obstacle (step S502). If the immediately adjacent pixel has already been determined to be an obstacle, it is verified whether the angle formed by the normal direction of the local surface of the pixel and the normal direction of the road surface is equal to or greater than the second threshold (step S503). . If it is equal to or greater than the threshold (Yes in step S503), it is determined that the pixel corresponds to an obstacle (step S504).
  • step S503 it is determined that the pixel is not an obstacle, that is, a pixel corresponding to the road surface (step S505).
  • FIG. 6 is a flowchart showing the overall operation of the obstacle detection apparatus 100.
  • the distance image acquisition unit 101 acquires a distance image (step S601).
  • the three-dimensional coordinate conversion unit 102 converts the distance information obtained for each pixel into three-dimensional coordinates on the object surface (step S602).
  • the road surface equation estimating unit 103 obtains and outputs an equation of a plane estimated to best describe the road surface plane (step S603).
  • the road surface height estimation unit 104 collates the road surface plane output from the road surface equation estimation unit 103 with the 3D coordinates of the point and measures the distance between the point and the road surface plane (step S604).
  • the local surface direction estimation unit 105 calculates the local surface orientation of each point on the image (step S605).
  • the road surface vicinity pixel selection unit 106 extracts pixels whose height estimated by the road surface height estimation unit is less than a predetermined first threshold as road surface vicinity pixels (step S606).
  • the obstacle detection unit 107 detects an obstacle based on the outputs of the local surface direction estimation unit 105 and the road surface vicinity pixel selection unit 106 (step S607).
  • the obstacle detection device 100 can detect an obstacle more accurately by the above operation, and can particularly detect an obstacle in a range close to the road surface with high accuracy. This is because, for pixels that can be determined as an obstacle relatively accurately based on the height from the road surface, a determination is made based on only the pixel value of the pixel, and pixels near the road surface are determined as an obstacle. This is because it is comprehensively determined whether or not the object is an obstacle based on the positional relationship with the pixel and the local surface direction.
  • pixels that are present near the road surface are not determined based on the height from the road surface, but are determined based on the characteristics of the obstacle itself that is difficult to superimpose noise, that is, the direction of the surface. is there. Also, based on the direction of the surface, only the pixels that are adjacent to the pixels that can be clearly determined to be obstacles based on the height from the road surface, or the pixels that are adjacent to the pixels that are determined to be obstacles based on the height from the road surface. This is because the use of pixels reduces the risk of erroneously determining an obstacle even if the surface direction cannot be obtained correctly with respect to the pixel corresponding to the road surface.
  • FIGS. 7-1 and 7-2, Fig. 8-1 and Fig. 8-2 show examples of processing results.
  • FIGS. 7-1 and 7-2 and FIGS. 8-1 and 8-2 are also in the same relationship as FIGS. 2-1 and 2-2, and the images of FIGS. FIGS. 7-2 and 8-2 are schematic diagrams more simply.
  • FIGS. 7-1 and 7-2 show the surface direction of each pixel obtained by the local surface direction estimation unit 105 in the situation of FIGS. 2-1 and 2-2 and the method of the road surface plane output by the road surface equation estimation unit 103.
  • An example of a result of extracting pixels whose angle with the line direction is equal to or greater than a second threshold is shown.
  • FIGS. 8-1 and 8-2 The obstacle detection results according to the present invention are shown in FIGS. 8-1 and 8-2. As can be seen from comparison with FIGS. 2-1 and 2-2 (b), it can be seen that the leg of the person in front and the leg of the person in the back of the left hand can be accurately detected to a point closer to the road surface.
  • the processing after the three-dimensional coordinate conversion unit 102 is expressed by the camera center coordinate system.
  • this is only an example, and it may be expressed by a coordinate system other than the camera center coordinate system.
  • a coordinate system other than the camera center coordinate system.
  • a world coordinate system is defined with respect to the horizontal plane and expressed in the world coordinate system. Since the transformation of the coordinate system can be performed by matrix operations well known to those skilled in the art, detailed description thereof is omitted.
  • the road surface equation estimation unit 103 calculates various plane equation candidates by selecting various three points by RANSAC, and calculates the candidate road surface with the maximum number of pixels whose distance from the plane is equal to or less than a predetermined threshold. The method of finally adopting the equation was explained. In this case, instead of selecting three points, various sets of three points may be selected to obtain an equation representing a plane by three-dimensional Hough transform.
  • step S502 it is determined in step S502 whether it has been determined that the pixel immediately above the pixel of interest corresponds to the obstacle. This may be confirmed as to whether or not it is determined that any one or more of the pixels adjacent to the pixel just above, the upper left, or the upper right is corresponding to the obstacle. By doing so, it becomes possible to better detect the vicinity of the lower part of the obliquely inclined obstacle.
  • the road surface equation estimation unit 103 estimates the parameters of the equation representing the road surface plane based on the three-dimensional coordinates.
  • the optical system internal parameters and external parameters of the distance image acquisition unit 101 are known and the road surface can be assumed to be substantially horizontal, only the parameters of the equation representing the road surface plane may be stored. That is, the road surface equation estimation unit 103 may be omitted from the obstacle detection apparatus 100 illustrated in FIG.
  • FIG. 9 is a block diagram showing a configuration of a computer 1000 for realizing the obstacle detection apparatus 100.
  • the computer 1000 includes an ECU (Electronic Control Unit) 1001, a camera 1002 and a hard disk 1003. By cooperating these units included in the computer 1000, each functional block included in the obstacle detection device 100 can be realized.
  • ECU Electronic Control Unit
  • the ECU 1001 controls the entire computer 1000 and includes, for example, a CPU, a RAM, a ROM, a signal processing circuit, a power supply circuit, and the like.
  • Functional blocks from the three-dimensional coordinate conversion unit 102 to the obstacle detection unit 107 are realized by the ECU 1001.
  • the distance image acquisition unit 101 is realized by the camera 1002.
  • the hard disk 1003 is a device for storing programs and data, and there is no problem even if it is configured by a storage medium other than the hard disk, such as SSD (Solid State Drive) realized by a flash memory.
  • SSD Solid State Drive
  • the hard disk 1003 stores a computer program for realizing each process and determination logic of the flowchart referred to in the description of the obstacle detection unit 107 from the three-dimensional coordinate conversion unit 102.
  • the ECU 1001 reads out the computer program from the hard disk 1003 and performs arithmetic processing, thereby realizing the above-described processes and determination logic.
  • microcomputer by making the functions executed by the ECU 1001 into hardware. Furthermore, some functions may be realized by hardware, and similar functions may be realized by cooperative operation of the hardware and the software program.
  • the ECU 1001, the camera 1002, and the hard disk 1003 are described as being realized by a single computer.
  • the ECU 1001, the camera 1002, and the hard disk 1003 may be realized by another computer and connected by means such as a bus or a cable compliant with the USB standard.
  • the connection may be a wired connection, but part or all of the connection may be a wireless connection.
  • the computer program has been described as being stored in the hard disk 1003 in advance.
  • a program for causing the computer 1000 to operate as all or part of the obstacle detection device 100 or to execute the above-described processing is a flexible disk, a CD-ROM (Compact Disc-Read-Only Memory), a DVD ( Stored in a computer-readable recording medium such as Digital Versatile Disc), MO (Magneto Optical Disk (Disc)) BD (Blu-ray Disc), etc., installed on another computer, and operates as the above-mentioned means Alternatively, the above-described steps may be executed.
  • the program may be stored in a disk device or the like of a server device on the Internet, and for example, the program may be superimposed on a carrier wave and downloaded to a computer to execute the program.
  • a distance image acquisition unit that acquires a distance image; Based on the pixel value of each pixel included in the distance image, a three-dimensional coordinate conversion unit that calculates three-dimensional coordinates of a point on the real world corresponding to each of the pixels;
  • a road surface equation estimating unit for estimating parameters of an equation representing a road surface based on the three-dimensional coordinates;
  • Road surface height estimation for estimating the height from the road surface plane for each pixel based on the parameters of the equation representing the road surface plane estimated by the road surface equation estimation unit and the three-dimensional coordinates calculated for each pixel
  • a road surface vicinity pixel selection unit that selects pixels whose estimated height of the road surface height estimation unit is less than a predetermined first threshold as road surface pixels;
  • a pixel that is not selected as a pixel near the road surface is determined to be an obstacle based on the pixel value of the pixel, and a pixel that is selected as a pixel near the road surface is determined only by the pixel value of the pixel.
  • Additional remark 2 It is an obstacle detection apparatus of Additional remark 1, Comprising: A local surface direction estimation unit that estimates a normal direction of a local surface in the real world from the three-dimensional coordinates calculated for each pixel; The obstacle detection unit determines whether or not the pixel selected as the road surface vicinity pixel is an obstacle in consideration of the estimation result of the local surface direction estimation unit. Detection device.
  • An obstacle detection apparatus of Additional remark 2, Comprising: The angle formed between the normal direction of the local surface estimated by the local surface direction estimation unit and the normal direction of the road surface plane indicated by the parameter of the equation representing the road surface plane is equal to or greater than a predetermined second threshold.
  • An obstacle detection device characterized in that if it is an obstacle, it is determined that the object is not an obstacle if it is less than a second threshold value.
  • the obstacle detection device (Supplementary note 4) The obstacle detection device according to any one of supplementary notes 1 to 3, The obstacle detection unit further satisfies the condition for the pixel selected as the pixel in the vicinity of the road surface on the condition that the pixel adjacent to the upper side of the pixel has been determined as an obstacle. In this case, the obstacle detection apparatus determines that the pixel is an obstacle.
  • the obstacle detection device is any one of a pixel adjacent to the upper side, a pixel adjacent to the upper right side, and a pixel adjacent to the upper left side of the pixel.
  • An obstacle detection device characterized in that a further condition is that one or more pixels have been determined as an obstacle, and the pixel is determined as an obstacle when the condition is further satisfied.
  • the obstacle detection device is An area used for calculation of a road surface plane is determined, a coefficient of an equation representing a plane passing through the plurality of points is calculated using a plurality of points randomly extracted from the determined area, and the calculated equation represents The process of counting the number of pixels whose distance from the plane is equal to or less than a predetermined third threshold is repeated N (an integer greater than or equal to 1) times, and the equation when the number of counted pixels is the largest is An obstacle detection apparatus characterized by determining an equation representing a true road surface plane.
  • a further condition is that a pixel adjacent to the upper side of the pixel has been determined as an obstacle, and the condition An obstacle detection method characterized in that the pixel is determined as an obstacle when the above is further satisfied.
  • the obstacle detection method according to any one of supplementary notes 7 to 9, In determining whether or not the obstacle is detected, for the pixels selected as the pixels near the road surface, the pixel adjacent to the upper side, the pixel adjacent to the upper right side, and the pixel adjacent to the upper left side of the pixel An obstacle detection method characterized by further determining that any one or more of the pixels have been determined as an obstacle and determining the pixel as an obstacle when the condition is further satisfied.
  • a distance image acquisition unit that acquires a distance image; Based on the pixel value of each pixel included in the distance image, a three-dimensional coordinate conversion unit that calculates three-dimensional coordinates of a point on the real world corresponding to each of the pixels;
  • a road surface equation estimating unit for estimating parameters of an equation representing a road surface based on the three-dimensional coordinates;
  • Road surface height estimation for estimating the height from the road surface plane for each pixel based on the parameters of the equation representing the road surface plane estimated by the road surface equation estimation unit and the three-dimensional coordinates calculated for each pixel
  • a road surface vicinity pixel selection unit that selects pixels whose estimated height of the road surface height estimation unit is less than a predetermined first threshold as road surface pixels;
  • a pixel that is not selected as a pixel near the road surface is determined to be an obstacle based on the pixel value of the pixel, and a pixel that is selected as a pixel near the road surface is determined only by the pixel value of the pixel.
  • the obstacle detection program according to supplementary note 13, wherein The computer, A local surface direction estimation unit that estimates a normal direction of a local surface in the real world from the three-dimensional coordinates calculated for each pixel; The obstacle detection unit further functions as an obstacle detection device that determines whether the pixel selected as the road vicinity pixel is an obstacle in consideration of the estimation result of the local surface direction estimation unit. Obstacle detection program characterized by causing
  • the obstacle detection program according to supplementary note 14, The angle formed between the normal direction of the local surface estimated by the local surface direction estimation unit and the normal direction of the road surface plane indicated by the parameter of the equation representing the road surface plane is equal to or greater than a predetermined second threshold.
  • An obstacle detection program characterized in that it is determined as an obstacle if it is less than the second threshold and is not an obstacle.
  • the obstacle detection program according to any one of supplementary notes 13 to 15, The obstacle detection unit further satisfies the condition for the pixel selected as the pixel in the vicinity of the road surface on the condition that the pixel adjacent to the upper side of the pixel has been determined as an obstacle.
  • An obstacle detection program characterized by determining the pixel as an obstacle.
  • the obstacle detection program according to any one of supplementary notes 13 to 15,
  • the obstacle detection unit is any one of a pixel adjacent to the upper side, a pixel adjacent to the upper right side, and a pixel adjacent to the upper left side of the pixel.
  • An obstacle detection program characterized in that a further condition is that one or more pixels have been determined as an obstacle, and the pixel is determined as an obstacle when the condition is further satisfied.
  • the obstacle detection program according to any one of supplementary notes 13 to 17,
  • the road surface equation estimator is An area used for calculation of a road surface plane is determined, a coefficient of an equation representing a plane passing through the plurality of points is calculated using a plurality of points randomly extracted from the determined area, and the calculated equation represents The process of counting the number of pixels whose distance from the plane is equal to or less than a predetermined third threshold is repeated N (an integer greater than or equal to 1) times, and the equation when the number of counted pixels is the largest is An obstacle detection program characterized by determining an equation representing a true road surface plane.

Abstract

An objective of the present invention is to more accurately sense, even in a range which is as close as possible to a road surface, an obstacle from a distance image which captures a road environment. A distance image is acquired. Three-dimensional coordinates are computed of points in the real world corresponding to each pixel included in the distance image on the basis of the pixel values of each pixel. A parameter is estimated of an equation which represents a road surface plane based on the three-dimensional coordinates. The height for each pixel from the road surface plane is estimated based on the estimated parameter of the equation which represents the road surface plane and the three-dimensional coordinates which are computed for each pixel. A pixel in which the height is less than a predetermined first threshold is selected as a near road surface pixel. A determination is made for a pixel which is not selected as the near road surface pixel being an obstacle based on the pixel value of the pixel. A determination is made for the pixel which is selected as the near road surface pixel as to whether said pixel is an obstacle based also on the location relation between said pixel and the pixel which is determined to be the obstacle as well as the pixel value of said pixel.

Description

障害物検知装置、障害物検知方法及び障害物検知プログラムObstacle detection device, obstacle detection method, and obstacle detection program
　本発明は、カメラや距離センサによって取得される距離画像に基づいた障害物検知に関する。 The present invention relates to obstacle detection based on a distance image acquired by a camera or a distance sensor.
　近年、パターン光を投影して三角測量の原理により距離計測するカメラや、ＴＯＦ（Time Of Flight）カメラなどといった、距離画像を取得できるセンサが登場している。これにより、一般的な単眼カメラでは難しかった障害物の検知が比較的容易にできるようになっている。なお、ここで障害物とは所定の距離よりも近くに存在する物体のことを指すものとする。 In recent years, sensors that can acquire distance images, such as cameras that project pattern light and measure distances based on the principle of triangulation, and time-of-flight (TOF) cameras, have appeared. This makes it possible to detect obstacles relatively easily, which was difficult with a general monocular camera. Here, the obstacle refers to an object that exists closer than a predetermined distance.
　このような障害物検知のための一般的な手法として、例えば特許文献１及び特許文献２に記載の手法が挙げられる。 As a general technique for such obstacle detection, for example, the techniques described in Patent Document 1 and Patent Document 2 can be cited.
　特許文献１には、距離画像に含まれる各画素に対応する実世界上の点の３次元座標値を求め、これに基づいて路面平面を検出し、路面平面から所定の高さ以上の点に対応する画素を対象物体として検出する物体検出装置が開示されている。 In Patent Document 1, a three-dimensional coordinate value of a point in the real world corresponding to each pixel included in a distance image is obtained, a road surface plane is detected based on this, and a point having a predetermined height or more is detected from the road surface plane. An object detection apparatus that detects a corresponding pixel as a target object is disclosed.
　また、特許文献２には、障害物をＨｏｕｇｈ変換により求める際のメモリ消費を低減するための手法が開示されている。具体的には、障害物に対応する面が路面に垂直な平面であると仮定する。そして、Ｚ成分を無視してＨｏｕｇｈ変換を行う画像処理装置が開示されている。更に、特許文献２には、水平平面の検出の際に、検出しようとする平面の候補に対して、垂直ベクトルの角度が所定の範囲内である候補を水平面と判断する画像処理装置が開示されている。 Patent Document 2 discloses a technique for reducing memory consumption when an obstacle is obtained by Hough transformation. Specifically, it is assumed that the surface corresponding to the obstacle is a plane perpendicular to the road surface. An image processing apparatus that performs Hough conversion while ignoring the Z component is disclosed. Further, Patent Document 2 discloses an image processing apparatus that, when detecting a horizontal plane, determines a candidate whose horizontal vector angle is within a predetermined range as a horizontal plane with respect to a plane candidate to be detected. ing.
特開平１０－１４３６５９号公報JP 10-143659 A 特許第４４２９４６１号Patent No. 4429461
　上述したように、一般的な物体検出装置や画像処理装置を用いることにより障害物の検知を行うことが可能となる。しかしながら、上述したような一般的な物体検出装置や画像処理装置を車両に搭載して、道路環境における車両周辺の障害物を検知する、という用途に用いる場合には下述するような課題が生じてしまうため問題となる。 As described above, obstacles can be detected by using a general object detection device or image processing device. However, when the general object detection device and the image processing device as described above are mounted on a vehicle and used for detecting an obstacle around the vehicle in a road environment, the following problems arise. This is a problem.
　最大の課題は、特許文献１に開示された物体検出装置を、道路環境を撮像した距離画像に対して適用しようとした場合、路面からの高さが一定以下の範囲の障害物を正確に検知できないことである。 The biggest problem is that when the object detection device disclosed in Patent Document 1 is applied to a distance image obtained by imaging the road environment, an obstacle with a height within a certain range from the road surface is accurately detected. It is impossible.
　なぜならば、アスファルトなどを成分とする道路の表面は光の反射率が低く、路面に対しカメラ光軸が浅い角度で交わるような場合、路面に対応する画素の距離値にノイズがのりやすいため、実際は路面に対応する点の３次元位置や路面平面を表す方程式の推定精度が低下するためである。この問題の対応策として、路面平面を誤って障害物と判定しないように路面からの高さに関する閾値を下げて設定することも考えられる。しかし閾値を下げてしまうと、路面平面からある程度の高さ以上の範囲に対しては正しく障害物として検出できるものの、閾値以下の範囲の障害物に対応する画素を路面領域と検出してしまうという新たな問題が発生する。 This is because the surface of a road that has asphalt or the like as a component has low light reflectance, and when the camera optical axis intersects at a shallow angle with respect to the road surface, noise tends to be applied to the distance value of the pixel corresponding to the road surface. This is because the estimation accuracy of the equation representing the three-dimensional position of the point corresponding to the road surface and the road surface plane actually decreases. As a countermeasure for this problem, it is conceivable to set the threshold value for the height from the road surface lower so as not to erroneously determine the road surface plane as an obstacle. However, if the threshold value is lowered, although it can be correctly detected as an obstacle for a range above a certain level from the road surface plane, pixels corresponding to obstacles in the range below the threshold value will be detected as a road surface area. New problems arise.
　この問題について具体的に説明するため、特許文献１に開示された物体検知の原理により障害物を検出した具体処理例を図２－１及び図２－２に示す。ここで、図２－１は、撮像した画像を用いた図であり、図２－２は、説明の便宜のために図２－１の画像をより簡略に模式化した図である。 In order to specifically explain this problem, examples of specific processing for detecting an obstacle based on the principle of object detection disclosed in Patent Document 1 are shown in FIGS. 2-1 and 2-2. Here, FIG. 2-1 is a diagram using captured images, and FIG. 2-2 is a diagram that schematically illustrates the image of FIG. 2-1 for convenience of explanation.
　そして、図２－１及び図２－２の（ａ）は、距離センサから見た実世界である道路環境の様子の一例を表している。 2-1 and 2-2 (a) show an example of the road environment in the real world as seen from the distance sensor.
　また、図２－１及び図２－２の（ｂ）は図２－１及び図２－２の（ａ）の道路環境において撮影した距離画像から特許文献１に開示された物体検知の原理により障害物を検出した結果の一例を表している。 FIGS. 2-1 and 2-2 (b) are based on the principle of object detection disclosed in Patent Document 1 from distance images taken in the road environment of FIGS. 2-1 and 2-2 (a). An example of the result of detecting an obstacle is shown.
　図２－１及び図２－２の（ａ）を参照すると、本例では歩行者として、手前に第１の人物１１が存在しており、その左側奥に第２の人物１１が存在している。 Referring to FIGS. 2-1 and 2-2 (a), in this example, as a pedestrian, a first person 11 is present in the foreground, and a second person 11 is present behind the left side. Yes.
　図２－１及び図２－２の（ｂ）の路面からの距離が一定以上であると判定した領域１３をみると、歩行者のふくらはぎから足先の範囲、およそ路面から３０ｃｍぐらいの範囲を誤って路面と判定してしまっている。もちろん、路面と障害物との判定に用いる、路面からの高さに関する閾値を下げれば脚首付近まで検知することはできる。しかし、閾値を下げてしまうと前述したように路面に対応する画素を障害物に誤検知する割合が増えてしまうという問題がある。 Looking at the region 13 in which the distance from the road surface in FIGS. 2-1 and 2-2 is determined to be a certain distance or more, the range of the pedestrian's calf to the toe, approximately 30 cm from the road surface. The road surface is mistakenly determined. Of course, if the threshold value for the height from the road surface, which is used for the determination of the road surface and the obstacle, is lowered, the vicinity of the ankle can be detected. However, if the threshold value is lowered, there is a problem that the ratio of erroneously detecting a pixel corresponding to the road surface as an obstacle increases as described above.
　一方、特許文献２に開示されている障害物検出の原理では、障害物の表面が、路面に垂直な平面であることを仮定している。そのため、斜めを向いた面を検出することができない。また、特許文献２では、面の垂直ベクトルの角度に基づき水平平面の検出を行うが、前述のように屋外の道路環境においては路面に対応する画素の距離値にノイズがのりやすく、面の法線方向を精度よく求めることが難しい。そのため、これにより道路面を正しく判定するのは難しい。 On the other hand, according to the principle of obstacle detection disclosed in Patent Document 2, it is assumed that the surface of the obstacle is a plane perpendicular to the road surface. For this reason, it is not possible to detect a surface facing obliquely. In Patent Document 2, the horizontal plane is detected based on the angle of the vertical vector of the surface. As described above, in the outdoor road environment, noise is easily applied to the distance value of the pixel corresponding to the road surface. It is difficult to accurately determine the line direction. For this reason, it is difficult to correctly determine the road surface.
　そこで、本発明は、道路環境をとらえた距離画像から、できるだけ路面に近い範囲についても障害物をより正確に検知することが可能な、障害物検知装置、障害物検知方法及び障害物検知プログラムを提供することを目的とする。 Therefore, the present invention provides an obstacle detection device, an obstacle detection method, and an obstacle detection program capable of more accurately detecting an obstacle even in a range as close to the road surface as possible from a distance image capturing a road environment. The purpose is to provide.
　本発明の第１の観点によれば、距離画像を取得する距離画像取得部と、前記距離画像に含まれる各画素の画素値に基づいて、該各画素のそれぞれに対応する実世界上の点の３次元座標を算出する３次元座標変換部と、前記３次元座標に基づき路面平面を表す方程式のパラメータを推定する路面方程式推定部と、前記路面方程式推定部が推定した路面平面を表す方程式のパラメータと、各画素に対して算出された前記３次元座標と、に基づき各画素毎に路面平面からの高さを推定する路面高さ推定部と、前記路面高さ推定部の推定した高さが予め定めた第１の閾値未満である画素を路面付近画素として選別する路面付近画素選別部と、前記路面付近画素に選別されなかった画素に対しては該画素の画素値に基づいて障害物であると判定し、路面付近画素に選別された画素に対しては、該画素の画素値だけではなく前記障害物であると判定された画素との位置関係にも基づいて障害物か否かを判定する障害物検知部と、を備えることを特徴とする障害物検知装置が提供される。 According to the first aspect of the present invention, a distance image acquisition unit that acquires a distance image, and a point in the real world corresponding to each of the pixels based on a pixel value of each pixel included in the distance image A three-dimensional coordinate conversion unit that calculates a three-dimensional coordinate, a road surface equation estimation unit that estimates a parameter of an equation representing a road surface based on the three-dimensional coordinate, and an equation that represents a road surface plane estimated by the road surface equation estimation unit A road surface height estimation unit that estimates the height from the road surface plane for each pixel based on the parameters and the three-dimensional coordinates calculated for each pixel, and the height estimated by the road surface height estimation unit A road surface pixel selecting unit that selects pixels having a value less than a predetermined first threshold as a road surface pixel, and for pixels that are not selected as the road surface pixels, an obstacle based on the pixel value of the pixel It is determined that the road An obstacle detection unit that determines whether a pixel is selected as a nearby pixel based on not only a pixel value of the pixel but also a positional relationship with the pixel determined to be the obstacle. An obstacle detection device is provided.
　本発明の第２の観点によれば、距離画像を取得し、前記距離画像に含まれる各画素の画素値に基づいて、該各画素のそれぞれに対応する実世界上の点の３次元座標を算出し、前記３次元座標に基づき路面平面を表す方程式のパラメータを推定し、前記推定した路面平面を表す方程式のパラメータと、各画素に対して算出された前記３次元座標と、に基づき各画素毎に路面平面からの高さを推定し、前記推定した高さが予め定めた第１の閾値未満である画素を路面付近画素として選別し、前記路面付近画素に選別されなかった画素に対しては該画素の画素値に基づいて障害物であると判定し、路面付近画素に選別された画素に対しては、該画素の画素値だけではなく前記障害物であると判定された画素との位置関係にも基づいて障害物か否かを判定する、ことを特徴とする障害物検知方法が提供される。 According to the second aspect of the present invention, a distance image is acquired, and based on a pixel value of each pixel included in the distance image, three-dimensional coordinates of a point on the real world corresponding to each pixel are obtained. Calculating, estimating a parameter of an equation representing a road plane based on the three-dimensional coordinates, and calculating each parameter based on the parameter of the equation representing the estimated road plane and the three-dimensional coordinates calculated for each pixel Estimate the height from the road surface every time, select the pixels whose estimated height is less than a predetermined first threshold as pixels near the road surface, and for pixels not selected as the road surface pixels Is determined to be an obstacle based on the pixel value of the pixel, and for pixels selected as pixels near the road surface, not only the pixel value of the pixel but also the pixel determined to be the obstacle Whether it is an obstacle based on the positional relationship Determining, obstacle detection and wherein the is provided.
　本発明の第３の観点によれば、距離画像を取得する距離画像取得部と、前記距離画像に含まれる各画素の画素値に基づいて、該各画素のそれぞれに対応する実世界上の点の３次元座標を算出する３次元座標変換部と、前記３次元座標に基づき路面平面を表す方程式のパラメータを推定する路面方程式推定部と、前記路面方程式推定部が推定した路面平面を表す方程式のパラメータと、各画素に対して算出された前記３次元座標と、に基づき各画素毎に路面平面からの高さを推定する路面高さ推定部と、前記路面高さ推定部の推定した高さが予め定めた第１の閾値未満である画素を路面付近画素として選別する路面付近画素選別部と、前記路面付近画素に選別されなかった画素に対しては該画素の画素値に基づいて障害物であると判定し、路面付近画素に選別された画素に対しては、該画素の画素値だけではなく前記障害物であると判定された画素との位置関係にも基づいて障害物か否かを判定する障害物検知部と、を備える障害物検知装置としてコンピュータを機能させることを特徴とする障害物検知プログラムが提供される。 According to the third aspect of the present invention, a distance image acquisition unit that acquires a distance image, and a point on the real world corresponding to each of the pixels based on a pixel value of each pixel included in the distance image A three-dimensional coordinate conversion unit that calculates a three-dimensional coordinate, a road surface equation estimation unit that estimates a parameter of an equation representing a road surface based on the three-dimensional coordinate, and an equation that represents a road surface plane estimated by the road surface equation estimation unit A road surface height estimation unit that estimates the height from the road surface plane for each pixel based on the parameters and the three-dimensional coordinates calculated for each pixel, and the height estimated by the road surface height estimation unit A road surface pixel selecting unit that selects pixels having a value less than a predetermined first threshold as a road surface pixel, and for pixels that are not selected as the road surface pixels, an obstacle based on the pixel value of the pixel It is determined that the road An obstacle detection unit that determines whether a pixel is selected as a nearby pixel based on not only a pixel value of the pixel but also a positional relationship with the pixel determined to be the obstacle. An obstacle detection program is provided that causes a computer to function as an obstacle detection device.
　本発明によれば、路面に対応する点の距離値や推定した路面方程式がノイズの影響を受けていることを考慮して判定を行い、道路環境において撮像した距離画像からできるだけ路面に近い範囲についても障害物をより正確に検知することが可能となる。 According to the present invention, the distance value of the point corresponding to the road surface and the estimated road surface equation are determined in consideration of the influence of noise, and the range as close to the road surface as possible from the distance image captured in the road environment is determined. It is also possible to detect obstacles more accurately.
本発明の実施形態の基本的構成を表す図である。It is a figure showing the basic composition of the embodiment of the present invention. 距離画像取得部から見た実世界の様子の一例を表す図、および前記実世界の状態に対する一般的手法による障害物検知結果例を表す図である。It is a figure showing an example of the state of the real world seen from the distance image acquisition part, and a figure showing the example of an obstacle detection result by the general method with respect to the state of the said real world. 図２－１を簡略化した模式図である。FIG. 2 is a simplified schematic diagram of FIG. 2-1. 路面方程式推定部の動作の一例を表すフローチャートである。It is a flowchart showing an example of operation | movement of a road surface equation estimation part. ３点又は４点の選択の仕方の一例を表す図である。It is a figure showing an example of the method of selection of 3 points | pieces or 4 points | pieces. 障害物検知部の動作の一例を表すフローチャートである。It is a flowchart showing an example of operation | movement of an obstruction detection part. 本発明の実施形態の動作の一例を表すフローチャートである。It is a flowchart showing an example of operation | movement of embodiment of this invention. 面の法線方向と路面平面の法線方向とのなす角度が第２の閾値以上の画素の検知結果例を表す図である。It is a figure showing the example of a detection result of the pixel whose angle which the normal line direction of a surface and the normal line direction of a road surface plane make is more than the 2nd threshold. 図７－１を簡略化した模式図である。FIG. 7 is a simplified schematic diagram of FIG. 本発明の実施形態による障害物検知結果例を表す図である。It is a figure showing the example of an obstacle detection result by embodiment of this invention. 図８－１を簡略化した模式図である。FIG. 8 is a simplified schematic diagram of FIG. 8-1. 本発明の実施形態を実現するための構成の一例を表す図である。It is a figure showing an example of the composition for realizing the embodiment of the present invention.
１０１　距離画像取得部
１０２　３次元座標変換部
１０３　路面方程式推定部
１０４　路面高さ推定部
１０５　局所面方向推定部
１０６　路面付近画素選別部
１０７　障害物検知部
４０１　着目画素
４０２　選択する周囲の画素
１００１　ＥＣＵ
１００２　カメラ
１００３　ハードディスク DESCRIPTION OF SYMBOLS 101 Distance image acquisition part 102 Three-dimensional coordinate transformation part 103 Road surface equation estimation part 104 Road surface height estimation part 105 Local surface direction estimation part 106 Road surface vicinity pixel selection part 107 Obstacle detection part 401 The attention pixel 402 The surrounding pixel 1001 ECU to select
1002 Camera 1003 Hard disk
　本発明の実施の形態について図面を参照して詳細に説明する。 Embodiments of the present invention will be described in detail with reference to the drawings.
　図１を参照すると、本発明の実施の形態である障害物検知装置１００は、距離画像取得部１０１、３次元座標変換部１０２、路面方程式推定部１０３、路面高さ推定部１０４、局所面方向推定部１０５、路面付近画素選別部１０６及び障害物検知部１０７を含む。 Referring to FIG. 1, an obstacle detection apparatus 100 according to an embodiment of the present invention includes a distance image acquisition unit 101, a three-dimensional coordinate conversion unit 102, a road surface equation estimation unit 103, a road surface height estimation unit 104, and a local surface direction. An estimation unit 105, a road surface vicinity pixel selection unit 106, and an obstacle detection unit 107 are included.
　障害物検知装置１００は、距離画像に基づき、路面（平面）を含むシーン中に存在する障害物を検知する障害物検知装置に関する。 The obstacle detection device 100 relates to an obstacle detection device that detects an obstacle present in a scene including a road surface (plane) based on a distance image.
　具体的には、路面に対応する点の距離値や推定した路面方程式がノイズの影響を受けていることを考慮して、路面付近に存在する画素については路面からの高さに基づいて判定せずに、ノイズが重畳しにくい障害物自体の特性に基づいて判定を行う。すなわち、障害物検知部１０７は、路面から一定以上上部に存在すると推定された点については該点のみに基づいて障害物に判定し、路面から一定の距離値未満に存在すると推定された点については、その上部の画素が障害物と判定済みで、かつ面の局所的法線方向と路面の法線方向とのなす角度が閾値以上である場合に障害物と判定を行う。 Specifically, considering that the distance value of the point corresponding to the road surface and the estimated road surface equation are affected by noise, the pixels near the road surface should be determined based on the height from the road surface. In addition, the determination is made based on the characteristics of the obstacle itself in which noise is difficult to be superimposed. That is, the obstacle detection unit 107 determines an obstacle that is estimated to exist above a certain level from the road surface as an obstacle based only on the point, and is estimated to be present below a certain distance value from the road surface. Is determined to be an obstacle when the upper pixel has been determined to be an obstacle and the angle formed by the local normal direction of the surface and the normal direction of the road surface is equal to or greater than a threshold value.
　続いて、障害物検知装置１００に含まれる各部の動作処理について詳細に説明する。 Subsequently, an operation process of each unit included in the obstacle detection apparatus 100 will be described in detail.
　距離画像取得部１０１は、距離画像を取得したカメラから撮像したシーン中の物体までの距離を画素値に変換した距離画像を取得する。そして、距離画像取得部１０１は、取得した距離画像を３次元座標変換部１０２に対して出力する。距離画像取得部１０１は、任意のモジュールにより実現可能であるが、例えばＴＯＦカメラもしくは近赤外線パターン投光して三角測量の原理により距離計測する距離センサ等により実現できる。また、離画像取得部１０１自体はカメラや距離センサとしての機能を有しておらず、外部に接続されたカメラや距離センサから、距離画像を受け取るようにしてもよい。 The distance image acquisition unit 101 acquires a distance image obtained by converting the distance from the camera that has acquired the distance image to the object in the captured scene into a pixel value. Then, the distance image acquisition unit 101 outputs the acquired distance image to the three-dimensional coordinate conversion unit 102. The distance image acquisition unit 101 can be realized by an arbitrary module. For example, the distance image acquisition unit 101 can be realized by a TOF camera or a distance sensor that measures a distance according to the principle of triangulation by projecting a near infrared pattern. Further, the separated image acquisition unit 101 itself does not have a function as a camera or a distance sensor, and may receive a distance image from a camera or a distance sensor connected to the outside.
　３次元座標変換部１０２は、距離画像取得部１０１が出力した距離画像の画素値により表される距離値を、対応するシーン中物体上の点の、３Ｄ座標に変換し出力する。 The three-dimensional coordinate conversion unit 102 converts the distance value represented by the pixel value of the distance image output from the distance image acquisition unit 101 into 3D coordinates of a point on the corresponding object in the scene and outputs the converted 3D coordinate.
　この３次元座標変換部１０２による３Ｄ座標変換の具体的計算方法について説明する。今回は、カメラ中心座標系における３Ｄ座標変換を行う。また、以下の説明では、透視射影モデルを仮定して説明する。 A specific calculation method of 3D coordinate conversion by the 3D coordinate conversion unit 102 will be described. This time, 3D coordinate transformation in the camera center coordinate system is performed. In the following description, a description is given assuming a perspective projection model.
　距離画像取得部１０１の焦点距離をｆ(cm)、距離画像の横幅の画素数をａ（画素）、距離画像の高さ方向の画素数をｂ（画素）、画像面の横幅をω（cm）、画像面の縦長をｈ（cm）、とする。また、座標系として、カメラ中心を原点、距離画像取得部１０１の光軸方向をＺ（軸）、画像面の縦軸に平行な軸をＹ（軸）、画像面の横軸に平行な軸をＸ（軸）とする。このとき、距離画像中の座標（ｘ，ｙ）の画素に対応するシーン中の点までの距離値をＤとすると、以下の関係が成り立つ。 The focal length of the distance image acquisition unit 101 is f (cm), the number of horizontal pixels of the distance image is a (pixel), the number of pixels in the height direction of the distance image is b (pixel), and the horizontal width of the image plane is ω (cm). ), And the vertical length of the image plane is h (cm). As a coordinate system, the camera center is the origin, the optical axis direction of the distance image acquisition unit 101 is Z (axis), the axis parallel to the vertical axis of the image plane is Y (axis), and the axis is parallel to the horizontal axis of the image plane. Is X (axis). At this time, if the distance value to the point in the scene corresponding to the pixel at the coordinates (x, y) in the distance image is D, the following relationship holds.
Figure JPOXMLDOC01-appb-M000001
Figure JPOXMLDOC01-appb-M000001
なお、上式においてαはスケールを表す定数を表す。これをαについて解くと、 In the above formula, α represents a constant representing a scale. Solving for α,
Figure JPOXMLDOC01-appb-M000002
Figure JPOXMLDOC01-appb-M000002
　が得られる。よって、αを代入することにより、距離画像の画素値からカメラ中心座標系における３次元位置（Ｘ,Ｙ,Ｚ）に変換することができる。 Is obtained. Therefore, by substituting α, the pixel value of the distance image can be converted into a three-dimensional position (X, Y, Z) in the camera center coordinate system.
　路面方程式推定部１０３は、３次元座標変換部１０２の出力する３次元座標を取得し、距離画像の下部付近に存在する路面平面を表す方程式を推定し、出力する。路面方程式は、３次元座標変換部１０２の出力する３次元座標と同じ座標系、すなわちカメラ中心座標系で表現するのが簡便である。 The road surface equation estimation unit 103 acquires the three-dimensional coordinates output from the three-dimensional coordinate conversion unit 102, and estimates and outputs an equation representing a road surface plane existing near the lower part of the distance image. It is easy to express the road surface equation in the same coordinate system as the three-dimensional coordinate output from the three-dimensional coordinate conversion unit 102, that is, the camera center coordinate system.
　路面方程式推定部１０３による路面方程式の推定手順を、図３を参照して説明する。図３は路面平面の方程式を算出する際の動作を表すフローチャートである。 The road surface equation estimation procedure by the road surface equation estimation unit 103 will be described with reference to FIG. FIG. 3 is a flowchart showing the operation when calculating the road surface equation.
　まず、路面平面の算出に利用する距離画像中の領域を定める（ステップＳ３０１）。領域の決定方法としては任意の方法を用いることが可能である。例えば、距離画像取得部１０１のおおよその仰角・俯角が既知の場合は、距離画像中における水平線の位置がおおよそ定まる。そのため、水平線よりも下側の画素を路面平面の算出に利用する領域と定めればよい。また、より単純に決定してもよく、例えば距離画像の固定的な領域を利用するようにしてもよい。 First, an area in the distance image used for calculating the road surface plane is determined (step S301). Any method can be used as the method of determining the region. For example, when the approximate elevation angle and depression angle of the distance image acquisition unit 101 are known, the position of the horizontal line in the distance image is roughly determined. Therefore, the pixel below the horizontal line may be determined as an area used for calculating the road surface plane. Further, it may be determined more simply, for example, a fixed area of the distance image may be used.
　次に、ステップＳ３０１において路面平面の算出に利用すると決定した領域に含まれている画素の中から３点をランダムに選出する（ステップＳ３０２）。 Next, three points are randomly selected from the pixels included in the region determined to be used for calculating the road surface in step S301 (step S302).
　続いて、ステップＳ３０２において選出した３点に対応する３次元座標値に基づき、３点を通る平面を表す方程式の係数を算出する（ステップＳ３０３）。 Subsequently, a coefficient of an equation representing a plane passing through the three points is calculated based on the three-dimensional coordinate values corresponding to the three points selected in step S302 (step S303).
　次に、ステップＳ３０３で算出した方程式が表す平面からの距離が、予め定めた閾値である路面方程式推定閾値（本発明の「第３の閾値」に相当）以下である画素の数を計数する。そして、計数結果である画素数をステップＳ３０３で算出した方程式と対応づけて記憶する（ステップＳ３０４）。なお、路面方程式推定閾値をどのような値にするかは、実装後の使用環境等に応じて任意に定めることが可能である。 Next, the number of pixels whose distance from the plane represented by the equation calculated in step S303 is equal to or less than a predetermined road surface equation estimation threshold (corresponding to the “third threshold” of the present invention) is counted. Then, the number of pixels as the counting result is stored in association with the equation calculated in step S303 (step S304). In addition, what kind of value the road surface equation estimation threshold value is made can be arbitrarily determined according to the usage environment after mounting.
　そして、ステップＳ３０２からステップＳ３０４までの処理をＮ回繰り返したか否かを判定する（ステップＳ３０５）。ここでＮは、１以上の任意の整数である。Ｎの数を大きく設定した方がより適切な結果が得られる可能性が上がるが、その分処理数も増えることとなるので、状況に応じて適切な数を設定することが好ましい。 Then, it is determined whether or not the processing from step S302 to step S304 has been repeated N times (step S305). Here, N is an arbitrary integer of 1 or more. Setting a larger number of N increases the possibility of obtaining a more appropriate result, but the number of processing increases accordingly, so it is preferable to set an appropriate number according to the situation.
　ここで、ステップＳ３０２からステップＳ３０４までの処理の繰り返し回数がＮ回未満の場合は（ステップＳ３０５においてＮｏ）、新たにランダムな３点を選び、再びステップＳ３０２から処理を繰り返す。 Here, if the number of repetitions of the processing from step S302 to step S304 is less than N (No in step S305), three new random points are selected, and the processing is repeated from step S302 again.
　一方、Ｎ回の処理を終えた際は（ステップＳ３０５においてＹｅｓ）、ステップＳ３０４で画素数がもっとも多かった場合の方程式を、真の路面平面を表す方程式として判定し出力する（ステップＳ３０６）。ステップＳ３０６で、画素数がもっとも多かった場合の方程式を出力するのは、もっとも路面平面を正しく表す方程式ほど路面平面との距離が閾値以下の画素数が多いはず、との考えに基づいている。なお、ステップＳ３０２～ステップＳ３０５のようなランダムサンプリングによる手法はＲＡＮＳＡＣ(RANdom SAmple Consensus)と呼ばれる。 On the other hand, when N times of processing have been completed (Yes in step S305), the equation when the number of pixels is the largest in step S304 is determined and output as an equation representing the true road surface plane (step S306). In step S306, the equation when the number of pixels is the largest is output based on the idea that the equation that correctly represents the road surface plane should have a larger number of pixels whose distance from the road surface plane is less than or equal to the threshold value. Note that the random sampling method such as steps S302 to S305 is called RANSAC (RANdom SAmple Consensus).
　また、距離画像取得部１０１のおおよその取り付け角度が既知であることから、路面平面を表す方程式の係数として期待される値が算出できるような場合はステップを一部省略するということも考えられる。具体的には、ステップＳ３０３実行後に得られた路面方程式の係数が、明らかに期待される値とは異なる場合には、該路面方程式が路面平面を表す方程式である可能性は極めて低いと考えて、該路面方程式に関してはステップＳ３０４を飛ばすようにしてもよい。これにより、処理時間を若干削減することができる。 Also, since the approximate mounting angle of the distance image acquisition unit 101 is known, it is conceivable that some of the steps may be omitted when an expected value can be calculated as an equation coefficient representing the road surface plane. Specifically, when the coefficient of the road surface equation obtained after execution of step S303 is clearly different from the expected value, the possibility that the road surface equation is an equation representing the road surface plane is considered extremely low. For the road surface equation, step S304 may be skipped. As a result, the processing time can be slightly reduced.
　路面高さ推定部１０４は、距離画像の各点に対して、対応するシーン中の点と、ステップＳ３０６により出力された路面平面との距離を算出する。具体的には、路面平面の方程式を、 The road surface height estimation unit 104 calculates, for each point in the distance image, the distance between the corresponding point in the scene and the road surface plane output in step S306. Specifically, the road plane equation is
Figure JPOXMLDOC01-appb-M000003
Figure JPOXMLDOC01-appb-M000003
とし、実世界中の点の座標を（Ｘ１，Ｙ１，Ｚ１）とすると、
路面平面とシーン中の点との距離Ｄ１は、 And the coordinates of points in the real world are (X 1 , Y 1 , Z 1 ),
The distance D 1 between the road surface plane and the point in the scene is
Figure JPOXMLDOC01-appb-M000004
Figure JPOXMLDOC01-appb-M000004
により求められる。 Is required.
　局所面方向推定部１０５は、距離画像中の各点に対して、対応するシーン中の点付近の局所的な面の向き（法線方向）を算出する。面の法線方向は以下の手順により算出する。まず、距離画像中の着目する点の近傍に存在する点の中から３点を選択する。そして、３点を通る平面の方程式、 The local surface direction estimation unit 105 calculates, for each point in the distance image, a local surface direction (normal direction) near the corresponding point in the scene. The normal direction of the surface is calculated by the following procedure. First, three points are selected from points existing in the vicinity of the point of interest in the distance image. And the equation of the plane passing through 3 points,
Figure JPOXMLDOC01-appb-M000005
Figure JPOXMLDOC01-appb-M000005
を算出すれば、ベクトル（Ｌ、Ｍ、Ｎ）が法線方向を表す。 Is calculated, the vector (L, M, N) represents the normal direction.
　３点の選択の仕方としては、種々の方法が考えられる。例えば、図４（ａ）に示すように、着目画素４０１の周囲の固定的な相対位置関係にある画素４０２を選択するようにしてもよいし、ランダムに選択するようにしてもよい。また、３点を選択するのではなく、図４（ｂ）のように着目画素４０１の周囲にある４点の画素４０２を選択し、最小二乗法を用いて係数（Ｌ、Ｍ、Ｎ）を求めるようにしてもよい。 種 々 There are various ways to select the three points. For example, as shown in FIG. 4A, a pixel 402 that is in a fixed relative positional relationship around the pixel of interest 401 may be selected, or may be selected randomly. Instead of selecting three points, four pixels 402 around the pixel of interest 401 are selected as shown in FIG. 4B, and coefficients (L, M, N) are calculated using the least square method. You may make it ask.
　路面付近画素選別部１０６は、路面高さ推定部１０４の演算結果に基づき、ステップＳ３０６により出力された路面平面と実世界上の点との距離が予め定めた第１の閾値未満である画素を路面付近画素として選別する。そして、路面付近画素選別部１０６は、路面付近画素と、路面付近画素に選別されなかった画素のそれぞれを障害物検知部１０７に対して出力する。 Based on the calculation result of the road surface height estimating unit 104, the road surface vicinity pixel selecting unit 106 selects pixels whose distance between the road surface plane output in step S306 and a point on the real world is less than a predetermined first threshold. Sort as a pixel near the road surface. Then, the road surface vicinity pixel selection unit 106 outputs to the obstacle detection unit 107 each of the road surface vicinity pixel and the pixel that has not been selected as the road surface vicinity pixel.
　障害物検知部１０７は、局所面方向推定部１０５および路面付近画素選別部１０６の出力に基づいて障害物を検知する。障害物検知処理を表すフローチャートである図５を参照しつつ説明する。 The obstacle detection unit 107 detects an obstacle based on the outputs of the local surface direction estimation unit 105 and the road surface vicinity pixel selection unit 106. This will be described with reference to FIG. 5 which is a flowchart showing the obstacle detection process.
　まず、路面付近画素選別部１０６により路面付近画素に選別されなかった画素（すなわち、路面平面と実世界上の点との距離が予め定めた第１の閾値以上である画素）を全て障害物に対応していると判定する（ステップＳ５０１）。以下、それ以外の路面付近画素（すなわち、路面平面と実世界上の点との距離が予め定めた第１の閾値未満である画素）に関して画像の上部から下部に向かってラスタスキャン順に１画素ずつ着目し、以下の処理を行う。 First, all the pixels that are not selected as road surface pixels by the road surface pixel selection unit 106 (that is, pixels whose distance between the road surface plane and a point in the real world is equal to or greater than a predetermined first threshold) are used as obstacles. It determines with corresponding (step S501). Hereinafter, other pixels in the vicinity of the road surface (that is, pixels in which the distance between the road surface plane and a point in the real world is less than a predetermined first threshold) pixel by pixel from the top to the bottom of the image in raster scan order. Pay attention to the following process.
　まず、着目した画素の真上に隣接する画素が障害物に対応していると判定済みか否かを確認する（ステップＳ５０２）。真上に隣接する画素が障害物に判定済みの場合は、該画素の局所面の法線方向と路面の法線方向とのなす角が第２の閾値以上であるか検証する（ステップＳ５０３）。閾値以上であれば（ステップＳ５０３においてＹｅｓ）、該画素を障害物に対応すると判定する（ステップＳ５０４）。 First, it is confirmed whether or not it is determined that a pixel adjacent immediately above the focused pixel corresponds to an obstacle (step S502). If the immediately adjacent pixel has already been determined to be an obstacle, it is verified whether the angle formed by the normal direction of the local surface of the pixel and the normal direction of the road surface is equal to or greater than the second threshold (step S503). . If it is equal to or greater than the threshold (Yes in step S503), it is determined that the pixel corresponds to an obstacle (step S504).
　それ以外の場合は（ステップＳ５０２においてＮｏ又はステップＳ５０３においてＮｏ）、障害物でない、すなわち路面に対応する画素であると判断する（ステップＳ５０５）。 Otherwise (No in step S502 or No in step S503), it is determined that the pixel is not an obstacle, that is, a pixel corresponding to the road surface (step S505).
　以上の説明では、障害物検知装置１００が有する各部の詳細な動作処理について説明した。続いて、障害物検知装置１００全体の動作について図面を参照して詳細に説明する。図６は障害物検知装置１００全体の動作を表すフローチャートである。 In the above description, the detailed operation processing of each unit included in the obstacle detection apparatus 100 has been described. Next, the overall operation of the obstacle detection apparatus 100 will be described in detail with reference to the drawings. FIG. 6 is a flowchart showing the overall operation of the obstacle detection apparatus 100.
　図６を参照すると、まず、距離画像取得部１０１が距離画像を取得する（ステップＳ６０１）。 Referring to FIG. 6, first, the distance image acquisition unit 101 acquires a distance image (step S601).
　次に、３次元座標変換部１０２が画素毎に得られている距離情報を、物体表面の３次元座標に変換する（ステップＳ６０２）。 Next, the three-dimensional coordinate conversion unit 102 converts the distance information obtained for each pixel into three-dimensional coordinates on the object surface (step S602).
　そして、路面方程式推定部１０３は路面平面をもっともよく記述していると推定される平面の方程式を求め出力する（ステップＳ６０３）。 Then, the road surface equation estimating unit 103 obtains and outputs an equation of a plane estimated to best describe the road surface plane (step S603).
　また、路面高さ推定部１０４は、路面方程式推定部１０３が出力した路面平面と点の３Ｄ座標とを照合して点と路面平面間の距離を計測する（ステップＳ６０４）。 Also, the road surface height estimation unit 104 collates the road surface plane output from the road surface equation estimation unit 103 with the 3D coordinates of the point and measures the distance between the point and the road surface plane (step S604).
　局所面方向推定部１０５は、画像上の各点の局所的な面の向きを算出する（ステップＳ６０５）。 The local surface direction estimation unit 105 calculates the local surface orientation of each point on the image (step S605).
　路面付近画素選別部１０６は前記路面高さ推定手段の推定した高さが予め定めた第１の閾値未満である画素を路面付近画素として抽出する（ステップＳ６０６）。 The road surface vicinity pixel selection unit 106 extracts pixels whose height estimated by the road surface height estimation unit is less than a predetermined first threshold as road surface vicinity pixels (step S606).
　障害物検知部１０７は、局所面方向推定部１０５および路面付近画素選別部１０６の出力に基づいて障害物を検知する（ステップＳ６０７）。 The obstacle detection unit 107 detects an obstacle based on the outputs of the local surface direction estimation unit 105 and the road surface vicinity pixel selection unit 106 (step S607).
　障害物検知装置１００は、以上の動作によって、障害物をより正確に検知することができ、特に路面に近い範囲の障害物を精度良く検知することが可能である。なぜならば、路面からの高さに基づいて比較的正確に障害物と判断できる画素に対しては該画素の画素値のみを基準として判定を行い、路面付近の画素については、障害物に判定済みの画素との位置関係、および局所面方向とに基づいて総合的に障害物か否かを判断するからである。 The obstacle detection device 100 can detect an obstacle more accurately by the above operation, and can particularly detect an obstacle in a range close to the road surface with high accuracy. This is because, for pixels that can be determined as an obstacle relatively accurately based on the height from the road surface, a determination is made based on only the pixel value of the pixel, and pixels near the road surface are determined as an obstacle. This is because it is comprehensively determined whether or not the object is an obstacle based on the positional relationship with the pixel and the local surface direction.
　また、路面に対応する点の距離値や推定した路面方程式がノイズの影響を受けていることを考慮しているからである。具体的には、路面付近に存在する画素については路面からの高さに基づいて判定するのではなく、ノイズが重畳しにくい障害物自体の特性、すなわち面の方向に基づいて判定を行うからである。また、路面からの高さに基づいて明らかに障害物であると判断できる画素と隣接している画素、もしくはそれによって障害物と判断された画素に隣接する画素のみを面の方向に基づく判定対象画素とすることによって、実際は路面に対応する画素に対して面方向が正しく求められない場合であっても、誤って障害物に判定する危険を低減しているためである。 This is also because the distance value of the point corresponding to the road surface and the estimated road surface equation are considered to be affected by noise. Specifically, pixels that are present near the road surface are not determined based on the height from the road surface, but are determined based on the characteristics of the obstacle itself that is difficult to superimpose noise, that is, the direction of the surface. is there. Also, based on the direction of the surface, only the pixels that are adjacent to the pixels that can be clearly determined to be obstacles based on the height from the road surface, or the pixels that are adjacent to the pixels that are determined to be obstacles based on the height from the road surface. This is because the use of pixels reduces the risk of erroneously determining an obstacle even if the surface direction cannot be obtained correctly with respect to the pixel corresponding to the road surface.
　図７－１及び図７－２、図８－１及び図８－２に処理結果の一例を示す。ここで、図７－１及び図７－２並びに図８－１及び図８－２も図２－１及び図２－２と同じ関係にあり、図７－１及び図８－１の画像をより簡略に模式化した図が図７－２及び図８－２である。 Fig. 7-1 and Fig. 7-2, Fig. 8-1 and Fig. 8-2 show examples of processing results. Here, FIGS. 7-1 and 7-2 and FIGS. 8-1 and 8-2 are also in the same relationship as FIGS. 2-1 and 2-2, and the images of FIGS. FIGS. 7-2 and 8-2 are schematic diagrams more simply.
　図７－１及び図７－２は図２－１及び図２－２の状況における局所面方向推定部１０５により求められる画素毎の面方向と、路面方程式推定部１０３の出力する路面平面の法線方向とのなす角度が第２の閾値以上の画素を抽出した結果例を示す。 FIGS. 7-1 and 7-2 show the surface direction of each pixel obtained by the local surface direction estimation unit 105 in the situation of FIGS. 2-1 and 2-2 and the method of the road surface plane output by the road surface equation estimation unit 103. An example of a result of extracting pixels whose angle with the line direction is equal to or greater than a second threshold is shown.
　また、本願発明による障害物検知結果を図８－１及び図８－２に示す。図２－１及び図２－２（ｂ）と比較すると分かるように、手前の人物の脚や左手奥にいる人物の脚がより路面に近いところまで正確に検知できていることがわかる。 The obstacle detection results according to the present invention are shown in FIGS. 8-1 and 8-2. As can be seen from comparison with FIGS. 2-1 and 2-2 (b), it can be seen that the leg of the person in front and the leg of the person in the back of the left hand can be accurately detected to a point closer to the road surface.
　また、上述した実施形態は、本発明の好適な実施形態ではあるが、上記実施形態のみに本発明の範囲を限定するものではなく、本発明の要旨を逸脱しない範囲において種々の変更を施した形態での実施が可能である。具体的な変更例として下記のような変形例が考えられる。 Moreover, although the above-described embodiment is a preferred embodiment of the present invention, the scope of the present invention is not limited only to the above-described embodiment, and various modifications are made without departing from the gist of the present invention. Implementation in the form is possible. The following modifications can be considered as specific modifications.
　上記の説明では、３次元座標変換部１０２以降の処理は、カメラ中心座標系により表現していた。しかしこれはあくまで一例に過ぎず、カメラ中心座標系以外の座標系により表現するようにしてもよい。例えば、水平面を基準としたワールド座標系を定義して、該ワールド座標系にて表現するのでも全く問題ない。座標系の変換は、当業者にとってよく知られた行列演算により行うことができるため詳細な説明は省略する。 In the above description, the processing after the three-dimensional coordinate conversion unit 102 is expressed by the camera center coordinate system. However, this is only an example, and it may be expressed by a coordinate system other than the camera center coordinate system. For example, there is no problem if a world coordinate system is defined with respect to the horizontal plane and expressed in the world coordinate system. Since the transformation of the coordinate system can be performed by matrix operations well known to those skilled in the art, detailed description thereof is omitted.
　また、上述の路面方程式推定部１０３では、ＲＡＮＳＡＣによりさまざまな３点を選んで平面の方程式の候補を算出し、平面からの距離が予め定めた閾値以下の画素数が最大となる路面の候補の方程式を最終的に採用する方式について説明した。この点、３点を選ぶのではなく、３点の組をさまざまに選んで３次元のＨｏｕｇｈ変換により平面を表す方程式を求めるようにしてもよい。 Further, the road surface equation estimation unit 103 calculates various plane equation candidates by selecting various three points by RANSAC, and calculates the candidate road surface with the maximum number of pixels whose distance from the plane is equal to or less than a predetermined threshold. The method of finally adopting the equation was explained. In this case, instead of selecting three points, various sets of three points may be selected to obtain an equation representing a plane by three-dimensional Hough transform.
　また、上述の障害物検知部１０７では、ステップＳ５０２にて、着目した画素の真上に隣接する画素が障害物に対応していると判定済みかを確認するとしている。これを、着目した画素の真上もしくは左上もしくは右上に隣接する画素のいずれか１つ以上が障害物に対応していると判定済みかを確認するものとしてもよい。そのようにすることによって、斜めに傾いた障害物の下部付近をより良好に検知することが可能となる。 In the obstacle detection unit 107 described above, it is determined in step S502 whether it has been determined that the pixel immediately above the pixel of interest corresponds to the obstacle. This may be confirmed as to whether or not it is determined that any one or more of the pixels adjacent to the pixel just above, the upper left, or the upper right is corresponding to the obstacle. By doing so, it becomes possible to better detect the vicinity of the lower part of the obliquely inclined obstacle.
　また、上述の障害物検知部１０７では、路面方程式推定部１０３により前記３次元座標に基づき路面平面を表す方程式のパラメータを推定するものとした。この点、距離画像取得部１０１の光学系内部パラメータおよび外部パラメータが既知で、かつ路面がほぼ水平と仮定できる場合であれば、路面平面を表す方程式のパラメータのみを記憶するようにしてもよい。すなわち、図１に表される障害物検知装置１００から、路面方程式推定部１０３を除いた構成としても構わない。 In the above-described obstacle detection unit 107, the road surface equation estimation unit 103 estimates the parameters of the equation representing the road surface plane based on the three-dimensional coordinates. In this regard, if the optical system internal parameters and external parameters of the distance image acquisition unit 101 are known and the road surface can be assumed to be substantially horizontal, only the parameters of the equation representing the road surface plane may be stored. That is, the road surface equation estimation unit 103 may be omitted from the obstacle detection apparatus 100 illustrated in FIG.
　次に、上述した障害物検知装置１００を実現する為の構成例について図９を参照して説明する。図９は、障害物検知装置１００を実現する為のコンピュータ１０００の構成を表すブロック図である。 Next, a configuration example for realizing the obstacle detection device 100 described above will be described with reference to FIG. FIG. 9 is a block diagram showing a configuration of a computer 1000 for realizing the obstacle detection apparatus 100.
　図９を参照すると、コンピュータ１０００は、ＥＣＵ（Electronic Control Unit）１００１、カメラ１００２及びハードディスク１００３を含んでいる。コンピュータ１０００が有するこれらの各部が協働することにより、障害物検知装置１００が有する各機能ブロックを実現することが可能となる。 Referring to FIG. 9, the computer 1000 includes an ECU (Electronic Control Unit) 1001, a camera 1002 and a hard disk 1003. By cooperating these units included in the computer 1000, each functional block included in the obstacle detection device 100 can be realized.
　ＥＣＵ１００１は、コンピュータ１０００全体の制御を行うものであり、例えばＣＰＵ、ＲＡＭ、ＲＯＭ、信号処理回路及び電源回路等を含む。 The ECU 1001 controls the entire computer 1000 and includes, for example, a CPU, a RAM, a ROM, a signal processing circuit, a power supply circuit, and the like.
　３次元座標変換部１０２から障害物検知部１０７までの機能ブロックはＥＣＵ１００１により実現される。また、距離画像取得部１０１は、カメラ１００２により実現される。 Functional blocks from the three-dimensional coordinate conversion unit 102 to the obstacle detection unit 107 are realized by the ECU 1001. The distance image acquisition unit 101 is realized by the camera 1002.
　また、ハードディスク１００３はプログラムやデータを記憶するための装置であり、フラッシュメモリにより実現されるＳＳＤ（Solid State Drive）等の、ハードディスク以外の記憶媒体により構成しても問題ない。 The hard disk 1003 is a device for storing programs and data, and there is no problem even if it is configured by a storage medium other than the hard disk, such as SSD (Solid State Drive) realized by a flash memory.
　更に、ハードディスク１００３には、３次元座標変換部１０２から障害物検知部１０７の説明において参照したフローチャートの各処理や判定ロジックを実現可能とするためのコンピュータ・プログラムが格納される。 Further, the hard disk 1003 stores a computer program for realizing each process and determination logic of the flowchart referred to in the description of the obstacle detection unit 107 from the three-dimensional coordinate conversion unit 102.
　そして、ＥＣＵ１００１が、ハードディスク１００３よりコンピュータ・プログラムを読み出して、演算処理を行うことにより上述の各処理や判定ロジックが実現される。 The ECU 1001 reads out the computer program from the hard disk 1003 and performs arithmetic processing, thereby realizing the above-described processes and determination logic.
　また、ＥＣＵ１００１で実行される機能をハードウェア化してマイコンを構成することも可能である。さらには、一部の機能をハードウェアで実現し、それらのハードウェアとソフトウェア・プログラムの協調動作により同様の機能を実現してもよい。 Further, it is also possible to configure a microcomputer by making the functions executed by the ECU 1001 into hardware. Furthermore, some functions may be realized by hardware, and similar functions may be realized by cooperative operation of the hardware and the software program.
　また、上記実施形態では、ＥＣＵ１００１、カメラ１００２及びハードディスク１００３が単一のコンピュータにより実現されているものとして説明した。しかし、ＥＣＵ１００１、カメラ１００２及びハードディスク１００３を別のコンピュータにより実現し、バスやＵＳＢ規格に準拠したケーブル等の手段を用いて接続するようにしてもよい。また、接続は有線接続であってもよいがその一部又は全部を無線による接続としてもよい。 In the above embodiment, the ECU 1001, the camera 1002, and the hard disk 1003 are described as being realized by a single computer. However, the ECU 1001, the camera 1002, and the hard disk 1003 may be realized by another computer and connected by means such as a bus or a cable compliant with the USB standard. The connection may be a wired connection, but part or all of the connection may be a wireless connection.
　更に、上記実施形態では、コンピュータ・プログラムが、ハードディスク１００３に予め記憶されているものとして説明した。しかし、コンピュータ１０００を、障害物検知装置１００の全部又は一部として動作させ、あるいは、上述の処理を実行させるためのプログラムを、フレキシブルディスク、ＣＤ－ＲＯＭ（Compact Disc Read-Only Memory）、ＤＶＤ（Digital Versatile Disc)、ＭＯ（Magneto Optical Disk(Disc)）ＢＤ（Blu-ray Disc）等のコンピュータ読み取り可能な記録媒体に格納して配布し、これを別のコンピュータにインストールし、上述の手段として動作させ、あるいは、上述の工程を実行させてもよい。 Furthermore, in the above-described embodiment, the computer program has been described as being stored in the hard disk 1003 in advance. However, a program for causing the computer 1000 to operate as all or part of the obstacle detection device 100 or to execute the above-described processing is a flexible disk, a CD-ROM (Compact Disc-Read-Only Memory), a DVD ( Stored in a computer-readable recording medium such as Digital Versatile Disc), MO (Magneto Optical Disk (Disc)) BD (Blu-ray Disc), etc., installed on another computer, and operates as the above-mentioned means Alternatively, the above-described steps may be executed.
　さらに、インターネット上のサーバ装置が有するディスク装置等にプログラムを格納しておき、例えば、搬送波にプログラムを重畳させて、コンピュータにダウンロード等してプログラムを実行してもよい。 Furthermore, the program may be stored in a disk device or the like of a server device on the Internet, and for example, the program may be superimposed on a carrier wave and downloaded to a computer to execute the program.
　本願は、日本の特願２０１１－１９７２５６（２０１１年９月９日に出願）に基づいたものであり、又、特願２０１１－１９７２５６に基づくパリ条約の優先権を主張するものである。特願２０１１－１９７２５６の開示内容は、特願２０１１－１９７２５６を参照することにより本明細書に援用される。 This application is based on Japanese Patent Application No. 2011-197256 (filed on September 9, 2011), and claims the priority of the Paris Convention based on Japanese Patent Application No. 2011-197256. The disclosed contents of Japanese Patent Application No. 2011-197256 are incorporated herein by reference to Japanese Patent Application No. 2011-197256.
　本発明の代表的な実施の形態が詳細に述べられたが、様々な変更(changes)、置き換え(substitutions)及び選択(alternatives)が請求項で定義された発明の精神と範囲から逸脱することなくなされることが理解されるべきである。また、仮にクレームが出願手続きにおいて補正されたとしても、クレームされた発明の均等の範囲は維持されるものと発明者は意図する。 Although exemplary embodiments of the present invention have been described in detail, various changes, substitutions and alternatives may be made without departing from the spirit and scope of the invention as defined in the claims. It should be understood that this is done. Moreover, even if the claim is amended in the application procedure, the inventor intends that the equivalent scope of the claimed invention is maintained.
　上記の実施形態の一部又は全部は、以下の付記のようにも記載されうるが、以下には限られない。 Some or all of the above embodiments can be described as in the following supplementary notes, but are not limited thereto.
　　（付記１）　距離画像を取得する距離画像取得部と、
　前記距離画像に含まれる各画素の画素値に基づいて、該各画素のそれぞれに対応する実世界上の点の３次元座標を算出する３次元座標変換部と、
　前記３次元座標に基づき路面平面を表す方程式のパラメータを推定する路面方程式推定部と、
　前記路面方程式推定部が推定した路面平面を表す方程式のパラメータと、各画素に対して算出された前記３次元座標と、に基づき各画素毎に路面平面からの高さを推定する路面高さ推定部と、
　前記路面高さ推定部の推定した高さが予め定めた第１の閾値未満である画素を路面付近画素として選別する路面付近画素選別部と、
　前記路面付近画素に選別されなかった画素に対しては該画素の画素値に基づいて障害物であると判定し、路面付近画素に選別された画素に対しては、該画素の画素値だけではなく前記障害物であると判定された画素との位置関係にも基づいて障害物か否かを判定する障害物検知部と、
　を備えることを特徴とする障害物検知装置。 (Supplementary Note 1) A distance image acquisition unit that acquires a distance image;
Based on the pixel value of each pixel included in the distance image, a three-dimensional coordinate conversion unit that calculates three-dimensional coordinates of a point on the real world corresponding to each of the pixels;
A road surface equation estimating unit for estimating parameters of an equation representing a road surface based on the three-dimensional coordinates;
Road surface height estimation for estimating the height from the road surface plane for each pixel based on the parameters of the equation representing the road surface plane estimated by the road surface equation estimation unit and the three-dimensional coordinates calculated for each pixel And
A road surface vicinity pixel selection unit that selects pixels whose estimated height of the road surface height estimation unit is less than a predetermined first threshold as road surface pixels;
A pixel that is not selected as a pixel near the road surface is determined to be an obstacle based on the pixel value of the pixel, and a pixel that is selected as a pixel near the road surface is determined only by the pixel value of the pixel. An obstacle detection unit that determines whether the object is an obstacle based on a positional relationship with the pixel that is determined to be the obstacle,
An obstacle detection device comprising:
　　（付記２）　付記１に記載の障害物検知装置であって、
　前記各画素に対して算出された前記３次元座標から実世界における局所的な面の法線方向を推定する局所面方向推定部を更に備え、
　前記障害物検知部は、前記路面付近画素として選別された画素に対しては、前記局所面方向推定部の推定結果をも考慮して障害物か否かを判定することを特徴とする障害物検知装置。 (Additional remark 2) It is an obstacle detection apparatus of Additional remark 1, Comprising:
A local surface direction estimation unit that estimates a normal direction of a local surface in the real world from the three-dimensional coordinates calculated for each pixel;
The obstacle detection unit determines whether or not the pixel selected as the road surface vicinity pixel is an obstacle in consideration of the estimation result of the local surface direction estimation unit. Detection device.
　（付記３）　付記２に記載の障害物検知装置であって、
　前記局所面方向推定部の推定した局所的な面の法線方向と、前記路面平面を表す方程式のパラメータが示す路面平面の法線方向とのなす角度が予め定めた第２の閾値以上であったならば障害物に判定し、第２の閾値未満の場合は障害物でないと判定することを特徴とする障害物検知装置。 (Additional remark 3) It is an obstacle detection apparatus of Additional remark 2, Comprising:
The angle formed between the normal direction of the local surface estimated by the local surface direction estimation unit and the normal direction of the road surface plane indicated by the parameter of the equation representing the road surface plane is equal to or greater than a predetermined second threshold. An obstacle detection device characterized in that if it is an obstacle, it is determined that the object is not an obstacle if it is less than a second threshold value.
　（付記４）　付記１乃至３の何れか１に記載の障害物検知装置であって、
　前記障害物検知部は、前記路面付近画素として選別された画素に対しては、該画素の上側に隣接する画素が障害物に判定済であることを更なる条件とし、該条件を更に満たした場合に該画素を障害物に判定することを特徴とする障害物検知装置。 (Supplementary note 4) The obstacle detection device according to any one of supplementary notes 1 to 3,
The obstacle detection unit further satisfies the condition for the pixel selected as the pixel in the vicinity of the road surface on the condition that the pixel adjacent to the upper side of the pixel has been determined as an obstacle. In this case, the obstacle detection apparatus determines that the pixel is an obstacle.
　（付記５）　付記１乃至３の何れか１に記載の障害物検知装置であって、
　前記障害物検知部は、前記路面付近画素として選別された画素に対しては、該画素の、上側に隣接する画素、右斜め上側に隣接する画素及び左斜め上側に隣接する画素の何れか１つ以上の画素が障害物に判定済であることを更なる条件とし、該条件を更に満たした場合に該画素を障害物に判定することを特徴とする障害物検知装置。 (Appendix 5) The obstacle detection device according to any one of appendices 1 to 3,
For the pixel selected as the road surface vicinity pixel, the obstacle detection unit is any one of a pixel adjacent to the upper side, a pixel adjacent to the upper right side, and a pixel adjacent to the upper left side of the pixel. An obstacle detection device characterized in that a further condition is that one or more pixels have been determined as an obstacle, and the pixel is determined as an obstacle when the condition is further satisfied.
　（付記６）　付記１乃至５の何れか１に記載の障害物検知装置であって、
　前記路面方程式推定部は、
　路面平面の算出に利用する領域を決定し、該決定した領域内からランダムに抽出した複数の点を用いて該複数の点を通る平面を表す方程式の係数を算出し、該算出した方程式が表す平面からの距離が、予め定めた第３の閾値以下である画素の数を計数する、と言う処理をＮ（１以上の整数）回繰り返し、計数した画素の数がもっとも多かった場合の方程式を真の路面平面を表す方程式として判定することを特徴とする障害物検知装置。 (Appendix 6) The obstacle detection device according to any one of appendices 1 to 5,
The road surface equation estimator is
An area used for calculation of a road surface plane is determined, a coefficient of an equation representing a plane passing through the plurality of points is calculated using a plurality of points randomly extracted from the determined area, and the calculated equation represents The process of counting the number of pixels whose distance from the plane is equal to or less than a predetermined third threshold is repeated N (an integer greater than or equal to 1) times, and the equation when the number of counted pixels is the largest is An obstacle detection apparatus characterized by determining an equation representing a true road surface plane.
　　（付記７）　距離画像を取得し、
　前記距離画像に含まれる各画素の画素値に基づいて、該各画素のそれぞれに対応する実世界上の点の３次元座標を算出し、
　前記３次元座標に基づき路面平面を表す方程式のパラメータを推定し、
　前記推定した路面平面を表す方程式のパラメータと、各画素に対して算出された前記３次元座標と、に基づき各画素毎に路面平面からの高さを推定し、
　前記推定した高さが予め定めた第１の閾値未満である画素を路面付近画素として選別し、
　前記路面付近画素に選別されなかった画素に対しては該画素の画素値に基づいて障害物であると判定し、路面付近画素に選別された画素に対しては、該画素の画素値だけではなく前記障害物であると判定された画素との位置関係にも基づいて障害物か否かを判定する、
　ことを特徴とする障害物検知方法。 (Appendix 7) Obtain a distance image,
Based on the pixel value of each pixel included in the distance image, calculate a three-dimensional coordinate of a point on the real world corresponding to each of the pixels,
Estimating parameters of an equation representing a road surface based on the three-dimensional coordinates;
Estimating the height from the road plane for each pixel based on the parameters of the equation representing the estimated road plane and the three-dimensional coordinates calculated for each pixel;
Selecting pixels whose estimated height is less than a predetermined first threshold as pixels near the road surface;
A pixel that is not selected as a pixel near the road surface is determined to be an obstacle based on the pixel value of the pixel, and a pixel that is selected as a pixel near the road surface is determined only by the pixel value of the pixel Determining whether it is an obstacle based also on the positional relationship with the pixel determined to be the obstacle without,
An obstacle detection method characterized by that.
　　（付記８）　付記７に記載の障害物検知方法であって、
　前記各画素に対して算出された前記３次元座標から実世界における局所的な面の法線方向を推定する局所面方向推定をし、
　前記障害物か否かの判定において、前記路面付近画素として選別された画素に対しては、前記局所面方向推定の推定結果をも考慮して障害物か否かを判定することを特徴とする障害物検知方法。 (Appendix 8) The obstacle detection method according to appendix 7,
Local surface direction estimation for estimating a normal direction of a local surface in the real world from the three-dimensional coordinates calculated for each pixel;
In determining whether or not the object is an obstacle, it is determined whether or not the pixel selected as the road surface vicinity pixel is an obstacle in consideration of the estimation result of the local surface direction estimation. Obstacle detection method.
　（付記９）　付記８に記載の障害物検知方法であって、
　前記局所面方向推定した局所的な面の法線方向と、前記路面平面を表す方程式のパラメータが示す路面平面の法線方向とのなす角度が予め定めた第２の閾値以上であったならば障害物に判定し、第２の閾値未満の場合は障害物でないと判定することを特徴とする障害物検知方法。 (Supplementary note 9) The obstacle detection method according to supplementary note 8,
If the angle formed by the normal direction of the local surface estimated by the local surface direction and the normal direction of the road surface plane indicated by the parameter of the equation representing the road surface plane is equal to or greater than a predetermined second threshold value An obstacle detection method, wherein an obstacle is determined, and if it is less than a second threshold, it is determined that the object is not an obstacle.
　（付記１０）　付記７乃至９の何れか１に記載の障害物検知方法であって、
　前記障害物検知か否かの判定において、前記路面付近画素として選別された画素に対しては、該画素の上側に隣接する画素が障害物に判定済であることを更なる条件とし、該条件を更に満たした場合に該画素を障害物に判定することを特徴とする障害物検知方法。 (Supplementary note 10) The obstacle detection method according to any one of supplementary notes 7 to 9,
In the determination as to whether or not the obstacle is detected, for the pixel selected as the pixel near the road surface, a further condition is that a pixel adjacent to the upper side of the pixel has been determined as an obstacle, and the condition An obstacle detection method characterized in that the pixel is determined as an obstacle when the above is further satisfied.
　（付記１１）　付記７乃至９の何れか１に記載の障害物検知方法であって、
　前記障害物検知か否かの判定において、前記路面付近画素として選別された画素に対しては、該画素の、上側に隣接する画素、右斜め上側に隣接する画素及び左斜め上側に隣接する画素の何れか１つ以上の画素が障害物に判定済であることを更なる条件とし、該条件を更に満たした場合に該画素を障害物に判定することを特徴とする障害物検知方法。 (Supplementary note 11) The obstacle detection method according to any one of supplementary notes 7 to 9,
In determining whether or not the obstacle is detected, for the pixels selected as the pixels near the road surface, the pixel adjacent to the upper side, the pixel adjacent to the upper right side, and the pixel adjacent to the upper left side of the pixel An obstacle detection method characterized by further determining that any one or more of the pixels have been determined as an obstacle and determining the pixel as an obstacle when the condition is further satisfied.
　（付記１２）　付記７乃至１１の何れか１に記載の障害物検知方法であって、
　前記路面方程式推定の推定では、
　路面平面の算出に利用する領域を決定し、該決定した領域内からランダムに抽出した複数の点を用いて該複数の点を通る平面を表す方程式の係数を算出し、該算出した方程式が表す平面からの距離が、予め定めた第３の閾値以下である画素の数を計数する、と言う処理をＮ（１以上の整数）回繰り返し、計数した画素の数がもっとも多かった場合の方程式を真の路面平面を表す方程式として判定することを特徴とする障害物検知方法。 (Supplementary note 12) The obstacle detection method according to any one of supplementary notes 7 to 11,
In the estimation of the road surface equation estimation,
An area used for calculation of a road surface plane is determined, a coefficient of an equation representing a plane passing through the plurality of points is calculated using a plurality of points randomly extracted from the determined area, and the calculated equation represents The process of counting the number of pixels whose distance from the plane is equal to or less than a predetermined third threshold is repeated N (an integer greater than or equal to 1) times, and the equation when the number of counted pixels is the largest is An obstacle detection method characterized by determining an equation representing a true road surface plane.
　　（付記１３）　距離画像を取得する距離画像取得部と、
　前記距離画像に含まれる各画素の画素値に基づいて、該各画素のそれぞれに対応する実世界上の点の３次元座標を算出する３次元座標変換部と、
　前記３次元座標に基づき路面平面を表す方程式のパラメータを推定する路面方程式推定部と、
　前記路面方程式推定部が推定した路面平面を表す方程式のパラメータと、各画素に対して算出された前記３次元座標と、に基づき各画素毎に路面平面からの高さを推定する路面高さ推定部と、
　前記路面高さ推定部の推定した高さが予め定めた第１の閾値未満である画素を路面付近画素として選別する路面付近画素選別部と、
　前記路面付近画素に選別されなかった画素に対しては該画素の画素値に基づいて障害物であると判定し、路面付近画素に選別された画素に対しては、該画素の画素値だけではなく前記障害物であると判定された画素との位置関係にも基づいて障害物か否かを判定する障害物検知部と、
　を備える障害物検知装置としてコンピュータを機能させることを特徴とする障害物検知プログラム。 (Supplementary Note 13) A distance image acquisition unit that acquires a distance image;
Based on the pixel value of each pixel included in the distance image, a three-dimensional coordinate conversion unit that calculates three-dimensional coordinates of a point on the real world corresponding to each of the pixels;
A road surface equation estimating unit for estimating parameters of an equation representing a road surface based on the three-dimensional coordinates;
Road surface height estimation for estimating the height from the road surface plane for each pixel based on the parameters of the equation representing the road surface plane estimated by the road surface equation estimation unit and the three-dimensional coordinates calculated for each pixel And
A road surface vicinity pixel selection unit that selects pixels whose estimated height of the road surface height estimation unit is less than a predetermined first threshold as road surface pixels;
A pixel that is not selected as a pixel near the road surface is determined to be an obstacle based on the pixel value of the pixel, and a pixel that is selected as a pixel near the road surface is determined only by the pixel value of the pixel. An obstacle detection unit that determines whether the object is an obstacle based on a positional relationship with the pixel that is determined to be the obstacle,
An obstacle detection program for causing a computer to function as an obstacle detection apparatus comprising:
　　（付記１４）　付記１３に記載の障害物検知プログラムであって、
　前記コンピュータを、
　前記各画素に対して算出された前記３次元座標から実世界における局所的な面の法線方向を推定する局所面方向推定部を更に備え、
　前記障害物検知部は、前記路面付近画素として選別された画素に対しては、前記局所面方向推定部の推定結果をも考慮して障害物か否かを判定する障害物検知装置として更に機能させることを特徴とする障害物検知プログラム。 (Supplementary note 14) The obstacle detection program according to supplementary note 13, wherein
The computer,
A local surface direction estimation unit that estimates a normal direction of a local surface in the real world from the three-dimensional coordinates calculated for each pixel;
The obstacle detection unit further functions as an obstacle detection device that determines whether the pixel selected as the road vicinity pixel is an obstacle in consideration of the estimation result of the local surface direction estimation unit. Obstacle detection program characterized by causing
　（付記１５）　付記１４に記載の障害物検知プログラムであって、
　前記局所面方向推定部の推定した局所的な面の法線方向と、前記路面平面を表す方程式のパラメータが示す路面平面の法線方向とのなす角度が予め定めた第２の閾値以上であったならば障害物に判定し、第２の閾値未満の場合は障害物でないと判定することを特徴とする障害物検知プログラム。 (Supplementary note 15) The obstacle detection program according to supplementary note 14,
The angle formed between the normal direction of the local surface estimated by the local surface direction estimation unit and the normal direction of the road surface plane indicated by the parameter of the equation representing the road surface plane is equal to or greater than a predetermined second threshold. An obstacle detection program characterized in that it is determined as an obstacle if it is less than the second threshold and is not an obstacle.
　（付記１６）　付記１３乃至１５の何れか１に記載の障害物検知プログラムであって、
　前記障害物検知部は、前記路面付近画素として選別された画素に対しては、該画素の上側に隣接する画素が障害物に判定済であることを更なる条件とし、該条件を更に満たした場合に該画素を障害物に判定することを特徴とする障害物検知プログラム。 (Supplementary note 16) The obstacle detection program according to any one of supplementary notes 13 to 15,
The obstacle detection unit further satisfies the condition for the pixel selected as the pixel in the vicinity of the road surface on the condition that the pixel adjacent to the upper side of the pixel has been determined as an obstacle. An obstacle detection program characterized by determining the pixel as an obstacle.
　（付記１７）　付記１３乃至１５の何れか１に記載の障害物検知プログラムであって、
　前記障害物検知部は、前記路面付近画素として選別された画素に対しては、該画素の、上側に隣接する画素、右斜め上側に隣接する画素及び左斜め上側に隣接する画素の何れか１つ以上の画素が障害物に判定済であることを更なる条件とし、該条件を更に満たした場合に該画素を障害物に判定することを特徴とする障害物検知プログラム。 (Supplementary note 17) The obstacle detection program according to any one of supplementary notes 13 to 15,
For the pixel selected as the road surface vicinity pixel, the obstacle detection unit is any one of a pixel adjacent to the upper side, a pixel adjacent to the upper right side, and a pixel adjacent to the upper left side of the pixel. An obstacle detection program characterized in that a further condition is that one or more pixels have been determined as an obstacle, and the pixel is determined as an obstacle when the condition is further satisfied.
　（付記１８）　付記１３乃至１７の何れか１に記載の障害物検知プログラムであって、
　前記路面方程式推定部は、
　路面平面の算出に利用する領域を決定し、該決定した領域内からランダムに抽出した複数の点を用いて該複数の点を通る平面を表す方程式の係数を算出し、該算出した方程式が表す平面からの距離が、予め定めた第３の閾値以下である画素の数を計数する、と言う処理をＮ（１以上の整数）回繰り返し、計数した画素の数がもっとも多かった場合の方程式を真の路面平面を表す方程式として判定することを特徴とする障害物検知プログラム。 (Supplementary note 18) The obstacle detection program according to any one of supplementary notes 13 to 17,
The road surface equation estimator is
An area used for calculation of a road surface plane is determined, a coefficient of an equation representing a plane passing through the plurality of points is calculated using a plurality of points randomly extracted from the determined area, and the calculated equation represents The process of counting the number of pixels whose distance from the plane is equal to or less than a predetermined third threshold is repeated N (an integer greater than or equal to 1) times, and the equation when the number of counted pixels is the largest is An obstacle detection program characterized by determining an equation representing a true road surface plane.

Claims (10)

  1. 　距離画像を取得する距離画像取得部と、
    　前記距離画像に含まれる各画素の画素値に基づいて、該各画素のそれぞれに対応する実世界上の点の３次元座標を算出する３次元座標変換部と、
    　前記３次元座標に基づき路面平面を表す方程式のパラメータを推定する路面方程式推定部と、
    　前記路面方程式推定部が推定した路面平面を表す方程式のパラメータと、各画素に対して算出された前記３次元座標と、に基づき各画素毎に路面平面からの高さを推定する路面高さ推定部と、
    　前記路面高さ推定部の推定した高さが予め定めた第１の閾値未満である画素を路面付近画素として選別する路面付近画素選別部と、
    　前記路面付近画素に選別されなかった画素に対しては該画素の画素値に基づいて障害物であると判定し、路面付近画素に選別された画素に対しては、該画素の画素値だけではなく前記障害物であると判定された画素との位置関係にも基づいて障害物か否かを判定する障害物検知部と、
    　を備えることを特徴とする障害物検知装置。     A distance image acquisition unit for acquiring a distance image;
    Based on the pixel value of each pixel included in the distance image, a three-dimensional coordinate conversion unit that calculates three-dimensional coordinates of a point on the real world corresponding to each of the pixels;
    A road surface equation estimating unit for estimating parameters of an equation representing a road surface based on the three-dimensional coordinates;
    Road surface height estimation for estimating the height from the road surface plane for each pixel based on the parameters of the equation representing the road surface plane estimated by the road surface equation estimation unit and the three-dimensional coordinates calculated for each pixel And
    A road surface vicinity pixel selection unit that selects pixels whose estimated height of the road surface height estimation unit is less than a predetermined first threshold as road surface pixels;
    A pixel that is not selected as a pixel near the road surface is determined to be an obstacle based on the pixel value of the pixel, and a pixel that is selected as a pixel near the road surface is determined only by the pixel value of the pixel An obstacle detection unit that determines whether the object is an obstacle based on a positional relationship with the pixel that is determined to be the obstacle,
    An obstacle detection device comprising:
  2. 　請求項１に記載の障害物検知装置であって、
    　前記各画素に対して算出された前記３次元座標から実世界における局所的な面の法線方向を推定する局所面方向推定部を更に備え、
    　前記障害物検知部は、前記路面付近画素として選別された画素に対しては、前記局所面方向推定部の推定結果をも考慮して障害物か否かを判定することを特徴とする障害物検知装置。     The obstacle detection device according to claim 1,
    A local surface direction estimation unit that estimates a normal direction of a local surface in the real world from the three-dimensional coordinates calculated for each pixel;
    The obstacle detection unit determines whether or not the pixel selected as the road surface vicinity pixel is an obstacle in consideration of the estimation result of the local surface direction estimation unit. Detection device.
  3. 　請求項２に記載の障害物検知装置であって、
    　前記局所面方向推定部の推定した局所的な面の法線方向と、前記路面平面を表す方程式のパラメータが示す路面平面の法線方向とのなす角度が予め定めた第２の閾値以上であったならば障害物に判定し、第２の閾値未満の場合は障害物でないと判定することを特徴とする障害物検知装置。     The obstacle detection device according to claim 2,
    The angle formed between the normal direction of the local surface estimated by the local surface direction estimation unit and the normal direction of the road surface plane indicated by the parameter of the equation representing the road surface plane is equal to or greater than a predetermined second threshold. An obstacle detection device characterized in that if it is an obstacle, it is determined that the object is not an obstacle if it is less than a second threshold value.
  4. 　請求項１乃至３の何れか１項に記載の障害物検知装置であって、
    　前記障害物検知部は、前記路面付近画素として選別された画素に対しては、該画素の上側に隣接する画素が障害物に判定済であることを更なる条件とし、該条件を更に満たした場合に該画素を障害物に判定することを特徴とする障害物検知装置。     The obstacle detection device according to any one of claims 1 to 3,
    The obstacle detection unit further satisfies the condition for the pixel selected as the pixel in the vicinity of the road surface on the condition that the pixel adjacent to the upper side of the pixel has been determined as an obstacle. In this case, the obstacle detection apparatus determines that the pixel is an obstacle.
  5. 　請求項１乃至３の何れか１項に記載の障害物検知装置であって、
    　前記障害物検知部は、前記路面付近画素として選別された画素に対しては、該画素の、上側に隣接する画素、右斜め上側に隣接する画素及び左斜め上側に隣接する画素の何れか１つ以上の画素が障害物に判定済であることを更なる条件とし、該条件を更に満たした場合に該画素を障害物に判定することを特徴とする障害物検知装置。     The obstacle detection device according to any one of claims 1 to 3,
    For the pixel selected as the road surface vicinity pixel, the obstacle detection unit is any one of a pixel adjacent to the upper side, a pixel adjacent to the upper right side, and a pixel adjacent to the upper left side of the pixel. An obstacle detection device characterized in that a further condition is that one or more pixels have been determined as an obstacle, and the pixel is determined as an obstacle when the condition is further satisfied.
  6. 　請求項１乃至５の何れか１項に記載の障害物検知装置であって、
    　前記路面方程式推定部は、
    　路面平面の算出に利用する領域を決定し、該決定した領域内からランダムに抽出した複数の点を用いて該複数の点を通る平面を表す方程式の係数を算出し、該算出した方程式が表す平面からの距離が、予め定めた第３の閾値以下である画素の数を計数する、と言う処理をＮ（１以上の整数）回繰り返し、計数した画素の数がもっとも多かった場合の方程式を真の路面平面を表す方程式として判定することを特徴とする障害物検知装置。     The obstacle detection device according to any one of claims 1 to 5,
    The road surface equation estimator is
    An area used for calculation of a road surface plane is determined, a coefficient of an equation representing a plane passing through the plurality of points is calculated using a plurality of points randomly extracted from the determined area, and the calculated equation represents The process of counting the number of pixels whose distance from the plane is equal to or less than a predetermined third threshold is repeated N (an integer greater than or equal to 1) times, and the equation when the number of counted pixels is the largest is An obstacle detection apparatus characterized by determining an equation representing a true road surface plane.
  7. 　距離画像を取得し、
    　前記距離画像に含まれる各画素の画素値に基づいて、該各画素のそれぞれに対応する実世界上の点の３次元座標を算出し、
    　前記３次元座標に基づき路面平面を表す方程式のパラメータを推定し、
    　前記推定した路面平面を表す方程式のパラメータと、各画素に対して算出された前記３次元座標と、に基づき各画素毎に路面平面からの高さを推定し、
    　前記推定した高さが予め定めた第１の閾値未満である画素を路面付近画素として選別し、
    　前記路面付近画素に選別されなかった画素に対しては該画素の画素値に基づいて障害物であると判定し、路面付近画素に選別された画素に対しては、該画素の画素値だけではなく前記障害物であると判定された画素との位置関係にも基づいて障害物か否かを判定する、
    　ことを特徴とする障害物検知方法。     Get a distance image,
    Based on the pixel value of each pixel included in the distance image, calculate a three-dimensional coordinate of a point on the real world corresponding to each of the pixels,
    Estimating parameters of an equation representing a road surface based on the three-dimensional coordinates;
    Estimating the height from the road plane for each pixel based on the parameters of the equation representing the estimated road plane and the three-dimensional coordinates calculated for each pixel;
    Selecting pixels whose estimated height is less than a predetermined first threshold as pixels near the road surface;
    A pixel that is not selected as a pixel near the road surface is determined to be an obstacle based on the pixel value of the pixel, and a pixel that is selected as a pixel near the road surface is determined only by the pixel value of the pixel. Determining whether it is an obstacle based also on the positional relationship with the pixel determined to be the obstacle without,
    An obstacle detection method characterized by that.
  8. 　請求項７に記載の障害物検知方法であって、
    　前記各画素に対して算出された前記３次元座標から実世界における局所的な面の法線方向を推定する局所面方向推定をし、
    　前記障害物か否かの判定において、前記路面付近画素として選別された画素に対しては、前記局所面方向推定の推定結果をも考慮して障害物か否かを判定することを特徴とする障害物検知方法。     The obstacle detection method according to claim 7,
    Local surface direction estimation for estimating a normal direction of a local surface in the real world from the three-dimensional coordinates calculated for each pixel;
    In determining whether or not the object is an obstacle, it is determined whether or not the pixel selected as the road surface vicinity pixel is an obstacle in consideration of the estimation result of the local surface direction estimation. Obstacle detection method.
  9. 　距離画像を取得する距離画像取得部と、
    　前記距離画像に含まれる各画素の画素値に基づいて、該各画素のそれぞれに対応する実世界上の点の３次元座標を算出する３次元座標変換部と、
    　前記３次元座標に基づき路面平面を表す方程式のパラメータを推定する路面方程式推定部と、
    　前記路面方程式推定部が推定した路面平面を表す方程式のパラメータと、各画素に対して算出された前記３次元座標と、に基づき各画素毎に路面平面からの高さを推定する路面高さ推定部と、
    　前記路面高さ推定部の推定した高さが予め定めた第１の閾値未満である画素を路面付近画素として選別する路面付近画素選別部と、
    　前記路面付近画素に選別されなかった画素に対しては該画素の画素値に基づいて障害物であると判定し、路面付近画素に選別された画素に対しては、該画素の画素値だけではなく前記障害物であると判定された画素との位置関係にも基づいて障害物か否かを判定する障害物検知部と、
    　を備える障害物検知装置としてコンピュータを機能させることを特徴とする障害物検知プログラム。     A distance image acquisition unit for acquiring a distance image;
    Based on the pixel value of each pixel included in the distance image, a three-dimensional coordinate conversion unit that calculates three-dimensional coordinates of a point on the real world corresponding to each of the pixels;
    A road surface equation estimating unit for estimating parameters of an equation representing a road surface based on the three-dimensional coordinates;
    Road surface height estimation for estimating the height from the road surface plane for each pixel based on the parameters of the equation representing the road surface plane estimated by the road surface equation estimation unit and the three-dimensional coordinates calculated for each pixel And
    A road surface vicinity pixel selection unit that selects pixels whose estimated height of the road surface height estimation unit is less than a predetermined first threshold as road surface pixels;
    A pixel that is not selected as a pixel near the road surface is determined to be an obstacle based on the pixel value of the pixel, and a pixel that is selected as a pixel near the road surface is determined only by the pixel value of the pixel. An obstacle detection unit that determines whether the object is an obstacle based on a positional relationship with the pixel that is determined to be the obstacle,
    An obstacle detection program for causing a computer to function as an obstacle detection apparatus comprising:
  10. 　請求項９に記載の障害物検知プログラムであって、
    　前記コンピュータを、
    　前記各画素に対して算出された前記３次元座標から実世界における局所的な面の法線方向を推定する局所面方向推定部を更に備え、
    　前記障害物検知部は、前記路面付近画素として選別された画素に対しては、前記局所面方向推定部の推定結果をも考慮して障害物か否かを判定する障害物検知装置として更に機能させることを特徴とする障害物検知プログラム。     The obstacle detection program according to claim 9,
    The computer,
    A local surface direction estimation unit that estimates a normal direction of a local surface in the real world from the three-dimensional coordinates calculated for each pixel;
    The obstacle detection unit further functions as an obstacle detection device that determines whether the pixel selected as the road vicinity pixel is an obstacle in consideration of the estimation result of the local surface direction estimation unit. Obstacle detection program characterized by causing
PCT/JP2012/071951 2011-09-09 2012-08-30 Obstacle sensing device, obstacle sensing method, and obstacle sensing program WO2013035612A1 (en)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2011197256 2011-09-09
JP2011-197256 2011-09-09

Publications (1)

Publication Number Publication Date
WO2013035612A1 true WO2013035612A1 (en) 2013-03-14

Family

ID=47832059

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2012/071951 WO2013035612A1 (en) 2011-09-09 2012-08-30 Obstacle sensing device, obstacle sensing method, and obstacle sensing program

Country Status (1)

Country Link
WO (1) WO2013035612A1 (en)

Cited By (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017036995A (en) * 2015-08-10 2017-02-16 朝日航洋株式会社 Laser measurement system and laser measurement method
JP6164546B1 (en) * 2016-11-07 2017-07-19 クモノスコーポレーション株式会社 Surveying method and surveying device
CN108885791A (en) * 2018-07-06 2018-11-23 深圳前海达闼云端智能科技有限公司 ground detection method, related device and computer readable storage medium
JP2020098188A (en) * 2018-09-27 2020-06-25 バイドゥ オンライン ネットワーク テクノロジー （ベイジン） カンパニー リミテッド Obstacle detection method, obstacle detection device, electronic apparatus, vehicle and storage medium
WO2021005659A1 (en) * 2019-07-05 2021-01-14 パナソニックセミコンダクターソリューションズ株式会社 Information processing system, sensor system, information processing method, and program
CN112561874A (en) * 2020-12-11 2021-03-26 杭州海康威视数字技术股份有限公司 Blocking object detection method and device and monitoring camera
JP2021085828A (en) * 2019-11-29 2021-06-03 株式会社豊田自動織機 Obstacle detector
JPWO2021199609A1 (en) * 2020-04-03 2021-10-07

Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10143659A (en) * 1996-11-06 1998-05-29 Komatsu Ltd Object detector
JP2002352249A (en) * 2001-05-23 2002-12-06 Toshiba Corp Image processor and its method
JP2005258941A (en) * 2004-03-12 2005-09-22 Toyota Central Res & Dev Lab Inc Device for detecting obstacle
JP2006349607A (en) * 2005-06-20 2006-12-28 Toyota Central Res & Dev Lab Inc Distance measuring device
JP2008026998A (en) * 2006-07-18 2008-02-07 Sumitomo Electric Ind Ltd Obstacle location calculation system and method
JP2009140023A (en) * 2007-12-03 2009-06-25 Mazda Motor Corp Vehicle obstacle detection unit
JP2010181919A (en) * 2009-02-03 2010-08-19 Toyohashi Univ Of Technology Three-dimensional shape specifying device, three-dimensional shape specifying method, three-dimensional shape specifying program

Patent Citations (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH10143659A (en) * 1996-11-06 1998-05-29 Komatsu Ltd Object detector
JP2002352249A (en) * 2001-05-23 2002-12-06 Toshiba Corp Image processor and its method
JP2005258941A (en) * 2004-03-12 2005-09-22 Toyota Central Res & Dev Lab Inc Device for detecting obstacle
JP2006349607A (en) * 2005-06-20 2006-12-28 Toyota Central Res & Dev Lab Inc Distance measuring device
JP2008026998A (en) * 2006-07-18 2008-02-07 Sumitomo Electric Ind Ltd Obstacle location calculation system and method
JP2009140023A (en) * 2007-12-03 2009-06-25 Mazda Motor Corp Vehicle obstacle detection unit
JP2010181919A (en) * 2009-02-03 2010-08-19 Toyohashi Univ Of Technology Three-dimensional shape specifying device, three-dimensional shape specifying method, three-dimensional shape specifying program

Cited By (18)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2017036995A (en) * 2015-08-10 2017-02-16 朝日航洋株式会社 Laser measurement system and laser measurement method
JP6164546B1 (en) * 2016-11-07 2017-07-19 クモノスコーポレーション株式会社 Surveying method and surveying device
JP2018077065A (en) * 2016-11-07 2018-05-17 クモノスコーポレーション株式会社 Survey method and location survey device
CN108072353A (en) * 2016-11-07 2018-05-25 蛛巢株式会社 Measuring method and measuring device
CN108072353B (en) * 2016-11-07 2020-06-02 蛛巢株式会社 Measuring method and measuring device
US10690478B2 (en) 2016-11-07 2020-06-23 Kumonos Corporation Survey method and survey apparatus
CN108885791A (en) * 2018-07-06 2018-11-23 深圳前海达闼云端智能科技有限公司 ground detection method, related device and computer readable storage medium
US11393219B2 (en) 2018-09-27 2022-07-19 Apollo Intelligent Driving Technology (Beijing) Co., Ltd. Method and apparatus for detecting obstacle, electronic device, vehicle and storage medium
JP2020098188A (en) * 2018-09-27 2020-06-25 バイドゥ オンライン ネットワーク テクノロジー （ベイジン） カンパニー リミテッド Obstacle detection method, obstacle detection device, electronic apparatus, vehicle and storage medium
JP7395301B2 (en) 2018-09-27 2023-12-11 アポロ インテリジェント ドライビング テクノロジー（ペキン）カンパニー リミテッド Obstacle detection method, obstacle detection device, electronic equipment, vehicle and storage medium
WO2021005659A1 (en) * 2019-07-05 2021-01-14 パナソニックセミコンダクターソリューションズ株式会社 Information processing system, sensor system, information processing method, and program
JP7372970B2 (en) 2019-07-05 2023-11-01 ヌヴォトンテクノロジージャパン株式会社 Information processing system, sensor system, information processing method and program
JPWO2021005659A1 (en) * 2019-07-05 2021-01-14
JP2021085828A (en) * 2019-11-29 2021-06-03 株式会社豊田自動織機 Obstacle detector
JP7230787B2 (en) 2019-11-29 2023-03-01 株式会社豊田自動織機 Obstacle detection device
JPWO2021199609A1 (en) * 2020-04-03 2021-10-07
WO2021199609A1 (en) * 2020-04-03 2021-10-07 日立Astemo株式会社 Object sensing device and object sensing method
CN112561874A (en) * 2020-12-11 2021-03-26 杭州海康威视数字技术股份有限公司 Blocking object detection method and device and monitoring camera

Similar Documents

Publication Publication Date Title
WO2013035612A1 (en) Obstacle sensing device, obstacle sensing method, and obstacle sensing program
US8976999B2 (en) Vehicle detection apparatus
WO2017057058A1 (en) Information processing device, information processing method, and program
RU2016150826A (en) DEPTH EVALUATION USING A MULTI-FULL STEREO IMAGE AND A CALIBRATED PROJECTOR
US10650535B2 (en) Measurement device and measurement method
JP5834933B2 (en) Vehicle position calculation device
EP3716210B1 (en) Three-dimensional point group data generation method, position estimation method, three-dimensional point group data generation device, and position estimation device
US20150310296A1 (en) Foreground region extraction device
JP2015184929A (en) Three-dimensional object detection apparatus, three-dimensional object detection method and three-dimensional object detection program
JP6638723B2 (en) Image analysis device, image analysis method, and image analysis program
US20200193184A1 (en) Image processing device and image processing method
JP2012252501A (en) Traveling path recognition device and traveling path recognition program
JP2018073275A (en) Image recognition device
JP6577595B2 (en) Vehicle external recognition device
JP2005156199A (en) Vehicle detection method and vehicle detector
JP2010020404A (en) Image processor and method thereof
JP5903901B2 (en) Vehicle position calculation device
JP5785515B2 (en) Pedestrian detection device and method, and vehicle collision determination device
JP7064400B2 (en) Object detection device
JP5891751B2 (en) Inter-image difference device and inter-image difference method
JP5272818B2 (en) Image processing apparatus and method, and program
JP6677142B2 (en) Parking frame recognition device
JP2021043141A (en) Object distance estimating device and object distance estimating method
JP3931885B2 (en) Obstacle detection device
WO2016181672A1 (en) Image analysis device, image analysis method, and image analysis program

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 12830638

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 12830638

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: JP