JP2011013803A - Peripheral shape detection device, autonomous mobile device, operation auxiliary device for mobile body, peripheral shape detection method, control method for the autonomous mobile device and operation auxiliary method for the mobile body - Google Patents

Abstract

PROBLEM TO BE SOLVED: To prevent detecting an upper face of grass or the like as a travelable plane in a course wherein the grass or the like is present, for example, in an autonomous mobile device.SOLUTION: This peripheral shape detection device includes: an imaging means imaging a stereo image including a standard image and a reference image by a plurality of cameras 3aL, 3aR; and a control means extracting a plane area inside the stereo image based on the stereo image obtained by the imaging means, performing object detection processing to an area except the extracted plane area, and detecting a movable area. The peripheral shape detection device distinguishes that a point having a ratio between a reflectance of a first wavelength band and a reflectance of a second wavelength band higher than a prescribed value is a plant leaf, excludes an area distinguished that the area is the plant leaf of the image imaged by the imaging means, and performs the extraction of the plane area and the object detection processing about a residual area.

Description

  The present invention relates to a surrounding shape detection device, an autonomous moving device that detects a surrounding shape and performs autonomous movement, a surrounding shape detection method, a control method for the autonomous moving device, and a moving body that assists maneuvering of a moving body performed by a human The present invention relates to a steering assist device and a steering assist method for a moving body.

  2. Description of the Related Art Conventionally, there has been proposed an autonomous mobile device that is equipped with a surrounding shape detection and control device, detects the surrounding shape by surrounding shape detection, and is controlled by the control device based on the detection result to perform autonomous movement. This autonomous mobile device moves on a travel route determined by itself so as to reach a predetermined target point while avoiding surrounding obstacles. In addition, such a control device can also assist in maneuvering a moving body performed by a human.

  In a control device that performs autonomous movement control and steering assistance, information on the surrounding situation is obtained, and based on this information, a course that can safely go to the destination is determined. Then, the control device controls the moving body so as to move on the determined route when performing autonomous movement, and transmits the determined route to the operator by display means or the like when performing steering assistance. .

  In order to realize the autonomous movement of the moving body outside the road or on rough terrain, the control device exists on the ground or the ground by a sensor 102 such as a laser range finder equipped on the moving body 101 as shown in FIG. The structure 103 and the three-dimensional shape of the tree are scanned and measured, and a flat portion with a small change in undulation is detected. Then, as shown in FIG. 10, the control device determines a course so as to pass through the flat portion based on the measurement result, and controls the moving body along the course.

  In addition, the control device detects the shape of the ground and surroundings on the ground to improve safety when performing steering assistance, and avoids or stops when approaching a dangerous distance. Control the moving body or inform the operator of the danger.

  As a method for detecting a plane portion that can travel from the surrounding shape, Patent Document 1 measures a three-dimensional position of the surroundings by using a stereo camera in which two or more cameras are combined, and uses the distribution of the positions to obtain a plane. A method is described in which voting is performed in a parameter space, a parameter having the most votes is selected, and a portion matching the parameter is set as a detection plane.

  Further, in Patent Document 2, as a method for detecting a plane portion from a surrounding shape, a projection based on a plane parameter assuming a reference image out of a standard image and a reference image captured by two cameras fixed to a moving body. This comparison is performed under image processing that obtains the result (an image of the absolute value of the difference) of the estimated reference image by comparing it with the actual reference image. A method for estimating a plane parameter so that the result has the highest degree of coincidence is described. In this method, by using the obtained plane parameter, a portion having a small difference is detected as a plane in the result of comparing the estimated reference image with the actual reference image (an image of the absolute value of the difference) as before.

JP 2003-271975 A JP 2006-54681 A

  By the way, as shown in FIGS. 11A and 11B, on the topography (parks, rivers, etc.) where the grass 104 is thick on the road or the like, the upper surface of the grass 104 is shown in FIG. A smooth surface may be formed with some variation, and there may be a large area. In such a case, in the conventional autonomous mobile device described in Patent Literature 1 and Patent Literature 2, there is a possibility that the upper surface of the grass 104 is detected as a travelable plane.

  Therefore, the present invention has been proposed in view of the above circumstances, and for example, surrounding shape detection in which the upper surface of grass or the like is detected as a travelable plane on a path where grass or the like exists. An apparatus and a surrounding shape detection method are provided, and an autonomous mobile device and a control method for the autonomous mobile device that can perform smooth movement by using such a surrounding shape detection device are provided. An object of the present invention is to provide such a steering assist device for a mobile object and a steering assist method for a mobile object.

  In order to solve the above-described problems and achieve the object, the present invention has any one of the following configurations.

[Configuration 1]
The surrounding shape detection apparatus according to the present invention extracts an image capturing unit that captures a stereo image including a standard image and a reference image by a plurality of cameras, and a planar area in the stereo image based on the stereo image obtained by the image capturing unit. And a control means for detecting a movable region by performing object detection processing on a region other than the extracted planar region, and the camera that captures the reference image has a high reflectance in the spectral reflectance of the plant leaf A function of capturing an image of a first wavelength band and a function of capturing an image of a second wavelength band having a locally low reflectance in the spectral reflectance of a plant leaf; For each point of the captured image, the ratio of the reflectance in the first wavelength band and the reflectance in the second wavelength band is obtained, and a point where this reflectance is higher than a predetermined value is determined to be a plant leaf. And for imaging means Of the captured image Ri, excluding the determined region as a leaf of the plant, the remaining region, is characterized in carrying out the extraction and object detection processing of the planar region.

[Configuration 2]
The surrounding shape detection device according to the present invention is a sampled and sampled three-dimensional data of three or more points from a three-dimensional data group, a camera for imaging the surroundings, shape measuring means for acquiring the surrounding shape as a three-dimensional data group, and Control means for calculating a plane from the three-dimensional data by Hough transform and extracting a sample point close to the plane as a movable region, and the camera has a first wavelength having a high reflectance in the spectral reflectance of the plant leaf A function of capturing an image of a band and a function of capturing an image of a second wavelength band in which the reflectance is locally low in the spectral reflectance of a plant leaf. For each point, the ratio between the reflectance of the first wavelength band and the reflectance of the second wavelength band is obtained, and the point where this reflectance is higher than a predetermined value is determined to be a plant leaf, and the shape measuring means Obtained by Of around the shape, in which excludes the determined region as a leaf of a plant, to extract the plane for the remaining regions, and detects the moving region.

[Configuration 3]
An autonomous mobile device according to the present invention is mounted on a traveling unit that is movable, an imaging unit that captures a stereo image including a reference image and a reference image by a plurality of cameras mounted on the traveling unit, and a traveling unit. A plane area in the stereo image is extracted based on the stereo image obtained by the image pickup means, and an object detection process is performed on an area other than the extracted plane area to detect a movable area and move in the movable area. A camera that captures a reference image and determines the planned path and controls the traveling unit to move on the planned travel path. The camera that captures the reference image is an image in the first wavelength band having a high reflectance in the spectral reflectance of the plant leaf. And a function of capturing an image of a second wavelength band having a locally low reflectance in the spectral reflectance of a plant leaf, and the control means is configured to control the image captured by the imaging means. For a point, the ratio between the reflectance of the first wavelength band and the reflectance of the second wavelength band is obtained, and a point having this reflectance higher than a predetermined value is determined to be a plant leaf, and imaged by the imaging means In the obtained image, a region determined to be a leaf of a plant is excluded, and a planar region is extracted and an object detection process is performed on the remaining region.

[Configuration 4]
An autonomous mobile device according to the present invention includes a traveling unit that is movable, a camera that is mounted on the traveling unit and images the surroundings of the traveling unit, and a shape of the surroundings of the traveling unit that is mounted on the traveling unit. The shape measuring means to be acquired as a group, and three or more three-dimensional data are sampled from the three-dimensional data group mounted on the traveling unit, a plane is calculated from the sampled three-dimensional data by Hough transform, and a sample point close to the plane is calculated The camera has a control means for extracting as a movable region, and the camera has a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of the plant leaf and a reflectance in the spectral reflectance of the plant leaf. A function of capturing an image of the second wavelength band that is locally low, and the control means includes a reflectance of the first wavelength band and a second wavelength band for each point of the image captured by the camera. Reflectivity The point where this reflectance is higher than a predetermined value is determined to be a plant leaf, and from the surrounding shapes acquired by the shape measuring means, the region determined to be a plant leaf is excluded. Then, a plane is extracted from the remaining area, and a movable area is detected.

[Configuration 5]
A steering assist device for a moving body according to the present invention includes an imaging unit that captures a stereo image including a reference image and a reference image by a plurality of cameras mounted on a steerable moving body, and an imaging unit that is mounted on the moving body. Control means for extracting a plane area in the stereo image based on the obtained stereo image and performing object detection processing on an area other than the extracted plane area to detect a movable area, and controlled by the control means And a camera that captures a reference image and has a function of capturing an image in a first wavelength band having a high reflectance in the spectral reflectance of the plant leaf and a local reflectance in the spectral reflectance of the plant leaf. And a function of capturing an image of the second wavelength band that is low in frequency, and the control means has a reflectance of the first wavelength band and a second wavelength band for each point of the image captured by the imaging means. Anti The point where the reflectance is higher than a predetermined value is determined to be a plant leaf, and the region determined to be a plant leaf is excluded from the image captured by the imaging means. The remaining area is subjected to extraction of a plane area and object detection processing.

[Configuration 6]
The moving body steering assist device according to the present invention is mounted on a steerable moving body and captures a surrounding image of the moving body mounted on the moving body as a three-dimensional data group. A shape measurement means that is mounted on a moving object, samples three or more three-dimensional data from a three-dimensional data group, calculates a plane from the sampled three-dimensional data by Hough transform, and moves a sample point close to the plane And a display unit controlled by the control unit. The camera has a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of the plant leaf and the leaf of the plant. And having a function of capturing an image of the second wavelength band in which the reflectance is locally low in the spectral reflectance, and the control unit is configured to counteract the first wavelength band for each point of the image captured by the imaging unit. The ratio between the reflectance and the reflectance of the second wavelength band is determined, a point where this reflectance is higher than a predetermined value is determined to be a plant leaf, and among the surrounding shapes acquired by the shape measuring means, the plant A region determined to be a leaf is excluded, a plane is extracted from the remaining region, and a movable region is detected.

[Configuration 7]
In the surrounding shape detection method according to the present invention, a stereo image including a base image and a reference image is captured by a plurality of cameras, and a plane area in the stereo image is extracted based on the stereo image, and other than the extracted plane area A surrounding shape detection method for performing an object detection process on a region of the above, and as a camera for capturing a reference image, a function of capturing an image in a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf; Using the one having a function of capturing an image of the second wavelength band having a locally low reflectance in the spectral reflectance of the leaf of the plant, the reflectance of the first wavelength band for each point of the reference image A ratio with the reflectance of the second wavelength band is obtained, a point where the reflectance is higher than a predetermined value is determined to be a plant leaf, and an area determined to be a plant leaf in the reference image is determined. Exclude the remaining area Te, and is characterized in carrying out the extraction and object detection processing of the planar region.

[Configuration 8]
The surrounding shape detection method according to the present invention acquires a surrounding shape as a three-dimensional data group, samples three or more three-dimensional data from the three-dimensional data group, and calculates a plane by Hough transform from the sampled three-dimensional data. A surrounding shape detection method for extracting a sample point close to the plane as a movable region, and a function for capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf and the leaf of the plant Using a camera having a function of capturing an image in the second wavelength band where the reflectance is locally low in spectral reflectance, the reflectance in the first wavelength band for each point of the image captured by this camera And the reflectance of the second wavelength band is determined, a point where the reflectance is higher than a predetermined value is determined to be a leaf of the plant, and among the surrounding shapes acquired by the shape measuring means, Leaves Excluding the determined area, the remaining area by extracting plane, is characterized in that to detect the moving region.

[Configuration 9]
The autonomous mobile device control method according to the present invention includes a traveling unit that is movable, and an imaging unit that captures a stereo image including a reference image and a reference image using a plurality of cameras mounted on the traveling unit. A method for controlling an autonomous mobile device, which functions as a camera for capturing a reference image and captures an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf and the spectral reflectance of the plant leaf 2 having a function of capturing an image of the second wavelength band whose reflectance is locally low, and for each point of the image captured by the imaging means, the reflectance of the first wavelength band and the second The ratio of the reflectance with the reflectance of the wavelength band is determined to be a leaf of the plant at a point where the reflectance is higher than a predetermined value, and it is determined to be the leaf of the plant among the images captured by the imaging means. The remaining area and the remaining area , Extracting a planar area in the stereo image, performing object detection processing on an area other than the extracted planar area, detecting a movable area, determining a planned moving path in the movable area, Is controlled to move on the planned movement path.

[Configuration 10]
The control method of the autonomous mobile device according to the present invention includes a traveling unit that is movable, a camera that is mounted on the traveling unit and images the surroundings of the traveling unit, and a shape of the surroundings of the traveling unit that is mounted on the traveling unit. A method for controlling an autonomous mobile device having a shape measuring means for acquiring a dimensional data group, the camera having a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf, The first wavelength band for each point of the image captured by the camera using the one having the function of capturing an image of the second wavelength band whose reflectance is locally low in the spectral reflectance of the plant leaf The ratio between the reflectance of the second wavelength band and the reflectance of the second wavelength band is determined, a point where the reflectance is higher than a predetermined value is determined to be a plant leaf, and among the surrounding shapes acquired by the shape measuring means , The area identified as the leaf of the plant For the remaining area, three or more points of three-dimensional data are sampled from the three-dimensional data group, a plane is calculated from the sampled three-dimensional data by Hough transform, and sample points close to the plane are extracted as movable areas. At the same time, it is characterized in that a planned travel path is determined within the movable region, and the travel unit is controlled to move on the planned travel path.

[Configuration 11]
The mobile object steering assist method according to the present invention uses an imaging unit that captures a stereo image including a reference image and a reference image by a plurality of cameras mounted on a steerable mobile object. A steering assist method for extracting a plane area in a stereo image, performing object detection processing on an area other than the extracted plane area, detecting a movable area, and assisting a driver of a moving object. As a camera for capturing a reference image, a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf and a second local reflectance in the spectral reflectance of the plant leaf are low. The ratio of the reflectance of the first wavelength band to the reflectance of the second wavelength band is determined for each point of the image captured by the imaging means using a device having a function of capturing an image of the wavelength band of Find this reflectance A point higher than a predetermined value is determined to be a plant leaf, and an area determined to be a plant leaf is excluded from the image captured by the imaging unit, and a planar region is extracted from the remaining region and An object detection process is performed.

[Configuration 12]
A method for assisting steering of a moving body according to the present invention includes a camera that is mounted on a movable body that can be steered and that images the surroundings of the moving body, and acquires the shape of the surroundings of the moving body that is mounted on the moving body as a three-dimensional data group. Using shape measurement means, sample 3D data of 3 or more points from 3D data group, calculate plane by Hough transform from sampled 3D data, and extract sample points close to that plane as movable area A steering assist method for assisting a driver of a moving body, and as a camera, a function of capturing an image of a first wavelength band having a high reflectance in a spectral reflectance of a plant leaf and a spectral distribution of the plant leaf With respect to each point of the image captured by the camera using the one having a function of capturing an image of the second wavelength band whose reflectance is locally low, the reflectance of the first wavelength band and the first 2 A ratio with the reflectance of the long band is obtained, a point where this reflectance is higher than a predetermined value is determined to be a plant leaf, and among the surrounding shapes acquired by the shape measuring means, the leaf is a plant The discriminated area is excluded, a plane is extracted from the remaining area, and a movable area is detected.

  In the present invention, in detecting a plane portion that can be traveled, a green leaf region such as grass is excluded, and a plane portion is detected for the remaining region. And this plane part is detected as a driving | running | working area | region.

  Therefore, according to the present invention, it is possible to prevent the upper surface such as grass from being detected as a travelable plane even in parks, river fields, and the like, and it is possible to control the moving body to correctly reach the destination. Become.

  That is, the present invention provides, for example, a surrounding shape detection device and a surrounding shape detection method in which it is prevented that the upper surface of the grass and the like is detected as a travelable plane in a course where grass and the like exist. Provided are an autonomous mobile device and a method for controlling the autonomous mobile device that can perform smooth movement by using such a surrounding shape detection device, and also provides a steering assist device and a mobile object for such a mobile object. It is possible to provide a steering assist method.

It is a side view which shows the external appearance structure of the autonomous mobile device which concerns on this invention. It is a block diagram which shows the structure of the autonomous mobile device which concerns on this invention. It is a flowchart which shows the procedure of the plane area | region in this invention. It is a flowchart which shows the other example of the procedure of the plane area | region in this invention. It is a flowchart which shows the pre-process of step S201 of FIG. It is a flowchart which shows the procedure for controlling the movement of a driving | running | working part in the autonomous mobile device which concerns on this invention. It is a top view which shows the structure of the camera in the autonomous mobile device which concerns on this invention. It is a top view which shows the other structure of the camera in the autonomous mobile apparatus which concerns on this invention. It is a side view which shows the external appearance structure of the conventional autonomous mobile apparatus. It is a top view which shows the movement route plan in the conventional autonomous mobile apparatus. It is the top view (a) which shows a mode that the conventional autonomous mobile device moves across vegetation, and the perspective view (b) which shows the mode of the vegetation seen from the autonomous mobile device. It is a perspective view which shows the state which comprises the gentle surface, while the upper surface of a grassy spot has a certain amount of variation.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

[Configuration of autonomous mobile device]
FIG. 1 is a side view showing an external configuration of an autonomous mobile device according to the present invention equipped with a surrounding shape detection device according to the present invention.

  The surrounding shape detection method according to the present invention is executed in the surrounding shape detection device according to the present invention, and the autonomous mobile device control method according to the present invention is executed in the autonomous mobile device according to the present invention.

  As shown in FIG. 1, the autonomous mobile device according to the present invention has a traveling unit 1 that can be moved. A plurality of cameras including stereo cameras 3aL and 3aR are used as a reference image and a reference image. The image pickup means for picking up images is mounted and configured.

  The traveling unit 1 may be equipped with a laser range finder (LRF) 2 that measures the shape around the traveling unit 1. The laser range finder 2 is a line scan (two-dimensional scan) type or surface scan (three-dimensional scan) type laser range finder, and can obtain distance information from the traveling unit 1 to the surrounding shape. .

  Of the plurality of cameras 3aL and 3aR, at least one camera 3aL that captures a reference image has a function of capturing an image in a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf. 3a and a second camera 3b having a function of capturing an image of a second wavelength band having a locally low reflectance in the spectral reflectance of the plant leaf. That is, one camera 3aL has a function of photographing the same image separately into two wavelength images.

  The first wavelength band having a high reflectance in the spectral reflectance of a plant leaf is, for example, 800 nm to 1300 nm. The first camera 3a includes a first filter that transmits light in a wavelength region having a width of 100 nm or more in the first wavelength band, and is configured to take an image of light that has passed through the first filter. ing. Note that if the wavelength width of the transmitted light in the first filter is excessively narrowed, the amount of light to be captured decreases, the noise increases relatively, and the S / N deteriorates.

  The second wavelength band in which the reflectance of the spectral reflectance of the plant is locally low is a wavelength band in which the absorption of the leaf component occurs, for example, the absorption band of chlorophyll is 640 nm ± 40 nm. . The second camera 3b includes a second filter that transmits light having a wavelength range of about 30 nm to 100 nm in the second wavelength band, and images the light transmitted through the second filter. It is configured. If the wavelength width of the transmitted light in the second filter is excessively narrowed, the amount of light to be imaged decreases, the noise increases relatively, and the S / N deteriorates. Therefore, a filter that transmits a width of about 80 nm with respect to the center wavelength. Is preferred.

  The second wavelength band may be a water absorption band. For example, 1240 nm ± 10 nm (the transmission width of the filter is 10 nm to 50 nm), 1450 nm ± 50 nm (the transmission width of the filter is 50 nm to 150 nm), 1940 nm ± 100 nm (the transmission width of the filter is 50 nm to 250 nm), etc. .

  In this way, the first and second cameras 3a and 3b can measure the amount of light from the imaging target with the two-dimensional light receiving element and measure a value proportional to the amount of light as the luminance value.

  FIG. 2 is a block diagram showing the configuration of the autonomous mobile device according to the present invention.

  As shown in FIG. 2, a control unit 21 is mounted in the traveling unit 1. The control unit 21 extracts a plane area in the stereo image based on the stereo image obtained by the imaging unit, and performs object detection processing on an area other than the extracted plane area. Further, the control means 21 detects the plane area as a movable area, determines a planned movement path within the movable area, and controls the traveling unit 1 to move on the planned movement path.

  A laser range finder 2, a gyro 22 and a GPS (receiver) 23 may be connected to the control means 21 in addition to the imaging means. Based on information from the first and second cameras 3a and 3b, the control means 21 detects the leaves (vegetation) of the plant as will be described later. Further, the control unit 21 extracts a planar area in the stereo image based on the stereo image obtained by the imaging unit, as will be described later. The control unit 21 converts information from the laser range finder 2 and the gyro 22 into camera coordinates at the time of shooting by the imaging unit. And the control means 21 acquires corresponding vegetation information on the stereo image obtained by an imaging means.

  Furthermore, the control means 21 determines the presence or absence of an obstacle, that is, whether the vehicle is allowed to pass, based on the detected position of the plant leaf (vegetation) and the extracted plane area, and displays a map indicating the accessible region. create. At this time, regarding the extraction of the planar region, the planar region is extracted for the remaining part excluding the plant leaf (vegetation) region.

  Based on the map showing the passable area created in this way, the control means 21 generates a planned movement plan and corrects the orbital deviation based on the information from the GPS (receiver) 23. I do. Based on this calculation result, the control means 21 calculates the steering instruction angle and the accelerator instruction amount.

  A brake drive motor 24, an accelerator drive motor 25, and a steering drive motor 26 are connected to the control means 21. The control means 21 controls the brake drive motor 24, the accelerator drive motor 25, and the steering drive motor 26 based on the calculated steering instruction angle and accelerator instruction amount. The brake drive motor 24 drives the brake 27. The accelerator drive motor 25 drives the accelerator 28. The steering drive motor 26 drives the steering 29. The traveling unit 1 moves on the planned travel path by driving the brake 27, the accelerator 28, and the steering 29.

[Detection of plant leaves (vegetation)]
The control means 21 obtains the ratio of the luminance in the first wavelength band and the luminance in the second wavelength band for each point of the image picked up by the image pickup means, and determines the point where the luminance ratio is higher than a predetermined value as a plant. It is determined that it is a leaf. That is, the control means obtains the ratio of the luminance of the first wavelength band and the luminance of the second wavelength band at the same corresponding point in space, and discriminates the leaves of the plant based on the ratio. Do.

  In order to make this determination, the reflectance ratio RefRate (reflectance in the first wavelength band / reflectance in the second wavelength band) is calculated for each point at the same position in space based on the ratio of the luminance values. Ask.

・・・・（式１）
反射率比RefRateは、計測された各波長帯の輝度値の比（Ｖ_１／Ｖ_２）に、係数（RefBase）を掛け、その時の露光時間の比（Ｅ_１／Ｅ_２）及びゲインの比（Ｋ_１／Ｋ_２）で除算することで求める。係数（RefBase）はカメラ３ａＬ，３ｂの受光素子の効率やゲイン係数、光源の分光光量比を補正するための値であり、既知反射率を持つ校正用の撮影対象を事前に、もしくは、計測と同時に撮影することにより求めることができる。

... (Formula 1)
The reflectance ratio RefRate is obtained by multiplying the ratio (V ₁ / V ₂ ) of the measured luminance values of each wavelength band by a coefficient (RefBase), and the ratio of the exposure time (E ₁ / E ₂ ) and the gain ratio at that time It is obtained by dividing by (K ₁ / K ₂ ). The coefficient (RefBase) is a value for correcting the efficiency and gain coefficient of the light receiving elements of the cameras 3aL and 3b and the spectral light amount ratio of the light source. It can be obtained by photographing at the same time.

  Then, the control means 21 determines that the reflectance ratio is a predetermined value between 2.0 and 3.0, for example, a location where the reflectance ratio is greater than 2.5 as a plant leaf (vegetation). As shown in [Table 1] below, the reflectance ratio between the reflectance of NIR (near infrared light (850 nm to 1000 nm)) and the reflectance of VISR (visible red light (600 nm to 680 nm)) under sunlight. Since it is 3.0 or more for plants and 2.0 or less for non-plants, the presence or absence of plants is determined by setting a predetermined value between 2.0 and 3.0 as a threshold value. Can do.

なお、制御手段２１は、撮像手段により得られたステレオ画像のうち、植物の葉であると判別された箇所については、この形状を避けるか否かを判断するための高さの閾値を変更し、この形状を避けるか否かを判断するようにしてもよい。

Note that the control means 21 changes the height threshold for determining whether or not to avoid this shape in the stereo image obtained by the imaging means for a portion determined to be a plant leaf. It may be determined whether to avoid this shape.

[Plane area extraction]
FIG. 3 is a flowchart showing the procedure of the planar area in the present invention.

  Extraction of a planar area is performed as follows.

  That is, as shown in FIG. 3, first, a standard image and a reference image including a road plane area on which the traveling unit 1 travels are transmitted for a predetermined time by using one camera (reference camera) 3aL and the other camera (reference camera) 3aR. Images are taken at intervals, and after predetermined image processing is performed by the control means 21, they are stored as image information for each time (step S101).

  Next, a projective transformation matrix for performing two-dimensional projective transformation from the road plane area in the reference image to the road plane area in the standard image is dynamically calculated (step S102).

  Then, a road plane region is extracted from the reference image using the projective transformation matrix calculated in step S102 (step S103).

  Next, the normal vector of the road plane and the distance from the optical center of one camera (reference camera) 3aL to the road plane are calculated as plane parameters (step S104).

  Next, using the plane parameter calculated in step S104, the road plane area extracted in step S103 is converted to a road surface to detect an object that may become an obstacle, or extracted in step S103. An object is detected by performing stereo measurement on an area other than the road plane area (step S105).

  Then, a relative velocity vector of each object with respect to the traveling unit 1 is calculated from the position change of each object for each time (step S106).

  Further, a virtual projection plane (VPP) image that is an image of the road surface viewed from above is generated using the normal vector that is the inclination of the road plane calculated in step S104 (step S107).

  Then, the speed vector of the traveling unit 1 is calculated from the position at each time of the VPP image generated in step S107 (step S108).

  Furthermore, the absolute velocity vector of each object is calculated using the relative velocity vector of the object calculated in step S106 and the velocity vector of the traveling unit 1 calculated in step S108 (step S109).

[Other examples of planar area extraction]
FIG. 4 is a flowchart showing another example of the procedure of the planar area in the present invention.

  Extraction of a planar region in the present invention may be performed using Hough Transform as follows.

  There are several methods for Hough transform called Forward Position, Backward Position, Feature Point Pairs, etc. Hereinafter, for the sake of simplicity, these Hough transforms will be described as examples of straight line detection in a two-dimensional image plane. It is assumed that an edge (feature point expected to be on a straight line) is detected from the original image by some image processing. Further, the number of feature points is M, the size of the original image is L × L, and the size of the parameter space (voting box) of the vote destination is l (el) × l (el).

  In the Forward Position, the feature points of the original image are sequentially traced and voted for each slot through which a curve passes when each point is mapped in the parameter space. The complexity of the algorithm is O (Ml).

  In Backward Position, each slot in the parameter space is sequentially operated, and the number of feature points on a straight line obtained by inversely mapping each slot to the original image is used as a vote for that slot. The complexity of the algorithm is O (L12).

  In Feature Point Pairs, every combination of two feature points is traced sequentially, and a vote is calculated for the slot to which the parameter calculated from the pair of the two points belongs. The complexity of the algorithm is O (M2).

  However, the method of extracting a plane from three-dimensional data by these Hough transforms requires an extremely large amount of computation and memory used. This is because the number of dimensions is one more than two dimensions and all data points have meaning as feature points. Accordingly, these Hough transforms are not practical in terms of the amount of calculation and the memory used in the task of plane detection from three-dimensional data.

  There is also a Hough transform called Randomized Hough Transform (RHT). This randomized Hough transform is a derivation of the above-mentioned Feature Point Pairs, and in the above-described two-dimensional image plane, processing is performed only for K combinations by random sampling, not all combinations of two feature points. It is. The complexity of this algorithm is O (K).

  In the present invention, the extraction of a plane area is performed by sampling data randomly from three-dimensional data to calculate a plane parameter, and by voting the plane parameter directly to the voting space (Randomized Hough Transform). It is determined whether it is a plane. That is, from the three pieces of three-dimensional data, the plane vector (θ, φ, d) where the direction of the normal vector (θ, φ) is d and the distance from the origin is d, that is, a plane equation is obtained. is there.

  In this procedure, first, the stereo image obtained by the imaging means, or the parallax or distance information obtained by the laser range finder 2 is converted into three-dimensional data by conversion based on appropriate calibration, and this three-dimensional data is converted. Prepare an array of data.

  First, from N pieces of data (three-dimensional data) {p1, p2, p3,..., PN}, M sets of subsets {p11, p21, p31,..., PN1}, {p12, p22, p32,. ,..., {P1M, p2M, p3M,..., PNM} are extracted (step S201). The “total number of votes” at this time is M.

  Next, parameters (θi, φi, di) are calculated from each subset {p1i, p2i, p3i,..., PNi} (step S202). Next, the calculated parameters are quantized to determine a voting destination grid, and voting is performed for each parameter individually (step S203).

  Finally, a parameter having a large number of votes is taken as a solution. That is, a plane having this parameter is determined as a plane (step S204).

  Here, the process performed before extracting data in step S201 will be described.

  FIG. 5 is a flowchart showing a pre-process of step S201 in FIG.

  Here, a method of processing a three-dimensional coordinate data group obtained in a form such as a distance image or a parallax image will be described. As shown in FIG. 5, data from the imaging means (cameras 3aL, 3aR) or the laser range finder 2 is input, and calibration or parameter conversion is performed to generate a three-dimensional data array (steps). S301). Here, together with the three-dimensional data group converted into three-dimensional coordinates, a reliability parameter indicating the reliability r of this three-dimensional data is input from the cameras 3aL, 3aR or the laser range finder 2 (step S302). In this embodiment, a distance image based on a stereo image or the like is used as an input image, and a reliability parameter obtained in the process of stereo distance calculation can be used effectively. Further, when such a reliability parameter cannot be used, it can be directly applied to three-dimensional data not accompanied by the reliability parameter by regarding it as a constant.

  The reliability parameter can be obtained by using various evaluation values, but in this embodiment, there are two methods of obtaining the reliability parameter based on the variance value of the template and how to obtain the reliability parameter based on the matching score. Will be described.

  As described above, the three-dimensional data in this embodiment has a distance image as a stereo image as an input. In stereo distance measurement, the correspondence between pixels in a stereo image is searched by template matching. For example, the left image is set as a reference image, and the right image is set as a reference image, and a luminance dispersion value of pixels in the matching template is obtained. Then, the variance values of all the templates to be searched can be calculated, and the size of the variance value can be used as the reliability parameter. That is, the higher the variance value, the higher the reliability, and the lower the variance value, the lower the reliability.

  In addition, as a method of obtaining a reliability image based on the matching score, the template in the base image and the area selected along the epipolar line of the reference image are compared, and the matching score is determined from the difference in luminance of pixels in these areas. A value can be calculated. Then, the reliability parameter based on the matching score can be obtained from the matching score value, the peak width near the minimum matching score value, the steepness value of the graph, and the like. That is, the lower the matching score value, the smaller the peak width near the minimum score value, and the larger the steepness value of the graph, the higher the reliability.

  In this way, a reliability parameter is calculated for each three-dimensional parameter, and input data is selected by discarding data whose reliability is lower than a predetermined threshold value by this reliability parameter (step S303). As in this embodiment, by selecting three-dimensional data based on the reliability parameter, it is possible to improve reliability and stability and prevent performance deterioration.

[Operation of autonomous mobile device]
When the autonomous mobile device starts operation, it performs imaging by the stereo cameras 3aL and 3aR. At this time, measurement (scanning) by the laser range finder 2 may be performed in parallel. Then, in the images obtained by the first and second cameras 3a and 3b, the plant area and the non-plant area are discriminated. Moreover, the discrimination result of a plant area | region and a non-plant area | region is matched with the stereo image obtained by stereo camera 3aL and 3aR. At this time, a region that is a plant region and can travel (a region where the height of the plant is a predetermined height or less) may be detected.

  Further, in extracting a plane area that can be traveled from the stereo images obtained by the stereo cameras 3aL and 3aR, the area of the plant leaf (vegetation) is excluded from the target image in advance, and the remaining part of the plane area is extracted. Perform extraction. At this time, a planar area that is a non-plant area and can travel can be detected.

  FIG. 6 is a flowchart showing a procedure for controlling the movement of the traveling unit 1.

  In this autonomous mobile device, as shown in FIG. 6, when the control means 21 starts the processing in step st11, the control unit 21 proceeds to step st13 during the operation of the moving body 1 (step st12), and is photographed by the stereo cameras 3aL and 3aR. I do.

  Next, in step st14, the attitude of the traveling unit 1 at the time of photographing is measured and stored by the gyro 22. Next, in step st15, the presence or absence of a plant leaf (vegetation) is determined and stored as a vegetation determination image. In step st16, the process waits until the image capturing time of the next frame, and proceeds to step st17 where line scanning is performed by the laser range finder 2. In step st18, the gyro 22 measures and stores the attitude of the traveling unit 1 during the line scan.

  In step st19, each point in one line scan is set as a distance measuring point, and the process proceeds to step st20, where the posture of the traveling unit 1 during imaging, the posture of the traveling unit 1 during line scanning, and one camera 3aL and the laser range. Based on the relative positional relationship with the finder 2, the corresponding position on the image is calculated as described above.

  Then, in step st21, the control means 21 refers to the stored vegetation determination image, excludes the plant leaf (vegetation) region, and in step st22, extracts the planar portion of the remaining region. That is, in the process in step st22, the leaf (vegetation) area of the plant is excluded from the calculation process of the planar area as a calculation object. Here, when performing the planar region extraction processing shown in FIG. 3, the region on the image corresponding to the plant leaf (vegetation) region is set as a non-target mask, and image difference processing is performed in this mask region. It can be realized by not. In addition, when a plane is detected by the procedure shown in FIG. 4, it can be realized by not using the measurement results of the surroundings corresponding to the area determined to be a plant leaf (vegetation) area for voting.

  In step st23, the processing after step st19 is looped until the next line scan, and in step st24, the processing after step st16 is looped until the next image capturing time, and the process proceeds to step st25.

  In step st25, a movement plan is generated based on the stored obstacle map. In step st26, the current position is measured by the gyro 22 and the GPS (receiver) 23, and the process proceeds to step st27.

  In step st27, the steering and the brake are controlled to correct the track of the traveling unit 1, and in step st28, the operation loop is returned to step st2.

[Other Embodiments]
FIG. 7 is a plan view showing the configuration of the camera in the autonomous mobile device according to the present invention.

  In the autonomous mobile device according to the present invention, one camera 3aL (3a, 3b) that performs imaging in two wavelength bands is configured to transmit light incident on the objective lens 4 from an imaging target by a dichroic mirror 5 as shown in FIG. It can be configured to divide into two wavelength bands and to pick up images with the image pickup devices 6 and 7 respectively. That is, light incident on the objective lens 4 is divided into a first wavelength band and a second wavelength band by an optical element having wavelength selectivity such as a dichroic mirror 5. The light in the first wavelength band is imaged by the first image sensor 6 through the first filter 8 and the condenser lens 9 that transmit the first wavelength band light. The light in the second wavelength band is imaged by the second image sensor 7 through the second filter 10 and the condenser lens 11 that transmit the second wavelength band light.

  FIG. 8 is a plan view showing another configuration of the camera in the autonomous mobile device according to the present invention.

  Further, as shown in FIG. 8, the plurality of cameras are provided with stereo cameras 3aL and 3aR that perform imaging in the first wavelength band, and a second camera 3b that performs imaging in the second wavelength band separately from these. May be configured. The camera 3aL that captures an image in the first wavelength band is a first camera 3a that captures an image with the first image sensor 6 via the first filter 8 and the objective lens 4 that transmit the first wavelength band light. Configured to work. The second camera 3b that captures an image in the second wavelength band is configured to capture an image with the second imaging element 7 via the second filter 10 and the objective lens 4 that transmit light in the second wavelength band. Has been.

  These cameras 3aL, 3aR, and 3b are positioned and installed on the same holder 12.

  The present invention relates to a surrounding shape detecting device and a surrounding shape detecting method for detecting a surrounding shape, an autonomous moving device for detecting a surrounding shape and performing autonomous movement, a control method for the autonomous moving device, and a moving body performed by a human. The present invention is applied to a steering assist device for a mobile object that assists in steering and a steering assist method for a mobile object.

DESCRIPTION OF SYMBOLS 1 Traveling part 2 Laser range finder 3aL, 3aR Stereo camera 3a 1st camera 3b 2nd camera 21 Control means 103 Obstacle

Claims (12)

Imaging means for imaging a stereo image including a standard image and a reference image by a plurality of cameras;
Control means for extracting a plane area in the stereo image based on the stereo image obtained by the imaging means, and performing object detection processing on an area other than the extracted plane area to detect a movable area. And
The camera that captures the reference image has a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf, and a first local reflectance in the spectral reflectance of the plant leaf. Having a function of taking an image of two wavelength bands,
The control means obtains the ratio of the reflectance in the first wavelength band and the reflectance in the second wavelength band for each point of the image captured by the imaging means, and the reflectance is higher than a predetermined value. The point is determined to be a plant leaf, the region determined to be a plant leaf is excluded from the image captured by the imaging means, and the planar region extraction and object detection processing is performed on the remaining region. A surrounding shape detection apparatus characterized by A camera that captures the surroundings;
A shape measuring means for acquiring a surrounding shape as a three-dimensional data group;
Control means for sampling three or more points of three-dimensional data from the three-dimensional data group, calculating a plane from the sampled three-dimensional data by Hough transform, and extracting a sample point close to the plane as a movable region; ,
The camera has a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf, and a second wavelength band having a locally low reflectance in the spectral reflectance of the plant leaf. A function of capturing an image,
The control means obtains a ratio between the reflectance of the first wavelength band and the reflectance of the second wavelength band for each point of the image captured by the camera, and the reflectance is higher than a predetermined value. Can be moved by extracting the plane for the remaining area, excluding the area determined to be a plant leaf from the surrounding shapes acquired by the shape measuring means A surrounding shape detection device characterized by detecting an area. A traveling section made movable,
Imaging means for imaging a stereo image including a standard image and a reference image by a plurality of cameras mounted on the traveling unit;
Based on the stereo image obtained by the imaging means mounted on the traveling unit, a plane area in the stereo image is extracted, and an object detection process is performed on an area other than the extracted plane area to move it. Control means for detecting a possible area, determining a planned travel path in the movable area, and controlling the travel unit to move on the planned travel path,
The camera that captures the reference image has a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf, and a first local reflectance in the spectral reflectance of the plant leaf. Having a function of taking an image of two wavelength bands,
The control means obtains the ratio of the reflectance in the first wavelength band and the reflectance in the second wavelength band for each point of the image captured by the imaging means, and the reflectance is higher than a predetermined value. The point is determined to be a plant leaf, the region determined to be a plant leaf is excluded from the image captured by the imaging means, and the planar region extraction and object detection processing is performed on the remaining region. An autonomous mobile device characterized by A traveling section made movable,
A camera mounted on the traveling unit and imaging the periphery of the traveling unit;
Shape measuring means mounted on the traveling unit and acquiring the shape around the traveling unit as a three-dimensional data group;
The three-dimensional data of three or more points is sampled from the three-dimensional data group, and a plane is calculated from the sampled three-dimensional data by Hough transform, and a sample point close to the plane is set as a movable region. And a control means for extracting,
The camera has a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf, and a second wavelength band having a locally low reflectance in the spectral reflectance of the plant leaf. A function of capturing an image,
The control means obtains a ratio between the reflectance of the first wavelength band and the reflectance of the second wavelength band for each point of the image captured by the camera, and the reflectance is higher than a predetermined value. Can be moved by extracting the plane for the remaining area, excluding the area determined to be a plant leaf from the surrounding shapes acquired by the shape measuring means An autonomous mobile device characterized by detecting an area. Imaging means for imaging a stereo image including a base image and a reference image by a plurality of cameras mounted on a steerable moving body;
Based on the stereo image obtained by the imaging means mounted on the moving body, a plane area in the stereo image is extracted, and object detection processing is performed on an area other than the extracted plane area to move the area. A control means for detecting a possible area; and a display means controlled by the control means,
The camera that captures the reference image has a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf, and a first local reflectance in the spectral reflectance of the plant leaf. Having a function of taking an image of two wavelength bands,
The control means obtains the ratio of the reflectance in the first wavelength band and the reflectance in the second wavelength band for each point of the image captured by the imaging means, and the reflectance is higher than a predetermined value. The point is determined to be a plant leaf, the region determined to be a plant leaf is excluded from the image captured by the imaging means, and the planar region extraction and object detection processing is performed on the remaining region. A steering assist device for a moving object. A camera mounted on a steerable mobile body and imaging the surroundings of the mobile body,
A shape measuring means mounted on the moving body and acquiring a shape around the moving body as a three-dimensional data group;
Mounted on the moving body, samples three or more three-dimensional data from the three-dimensional data group, calculates a plane by Hough transform from the sampled three-dimensional data, and sets a sample point close to the plane as a movable region Control means for extracting; and display means controlled by the control means,
The camera has a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf, and a second wavelength band having a locally low reflectance in the spectral reflectance of the plant leaf. A function of capturing an image,
The control means obtains the ratio of the reflectance in the first wavelength band and the reflectance in the second wavelength band for each point of the image captured by the imaging means, and the reflectance is higher than a predetermined value. It is determined that the point is a plant leaf, out of the surrounding shape obtained by the shape measuring means, the region determined to be a plant leaf is excluded, and a plane is extracted for the remaining region, A steering assist device for a moving body, characterized by detecting a movable region. A stereo image including a reference image and a reference image is captured by a plurality of cameras,
A surrounding shape detection method for extracting a plane area in the stereo image based on the stereo image and performing object detection processing on an area other than the extracted plane area,
As a camera for capturing the reference image, a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf and a locally low reflectance in the spectral reflectance of the plant leaf 2 having a function of capturing an image in the second wavelength band, and for each point of the reference image, a ratio between the reflectance in the first wavelength band and the reflectance in the second wavelength band is obtained. A point where the reflectance is higher than a predetermined value is determined to be a plant leaf, the region determined to be a plant leaf is excluded from the reference image, and the extraction of the planar region for the remaining region and A surrounding shape detection method characterized by performing object detection processing. Acquire the surrounding shape as a 3D data group,
A surrounding shape detection method in which three or more points of three-dimensional data are sampled from the three-dimensional data group, a plane is calculated from the sampled three-dimensional data by Hough transform, and sample points close to the plane are extracted as a movable region. There,
A function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf, and an image of a second wavelength band having a locally low reflectance in the spectral reflectance of a plant leaf The ratio of the reflectance of the first wavelength band to the reflectance of the second wavelength band is obtained for each point of the image captured by the camera using a camera having a function, and this reflectance is a predetermined value. A higher point is determined to be a plant leaf, out of the surrounding shapes acquired by the shape measuring means, the region determined to be a plant leaf is excluded, and a plane is extracted for the remaining region A surrounding shape detection method characterized by detecting a movable region. A method for controlling an autonomous mobile device having a traveling unit that is movable and an imaging unit that captures a stereo image including a reference image and a reference image by a plurality of cameras mounted on the traveling unit,
As a camera for capturing the reference image, a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf and a locally low reflectance in the spectral reflectance of the plant leaf Using the one having a function of capturing an image in the wavelength band of 2,
For each point of the image picked up by the image pickup means, the ratio of the reflectance of the first wavelength band to the reflectance of the second wavelength band is obtained, and the point where this reflectance is higher than a predetermined value is determined as the leaf of the plant. Is extracted, and from the image captured by the imaging means, the region determined to be a plant leaf is excluded, and for the remaining region, the planar region in the stereo image is extracted and extracted. An object detection process is performed on an area other than a plane area, a movable area is detected, a planned movement path is determined in the movable area, and the traveling unit is controlled to move on the planned movement path. A method for controlling an autonomous mobile device. A traveling unit that is movable, a camera that is mounted on the traveling unit and images the periphery of the traveling unit, and a shape measuring unit that is mounted on the traveling unit and acquires the shape of the periphery of the traveling unit as a three-dimensional data group A method for controlling an autonomous mobile device comprising:
The camera has a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf, and a second wavelength band having a locally low reflectance in the spectral reflectance of the plant leaf. With what has a function to capture an image,
For each point of the image captured by the camera, the ratio of the reflectance of the first wavelength band to the reflectance of the second wavelength band is obtained, and the point at which this reflectance is higher than a predetermined value is determined by the plant leaf. It is determined that there is a region, and the region determined to be a plant leaf is excluded from the surrounding shapes acquired by the shape measuring means, and the remaining region is three-dimensional or more three-dimensional from the three-dimensional data group. The data is sampled, a plane is calculated from the sampled three-dimensional data by Hough transform, sample points close to the plane are extracted as a movable area, a planned movement path is determined in the movable area, and the traveling A control method for an autonomous mobile device, characterized in that a part is controlled to move on the planned travel path. Extracting a planar area in the stereo image based on the stereo image using an imaging unit that captures a stereo image including a standard image and a reference image by a plurality of cameras mounted on a steerable mobile body, An operation assisting method for performing object detection processing on an area other than the extracted planar area, detecting a movable area, and assisting a driver of the moving body,
As a camera for capturing the reference image, a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf and a locally low reflectance in the spectral reflectance of the plant leaf Using the one having a function of capturing an image in the wavelength band of 2,
For each point of the image picked up by the image pickup means, the ratio of the reflectance of the first wavelength band to the reflectance of the second wavelength band is obtained, and the point where this reflectance is higher than a predetermined value is determined as the leaf of the plant. A region that is determined to be a plant leaf is excluded from the image captured by the imaging unit, and the planar region extraction and object detection processing are performed on the remaining region. A method for assisting the maneuvering of a moving object. Using the camera mounted on the steerable moving body and imaging the surroundings of the moving body, and the shape measuring means mounted on the moving body and acquiring the surrounding shape of the moving body as a three-dimensional data group, the 3 Three or more points of three-dimensional data are sampled from a group of three-dimensional data, a plane is calculated from the sampled three-dimensional data by Hough transform, sample points close to the plane are extracted as movable regions, and the moving object is controlled A steering assistance method for assisting a person,
The camera has a function of capturing an image of a first wavelength band having a high reflectance in the spectral reflectance of a plant leaf, and a second wavelength band having a locally low reflectance in the spectral reflectance of the plant leaf. With what has a function to capture an image,
For each point of the image captured by the camera, the ratio of the reflectance of the first wavelength band to the reflectance of the second wavelength band is obtained, and the point at which this reflectance is higher than a predetermined value is determined by the plant leaf. It is determined that there is a region, and the region determined to be a plant leaf is excluded from the surrounding shapes acquired by the shape measuring unit, and a plane is extracted from the remaining region to detect a movable region. A method for assisting in maneuvering a moving body.

Images