WO2023095489A1 - External environment recognition device - Google Patents
External environment recognition device Download PDFInfo
- Publication number
- WO2023095489A1 WO2023095489A1 PCT/JP2022/038648 JP2022038648W WO2023095489A1 WO 2023095489 A1 WO2023095489 A1 WO 2023095489A1 JP 2022038648 W JP2022038648 W JP 2022038648W WO 2023095489 A1 WO2023095489 A1 WO 2023095489A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- landmark
- unit
- recognition device
- external world
- height
- Prior art date
Links
- 238000001514 detection method Methods 0.000 claims description 72
- 150000001875 compounds Chemical class 0.000 claims description 43
- 238000004364 calculation method Methods 0.000 claims description 17
- 238000005259 measurement Methods 0.000 claims description 14
- 238000000034 method Methods 0.000 description 66
- 238000012986 modification Methods 0.000 description 36
- 230000004048 modification Effects 0.000 description 36
- 238000012545 processing Methods 0.000 description 32
- 238000010586 diagram Methods 0.000 description 13
- 230000006870 function Effects 0.000 description 11
- 238000013527 convolutional neural network Methods 0.000 description 8
- 238000013459 approach Methods 0.000 description 6
- 238000003384 imaging method Methods 0.000 description 4
- 238000010801 machine learning Methods 0.000 description 3
- 230000006399 behavior Effects 0.000 description 2
- 230000015654 memory Effects 0.000 description 2
- 239000007787 solid Substances 0.000 description 2
- 241001465754 Metazoa Species 0.000 description 1
- 206010039203 Road traffic accident Diseases 0.000 description 1
- 238000004422 calculation algorithm Methods 0.000 description 1
- 230000000694 effects Effects 0.000 description 1
- 238000011156 evaluation Methods 0.000 description 1
- 230000002265 prevention Effects 0.000 description 1
- 238000007637 random forest analysis Methods 0.000 description 1
- 238000012706 support-vector machine Methods 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
- G01C3/00—Measuring distances in line of sight; Optical rangefinders
- G01C3/02—Details
- G01C3/06—Use of electric means to obtain final indication
Definitions
- the present invention relates to an external world recognition device.
- Patent Literature 1 describes a pedestrian distance calculation method using a stereo camera that has a field-of-view overlapping area and a field-of-view non-overlapping area.
- Patent Literature 1 describes a stereo camera device that calculates the distance of a moving object in a monocular region using road surface height calculated from parallax information.
- the stereo camera device obtains coordinates obtained by subtracting (lowering) the height of the stairs relative to the road surface from the vertical coordinates on the image of the feet of the pedestrian on the stairs. Then, in the stereo camera device, the obtained coordinates are extended to the stereo area, and the distance to the pedestrian is obtained from the parallax information of the road surface at that position.
- the parallax of the road surface is calculated based on the texture information of the road surface in the acquired image, and this parallax information is used to calculate the distance to the pedestrian.
- this parallax information is used to calculate the distance to the pedestrian.
- the present invention has been made in view of the above problems, and its purpose is to provide an external world recognition device that can accurately calculate the distance to an object regardless of the usage environment.
- an external world recognition device mounted on a vehicle, comprising: an image acquisition unit for acquiring an image of a landmark and an object; a landmark information acquisition unit that acquires information; and a distance estimation unit that estimates the distance to the object based on the three-dimensional information of the landmark and the sizes of the landmark and the object in the image. and.
- the distance to the object can be accurately calculated regardless of the usage environment. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.
- FIG. 1 is a functional block diagram showing a schematic configuration of an external world recognition device according to a first embodiment of the present invention
- FIG. FIG. 2 is a plan view showing an example of a state in which the external world recognition device of FIG. 1 is mounted on a vehicle
- 2 is a flowchart showing landmark information acquisition processing of the external world recognition device of FIG. 1; An example of the image which the external world recognition apparatus of FIG. 1 acquired by landmark information acquisition processing.
- 2 is a flowchart showing distance estimation processing of the external world recognition device of FIG. 1; An example of the image which the external world recognition apparatus of FIG. 1 acquired by the distance estimation process.
- the functional block diagram which shows schematic structure of the external world recognition apparatus of the modification of 1st Embodiment.
- FIG. 8 is a flowchart showing distance estimation processing of the external world recognition device of FIG. 7;
- FIG. 8 is a view for explaining proximity determination processing of the external world recognition device of FIG. 7;
- FIG. 8 is a diagram showing another example of proximity determination processing of the external world recognition device of FIG. 7 ;
- FIG. 5 is a functional block diagram showing a schematic configuration of an external world recognition device according to another modification of the first embodiment;
- the functional block diagram which shows schematic structure of the external world recognition apparatus which concerns on 2nd Embodiment of this invention.
- the functional block diagram which shows schematic structure of the external world recognition apparatus which concerns on 3rd Embodiment of this invention.
- FIG. 1 is a functional block diagram showing a schematic configuration of an external world recognition device 1 according to the first embodiment of the invention.
- 2 is a plan view showing an example of a state in which the external world recognition device 1 of FIG. 1 is mounted on a vehicle V.
- FIG. 3 is a flowchart showing landmark information acquisition processing of the external world recognition device 1 of FIG.
- FIG. 4 is an example of an image acquired by the external world recognition device 1 of FIG. 1 in the landmark information acquisition process.
- FIG. 5 is a flowchart showing distance estimation processing of the external world recognition device 1 of FIG.
- FIG. 6 is an example of an image acquired by the external world recognition device 1 of FIG. 1 through distance estimation processing.
- the external world recognition device 1 is mounted on a vehicle V (hereinafter also referred to as own vehicle V). As shown in FIG. 1 , the external world recognition device 1 has an image acquisition section 10 , a landmark information acquisition section 30 and a distance estimation section 50 . Although illustration is omitted, the external world recognition device 1 has a configuration in which a CPU, a RAM, a ROM, and the like are connected via a bus, and the CPU executes various control programs stored in the ROM to control the entire system. control behavior.
- the image acquisition unit 10 acquires images from a pair of cameras 10a and 10b (FIG. 2) functioning as stereo cameras, and the distance estimation unit 50 uses the images to determine the pedestrian 200 as the target object. and the own vehicle V will be described.
- a “landmark” used in this specification means each three-dimensional object (for example, reference numerals 100, 101 and 102 shown in FIG. 4) photographed by the pair of cameras 10a and 10b.
- "three-dimensional information of landmarks” means including at least one of the height and width of the landmarks 100, 101, and 102 themselves.
- the "three-dimensional information of the landmarks” includes the distance between the landmarks 100, 101, and 102 and the vehicle V ( depth).
- the image acquisition unit 10 acquires images of landmarks 100, 101 and 102 and a pedestrian (object) 200 from a pair of cameras 10a and 10b. Specifically, the image acquisition unit 10 captures an image captured by a pair of cameras 10a and 10b, and divides a compound eye region with overlapping fields of view and a monocular region (a left monocular region and a right monocular region) with non-overlapping fields of view. Get the containing image.
- the compound eye area of the image will be referred to as compound eye area image CA
- the left monocular area and right monocular area of the image will be referred to as left monocular area image LA and right monocular area image RA, respectively.
- the pair of cameras 10a and 10b are configured as one stereo camera mounted on a vehicle V.
- the pair of cameras 10a and 10b are composed of CCD or CMOS image sensors, and are directed to the front of the vehicle V.
- the pair of cameras 10a and 10b are installed so that each of the cameras 10a and 10b captures an image in front of the vehicle V at a predetermined depression angle (that is, an imaging area), and the angles of depression overlap with each other. For example, if the camera 10a is the left camera and the camera 10b is the right camera, the left side of the depression angle of the camera 10a and the right side of the camera 10b overlap each other.
- the center front area of the vehicle V becomes a stereo area (also called a compound eye area) defined by the imaging area of the camera 10a and the imaging area of the camera 10b.
- the left front area of the vehicle V is a left monocular area defined by the left portion of the photographing area of the camera 10b.
- the right front area of the vehicle V is a right monocular area defined by the right side of the imaging area of the camera 10a. Images captured by the pair of cameras 10 a and 10 b are input to the image acquisition section 10 .
- the image acquisition unit 10 acquires two images captured by a pair of cameras 10a and 10b, and acquires parallax information by applying a known parallax calculation algorithm to the acquired left and right images.
- the landmark information acquisition unit 30 includes a compound eye region landmark detection unit 32, a size information measurement unit 33, and a landmark information storage unit 36.
- a landmark information acquisition unit 30 acquires three-dimensional information of the landmarks 100 , 101 and 102 .
- the landmark information acquisition unit 30 acquires three-dimensional information of the landmarks 100, 101, and 102 positioned in the compound eye area image CA.
- the compound eye area landmark detection unit 32 detects landmarks 100, 101, and 102 from the compound eye area image CA acquired by the image acquisition unit 10.
- the compound eye area landmark detection unit 32 may analyze the texture of the compound eye area image CA transmitted from the image acquisition unit 10 and detect the landmarks 100, 101 and 102 from this.
- the compound eye area landmark detection unit 32 may detect the landmarks 100 , 101 , and 102 based on the parallax information acquired by the image acquisition unit 10 .
- the landmarks 100, 101, and 102 may be detected by clustering disparity information or the like, or the landmarks 100, 101, and 102 may be detected using a convolutional neural network as statistical machine learning. You may
- the size information measuring unit 33 measures the actual sizes of the landmarks 100, 101, and 102 included in the three-dimensional information from the compound eye area image CA. Specifically, the size information measurement unit 33 calculates the actual heights of the landmarks 100, 101, and 102 detected by the compound eye area landmark detection unit 32 from the compound eye area image CA. to calculate the Note that the size information measuring unit 33 measures not only the actual vertical size (actual height) of the landmarks 100, 101, and 102, but also the actual horizontal size (actual height) of the landmarks 100, 101, and 102. actual width) may be measured. Also, the size information measuring unit 33 may measure the actual size (actual height or actual width) of a part of the landmarks 100 , 101 , 102 .
- the landmark information storage unit 36 registers at least one of the size information of the landmarks 100, 101, and 102 measured by the size information measuring unit 33 as landmark information used for distance calculation, which will be described later. Specifically, the landmark information storage unit 36 stores size information (actual height, actual width, etc.) of the landmarks 100, 101, and 102 that have been measured, as well as size information of the photographed landmarks 100, 101, and 102. Stores texture information.
- the distance estimation unit 50 estimates the distance to the pedestrian 200 located in the monocular area image (left monocular area image LA, right monocular area image RA). Specifically, the distance estimating unit 50 uses the landmarks 100, 101, and 102 detected by the landmark information acquiring unit 30 to use the monocular area images (left monocular area image LA, right The distance to the pedestrian 200 included in the monocular area image RA) is estimated. More specifically, the distance estimation unit 50 estimates the distance from the vehicle V to the pedestrian 200 based on the three-dimensional information of the landmark 100 and the sizes of the landmark 100 and the pedestrian 200 in the image. do. In this embodiment, a case where the landmark 100 is used as a landmark for estimating the distance between the pedestrian 200 and the own vehicle V will be described. As shown in FIG. 1 , the distance estimation unit 50 includes a monocular area landmark detection unit 52 , a monocular area object detection unit 54 , a size estimation unit 58 and a distance calculation unit 60 .
- the monocular area landmark detection unit 52 detects landmarks 100 from the monocular area images (left monocular area image LA, right monocular area image RA) acquired by the image acquisition unit 10 . Specifically, the monocular area landmark detection unit 52 re-detects the landmark 100 based on the texture information of the landmark 100 stored in the landmark information storage unit 36 . For example, the monocular area landmark detection unit 52 may detect by template matching using the texture information of the landmark 100 stored in the landmark information storage unit 36 as a template. Also, the monocular area landmark detection unit 52 may detect the landmark 100 by tracking processing using a convolutional neural network.
- the monocular area landmark detection unit 52 detects the vertical length of the landmark 100 in the image from the monocular area images (the left monocular area image LA and the right monocular area image RA) of the landmark 100 acquired from the image acquisition unit 10 .
- height i.e., image height
- the monocular area object detection unit 54 detects an object for distance estimation from the monocular area images (left monocular area image LA, right monocular area image RA) acquired by the image acquisition unit 10 .
- the monocular region target object detection unit 54 detects, for example, a pedestrian 200 by known statistical machine learning when the target is the pedestrian 200 .
- the monocular region object detection unit 54 may detect the pedestrian 200 using classical machine learning such as random forest, support vector machine, and real Adaboost, or may detect the pedestrian 200 using a convolutional neural network. A pedestrian 200 may be detected by using this.
- the monocular area object detection unit 54 calculates the vertical length (that is, the image height) of the pedestrian 200 in the image based on the monocular area image of the pedestrian 200 acquired from the image acquisition unit 10 .
- the size estimation unit 58 estimates the landmark 100 based on the image size of the landmark 100 in the monocular area image (left monocular area image LA, etc.) and the image size of the pedestrian 200 in the monocular area image (left monocular area image LA, etc.).
- the size of pedestrian 200 is estimated from the size of .
- the size estimation unit 58 estimates the size (height, etc.) of the pedestrian 200 using the size (height, etc.) of the landmark 100 stored in the landmark information storage unit 36 . Specifically, based on the image height of the landmark 100 and the image height of the pedestrian 200, the size estimation unit 58 calculates the height of the pedestrian 200 from the actual height of the landmark 100 (hereinafter also referred to as height). ).
- the size estimation unit 58 multiplies the actual height of the landmark 100 by the ratio of the height of the image of the pedestrian 200 to the height of the image of the landmark 100 to obtain the height of the pedestrian 200. to estimate The size information of pedestrian 200 may be estimated only once.
- the distance calculation unit 60 A distance to the pedestrian 200 is calculated. Specifically, the distance calculation unit 60 calculates the distance from the own vehicle V to the pedestrian 200 based on the height of the pedestrian 200 estimated by the size estimation unit 58 . More specifically, the distance calculation unit 60 calculates the size of the vehicle based on the height of the pedestrian 200 estimated by the size estimation unit 58 and the image height of the pedestrian 200 detected by the monocular area object detection unit 54. The distance between V and pedestrian 200 is calculated.
- FIGS. 3 to 6 an operation example of the external world recognition device 1 of this embodiment will be described in landmark information acquisition processing (FIGS. 3 and 4) and distance estimation processing (FIGS. 5 and 6). I will explain separately.
- the external world recognition device 1 using a pair of cameras 10a and 10b functioning as stereo cameras installed to monitor the front of the vehicle V will be described.
- a case of calculating the distance between the own vehicle V and the pedestrian 200 based on the height of the pedestrian 200 photographed in the left monocular area image LA shown in FIG. 6 will be described.
- the external world recognition device 1 performs image acquisition processing (P101), parallax calculation processing (P102), solid object detection processing (P103), and three-dimensional information acquisition processing (P104) in the landmark information acquisition processing.
- the landmark registration process (P105) is executed in order.
- the image acquisition unit 10 acquires images captured by a pair of cameras 10a and 10b as shown in FIG. Acquire an area image RA).
- the image acquisition unit 10 calculates the parallax in the compound eye area image CA. For example, a window area of 5 ⁇ 5 pixels is set for the parallax calculation. Then, the parallax is calculated by horizontally scanning the image of the right camera 10b using the SAD as an evaluation value, with the image of the left camera 10a of the pair of cameras 10a and 10b as a reference.
- the compound eye region landmark detection unit 32 uses the parallax calculated in the parallax calculation process (P102) to detect landmarks 100, 101, and 102 as shown in FIG. To detect. Specifically, since the areas of the landmarks 100, 101, and 102 are at the same distance, clustering processing is performed on the parallax to roughly extract the areas where the landmarks 100, 101, and 102 exist. be done. Next, since the distance value changes between each of the landmarks 100, 101, and 102 and the boundary of the background, by detecting the parallax change point in the roughly extracted region, the landmarks 100, 101, and 102 The existing areas are detected in detail. FIG.
- FIG. 4 shows the detection result of the three-dimensional object detection process (P103) for the landmark 100 among the plurality of landmarks 100, 101, and 102.
- the size information measuring unit 33 measures the height direction size ( That is, the actual height) is calculated.
- a method of estimating the size (actual height) with respect to the landmark 100 among the plurality of landmarks 100, 101, and 102 will be described below.
- the rectangle B100 which is the detection result of the three-dimensional object detection process (P103), is used.
- T100 in FIG. 4 is the pixel position of the upper edge of the rectangle B100
- L100 in FIG. 4 indicates the pixel position of the lower edge of the rectangle B100.
- D_med be the disparity median value at the landmark 100 .
- D_med is obtained by calculating the median value of disparities contained within rectangle B100.
- BaseLine in Expression (1) is the base line length of the camera 10a and the camera 10b, and abs indicates an absolute value operator.
- the landmark information storage unit 36 registers the actual height information of the landmark 100 calculated in the three-dimensional information acquisition process (P104). Also, the landmark information storage unit 36 registers the texture information of the compound eye area image CA analyzed by the compound eye area landmark detection unit 32 . The landmark information storage unit 36 also stores a compound eye area image CA for the rectangle B100 detected in the three-dimensional object detection process (P103).
- FIG. 5 the distance estimation process will be described with reference to FIGS. 5 and 6.
- the external world recognition device 1 performs image acquisition processing (P111), landmark detection processing (P112), pedestrian detection processing (P113), size estimation processing (P115), distance The estimation process (P116) is executed in order.
- the image acquisition unit 10 acquires images captured by a pair of cameras 10a and 10b as shown in FIG. Acquire an area image RA).
- the monocular area landmark detection unit 52 detects the landmark 100 again.
- the landmark 100 was photographed in the compound eye area image CA in FIG. 4, but as the vehicle V moves, it is photographed in the left monocular area image LA in FIG.
- the landmark 100 photographed in the left monocular area image LA shown in FIG. 6 is re-detected. do.
- the area detected in this process is represented by a rectangle B101 as shown in FIG.
- the monocular area object detection unit 54 detects the pedestrian 200 photographed in the left monocular area image LA of FIG.
- a convolutional neural network is used for this detection.
- the convolutional neural network used detects the pedestrian 200 included in the input predetermined image.
- the convolutional neural network not only locates the pedestrian 200, but also calculates the identification score at the same time.
- the identification score is a point for the type of pedestrian 200, and a high point for a certain type indicates that the region is likely to be of that type.
- the pedestrian 200 is detected by inputting the left monocular area image LA into the convolutional neural network and adopting only detection results with a high identification score for the type of pedestrian.
- the detection result is represented by a rectangle B201 as shown in FIG.
- the size estimation unit 58 estimates the height of the pedestrian 200 using the landmark 100 information.
- the actual height of the landmark 100 is stored in the landmark information storage unit .
- the monocular area landmark detection unit 52 detects the image height of the landmark 100 (that is, the height of the rectangle B101 on the image) IMG_Height_Land(pix).
- the monocular area object detection unit 54 detects the image height of the pedestrian 200 (that is, the height of the rectangle B201 on the image) IMG_Height_Ped(pix).
- the distance calculation unit 60 calculates the height (Height_Ped) of the pedestrian 200 and the image height (IMG_Height_Ped(pix)) of the rectangle B201 detected in the pedestrian detection process (P113). , is used to obtain the distance from the own vehicle V to the pedestrian 200 .
- f is the focal length (pix) of the pair of cameras 10a and 10b. Note that once the height of the pedestrian is calculated in the size estimation process (P115), the distance estimation process can be performed using the height of the pedestrian 200 in another image frame after the image shown in FIG. At (P116), the distance between the vehicle V and the pedestrian 200 can be obtained.
- the external world recognition device 1 of this embodiment utilizes the actual height of any landmark 100 photographed in the compound eye area image CA to perform walking in the monocular area image (for example, the left monocular area image LA).
- the distance between the person 200 and the own vehicle V is calculated. Since the landmark 100 is a three-dimensional object and has texture information, parallax can be obtained more stably than the road surface. Therefore, it is possible to calculate the distance to the pedestrian 200 with higher accuracy even in a situation where it is difficult to obtain the parallax with respect to the road surface, such as at night or in rainy weather.
- the external world recognition device 1 of this embodiment measures the actual height of the landmark 100 in the compound eye area image CA, and uses the result to estimate the height of the pedestrian 200 .
- Pedestrian 200 changes its size in the horizontal direction according to the movement of the hand during walking.
- the height of the pedestrian 200 does not change or fluctuates little even when the pedestrian 200 walks. Therefore, by estimating the height of pedestrian 200 and obtaining the distance between pedestrian 200 and own vehicle V based on the estimated height, the distance to pedestrian 200 can be obtained with high accuracy.
- the mounting posture of the camera on the road surface can be accurately estimated. There is a need.
- the distance between the own vehicle V and the pedestrian 200 can be calculated without being affected by changes in the posture of the camera. can be estimated.
- FIG. 7 is a functional block diagram showing a schematic configuration of an external world recognition device 1d according to Modification 1 of the first embodiment.
- FIG. 8 is a flowchart showing distance estimation processing of the external world recognition device 1d of FIG.
- FIG. 9 is a diagram for explaining proximity determination processing of the external world recognition device 1d of FIG.
- the external world recognition device 1d according to Modification 1 differs from the external world recognition device 1 described above in that it includes a proximity determination unit 56, which will be described later.
- configurations having the same or similar functions as those of the external world recognition device 1 described above are denoted by the same reference numerals, and descriptions thereof are omitted, and different portions are described.
- the distance estimation unit 50 includes a proximity determination unit 56.
- the proximity determination unit 56 determines whether or not the landmark 100 detected by the monocular area landmark detection unit 52 and the pedestrian 200 detected by the monocular area object detection unit 54 are three-dimensionally close. Specifically, as shown in FIG. 9, the proximity determination unit 56 determines whether the lower end of the landmark 100 in the left monocular area image LA (monocular area image) and the pedestrian 200 in the left monocular area image LA (monocular area image) Calculate the difference in the height direction from the bottom edge. If the difference calculated by the proximity determination unit 56 is within a margin (proximity determination threshold value) ⁇ described later, and the size estimation unit 58 determines that the landmark 100 and the pedestrian 200 are close to each other, , to estimate the height of the pedestrian 200 .
- a margin proximity determination threshold value
- the external world recognition device 1d performs proximity determination processing ( P114) is executed.
- the proximity determination unit 56 spatially determines the landmark 100 detected by the monocular area landmark detection unit 52 and the pedestrian 200 detected by the monocular area object detection unit 54. It is determined whether or not it exists at a close position.
- the proximity determination unit 56 uses the lower end of the landmark 100.
- FIG. 9 the proximity determination unit 56 sets a predetermined margin ⁇ (for example, 5 pix) in the height direction around the lower end of the landmark 100, and the lower end of the pedestrian 200 is included in the margin ⁇ .
- the proximity determination unit 56 determines that the landmark 100 and the pedestrian 200 are at approximately the same distance from the host vehicle V, and determines that the two are close to each other. It is determined that When the proximity determination unit 56 determines that the landmark 100 and the pedestrian 200 are close to each other, in the size estimation process (P115), the size estimation unit 58 determines the size of the landmark 100 stored in the landmark information storage unit 36. The height of the pedestrian 200 is estimated using the actual height. Note that the above margin ⁇ (eg, 5 pix) may be stored in the landmark information storage unit 36 .
- the proximity determination unit 56 determines whether or not the landmark 100 is used for estimating the height of the pedestrian 200.
- the ratio of the size (ie, image height) of the pedestrian 200 and the landmark 100 on the image is used.
- the ratio of the image heights of the two may deviate from the ratio of the actual heights of the two, and the height of the pedestrian 200 cannot be accurately estimated.
- the height of the pedestrian 200 can be estimated more accurately by the proximity determination unit 56 performing the proximity determination process (P114) as in the first modification.
- FIG. 10 is a diagram showing another example of the proximity determination process (P114) of the external world recognition device 1d of FIG.
- the landmark registration process (P105) is described in the upper part, and the proximity determination process (P114) is described in the lower part.
- the example shown in FIG. 10 differs from the external world recognition device 1d according to Modification 1 described above in terms of landmark registration processing (P105) and proximity determination processing (P114).
- the same reference numerals as those of the external world recognition device 1d (FIG. 7) according to Modification 1 are attached and the description thereof will be made.
- the proximity determination unit 56 executes the proximity determination process (P114) based on the parallax around the landmark 100 detected by the compound eye region landmark detection unit 32.
- the compound eye area landmark detection unit 32 detects the inclination of the surface on which the landmark 100 is on the ground based on the parallax information around the surface on which the detected landmark 100 is on the ground. change may be calculated. As a result, the location where the inclination of the surface on which the landmark 100 is grounded is identified.
- the size information measurement unit 33 measures the image height (Obj_Height) of the landmark 100 detected by the compound eye area landmark detection unit 32 and the image from the center position of the landmark 100 to the point where the inclination changes.
- the landmark information storage unit 36 stores, for example, the value of Plane_rate as the proximity determination radius in addition to the predetermined margin ⁇ (landmark registration process (P105)).
- the proximity determination unit 56 uses the image height (Obj_Height2) of the landmark 100 re-detected by the monocular area landmark detection unit 52 in the proximity determination process (P114), Calculate the margin ⁇ 2 (Plane_rate ⁇ Obj_Height2). Then, the proximity determination unit 56 determines that the pedestrian 200 is close to the landmark 100 when the pedestrian 200 approaches within ⁇ 2 (pix) in the horizontal direction in addition to the margin ⁇ (pix) in the vertical direction. judge.
- the height of the pedestrian 200 is estimated on the assumption that the pedestrian 200 and the landmark 100 are at the same distance from the own vehicle V. However, if the inclination of the surface where the landmark 100 touches the ground changes in the middle, if only the vertical margin ⁇ is referred to in the proximity determination process (P114), the pedestrian 200 and the landmark 100 are not at the same distance. In spite of this, it may be determined that the landmark 100 and the pedestrian 200 are close to each other. Therefore, based on the parallax information, the area ( ⁇ 2 in the lower part of FIG. 10) up to the point where the inclination of the road surface does not change is calculated, and based on this result, the proximity determination process (P114) is performed, so that both are at equal distances or not. Therefore, the height of pedestrian 200 can be estimated more accurately.
- the external world recognition device 1e according to Modification 2 differs from the external world recognition device 1 described above in that it includes a landmark information measurement unit 34, which will be described later.
- a landmark information measurement unit 34 which will be described later.
- FIG. 11 is a functional block diagram showing a schematic configuration of an external world recognition device 1e according to modification 2 of the first embodiment.
- the landmark information acquisition section 30 includes a landmark information measurement section 34 .
- the landmark information measurement unit 34 determines whether the landmarks 100, 101, and 102 detected by the compound eye area landmark detection unit 32 have moved from the compound eye area image CA. judge. Then, the landmark information storage unit 36 stores the landmarks determined to be moving by the landmark information measurement unit 34 among the landmarks 100, 101, and 102 included in the compound eye area image CA (for example, the landmarks in FIG. The mark 101) is stored as a landmark for estimating the distance to the pedestrian 200.
- the external world recognition device 1e not only can stationary three-dimensional objects be registered as landmarks, but also three-dimensional objects such as other pedestrians and vehicles can be used as landmarks as mobile objects. Since the landmark 100 shown in FIG. 4 is a stationary object, if the pedestrian 200, who is the target of distance calculation with respect to the own vehicle V, is stationary, the two do not approach each other. Therefore, the height of stationary pedestrian 200 cannot be estimated based on landmark 100, which is a stationary object. On the other hand, by using a moving object (for example, the landmark 101) as in Modification 2, even when the pedestrian 200, which is the object of distance calculation, is stationary, the landmark 101 is the pedestrian. It can get close to 200. Therefore, it becomes possible to estimate the height of the stationary pedestrian 200 as well.
- a moving object for example, the landmark 101
- the external world recognition device according to Modification 3 differs from the external world recognition device described above in terms of the function of the landmark information storage unit 36 .
- the external world recognition device 1 according to Modification 3 will be described with the same reference numerals as the external world recognition device 1 described above, with reference to FIGS.
- the landmark information storage unit 36 shown in FIG. ) above (for example, the landmarks 100 and 102 in FIG. 4) are stored as landmarks for estimating the distance to the pedestrian 200 .
- the actual height of the landmark eg , 100 and 102 in FIG. 4 are registered in the landmark information storage unit 36 in the landmark registration process (P105).
- the landmark detection process P112
- the landmarks registered in the landmark registration process (P105) and having an actual height equal to or greater than the height threshold for example, the landmarks 100 and 102 in FIG. 4) are detected again. .
- the detected position may be misaligned.
- the image height of the landmark detected by the monocular area landmark detection unit 52 changes from the true image height of the landmark.
- the estimated height of pedestrian 200 may be deviated.
- the external world recognition device according to Modification 3 uses only landmarks having a height equal to or greater than a predetermined height threshold (for example, 1 m). Therefore, the difference between the height of the detected landmark image and the true height of the image can be reduced, and the height of the pedestrian can be estimated more accurately.
- the landmark information storage unit 36 selects the plurality of landmarks 100, 101, and 102 measured by the size information measuring unit 33 in descending order of actual height. (for example, the number corresponding to 60% of the plurality of landmarks) (for example, the landmarks 100 and 102) may be stored as landmarks for estimating the distance to the pedestrian 200.
- FIG. As a result, many options of landmarks used for distance estimation to the pedestrian 200 can be held.
- the external world recognition device according to Modified Example 4 differs from the external world recognition device 1e (FIG. 11) according to Modified Example 2 in terms of the functions of the landmark information measurement unit 34 and the landmark information storage unit 36.
- the external world recognition device according to Modified Example 4 will be described with reference to FIGS.
- the landmark information measurement unit 34 calculates the contrast information (edge component , color, luminance difference, etc.). Then, of the landmarks 100, 101, and 102 included in the compound eye area image CA, the landmark information storage unit 36 selects a landmark having contrast information equal to or greater than a predetermined threshold value (for example, the landmark 100 in FIG. 4), It is stored as a landmark for estimating the distance to pedestrian 200 .
- a predetermined threshold value for example, the landmark 100 in FIG. 4
- the landmark information measurement unit 34 applies a Sobel filter to each of the landmarks 100, 101, and 102 in the landmark registration process (P105) to obtain edge components (contrast information).
- the landmark information storage unit 36 selects landmarks (for example, the landmark 100) whose density of edge components (the number of edges) is equal to or greater than a predetermined value within the areas of the landmarks 100, 101, and 102. register.
- a landmark for example, landmark 100
- whose color information (contrast information) is equal to or greater than a predetermined value may be registered.
- pixels with saturation and brightness equal to or greater than a predetermined value are counted in a hue region that can be regarded as the red component of an image, and if the number of pixels is equal to or greater than a predetermined value, it is registered as a landmark. good too.
- the landmark detection process P112
- the landmark 100 can be re-detected more stably, and erroneous detection of other three-dimensional objects as landmarks can be avoided. Thereby, the height of pedestrian 200 can be estimated with higher accuracy.
- the landmark information measurement unit 34 analyzes the luminance difference between the lower ends of the landmarks 100, 101, and 102 and the road surface.
- the landmark information storage unit 36 stores landmarks (for example, landmarks) in which the number of pixels whose brightness difference between the lower end and the road surface is equal to or greater than a predetermined value among the landmarks 100, 101, and 102 is equal to or greater than a predetermined value. store the mark 100). This makes it easier for the monocular area landmark detection unit 52 to re-detect the same area as the area detected by the compound eye area landmark detection unit 32, thereby estimating the height of the pedestrian 200 with higher accuracy. can be done.
- the external world recognition device includes the proximity determination unit 56
- the proximity determination unit 56 when using the lower end of the landmark 100 in the proximity determination process (P114), it is possible to register the landmark 100 that is highly separable from the road surface. , the proximity determination can be performed more accurately.
- the landmark information storage unit 36 can register a specific part of the landmark in the landmark registration process (P105). Specifically, the landmark information measuring unit 34 measures, for example, a portion of the landmark 100 where the edge component is strong as the upper end of the landmark. Then, in the processing, the landmark information storage unit 36 can register the portion as the upper end of the landmark. This makes it easier for the monocular area landmark detection section 52 to re-detect the same portion as the portion detected by the compound eye area landmark detection section 32 . In this way, by registering, for example, a specific portion of the landmark 100 based on a portion with a strong edge component, the landmark can be stably re-detected by the monocular area landmark detection unit 52. Pedestrian distance can be estimated with high accuracy.
- the external world recognition device according to modification 5 differs from the external world recognition device 1 described above in terms of the function of the size estimation unit 58 .
- the external world recognition device according to Modification 5 will be described with the same reference numerals as the external world recognition device 1 described above, with reference to FIGS.
- the size estimation unit 58 uses the plurality of landmarks to estimate the pedestrian 200. Estimate the size of Specifically, the size estimation unit 58 estimates the height (Height_Ped) of the pedestrian 200 when approaching the landmark 100 in a certain image frame. Then, the distance calculation unit 60 calculates the distance between the vehicle V and the pedestrian 200 based on the Height_Ped. After that, in another image frame, when the pedestrian 200 approaches the landmark 101, the size estimator 58 calculates the pedestrian's Update height Height_Ped.
- Height_Ped_2 is the height of the pedestrian 200 estimated from the actual height of the landmark 101, the image height of the landmark 101, and the height of the pedestrian 200 image.
- size estimation unit 58 updates the height of pedestrian 200 each time pedestrian 200 approaches a plurality of landmarks. As a result, even if the initial height estimation result contains an error, the height can be updated so as to approach the correct value each time it approaches each landmark, and the distance of the pedestrian can be estimated more accurately. can.
- FIG. 12 is a functional block diagram showing a schematic configuration of the external world recognition device 1a according to the second embodiment of the present invention.
- the external world recognition device 1a according to the second embodiment differs from the external world recognition device 1d (Modification 1) in that it includes a landmark selection unit 40 and a course estimation unit 70, which will be described later.
- Configurations having the same or similar functions as those of the external world recognition device 1d described above are denoted by the same reference numerals as those of the external world recognition device 1d, and descriptions thereof are omitted. Different parts will be explained.
- the external world recognition device 1a includes a landmark selection unit 40 and a traveling route estimation unit 70.
- Landmark selection unit 40 is included in landmark information acquisition unit 30 .
- the landmark information acquisition unit 30 of the external world recognition device 1a acquires the three-dimensional information of the plurality of landmarks 100, 101, and 102
- the landmark information acquisition unit 30 determines the location of the plurality of landmarks according to the movement of the vehicle V. Select a landmark (eg, landmark 100) to be used for distance estimation.
- the landmark selection unit 40 of the landmark information acquisition unit 30 selects the landmark (for example, the landmark 100) used for distance estimation from a plurality of landmarks.
- the landmark selection unit 40 selects a landmark 100 existing in the traveling direction of the vehicle V estimated by the traveling path estimation unit 70 described later as a landmark used for estimating the distance to the pedestrian 200. select.
- the landmark information storage section 36 stores the landmark 100 selected by the landmark selection section 40 .
- the traveling path estimation unit 70 estimates the traveling direction of the vehicle V from the movement of the vehicle V (that is, behavior information). Specifically, the traveling route estimation unit 70 predicts the traveling route of the own vehicle V based on the speed of the vehicle V, the yaw rate, and the like. For example, the traveling path estimator 70 estimates the turning radius of the vehicle V from the values of the vehicle speed and the yaw rate, and sets the locus on the circumference as the traveling path. Note that, when the vehicle V is traveling straight ahead, the traveling path estimation unit 70 may estimate a trajectory along the straight traveling direction as the traveling path.
- the landmark registration process P105
- only the landmark for example, the landmark 100
- the traveling direction estimated by the traveling path estimation unit 70 is registered as a landmark.
- the landmark information storage unit 36 may store landmarks 100 that are within a predetermined distance (20 m) from the course of the vehicle V. FIG.
- FIG. 13 is a functional block diagram showing a schematic configuration of an external world recognition device 1b according to the third embodiment of the present invention.
- the external world recognition device 1b according to the third embodiment differs from the external world recognition device 1d (Modification 1) in that it includes a landmark selection unit 40 and a surrounding environment recognition unit 90, which will be described later.
- Configurations having the same or similar functions as those of the external world recognition device 1d described above are denoted by the same reference numerals as those of the external world recognition device 1d, and descriptions thereof are omitted. Different parts will be explained.
- the external world recognition device 1b includes a landmark selection unit 40 and a surrounding environment recognition unit 90 that recognizes the surrounding environment of the vehicle V.
- Landmark selection unit 40 is included in landmark information acquisition unit 30 .
- the landmark information acquisition unit 30 of the external world recognition device 1b recognizes the landmarks (for example, the landmark 100) existing in the area recognized as the sidewalk by the surrounding environment recognition unit 90 up to the pedestrian 200. as landmarks for distance estimation.
- the landmark selection unit 40 of the landmark information acquisition unit 30 selects the landmark (for example, the landmark 100) used for estimating the distance to the pedestrian 200.
- the landmark information storage unit 36 stores the landmarks selected by the landmark selection unit 40 .
- the surrounding environment recognition unit 90 estimates type information for each pixel of the images of the pair of cameras 10 a and 10 b acquired by the image acquisition unit 10 .
- the type information includes not only information on objects such as vehicles and pedestrians, but also information on road surfaces, sidewalks, and the like.
- a convolutional neural network is used to estimate the type information for each pixel.
- the surrounding environment recognition unit 90 estimates the type information corresponding to each pixel by using a model learned from the correct value data to which the type information for each pixel is added. As a result, the surrounding environment recognition unit 90 can recognize the sidewalk area from the images captured by the pair of cameras 10a and 10b.
- the landmark registration process is executed using the type information estimated by the surrounding environment recognition unit 90.
- the landmark information storage unit 36 stores the landmark 100 that exists on the pixel that the surrounding environment recognition unit 90 has estimated as the sidewalk.
- the pair of cameras 10a and 10b are used as one stereo camera in order to acquire the three-dimensional information (size information) of the landmarks 100, 101, and 102 in the landmark information acquisition unit 30.
- a three-dimensional sensor for example, lidar, millimeter wave radar, ultrasonic sensor, etc.
- the pair of cameras 10a and 10b may be installed so as not to have a stereo area.
- the camera 10 a may capture only the left monocular region
- the camera 10 b may capture only the right monocular region
- the images captured in these regions may be input to the image acquisition unit 10 .
- the distance between the vehicle V and the pedestrian 200 may be estimated using specific landmarks described on a digital map or the like and using three-dimensional information registered in the digital map. . Further, in the above-described embodiment, a detailed description was given from the viewpoint of estimating the height of pedestrian 200, but based on the width of pedestrian 200, the distance between own vehicle V and pedestrian 200 is calculated. good too.
- the distance estimating unit 50 estimates the distance to the pedestrian 200 based on the actual width of the landmark 100 and the lateral length (i.e. image width) of the landmark 100 and the pedestrian 200 in the image.
- the method of estimating the distance to the pedestrian 200 has been described by taking the pedestrian 200 as an example of the object, but various objects such as vehicles and animals can be treated as objects for distance estimation.
- the distance between these and the host vehicle V can be estimated based on the heights of the V and the like.
- the present invention is not limited to the above embodiments, and includes various modifications.
- the above embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described.
- it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.
- each of the above configurations, functions, processing units, processing means, etc. may be realized by hardware, for example, by designing them in integrated circuits, in part or in whole.
- each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function.
- Information such as programs, tapes, and files that implement each function can be stored in recording devices such as memories, hard disks, SSDs (solid state drives), or recording media such as IC cards, SD cards, and DVDs.
- control lines and information lines indicate what is considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In practice, it may be considered that almost all configurations are interconnected.
- 1, 1a, 1b, 1d, 1e external world recognition device 10 image acquisition unit, 10a, 10b pair of cameras (stereo cameras), 30 landmark information acquisition unit, 32 compound eye region landmark detection unit, 33 size information measurement unit , 34 landmark information measurement unit, 36 landmark information storage unit, 40 landmark selection unit, 50 distance estimation unit, 52 monocular area landmark detection unit, 54 monocular area object detection unit, 56 proximity determination unit, 58 size Estimation unit 60 Distance calculation unit 70 Course estimation unit 90 Surrounding environment recognition unit 100, 101, 102 Landmark 200 Pedestrian (object) CA Compound eye area image LA Left monocular area image (monocular area image ), RA right monocular area image (monocular area image), V vehicle, ⁇ margin (proximity judgment threshold)
Landscapes
- Physics & Mathematics (AREA)
- Electromagnetism (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Radar, Positioning & Navigation (AREA)
- Remote Sensing (AREA)
- Image Processing (AREA)
- Measurement Of Optical Distance (AREA)
- Image Analysis (AREA)
- Traffic Control Systems (AREA)
Abstract
Provided is an external environment recognition device capable of accurately calculating the distance to an object regardless of the usage environment. The external environment recognition device is mounted on a vehicle. The external environment recognition device has an image obtaining unit that obtains an image of a landmark and an object; a landmark information obtaining unit that obtains three dimensional information about the landmark; and a distance estimation unit that estimates the distance to the object on the basis of the three dimensional information about the landmark and the sizes of the landmark and the object in the image.
Description
本発明は、外界認識装置に関する。
The present invention relates to an external world recognition device.
従来から、カメラを用いて車両の前方を横切る歩行者等を検知し、必要に応じてドライバへの警報や自動ブレーキを行うシステムが知られている。特に、自動運転の実現や交通事故の防止の観点から、歩行者等を検出してブレーキ制御をおこなう自動ブレーキ装置に大きな関心が寄せられている。自動ブレーキ装置では、自車両から歩行者までの距離を精度高く算出する必要がある。また、自動ブレーキ装置では、歩行者の急な飛び出しに対応するため、より広い画角領域において歩行者の距離を求めることが必要となる。
Conventionally, systems are known that use cameras to detect pedestrians crossing the front of a vehicle, warn the driver and automatically brake as necessary. In particular, from the viewpoint of realization of automatic driving and prevention of traffic accidents, there is a great deal of interest in an automatic braking device that detects a pedestrian or the like and performs braking control. In the automatic braking device, it is necessary to calculate the distance from the own vehicle to the pedestrian with high accuracy. In addition, in the automatic braking device, it is necessary to obtain the distance to the pedestrian in a wider field of view in order to cope with the sudden movement of the pedestrian.
例えば特許文献1には、視野重複領域と視野非重複領域を備えるステレオカメラを利用した歩行者の距離算出方法が記載されている。具体的には、特許文献1には、視差情報から算出した路面高さを用いて、単眼領域の移動体の距離を算出するステレオカメラ装置が記載されている。当該ステレオカメラ装置では、階段の上にいる歩行者の足元の画像上の垂直座標から、路面に対する階段の高さを引いた(低くした)座標を得ている。そして、当該ステレオカメラ装置では、この得られた座標をステレオ領域まで伸ばして、その位置の路面の視差情報により歩行者の距離を求めている。
For example, Patent Literature 1 describes a pedestrian distance calculation method using a stereo camera that has a field-of-view overlapping area and a field-of-view non-overlapping area. Specifically, Patent Literature 1 describes a stereo camera device that calculates the distance of a moving object in a monocular region using road surface height calculated from parallax information. The stereo camera device obtains coordinates obtained by subtracting (lowering) the height of the stairs relative to the road surface from the vertical coordinates on the image of the feet of the pedestrian on the stairs. Then, in the stereo camera device, the obtained coordinates are extended to the stereo area, and the distance to the pedestrian is obtained from the parallax information of the road surface at that position.
ところで、特許文献1に記載のステレオカメラ装置では、取得された画像における路面のテクスチャ情報に基づいて当該路面の視差を算出し、この視差情報を歩行者の距離算出に利用している。しかしながら、例えば夜間や雨天等の、路面のテクスチャが鮮明ではない使用環境の場合、路面の視差情報を得ることが難しくなる可能性があり、歩行者の距離を正確に算出できないおそれがある。
By the way, in the stereo camera device described in Patent Document 1, the parallax of the road surface is calculated based on the texture information of the road surface in the acquired image, and this parallax information is used to calculate the distance to the pedestrian. However, in a usage environment where the texture of the road surface is not clear, such as at night or in rainy weather, it may be difficult to obtain the parallax information of the road surface, and the distance to the pedestrian may not be calculated accurately.
本発明は、上述の課題に鑑みてなされたものであり、その目的は、対象物までの距離を、使用環境に関わらず正確に算出することができる外界認識装置を提供することである。
The present invention has been made in view of the above problems, and its purpose is to provide an external world recognition device that can accurately calculate the distance to an object regardless of the usage environment.
上記目的を達成するために、本発明に係る外界認識装置は、車両に搭載される外界認識装置であって、ランドマークおよび対象物の画像を取得する画像取得部と、前記ランドマークの3次元情報を取得するランドマーク情報取得部と、前記ランドマークの3次元情報と、前記画像内における前記ランドマークおよび前記対象物の大きさと、に基づき、前記対象物までの距離を推定する距離推定部と、を有することを特徴とする。
In order to achieve the above object, an external world recognition device according to the present invention is an external world recognition device mounted on a vehicle, comprising: an image acquisition unit for acquiring an image of a landmark and an object; a landmark information acquisition unit that acquires information; and a distance estimation unit that estimates the distance to the object based on the three-dimensional information of the landmark and the sizes of the landmark and the object in the image. and.
本発明の外界認識装置によれば、対象物までの距離を、使用環境に関わらず正確に算出することができる。上記以外の課題、構成および効果は、以下の実施形態の説明により明らかにされる。
According to the external world recognition device of the present invention, the distance to the object can be accurately calculated regardless of the usage environment. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.
以下、本発明の実施形態について図面を参照し説明する。
Hereinafter, embodiments of the present invention will be described with reference to the drawings.
<第1実施形態>
図1は、本発明の第1実施形態に係る外界認識装置1の概略構成を示す機能ブロック図である。図2は、図1の外界認識装置1が車両Vに搭載された状態の一例を示す平面図である。図3は、図1の外界認識装置1のランドマーク情報取得処理を示すフローチャートである。図4は、図1の外界認識装置1がランドマーク情報取得処理で取得した画像の一例である。図5は、図1の外界認識装置1の距離推定処理を示すフローチャートである。図6は、図1の外界認識装置1が距離推定処理で取得した画像の一例である。 <First embodiment>
FIG. 1 is a functional block diagram showing a schematic configuration of an external world recognition device 1 according to the first embodiment of the invention. 2 is a plan view showing an example of a state in which the external world recognition device 1 of FIG. 1 is mounted on a vehicle V. FIG. FIG. 3 is a flowchart showing landmark information acquisition processing of the external world recognition device 1 of FIG. FIG. 4 is an example of an image acquired by the external world recognition device 1 of FIG. 1 in the landmark information acquisition process. FIG. 5 is a flowchart showing distance estimation processing of the external world recognition device 1 of FIG. FIG. 6 is an example of an image acquired by the external world recognition device 1 of FIG. 1 through distance estimation processing.
図1は、本発明の第1実施形態に係る外界認識装置1の概略構成を示す機能ブロック図である。図2は、図1の外界認識装置1が車両Vに搭載された状態の一例を示す平面図である。図3は、図1の外界認識装置1のランドマーク情報取得処理を示すフローチャートである。図4は、図1の外界認識装置1がランドマーク情報取得処理で取得した画像の一例である。図5は、図1の外界認識装置1の距離推定処理を示すフローチャートである。図6は、図1の外界認識装置1が距離推定処理で取得した画像の一例である。 <First embodiment>
FIG. 1 is a functional block diagram showing a schematic configuration of an external world recognition device 1 according to the first embodiment of the invention. 2 is a plan view showing an example of a state in which the external world recognition device 1 of FIG. 1 is mounted on a vehicle V. FIG. FIG. 3 is a flowchart showing landmark information acquisition processing of the external world recognition device 1 of FIG. FIG. 4 is an example of an image acquired by the external world recognition device 1 of FIG. 1 in the landmark information acquisition process. FIG. 5 is a flowchart showing distance estimation processing of the external world recognition device 1 of FIG. FIG. 6 is an example of an image acquired by the external world recognition device 1 of FIG. 1 through distance estimation processing.
外界認識装置1は、車両V(以下、自車両Vともいう)に搭載される。図1に示すように、外界認識装置1は、画像取得部10と、ランドマーク情報取得部30と、距離推定部50と、を有する。図示は省略するが、外界認識装置1は、CPU、RAM、ROMなどがバスを介して接続された構成を有し、CPUがROMに格納された各種制御プログラムを実行することで、システム全体の動作を制御する。本実施形態では、画像取得部10が、ステレオカメラとして機能する一対のカメラ10a、10b(図2)から画像を取得し、距離推定部50が、当該画像を用いて対象物としての歩行者200と自車両Vとの距離を推定する場合について説明する。本明細書で使用する「ランドマーク」とは、一対のカメラ10a、10bにより撮影される各立体物(例えば、図4に示す符号100、101、102)を意味する。本明細書において、「ランドマークの3次元情報」とは、ランドマーク100、101、102自体の高さ及び幅の少なくともいずれか一方を含む意味である。なお、「ランドマークの3次元情報」は、ランドマーク100、101、102自体の高さ及び幅の少なくともいずれか一方に加え、当該ランドマーク100、101、102と車両Vとの間の距離(奥行き)を含んでもよい。
The external world recognition device 1 is mounted on a vehicle V (hereinafter also referred to as own vehicle V). As shown in FIG. 1 , the external world recognition device 1 has an image acquisition section 10 , a landmark information acquisition section 30 and a distance estimation section 50 . Although illustration is omitted, the external world recognition device 1 has a configuration in which a CPU, a RAM, a ROM, and the like are connected via a bus, and the CPU executes various control programs stored in the ROM to control the entire system. control behavior. In this embodiment, the image acquisition unit 10 acquires images from a pair of cameras 10a and 10b (FIG. 2) functioning as stereo cameras, and the distance estimation unit 50 uses the images to determine the pedestrian 200 as the target object. and the own vehicle V will be described. A “landmark” used in this specification means each three-dimensional object (for example, reference numerals 100, 101 and 102 shown in FIG. 4) photographed by the pair of cameras 10a and 10b. In this specification, "three-dimensional information of landmarks" means including at least one of the height and width of the landmarks 100, 101, and 102 themselves. In addition to at least one of the height and width of the landmarks 100, 101, and 102 themselves, the "three-dimensional information of the landmarks" includes the distance between the landmarks 100, 101, and 102 and the vehicle V ( depth).
図4及び図6に示すように、画像取得部10は、一対のカメラ10a、10bから、ランドマーク100、101、102および歩行者(対象物)200の画像を取得する。具体的には、画像取得部10は、一対のカメラ10a、10bによって撮像された画像であって、視野が重複する複眼領域と視野が重複しない単眼領域(左単眼領域及び右単眼領域)とを含む画像を取得する。以下、当該画像の複眼領域を複眼領域画像CAと記載し、当該画像の左単眼領域及び右単眼領域をそれぞれ、左単眼領域画像LA及び右単眼領域画像RAと記載する。図2に示すように、一対のカメラ10a、10bは、車両Vに搭載された1台のステレオカメラとして構成されている。具体的には、一対のカメラ10a、10bは、CCDやCMOSイメージセンサで構成されており、自車両Vの前方に向けられている。また、一対のカメラ10a、10bは、それぞれが自車両Vの前方を所定の俯角(即ち撮影領域)で撮影するとともに、それぞれの俯角が重なるように設置されている。例えば、カメラ10aを左カメラ、カメラ10bを右カメラとした場合、カメラ10aの俯角の左側部分と、カメラ10bの右側部分とが重なるようになっている。これにより、本実施形態では、自車両Vの中央前方領域が、カメラ10aの撮影領域とカメラ10bの撮影領域とで画成されるステレオ領域(複眼領域ともいう)となる。また、自車両Vの左前方領域は、カメラ10bの撮影領域の左側部分で画成される左単眼領域となっている。自車両Vの右前方領域は、カメラ10aの撮影領域の右側部分で画成される右単眼領域となっている。一対のカメラ10a、10bにより撮影された画像は、画像取得部10に入力される。画像取得部10は、一対のカメラ10a、10bにより撮影された2枚の画像を取得し、取得した左右の画像に対し公知の視差算出アルゴリズムを適用し視差情報を取得する。
As shown in FIGS. 4 and 6, the image acquisition unit 10 acquires images of landmarks 100, 101 and 102 and a pedestrian (object) 200 from a pair of cameras 10a and 10b. Specifically, the image acquisition unit 10 captures an image captured by a pair of cameras 10a and 10b, and divides a compound eye region with overlapping fields of view and a monocular region (a left monocular region and a right monocular region) with non-overlapping fields of view. Get the containing image. Hereinafter, the compound eye area of the image will be referred to as compound eye area image CA, and the left monocular area and right monocular area of the image will be referred to as left monocular area image LA and right monocular area image RA, respectively. As shown in FIG. 2, the pair of cameras 10a and 10b are configured as one stereo camera mounted on a vehicle V. As shown in FIG. Specifically, the pair of cameras 10a and 10b are composed of CCD or CMOS image sensors, and are directed to the front of the vehicle V. As shown in FIG. Also, the pair of cameras 10a and 10b are installed so that each of the cameras 10a and 10b captures an image in front of the vehicle V at a predetermined depression angle (that is, an imaging area), and the angles of depression overlap with each other. For example, if the camera 10a is the left camera and the camera 10b is the right camera, the left side of the depression angle of the camera 10a and the right side of the camera 10b overlap each other. As a result, in the present embodiment, the center front area of the vehicle V becomes a stereo area (also called a compound eye area) defined by the imaging area of the camera 10a and the imaging area of the camera 10b. Also, the left front area of the vehicle V is a left monocular area defined by the left portion of the photographing area of the camera 10b. The right front area of the vehicle V is a right monocular area defined by the right side of the imaging area of the camera 10a. Images captured by the pair of cameras 10 a and 10 b are input to the image acquisition section 10 . The image acquisition unit 10 acquires two images captured by a pair of cameras 10a and 10b, and acquires parallax information by applying a known parallax calculation algorithm to the acquired left and right images.
図1に示すように、ランドマーク情報取得部30は、複眼領域ランドマーク検出部32と、大きさ情報計測部33と、ランドマーク情報記憶部36と、を含む。ランドマーク情報取得部30は、ランドマーク100、101、102の3次元情報を取得する。具体的には、ランドマーク情報取得部30は、複眼領域画像CAに位置するランドマーク100、101、102の3次元情報を取得する。
As shown in FIG. 1, the landmark information acquisition unit 30 includes a compound eye region landmark detection unit 32, a size information measurement unit 33, and a landmark information storage unit 36. A landmark information acquisition unit 30 acquires three-dimensional information of the landmarks 100 , 101 and 102 . Specifically, the landmark information acquisition unit 30 acquires three-dimensional information of the landmarks 100, 101, and 102 positioned in the compound eye area image CA.
図4に示すように、複眼領域ランドマーク検出部32は、画像取得部10で取得された複眼領域画像CAからランドマーク100、101、102を検出する。例えば、複眼領域ランドマーク検出部32は、画像取得部10から送信された複眼領域画像CAのテクスチャを解析し、これによりランドマーク100、101、102を検出してもよい。また、複眼領域ランドマーク検出部32は、画像取得部10が取得した視差情報に基づいて、ランドマーク100、101、102を検出してもよい。具体的には、視差情報等をクラスタリングすることによりランドマーク100、101、102を検出してもよいし、統計的な機械学習として畳み込みニューラルネットワークを利用してランドマーク100、101、102を検出してもよい。
As shown in FIG. 4, the compound eye area landmark detection unit 32 detects landmarks 100, 101, and 102 from the compound eye area image CA acquired by the image acquisition unit 10. FIG. For example, the compound eye area landmark detection unit 32 may analyze the texture of the compound eye area image CA transmitted from the image acquisition unit 10 and detect the landmarks 100, 101 and 102 from this. Moreover, the compound eye area landmark detection unit 32 may detect the landmarks 100 , 101 , and 102 based on the parallax information acquired by the image acquisition unit 10 . Specifically, the landmarks 100, 101, and 102 may be detected by clustering disparity information or the like, or the landmarks 100, 101, and 102 may be detected using a convolutional neural network as statistical machine learning. You may
大きさ情報計測部33は、複眼領域画像CAから、3次元情報に含まれるランドマーク100、101、102の実際の大きさを計測する。具体的には、大きさ情報計測部33は、複眼領域ランドマーク検出部32によって検出されたランドマーク100、101、102に対し、複眼領域画像CAから当該ランドマーク100、101、102の実高さを算出する。なお、大きさ情報計測部33は、ランドマーク100、101、102の縦方向の実際の大きさ(実高さ)だけでなく、ランドマーク100、101、102の横方向の実際の大きさ(実幅)を計測してもよい。また、大きさ情報計測部33は、ランドマーク100、101、102のうちの一部の領域における実際の大きさ(実高さや実幅)を計測してもよい。ランドマーク情報記憶部36は、大きさ情報計測部33において計測されたランドマーク100、101、102の大きさ情報の少なくとも一つを、後述する距離算出に用いるランドマーク情報として登録する。具体的には、ランドマーク情報記憶部36は、計測したランドマーク100、101、102の大きさ情報(実高さ、実幅等)に加えて、撮影されたランドマーク100、101、102のテクスチャ情報を記憶する。
The size information measuring unit 33 measures the actual sizes of the landmarks 100, 101, and 102 included in the three-dimensional information from the compound eye area image CA. Specifically, the size information measurement unit 33 calculates the actual heights of the landmarks 100, 101, and 102 detected by the compound eye area landmark detection unit 32 from the compound eye area image CA. to calculate the Note that the size information measuring unit 33 measures not only the actual vertical size (actual height) of the landmarks 100, 101, and 102, but also the actual horizontal size (actual height) of the landmarks 100, 101, and 102. actual width) may be measured. Also, the size information measuring unit 33 may measure the actual size (actual height or actual width) of a part of the landmarks 100 , 101 , 102 . The landmark information storage unit 36 registers at least one of the size information of the landmarks 100, 101, and 102 measured by the size information measuring unit 33 as landmark information used for distance calculation, which will be described later. Specifically, the landmark information storage unit 36 stores size information (actual height, actual width, etc.) of the landmarks 100, 101, and 102 that have been measured, as well as size information of the photographed landmarks 100, 101, and 102. Stores texture information.
距離推定部50は、単眼領域画像(左単眼領域画像LA、右単眼領域画像RA)に位置する歩行者200までの距離を推定する。具体的には、距離推定部50は、ランドマーク情報取得部30において検出したランドマーク100、101、102を利用して、画像取得部10で取得した単眼領域画像(左単眼領域画像LA、右単眼領域画像RA)に含まれる歩行者200までの距離を推定する。より具体的には、距離推定部50は、ランドマーク100の3次元情報と、画像内におけるランドマーク100および歩行者200の大きさと、に基づき、自車両Vから歩行者200までの距離を推定する。本実施形態では、ランドマーク100を、歩行者200と自車両Vとの距離を推定するランドマークとする場合について説明する。図1に示すように、距離推定部50は、単眼領域ランドマーク検出部52と、単眼領域対象物検出部54と、大きさ推定部58と、距離算出部60と、を含む。
The distance estimation unit 50 estimates the distance to the pedestrian 200 located in the monocular area image (left monocular area image LA, right monocular area image RA). Specifically, the distance estimating unit 50 uses the landmarks 100, 101, and 102 detected by the landmark information acquiring unit 30 to use the monocular area images (left monocular area image LA, right The distance to the pedestrian 200 included in the monocular area image RA) is estimated. More specifically, the distance estimation unit 50 estimates the distance from the vehicle V to the pedestrian 200 based on the three-dimensional information of the landmark 100 and the sizes of the landmark 100 and the pedestrian 200 in the image. do. In this embodiment, a case where the landmark 100 is used as a landmark for estimating the distance between the pedestrian 200 and the own vehicle V will be described. As shown in FIG. 1 , the distance estimation unit 50 includes a monocular area landmark detection unit 52 , a monocular area object detection unit 54 , a size estimation unit 58 and a distance calculation unit 60 .
図6に示すように、単眼領域ランドマーク検出部52は、画像取得部10で取得した単眼領域画像(左単眼領域画像LA、右単眼領域画像RA)からランドマーク100を検出する。具体的には、単眼領域ランドマーク検出部52は、ランドマーク情報記憶部36で記憶したランドマーク100のテクスチャ情報に基づきランドマーク100を再検出する。例えば、単眼領域ランドマーク検出部52は、ランドマーク情報記憶部36で記憶したランドマーク100のテクスチャ情報をテンプレートとしたテンプレートマッチングにより検出してもよい。また、単眼領域ランドマーク検出部52は、畳み込みニューラルネットワークを利用した追跡処理によりランドマーク100を検出してもよい。また、単眼領域ランドマーク検出部52は、画像取得部10から取得したランドマーク100の単眼領域画像(左単眼領域画像LA、右単眼領域画像RA)から、画像内におけるランドマーク100の縦方向長さ(即ち画像高さ)を算出する。
As shown in FIG. 6, the monocular area landmark detection unit 52 detects landmarks 100 from the monocular area images (left monocular area image LA, right monocular area image RA) acquired by the image acquisition unit 10 . Specifically, the monocular area landmark detection unit 52 re-detects the landmark 100 based on the texture information of the landmark 100 stored in the landmark information storage unit 36 . For example, the monocular area landmark detection unit 52 may detect by template matching using the texture information of the landmark 100 stored in the landmark information storage unit 36 as a template. Also, the monocular area landmark detection unit 52 may detect the landmark 100 by tracking processing using a convolutional neural network. Further, the monocular area landmark detection unit 52 detects the vertical length of the landmark 100 in the image from the monocular area images (the left monocular area image LA and the right monocular area image RA) of the landmark 100 acquired from the image acquisition unit 10 . height (i.e., image height).
単眼領域対象物検出部54は、画像取得部10で取得した単眼領域画像(左単眼領域画像LA、右単眼領域画像RA)から、距離推定の対象物を検出する。単眼領域対象物検出部54は、例えば歩行者200が検出対象である場合には公知の統計的な機械学習により検出する。具体的には、単眼領域対象物検出部54は、ランダムフォレスト、サポートベクターマシン、リアルアダブーストなどの古典的な機械学習を利用して歩行者200を検出してもよいし、畳み込みニューラルネットワークを利用して歩行者200を検出してもよい。また、単眼領域対象物検出部54は、画像取得部10から取得した歩行者200の単眼領域画像に基づき、画像内における歩行者200の縦方向長さ(即ち画像高さ)を算出する。
The monocular area object detection unit 54 detects an object for distance estimation from the monocular area images (left monocular area image LA, right monocular area image RA) acquired by the image acquisition unit 10 . The monocular region target object detection unit 54 detects, for example, a pedestrian 200 by known statistical machine learning when the target is the pedestrian 200 . Specifically, the monocular region object detection unit 54 may detect the pedestrian 200 using classical machine learning such as random forest, support vector machine, and real Adaboost, or may detect the pedestrian 200 using a convolutional neural network. A pedestrian 200 may be detected by using this. Further, the monocular area object detection unit 54 calculates the vertical length (that is, the image height) of the pedestrian 200 in the image based on the monocular area image of the pedestrian 200 acquired from the image acquisition unit 10 .
大きさ推定部58は、単眼領域画像(左単眼領域画像LA等)のランドマーク100の画像サイズ及び単眼領域画像(左単眼領域画像LA等)の歩行者200の画像サイズに基づき、ランドマーク100の大きさから歩行者200の大きさを推定する。大きさ推定部58は、ランドマーク情報記憶部36において記憶したランドマーク100の大きさ(高さ等)を利用して、歩行者200の大きさ(高さ等)を推定する。具体的には、大きさ推定部58は、ランドマーク100の画像高さ及び歩行者200の画像高さに基づき、ランドマーク100の実高さから歩行者200の高さ(以下、身長ともいう)を推定する。より具体的には、大きさ推定部58は、ランドマーク100の画像高さに対する歩行者200の画像高さの比率を、ランドマーク100の実高さに乗算することにより、歩行者200の身長を推定する。歩行者200の大きさ情報は、一度推定するだけでもよい。
The size estimation unit 58 estimates the landmark 100 based on the image size of the landmark 100 in the monocular area image (left monocular area image LA, etc.) and the image size of the pedestrian 200 in the monocular area image (left monocular area image LA, etc.). The size of pedestrian 200 is estimated from the size of . The size estimation unit 58 estimates the size (height, etc.) of the pedestrian 200 using the size (height, etc.) of the landmark 100 stored in the landmark information storage unit 36 . Specifically, based on the image height of the landmark 100 and the image height of the pedestrian 200, the size estimation unit 58 calculates the height of the pedestrian 200 from the actual height of the landmark 100 (hereinafter also referred to as height). ). More specifically, the size estimation unit 58 multiplies the actual height of the landmark 100 by the ratio of the height of the image of the pedestrian 200 to the height of the image of the landmark 100 to obtain the height of the pedestrian 200. to estimate The size information of pedestrian 200 may be estimated only once.
距離算出部60は、大きさ推定部58において推定された歩行者200の実際の大きさと単眼領域画像(左単眼領域画像LA、右単眼領域画像RA)における歩行者200の画像サイズとに基づき、歩行者200までの距離を算出する。具体的には、距離算出部60は、大きさ推定部58で推定された歩行者200の身長に基づき、自車両Vから歩行者200までの距離を算出する。より具体的には、距離算出部60は、大きさ推定部58で推定された歩行者200の身長と単眼領域対象物検出部54で検出された歩行者200の画像高さとに基づき、自車両Vと歩行者200との間の距離を算出する。
Based on the actual size of the pedestrian 200 estimated by the size estimation unit 58 and the image size of the pedestrian 200 in the monocular area images (left monocular area image LA, right monocular area image RA), the distance calculation unit 60 A distance to the pedestrian 200 is calculated. Specifically, the distance calculation unit 60 calculates the distance from the own vehicle V to the pedestrian 200 based on the height of the pedestrian 200 estimated by the size estimation unit 58 . More specifically, the distance calculation unit 60 calculates the size of the vehicle based on the height of the pedestrian 200 estimated by the size estimation unit 58 and the image height of the pedestrian 200 detected by the monocular area object detection unit 54. The distance between V and pedestrian 200 is calculated.
次に、図3から図6を参照して、本実施形態の外界認識装置1の動作例を、ランドマーク情報取得処理(図3、図4)及び距離推定処理(図5、図6)に分けて説明する。ここでは、図2に示すように、車両Vの前方を監視するように設置されたステレオカメラとして機能する一対のカメラ10a、10bを利用する外界認識装置1に関して説明する。また、図6に示す左単眼領域画像LA内に撮影された歩行者200の身長に基づいて、自車両Vと歩行者200との間の距離を算出する場合を説明する。
Next, with reference to FIGS. 3 to 6, an operation example of the external world recognition device 1 of this embodiment will be described in landmark information acquisition processing (FIGS. 3 and 4) and distance estimation processing (FIGS. 5 and 6). I will explain separately. Here, as shown in FIG. 2, the external world recognition device 1 using a pair of cameras 10a and 10b functioning as stereo cameras installed to monitor the front of the vehicle V will be described. Also, a case of calculating the distance between the own vehicle V and the pedestrian 200 based on the height of the pedestrian 200 photographed in the left monocular area image LA shown in FIG. 6 will be described.
まず、図3及び図4を参照してランドマーク情報取得処理について説明する。
First, the landmark information acquisition process will be described with reference to FIGS.
図3に示すように、外界認識装置1は、ランドマーク情報取得処理において、画像取得処理(P101)、視差計算処理(P102)、立体物検出処理(P103)、3次元情報取得処理(P104)、ランドマーク登録処理(P105)を順に実行する。
As shown in FIG. 3, the external world recognition device 1 performs image acquisition processing (P101), parallax calculation processing (P102), solid object detection processing (P103), and three-dimensional information acquisition processing (P104) in the landmark information acquisition processing. , the landmark registration process (P105) is executed in order.
画像取得処理(P101)において、画像取得部10は、図4に示すような一対のカメラ10a、10bで撮影された画像、即ち複眼領域画像CA及び単眼領域画像(左単眼領域画像LA、右単眼領域画像RA)を取得する。
In the image acquisition process (P101), the image acquisition unit 10 acquires images captured by a pair of cameras 10a and 10b as shown in FIG. Acquire an area image RA).
次いで、視差計算処理(P102)において、画像取得部10は、複眼領域画像CAにおける視差を計算する。例えば、当該視差の算出には5×5ピクセルの窓領域が設定される。そして、一対のカメラ10a、10bのうち左側のカメラ10aの各画像を基準に、右側のカメラ10bの画像を、SADを評価値として水平方向に走査することで、視差が計算される。
Next, in the parallax calculation process (P102), the image acquisition unit 10 calculates the parallax in the compound eye area image CA. For example, a window area of 5×5 pixels is set for the parallax calculation. Then, the parallax is calculated by horizontally scanning the image of the right camera 10b using the SAD as an evaluation value, with the image of the left camera 10a of the pair of cameras 10a and 10b as a reference.
次いで、立体物検出処理(P103)において、複眼領域ランドマーク検出部32は、視差計算処理(P102)で計算した視差を利用して、図4に示すように、ランドマーク100、101、102を検出する。具体的には、各ランドマーク100、101、102の領域は同一距離になることから、視差に対してクラスタリング処理を実施することで、各ランドマーク100、101、102の存在する領域が粗く抽出される。次いで、各ランドマーク100、101、102と背景の境界とで距離の値が変化することから、粗く抽出した領域に対して視差の変化点を検出することで、ランドマーク100、101、102の存在する領域が詳細に検出される。図4では、複数のランドマーク100、101、102のうち、ランドマーク100に対する立体物検出処理(P103)の検出結果を示す。立体物検出処理(P103)では、図4に示すような矩形B100が検出結果となる。
Next, in the three-dimensional object detection process (P103), the compound eye region landmark detection unit 32 uses the parallax calculated in the parallax calculation process (P102) to detect landmarks 100, 101, and 102 as shown in FIG. To detect. Specifically, since the areas of the landmarks 100, 101, and 102 are at the same distance, clustering processing is performed on the parallax to roughly extract the areas where the landmarks 100, 101, and 102 exist. be done. Next, since the distance value changes between each of the landmarks 100, 101, and 102 and the boundary of the background, by detecting the parallax change point in the roughly extracted region, the landmarks 100, 101, and 102 The existing areas are detected in detail. FIG. 4 shows the detection result of the three-dimensional object detection process (P103) for the landmark 100 among the plurality of landmarks 100, 101, and 102. FIG. In the three-dimensional object detection process (P103), a rectangle B100 as shown in FIG. 4 is the detection result.
次いで、3次元情報取得処理(P104)において、大きさ情報計測部33は、立体物検出処理(P103)で検出した各ランドマーク100、101、102のそれぞれに対し、高さ方向の大きさ(即ち実高さ)を計算する。以降では、複数のランドマーク100、101、102のうち、ランドマーク100に対する大きさ(実高さ)を推定する方法について説明する。ランドマーク100の実高さの推定には、立体物検出処理(P103)の検出結果である矩形B100を利用する。図4のT100は、矩形B100の上端のピクセル位置であり、図4のL100は矩形B100の下端のピクセル位置を示している。また、ランドマーク100における視差中央値をD_medとする。D_medは矩形B100内に含まれる視差の中央値を計算することで取得される。ランドマーク100の実高さHeightは、「Height=BaseLine×abs(T100-L100)/D_med」(式(1))を計算することで算出される。ここで、式(1)におけるBaseLineはカメラ10aとカメラ10bの基線長であり、absは絶対値演算子を示している。
Next, in the three-dimensional information acquisition process (P104), the size information measuring unit 33 measures the height direction size ( That is, the actual height) is calculated. A method of estimating the size (actual height) with respect to the landmark 100 among the plurality of landmarks 100, 101, and 102 will be described below. For estimating the actual height of the landmark 100, the rectangle B100, which is the detection result of the three-dimensional object detection process (P103), is used. T100 in FIG. 4 is the pixel position of the upper edge of the rectangle B100, and L100 in FIG. 4 indicates the pixel position of the lower edge of the rectangle B100. Also, let D_med be the disparity median value at the landmark 100 . D_med is obtained by calculating the median value of disparities contained within rectangle B100. The actual height Height of the landmark 100 is calculated by calculating "Height=BaseLine×abs(T100-L100)/D_med" (formula (1)). Here, BaseLine in Expression (1) is the base line length of the camera 10a and the camera 10b, and abs indicates an absolute value operator.
次いで、ランドマーク登録処理(P105)において、ランドマーク情報記憶部36は、3次元情報取得処理(P104)で計算したランドマーク100の実高さの情報を登録する。また、ランドマーク情報記憶部36は、複眼領域ランドマーク検出部32で解析された複眼領域画像CAのテクスチャ情報を登録する。また、ランドマーク情報記憶部36は、立体物検出処理(P103)で検出した矩形B100に対する複眼領域画像CAを記憶する。
Next, in the landmark registration process (P105), the landmark information storage unit 36 registers the actual height information of the landmark 100 calculated in the three-dimensional information acquisition process (P104). Also, the landmark information storage unit 36 registers the texture information of the compound eye area image CA analyzed by the compound eye area landmark detection unit 32 . The landmark information storage unit 36 also stores a compound eye area image CA for the rectangle B100 detected in the three-dimensional object detection process (P103).
次いで、図5及び図6を参照して距離推定処理について説明する。
Next, the distance estimation process will be described with reference to FIGS. 5 and 6. FIG.
図5に示すように、外界認識装置1は、距離推定処理において、画像取得処理(P111)、ランドマーク検出処理(P112)、歩行者検出処理(P113)、大きさ推定処理(P115)、距離推定処理(P116)を順に実行する。
As shown in FIG. 5, in the distance estimation process, the external world recognition device 1 performs image acquisition processing (P111), landmark detection processing (P112), pedestrian detection processing (P113), size estimation processing (P115), distance The estimation process (P116) is executed in order.
画像取得処理(P111)において、画像取得部10は、図6に示すような一対のカメラ10a、10bで撮影された画像、即ち複眼領域画像CA及び単眼領域画像(左単眼領域画像LA、右単眼領域画像RA)を取得する。
In the image acquisition process (P111), the image acquisition unit 10 acquires images captured by a pair of cameras 10a and 10b as shown in FIG. Acquire an area image RA).
次いで、ランドマーク検出処理(P112)において、単眼領域ランドマーク検出部52は、ランドマーク100を再検出する。ランドマーク100は、図4では複眼領域画像CAにて撮影されていたが、車両Vの移動に伴い図6では左単眼領域画像LAにて撮影されている。ここでは、ランドマーク情報記憶部36に記憶されたランドマーク100のテクスチャ情報をテンプレートとしてテンプレートマッチングを実施することで、図6に示す左単眼領域画像LAにて撮影されたランドマーク100を再検出する。本処理において検出された領域は、図6に示すように矩形B101により表現される。
Next, in the landmark detection process (P112), the monocular area landmark detection unit 52 detects the landmark 100 again. The landmark 100 was photographed in the compound eye area image CA in FIG. 4, but as the vehicle V moves, it is photographed in the left monocular area image LA in FIG. Here, by executing template matching using the texture information of the landmark 100 stored in the landmark information storage unit 36 as a template, the landmark 100 photographed in the left monocular area image LA shown in FIG. 6 is re-detected. do. The area detected in this process is represented by a rectangle B101 as shown in FIG.
次いで、歩行者検出処理(P113)において、単眼領域対象物検出部54は、図6の左単眼領域画像LAに撮影された歩行者200を検出する。この検出には畳み込みニューラルネットワークを利用する。利用する畳み込みニューラルネットワークでは、入力された所定の画像に対し、当該画像に含まれる歩行者200を検出する。また、畳み込みニューラルネットワークは歩行者200の位置を特定するだけでなく、識別スコアを同時に算出する。識別スコアとは、歩行者200の種別に対するポイントであり、ある種別に対するポイントが高いということは、その領域がその種別である可能性が高いことを示す。歩行者検出処理(P113)では、左単眼領域画像LAを畳み込みニューラルネットワークに入力して、歩行者種別に対する識別スコアが高い検出結果のみを採用することで、歩行者200を検出する。検出結果は、図6に示すように矩形B201により表現される。
Next, in the pedestrian detection process (P113), the monocular area object detection unit 54 detects the pedestrian 200 photographed in the left monocular area image LA of FIG. A convolutional neural network is used for this detection. The convolutional neural network used detects the pedestrian 200 included in the input predetermined image. In addition, the convolutional neural network not only locates the pedestrian 200, but also calculates the identification score at the same time. The identification score is a point for the type of pedestrian 200, and a high point for a certain type indicates that the region is likely to be of that type. In the pedestrian detection process (P113), the pedestrian 200 is detected by inputting the left monocular area image LA into the convolutional neural network and adopting only detection results with a high identification score for the type of pedestrian. The detection result is represented by a rectangle B201 as shown in FIG.
大きさ推定処理(P115)において、大きさ推定部58は、ランドマーク100の情報を利用して歩行者200の身長を推定する。上述したランドマーク登録処理(P105)において、ランドマーク情報記憶部36に、ランドマーク100の実高さHeightが記憶されている。また、ランドマーク検出処理(P112)において、単眼領域ランドマーク検出部52により、ランドマーク100の画像高さ(即ち、矩形B101の画像上の高さ)IMG_Height_Land(pix)が検出されている。また、歩行者検出処理(P113)において、単眼領域対象物検出部54により、歩行者200の画像高さ(即ち、矩形B201の画像上の高さ)IMG_Height_Ped(pix)が検出されている。大きさ推定処理(P115)では、歩行者200の身長Height_Pedを、「Height_Ped=(IMG_Height_Ped/IMG_Height_Land)×Height」(式(2))により推定する。つまり、ランドマーク100の実高さ(Height)に、ランドマーク100の画像高さ(IMG_Height_Land)及び歩行者200の画像高さ(IMG_Height_Ped)の比率を適用することで、歩行者200の身長を推定する。
In the size estimation process (P115), the size estimation unit 58 estimates the height of the pedestrian 200 using the landmark 100 information. In the landmark registration process (P105) described above, the actual height of the landmark 100 is stored in the landmark information storage unit . Further, in the landmark detection process (P112), the monocular area landmark detection unit 52 detects the image height of the landmark 100 (that is, the height of the rectangle B101 on the image) IMG_Height_Land(pix). Also, in the pedestrian detection process (P113), the monocular area object detection unit 54 detects the image height of the pedestrian 200 (that is, the height of the rectangle B201 on the image) IMG_Height_Ped(pix). In the size estimation process (P115), the height Height_Ped of the pedestrian 200 is estimated by "Height_Ped=(IMG_Height_Ped/IMG_Height_Land)×Height" (formula (2)). That is, by applying the ratio of the image height (IMG_Height_Land) of the landmark 100 and the image height (IMG_Height_Ped) of the pedestrian 200 to the actual height (Height) of the landmark 100, the height of the pedestrian 200 is estimated. do.
次いで、距離推定処理(P116)において、距離算出部60は、歩行者200の身長(Height_Ped)と、歩行者検出処理(P113)で検出された矩形B201の画像高さ(IMG_Height_Ped(pix))と、を利用して自車両Vから歩行者200までの距離を求める。自車両Vから歩行者200までの距離Distは、「Dist=f×(Height_Ped/IMG_Height_Ped)」(式(3))に基づき算出される。ここで、fは、一対のカメラ10a、10bの焦点距離(pix)である。なお、大きさ推定処理(P115)で歩行者の身長を一度算出しておけば、図6に示した画像以降の別の画像フレームにおける歩行者200の画像高さを利用して、距離推定処理(P116)で自車両Vと歩行者200との間の距離を求めることができる。
Next, in the distance estimation process (P116), the distance calculation unit 60 calculates the height (Height_Ped) of the pedestrian 200 and the image height (IMG_Height_Ped(pix)) of the rectangle B201 detected in the pedestrian detection process (P113). , is used to obtain the distance from the own vehicle V to the pedestrian 200 . The distance Dist from the host vehicle V to the pedestrian 200 is calculated based on "Dist=f×(Height_Ped/IMG_Height_Ped)" (Formula (3)). Here, f is the focal length (pix) of the pair of cameras 10a and 10b. Note that once the height of the pedestrian is calculated in the size estimation process (P115), the distance estimation process can be performed using the height of the pedestrian 200 in another image frame after the image shown in FIG. At (P116), the distance between the vehicle V and the pedestrian 200 can be obtained.
このように、本実施形態の外界認識装置1は、複眼領域画像CAにて撮影された任意のランドマーク100の実高さを活用して、単眼領域画像(例えば左単眼領域画像LA)における歩行者200と自車両Vとの間の距離を算出する。ランドマーク100は立体物であり、テクスチャ情報を有するため、路面に比べて安定的に視差を取得することができる。このため、夜間や雨天などで路面に対する視差を取得し難い状況であっても、より高精度に歩行者200までの距離を算出することができる。
In this way, the external world recognition device 1 of this embodiment utilizes the actual height of any landmark 100 photographed in the compound eye area image CA to perform walking in the monocular area image (for example, the left monocular area image LA). The distance between the person 200 and the own vehicle V is calculated. Since the landmark 100 is a three-dimensional object and has texture information, parallax can be obtained more stably than the road surface. Therefore, it is possible to calculate the distance to the pedestrian 200 with higher accuracy even in a situation where it is difficult to obtain the parallax with respect to the road surface, such as at night or in rainy weather.
また、本実施形態の外界認識装置1では、複眼領域画像CAのランドマーク100の実高さを計測し、その結果を利用して歩行者200の身長を推定する。歩行者200は、歩行時の手の動きにより自身の横方向の大きさが変化する。これに対し、歩行者200の身長は、当該歩行者200が歩行するときであっても、変動しない若しくは変動量が少ない。そのため、歩行者200の身長を推定し、推定した身長に基づき歩行者200と自車両Vとの間の距離を求めることで、歩行者200までの距離を高精度に求めることができる。また、路面に対する歩行者200の接地位置とカメラの取り付け姿勢とから、歩行者200と自車両Vとの間の距離を推定する手法を採用した場合、路面に対するカメラの取り付け姿勢を正確に推定する必要がある。これに対し、本実施形態のように、歩行者200の身長を推定して距離を求めることで、カメラの姿勢変化の影響を受けることなく、自車両Vと歩行者200との間の距離を推定することができる。
In addition, the external world recognition device 1 of this embodiment measures the actual height of the landmark 100 in the compound eye area image CA, and uses the result to estimate the height of the pedestrian 200 . Pedestrian 200 changes its size in the horizontal direction according to the movement of the hand during walking. On the other hand, the height of the pedestrian 200 does not change or fluctuates little even when the pedestrian 200 walks. Therefore, by estimating the height of pedestrian 200 and obtaining the distance between pedestrian 200 and own vehicle V based on the estimated height, the distance to pedestrian 200 can be obtained with high accuracy. Further, when a method of estimating the distance between the pedestrian 200 and the own vehicle V from the contact position of the pedestrian 200 on the road surface and the mounting posture of the camera on the road surface is adopted, the mounting posture of the camera on the road surface can be accurately estimated. There is a need. In contrast, by estimating the height of the pedestrian 200 and obtaining the distance as in the present embodiment, the distance between the own vehicle V and the pedestrian 200 can be calculated without being affected by changes in the posture of the camera. can be estimated.
<変形例1>
次いで、本発明の第1実施形態の変形例1について説明する。 <Modification 1>
Next, Modification 1 of the first embodiment of the present invention will be described.
次いで、本発明の第1実施形態の変形例1について説明する。 <Modification 1>
Next, Modification 1 of the first embodiment of the present invention will be described.
図7は、第1実施形態の変形例1に係る外界認識装置1dの概略構成を示す機能ブロック図である。図8は、図7の外界認識装置1dの距離推定処理を示すフローチャートである。図9は、図7の外界認識装置1dの近接判定処理を説明する図である。変形例1に係る外界認識装置1dは、上述の外界認識装置1に対して、後述する近接判定部56を含む点で異なる。以下、上述の外界認識装置1と同じ又は類似する機能を有する構成については、同一の符号を付してその説明を省略し、異なる部分について説明する。
FIG. 7 is a functional block diagram showing a schematic configuration of an external world recognition device 1d according to Modification 1 of the first embodiment. FIG. 8 is a flowchart showing distance estimation processing of the external world recognition device 1d of FIG. FIG. 9 is a diagram for explaining proximity determination processing of the external world recognition device 1d of FIG. The external world recognition device 1d according to Modification 1 differs from the external world recognition device 1 described above in that it includes a proximity determination unit 56, which will be described later. Hereinafter, configurations having the same or similar functions as those of the external world recognition device 1 described above are denoted by the same reference numerals, and descriptions thereof are omitted, and different portions are described.
図7に示すように、距離推定部50は、近接判定部56を含む。近接判定部56は、単眼領域ランドマーク検出部52で検出されたランドマーク100と単眼領域対象物検出部54で検出された歩行者200とが3次元的に近いか否かを判定する。具体的には、図9に示すように、近接判定部56は、左単眼領域画像LA(単眼領域画像)におけるランドマーク100の下端と左単眼領域画像LA(単眼領域画像)における歩行者200の下端との高さ方向における差を算出する。そして、大きさ推定部58は、近接判定部56により算出された上記差が後述するマージン(近接判定閾値)δ内にあり、ランドマーク100と歩行者200が近接していると判定された場合、歩行者200の身長を推定する。
As shown in FIG. 7, the distance estimation unit 50 includes a proximity determination unit 56. The proximity determination unit 56 determines whether or not the landmark 100 detected by the monocular area landmark detection unit 52 and the pedestrian 200 detected by the monocular area object detection unit 54 are three-dimensionally close. Specifically, as shown in FIG. 9, the proximity determination unit 56 determines whether the lower end of the landmark 100 in the left monocular area image LA (monocular area image) and the pedestrian 200 in the left monocular area image LA (monocular area image) Calculate the difference in the height direction from the bottom edge. If the difference calculated by the proximity determination unit 56 is within a margin (proximity determination threshold value) δ described later, and the size estimation unit 58 determines that the landmark 100 and the pedestrian 200 are close to each other, , to estimate the height of the pedestrian 200 .
図8に示すように、変形例1に係る外界認識装置1dは、距離推定処理において、歩行者検出処理(P113)の後、且つ、大きさ推定処理(P115)の前に、近接判定処理(P114)を実行する。近接判定処理(P114)において、近接判定部56は、単眼領域ランドマーク検出部52で検出されたランドマーク100と、単眼領域対象物検出部54で検出された歩行者200と、が空間的に近い位置に存在するか否かを判定する。例えば、近接判定処理(P114)において、近接判定部56はランドマーク100の下端を利用する。図9に示すように、近接判定部56は、ランドマーク100の下端を中心に、高さ方向における所定のマージンδ(例えば5pix)を設定し、当該マージンδ内に歩行者200の下端が含まれるかどうかを判定する。近接判定部56は、当該マージンδ内に歩行者200の下端が含まれる場合、ランドマーク100と歩行者200が自車両Vから見てほぼ等距離に存在していると判断し、両者が近接していると判定する。近接判定部56によりランドマーク100と歩行者200が近接すると判定された場合、大きさ推定処理(P115)において、大きさ推定部58は、ランドマーク情報記憶部36に記憶されたランドマーク100の実高さを利用して、歩行者200の身長を推定する。なお、上述したマージンδ(例えば5pix)は、ランドマーク情報記憶部36に記憶されてよい。
As shown in FIG. 8, the external world recognition device 1d according to Modification 1 performs proximity determination processing ( P114) is executed. In the proximity determination process (P114), the proximity determination unit 56 spatially determines the landmark 100 detected by the monocular area landmark detection unit 52 and the pedestrian 200 detected by the monocular area object detection unit 54. It is determined whether or not it exists at a close position. For example, in the proximity determination process (P114), the proximity determination unit 56 uses the lower end of the landmark 100. FIG. As shown in FIG. 9, the proximity determination unit 56 sets a predetermined margin δ (for example, 5 pix) in the height direction around the lower end of the landmark 100, and the lower end of the pedestrian 200 is included in the margin δ. determine whether or not When the lower end of the pedestrian 200 is included in the margin δ, the proximity determination unit 56 determines that the landmark 100 and the pedestrian 200 are at approximately the same distance from the host vehicle V, and determines that the two are close to each other. It is determined that When the proximity determination unit 56 determines that the landmark 100 and the pedestrian 200 are close to each other, in the size estimation process (P115), the size estimation unit 58 determines the size of the landmark 100 stored in the landmark information storage unit 36. The height of the pedestrian 200 is estimated using the actual height. Note that the above margin δ (eg, 5 pix) may be stored in the landmark information storage unit 36 .
本変形例1の外界認識装置1dでは、近接判定部56が、ランドマーク100を歩行者200の身長推定に利用するか否かを判定する。歩行者200の身長を推定する際、歩行者200とランドマーク100の画像上のサイズ(即ち画像高さ)の比率を利用する。しかし、歩行者200とランドマーク100が離れた位置に存在する場合、両者の画像高さの比率が両者の実高さの比率から乖離することがあり、正確に歩行者200の身長を推定できないことがある。そのため、本変形例1のように、近接判定部56が近接判定処理(P114)を行うことで、より正確に歩行者200の身長を推定することができる。
In the external world recognition device 1d of Modification 1, the proximity determination unit 56 determines whether or not the landmark 100 is used for estimating the height of the pedestrian 200. When estimating the height of the pedestrian 200, the ratio of the size (ie, image height) of the pedestrian 200 and the landmark 100 on the image is used. However, when the pedestrian 200 and the landmark 100 exist at distant positions, the ratio of the image heights of the two may deviate from the ratio of the actual heights of the two, and the height of the pedestrian 200 cannot be accurately estimated. Sometimes. Therefore, the height of the pedestrian 200 can be estimated more accurately by the proximity determination unit 56 performing the proximity determination process (P114) as in the first modification.
次いで、変形例1に係る外界認識装置1dの別の例を説明する。図10は、図7の外界認識装置1dの近接判定処理(P114)に係る別の例を示した図である。図10では、上段部分にランドマーク登録処理(P105)が記載され、下段部分に近接判定処理(P114)が記載されている。図10に示す例では、上述の変形例1に係る外界認識装置1dに対し、ランドマーク登録処理(P105)、近接判定処理(P114)の点で異なる。図10に示す例では、変形例1に係る外界認識装置1d(図7)と同一の符号を付してその説明をする。
Next, another example of the external world recognition device 1d according to Modification 1 will be described. FIG. 10 is a diagram showing another example of the proximity determination process (P114) of the external world recognition device 1d of FIG. In FIG. 10, the landmark registration process (P105) is described in the upper part, and the proximity determination process (P114) is described in the lower part. The example shown in FIG. 10 differs from the external world recognition device 1d according to Modification 1 described above in terms of landmark registration processing (P105) and proximity determination processing (P114). In the example shown in FIG. 10, the same reference numerals as those of the external world recognition device 1d (FIG. 7) according to Modification 1 are attached and the description thereof will be made.
図10に示す例では、近接判定部56は、近接判定処理(P114)を、複眼領域ランドマーク検出部32で検出されたランドマーク100の周辺の視差に基づき実行する。例えば、複眼領域ランドマーク検出部32は、図10の上段部分に示すように、検出されたランドマーク100が接地する面の周辺の視差情報から、ランドマーク100が接地している面の傾きの変化を計算してもよい。これにより、ランドマーク100が接地した面の傾きが変化する箇所が特定される。その後、大きさ情報計測部33は、複眼領域ランドマーク検出部32で検出されたランドマーク100の画像高さ(Obj_Height)と、ランドマーク100の中心位置から傾きが変化する箇所までの画像上での画像長さ(Plane_Distance)との比率Plane_rateを、「Plane_rate=(Plane_Distance/Obj_Height)」(式(4))により計算する。そして、ランドマーク情報記憶部36は、例えば上記所定のマージンδに加え、Plane_rateの値を近接判定半径として記憶する(ランドマーク登録処理(P105))。
In the example shown in FIG. 10, the proximity determination unit 56 executes the proximity determination process (P114) based on the parallax around the landmark 100 detected by the compound eye region landmark detection unit 32. For example, as shown in the upper part of FIG. 10, the compound eye area landmark detection unit 32 detects the inclination of the surface on which the landmark 100 is on the ground based on the parallax information around the surface on which the detected landmark 100 is on the ground. change may be calculated. As a result, the location where the inclination of the surface on which the landmark 100 is grounded is identified. After that, the size information measurement unit 33 measures the image height (Obj_Height) of the landmark 100 detected by the compound eye area landmark detection unit 32 and the image from the center position of the landmark 100 to the point where the inclination changes. The ratio Plane_rate to the image length (Plane_Distance) of is calculated by "Plane_rate=(Plane_Distance/Obj_Height)" (equation (4)). Then, the landmark information storage unit 36 stores, for example, the value of Plane_rate as the proximity determination radius in addition to the predetermined margin δ (landmark registration process (P105)).
図10の下段部分に示すように、近接判定部56は、近接判定処理(P114)において、単眼領域ランドマーク検出部52で再検出されたランドマーク100の画像高さ(Obj_Height2)を利用し、マージンδ2(Plane_rate×Obj_Height2)を計算する。そして、近接判定部56は、縦方向のマージンδ(pix)に加え、横方向にδ2(pix)以内に歩行者200が近づいた場合に、ランドマーク100と歩行者200が近接していると判定する。
As shown in the lower part of FIG. 10, the proximity determination unit 56 uses the image height (Obj_Height2) of the landmark 100 re-detected by the monocular area landmark detection unit 52 in the proximity determination process (P114), Calculate the margin δ2 (Plane_rate×Obj_Height2). Then, the proximity determination unit 56 determines that the pedestrian 200 is close to the landmark 100 when the pedestrian 200 approaches within δ2 (pix) in the horizontal direction in addition to the margin δ (pix) in the vertical direction. judge.
大きさ推定処理(P115)では、歩行者200とランドマーク100が自車両Vから等しい距離に存在すると仮定して歩行者200の身長を推定する。しかし、ランドマーク100が接地する面の傾きが途中で変化していた場合、近接判定処理(P114)において縦方向のマージンδだけを参照すると、歩行者200とランドマーク100が等しい距離にいないにも関わらず、ランドマーク100と歩行者200が近接していると判定されるおそれがある。そのため、視差情報に基づいて路面の傾きが変化していない箇所までの領域(図10の下段部分におけるδ2)を算出し、この結果に基づいて近接判定処理(P114)を実施することにより、両者が等しい距離に存在しているか正しく判定できる。よって、歩行者200の身長をより正確に推定することができる。
In the size estimation process (P115), the height of the pedestrian 200 is estimated on the assumption that the pedestrian 200 and the landmark 100 are at the same distance from the own vehicle V. However, if the inclination of the surface where the landmark 100 touches the ground changes in the middle, if only the vertical margin δ is referred to in the proximity determination process (P114), the pedestrian 200 and the landmark 100 are not at the same distance. In spite of this, it may be determined that the landmark 100 and the pedestrian 200 are close to each other. Therefore, based on the parallax information, the area (δ2 in the lower part of FIG. 10) up to the point where the inclination of the road surface does not change is calculated, and based on this result, the proximity determination process (P114) is performed, so that both are at equal distances or not. Therefore, the height of pedestrian 200 can be estimated more accurately.
<変形例2>
次いで、本発明の第1実施形態の変形例2について説明する。 <Modification 2>
Next, Modification 2 of the first embodiment of the present invention will be described.
次いで、本発明の第1実施形態の変形例2について説明する。 <Modification 2>
Next, Modification 2 of the first embodiment of the present invention will be described.
変形例2に係る外界認識装置1eは、上述の外界認識装置1に対して、後述するランドマーク情報計測部34を含む点で異なる。以下、上述の外界認識装置1と同じ又は類似する機能を有する構成については、同一の符号を付してその説明を省略し、異なる部分について説明する。
The external world recognition device 1e according to Modification 2 differs from the external world recognition device 1 described above in that it includes a landmark information measurement unit 34, which will be described later. Hereinafter, configurations having the same or similar functions as those of the external world recognition device 1 described above are denoted by the same reference numerals, and descriptions thereof are omitted, and different portions are described.
図11は、第1実施形態の変形例2に係る外界認識装置1eの概略構成を示す機能ブロック図である。図11に示すように、ランドマーク情報取得部30は、ランドマーク情報計測部34を含む。
FIG. 11 is a functional block diagram showing a schematic configuration of an external world recognition device 1e according to modification 2 of the first embodiment. As shown in FIG. 11 , the landmark information acquisition section 30 includes a landmark information measurement section 34 .
ランドマーク情報計測部34は、複眼領域ランドマーク検出部32によって検出されたランドマーク100、101、102に対し、複眼領域画像CAから当該ランドマーク100、101、102が移動しているか否かを判定する。そして、ランドマーク情報記憶部36は、複眼領域画像CAに含まれるランドマーク100、101、102のうち、ランドマーク情報計測部34により移動していると判定されたランドマーク(例えば図4のランドマーク101)を、歩行者200までの距離を推定するランドマークとして記憶する。
The landmark information measurement unit 34 determines whether the landmarks 100, 101, and 102 detected by the compound eye area landmark detection unit 32 have moved from the compound eye area image CA. judge. Then, the landmark information storage unit 36 stores the landmarks determined to be moving by the landmark information measurement unit 34 among the landmarks 100, 101, and 102 included in the compound eye area image CA (for example, the landmarks in FIG. The mark 101) is stored as a landmark for estimating the distance to the pedestrian 200. FIG.
変形例2に係る外界認識装置1eでは、静止した立体物をランドマークとして登録するだけではなく、移動体として、例えばその他の歩行者や車両などの立体物をランドマークとして利用することができる。図4に示すランドマーク100は静止物であるため、自車両Vに対する距離算出の対象である歩行者200が静止していると、両者が近づくことが無い。このため、静止物であるランドマーク100に基づいて、静止している歩行者200の身長を推定することはできない。これに対し、本変形例2のように、移動体(例えばランドマーク101)を利用することで、距離算出の対象である歩行者200が静止していた場合においても、ランドマーク101が歩行者200に近づくことがある。よって、静止した歩行者200に対しても身長を推定することができるようになる。
In the external world recognition device 1e according to Modification 2, not only can stationary three-dimensional objects be registered as landmarks, but also three-dimensional objects such as other pedestrians and vehicles can be used as landmarks as mobile objects. Since the landmark 100 shown in FIG. 4 is a stationary object, if the pedestrian 200, who is the target of distance calculation with respect to the own vehicle V, is stationary, the two do not approach each other. Therefore, the height of stationary pedestrian 200 cannot be estimated based on landmark 100, which is a stationary object. On the other hand, by using a moving object (for example, the landmark 101) as in Modification 2, even when the pedestrian 200, which is the object of distance calculation, is stationary, the landmark 101 is the pedestrian. It can get close to 200. Therefore, it becomes possible to estimate the height of the stationary pedestrian 200 as well.
<変形例3>
次いで、本発明の第1実施形態の変形例3について説明する。 <Modification 3>
Next, Modification 3 of the first embodiment of the present invention will be described.
次いで、本発明の第1実施形態の変形例3について説明する。 <Modification 3>
Next, Modification 3 of the first embodiment of the present invention will be described.
変形例3に係る外界認識装置は、上述の外界認識装置に対して、ランドマーク情報記憶部36の機能の点で異なる。以下、変形例3に係る外界認識装置1を、図1から図6を参照しつつ、上述の外界認識装置1と同一の符号を付して説明する。
The external world recognition device according to Modification 3 differs from the external world recognition device described above in terms of the function of the landmark information storage unit 36 . Hereinafter, the external world recognition device 1 according to Modification 3 will be described with the same reference numerals as the external world recognition device 1 described above, with reference to FIGS.
変形例3において、図1に示すランドマーク情報記憶部36は、大きさ情報計測部33で計測の対象とされたランドマーク100、101、102のうち、実高さが高さ閾値(例えば1m)以上のランドマーク(例えば、図4のランドマーク100、102)を、歩行者200までの距離を推定するランドマークとして記憶する。このように、変形例3では、3次元情報取得処理(P104)で計測したランドマーク100、101、102のうち、実高さが所定の高さ閾値以上(例えば1m以上)のランドマーク(例えば、図4のランドマーク100、102)のみを、ランドマーク登録処理(P105)でランドマーク情報記憶部36に登録する。そして、ランドマーク検出処理(P112)では、ランドマーク登録処理(P105)で登録した、高さ閾値以上の実高さを有するランドマーク(例えば、図4のランドマーク100、102)を再検出する。
In modification 3, the landmark information storage unit 36 shown in FIG. ) above (for example, the landmarks 100 and 102 in FIG. 4) are stored as landmarks for estimating the distance to the pedestrian 200 . Thus, in Modification 3, among the landmarks 100, 101, and 102 measured in the three-dimensional information acquisition process (P104), the actual height of the landmark (eg , 100 and 102 in FIG. 4) are registered in the landmark information storage unit 36 in the landmark registration process (P105). Then, in the landmark detection process (P112), the landmarks registered in the landmark registration process (P105) and having an actual height equal to or greater than the height threshold (for example, the landmarks 100 and 102 in FIG. 4) are detected again. .
ランドマーク検出処理(P112)において実高さの低いランドマークを再検出する際にテンプレートマッチングなどの処理を用いると、検知位置のズレが発生してしまうことがある。検知位置のズレが発生した場合、単眼領域ランドマーク検出部52により検出したランドマークの画像高さが、当該ランドマークの真の画像高さから変化してしまい、大きさ推定処理(P115)で推定する歩行者200の身長にズレが生じることがある。この点で、変形例3に係る外界認識装置では、所定の高さ閾値(例えば1m)以上のランドマークのみを利用する。このため、真の画像高さに対する、検出されたランドマーク画像高さの乖離を小さくすることができ、より正確に歩行者の身長を推定できる。
If a process such as template matching is used when re-detecting a landmark with a low actual height in the landmark detection process (P112), the detected position may be misaligned. When the detection position is misaligned, the image height of the landmark detected by the monocular area landmark detection unit 52 changes from the true image height of the landmark. The estimated height of pedestrian 200 may be deviated. In this regard, the external world recognition device according to Modification 3 uses only landmarks having a height equal to or greater than a predetermined height threshold (for example, 1 m). Therefore, the difference between the height of the detected landmark image and the true height of the image can be reduced, and the height of the pedestrian can be estimated more accurately.
変形例3の別の例として、ランドマーク情報記憶部36は、大きさ情報計測部33で計測の対象となった複数のランドマーク100、101、102のうち、実高さが高い方から所定の数(例えば複数のランドマークのうち60%に相当する数)のランドマーク(例えばランドマーク100、102)を、歩行者200までの距離を推定するランドマークとして記憶してもよい。これにより、歩行者200までの距離推定に使用するランドマークの選択肢を多く保持することができる。
As another example of Modified Example 3, the landmark information storage unit 36 selects the plurality of landmarks 100, 101, and 102 measured by the size information measuring unit 33 in descending order of actual height. (for example, the number corresponding to 60% of the plurality of landmarks) (for example, the landmarks 100 and 102) may be stored as landmarks for estimating the distance to the pedestrian 200. FIG. As a result, many options of landmarks used for distance estimation to the pedestrian 200 can be held.
<変形例4>
次いで、本発明の第1実施形態の変形例4について説明する。 <Modification 4>
Next, Modification 4 of the first embodiment of the present invention will be described.
次いで、本発明の第1実施形態の変形例4について説明する。 <Modification 4>
Next, Modification 4 of the first embodiment of the present invention will be described.
変形例4に係る外界認識装置は、変形例2の外界認識装置1e(図11)に対して、ランドマーク情報計測部34及びランドマーク情報記憶部36の機能の点で異なる。以下、変形例4に係る外界認識装置を、図3から図6及び図11を参照しつつ、上述の外界認識装置1eと同一の符号を付して説明する。
The external world recognition device according to Modified Example 4 differs from the external world recognition device 1e (FIG. 11) according to Modified Example 2 in terms of the functions of the landmark information measurement unit 34 and the landmark information storage unit 36. Hereinafter, the external world recognition device according to Modified Example 4 will be described with reference to FIGS.
変形例4において、ランドマーク情報計測部34は、複眼領域ランドマーク検出部32によって検出されたランドマーク100、101、102に対し、複眼領域画像CAに基づき、ランドマーク100のコントラスト情報(エッジ成分、色、輝度差等)を算出する。そして、ランドマーク情報記憶部36は、複眼領域画像CAに含まれるランドマーク100、101、102のうち、所定の閾値以上のコントラスト情報を有するランドマーク(例えば、図4のランドマーク100)を、歩行者200までの距離を推定するランドマークとして記憶する。
In Modified Example 4, the landmark information measurement unit 34 calculates the contrast information (edge component , color, luminance difference, etc.). Then, of the landmarks 100, 101, and 102 included in the compound eye area image CA, the landmark information storage unit 36 selects a landmark having contrast information equal to or greater than a predetermined threshold value (for example, the landmark 100 in FIG. 4), It is stored as a landmark for estimating the distance to pedestrian 200 .
具体的には、変形例4において、ランドマーク情報計測部34は、ランドマーク登録処理(P105)において、各ランドマーク100、101、102に対して、ソベルフィルタを適用することでエッジ成分(コントラスト情報)を抽出する。そして、当該処理において、ランドマーク情報記憶部36は、ランドマーク100、101、102の領域内において、エッジ成分の密度(エッジの数)が所定の値以上のランドマーク(例えばランドマーク100)を登録する。もしくは、色情報(コントラスト情報)が所定の値以上のランドマーク(例えばランドマーク100)が登録されてもよい。具体的には、画像の赤成分とみなすことができる色相領域に対して、彩度と明度が所定値以上の画素をカウントし、画素数が所定以上であった場合にランドマークとして登録してもよい。これにより、ランドマーク検出処理(P112)において、ランドマーク100をより安定して再検出することができ、他の立体物を誤ってランドマークとして検出することを回避できる。これにより、より精度高く歩行者200の身長を推定できる。
Specifically, in Modified Example 4, the landmark information measurement unit 34 applies a Sobel filter to each of the landmarks 100, 101, and 102 in the landmark registration process (P105) to obtain edge components (contrast information). In this process, the landmark information storage unit 36 selects landmarks (for example, the landmark 100) whose density of edge components (the number of edges) is equal to or greater than a predetermined value within the areas of the landmarks 100, 101, and 102. register. Alternatively, a landmark (for example, landmark 100) whose color information (contrast information) is equal to or greater than a predetermined value may be registered. Specifically, pixels with saturation and brightness equal to or greater than a predetermined value are counted in a hue region that can be regarded as the red component of an image, and if the number of pixels is equal to or greater than a predetermined value, it is registered as a landmark. good too. As a result, in the landmark detection process (P112), the landmark 100 can be re-detected more stably, and erroneous detection of other three-dimensional objects as landmarks can be avoided. Thereby, the height of pedestrian 200 can be estimated with higher accuracy.
また、ランドマーク情報計測部34は、ランドマーク登録処理(P105)において、ランドマーク100、101、102の下端と路面の輝度差を解析する。そして、当該処理において、ランドマーク情報記憶部36は、ランドマーク100、101、102のうち、これらの下端と路面の輝度差が所定値以上のピクセル数が一定値以上であるランドマーク(例えばランドマーク100)を記憶する。これにより、複眼領域ランドマーク検出部32で検出された領域と同一箇所を、単眼領域ランドマーク検出部52において再検出することが容易になり、より高精度に歩行者200の身長を推定することができる。また、変形例4に係る外界認識装置が近接判定部56を含む場合、近接判定処理(P114)でランドマーク100の下端を利用する際に路面との分離性が高いランドマーク100を登録することで、より正確に近接判定を実施することができる。
Also, in the landmark registration process (P105), the landmark information measurement unit 34 analyzes the luminance difference between the lower ends of the landmarks 100, 101, and 102 and the road surface. In this process, the landmark information storage unit 36 stores landmarks (for example, landmarks) in which the number of pixels whose brightness difference between the lower end and the road surface is equal to or greater than a predetermined value among the landmarks 100, 101, and 102 is equal to or greater than a predetermined value. store the mark 100). This makes it easier for the monocular area landmark detection unit 52 to re-detect the same area as the area detected by the compound eye area landmark detection unit 32, thereby estimating the height of the pedestrian 200 with higher accuracy. can be done. Further, when the external world recognition device according to the fourth modification includes the proximity determination unit 56, when using the lower end of the landmark 100 in the proximity determination process (P114), it is possible to register the landmark 100 that is highly separable from the road surface. , the proximity determination can be performed more accurately.
また、ランドマーク情報記憶部36は、ランドマーク登録処理(P105)において、ランドマークにおける特定の部位を登録することができる。具体的には、ランドマーク情報計測部34は、例えばランドマーク100においてエッジ成分が強い部分をランドマークの上端として計測する。そして、当該処理において、ランドマーク情報記憶部36は、当該部分をランドマークの上端として登録することができる。これにより、複眼領域ランドマーク検出部32で検出された部分と同一箇所を、単眼領域ランドマーク検出部52において再検出することが容易になる。このように、エッジ成分の強い部分に基づいて、例えばランドマーク100の特定の部分を登録することにより、単眼領域ランドマーク検出部52にてランドマークを安定して再検出することができ、より高精度で歩行者の距離を推定することができる。
Also, the landmark information storage unit 36 can register a specific part of the landmark in the landmark registration process (P105). Specifically, the landmark information measuring unit 34 measures, for example, a portion of the landmark 100 where the edge component is strong as the upper end of the landmark. Then, in the processing, the landmark information storage unit 36 can register the portion as the upper end of the landmark. This makes it easier for the monocular area landmark detection section 52 to re-detect the same portion as the portion detected by the compound eye area landmark detection section 32 . In this way, by registering, for example, a specific portion of the landmark 100 based on a portion with a strong edge component, the landmark can be stably re-detected by the monocular area landmark detection unit 52. Pedestrian distance can be estimated with high accuracy.
<変形例5>
次いで、本発明の第1実施形態の変形例5について説明する。 <Modification 5>
Next, Modification 5 of the first embodiment of the present invention will be described.
次いで、本発明の第1実施形態の変形例5について説明する。 <Modification 5>
Next, Modification 5 of the first embodiment of the present invention will be described.
変形例5に係る外界認識装置は、上述の外界認識装置1に対して、大きさ推定部58の機能の点で異なる。以下、変形例5に係る外界認識装置を、図1から図10を参照しつつ、上述の外界認識装置1と同一の符号を付して説明する。
The external world recognition device according to modification 5 differs from the external world recognition device 1 described above in terms of the function of the size estimation unit 58 . Hereinafter, the external world recognition device according to Modification 5 will be described with the same reference numerals as the external world recognition device 1 described above, with reference to FIGS.
変形例5に係る外界認識装置では、ランドマーク情報記憶部36に複数のランドマーク100、101、102が登録された場合、大きさ推定部58は、複数のランドマークを利用して歩行者200のサイズを推定する。具体的には、ある画像フレームにおいて、ランドマーク100と近接した際に、大きさ推定部58は、歩行者200の身長(Height_Ped)を推定する。そして、距離算出部60は、当該Height_Pedに基づき、自車両Vと歩行者200との間の距離を算出する。その後、別の画像フレームにおいて、歩行者200がランドマーク101と近接した際に、大きさ推定部58は、「Height_Ped=(Height_Ped+Height_Ped_2)/2」(式(5))により、歩行者の身長Height_Pedを更新する。ここで、Height_Ped_2は、ランドマーク101の実高さと、ランドマーク101画像高さ及び歩行者200の画像高さから推定した歩行者200の身長である。大きさ推定部58は、大きさ推定処理(P115)において、複数のランドマークと近接する毎に歩行者200の身長を更新する。これにより、初期の身長推定結果が誤りを含む場合においても、各ランドマークと近接する毎により正しい値に近づくように身長を更新することができ、より正確に歩行者の距離を推定することができる。
In the external world recognition device according to Modification 5, when a plurality of landmarks 100, 101, and 102 are registered in the landmark information storage unit 36, the size estimation unit 58 uses the plurality of landmarks to estimate the pedestrian 200. Estimate the size of Specifically, the size estimation unit 58 estimates the height (Height_Ped) of the pedestrian 200 when approaching the landmark 100 in a certain image frame. Then, the distance calculation unit 60 calculates the distance between the vehicle V and the pedestrian 200 based on the Height_Ped. After that, in another image frame, when the pedestrian 200 approaches the landmark 101, the size estimator 58 calculates the pedestrian's Update height Height_Ped. Here, Height_Ped_2 is the height of the pedestrian 200 estimated from the actual height of the landmark 101, the image height of the landmark 101, and the height of the pedestrian 200 image. In the size estimation process (P115), size estimation unit 58 updates the height of pedestrian 200 each time pedestrian 200 approaches a plurality of landmarks. As a result, even if the initial height estimation result contains an error, the height can be updated so as to approach the correct value each time it approaches each landmark, and the distance of the pedestrian can be estimated more accurately. can.
<第2実施形態>
次いで、本発明の第2実施形態について説明する。 <Second embodiment>
Next, a second embodiment of the invention will be described.
次いで、本発明の第2実施形態について説明する。 <Second embodiment>
Next, a second embodiment of the invention will be described.
図12は、本発明の第2実施形態に係る外界認識装置1aの概略構成を示す機能ブロック図である。第2実施形態に係る外界認識装置1aは、上述の外界認識装置1d(変形例1)に対して、後述するランドマーク選択部40及び進行路推定部70を含む点で異なる。上述の外界認識装置1dと同じ又は類似する機能を有する構成については、当該外界認識装置1dと同一の符号を付してその説明を省略し、図2から図6及び図12を参照して、異なる部分について説明する。
FIG. 12 is a functional block diagram showing a schematic configuration of the external world recognition device 1a according to the second embodiment of the present invention. The external world recognition device 1a according to the second embodiment differs from the external world recognition device 1d (Modification 1) in that it includes a landmark selection unit 40 and a course estimation unit 70, which will be described later. Configurations having the same or similar functions as those of the external world recognition device 1d described above are denoted by the same reference numerals as those of the external world recognition device 1d, and descriptions thereof are omitted. Different parts will be explained.
図12に示すように、外界認識装置1aは、ランドマーク選択部40及び進行路推定部70を含む。ランドマーク選択部40はランドマーク情報取得部30に含まれる。第2実施形態に係る外界認識装置1aのランドマーク情報取得部30は、複数のランドマーク100、101、102の3次元情報を取得した場合、車両Vの動きに応じて、複数のランドマークの中から距離の推定に用いるランドマーク(例えばランドマーク100)を選択する。具体的には、ランドマーク情報取得部30のランドマーク選択部40が、複数のランドマークの中から距離の推定に用いる上記ランドマーク(例えばランドマーク100)を選択する。より具体的には、ランドマーク選択部40は、後述する進行路推定部70により推定された車両Vの進行方向に存在するランドマーク100を、歩行者200までの距離の推定に用いるランドマークとして選択する。そして、ランドマーク情報記憶部36は、ランドマーク選択部40により選択されたランドマーク100を記憶する。
As shown in FIG. 12, the external world recognition device 1a includes a landmark selection unit 40 and a traveling route estimation unit 70. Landmark selection unit 40 is included in landmark information acquisition unit 30 . When the landmark information acquisition unit 30 of the external world recognition device 1a according to the second embodiment acquires the three-dimensional information of the plurality of landmarks 100, 101, and 102, the landmark information acquisition unit 30 determines the location of the plurality of landmarks according to the movement of the vehicle V. Select a landmark (eg, landmark 100) to be used for distance estimation. Specifically, the landmark selection unit 40 of the landmark information acquisition unit 30 selects the landmark (for example, the landmark 100) used for distance estimation from a plurality of landmarks. More specifically, the landmark selection unit 40 selects a landmark 100 existing in the traveling direction of the vehicle V estimated by the traveling path estimation unit 70 described later as a landmark used for estimating the distance to the pedestrian 200. select. The landmark information storage section 36 stores the landmark 100 selected by the landmark selection section 40 .
進行路推定部70は、車両Vの動き(即ち挙動情報)から車両Vの進行方向を推定する。具体的には、進行路推定部70は、車両Vの速度やヨーレートなどに基づき自車両Vの進行路を予測する。例えば進行路推定部70は、車速とヨーレートの値から車両Vの回転半径を推定して、その円周上の軌跡を進行路とする。なお、進行路推定部70は、車両Vが直進している場合、その直進方向に沿う軌跡を進行路と推定してもよい。
The traveling path estimation unit 70 estimates the traveling direction of the vehicle V from the movement of the vehicle V (that is, behavior information). Specifically, the traveling route estimation unit 70 predicts the traveling route of the own vehicle V based on the speed of the vehicle V, the yaw rate, and the like. For example, the traveling path estimator 70 estimates the turning radius of the vehicle V from the values of the vehicle speed and the yaw rate, and sets the locus on the circumference as the traveling path. Note that, when the vehicle V is traveling straight ahead, the traveling path estimation unit 70 may estimate a trajectory along the straight traveling direction as the traveling path.
このように、第2実施形態に係る外界認識装置1aでは、ランドマーク登録処理(P105)において、進行路推定部70が推定した進行方向に存在するランドマーク(例えばランドマーク100)だけが、ランドマーク情報記憶部36に登録される。例えば、車両Vの進行路との距離が所定値(20m)以内に存在するランドマーク100が、ランドマーク情報記憶部36に記憶されてよい。このように、車両Vの進行路情報に基づき、登録されるランドマークを選択することで、車両Vと衝突するリスクのある歩行者200との距離推定に利用するランドマークのみを登録することができる。これにより、登録するランドマークを削減することでメモリの使用量や、ランドマーク検出処理(P112)で検出するランドマークの数を制限することができ、外界認識装置1aにおける処理負荷を低減することができる。
As described above, in the external world recognition device 1a according to the second embodiment, in the landmark registration process (P105), only the landmark (for example, the landmark 100) existing in the traveling direction estimated by the traveling path estimation unit 70 is registered as a landmark. Registered in the mark information storage unit 36 . For example, the landmark information storage unit 36 may store landmarks 100 that are within a predetermined distance (20 m) from the course of the vehicle V. FIG. By selecting the landmarks to be registered based on the traveling route information of the vehicle V in this manner, it is possible to register only the landmarks used for estimating the distance between the pedestrian 200 who is at risk of colliding with the vehicle V. can. As a result, by reducing the number of registered landmarks, it is possible to limit the amount of memory used and the number of landmarks detected in the landmark detection process (P112), thereby reducing the processing load on the external world recognition device 1a. can be done.
<第3実施形態>
次いで、本発明の第3実施形態について説明する。 <Third Embodiment>
Next, a third embodiment of the invention will be described.
次いで、本発明の第3実施形態について説明する。 <Third Embodiment>
Next, a third embodiment of the invention will be described.
図13は、本発明の第3実施形態に係る外界認識装置1bの概略構成を示す機能ブロック図である。第3実施形態に係る外界認識装置1bは、上述の外界認識装置1d(変形例1)に対して、後述するランドマーク選択部40及び周辺環境認識部90を含む点で異なる。上述の外界認識装置1dと同じ又は類似する機能を有する構成については、当該外界認識装置1dと同一の符号を付してその説明を省略し、図2から図6及び図13を参照して、異なる部分について説明する。
FIG. 13 is a functional block diagram showing a schematic configuration of an external world recognition device 1b according to the third embodiment of the present invention. The external world recognition device 1b according to the third embodiment differs from the external world recognition device 1d (Modification 1) in that it includes a landmark selection unit 40 and a surrounding environment recognition unit 90, which will be described later. Configurations having the same or similar functions as those of the external world recognition device 1d described above are denoted by the same reference numerals as those of the external world recognition device 1d, and descriptions thereof are omitted. Different parts will be explained.
図13に示すように、外界認識装置1bは、ランドマーク選択部40及び車両Vの周辺環境を認識する周辺環境認識部90を含む。ランドマーク選択部40はランドマーク情報取得部30に含まれる。第3実施形態に係る外界認識装置1bのランドマーク情報取得部30は、周辺環境認識部90により歩道であると認識された領域に存在するランドマーク(例えばランドマーク100)を、歩行者200までの距離の推定に用いるランドマークとして選択する。具体的には、ランドマーク情報取得部30のランドマーク選択部40が、歩行者200までの距離の推定に用いる上記ランドマーク(例えばランドマーク100)を選択する。そして、ランドマーク情報記憶部36は、ランドマーク選択部40により選択されたランドマークを記憶する。
As shown in FIG. 13, the external world recognition device 1b includes a landmark selection unit 40 and a surrounding environment recognition unit 90 that recognizes the surrounding environment of the vehicle V. Landmark selection unit 40 is included in landmark information acquisition unit 30 . The landmark information acquisition unit 30 of the external world recognition device 1b according to the third embodiment recognizes the landmarks (for example, the landmark 100) existing in the area recognized as the sidewalk by the surrounding environment recognition unit 90 up to the pedestrian 200. as landmarks for distance estimation. Specifically, the landmark selection unit 40 of the landmark information acquisition unit 30 selects the landmark (for example, the landmark 100) used for estimating the distance to the pedestrian 200. FIG. The landmark information storage unit 36 stores the landmarks selected by the landmark selection unit 40 .
周辺環境認識部90は、画像取得部10が取得した一対のカメラ10a、10bの画像の各ピクセルに対して種別情報を推定する。ここで、種別情報は、車両や歩行者などの物体だけでなく、路面、歩道などの情報を含む。各ピクセルに対する種別情報の推定には畳み込みニューラルネットワークを利用する。周辺環境認識部90は、各ピクセルに対する種別情報が付与された正解値データにより学習されたモデルを利用し、各ピクセルに対応した種別情報を推定する。これにより、周辺環境認識部90は、一対のカメラ10a、10bの画像から歩道である領域を認識することができる。
The surrounding environment recognition unit 90 estimates type information for each pixel of the images of the pair of cameras 10 a and 10 b acquired by the image acquisition unit 10 . Here, the type information includes not only information on objects such as vehicles and pedestrians, but also information on road surfaces, sidewalks, and the like. A convolutional neural network is used to estimate the type information for each pixel. The surrounding environment recognition unit 90 estimates the type information corresponding to each pixel by using a model learned from the correct value data to which the type information for each pixel is added. As a result, the surrounding environment recognition unit 90 can recognize the sidewalk area from the images captured by the pair of cameras 10a and 10b.
このように、第3実施形態に係る外界認識装置1bでは、ランドマーク登録処理(P105)において、周辺環境認識部90が推定した種別情報を利用してランドマークの登録処理を実行する。具体的には、ランドマーク情報記憶部36は、周辺環境認識部90が歩道だと推定したピクセル上に存在するランドマーク100を記憶する。これにより、歩行者200と近接する可能性が高いランドマーク100だけを登録することができ、外界認識装置1bにおける処理負荷を低減することができる。
Thus, in the external world recognition device 1b according to the third embodiment, in the landmark registration process (P105), the landmark registration process is executed using the type information estimated by the surrounding environment recognition unit 90. Specifically, the landmark information storage unit 36 stores the landmark 100 that exists on the pixel that the surrounding environment recognition unit 90 has estimated as the sidewalk. As a result, only landmarks 100 that are likely to be close to pedestrian 200 can be registered, and the processing load on external world recognition device 1b can be reduced.
上述した本実施形態では、ランドマーク情報取得部30においてランドマーク100、101、102の3次元情報(大きさ情報)を取得するために、一対のカメラ10a、10bを1台のステレオカメラとして用いている。しかし、ランドマークの3次元情報(大きさ)を取得するために、3次元センサ(例えば、Lidar、ミリ波レーダ、超音波センサ等)を用いてもよい。この場合、一対のカメラ10a、10bは、ステレオ領域を有さないように設置されてもよい。例えば、カメラ10aは左単眼領域のみを撮影し、カメラ10bは右単眼領域のみを撮影し、これらの領域で撮影された画像が、画像取得部10に入力されてもよい。
In the above-described embodiment, the pair of cameras 10a and 10b are used as one stereo camera in order to acquire the three-dimensional information (size information) of the landmarks 100, 101, and 102 in the landmark information acquisition unit 30. ing. However, a three-dimensional sensor (for example, lidar, millimeter wave radar, ultrasonic sensor, etc.) may be used to acquire the three-dimensional information (size) of the landmark. In this case, the pair of cameras 10a and 10b may be installed so as not to have a stereo area. For example, the camera 10 a may capture only the left monocular region, the camera 10 b may capture only the right monocular region, and the images captured in these regions may be input to the image acquisition unit 10 .
また、デジタルマップなどに記載された特定のランドマークを利用し、デジタルマップにおいて登録されている3次元情報を利用して、自車両Vと歩行者200との間の距離を推定してもよい。また、上記実施形態では、歩行者200の身長を推定するという観点に対して詳細に説明したが、歩行者200の幅に基づき、自車両Vと歩行者200との間の距離を算出してもよい。例えば、距離推定部50は、ランドマーク100の実幅と、画像内におけるランドマーク100および歩行者200の横方向長さ(即ち画像幅)と、に基づき、歩行者200までの距離を推定してもよい。また、対象物として歩行者200を一例として、歩行者200までの距離を推定する方法に関して述べたが、車両、動物など様々な物体を距離推定の対象物として扱うことができ、車両、動物などの高さ等に基づき、これらと自車両Vとの間の距離を推定することができる。
Also, the distance between the vehicle V and the pedestrian 200 may be estimated using specific landmarks described on a digital map or the like and using three-dimensional information registered in the digital map. . Further, in the above-described embodiment, a detailed description was given from the viewpoint of estimating the height of pedestrian 200, but based on the width of pedestrian 200, the distance between own vehicle V and pedestrian 200 is calculated. good too. For example, the distance estimating unit 50 estimates the distance to the pedestrian 200 based on the actual width of the landmark 100 and the lateral length (i.e. image width) of the landmark 100 and the pedestrian 200 in the image. may In addition, the method of estimating the distance to the pedestrian 200 has been described by taking the pedestrian 200 as an example of the object, but various objects such as vehicles and animals can be treated as objects for distance estimation. The distance between these and the host vehicle V can be estimated based on the heights of the V and the like.
なお、本発明は上記の実施形態に限定されるものではなく、様々な変形例が含まれる。例えば、上記の実施形態は本発明を分かりやすく説明するために詳細に説明したものであり、必ずしも説明した全ての構成を備えるものに限定されるものではない。また、或る実施形態の構成の一部を他の実施形態の構成に置き換えることが可能であり、また、或る実施形態の構成に他の実施形態の構成を加えることも可能である。また、各実施形態の構成の一部について、他の構成の追加・削除・置換をすることが可能である。
It should be noted that the present invention is not limited to the above embodiments, and includes various modifications. For example, the above embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described. Moreover, it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Moreover, it is possible to add, delete, or replace a part of the configuration of each embodiment with another configuration.
また、上記の各構成、機能、処理部、処理手段等は、それらの一部又は全部を、例えば集積回路にて設計する等によりハードウェアによって実現してもよい。また、上記の各構成、機能等は、プロセッサがそれぞれの機能を実現するプログラムを解釈し、実行することによりソフトウェアによって実現してもよい。各機能を実現するプログラム、テープ、ファイル等の情報は、メモリや、ハードディスク、SSD(solid state drive)等の記録装置、又は、ICカード、SDカード、DVD等の記録媒体に置くことができる。
In addition, each of the above configurations, functions, processing units, processing means, etc., may be realized by hardware, for example, by designing them in integrated circuits, in part or in whole. Moreover, each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function. Information such as programs, tapes, and files that implement each function can be stored in recording devices such as memories, hard disks, SSDs (solid state drives), or recording media such as IC cards, SD cards, and DVDs.
また、制御線や情報線は説明上必要と考えられるものを示しており、製品上必ずしも全ての制御線や情報線を示しているとは限らない。実際には殆ど全ての構成が相互に接続されていると考えてもよい。
In addition, the control lines and information lines indicate what is considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In practice, it may be considered that almost all configurations are interconnected.
1、1a、1b、1d、1e 外界認識装置、10 画像取得部、10a、10b 一対のカメラ(ステレオカメラ)、30 ランドマーク情報取得部、32 複眼領域ランドマーク検出部、33 大きさ情報計測部、34 ランドマーク情報計測部、36 ランドマーク情報記憶部、40 ランドマーク選択部、50 距離推定部、52 単眼領域ランドマーク検出部、54 単眼領域対象物検出部、56 近接判定部、58 大きさ推定部、60 距離算出部、70 進行路推定部、90 周辺環境認識部、100、101、102 ランドマーク、200 歩行者(対象物)、CA 複眼領域画像、LA 左単眼領域画像(単眼領域画像)、RA 右単眼領域画像(単眼領域画像)、V 車両、δ マージン(近接判定閾値)
1, 1a, 1b, 1d, 1e external world recognition device, 10 image acquisition unit, 10a, 10b pair of cameras (stereo cameras), 30 landmark information acquisition unit, 32 compound eye region landmark detection unit, 33 size information measurement unit , 34 landmark information measurement unit, 36 landmark information storage unit, 40 landmark selection unit, 50 distance estimation unit, 52 monocular area landmark detection unit, 54 monocular area object detection unit, 56 proximity determination unit, 58 size Estimation unit 60 Distance calculation unit 70 Course estimation unit 90 Surrounding environment recognition unit 100, 101, 102 Landmark 200 Pedestrian (object) CA Compound eye area image LA Left monocular area image (monocular area image ), RA right monocular area image (monocular area image), V vehicle, δ margin (proximity judgment threshold)
Claims (12)
- 車両に搭載される外界認識装置であって、
ランドマークおよび対象物の画像を取得する画像取得部と、
前記ランドマークの3次元情報を取得するランドマーク情報取得部と、
前記ランドマークの3次元情報と、前記画像内における前記ランドマークおよび前記対象物の大きさと、に基づき、前記対象物までの距離を推定する距離推定部と、を有することを特徴とする外界認識装置。 An external world recognition device mounted on a vehicle,
an image acquisition unit that acquires images of landmarks and objects;
a landmark information acquisition unit that acquires three-dimensional information of the landmark;
and a distance estimating unit that estimates a distance to the object based on three-dimensional information of the landmark and sizes of the landmark and the object in the image. Device. - 請求項1に記載の外界認識装置において、
前記画像取得部は、ステレオカメラによって撮像された画像であって、視野が重複する複眼領域と視野が重複しない単眼領域とを含む画像を取得し、
前記ランドマーク情報取得部は、前記複眼領域に位置するランドマークの3次元情報を取得し、
前記距離推定部は、前記単眼領域に位置する前記対象物までの距離を推定する、ことを特徴とする外界認識装置。 In the external world recognition device according to claim 1,
The image acquisition unit acquires an image captured by a stereo camera and including a compound eye region with overlapping fields of view and a monocular region with non-overlapping fields of view,
The landmark information acquisition unit acquires three-dimensional information of landmarks located in the compound eye region,
The external world recognition device, wherein the distance estimation unit estimates a distance to the object located in the monocular area. - 請求項2に記載の外界認識装置において、
前記ランドマーク情報取得部は、
前記複眼領域から前記ランドマークを検出する複眼領域ランドマーク検出部と、
前記複眼領域から、前記3次元情報に含まれる前記ランドマークの大きさを計測する大きさ情報計測部と、を備え、
前記距離推定部は、
前記単眼領域から前記ランドマークを検出する単眼領域ランドマーク検出部と、
前記単眼領域から前記対象物を検出する単眼領域対象物検出部と、
前記単眼領域における前記ランドマークの画像サイズ及び前記単眼領域における前記対象物の画像サイズに基づき、前記ランドマークの大きさから前記対象物の大きさを推定する大きさ推定部と、
前記大きさ推定部において推定された前記対象物の大きさと前記単眼領域における前記対象物の画像サイズとに基づき、前記距離を算出する距離算出部と、を有することを特徴とする外界認識装置。 In the external world recognition device according to claim 2,
The landmark information acquisition unit
a compound eye area landmark detection unit that detects the landmark from the compound eye area;
a size information measuring unit that measures the size of the landmark included in the three-dimensional information from the compound eye region;
The distance estimation unit
a monocular area landmark detection unit that detects the landmark from the monocular area;
a monocular area object detection unit that detects the object from the monocular area;
a size estimation unit that estimates the size of the object from the size of the landmark based on the image size of the landmark in the monocular region and the image size of the object in the monocular region;
and a distance calculation unit that calculates the distance based on the size of the object estimated by the size estimation unit and the image size of the object in the monocular region. - 請求項3に記載の外界認識装置において、
前記大きさ情報計測部は、前記複眼領域から前記ランドマークの実高さを算出し、
前記単眼領域ランドマーク検出部は、前記単眼領域から前記ランドマークの画像高さを算出し、
前記単眼領域対象物検出部は、前記単眼領域から前記対象物の画像高さを算出し、
前記大きさ推定部は、前記ランドマークの画像高さ及び前記対象物の画像高さに基づき、前記ランドマークの実高さから前記対象物の高さを推定し、
前記距離算出部は、推定された前記高さに基づき前記距離を算出する、ことを特徴とする外界認識装置。 In the external world recognition device according to claim 3,
The size information measuring unit calculates an actual height of the landmark from the compound eye area,
The monocular area landmark detection unit calculates an image height of the landmark from the monocular area,
The monocular area object detection unit calculates an image height of the object from the monocular area,
The size estimating unit estimates the height of the object from the actual height of the landmark based on the image height of the landmark and the image height of the object,
The external world recognition device, wherein the distance calculation unit calculates the distance based on the estimated height. - 請求項4に記載の外界認識装置において、
前記大きさ推定部は、前記ランドマークの画像高さに対する前記対象物の画像高さの比率を、前記ランドマークの実高さに乗算することにより、前記対象物の高さを推定する、ことを特徴とする外界認識装置。 In the external world recognition device according to claim 4,
The size estimation unit estimates the height of the object by multiplying the actual height of the landmark by a ratio of the image height of the object to the image height of the landmark. An external world recognition device characterized by - 請求項4に記載の外界認識装置において、
前記距離推定部は、前記単眼領域における前記ランドマークの下端と前記単眼領域における前記対象物の下端との高さ方向における差を算出する近接判定部を含み、
前記大きさ推定部は、前記近接判定部により算出された前記差が近接判定閾値内にある場合、前記対象物の高さを推定する、ことを特徴とする外界認識装置。 In the external world recognition device according to claim 4,
The distance estimating unit includes a proximity determining unit that calculates a height difference between the lower end of the landmark in the monocular region and the lower end of the object in the monocular region,
The external world recognition device, wherein the size estimation unit estimates the height of the object when the difference calculated by the proximity determination unit is within a proximity determination threshold value. - 請求項2に記載の外界認識装置において、
前記ランドマーク情報取得部は、
前記複眼領域から、前記ランドマークが移動しているか否かを判定するランドマーク情報計測部と、
前記複眼領域に含まれるランドマークのうち、前記ランドマーク情報計測部により移動していると判定されたランドマークを、前記対象物までの距離を推定するランドマークとして記憶するランドマーク情報記憶部とを含む、ことを特徴とする外界認識装置。 In the external world recognition device according to claim 2,
The landmark information acquisition unit
a landmark information measuring unit that determines whether or not the landmark has moved from the compound eye area;
a landmark information storage unit for storing, among landmarks included in the compound eye region, landmarks determined to be moving by the landmark information measurement unit as landmarks for estimating a distance to the object; An external world recognition device comprising: - 請求項4に記載の外界認識装置において、
前記ランドマーク情報取得部は、
前記実高さが高さ閾値以上のランドマークを、前記対象物までの距離を推定するランドマークとして記憶するランドマーク情報記憶部を含む、ことを特徴とする外界認識装置。 In the external world recognition device according to claim 4,
The landmark information acquisition unit
An external world recognition device, comprising: a landmark information storage unit that stores a landmark whose actual height is equal to or greater than a height threshold as a landmark for estimating a distance to the object. - 請求項2に記載の外界認識装置において、
前記ランドマーク情報取得部は、
前記複眼領域から、前記ランドマークのコントラスト情報を算出するランドマーク情報計測部と、
前記複眼領域に含まれるランドマークのうち、所定の閾値以上の前記コントラスト情報を有するランドマークを、前記対象物までの距離を推定するランドマークとして記憶するランドマーク情報記憶部とを含む、ことを特徴とする外界認識装置。 In the external world recognition device according to claim 2,
The landmark information acquisition unit
a landmark information measuring unit that calculates contrast information of the landmark from the compound eye area;
and a landmark information storage unit that stores, among the landmarks included in the compound eye area, landmarks having the contrast information equal to or greater than a predetermined threshold value as landmarks for estimating the distance to the object. Characteristic external recognition device. - 請求項1に記載の外界認識装置において、
前記ランドマーク情報取得部は、複数のランドマークの3次元情報を取得した場合、前記車両の動きに応じて、前記複数のランドマークの中から前記距離の推定に用いるランドマークを選択する、ことを特徴とする外界認識装置。 In the external world recognition device according to claim 1,
When the landmark information acquisition unit acquires the three-dimensional information of a plurality of landmarks, the landmark information acquisition unit selects a landmark to be used for estimating the distance from the plurality of landmarks according to the movement of the vehicle. An external world recognition device characterized by - 請求項10に記載の外界認識装置において、
前記車両の動きから前記車両の進行方向を推定する進行路推定部を備え、
前記ランドマーク情報取得部は、前記進行路推定部により推定された前記車両の進行方向に存在するランドマークを、前記対象物までの前記距離の推定に用いるランドマークとして選択する、ことを特徴とする外界認識装置。 In the external world recognition device according to claim 10,
A traveling path estimation unit for estimating the traveling direction of the vehicle from the movement of the vehicle,
The landmark information acquiring unit selects a landmark existing in the traveling direction of the vehicle estimated by the traveling route estimating unit as a landmark used for estimating the distance to the object. A device that recognizes the external world. - 請求項1に記載の外界認識装置において、
前記車両の周辺環境を認識する周辺環境認識部を備え、
前記ランドマーク情報取得部は、前記周辺環境認識部により歩道であると認識された領域に存在するランドマークを、前記対象物までの前記距離の推定に用いるランドマークとして選択する、ことを特徴とする外界認識装置。 In the external world recognition device according to claim 1,
A surrounding environment recognition unit that recognizes the surrounding environment of the vehicle,
The landmark information acquisition unit selects a landmark existing in an area recognized as a sidewalk by the surrounding environment recognition unit as a landmark used for estimating the distance to the object. A device that recognizes the external world.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
DE112022004418.7T DE112022004418T5 (en) | 2021-11-26 | 2022-10-17 | EXTERNAL ENVIRONMENT DETECTION DEVICE |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2021192274A JP2023078934A (en) | 2021-11-26 | 2021-11-26 | Environment recognition device |
JP2021-192274 | 2021-11-26 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2023095489A1 true WO2023095489A1 (en) | 2023-06-01 |
Family
ID=86539281
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2022/038648 WO2023095489A1 (en) | 2021-11-26 | 2022-10-17 | External environment recognition device |
Country Status (3)
Country | Link |
---|---|
JP (1) | JP2023078934A (en) |
DE (1) | DE112022004418T5 (en) |
WO (1) | WO2023095489A1 (en) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2014171052A1 (en) * | 2013-04-16 | 2014-10-23 | コニカミノルタ株式会社 | Image processing method, image processing device, image-capture device, and image processing program |
WO2017090410A1 (en) * | 2015-11-25 | 2017-06-01 | 日立オートモティブシステムズ株式会社 | Stereo camera device |
JP2018132477A (en) * | 2017-02-17 | 2018-08-23 | 日本電信電話株式会社 | Depth estimation device, dimension estimation device, depth estimation method, dimension estimation method, and program |
WO2021070537A1 (en) * | 2019-10-08 | 2021-04-15 | 日立Astemo株式会社 | Object recognition device |
-
2021
- 2021-11-26 JP JP2021192274A patent/JP2023078934A/en active Pending
-
2022
- 2022-10-17 DE DE112022004418.7T patent/DE112022004418T5/en active Pending
- 2022-10-17 WO PCT/JP2022/038648 patent/WO2023095489A1/en active Application Filing
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2014171052A1 (en) * | 2013-04-16 | 2014-10-23 | コニカミノルタ株式会社 | Image processing method, image processing device, image-capture device, and image processing program |
WO2017090410A1 (en) * | 2015-11-25 | 2017-06-01 | 日立オートモティブシステムズ株式会社 | Stereo camera device |
JP2018132477A (en) * | 2017-02-17 | 2018-08-23 | 日本電信電話株式会社 | Depth estimation device, dimension estimation device, depth estimation method, dimension estimation method, and program |
WO2021070537A1 (en) * | 2019-10-08 | 2021-04-15 | 日立Astemo株式会社 | Object recognition device |
Also Published As
Publication number | Publication date |
---|---|
DE112022004418T5 (en) | 2024-07-18 |
JP2023078934A (en) | 2023-06-07 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US11087148B2 (en) | Barrier and guardrail detection using a single camera | |
JP7025912B2 (en) | In-vehicle environment recognition device | |
JP4650079B2 (en) | Object detection apparatus and method | |
US9846812B2 (en) | Image recognition system for a vehicle and corresponding method | |
US10129521B2 (en) | Depth sensing method and system for autonomous vehicles | |
US7612800B2 (en) | Image processing apparatus and method | |
US20180137376A1 (en) | State estimating method and apparatus | |
US8428305B2 (en) | Method for detecting a clear path through topographical variation analysis | |
US9690993B2 (en) | Method and device for analyzing trafficability | |
US9697421B2 (en) | Stereoscopic camera apparatus | |
US20090052742A1 (en) | Image processing apparatus and method thereof | |
JP6711395B2 (en) | Image processing device, imaging device, mobile device control system, mobile device, image processing method, and program | |
JP6816401B2 (en) | Image processing device, imaging device, mobile device control system, image processing method, and program | |
CN111164648B (en) | Position estimating device and position estimating method for mobile body | |
JP2016099650A (en) | Travel lane recognition apparatus and travel support system using the same | |
Petrovai et al. | A stereovision based approach for detecting and tracking lane and forward obstacles on mobile devices | |
WO2019065970A1 (en) | Vehicle exterior recognition device | |
CN113874914A (en) | Method for determining an operating angle between a tractor and a trailer of a tractor | |
US20200193184A1 (en) | Image processing device and image processing method | |
EP3410345B1 (en) | Information processing apparatus and non-transitory recording medium storing thereon a computer program | |
KR20180047149A (en) | Apparatus and method for risk alarming of collision | |
US20220153259A1 (en) | Autonomous parking systems and methods for vehicles | |
Ma et al. | A real time object detection approach applied to reliable pedestrian detection | |
KR101595317B1 (en) | Precise positioning of the vehicle for detecting a road surface display method and system | |
WO2023095489A1 (en) | External environment recognition device |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22898273 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 112022004418 Country of ref document: DE |