WO2023095489A1 - Dispositif de reconnaissance d'environnement externe - Google Patents

Dispositif de reconnaissance d'environnement externe Download PDF

Info

Publication number
WO2023095489A1
WO2023095489A1 PCT/JP2022/038648 JP2022038648W WO2023095489A1 WO 2023095489 A1 WO2023095489 A1 WO 2023095489A1 JP 2022038648 W JP2022038648 W JP 2022038648W WO 2023095489 A1 WO2023095489 A1 WO 2023095489A1
Authority
WO
WIPO (PCT)
Prior art keywords
landmark
unit
recognition device
external world
height
Prior art date
Application number
PCT/JP2022/038648
Other languages
English (en)
Japanese (ja)
Inventor
健 遠藤
春樹 的野
健 永崎
Original Assignee
日立Astemo株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 日立Astemo株式会社 filed Critical 日立Astemo株式会社
Publication of WO2023095489A1 publication Critical patent/WO2023095489A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01C—MEASURING DISTANCES, LEVELS OR BEARINGS; SURVEYING; NAVIGATION; GYROSCOPIC INSTRUMENTS; PHOTOGRAMMETRY OR VIDEOGRAMMETRY
    • G01C3/00—Measuring distances in line of sight; Optical rangefinders
    • G01C3/02—Details
    • G01C3/06—Use of electric means to obtain final indication

Definitions

  • the present invention relates to an external world recognition device.
  • Patent Literature 1 describes a pedestrian distance calculation method using a stereo camera that has a field-of-view overlapping area and a field-of-view non-overlapping area.
  • Patent Literature 1 describes a stereo camera device that calculates the distance of a moving object in a monocular region using road surface height calculated from parallax information.
  • the stereo camera device obtains coordinates obtained by subtracting (lowering) the height of the stairs relative to the road surface from the vertical coordinates on the image of the feet of the pedestrian on the stairs. Then, in the stereo camera device, the obtained coordinates are extended to the stereo area, and the distance to the pedestrian is obtained from the parallax information of the road surface at that position.
  • the parallax of the road surface is calculated based on the texture information of the road surface in the acquired image, and this parallax information is used to calculate the distance to the pedestrian.
  • this parallax information is used to calculate the distance to the pedestrian.
  • the present invention has been made in view of the above problems, and its purpose is to provide an external world recognition device that can accurately calculate the distance to an object regardless of the usage environment.
  • an external world recognition device mounted on a vehicle, comprising: an image acquisition unit for acquiring an image of a landmark and an object; a landmark information acquisition unit that acquires information; and a distance estimation unit that estimates the distance to the object based on the three-dimensional information of the landmark and the sizes of the landmark and the object in the image. and.
  • the distance to the object can be accurately calculated regardless of the usage environment. Problems, configurations, and effects other than those described above will be clarified by the following description of the embodiments.
  • FIG. 1 is a functional block diagram showing a schematic configuration of an external world recognition device according to a first embodiment of the present invention
  • FIG. FIG. 2 is a plan view showing an example of a state in which the external world recognition device of FIG. 1 is mounted on a vehicle
  • 2 is a flowchart showing landmark information acquisition processing of the external world recognition device of FIG. 1; An example of the image which the external world recognition apparatus of FIG. 1 acquired by landmark information acquisition processing.
  • 2 is a flowchart showing distance estimation processing of the external world recognition device of FIG. 1; An example of the image which the external world recognition apparatus of FIG. 1 acquired by the distance estimation process.
  • the functional block diagram which shows schematic structure of the external world recognition apparatus of the modification of 1st Embodiment.
  • FIG. 8 is a flowchart showing distance estimation processing of the external world recognition device of FIG. 7;
  • FIG. 8 is a view for explaining proximity determination processing of the external world recognition device of FIG. 7;
  • FIG. 8 is a diagram showing another example of proximity determination processing of the external world recognition device of FIG. 7 ;
  • FIG. 5 is a functional block diagram showing a schematic configuration of an external world recognition device according to another modification of the first embodiment;
  • the functional block diagram which shows schematic structure of the external world recognition apparatus which concerns on 2nd Embodiment of this invention.
  • the functional block diagram which shows schematic structure of the external world recognition apparatus which concerns on 3rd Embodiment of this invention.
  • FIG. 1 is a functional block diagram showing a schematic configuration of an external world recognition device 1 according to the first embodiment of the invention.
  • 2 is a plan view showing an example of a state in which the external world recognition device 1 of FIG. 1 is mounted on a vehicle V.
  • FIG. 3 is a flowchart showing landmark information acquisition processing of the external world recognition device 1 of FIG.
  • FIG. 4 is an example of an image acquired by the external world recognition device 1 of FIG. 1 in the landmark information acquisition process.
  • FIG. 5 is a flowchart showing distance estimation processing of the external world recognition device 1 of FIG.
  • FIG. 6 is an example of an image acquired by the external world recognition device 1 of FIG. 1 through distance estimation processing.
  • the external world recognition device 1 is mounted on a vehicle V (hereinafter also referred to as own vehicle V). As shown in FIG. 1 , the external world recognition device 1 has an image acquisition section 10 , a landmark information acquisition section 30 and a distance estimation section 50 . Although illustration is omitted, the external world recognition device 1 has a configuration in which a CPU, a RAM, a ROM, and the like are connected via a bus, and the CPU executes various control programs stored in the ROM to control the entire system. control behavior.
  • the image acquisition unit 10 acquires images from a pair of cameras 10a and 10b (FIG. 2) functioning as stereo cameras, and the distance estimation unit 50 uses the images to determine the pedestrian 200 as the target object. and the own vehicle V will be described.
  • a “landmark” used in this specification means each three-dimensional object (for example, reference numerals 100, 101 and 102 shown in FIG. 4) photographed by the pair of cameras 10a and 10b.
  • "three-dimensional information of landmarks” means including at least one of the height and width of the landmarks 100, 101, and 102 themselves.
  • the "three-dimensional information of the landmarks” includes the distance between the landmarks 100, 101, and 102 and the vehicle V ( depth).
  • the image acquisition unit 10 acquires images of landmarks 100, 101 and 102 and a pedestrian (object) 200 from a pair of cameras 10a and 10b. Specifically, the image acquisition unit 10 captures an image captured by a pair of cameras 10a and 10b, and divides a compound eye region with overlapping fields of view and a monocular region (a left monocular region and a right monocular region) with non-overlapping fields of view. Get the containing image.
  • the compound eye area of the image will be referred to as compound eye area image CA
  • the left monocular area and right monocular area of the image will be referred to as left monocular area image LA and right monocular area image RA, respectively.
  • the pair of cameras 10a and 10b are configured as one stereo camera mounted on a vehicle V.
  • the pair of cameras 10a and 10b are composed of CCD or CMOS image sensors, and are directed to the front of the vehicle V.
  • the pair of cameras 10a and 10b are installed so that each of the cameras 10a and 10b captures an image in front of the vehicle V at a predetermined depression angle (that is, an imaging area), and the angles of depression overlap with each other. For example, if the camera 10a is the left camera and the camera 10b is the right camera, the left side of the depression angle of the camera 10a and the right side of the camera 10b overlap each other.
  • the center front area of the vehicle V becomes a stereo area (also called a compound eye area) defined by the imaging area of the camera 10a and the imaging area of the camera 10b.
  • the left front area of the vehicle V is a left monocular area defined by the left portion of the photographing area of the camera 10b.
  • the right front area of the vehicle V is a right monocular area defined by the right side of the imaging area of the camera 10a. Images captured by the pair of cameras 10 a and 10 b are input to the image acquisition section 10 .
  • the image acquisition unit 10 acquires two images captured by a pair of cameras 10a and 10b, and acquires parallax information by applying a known parallax calculation algorithm to the acquired left and right images.
  • the landmark information acquisition unit 30 includes a compound eye region landmark detection unit 32, a size information measurement unit 33, and a landmark information storage unit 36.
  • a landmark information acquisition unit 30 acquires three-dimensional information of the landmarks 100 , 101 and 102 .
  • the landmark information acquisition unit 30 acquires three-dimensional information of the landmarks 100, 101, and 102 positioned in the compound eye area image CA.
  • the compound eye area landmark detection unit 32 detects landmarks 100, 101, and 102 from the compound eye area image CA acquired by the image acquisition unit 10.
  • the compound eye area landmark detection unit 32 may analyze the texture of the compound eye area image CA transmitted from the image acquisition unit 10 and detect the landmarks 100, 101 and 102 from this.
  • the compound eye area landmark detection unit 32 may detect the landmarks 100 , 101 , and 102 based on the parallax information acquired by the image acquisition unit 10 .
  • the landmarks 100, 101, and 102 may be detected by clustering disparity information or the like, or the landmarks 100, 101, and 102 may be detected using a convolutional neural network as statistical machine learning. You may
  • the size information measuring unit 33 measures the actual sizes of the landmarks 100, 101, and 102 included in the three-dimensional information from the compound eye area image CA. Specifically, the size information measurement unit 33 calculates the actual heights of the landmarks 100, 101, and 102 detected by the compound eye area landmark detection unit 32 from the compound eye area image CA. to calculate the Note that the size information measuring unit 33 measures not only the actual vertical size (actual height) of the landmarks 100, 101, and 102, but also the actual horizontal size (actual height) of the landmarks 100, 101, and 102. actual width) may be measured. Also, the size information measuring unit 33 may measure the actual size (actual height or actual width) of a part of the landmarks 100 , 101 , 102 .
  • the landmark information storage unit 36 registers at least one of the size information of the landmarks 100, 101, and 102 measured by the size information measuring unit 33 as landmark information used for distance calculation, which will be described later. Specifically, the landmark information storage unit 36 stores size information (actual height, actual width, etc.) of the landmarks 100, 101, and 102 that have been measured, as well as size information of the photographed landmarks 100, 101, and 102. Stores texture information.
  • the distance estimation unit 50 estimates the distance to the pedestrian 200 located in the monocular area image (left monocular area image LA, right monocular area image RA). Specifically, the distance estimating unit 50 uses the landmarks 100, 101, and 102 detected by the landmark information acquiring unit 30 to use the monocular area images (left monocular area image LA, right The distance to the pedestrian 200 included in the monocular area image RA) is estimated. More specifically, the distance estimation unit 50 estimates the distance from the vehicle V to the pedestrian 200 based on the three-dimensional information of the landmark 100 and the sizes of the landmark 100 and the pedestrian 200 in the image. do. In this embodiment, a case where the landmark 100 is used as a landmark for estimating the distance between the pedestrian 200 and the own vehicle V will be described. As shown in FIG. 1 , the distance estimation unit 50 includes a monocular area landmark detection unit 52 , a monocular area object detection unit 54 , a size estimation unit 58 and a distance calculation unit 60 .
  • the monocular area landmark detection unit 52 detects landmarks 100 from the monocular area images (left monocular area image LA, right monocular area image RA) acquired by the image acquisition unit 10 . Specifically, the monocular area landmark detection unit 52 re-detects the landmark 100 based on the texture information of the landmark 100 stored in the landmark information storage unit 36 . For example, the monocular area landmark detection unit 52 may detect by template matching using the texture information of the landmark 100 stored in the landmark information storage unit 36 as a template. Also, the monocular area landmark detection unit 52 may detect the landmark 100 by tracking processing using a convolutional neural network.
  • the monocular area landmark detection unit 52 detects the vertical length of the landmark 100 in the image from the monocular area images (the left monocular area image LA and the right monocular area image RA) of the landmark 100 acquired from the image acquisition unit 10 .
  • height i.e., image height
  • the monocular area object detection unit 54 detects an object for distance estimation from the monocular area images (left monocular area image LA, right monocular area image RA) acquired by the image acquisition unit 10 .
  • the monocular region target object detection unit 54 detects, for example, a pedestrian 200 by known statistical machine learning when the target is the pedestrian 200 .
  • the monocular region object detection unit 54 may detect the pedestrian 200 using classical machine learning such as random forest, support vector machine, and real Adaboost, or may detect the pedestrian 200 using a convolutional neural network. A pedestrian 200 may be detected by using this.
  • the monocular area object detection unit 54 calculates the vertical length (that is, the image height) of the pedestrian 200 in the image based on the monocular area image of the pedestrian 200 acquired from the image acquisition unit 10 .
  • the size estimation unit 58 estimates the landmark 100 based on the image size of the landmark 100 in the monocular area image (left monocular area image LA, etc.) and the image size of the pedestrian 200 in the monocular area image (left monocular area image LA, etc.).
  • the size of pedestrian 200 is estimated from the size of .
  • the size estimation unit 58 estimates the size (height, etc.) of the pedestrian 200 using the size (height, etc.) of the landmark 100 stored in the landmark information storage unit 36 . Specifically, based on the image height of the landmark 100 and the image height of the pedestrian 200, the size estimation unit 58 calculates the height of the pedestrian 200 from the actual height of the landmark 100 (hereinafter also referred to as height). ).
  • the size estimation unit 58 multiplies the actual height of the landmark 100 by the ratio of the height of the image of the pedestrian 200 to the height of the image of the landmark 100 to obtain the height of the pedestrian 200. to estimate The size information of pedestrian 200 may be estimated only once.
  • the distance calculation unit 60 A distance to the pedestrian 200 is calculated. Specifically, the distance calculation unit 60 calculates the distance from the own vehicle V to the pedestrian 200 based on the height of the pedestrian 200 estimated by the size estimation unit 58 . More specifically, the distance calculation unit 60 calculates the size of the vehicle based on the height of the pedestrian 200 estimated by the size estimation unit 58 and the image height of the pedestrian 200 detected by the monocular area object detection unit 54. The distance between V and pedestrian 200 is calculated.
  • FIGS. 3 to 6 an operation example of the external world recognition device 1 of this embodiment will be described in landmark information acquisition processing (FIGS. 3 and 4) and distance estimation processing (FIGS. 5 and 6). I will explain separately.
  • the external world recognition device 1 using a pair of cameras 10a and 10b functioning as stereo cameras installed to monitor the front of the vehicle V will be described.
  • a case of calculating the distance between the own vehicle V and the pedestrian 200 based on the height of the pedestrian 200 photographed in the left monocular area image LA shown in FIG. 6 will be described.
  • the external world recognition device 1 performs image acquisition processing (P101), parallax calculation processing (P102), solid object detection processing (P103), and three-dimensional information acquisition processing (P104) in the landmark information acquisition processing.
  • the landmark registration process (P105) is executed in order.
  • the image acquisition unit 10 acquires images captured by a pair of cameras 10a and 10b as shown in FIG. Acquire an area image RA).
  • the image acquisition unit 10 calculates the parallax in the compound eye area image CA. For example, a window area of 5 ⁇ 5 pixels is set for the parallax calculation. Then, the parallax is calculated by horizontally scanning the image of the right camera 10b using the SAD as an evaluation value, with the image of the left camera 10a of the pair of cameras 10a and 10b as a reference.
  • the compound eye region landmark detection unit 32 uses the parallax calculated in the parallax calculation process (P102) to detect landmarks 100, 101, and 102 as shown in FIG. To detect. Specifically, since the areas of the landmarks 100, 101, and 102 are at the same distance, clustering processing is performed on the parallax to roughly extract the areas where the landmarks 100, 101, and 102 exist. be done. Next, since the distance value changes between each of the landmarks 100, 101, and 102 and the boundary of the background, by detecting the parallax change point in the roughly extracted region, the landmarks 100, 101, and 102 The existing areas are detected in detail. FIG.
  • FIG. 4 shows the detection result of the three-dimensional object detection process (P103) for the landmark 100 among the plurality of landmarks 100, 101, and 102.
  • the size information measuring unit 33 measures the height direction size ( That is, the actual height) is calculated.
  • a method of estimating the size (actual height) with respect to the landmark 100 among the plurality of landmarks 100, 101, and 102 will be described below.
  • the rectangle B100 which is the detection result of the three-dimensional object detection process (P103), is used.
  • T100 in FIG. 4 is the pixel position of the upper edge of the rectangle B100
  • L100 in FIG. 4 indicates the pixel position of the lower edge of the rectangle B100.
  • D_med be the disparity median value at the landmark 100 .
  • D_med is obtained by calculating the median value of disparities contained within rectangle B100.
  • BaseLine in Expression (1) is the base line length of the camera 10a and the camera 10b, and abs indicates an absolute value operator.
  • the landmark information storage unit 36 registers the actual height information of the landmark 100 calculated in the three-dimensional information acquisition process (P104). Also, the landmark information storage unit 36 registers the texture information of the compound eye area image CA analyzed by the compound eye area landmark detection unit 32 . The landmark information storage unit 36 also stores a compound eye area image CA for the rectangle B100 detected in the three-dimensional object detection process (P103).
  • FIG. 5 the distance estimation process will be described with reference to FIGS. 5 and 6.
  • the external world recognition device 1 performs image acquisition processing (P111), landmark detection processing (P112), pedestrian detection processing (P113), size estimation processing (P115), distance The estimation process (P116) is executed in order.
  • the image acquisition unit 10 acquires images captured by a pair of cameras 10a and 10b as shown in FIG. Acquire an area image RA).
  • the monocular area landmark detection unit 52 detects the landmark 100 again.
  • the landmark 100 was photographed in the compound eye area image CA in FIG. 4, but as the vehicle V moves, it is photographed in the left monocular area image LA in FIG.
  • the landmark 100 photographed in the left monocular area image LA shown in FIG. 6 is re-detected. do.
  • the area detected in this process is represented by a rectangle B101 as shown in FIG.
  • the monocular area object detection unit 54 detects the pedestrian 200 photographed in the left monocular area image LA of FIG.
  • a convolutional neural network is used for this detection.
  • the convolutional neural network used detects the pedestrian 200 included in the input predetermined image.
  • the convolutional neural network not only locates the pedestrian 200, but also calculates the identification score at the same time.
  • the identification score is a point for the type of pedestrian 200, and a high point for a certain type indicates that the region is likely to be of that type.
  • the pedestrian 200 is detected by inputting the left monocular area image LA into the convolutional neural network and adopting only detection results with a high identification score for the type of pedestrian.
  • the detection result is represented by a rectangle B201 as shown in FIG.
  • the size estimation unit 58 estimates the height of the pedestrian 200 using the landmark 100 information.
  • the actual height of the landmark 100 is stored in the landmark information storage unit .
  • the monocular area landmark detection unit 52 detects the image height of the landmark 100 (that is, the height of the rectangle B101 on the image) IMG_Height_Land(pix).
  • the monocular area object detection unit 54 detects the image height of the pedestrian 200 (that is, the height of the rectangle B201 on the image) IMG_Height_Ped(pix).
  • the distance calculation unit 60 calculates the height (Height_Ped) of the pedestrian 200 and the image height (IMG_Height_Ped(pix)) of the rectangle B201 detected in the pedestrian detection process (P113). , is used to obtain the distance from the own vehicle V to the pedestrian 200 .
  • f is the focal length (pix) of the pair of cameras 10a and 10b. Note that once the height of the pedestrian is calculated in the size estimation process (P115), the distance estimation process can be performed using the height of the pedestrian 200 in another image frame after the image shown in FIG. At (P116), the distance between the vehicle V and the pedestrian 200 can be obtained.
  • the external world recognition device 1 of this embodiment utilizes the actual height of any landmark 100 photographed in the compound eye area image CA to perform walking in the monocular area image (for example, the left monocular area image LA).
  • the distance between the person 200 and the own vehicle V is calculated. Since the landmark 100 is a three-dimensional object and has texture information, parallax can be obtained more stably than the road surface. Therefore, it is possible to calculate the distance to the pedestrian 200 with higher accuracy even in a situation where it is difficult to obtain the parallax with respect to the road surface, such as at night or in rainy weather.
  • the external world recognition device 1 of this embodiment measures the actual height of the landmark 100 in the compound eye area image CA, and uses the result to estimate the height of the pedestrian 200 .
  • Pedestrian 200 changes its size in the horizontal direction according to the movement of the hand during walking.
  • the height of the pedestrian 200 does not change or fluctuates little even when the pedestrian 200 walks. Therefore, by estimating the height of pedestrian 200 and obtaining the distance between pedestrian 200 and own vehicle V based on the estimated height, the distance to pedestrian 200 can be obtained with high accuracy.
  • the mounting posture of the camera on the road surface can be accurately estimated. There is a need.
  • the distance between the own vehicle V and the pedestrian 200 can be calculated without being affected by changes in the posture of the camera. can be estimated.
  • FIG. 7 is a functional block diagram showing a schematic configuration of an external world recognition device 1d according to Modification 1 of the first embodiment.
  • FIG. 8 is a flowchart showing distance estimation processing of the external world recognition device 1d of FIG.
  • FIG. 9 is a diagram for explaining proximity determination processing of the external world recognition device 1d of FIG.
  • the external world recognition device 1d according to Modification 1 differs from the external world recognition device 1 described above in that it includes a proximity determination unit 56, which will be described later.
  • configurations having the same or similar functions as those of the external world recognition device 1 described above are denoted by the same reference numerals, and descriptions thereof are omitted, and different portions are described.
  • the distance estimation unit 50 includes a proximity determination unit 56.
  • the proximity determination unit 56 determines whether or not the landmark 100 detected by the monocular area landmark detection unit 52 and the pedestrian 200 detected by the monocular area object detection unit 54 are three-dimensionally close. Specifically, as shown in FIG. 9, the proximity determination unit 56 determines whether the lower end of the landmark 100 in the left monocular area image LA (monocular area image) and the pedestrian 200 in the left monocular area image LA (monocular area image) Calculate the difference in the height direction from the bottom edge. If the difference calculated by the proximity determination unit 56 is within a margin (proximity determination threshold value) ⁇ described later, and the size estimation unit 58 determines that the landmark 100 and the pedestrian 200 are close to each other, , to estimate the height of the pedestrian 200 .
  • a margin proximity determination threshold value
  • the external world recognition device 1d performs proximity determination processing ( P114) is executed.
  • the proximity determination unit 56 spatially determines the landmark 100 detected by the monocular area landmark detection unit 52 and the pedestrian 200 detected by the monocular area object detection unit 54. It is determined whether or not it exists at a close position.
  • the proximity determination unit 56 uses the lower end of the landmark 100.
  • FIG. 9 the proximity determination unit 56 sets a predetermined margin ⁇ (for example, 5 pix) in the height direction around the lower end of the landmark 100, and the lower end of the pedestrian 200 is included in the margin ⁇ .
  • the proximity determination unit 56 determines that the landmark 100 and the pedestrian 200 are at approximately the same distance from the host vehicle V, and determines that the two are close to each other. It is determined that When the proximity determination unit 56 determines that the landmark 100 and the pedestrian 200 are close to each other, in the size estimation process (P115), the size estimation unit 58 determines the size of the landmark 100 stored in the landmark information storage unit 36. The height of the pedestrian 200 is estimated using the actual height. Note that the above margin ⁇ (eg, 5 pix) may be stored in the landmark information storage unit 36 .
  • the proximity determination unit 56 determines whether or not the landmark 100 is used for estimating the height of the pedestrian 200.
  • the ratio of the size (ie, image height) of the pedestrian 200 and the landmark 100 on the image is used.
  • the ratio of the image heights of the two may deviate from the ratio of the actual heights of the two, and the height of the pedestrian 200 cannot be accurately estimated.
  • the height of the pedestrian 200 can be estimated more accurately by the proximity determination unit 56 performing the proximity determination process (P114) as in the first modification.
  • FIG. 10 is a diagram showing another example of the proximity determination process (P114) of the external world recognition device 1d of FIG.
  • the landmark registration process (P105) is described in the upper part, and the proximity determination process (P114) is described in the lower part.
  • the example shown in FIG. 10 differs from the external world recognition device 1d according to Modification 1 described above in terms of landmark registration processing (P105) and proximity determination processing (P114).
  • the same reference numerals as those of the external world recognition device 1d (FIG. 7) according to Modification 1 are attached and the description thereof will be made.
  • the proximity determination unit 56 executes the proximity determination process (P114) based on the parallax around the landmark 100 detected by the compound eye region landmark detection unit 32.
  • the compound eye area landmark detection unit 32 detects the inclination of the surface on which the landmark 100 is on the ground based on the parallax information around the surface on which the detected landmark 100 is on the ground. change may be calculated. As a result, the location where the inclination of the surface on which the landmark 100 is grounded is identified.
  • the size information measurement unit 33 measures the image height (Obj_Height) of the landmark 100 detected by the compound eye area landmark detection unit 32 and the image from the center position of the landmark 100 to the point where the inclination changes.
  • the landmark information storage unit 36 stores, for example, the value of Plane_rate as the proximity determination radius in addition to the predetermined margin ⁇ (landmark registration process (P105)).
  • the proximity determination unit 56 uses the image height (Obj_Height2) of the landmark 100 re-detected by the monocular area landmark detection unit 52 in the proximity determination process (P114), Calculate the margin ⁇ 2 (Plane_rate ⁇ Obj_Height2). Then, the proximity determination unit 56 determines that the pedestrian 200 is close to the landmark 100 when the pedestrian 200 approaches within ⁇ 2 (pix) in the horizontal direction in addition to the margin ⁇ (pix) in the vertical direction. judge.
  • the height of the pedestrian 200 is estimated on the assumption that the pedestrian 200 and the landmark 100 are at the same distance from the own vehicle V. However, if the inclination of the surface where the landmark 100 touches the ground changes in the middle, if only the vertical margin ⁇ is referred to in the proximity determination process (P114), the pedestrian 200 and the landmark 100 are not at the same distance. In spite of this, it may be determined that the landmark 100 and the pedestrian 200 are close to each other. Therefore, based on the parallax information, the area ( ⁇ 2 in the lower part of FIG. 10) up to the point where the inclination of the road surface does not change is calculated, and based on this result, the proximity determination process (P114) is performed, so that both are at equal distances or not. Therefore, the height of pedestrian 200 can be estimated more accurately.
  • the external world recognition device 1e according to Modification 2 differs from the external world recognition device 1 described above in that it includes a landmark information measurement unit 34, which will be described later.
  • a landmark information measurement unit 34 which will be described later.
  • FIG. 11 is a functional block diagram showing a schematic configuration of an external world recognition device 1e according to modification 2 of the first embodiment.
  • the landmark information acquisition section 30 includes a landmark information measurement section 34 .
  • the landmark information measurement unit 34 determines whether the landmarks 100, 101, and 102 detected by the compound eye area landmark detection unit 32 have moved from the compound eye area image CA. judge. Then, the landmark information storage unit 36 stores the landmarks determined to be moving by the landmark information measurement unit 34 among the landmarks 100, 101, and 102 included in the compound eye area image CA (for example, the landmarks in FIG. The mark 101) is stored as a landmark for estimating the distance to the pedestrian 200.
  • the external world recognition device 1e not only can stationary three-dimensional objects be registered as landmarks, but also three-dimensional objects such as other pedestrians and vehicles can be used as landmarks as mobile objects. Since the landmark 100 shown in FIG. 4 is a stationary object, if the pedestrian 200, who is the target of distance calculation with respect to the own vehicle V, is stationary, the two do not approach each other. Therefore, the height of stationary pedestrian 200 cannot be estimated based on landmark 100, which is a stationary object. On the other hand, by using a moving object (for example, the landmark 101) as in Modification 2, even when the pedestrian 200, which is the object of distance calculation, is stationary, the landmark 101 is the pedestrian. It can get close to 200. Therefore, it becomes possible to estimate the height of the stationary pedestrian 200 as well.
  • a moving object for example, the landmark 101
  • the external world recognition device according to Modification 3 differs from the external world recognition device described above in terms of the function of the landmark information storage unit 36 .
  • the external world recognition device 1 according to Modification 3 will be described with the same reference numerals as the external world recognition device 1 described above, with reference to FIGS.
  • the landmark information storage unit 36 shown in FIG. ) above (for example, the landmarks 100 and 102 in FIG. 4) are stored as landmarks for estimating the distance to the pedestrian 200 .
  • the actual height of the landmark eg , 100 and 102 in FIG. 4 are registered in the landmark information storage unit 36 in the landmark registration process (P105).
  • the landmark detection process P112
  • the landmarks registered in the landmark registration process (P105) and having an actual height equal to or greater than the height threshold for example, the landmarks 100 and 102 in FIG. 4) are detected again. .
  • the detected position may be misaligned.
  • the image height of the landmark detected by the monocular area landmark detection unit 52 changes from the true image height of the landmark.
  • the estimated height of pedestrian 200 may be deviated.
  • the external world recognition device according to Modification 3 uses only landmarks having a height equal to or greater than a predetermined height threshold (for example, 1 m). Therefore, the difference between the height of the detected landmark image and the true height of the image can be reduced, and the height of the pedestrian can be estimated more accurately.
  • the landmark information storage unit 36 selects the plurality of landmarks 100, 101, and 102 measured by the size information measuring unit 33 in descending order of actual height. (for example, the number corresponding to 60% of the plurality of landmarks) (for example, the landmarks 100 and 102) may be stored as landmarks for estimating the distance to the pedestrian 200.
  • FIG. As a result, many options of landmarks used for distance estimation to the pedestrian 200 can be held.
  • the external world recognition device according to Modified Example 4 differs from the external world recognition device 1e (FIG. 11) according to Modified Example 2 in terms of the functions of the landmark information measurement unit 34 and the landmark information storage unit 36.
  • the external world recognition device according to Modified Example 4 will be described with reference to FIGS.
  • the landmark information measurement unit 34 calculates the contrast information (edge component , color, luminance difference, etc.). Then, of the landmarks 100, 101, and 102 included in the compound eye area image CA, the landmark information storage unit 36 selects a landmark having contrast information equal to or greater than a predetermined threshold value (for example, the landmark 100 in FIG. 4), It is stored as a landmark for estimating the distance to pedestrian 200 .
  • a predetermined threshold value for example, the landmark 100 in FIG. 4
  • the landmark information measurement unit 34 applies a Sobel filter to each of the landmarks 100, 101, and 102 in the landmark registration process (P105) to obtain edge components (contrast information).
  • the landmark information storage unit 36 selects landmarks (for example, the landmark 100) whose density of edge components (the number of edges) is equal to or greater than a predetermined value within the areas of the landmarks 100, 101, and 102. register.
  • a landmark for example, landmark 100
  • whose color information (contrast information) is equal to or greater than a predetermined value may be registered.
  • pixels with saturation and brightness equal to or greater than a predetermined value are counted in a hue region that can be regarded as the red component of an image, and if the number of pixels is equal to or greater than a predetermined value, it is registered as a landmark. good too.
  • the landmark detection process P112
  • the landmark 100 can be re-detected more stably, and erroneous detection of other three-dimensional objects as landmarks can be avoided. Thereby, the height of pedestrian 200 can be estimated with higher accuracy.
  • the landmark information measurement unit 34 analyzes the luminance difference between the lower ends of the landmarks 100, 101, and 102 and the road surface.
  • the landmark information storage unit 36 stores landmarks (for example, landmarks) in which the number of pixels whose brightness difference between the lower end and the road surface is equal to or greater than a predetermined value among the landmarks 100, 101, and 102 is equal to or greater than a predetermined value. store the mark 100). This makes it easier for the monocular area landmark detection unit 52 to re-detect the same area as the area detected by the compound eye area landmark detection unit 32, thereby estimating the height of the pedestrian 200 with higher accuracy. can be done.
  • the external world recognition device includes the proximity determination unit 56
  • the proximity determination unit 56 when using the lower end of the landmark 100 in the proximity determination process (P114), it is possible to register the landmark 100 that is highly separable from the road surface. , the proximity determination can be performed more accurately.
  • the landmark information storage unit 36 can register a specific part of the landmark in the landmark registration process (P105). Specifically, the landmark information measuring unit 34 measures, for example, a portion of the landmark 100 where the edge component is strong as the upper end of the landmark. Then, in the processing, the landmark information storage unit 36 can register the portion as the upper end of the landmark. This makes it easier for the monocular area landmark detection section 52 to re-detect the same portion as the portion detected by the compound eye area landmark detection section 32 . In this way, by registering, for example, a specific portion of the landmark 100 based on a portion with a strong edge component, the landmark can be stably re-detected by the monocular area landmark detection unit 52. Pedestrian distance can be estimated with high accuracy.
  • the external world recognition device according to modification 5 differs from the external world recognition device 1 described above in terms of the function of the size estimation unit 58 .
  • the external world recognition device according to Modification 5 will be described with the same reference numerals as the external world recognition device 1 described above, with reference to FIGS.
  • the size estimation unit 58 uses the plurality of landmarks to estimate the pedestrian 200. Estimate the size of Specifically, the size estimation unit 58 estimates the height (Height_Ped) of the pedestrian 200 when approaching the landmark 100 in a certain image frame. Then, the distance calculation unit 60 calculates the distance between the vehicle V and the pedestrian 200 based on the Height_Ped. After that, in another image frame, when the pedestrian 200 approaches the landmark 101, the size estimator 58 calculates the pedestrian's Update height Height_Ped.
  • Height_Ped_2 is the height of the pedestrian 200 estimated from the actual height of the landmark 101, the image height of the landmark 101, and the height of the pedestrian 200 image.
  • size estimation unit 58 updates the height of pedestrian 200 each time pedestrian 200 approaches a plurality of landmarks. As a result, even if the initial height estimation result contains an error, the height can be updated so as to approach the correct value each time it approaches each landmark, and the distance of the pedestrian can be estimated more accurately. can.
  • FIG. 12 is a functional block diagram showing a schematic configuration of the external world recognition device 1a according to the second embodiment of the present invention.
  • the external world recognition device 1a according to the second embodiment differs from the external world recognition device 1d (Modification 1) in that it includes a landmark selection unit 40 and a course estimation unit 70, which will be described later.
  • Configurations having the same or similar functions as those of the external world recognition device 1d described above are denoted by the same reference numerals as those of the external world recognition device 1d, and descriptions thereof are omitted. Different parts will be explained.
  • the external world recognition device 1a includes a landmark selection unit 40 and a traveling route estimation unit 70.
  • Landmark selection unit 40 is included in landmark information acquisition unit 30 .
  • the landmark information acquisition unit 30 of the external world recognition device 1a acquires the three-dimensional information of the plurality of landmarks 100, 101, and 102
  • the landmark information acquisition unit 30 determines the location of the plurality of landmarks according to the movement of the vehicle V. Select a landmark (eg, landmark 100) to be used for distance estimation.
  • the landmark selection unit 40 of the landmark information acquisition unit 30 selects the landmark (for example, the landmark 100) used for distance estimation from a plurality of landmarks.
  • the landmark selection unit 40 selects a landmark 100 existing in the traveling direction of the vehicle V estimated by the traveling path estimation unit 70 described later as a landmark used for estimating the distance to the pedestrian 200. select.
  • the landmark information storage section 36 stores the landmark 100 selected by the landmark selection section 40 .
  • the traveling path estimation unit 70 estimates the traveling direction of the vehicle V from the movement of the vehicle V (that is, behavior information). Specifically, the traveling route estimation unit 70 predicts the traveling route of the own vehicle V based on the speed of the vehicle V, the yaw rate, and the like. For example, the traveling path estimator 70 estimates the turning radius of the vehicle V from the values of the vehicle speed and the yaw rate, and sets the locus on the circumference as the traveling path. Note that, when the vehicle V is traveling straight ahead, the traveling path estimation unit 70 may estimate a trajectory along the straight traveling direction as the traveling path.
  • the landmark registration process P105
  • only the landmark for example, the landmark 100
  • the traveling direction estimated by the traveling path estimation unit 70 is registered as a landmark.
  • the landmark information storage unit 36 may store landmarks 100 that are within a predetermined distance (20 m) from the course of the vehicle V. FIG.
  • FIG. 13 is a functional block diagram showing a schematic configuration of an external world recognition device 1b according to the third embodiment of the present invention.
  • the external world recognition device 1b according to the third embodiment differs from the external world recognition device 1d (Modification 1) in that it includes a landmark selection unit 40 and a surrounding environment recognition unit 90, which will be described later.
  • Configurations having the same or similar functions as those of the external world recognition device 1d described above are denoted by the same reference numerals as those of the external world recognition device 1d, and descriptions thereof are omitted. Different parts will be explained.
  • the external world recognition device 1b includes a landmark selection unit 40 and a surrounding environment recognition unit 90 that recognizes the surrounding environment of the vehicle V.
  • Landmark selection unit 40 is included in landmark information acquisition unit 30 .
  • the landmark information acquisition unit 30 of the external world recognition device 1b recognizes the landmarks (for example, the landmark 100) existing in the area recognized as the sidewalk by the surrounding environment recognition unit 90 up to the pedestrian 200. as landmarks for distance estimation.
  • the landmark selection unit 40 of the landmark information acquisition unit 30 selects the landmark (for example, the landmark 100) used for estimating the distance to the pedestrian 200.
  • the landmark information storage unit 36 stores the landmarks selected by the landmark selection unit 40 .
  • the surrounding environment recognition unit 90 estimates type information for each pixel of the images of the pair of cameras 10 a and 10 b acquired by the image acquisition unit 10 .
  • the type information includes not only information on objects such as vehicles and pedestrians, but also information on road surfaces, sidewalks, and the like.
  • a convolutional neural network is used to estimate the type information for each pixel.
  • the surrounding environment recognition unit 90 estimates the type information corresponding to each pixel by using a model learned from the correct value data to which the type information for each pixel is added. As a result, the surrounding environment recognition unit 90 can recognize the sidewalk area from the images captured by the pair of cameras 10a and 10b.
  • the landmark registration process is executed using the type information estimated by the surrounding environment recognition unit 90.
  • the landmark information storage unit 36 stores the landmark 100 that exists on the pixel that the surrounding environment recognition unit 90 has estimated as the sidewalk.
  • the pair of cameras 10a and 10b are used as one stereo camera in order to acquire the three-dimensional information (size information) of the landmarks 100, 101, and 102 in the landmark information acquisition unit 30.
  • a three-dimensional sensor for example, lidar, millimeter wave radar, ultrasonic sensor, etc.
  • the pair of cameras 10a and 10b may be installed so as not to have a stereo area.
  • the camera 10 a may capture only the left monocular region
  • the camera 10 b may capture only the right monocular region
  • the images captured in these regions may be input to the image acquisition unit 10 .
  • the distance between the vehicle V and the pedestrian 200 may be estimated using specific landmarks described on a digital map or the like and using three-dimensional information registered in the digital map. . Further, in the above-described embodiment, a detailed description was given from the viewpoint of estimating the height of pedestrian 200, but based on the width of pedestrian 200, the distance between own vehicle V and pedestrian 200 is calculated. good too.
  • the distance estimating unit 50 estimates the distance to the pedestrian 200 based on the actual width of the landmark 100 and the lateral length (i.e. image width) of the landmark 100 and the pedestrian 200 in the image.
  • the method of estimating the distance to the pedestrian 200 has been described by taking the pedestrian 200 as an example of the object, but various objects such as vehicles and animals can be treated as objects for distance estimation.
  • the distance between these and the host vehicle V can be estimated based on the heights of the V and the like.
  • the present invention is not limited to the above embodiments, and includes various modifications.
  • the above embodiments have been described in detail in order to explain the present invention in an easy-to-understand manner, and are not necessarily limited to those having all the configurations described.
  • it is possible to replace part of the configuration of one embodiment with the configuration of another embodiment and it is also possible to add the configuration of another embodiment to the configuration of one embodiment.
  • each of the above configurations, functions, processing units, processing means, etc. may be realized by hardware, for example, by designing them in integrated circuits, in part or in whole.
  • each of the above configurations, functions, etc. may be realized by software by a processor interpreting and executing a program for realizing each function.
  • Information such as programs, tapes, and files that implement each function can be stored in recording devices such as memories, hard disks, SSDs (solid state drives), or recording media such as IC cards, SD cards, and DVDs.
  • control lines and information lines indicate what is considered necessary for explanation, and not all control lines and information lines are necessarily indicated on the product. In practice, it may be considered that almost all configurations are interconnected.
  • 1, 1a, 1b, 1d, 1e external world recognition device 10 image acquisition unit, 10a, 10b pair of cameras (stereo cameras), 30 landmark information acquisition unit, 32 compound eye region landmark detection unit, 33 size information measurement unit , 34 landmark information measurement unit, 36 landmark information storage unit, 40 landmark selection unit, 50 distance estimation unit, 52 monocular area landmark detection unit, 54 monocular area object detection unit, 56 proximity determination unit, 58 size Estimation unit 60 Distance calculation unit 70 Course estimation unit 90 Surrounding environment recognition unit 100, 101, 102 Landmark 200 Pedestrian (object) CA Compound eye area image LA Left monocular area image (monocular area image ), RA right monocular area image (monocular area image), V vehicle, ⁇ margin (proximity judgment threshold)

Landscapes

  • Physics & Mathematics (AREA)
  • Electromagnetism (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Radar, Positioning & Navigation (AREA)
  • Remote Sensing (AREA)
  • Measurement Of Optical Distance (AREA)
  • Image Processing (AREA)
  • Traffic Control Systems (AREA)

Abstract

L'invention concerne un dispositif de reconnaissance d'environnement externe permettant de calculer avec précision la distance à un objet indépendamment de l'environnement d'utilisation. Le dispositif de reconnaissance d'environnement externe est monté sur un véhicule. Le dispositif de reconnaissance d'environnement externe comprend une unité d'obtention d'image qui obtient une image d'un point de repère et d'un objet ; une unité d'obtention d'informations de point de repère qui obtient des informations tridimensionnelles concernant le point de repère ; et une unité d'estimation de distance qui estime la distance à l'objet sur la base des informations tridimensionnelles concernant le point de repère et les tailles du point de repère et de l'objet dans l'image.
PCT/JP2022/038648 2021-11-26 2022-10-17 Dispositif de reconnaissance d'environnement externe WO2023095489A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021-192274 2021-11-26
JP2021192274A JP2023078934A (ja) 2021-11-26 2021-11-26 外界認識装置

Publications (1)

Publication Number Publication Date
WO2023095489A1 true WO2023095489A1 (fr) 2023-06-01

Family

ID=86539281

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/038648 WO2023095489A1 (fr) 2021-11-26 2022-10-17 Dispositif de reconnaissance d'environnement externe

Country Status (2)

Country Link
JP (1) JP2023078934A (fr)
WO (1) WO2023095489A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014171052A1 (fr) * 2013-04-16 2014-10-23 コニカミノルタ株式会社 Procede de traitement d'image, dispositif de traitement d'image, dispositif de capture d'image, et programme de traitement d'image
WO2017090410A1 (fr) * 2015-11-25 2017-06-01 日立オートモティブシステムズ株式会社 Dispositif de caméra stéréoscopique
JP2018132477A (ja) * 2017-02-17 2018-08-23 日本電信電話株式会社 深度推定装置、寸法推定装置、深度推定方法、寸法推定方法、及びプログラム
WO2021070537A1 (fr) * 2019-10-08 2021-04-15 日立Astemo株式会社 Dispositif de reconnaissance d'objet

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2014171052A1 (fr) * 2013-04-16 2014-10-23 コニカミノルタ株式会社 Procede de traitement d'image, dispositif de traitement d'image, dispositif de capture d'image, et programme de traitement d'image
WO2017090410A1 (fr) * 2015-11-25 2017-06-01 日立オートモティブシステムズ株式会社 Dispositif de caméra stéréoscopique
JP2018132477A (ja) * 2017-02-17 2018-08-23 日本電信電話株式会社 深度推定装置、寸法推定装置、深度推定方法、寸法推定方法、及びプログラム
WO2021070537A1 (fr) * 2019-10-08 2021-04-15 日立Astemo株式会社 Dispositif de reconnaissance d'objet

Also Published As

Publication number Publication date
JP2023078934A (ja) 2023-06-07

Similar Documents

Publication Publication Date Title
US11087148B2 (en) Barrier and guardrail detection using a single camera
JP7025912B2 (ja) 車載環境認識装置
JP4650079B2 (ja) 物体検出装置、および方法
US9846812B2 (en) Image recognition system for a vehicle and corresponding method
US10129521B2 (en) Depth sensing method and system for autonomous vehicles
US7612800B2 (en) Image processing apparatus and method
US20180137376A1 (en) State estimating method and apparatus
US8428305B2 (en) Method for detecting a clear path through topographical variation analysis
US9690993B2 (en) Method and device for analyzing trafficability
US9697421B2 (en) Stereoscopic camera apparatus
US20090052742A1 (en) Image processing apparatus and method thereof
JP6711395B2 (ja) 画像処理装置、撮像装置、移動体機器制御システム、移動体、画像処理方法、及びプログラム
JP6816401B2 (ja) 画像処理装置、撮像装置、移動体機器制御システム、画像処理方法、及びプログラム
CN111164648B (zh) 移动体的位置推断装置及位置推断方法
JP2016099650A (ja) 走行路認識装置及びそれを用いた走行支援システム
Petrovai et al. A stereovision based approach for detecting and tracking lane and forward obstacles on mobile devices
CN113874914A (zh) 用于求取牵引机与牵引机的挂车之间的运行角度
WO2019065970A1 (fr) Dispositif de reconnaissance extérieur de véhicule
US20200193184A1 (en) Image processing device and image processing method
EP3410345B1 (fr) Appareil de traitement d'informations et support d'enregistrement non transitoire stockant un programme informatique
KR20180047149A (ko) 충돌 위험 경고 장치 및 방법
US20220153259A1 (en) Autonomous parking systems and methods for vehicles
Ma et al. A real time object detection approach applied to reliable pedestrian detection
WO2023095489A1 (fr) Dispositif de reconnaissance d'environnement externe
JP5785515B2 (ja) 歩行者検出装置及び方法、並びに車両用衝突判定装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22898273

Country of ref document: EP

Kind code of ref document: A1