WO2023162761A1 - 障害物検知方法、プログラムおよび障害物検知装置 - Google Patents

障害物検知方法、プログラムおよび障害物検知装置 Download PDF

Info

Publication number
WO2023162761A1
WO2023162761A1 PCT/JP2023/004859 JP2023004859W WO2023162761A1 WO 2023162761 A1 WO2023162761 A1 WO 2023162761A1 JP 2023004859 W JP2023004859 W JP 2023004859W WO 2023162761 A1 WO2023162761 A1 WO 2023162761A1
Authority
WO
WIPO (PCT)
Prior art keywords
depth
pixels
depth pixel
pixel
value
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2023/004859
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
慧 上田
正樹 金丸
哲郎 奥山
良直 河合
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Nuvoton Technology Corp Japan
Original Assignee
Nuvoton Technology Corp Japan
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Nuvoton Technology Corp Japan filed Critical Nuvoton Technology Corp Japan
Priority to EP23759764.6A priority Critical patent/EP4485356A4/en
Priority to JP2024503041A priority patent/JPWO2023162761A1/ja
Priority to CN202380022873.XA priority patent/CN118871950A/zh
Publication of WO2023162761A1 publication Critical patent/WO2023162761A1/ja
Priority to US18/797,129 priority patent/US20240395008A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/20—Image preprocessing
    • G06V10/26—Segmentation of patterns in the image field; Cutting or merging of image elements to establish the pattern region, e.g. clustering-based techniques; Detection of occlusion
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/60—Analysis of geometric attributes
    • G06T7/64—Analysis of geometric attributes of convexity or concavity
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
    • G06V10/74—Image or video pattern matching; Proximity measures in feature spaces
    • G06V10/75—Organisation of the matching processes, e.g. simultaneous or sequential comparisons of image or video features; Coarse-fine approaches, e.g. multi-scale approaches; using context analysis; Selection of dictionaries
    • G06V10/751—Comparing pixel values or logical combinations thereof, or feature values having positional relevance, e.g. template matching
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds

Definitions

  • the present disclosure relates to an obstacle detection method, program, and obstacle detection device for detecting obstacles.
  • Patent Document 1 discloses a technique for generating a travelable route by determining locally flat voxels based on normal vectors associated with voxels.
  • Patent Document 1 for a group of objects including a rough surface that is globally flat but locally uneven, the local unevenness is determined to be an obstacle, and the vehicle travels. It can be judged impossible.
  • the present disclosure provides an obstacle detection method and the like capable of accurately detecting obstacles in a group of objects including a rough surface that is globally flat but locally uneven. .
  • An obstacle detection method is an obstacle detection method for detecting an obstacle, wherein a plurality of depth pixels in a depth image obtained by sensing an object group including a rough surface with a depth sensor include: a determination step of determining exclusion depth pixels to be excluded from obstacle candidates; and an obstacle detection step of detecting an obstacle based on depth pixels excluding the exclusion depth pixels among the plurality of depth pixels.
  • a depth pixel value of the depth pixel and each of a plurality of first peripheral depth pixels within a first number of pixels from the depth pixel (i) calculating an average value for the depth pixel based on the depth pixel value and associating the calculated average value with the depth pixel; (iii) a process of calculating a normal based on the average value associated with each of the plurality of second peripheral depth pixels within the range of the second number of pixels; (iv) determining whether or not the angle formed with the normal line is equal to or less than a predetermined angle; and perform the process of determining.
  • a program according to the present disclosure is a program that causes a computer to execute the above obstacle detection method.
  • An obstacle detection device is an obstacle detection device that detects an obstacle, and among a plurality of depth pixels in a depth image obtained by sensing an object group including a rough surface with a depth sensor, a determination unit that determines excluded depth pixels to be excluded from obstacle candidates; and a detection unit that detects an obstacle based on remaining depth pixels after the excluded depth pixels are excluded from the plurality of depth pixels.
  • the determining unit performs (i) a depth pixel value of the depth pixel and the depth of each of a plurality of peripheral depth pixels within a predetermined number of pixels from the depth pixel; (ii) a process of calculating an average value for the depth pixel based on the pixel value and associating the calculated average value with the depth pixel; (iii) the calculated normal and the normal of a predetermined reference plane; (iv) a process of determining whether or not the angle formed is equal to or less than a predetermined angle; and (iv) a process of determining the depth pixel as the excluded depth pixel if it is determined that the formed angle is equal to or less than the predetermined angle. and
  • an obstacle detection method and the like it is possible to accurately detect an obstacle in an object group including a rough surface that is flat in general but locally uneven. can be done.
  • FIG. 1 is a block diagram showing an example of an obstacle detection device according to an embodiment; FIG. It is a figure for demonstrating a rough surface.
  • 4 is a flow chart showing an example of an obstacle detection method according to the embodiment;
  • FIG. 4 is a flow chart illustrating an example of determining steps for determining excluded depth pixels according to an embodiment;
  • FIG. It is a figure for demonstrating the calculation method of an average depth pixel value.
  • FIG. 4 is a diagram schematically showing depth pixels used for calculating an average depth pixel value and depth pixels not used for calculating an average depth pixel value;
  • FIG. 11 is a diagram showing an example of effective second peripheral depth pixels used for calculating a normal calculation plane;
  • the planted crop group is determined to be a non-obstacle, and while working in the planted crop group, the upper body comes out from the tip surface of the planted crop group. It is necessary to detect a person who is present as an obstacle. For example, when an object with a height equal to or higher than the threshold is detected as an obstacle using a certain height as a threshold, if the threshold is set too high, a person cannot be detected, which may lead to a serious accident. On the other hand, if the threshold value is too low, many planted crops will be detected as obstacles, and the agricultural machinery may stop frequently, reducing work efficiency. If the threshold is determined based on the height in this way, there is a risk that undetected or erroneous detection will occur frequently.
  • the top surface of the planted crop group is flat, it is determined that the top surface of the planted crop group is a road surface on which it is possible to travel.
  • Crop groups can be determined as non-obstacles. Also, if there is a person whose upper body is protruding from the tip surface of the planted crop group, the part where the person is is not flat, and the person can be determined as an obstacle.
  • the height of the planted crops is not constant, and even if the top surface of the planted crops is flat globally, it has a rough surface including irregularities locally. Local irregularities may be detected as obstacles even when no one is present.
  • An obstacle detection method is an obstacle detection method for detecting an obstacle, and a plurality of depth pixels in a depth image obtained by sensing an object group including a rough surface with a depth sensor. a determination step of determining excluded depth pixels to be excluded from obstacle candidates; and an obstacle detection step of detecting an obstacle based on depth pixels other than the excluded depth pixels among the plurality of depth pixels.
  • the determining step includes: (i) a depth pixel value of the depth pixel and a plurality of first peripheral depth pixels within a first number of pixels from the depth pixel; (ii) a process of calculating an average value for the depth pixel based on each depth pixel value of the depth pixel and associating the calculated average value with the depth pixel; (ii) the average value associated with the depth pixel; (iii) a process of calculating a normal based on the average value associated with each of the plurality of second peripheral depth pixels within the range of the second number of pixels from the depth pixel; (iv) a process of determining whether or not an angle formed with a normal to a reference plane is equal to or less than a predetermined angle; and determining the depth pixel.
  • the depth pixel value of the depth pixel and the depth pixel values of the surrounding plurality of first peripheral depth pixels is used to compute the average value for that depth pixel, and the average value is used to compute the normal for determining the excluded depth pixels. That is, the values related to the depth pixels (for example, the depth pixel values or the coordinate values of the depth pixels) corresponding to the local irregularities are averaged so that the local irregularities become nearly flat (in other words, The angle formed by the calculated normal line and the normal line of the predetermined reference plane is less than or equal to the predetermined angle), and local unevenness can be excluded from obstacle candidates.
  • the average depth pixel value associated with the depth pixel and a second number of pixels from the depth pixel The normal may be calculated based on an average depth pixel value associated with each of the plurality of second surrounding depth pixels within the range.
  • the average depth pixel value of the depth pixel values of the surrounding plurality of first peripheral depth pixels is calculated. and the average depth pixel value is used to compute a normal for determining excluded depth pixels.
  • the depth pixel values of the depth pixels corresponding to the local unevenness are averaged to make the local unevenness nearly flat (in other words, the calculated normal and the normal of the predetermined reference plane).
  • the angle formed with the line is less than a predetermined angle), and local unevenness can be excluded from obstacle candidates.
  • the depth pixel value of the depth pixel and one or more depth pixel values having a difference between the depth pixel value and the depth pixel value of the plurality of first peripheral depth pixels is equal to or less than a predetermined threshold.
  • An average depth pixel value may be calculated for each depth pixel value of the first surrounding depth pixels, and the calculated average depth pixel value may be associated with the depth pixel.
  • the average depth pixel value associated with the depth pixel and the plurality of valid second peripheral depth pixels among the plurality of second peripheral depth pixels associated with each of the The normal may be calculated based on the average depth pixel value obtained. For example, if there is no valid second peripheral depth pixel or only one valid second peripheral depth pixel among the plurality of second peripheral depth pixels, the depth pixel may be determined as an obstacle candidate.
  • one coordinate value in three-dimensional coordinate values obtained by point group transformation from the depth pixel value of the depth pixel, and a plurality of coordinates within a first pixel number from the depth pixel calculating an average coordinate value between each depth pixel value of the first surrounding depth pixels and one coordinate value in the three-dimensional coordinate values obtained by point group transformation, associating the calculated average coordinate value with the depth pixel;
  • an average coordinate value associated with the depth pixel and an average coordinate value associated with each of a plurality of second peripheral depth pixels within a range of a second number of pixels from the depth pixel You may calculate a normal line based on and.
  • the coordinate values of the plurality of surrounding first peripheral depth pixels (for example, the coordinate values corresponding to the height ) is calculated, and the average coordinate is used to calculate the normal for determining the excluded depth pixels.
  • the coordinate values of the depth pixels corresponding to the locations with local unevenness are averaged, and the local unevenness becomes nearly flat (in other words, the calculated normal and the normal of the predetermined reference plane is equal to or less than a predetermined angle), local irregularities can be excluded from obstacle candidates. Note that there is a large difference between the coordinate values of depth pixels corresponding to places where people exist and the coordinate values of depth pixels corresponding to places where people do not exist.
  • a difference in depth pixel value between the one coordinate value of the depth pixel and the depth pixel among the plurality of first peripheral depth pixels is equal to or less than a predetermined threshold value.
  • An average coordinate value between each of the first peripheral depth pixels and the one coordinate value may be calculated, and the calculated average coordinate value may be associated with the depth pixel.
  • the depth pixels having a small difference in the coordinate value are averaged. It is possible to prevent the average coordinate values from being affected by outliers.
  • the average coordinate value associated with the depth pixels and the The normal may be calculated based on the average coordinate value. For example, if there is no valid second peripheral depth pixel or only one valid second peripheral depth pixel among the plurality of second peripheral depth pixels, the depth pixel may be determined as an obstacle candidate.
  • the group of objects may include a rough surface including unevenness formed by the tips of the group of planted crops.
  • the normal to the predetermined reference plane may be the normal to the ground on which the planted crop group grows. For example, further calculating the normal line of the ground based on an average value associated with the depth pixels corresponding to the ground and an average value associated with each of the plurality of second peripheral depth pixels.
  • the normal to the ground may be acquired by a sensor capable of acquiring the normal to the ground.
  • a program according to one aspect of the present disclosure is a program that causes a computer to execute the obstacle detection method described above.
  • An obstacle detection device is an obstacle detection device that detects an obstacle, and includes a plurality of depth pixels in a depth image obtained by sensing an object group including a rough surface with a depth sensor. Among them, a determination unit that determines excluded depth pixels to be excluded from obstacle candidates, and a detection unit that detects an obstacle based on the remaining depth pixels after the excluded depth pixels are excluded from the plurality of depth pixels.
  • the determination unit includes: (i) a depth pixel value of the depth pixel and a plurality of peripheral depth pixels within a predetermined number of pixels from the depth pixel; (ii) calculating an average value for the depth pixels based on each depth pixel value and associating the calculated average value with the depth pixel; and (ii) the average value associated with the depth pixel and the depth. (iii) a process of calculating a normal based on the average value associated with each of a plurality of peripheral depth pixels within a range of a predetermined number of pixels from the pixel; (iv) determining whether or not the angle formed with the normal line is equal to or less than a predetermined angle; and perform the process of determining.
  • an obstacle detection device that can accurately detect obstacles in a group of objects including a rough surface that is flat globally but has unevenness locally.
  • FIG. 1 is a block diagram showing an example of an obstacle detection device 10 according to an embodiment.
  • the obstacle detection device 10 is a device that detects obstacles.
  • the obstacle detection device 10 is mounted on an agricultural machine running in a field, and detects a person working in a group of planted crops in the field as an obstacle.
  • the detected obstacles are not limited to people, and may be other agricultural machines or the like.
  • Planted crops are, for example, grass crops such as rice, wheat, and millet.
  • a field is an example of a place where obstacle detection is performed.
  • the location where obstacle detection is performed may be a construction site or the like.
  • the obstacle detection device 10 may be mounted on a construction machine or the like running at a construction site, and may detect a person or the like working at the construction site as an obstacle.
  • the detected obstacle is not limited to a person, and may be another construction machine or the like.
  • the obstacle detection device 10 includes a determination unit 11 and a detection unit 12 as functional components.
  • the obstacle detection device 10 is a computer including a processor and memory.
  • the memory is ROM (Read Only Memory), RAM (Random Access Memory), etc., and can store programs executed by the processor.
  • the determination unit 11 and the detection unit 12 are implemented by a processor or the like that executes a program stored in a memory.
  • the determination unit 11 determines exclusion depth pixels to be excluded from obstacle candidates, among a plurality of depth pixels in a depth image obtained by sensing an object group including a rough surface with a depth sensor.
  • a depth sensor is mounted, for example, on an agricultural machine or a construction machine.
  • the depth sensor may be based on a stereo camera system or may be based on a ToF (Time of Flight) system.
  • a plurality of depth pixels in the depth image obtained by the depth sensor includes not only positional information in the depth image but also depth pixel value (that is, depth indicating depth) information. That is, depth pixels can be viewed as 3D points.
  • the group of objects will be described as including a rough surface including unevenness formed by the tips of the group of planted crops.
  • the depth sensor is installed on the ceiling side of an agricultural machine or construction machine, and can sense the tip surface (rough surface) of the planted crop group from obliquely above the planted crop group.
  • the tip surface rough surface
  • FIG. 2 is a diagram for explaining the rough surface.
  • FIG. 2 shows a cross-section of the rough surface.
  • each planted crop in the planted crop group does not have the same height and is locally uneven.
  • each planted crop in the planted crop group appears to have a substantially constant height.
  • a surface that is globally flat but locally uneven, such as the top surface of a planted crop group, is called a rough surface here.
  • the detection unit 12 detects an obstacle based on the depth pixels remaining after the exclusion depth pixels are excluded from the plurality of depth pixels. For example, the detection unit 12 outputs the detection result to another device (for example, a device that controls agricultural machinery or construction machinery).
  • another device for example, a device that controls agricultural machinery or construction machinery.
  • FIG. 3A is a flowchart showing an example of an obstacle detection method according to the embodiment. Since the obstacle detection method is a method executed by the obstacle detection device 10, FIG. 3A is also an example of a flowchart showing the operation of the obstacle detection device 10. FIG.
  • the determining unit 11 determines excluded depth pixels to be excluded from obstacle candidates, among a plurality of depth pixels in a depth image obtained by sensing an object group including a rough surface with a depth sensor (step S11: decision step).
  • step S11 determination step
  • the details of step S11 determination step
  • the details of the operation of the determination unit 11 will be described with reference to FIG. 3B.
  • FIG. 3B is a flowchart illustrating an example of determination steps for determining exclusion depth pixels according to the embodiment.
  • the determining unit 11 performs the processing from step S101 to step S107 shown in FIG. 3B for each of the plurality of depth pixels in the depth image.
  • a depth pixel to be processed among the plurality of depth pixels is referred to as a target depth pixel.
  • the determination unit 11 determines whether the target depth pixel is a valid depth pixel (step S101).
  • An invalid depth pixel is a depth pixel whose depth pixel value is abnormal due to the influence of noise or sunlight. Since it is unknown whether or not an obstacle exists at a position corresponding to such a depth pixel, and there is a possibility that an obstacle exists, the determining unit 11 determines if the target depth pixel is not a valid depth pixel (step If No in S101), the target depth pixel is determined as an obstacle candidate (step S102).
  • the determination unit 11 determines the depth pixel value of the target depth pixel and the range of the first number of pixels from the target depth pixel. Based on the depth pixel value of each of a plurality of first surrounding depth pixels, the average value of the target depth pixel is calculated, and the calculated average value is associated with the depth pixel (step S103). For example, the determination unit 11 determines the average depth pixel value of the depth pixel value of the target depth pixel and the depth pixel values of each of the plurality of first peripheral depth pixels within the range of the first number of pixels from the target depth pixel. and associate the calculated average depth pixel value with the depth pixel of interest.
  • the determining unit 11 determines one or more first depth pixels whose difference in depth pixel value between the target depth pixel and the target depth pixel among the plurality of first peripheral depth pixels is equal to or less than a predetermined threshold.
  • An average depth pixel value is calculated for each depth pixel value of the peripheral depth pixels, and the calculated average depth pixel value is associated with the depth pixel of interest.
  • a method for calculating the average depth pixel value will be described with reference to FIG.
  • FIG. 4 is a diagram for explaining the method of calculating the average depth pixel value.
  • FIG. 4 shows a depth pixel of interest (the location indicated as 40) and a plurality of first peripheral depth pixels within the range of the first number of pixels from the depth pixel of interest (areas around the location indicated as 40). 8 points), and each numerical value indicates a depth pixel value.
  • the first number of pixels is set to one. That is, the plurality of first peripheral depth pixels are set as depth pixels adjacent to the target depth pixel.
  • the first number of pixels is not limited to 1, and is appropriately set according to, for example, the type of object group (for example, the type of planted crops).
  • one or more first surrounding depth pixels whose difference in depth pixel value from the target depth pixel is equal to or less than a predetermined threshold are hatched.
  • the predetermined threshold is 10
  • one or more first peripheral depth pixels whose difference in depth pixel value from the target depth pixel (depth pixel value of 40) is 10 or less are the target depth pixels.
  • the predetermined threshold value is not limited to 10, and is appropriately set according to, for example, the type of object group (for example, the type of planted crops).
  • FIG. 5 is a diagram schematically showing depth pixels used for calculating the average depth pixel value and depth pixels not used for calculating the average depth pixel value.
  • depth pixel A is a target depth pixel
  • depth pixel B is a depth pixel used for calculating the average depth pixel value
  • depth pixel C is not used for calculating the average depth pixel value.
  • the difference between the height of the planted crop corresponding to depth pixel A and the height of the planted crop corresponding to depth pixel B is small.
  • the difference between the height of standing crops is large.
  • the depth pixel B has a smaller difference in depth pixel value than the depth pixel A
  • the depth pixel C has a larger difference in depth pixel value than the depth pixel A.
  • the depth pixel B has a smaller difference in depth pixel value than the depth pixel A, so it is used for the averaging process.
  • depth pixel C has a larger difference in depth pixel value than depth pixel A, so it is excluded from the averaging process.
  • depth pixels with small differences in depth pixel value are averaged. It is possible to prevent the average depth pixel value from being affected by an outlier (for example, the depth pixel value of depth pixel C).
  • the average depth pixel value is calculated and associated in the same manner. For example, for a depth pixel with a depth pixel value of 35 shown in FIG. The depth pixel value of one or more first surrounding depth pixels among the depth pixels below (not shown) whose difference in depth pixel value from the target depth pixel is equal to or less than a predetermined threshold (for example, 10). An average depth pixel value is calculated and the depth pixel of interest is associated with the average depth pixel value.
  • a predetermined threshold for example, 10
  • the determining unit 11 determines the average value associated with the target depth pixel and the second peripheral depth pixels within the range of the second number of pixels from the target depth pixel.
  • a normal line is calculated based on the associated average value (step S104). For example, the determination unit 11 determines the average depth pixel value associated with the target depth pixel, and the plurality of second peripheral depth pixels within the range of the second number of pixels from the target depth pixel. A normal is calculated based on the average depth pixel value.
  • Each of the plurality of second peripheral depth pixels is also processed in step S103 in the same manner as the target depth pixel, and the average depth pixel value is calculated and associated.
  • the determination unit 11 determines the average depth pixel value associated with the target depth pixel, and the average depth associated with each of the plurality of valid second peripheral depth pixels among the plurality of second peripheral depth pixels.
  • a normal calculation plane is calculated based on the pixel values, and the normal of the calculated normal calculation plane is calculated. Although the details will be described later, the normal is used when determining whether or not to determine the target depth pixel as an excluded depth pixel.
  • FIG. 1 an example of valid second peripheral depth pixels is shown in FIG.
  • FIG. 6 is a diagram showing an example of effective second peripheral depth pixels used for calculating the normal calculation plane. Locations marked with asterisks are depth pixels of interest, locations marked with circles are valid second surrounding depth pixels, and locations marked with crosses are non-valid second surrounding depth pixels. .
  • the second number of pixels is set to 1, that is, a plurality of second peripheral depth pixels are set as depth pixels adjacent to the target depth pixel.
  • the first peripheral depth pixel and the second peripheral depth pixel for the depth pixel of interest may be the same pixel.
  • the second number of pixels is not limited to 1, and may be appropriately set according to the type of object group (for example, the type of planted crops).
  • a second peripheral depth pixel that is not valid is a depth pixel whose depth pixel value is abnormal due to the influence of noise or sunlight, and is not used to calculate the normal calculation plane.
  • the determining unit 11 determines the average depth pixel value associated with the target depth pixel and the position of the target depth pixel in the depth image (that is, the 3D point of the target depth pixel), and the valid second peripherals the average depth pixel value associated with the depth (e.g., depth pixels to the left, bottom left, right, and top right of the depth pixel of interest shown in FIG. 6) and the positions in the depth image of the valid plurality of second surrounding depth pixels; Based on this, the normal calculation plane is calculated. If there are three or more valid second peripheral depth pixels, the equation of the normal calculation plane can be obtained by the least squares method.
  • Equation 1 the left side of Equation 1 can be expressed as in Equation 3 below.
  • the normal can be calculated from the outer product of the 3D points of the target depth pixel and the two valid second surrounding depth pixels. However, in this case, it is sensitive to outliers.
  • the normal line cannot be calculated if there is no or only one valid second peripheral depth pixel among the plurality of second peripheral depth pixels.
  • the normal calculation plane is computed, and the cross product This is because it is not possible to calculate
  • the determination unit 11 determines whether or not the normal can be calculated (step S105). It is determined as a candidate (step S102). This is because there is a possibility that an obstacle exists at a position corresponding to the target depth pixel because it cannot be determined whether or not to determine the target depth pixel as the excluded depth pixel.
  • the determining unit 11 determines whether or not the angle formed by the calculated normal line and the normal line of the predetermined reference plane is equal to or less than a predetermined angle (step S106). ).
  • the normal of the predetermined reference plane is, for example, the normal of the ground on which the planted crops grow.
  • the determination unit 11 determines the average value (average depth pixel value) associated with the depth pixels corresponding to the ground, and each of the plurality of second peripheral depth pixels (for example, the plurality of effective second peripheral depth pixels).
  • the normal to the ground may be calculated based on the associated average value (average depth pixel value).
  • the determining unit 11 may calculate the normal line of the ground using a sensor capable of acquiring the normal line of the ground, such as an inclination sensor mounted on an agricultural machine or a construction machine.
  • the predetermined angle may be appropriately set according to the type of object group (for example, the type of planted crops). If the angle formed by the calculated normal and the normal to the predetermined reference plane is equal to or less than the predetermined angle, the depth pixel periphery of the target is substantially parallel to the predetermined reference plane, and there are no obstacles. Probability is high. If the angle formed by the calculated normal line and the normal line of the predetermined reference plane is larger than the predetermined angle, the target depth pixel periphery is tilted with respect to the predetermined reference plane, and there is a possibility that there is an obstacle. There is a sensor capable of acquiring the normal line of the ground, such as an inclination sensor mounted on an agricultural machine or a construction machine.
  • the predetermined angle may be appropriately set according to the type of object group (for example, the type of planted crops
  • the determining unit 11 determines the target depth pixel as the excluded depth pixel. (Step S107). As described above, in this case, there is a high probability that there is no obstacle, so the depth pixel of interest is excluded from obstacle candidates. When determining that the angle formed by the calculated normal and the normal to the predetermined reference plane is larger than the predetermined angle (No in step S106), the determination unit 11 determines the target depth pixel as an obstacle candidate. (step S102). As mentioned above, in this case there may be an obstacle, so the target depth pixel is not excluded from the obstacle candidates.
  • the determination unit 11 determines exclusion depth pixels to be excluded from obstacle candidates.
  • the detection unit 12 detects an obstacle based on the depth pixels remaining after the exclusion depth pixels are excluded from the plurality of depth pixels (step S12: obstacle detection step). For example, the detection unit 12 uses the density of the 3D point cloud of the remaining depth pixels, the luminance value used for depth calculation in the remaining depth pixels, and the like to detect the presence or absence of obstacles, the type of obstacles, and the like. conduct. For example, the detection unit 12 may consider point groups that are close to each other as the same object and combine them (clustering), and detect clusters in which the number of point groups is equal to or greater than a certain value as an obstacle. Further, the detection unit 12 may detect an obstacle using an image captured by a camera (which may be a black and white image) and an image recognition algorithm using the image.
  • a camera which may be a black and white image
  • the average depth pixel value of the depth pixel values of the surrounding plurality of first surrounding depth pixels is are calculated and the average depth pixel value is used to calculate the normal for determining the excluded depth pixels.
  • the depth pixel values of the depth pixels corresponding to the local unevenness are averaged to make the local unevenness nearly flat (in other words, the calculated normal and the normal of the predetermined reference plane).
  • the angle formed with the line is less than a predetermined angle), and local unevenness can be excluded from obstacle candidates.
  • the present invention is not limited to this.
  • An example of determining the excluded depth pixels using the coordinate values of the depth pixels calculated from the depth pixel values of the depth pixels will be described below.
  • the determination unit 11 determines one coordinate value in the three-dimensional coordinate values obtained by point group transformation from the depth pixel value of the target depth pixel, and the third coordinate value from the target depth pixel. Calculate the average coordinate value of each of the depth pixel values of the plurality of first peripheral depth pixels within the range of one pixel number and one coordinate value in the three-dimensional coordinate values obtained by the point group transformation, and calculate the calculated average coordinate Map the value to the depth pixel of interest.
  • the three-dimensional coordinate values are the X coordinate value, Y coordinate value and Z coordinate value in the world coordinate system.
  • the X coordinate value and the Y coordinate value are horizontal coordinate values
  • the Z coordinate value is a vertical coordinate value
  • the origin is the position of the ground vertically below the depth sensor. That is, the Z coordinate value is a coordinate value corresponding to height.
  • the determination unit 11 determines the position of the target depth pixel in the depth image, the depth pixel value of the target depth pixel (distance from the depth sensor), the angle of the depth sensor in the optical axis direction, and the installation position of the depth sensor. By performing point group transformation using the height from the ground, it is possible to calculate the three-dimensional coordinate values of the target depth pixels.
  • One coordinate value in the three-dimensional coordinate value is the coordinate value corresponding to height, that is, the Z coordinate value.
  • the determining unit 11 selects one or more first depth pixels whose difference between the Z coordinate value of the target depth pixel and the target depth pixel among the plurality of first surrounding depth pixels is equal to or less than a predetermined threshold value.
  • An average coordinate value with respect to the Z coordinate value of each peripheral depth pixel is calculated, and the calculated average coordinate value is associated with the target depth pixel.
  • the depth pixel value can be replaced with the Z coordinate value
  • the average depth pixel value can be replaced with the average coordinate value.
  • the determining unit 11 determines the average coordinate value associated with the target depth pixel and the average coordinate value associated with each of the plurality of second peripheral depth pixels within the range of the second number of pixels from the target depth pixel. and the normal is calculated. For each of the plurality of second peripheral depth pixels, the average coordinate value is calculated and associated in the same manner as the target depth pixel. For example, the determining unit 11 determines the average coordinate value associated with the target depth pixel, and the average coordinate value associated with each of the plurality of effective second peripheral depth pixels among the plurality of second peripheral depth pixels. A normal calculation plane is calculated based on and a normal to the calculated normal calculation plane is calculated.
  • the determination unit 11 determines the average coordinate value associated with the target depth pixel and the three-dimensional coordinate value (position) of the target depth pixel, and a plurality of valid second peripheral depths (for example, the target depth shown in FIG. 6). For normal calculation, based on the average coordinate values associated with the left, lower left, right, and upper right depth pixels of the depth pixels of ) and the three-dimensional coordinate values (positions) of the plurality of valid second peripheral depth pixels Calculate the plane. The details of the method for calculating the normal are the same as the case of determining the excluded depth pixels using the depth pixel values of the depth pixels, so description thereof will be omitted.
  • the determination unit 11 determines whether or not the normal can be calculated, and if the normal cannot be calculated, determines the target depth pixel as an obstacle candidate.
  • the determination unit 11 determines whether or not the angle formed by the calculated normal line and the normal line of the predetermined reference plane is equal to or less than a predetermined angle.
  • the normal of the predetermined reference plane is, for example, the normal of the ground on which the planted crops grow.
  • the determination unit 11 calculates the average coordinate value associated with the depth pixels corresponding to the ground, and the average coordinate value associated with each of the plurality of second peripheral depth pixels (for example, the plurality of effective second peripheral depth pixels).
  • the normal to the ground may be calculated based on the coordinate values.
  • the determining unit 11 may calculate the normal line of the ground using a sensor capable of acquiring the normal line of the ground, such as an inclination sensor mounted on an agricultural machine or a construction machine.
  • the determination unit 11 determines that the angle formed by the calculated normal and the normal to the predetermined reference plane is equal to or less than a predetermined angle, the determination unit 11 determines the target depth pixel as an exclusion depth pixel.
  • the determination unit 11 may determine exclusion depth pixels to be excluded from the obstacle candidates using the coordinate values of the depth pixels calculated from the depth pixel values of the depth pixels.
  • each of the plurality of depth pixels is compared with the coordinate values (for example, the Z coordinate value) of the plurality of surrounding first peripheral depth pixels.
  • An average coordinate value is calculated and used to calculate a normal for determining excluded depth pixels.
  • the coordinate values of the depth pixels corresponding to the locations with local unevenness are averaged, and the local unevenness becomes nearly flat (in other words, the calculated normal and the normal of the predetermined reference plane is equal to or less than a predetermined angle), local irregularities can be excluded from obstacle candidates. Note that there is a large difference between the coordinate values of depth pixels corresponding to places where people exist and the coordinate values of depth pixels corresponding to places where people do not exist.
  • a predetermined threshold among the plurality of first peripheral depth pixels.
  • An example of calculating the average depth pixel value of each depth pixel value of one peripheral depth pixel has been described, but the present invention is not limited to this.
  • the average depth pixel value may be calculated including the depth pixel values of the first surrounding depth pixels whose difference in depth pixel value from the target depth pixel is not equal to or less than a predetermined threshold.
  • the difference between the coordinate value (for example, the Z coordinate value) of the target depth pixel and the target depth pixel among the plurality of first surrounding depth pixels is equal to or less than a predetermined threshold.
  • the average coordinate value may be calculated including the coordinate values of the first surrounding depth pixels whose difference in coordinate value from the target depth pixel is not equal to or less than a predetermined threshold.
  • the present disclosure can be implemented as a program for causing a processor to execute the steps included in the obstacle detection method.
  • the present disclosure can be implemented as a non-temporary computer-readable recording medium such as a CD-ROM recording the program.
  • each step is executed by executing the program using hardware resources such as the CPU, memory, and input/output circuits of the computer.
  • hardware resources such as the CPU, memory, and input/output circuits of the computer.
  • each step is executed by the CPU acquiring data from a memory or an input/output circuit, etc., performing an operation, or outputting the operation result to the memory, an input/output circuit, or the like.
  • each component included in the obstacle detection device 10 may be configured with dedicated hardware or realized by executing a software program suitable for each component.
  • Each component may be realized by reading and executing a software program recorded in a recording medium such as a hard disk or a semiconductor memory by a program execution unit such as a CPU or processor.
  • a part or all of the functions of the obstacle detection device 10 according to the above embodiment are typically implemented as an LSI, which is an integrated circuit. These may be made into one chip individually, or may be made into one chip so as to include part or all of them. Further, circuit integration is not limited to LSIs, and may be realized by dedicated circuits or general-purpose processors.
  • An FPGA Field Programmable Gate Array
  • a reconfigurable processor that can reconfigure the connections and settings of the circuit cells inside the LSI may be used.
  • the present disclosure also includes various modifications in which a person skilled in the art makes modifications to each embodiment of the present disclosure, as long as they do not deviate from the gist of the present disclosure.
  • the present disclosure can be applied to devices that detect obstacles in fields and the like.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Computing Systems (AREA)
  • Artificial Intelligence (AREA)
  • Health & Medical Sciences (AREA)
  • Databases & Information Systems (AREA)
  • Evolutionary Computation (AREA)
  • General Health & Medical Sciences (AREA)
  • Medical Informatics (AREA)
  • Software Systems (AREA)
  • Geometry (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
PCT/JP2023/004859 2022-02-24 2023-02-13 障害物検知方法、プログラムおよび障害物検知装置 Ceased WO2023162761A1 (ja)

Priority Applications (4)

Application Number Priority Date Filing Date Title
EP23759764.6A EP4485356A4 (en) 2022-02-24 2023-02-13 OBSTACLE DETECTION METHOD, PROGRAM, AND OBSTACLE DETECTION DEVICE
JP2024503041A JPWO2023162761A1 (https=) 2022-02-24 2023-02-13
CN202380022873.XA CN118871950A (zh) 2022-02-24 2023-02-13 障碍物检测方法、程序及障碍物检测装置
US18/797,129 US20240395008A1 (en) 2022-02-24 2024-08-07 Obstruction detection method, recording medium, and obstruction detection device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2022027286 2022-02-24
JP2022-027286 2022-02-24

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/797,129 Continuation US20240395008A1 (en) 2022-02-24 2024-08-07 Obstruction detection method, recording medium, and obstruction detection device

Publications (1)

Publication Number Publication Date
WO2023162761A1 true WO2023162761A1 (ja) 2023-08-31

Family

ID=87765773

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/004859 Ceased WO2023162761A1 (ja) 2022-02-24 2023-02-13 障害物検知方法、プログラムおよび障害物検知装置

Country Status (5)

Country Link
US (1) US20240395008A1 (https=)
EP (1) EP4485356A4 (https=)
JP (1) JPWO2023162761A1 (https=)
CN (1) CN118871950A (https=)
WO (1) WO2023162761A1 (https=)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011186749A (ja) * 2010-03-08 2011-09-22 Optex Co Ltd 距離画像における平面推定方法および距離画像カメラ
JP2021007386A (ja) * 2019-06-28 2021-01-28 株式会社クボタ 自動走行システム
JP2021085828A (ja) * 2019-11-29 2021-06-03 株式会社豊田自動織機 障害物検出装置

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113848931B (zh) * 2021-10-09 2022-09-27 上海联适导航技术股份有限公司 农机自动驾驶障碍物识别方法、系统、设备和存储介质

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011186749A (ja) * 2010-03-08 2011-09-22 Optex Co Ltd 距離画像における平面推定方法および距離画像カメラ
JP2021007386A (ja) * 2019-06-28 2021-01-28 株式会社クボタ 自動走行システム
JP2021085828A (ja) * 2019-11-29 2021-06-03 株式会社豊田自動織機 障害物検出装置

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
See also references of EP4485356A4

Also Published As

Publication number Publication date
US20240395008A1 (en) 2024-11-28
EP4485356A4 (en) 2025-06-11
CN118871950A (zh) 2024-10-29
JPWO2023162761A1 (https=) 2023-08-31
EP4485356A1 (en) 2025-01-01

Similar Documents

Publication Publication Date Title
US10101746B2 (en) Automated vehicle road model definition system
US10776953B2 (en) Method for identification of candidate points as possible characteristic points of a calibration pattern within an image of the calibration pattern
JP5943084B2 (ja) 画像処理装置、画像処理方法および画像処理プログラム
US8755562B2 (en) Estimation apparatus, control method thereof, and program
KR102073468B1 (ko) 비전 시스템에서 컬러 이미지에 대해 컬러 후보 포즈들의 점수화를 위한 시스템 및 방법
US20180357509A1 (en) Method for identification of characteristic points of a calibration pattern within a set of candidate points in an image of the calibration pattern
WO2011114947A1 (ja) 横断歩道検出装置、横断歩道検出方法及びプログラム
CN112597846A (zh) 车道线检测方法、装置、计算机设备和存储介质
US20150198735A1 (en) Method of Processing 3D Sensor Data to Provide Terrain Segmentation
CN113436336A (zh) 地面点云分割方法和装置及自动驾驶车辆
CN119152032B (zh) 一种用于结构光参数标定的激光光斑阵列检测方法及系统
CN119784817A (zh) 基于双目视觉的路面平整度检测方法及系统
CN119579418B (zh) 补点处理方法、装置、设备及存储介质
WO2023162761A1 (ja) 障害物検知方法、プログラムおよび障害物検知装置
Cho et al. Vision-based uncut crop edge detection for automated guidance of head-feeding combine
CN118230018A (zh) 地面断崖识别方法、装置、电子设备及可读存储介质
WO2023068034A1 (ja) 画像処理装置
CN118913220A (zh) 障碍物测距方法、装置、电子设备及存储介质
US12020464B2 (en) Method of determining an orientation of an object and a method and apparatus for tracking an object
CN113450335B (zh) 一种路沿检测方法、路沿检测装置及路面施工车辆
Bhowmick et al. A novel floor segmentation algorithm for mobile robot navigation
JP7064400B2 (ja) 物体検知装置
Orlov et al. Machine vision system for autonomous agricultural vehicle
CN120783253A (zh) 悬崖检测处理方法、自移动设备及存储介质
CN119741362A (zh) 货架检测方法、电子设备和计算机可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23759764

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2024503041

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 202380022873.X

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 2023759764

Country of ref document: EP

NENP Non-entry into the national phase

Ref country code: DE

ENP Entry into the national phase

Ref document number: 2023759764

Country of ref document: EP

Effective date: 20240924