WO2021237738A1 - 自动驾驶方法和装置、距离确定方法和装置 - Google Patents

自动驾驶方法和装置、距离确定方法和装置 Download PDF

Info

Publication number
WO2021237738A1
WO2021237738A1 PCT/CN2020/093521 CN2020093521W WO2021237738A1 WO 2021237738 A1 WO2021237738 A1 WO 2021237738A1 CN 2020093521 W CN2020093521 W CN 2020093521W WO 2021237738 A1 WO2021237738 A1 WO 2021237738A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
image area
distance
depth information
belonging
Prior art date
Application number
PCT/CN2020/093521
Other languages
English (en)
French (fr)
Inventor
王涛
李思晋
刘政哲
李鑫超
Original Assignee
深圳市大疆创新科技有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳市大疆创新科技有限公司 filed Critical 深圳市大疆创新科技有限公司
Priority to PCT/CN2020/093521 priority Critical patent/WO2021237738A1/zh
Priority to CN202080005812.9A priority patent/CN112912892A/zh
Publication of WO2021237738A1 publication Critical patent/WO2021237738A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/58—Recognition of moving objects or obstacles, e.g. vehicles or pedestrians; Recognition of traffic objects, e.g. traffic signs, traffic lights or roads
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/24—Classification techniques
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods

Definitions

  • the present invention relates to the field of automatic driving, in particular to an automatic driving method, a distance determining method, an automatic driving device, a distance determining device, a movable platform and a machine-readable storage medium.
  • the present invention proposes an automatic driving method, a distance determining method, an automatic driving device, a distance determining device, a movable platform and a machine-readable storage medium to solve technical problems in related technologies.
  • an automatic driving method is proposed, which is applied to a vehicle, and the vehicle is equipped with a camera device, and the camera device is used to obtain visible light images, and the method includes:
  • the distance between the object belonging to the first object category and the vehicle is determined.
  • a distance determination method is proposed, which is applied to a movable platform on which a camera device is mounted, and includes:
  • the visible light image is acquired by the camera device;
  • the distance between the object belonging to the first object category and the movable platform is determined.
  • an automatic driving device which is applied to a vehicle, and the vehicle is equipped with a camera device and a processor, and the processor is configured to execute the distance determination method described in any of the above embodiments Steps in.
  • a distance determining device is provided, which is applied to a movable platform on which a camera device and a processor are mounted, and the processor is configured to execute the method described in any of the above embodiments. The steps in the distance determination method.
  • a movable platform including:
  • a power system arranged in the body, and the power system is used to provide power for the movable platform;
  • a camera device which is provided in the body, and the camera device is used to obtain a visible light image
  • processors configured to execute the steps in the distance determination method described in any of the foregoing embodiments.
  • a machine-readable storage medium which is suitable for a removable platform.
  • the machine-readable storage medium stores a number of computer instructions, and the computer instructions are configured to execute any of the foregoing. Steps in the distance determination method described in an embodiment.
  • the distance between the object of the first object type and the vehicle can be accurately determined, so that the vehicle can accurately determine the distance between the object of the first object type and the vehicle. Responsive actions are helpful to ensure driving safety.
  • Fig. 1 is a schematic flowchart showing an automatic driving method according to an embodiment of the present disclosure.
  • Fig. 2 is a schematic flowchart showing another automatic driving method according to an embodiment of the present disclosure.
  • Fig. 3 is a schematic flowchart showing yet another automatic driving method according to an embodiment of the present disclosure.
  • Fig. 4 is a schematic diagram showing a neural network according to an embodiment of the present disclosure.
  • Fig. 5 is a schematic flowchart showing yet another automatic driving method according to an embodiment of the present disclosure.
  • Fig. 6 is a schematic flowchart showing yet another automatic driving method according to an embodiment of the present disclosure.
  • FIGS 7A to 7D are schematic diagrams of application scenarios of the automatic driving method shown in an embodiment of the present disclosure.
  • Fig. 8 is a schematic flowchart of a method for determining a distance according to an embodiment of the present disclosure.
  • Fig. 9 is a schematic flowchart showing another method for determining a distance according to an embodiment of the present disclosure.
  • Fig. 10 is a schematic flowchart showing another method for determining a distance according to an embodiment of the present disclosure.
  • Fig. 11 is a schematic flowchart showing another method for determining a distance according to an embodiment of the present disclosure.
  • Fig. 12 is a schematic flowchart showing yet another method for determining a distance according to an embodiment of the present disclosure.
  • Fig. 1 is a schematic flowchart showing an automatic driving method according to an embodiment of the present disclosure.
  • the automatic driving method shown in this embodiment may be applied to a vehicle.
  • the vehicle may be an unmanned vehicle.
  • the vehicle may be equipped with a camera, such as a camera, a video recorder, etc., which can acquire visible light images.
  • the automatic driving method may include the following steps:
  • step S101 the object types corresponding to multiple first image regions in the visible light image are identified through a preset target recognition algorithm; the object types include a first object type and a second object type;
  • step S102 the visible light image is mapped to a depth image, and the object types corresponding to the multiple second image areas in the depth image are determined according to the object types corresponding to the multiple first image areas;
  • step S103 determine the target image area of the object belonging to the first object type among the plurality of second image areas according to the object types corresponding to the plurality of second image areas in the depth image;
  • step S104 the distance between the object belonging to the first object category and the vehicle is determined according to the depth information of the target image area.
  • the target recognition algorithm may be a neural network trained in advance through machine learning, such as a convolutional neural network.
  • the neural network can be trained to recognize the object type corresponding to each area in the image.
  • the object type may include a first object type and a second object type.
  • the first object type may be a dynamic object
  • the second object type The category can be a static object.
  • the steps in the embodiments of the present disclosure can be mainly performed when objects of the second object type occlude objects of the first object type.
  • the first object type is a vehicle and the second object type is a road.
  • fence or green belt on the side The current self-driving car needs to determine the distance between it and the vehicle in the field of view to avoid a collision between two cars and a traffic accident.
  • the fence or green belt on the roadside partially obstructs the vehicles in its field of vision, the current autonomous vehicles will misidentify that the fence or green belt on the roadside is also a vehicle, resulting in misdetection of distance and misoperation.
  • the probability of misjudgment can be reduced, the decision-making accuracy of autonomous driving can be improved, and a better autonomous driving experience can be provided to users.
  • the object of the second object type occludes the object of the first object type
  • it can be selected according to needs, and is not limited to the following example.
  • the first preset algorithm for example, a model obtained through machine learning in advance
  • the specific situation may be that the object of the first object type in the image is the side image of the vehicle, and only If the front light is not displayed in the image, it can be determined that the object of the first object type is not blocked by the object of the second object type; if the object integrity of the first object type is less than the preset integrity, such as less than 95%, such as the above In the side image of the vehicle, the entire front of the vehicle is not displayed in the image, and it can be determined that the object of the first object type is occluded by the object of the second object type.
  • the mapping relationship between the visible light image and the depth image may be predetermined.
  • the first image area in the visible light image will also be mapped to the depth image to form the second image area, then the object type corresponding to the first image area is the object type corresponding to the second image area .
  • the target image area belonging to the first object type may be further determined in the second image area. Since the target image area is located in the depth image, the depth information of the target image area can be determined, and the target image area is composed of objects corresponding to the first object type, so objects of the first object type can be determined according to the depth information of the target image area According to the depth information of the object of the first object type, the distance between the object belonging to the first object type and the vehicle to which the method is applied can be determined.
  • the distance between the object of the first object type and the vehicle can be accurately determined, so that the vehicle can accurately respond according to the distance. Action is conducive to ensuring driving safety.
  • the depth image is obtained by binocular camera equipment or lidar mounted on the vehicle.
  • the vehicle may also be equipped with a binocular camera device or a lidar, and then the depth image may be obtained by using a binocular camera device or a lidar.
  • Fig. 2 is a schematic flowchart showing another automatic driving method according to an embodiment of the present disclosure. As shown in FIG. 2, before determining the distance between the object belonging to the first object category and the vehicle according to the depth information of the target image area, the method further includes:
  • step S105 according to the depth information of each point cloud in the depth image, cluster the point clouds in the depth image to generate a clustering result
  • step S106 correct the target image area according to the clustering result
  • the determining the distance between the object belonging to the first object type and the vehicle according to the depth information of the target image area includes:
  • step S1041 the distance between the object belonging to the first object category and the vehicle is determined according to the depth information of the corrected target image area.
  • the acquired depth map may include multiple point clouds, and each point cloud may have its own depth information. Then the point clouds may be clustered according to the depth information of the point clouds, for example, point clouds with close depths If the clustering is one type, the point clouds belonging to the same type in the clustering result are more likely to belong to the same object, and the target image area can be corrected according to the clustering result.
  • the target image area can be expanded so that the expanded target image area contains the first part of the point cloud.
  • Part of the point cloud and the second part of the point cloud help ensure that the corrected target image area contains every part of the object of the first object type, and then determine the object belonging to the first object type according to the depth information of the corrected target image area
  • the distance to the vehicle helps to ensure the accuracy of determining the distance.
  • the point clouds belonging to the same category in the clustering results are all located in the target image area, but the target image area includes not only the type A point cloud, but also a small part of other types of point clouds, then the target The image area is reduced so that the reduced target image contains all the point clouds of type A and does not contain other types of point clouds, which helps to ensure that the corrected target image area only contains objects of the first object type, and does not include other object types
  • the distance between the object belonging to the first object category and the vehicle is determined, which is beneficial to ensure the accuracy of the distance determination.
  • Fig. 3 is a schematic flowchart showing yet another automatic driving method according to an embodiment of the present disclosure.
  • the recognition of object types corresponding to multiple first image regions in the visible light image by using a preset target recognition algorithm includes:
  • step S1011 the confidence that each first image area in the visible light image belongs to each type of object is recognized by the target recognition algorithm
  • step S1012 the object type corresponding to each first image area is determined according to the confidence level.
  • the target recognition algorithm may be, for example, a neural network, and the neural network may be trained to recognize the confidence that each first image region in the image belongs to each object category.
  • Fig. 4 is a schematic diagram showing a neural network according to an embodiment of the present disclosure.
  • the neural network can be obtained by step-by-step training.
  • the neural network can include multiple sequentially connected modules, and the modules can have a forward propagation (Skip connection) relationship.
  • Each module includes a convolutional layer Conv, Batch normalization layer bn and linear rectification layer Relu.
  • the input of the neural network can be expressed as N*4*H*W, where H represents the height of the image, W represents the width of the image, N represents the number of images, and 4 represents the number of image channels, such as red (R), green ( There are 4 channels in G), Blue (B) and Depth.
  • the output of the neural network can be expressed as a tensor N*K*H*W, where the meaning of N is the same as the meaning of the corresponding parameter in the input, and K represents the identification of the type of object, where each type of object can be preset Identification, so that after determining the type of object, it can be indicated by the corresponding identification.
  • five types of objects can be roughly divided in advance, namely, vehicles, sky, road surface, dynamic objects, and static objects.
  • the sign of the sky can be set to 16, and the sign of the road can be set to 1.
  • the identification for cars is 19, the identification for trucks is 20, the identification for buses is 21, the identification for caravans is 22, the identification for trains is 24, the identification for trailers is 23, and the identification for tricycles is 28.
  • the identification of the construction vehicle is 27.
  • the building is marked as 5, the wall is marked as 6, the fence is marked as 7, the guardrail is marked as 8, the bridge is marked as 9, the tunnel is marked as 10, and the pillar is marked.
  • the mark of traffic light is 12
  • the mark of traffic sign is 13
  • the mark of plants is 14, and the mark of terrain is 15.
  • the identifier for pedestrians is 17
  • the identifier for starting hands is 18
  • the identifier for motorcycles is 25
  • the identifier for bicycles is 26.
  • K can be used to indicate the type of object in the output.
  • H and W in the output can represent the height and width of the object belonging to the K-corresponding object type in the image. Then, according to H and W, the corresponding object type of each K can be determined.
  • the area corresponding to the object in the image such as the first image area in the foregoing embodiment.
  • K in the output can also include the confidence that the first image area belongs to the object type (which can also be expressed as a probability).
  • the first image area belongs to the confidence level of each object type, so that the object type corresponding to each first image area can be determined according to the confidence level.
  • the first image area can be determined according to the confidence. Which object type should the image area correspond to? For example, the object type with the highest confidence level can be regarded as the object type corresponding to the first image area. For example, the first image area has a 20% confidence level for pillars, a 30% confidence level for plants, and a 50% confidence level for pedestrians. The maximum confidence level is 50%, and the corresponding object type is a pedestrian, then it can be determined that the first image area belongs to a pedestrian.
  • Fig. 5 is a schematic flowchart showing yet another automatic driving method according to an embodiment of the present disclosure. As shown in FIG. 5, before determining the distance between the object belonging to the first object category and the vehicle according to the depth information of the target image area, the method further includes:
  • step S107 a third image area of an object belonging to the second object type among the plurality of second image areas is determined according to the object types corresponding to the plurality of second image areas in the depth image;
  • the determining the distance between the object belonging to the first object category and the vehicle according to the depth information of the target image area includes:;
  • step S1042 the distance between the object belonging to the first object category and the vehicle is determined according to the depth information of the target image area and the depth information of the third image area.
  • Objects of the first object type are partially occluded by objects of the second object type.
  • the first image area contains both objects belonging to the first object type and objects belonging to the second object type.
  • the second image area mapped to the depth image also contains both objects belonging to the first object type and objects belonging to the second object type.
  • the object belonging to the second object type in the second image area is the third image area, and the object belonging to the first object type in the second image area is blocked. Then the distance between the object belonging to the first object type and the vehicle is not only
  • the depth information of the target image area can be considered, and the depth information of the third image area can also be considered, that is, the objects belonging to the first object category and the movable platform are determined according to the depth information of the target image area and the depth information of the third image area the distance.
  • this embodiment Comprehensively consider the depth information of the target image area and the depth information of the third image area to determine the distance between the object belonging to the first object category and the vehicle, so that in the case of misjudgment of some pixels, the misjudgment of pixels can also be considered Depth information helps to ensure the accuracy of determining the distance.
  • the first image area of the object belonging to the second object type and the first image area of the object belonging to the first object type are adjacent in the visible light image.
  • one situation is that some pixels in the object belonging to the first object type are misjudged as objects belonging to the second object type, and then these pixels are in the visible light image.
  • the corresponding first image area and another part of the pixels belonging to the first object category should be adjacent (specifically adjacent).
  • the distance between the object belonging to the first object category and the vehicle is determined according to the depth information of the target image area and the depth information of the third image area.
  • the judgment result is generally accurate, so there is no need to base it on
  • the depth information of the target image area and the depth information of the third image area determine the distance between the object belonging to the first object type and the vehicle, but only the distance between the object belonging to the first object type and the vehicle can be determined according to the depth information of the target image area .
  • Fig. 6 is a schematic flowchart showing yet another automatic driving method according to an embodiment of the present disclosure.
  • the determining the distance between the object belonging to the first object category and the vehicle according to the depth information of the target image area and the depth information of the third image area includes:
  • step S10421 the depth information of the target image area is weighted by the first weight, and the depth information of the third image area is weighted by the second weight, and the sum of the two is calculated to obtain The distance between the object belonging to the first object category and the vehicle.
  • the distance between the object belonging to the first object category and the vehicle is determined according to the depth information of the target image area and the depth information of the third image area.
  • the depth information of the target image area may be weighted by the first weight.
  • the first weight value and the second weight value can be set according to needs. In general, the first weight value can be set to be greater than the second weight value.
  • the target recognition algorithm is a convolutional neural network.
  • the convolutional neural network includes multiple sets of layer structures, and each set of layer structures includes a convolution layer, a batch normalization layer, and a linear rectification layer.
  • the convolutional neural network includes a residual network.
  • FIGS 7A to 7D are schematic diagrams of application scenarios of the automatic driving method shown in an embodiment of the present disclosure.
  • Figure 7A it is a schematic view of the vehicle's perspective. It can be seen in the figure that there are vehicles on the left front, and pedestrians on the right side of the vehicle, where the rear half of the vehicle is blocked by the plants in the green belt.
  • the corresponding color can be set for each object type in advance, and then the image shown in FIG. 7A can be input into the convolutional neural network in the above-mentioned embodiment, then it can be determined that the pixel in each area of the image belongs to the object Object type, and then color the pixels according to the object type.
  • the coloring result is shown in Figure 7B. Based on the coloring result shown in Figure 7B, vehicles, pedestrians, roads, traffic lights, plants in the green belt, etc. can be clearly distinguished object.
  • coloring in FIG. 7C is optional, and subsequent rendering operations may be performed after obtaining FIG. 7B, instead of performing subsequent rendering after obtaining FIG. 7C.
  • the image can be further rendered according to the coloring result, as shown in Fig. 7D, so that The rendered image is close to the color of the objects in the display scene, and can highlight the objects that are likely to affect driving such as vehicles and pedestrians, so that according to the rendering results, you can accurately determine which objects in the image are objects that need attention , And then measure the distance of these objects, so as to effectively avoid objects that are closer, so as to ensure driving safety in the process of automatic driving.
  • Fig. 8 is a schematic flowchart of a method for determining a distance according to an embodiment of the present disclosure.
  • the automatic driving method shown in this embodiment can be applied to a movable platform equipped with a camera device, such as a camera, a video recorder, etc., and the camera device can acquire visible light images.
  • a camera device such as a camera, a video recorder, etc.
  • the distance determination method may include the following steps:
  • step S201 the types of objects corresponding to the multiple first image regions in the visible light image are identified through a preset target recognition algorithm; the visible light image is acquired by the camera device;
  • step S202 map the visible light image to a depth image, and determine the object types corresponding to the multiple second image areas in the depth image according to the object types corresponding to the multiple first image areas;
  • step S203 determine the target image area belonging to the first object type among the multiple second image areas according to the object types corresponding to the multiple second image areas in the depth image;
  • step S204 the distance between the object belonging to the first object category and the movable platform is determined according to the depth information of the target image area.
  • the vehicle in which the vehicle is located can accurately respond to the distance, which is beneficial to ensure driving safety.
  • the movable platform is a car.
  • the first object type is a dynamic object.
  • the first object category includes automobiles.
  • the depth image is obtained by binocular camera equipment or lidar mounted on the movable platform.
  • Fig. 9 is a schematic flowchart showing another method for determining a distance according to an embodiment of the present disclosure. As shown in FIG. 9, before determining the distance between the object belonging to the first object category and the movable platform according to the depth information of the target image area, the method further includes:
  • step S205 clustering the point clouds in the depth image according to the depth information of each point cloud in the depth image to generate a clustering result
  • step S206 correct the target image area according to the clustering result
  • the determining the distance between the object belonging to the first object type and the movable platform according to the depth information of the target image area includes:
  • step S2041 the distance between the object belonging to the first object category and the movable platform is determined according to the corrected depth information of the target image area.
  • the acquired depth map may include multiple point clouds, and each point cloud may have its own depth information. Then the point clouds may be clustered according to the depth information of the point clouds, for example, point clouds with close depths If the clustering is one type, the point clouds belonging to the same type in the clustering result are more likely to belong to the same object, and the target image area can be corrected according to the clustering result.
  • the target image area can be expanded so that the expanded target image area contains the first part of the point cloud.
  • Part of the point cloud and the second part of the point cloud help ensure that the corrected target image area contains every part of the object of the first object type, and then determine the object belonging to the first object type according to the depth information of the corrected target image area
  • the distance to the vehicle helps to ensure the accuracy of determining the distance.
  • the point clouds belonging to the same category in the clustering results are all located in the target image area, but the target image area includes not only the type A point cloud, but also a small part of other types of point clouds, then the target The image area is reduced so that the reduced target image contains all the point clouds of type A and does not contain other types of point clouds, which helps to ensure that the corrected target image area only contains objects of the first object type, and does not include other object types
  • the distance between the object belonging to the first object category and the vehicle is determined, which is beneficial to ensure the accuracy of the distance determination.
  • Fig. 10 is a schematic flowchart showing another method for determining a distance according to an embodiment of the present disclosure.
  • the identification of object types corresponding to multiple first image regions in the visible light image through a preset target recognition algorithm includes:
  • step S2011 the confidence that each first image area in the visible light image belongs to each type of object is recognized by the target recognition algorithm
  • step S2012 the object type corresponding to each first image area is determined according to the confidence level.
  • the target recognition algorithm may be, for example, a neural network, and the neural network may be trained to recognize the confidence that each first image region in the image belongs to each object category.
  • a neural network as shown in Figure 4 can be used.
  • the first image area As for a certain first image area, it is difficult to 100% determine the type of object it belongs to, but it can be determined that it can belong to multiple object types, and the confidence of belonging to each object type, and then the first image can be determined according to the confidence Which object type should the area correspond to? For example, the object type with the highest confidence level can be regarded as the object type corresponding to the first image area. For example, the first image area has a 20% confidence level for pillars, 30% confidence level for plants, and 50% confidence level for pedestrians. The maximum confidence level is 50%, and the corresponding object type is a pedestrian, then it can be determined that the first image area belongs to a pedestrian.
  • Fig. 11 is a schematic flowchart showing another method for determining a distance according to an embodiment of the present disclosure. As shown in FIG. 11, before determining the distance between the object belonging to the first object category and the movable platform according to the depth information of the target image area, the method further includes:
  • step S207 determine a third image area belonging to the second object type among the plurality of second image areas according to the object types corresponding to the plurality of second image areas in the depth image;
  • the determining the distance between the object belonging to the first object type and the movable platform according to the depth information of the target image area includes:
  • step S2042 the distance between the object belonging to the first object category and the movable platform is determined according to the depth information of the target image area and the depth information of the third image area.
  • the first image area belonging to the second object type and the first image area belonging to the first object type are adjacent in the visible light image.
  • the first image area contains both objects belonging to the first object type and objects belonging to the second object type, which are mapped to the depth image
  • the second image area in, also contains both objects belonging to the first object type and objects belonging to the second object type.
  • an object belonging to the second object type in the second image area is in the third image area, and the object belonging to the first object type in the second image area is blocked, then the distance between the object belonging to the first object type and the vehicle, Not only can the depth information of the target image area be considered, but also the depth information of the third image area, that is, according to the depth information of the target image area and the depth information of the third image area to determine the objects and movable objects belonging to the first object category The distance of the platform.
  • this embodiment Comprehensively consider the depth information of the target image area and the depth information of the third image area to determine the distance between the object belonging to the first object category and the vehicle, so that in the case of misjudgment of some pixels, the misjudgment of pixels can also be considered Depth information helps to ensure the accuracy of determining the distance.
  • Fig. 12 is a schematic flowchart showing yet another method for determining a distance according to an embodiment of the present disclosure. As shown in FIG. 12, the determining the distance between the object belonging to the first object category and the movable platform according to the depth information of the target image area and the depth information of the third image area includes:
  • step S20421 the depth information of the target image area is weighted by the first weight, and the depth information of the third image area is weighted by the second weight, and the sum of the two is calculated to obtain The distance between the object belonging to the first object category and the movable platform.
  • the distance between the object belonging to the first object category and the vehicle is determined according to the depth information of the target image area and the depth information of the third image area.
  • the depth information of the target image area may be weighted by the first weight.
  • the first weight value and the second weight value can be set according to needs. In general, the first weight value can be set to be greater than the second weight value.
  • the target recognition algorithm is a convolutional neural network.
  • the convolutional neural network includes multiple sets of layer structures, and each set of layer structures includes a convolution layer, a batch normalization layer, and a linear rectification layer.
  • the convolutional neural network includes a residual network.
  • the embodiment of the present disclosure also proposes an automatic driving device, which is applied to a vehicle, and the vehicle is equipped with a camera device and a processor, and the processor is configured to execute the distance determination method described in any of the above embodiments. step.
  • the embodiment of the present disclosure also proposes a distance determining device, which is applied to a movable platform on which a camera device and a processor are mounted, and the processor is used to execute the distance as described in any of the above embodiments. Identify the steps in the method.
  • the embodiment of the present disclosure also proposes a movable platform, including:
  • a power system arranged in the body, and the power system is used to provide power for the movable platform;
  • a camera device which is provided in the body, and the camera device is used to obtain a visible light image
  • processors configured to execute the steps in the distance determination method described in any of the foregoing embodiments.
  • the movable platform is a drone, an autonomous vehicle, or the like.
  • the embodiment of the present disclosure also proposes a machine-readable storage medium suitable for a removable platform.
  • the machine-readable storage medium stores a number of computer instructions, and the computer instructions are configured to execute any of the above-mentioned embodiments. The steps in the distance determination method.
  • the systems, devices, modules, or units explained in the foregoing embodiments may be specifically implemented by computer chips or entities, or implemented by products with certain functions.
  • the functions are divided into various units and described separately.
  • the functions of each unit can be implemented in the same one or more software and/or hardware.
  • the embodiments of the present invention can be provided as a method, a system, or a computer program product. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware.
  • the present invention may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

Abstract

本公开涉及自动驾驶方法，包括：通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类；将可见光图像映射到深度图像，并根据多个第一图像区域所对应的物体种类确定深度图像中多个第二图像区域所对应的物体种类；根据深度图像中多个第二图像区域所对应的物体种类，确定多个第二图像区域中属于第一物体种类物体的目标图像区域；根据目标图像区域的深度信息，确定属于第一物体种类的物体与车辆的距离。根据本公开的实施例，即使在可见光图像中第一物体种类的物体被第二物体种类的物体所遮挡，也能够准确地确定第一物体种类的物体与车辆的距离。

Description

自动驾驶方法和装置、距离确定方法和装置 技术领域
本发明涉及自动驾驶领域，尤其涉及自动驾驶方法、距离确定方法、自动驾驶装置、距离确定装置、可移动平台和机器可读存储介质。
背景技术
为了实现车辆等交通工具的自动驾驶，识别出车辆周围的物体以便进行避让，是不可或缺的技术。
为了有效地进行避让，一方面需要识别出车辆周围的物体，另一方面还需要确定物体到车辆的距离。但是在实际行驶场景中，车辆周围会存在很多物体，物体之间存在遮挡的情况，这会导致对于被遮挡的物体，难以准确地确定其到车辆的距离。
发明内容
本发明提出了自动驾驶方法、距离确定方法、自动驾驶装置、距离确定装置、可移动平台和机器可读存储介质，以解决相关技术中的技术问题。
根据本公开实施例的第一方面，提出一种自动驾驶方法，应用于一车辆，所述车辆上搭载有摄像装置，所述摄像装置用于获取可见光图像，所述方法包括：
通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类；所述物体种类包括第一物体种类和第二物体种类；
将所述可见光图像映射到深度图像，并根据多个所述第一图像区域所对应的物体种类确定所述深度图像中多个第二图像区域所对应的物体种类；
根据所述深度图像中多个第二图像区域所对应的物体种类，确定多个所述第二图像区域中属于第一物体种类物体的目标图像区域；
根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离。
根据本公开实施例的第二方面，提出一种距离确定方法，应用于一可移动平台，所述可移动平台上搭载有摄像装置，包括：
通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类；所述可见光图像由所述摄像装置获取；
将所述可见光图像映射到深度图像，并根据多个所述第一图像区域所对应的物体种类确定所述深度图像中多个第二图像区域所对应的物体种类；
根据所述深度图像中多个第二图像区域所对应的物体种类，确定多个所述第二图像区域中属于第一物体种类的目标图像区域；
根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离。
根据本公开实施例的第三方面，提出一种自动驾驶装置，应用于一车辆，所述车辆上搭载有摄像装置以及处理器，所述处理器用于执行上述任一实施例所述距离确定方法中的步骤。
根据本公开实施例的第四方面，提出一种距离确定装置，应用于一可移动平台，所述可移动平台上搭载有摄像装置以及处理器，所述处理器用于执行上述任一实施例所述距离确定方法中的步骤。
根据本公开实施例的第五方面，提出一种可移动平台，包括：
机体；
动力系统，设于所述机体，所述动力系统用于为所述可移动平台提供动力；
摄像装置，设于所述机体，所述摄像装置用于获取可见光图像；
以及一个或多个处理器，用于执行上述任一实施例所述距离确定方法中的步骤。
根据本公开实施例的第六方面，提出一种机器可读存储介质，适用于可 移动平台，所述机器可读存储介质上存储有若干计算机指令，所述计算机指令被被配置为执行上述任一实施例所述距离确定方法中的步骤。
根据本公开的实施例，即使在可见光图像中第一物体种类的物体被第二物体种类的物体所遮挡，也能够准确地确定第一物体种类的物体与车辆的距离，以便车辆根据该距离准确地做出响应动作，有利于确保行驶安全。
附图说明
为了更清楚地说明本发明实施例中的技术方案，下面将对实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本发明的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1是根据本公开的实施例示出的一种自动驾驶方法的示意流程图。
图2是根据本公开的实施例示出的另一种自动驾驶方法的示意流程图。
图3是根据本公开的实施例示出的又一种自动驾驶方法的示意流程图。
图4是根据本公开的实施例示出的一种神经网络的示意图。
图5是根据本公开的实施例示出的又一种自动驾驶方法的示意流程图。
图6是根据本公开的实施例示出的又一种自动驾驶方法的示意流程图。
图7A至图7D是本公开的实施例所示的自动驾驶方法的应用场景示意图。
图8是根据本公开的实施例示出的一种距离确定方法的示意流程图。
图9是根据本公开的实施例示出的另一种距离确定方法的示意流程图。
图10是根据本公开的实施例示出的又一种距离确定方法的示意流程图。
图11是根据本公开的实施例示出的又一种距离确定方法的示意流程图。
图12是根据本公开的实施例示出的又一种距离确定方法的示意流程图。
具体实施方式
下面将结合本发明实施例中的附图，对本发明实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本发明一部分实施例，而不是全部的实施例。基于本发明中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本发明保护的范围。另外，在不冲突的情况下，下述的实施例及实施例中的特征可以相互组合。
图1是根据本公开的实施例示出的一种自动驾驶方法的示意流程图。本实施例所示的自动驾驶方法可以应用于车辆，所述车辆可以是无人驾驶车辆，在车辆上可以搭载有摄像装置，例如照相机、录像机等，摄像装置可以获取可见光图像。
如图1所示，所述自动驾驶方法可以包括以下步骤：
在步骤S101中，通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类；所述物体种类包括第一物体种类和第二物体种类；
在步骤S102中，将所述可见光图像映射到深度图像，并根据多个所述第一图像区域所对应的物体种类确定所述深度图像中多个第二图像区域所对应的物体种类；
在步骤S103中，根据所述深度图像中多个第二图像区域所对应的物体种类，确定多个所述第二图像区域中属于第一物体种类物体的目标图像区域；
在步骤S104中，根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离。
在一个实施例中，目标识别算法可以是预先通过机器学习训练得到的神经网络，例如卷积神经网络。神经网络可以被训练用于识别图像中每个区域对应的物体种类，例如所述物体种类可以包括第一物体种类和第二物体种类，具体地，第一物体种类可以是动态物体，第二物体种类可以是静态物体。
本公开所述实施例中的步骤(例如上述步骤S102)，主要可以在第二物体种类的物体遮挡第一物体种类的物体时进行，例如，第一物体种类是车辆，第二物体种类是路边的栅栏或绿化带。当前的自动驾驶汽车需要确定其与视野内的车辆的距离，以避免两车相撞而发生交通事故。但是当路边的栅栏或绿化带对其视野内的车辆部分遮挡时，当前的自动驾驶汽车会误识别路边的栅栏或绿化带也是车辆，从而发生距离的误测而发生误操作。通过本方案，可以降低误判的概率，提高自动驾驶的决策准确度，为用户提供更好的自动驾驶体验。
关于确定第二物体种类的物体遮挡第一物体种类的物体的方式，可以根据需要选择，并不限于下面示例的方式。例如在第一物体种类的物体对应的第一图像区域和第二物体种类的物体对应的第一图像区域相接的情况下，可以通过第一预设算法(例如预先通过机器学习得到的模型)识别第一物体种类的物体的完整度，如果第一物体种类的物体完整度大于预设完整度，例如大于95％，具体情况可以是图像中第一物体种类的物体为车辆侧面图像，其中只有前车灯未显示在图像中，那么可以确定第一物体种类的物体未被第二物体种类的物体遮挡；如果第一物体种类的物体完整度小于预设完整度，例如小于95％，例如上述车辆侧面图像中整个车头都未显示在图像中，可以确定第一物体种类的物体被第二物体种类的物体遮挡。
进一步可以根据第一物体种类的物体和第二物体种类的物体在图像中的关系，确定第一种类的物体是否被第二物体种类的物体所遮挡，并且可以将可见光图像映射到深度图像。其中，可见光图像和深度图像之间的映射关系(例如以矩阵的形式表示)可以是预先确定的。
在将可见光图像映射到深度图像后，可见光图像中的第一图像区域也会映射到深度图像中形成第二图像区域，那么第一图像区域对应的物体种类，就是第二图像区域对应的物体种类。
在确定深度图像中每个第二图像区域对应的物体种类后，可以进一步在 第二图像区域中确定属于第一物体种类的目标图像区域。由于目标图像区域位于深度图像中，所以目标图像区域的深度信息是可以确定的，而目标图像区域由对应第一物体种类的物体，因此可以根据目标图像区域的深度信息确定第一物体种类的物体的深度信息，进而根据第一物体种类的物体的深度信息可以确定属于第一物体种类的物体与本方法所应用于的车辆的距离。
据此，即使在可见光图像中第一物体种类的物体被第二物体种类的物体所遮挡，也能够准确地确定第一物体种类的物体与车辆的距离，以便车辆根据该距离准确地做出响应动作，有利于确保行驶安全。
可选地，所述深度图像由搭载在所述车辆上的双目摄像设备或者激光雷达获得。
在一个实施例中，在车辆上还可以搭载双目摄像设备，或者搭载激光雷达，那么可以通过双目摄像设备，或者激光雷达来获取所述深度图像。
图2是根据本公开的实施例示出的另一种自动驾驶方法的示意流程图。如图2所示，在根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离之前，所述方法还包括：
在步骤S105中，根据所述深度图像中各个点云的深度信息，对所述深度图像中的点云进行聚类生成聚类结果；
在步骤S106中，根据所述聚类结果对所述目标图像区域进行修正；
其中，所述根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离包括：
在步骤S1041中，根据修正后的目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离。
在一个实施例中，所采集的深度图可以包括多个点云，每个点云可以具备各自深度信息，那么可以根据点云的深度信息对点云进行聚类，例如将深度接近的点云聚类为一类，那么聚类结果中属于同一类的点云，较大概率属 于同一个物体，进而可以根据聚类结果对目标图像区域进行修正。
例如聚类结果中属于同一类的点云，其中第一部分点云位于目标图像区域内，第二部分点云位于目标图像区域外，那么可以扩大目标图像区域，使得扩大后的目标图像区域包含第一部分点云和第二部分点云，有利于确保修正后的目标图像区域包含第一物体种类的物体每个部分，进而根据修正后的目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离，有利于保证确定距离的准确性。
例如聚类结果中属于同一类的点云，简称A类点云，全部位于目标图像区域内，但是目标图像区域除了包括A类点云，还包括一小部分其他类点云，那么可以对目标图像区域进行缩小，使得缩小后的目标图像包含全部A类点云，并且不包含其他类点云，有利于确保修正后的目标图像区域仅包含第一物体种类的物体，而不包含其他物体种类的物体，进而根据修正后的目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离，有利于保证确定距离的准确性。
图3是根据本公开的实施例示出的又一种自动驾驶方法的示意流程图。如图3所示，所述通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类包括：
在步骤S1011中，通过所述目标识别算法识别所述可见光图像中每个第一图像区域属于每个物体种类的置信度；
在步骤S1012中，根据所述置信度确定每个第一图像区域对应的物体种类。
在一个实施例中，目标识别算法例如可以是神经网络，所述神经网络可以被训练用于识别图像中每个第一图像区域属于每个物体种类的置信度。
图4是根据本公开的实施例示出的一种神经网络的示意图。
如图4所示，神经网络可以是逐步训练得到的，神经网络可以包括多个 依次相连的模块，并且模块之间可以具有向前传播(Skip connection)关系，每个模块包括卷积层Conv、批量归一化层bn和线性整流层Relu。
神经网络的输入量可以表示为N*4*H*W，其中H表示图像的高度，W表示图像的宽度，N表示图像的数量，4表示图像的通道数，例如红(R)、绿(G)、蓝(B)以及深度共4个通道。
神经网络的输出量可以是表示为张量N*K*H*W，其中N的含义与输入量中对应参数的含义相同，K表示物体种类的标识，其中，可以预先设置每种物体种类的标识，以便确定出物体种类后，能够通过相应的标识进行表示。
例如，可以预先粗略划分五类物体，分别是车辆、天空、路面、动态物体、静态物体。其中，可以设置天空的标识为16，路面的标识为1。车辆的具体种类可以有多个，例如轿车标识为19，卡车的标识为20，公交车的标识为21，篷车的标识为22，火车的标识为24，拖车的标识为23，三轮车的标识为28，工程车辆的标识为27。静态物体的具体种类可以有多个，例如楼房的标识为5，墙的标识为6，栅栏的标识为7，护栏的标识为8，桥梁的标识为9，隧道的标识为10，柱子的标识为11，交通灯的标识为12，交通标识的为13，植物的标识为14，地形的标识为15。动态物体的具体种类可以有多个，例如行人的标识为17，起手的标识为18，摩托车的标识为25，自行车的标识为26。
进而可以在输出量中通过K表示物体种类，输出量中的H和W可以表示属于K对应物体种类的物体在图像中的高度和宽度，进而根据H和W可以确定每种K对应物体种类的物体在图像中对应的区域，例如上述实施例中的第一图像区域。
另外，输出量中的K除了可以包含物体种类的标识，还可以包含第一图像区域属于该物体种类的置信度(也可以表示为概率)，也即通过目标识别算法可以识别出可见光图像中每个第一图像区域属于每个物体种类的置信度， 从而可以根据置信度确定每个第一图像区域对应的物体种类。
也即对于某个第一图像区域，难以100％确定其所属的物体种类，而是可以确定其可以属于多个物体种类，以及属于每个物体种类的置信度，进而可以根据置信度确定第一图像区域到底应该对应哪个物体种类。例如可以将其中置信度最大的物体种类，作为第一图像区域对应的物体种类，例如第一图像区域20％的置信度属于柱子，30％的置信度属于植物，50％的置信度属于行人，其中最大的置信度为50％，对应的物体种类为行人，那么可以确定第一图像区域属于行人。
图5是根据本公开的实施例示出的又一种自动驾驶方法的示意流程图。如图5所示，在根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离之前，所述方法还包括：
在步骤S107中，根据所述深度图像中多个第二图像区域所对应的物体种类，确定多个所述第二图像区域中属于第二物体种类的物体的第三图像区域；
其中，所述根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离包括：；
在步骤S1042中，根据所述目标图像区域的深度信息与所述第三图像区域的深度信息确定属于第一物体种类的物体与所述车辆的距离。
在实际情况中，可以存在以下场景，第一物体种类的物体局部被第二物体种类的物体所遮挡，第一图像区域既包含属于第一物体种类的物体，又包含属于第二物体种类的物体，映射到深度图像中的第二图像区域，也既既包含属于第一物体种类的物体，又包含属于第二物体种类的物体。
例如第二图像区域中属于第二物体种类的物体为第三图像区域，对第二图像区域中属于第一物体种类的物体造成了遮挡，那么属于第一物体种类的物体与车辆的距离，不仅可以考虑目标图像区域的深度信息，还可以考虑第三图像区域的深度信息，也即根据根据目标图像区域的深度信息与第三图像 区域的深度信息确定属于第一物体种类的物体与可移动平台的距离。
由于识别算法本身存在一定的不准确性，在确定第一图像区域中属于第一物体种类的物体和属于第二物体种类的物体时，针对部分像素可能会产生错误的判断结果，将实际上应该属于第二物体种类的物体的像素，确定为属于第一物体种类的物体，或者将实际上应该属于第一物体种类的物体的像素，确定为属于第二物体种类的物体，因此，本实施例综合考虑目标图像区域的深度信息以及第三图像区域的深度信息来确定属于第一物体种类的物体与车辆的距离，以便在对部分像素误判的情况下，也能够考虑到被误判像素的深度信息，有利于确保确定距离的准确性。
可选地，属于第二物体种类的物体的第一图像区域和属于第一物体种类的物体的第一图像区域在所述可见光图像中相邻。
在一个实施例中，在上述出现误判的情况下，一种情况是是将属于第一物体种类的物体中部分像素误判为属于第二物体种类的物体，那么这部分像素在可见光图像中对应的第一图像区域，和属于第一物体种类的物体的另外一部分像素应该是相邻的(具体是相接的)。
也即在属于第二物体种类的物体的第一图像区域和属于第一物体种类的物体的第一图像区域在可见光图像中相邻的情况下，那么才较为可能出现误判的情况，因此才根据目标图像区域的深度信息与第三图像区域的深度信息确定属于第一物体种类的物体与车辆的距离。而在属于第二物体种类的物体的第一图像区域和属于第一物体种类的物体的第一图像区域在可见光图像中不相邻的情况下，那么判断结果一般是准确的，所以可以不必根据目标图像区域的深度信息与第三图像区域的深度信息确定属于第一物体种类的物体与车辆的距离，而只是根据目标图像区域的深度信息确定属于第一物体种类的物体与车辆的距离即可。
图6是根据本公开的实施例示出的又一种自动驾驶方法的示意流程图。 如图6所示，所述根据所述目标图像区域的深度信息与所述第三图像区域的深度信息确定属于第一物体种类的物体与所述车辆的距离包括：
在步骤S10421中，通过第一权值对所述目标图像区域的深度信息进行加权，以及通过第二权值对所述第三图像区域的深度信息进行加权，计算加权后的两者之和获得所述属于第一物体种类的物体与所述车辆的距离。
在一个实施例中，根据目标图像区域的深度信息与第三图像区域的深度信息确定属于第一物体种类的物体与车辆的距离，具体可以通过第一权值对目标图像区域的深度信息进行加权，以及通过第二权值对第三图像区域的深度信息进行加权，计算加权后的两者之和获得属于第一物体种类的物体与车辆的距离。其中，第一权值和第二权值可以根据需要设定，一般情况下，可以设定第一权值大于第二权值。
可选地，所述目标识别算法为卷积神经网络。
可选地，所述卷积神经网络包括多组层结构，每组层结构包含卷积层、批量归一化层和线性整流层。
可选地，所述卷积神经网络包括残差网络。
图7A至图7D是本公开的实施例所示的自动驾驶方法的应用场景示意图。
如图7A所示，为车辆视角的示意图，在图中可以看出，左前方有车辆，车辆靠右一些有行人，其中车辆被绿化带中的植物遮挡了后半部。
在一个实施例中，可以预先为每个物体种类设置对应的颜色，进而将图7A所示图像输入上述实施例中的卷积神经网络，那么可以确定图像中每个区域中的像素所属物体的物体种类，进而根据物体种类对像素进行着色，着色结果如图7B所示，基于图7B所示的着色结果，可以清楚地的区分出车辆、行人、路面、交通灯、绿化带中的植物等物体。
由于图7B所示图像中物体种类较多，可以对于其中的物体种类进行进一 步规律，例如归类为为上述实施例中的五类：车辆、天空、路面、动态物体、静态物体。进而确定图像中的像素属于这五类物体中的哪一类，然后根据所述的类别对应的颜色对像素进行着色，那么可以简化图像中的着色，得到如图7C所示的图像，如图7C所示，交通灯、植物、远处的山等都属于静态物体，因此可以通过相同颜色着色，天空通过一种颜色着色，路面通过一种颜色着色，那么在图像中就仅剩下左侧的车辆和车辆右侧的行人，那么对于车辆和属于静态物体的行人，可以通过不同的颜色进行着色，从而通过五种颜色即可将图像中每一类物体清晰的标识出来。
需要说明的是，图7C的着色是可选地，也可以在得到图7B后，就执行后续渲染操作，而不必在得到图7C后再进行后续渲染。
进而根据上述图7B或者图7C所示的图像，由于已经确定了每个像素所属的物体的物体类别，并进行了着色，那么可以根据着色结果进一步对图像进行渲染，如图7D所示，使得渲染后的图像与显示场景中物体的颜色较为接近，并且能够对车辆、行人等容易对驾驶产生影响的物体进行突出的渲染，以便根据渲染结果能够准确地确定图像中哪些物体是需要注意的物体，进而对这些物体进行测距，从而对距离较近的物体进行有效避让，以便确保自动驾驶过程中的行驶安全。
图8是根据本公开的实施例示出的一种距离确定方法的示意流程图。本实施例所示的自动驾驶方法可以应用于可移动平台，所述可移动平台上搭载有摄像装置，例如照相机、录像机等，摄像装置可以获取可见光图像。
如图8所示，所述距离确定方法可以包括以下步骤：
在步骤S201中，通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类；所述可见光图像由所述摄像装置获取；
在步骤S202中，将所述可见光图像映射到深度图像，并根据多个所述第一图像区域所对应的物体种类确定所述深度图像中多个第二图像区域所对应 的物体种类；
在步骤S203中，根据所述深度图像中多个第二图像区域所对应的物体种类，确定多个所述第二图像区域中属于第一物体种类的目标图像区域；
在步骤S204中，根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离。
根据本公开的实施例，即使在可见光图像中第一物体种类的物体被第二物体种类的物体所遮挡，也能够准确地确定第一物体种类的物体与可移动平台的距离，以便可移动平台所在车辆能够根据该距离准确地做出响应动作，有利于确保行驶安全。
可选地，所述可移动平台为汽车。
可选地，所述第一物体种类为动态物体。
可选地，所述第一物体种类包括汽车。
可选地，所述深度图像由搭载在所述可移动平台上的双目摄像设备或者激光雷达获得。
图9是根据本公开的实施例示出的另一种距离确定方法的示意流程图。如图9所示，在根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离之前，所述方法还包括：
在步骤S205中，根据所述深度图像中各个点云的深度信息，对所述深度图像中的点云进行聚类生成聚类结果；
在步骤S206中，根据所述聚类结果对所述目标图像区域进行修正；
其中，所述根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离包括：
在步骤S2041中，根据修正后的目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离。
在一个实施例中，所采集的深度图可以包括多个点云，每个点云可以具备各自深度信息，那么可以根据点云的深度信息对点云进行聚类，例如将深度接近的点云聚类为一类，那么聚类结果中属于同一类的点云，较大概率属于同一个物体，进而可以根据聚类结果对目标图像区域进行修正。
例如聚类结果中属于同一类的点云，其中第一部分点云位于目标图像区域内，第二部分点云位于目标图像区域外，那么可以扩大目标图像区域，使得扩大后的目标图像区域包含第一部分点云和第二部分点云，有利于确保修正后的目标图像区域包含第一物体种类的物体每个部分，进而根据修正后的目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离，有利于保证确定距离的准确性。
例如聚类结果中属于同一类的点云，简称A类点云，全部位于目标图像区域内，但是目标图像区域除了包括A类点云，还包括一小部分其他类点云，那么可以对目标图像区域进行缩小，使得缩小后的目标图像包含全部A类点云，并且不包含其他类点云，有利于确保修正后的目标图像区域仅包含第一物体种类的物体，而不包含其他物体种类的物体，进而根据修正后的目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离，有利于保证确定距离的准确性。
图10是根据本公开的实施例示出的又一种距离确定方法的示意流程图。如图10所示，所述通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类包括：
在步骤S2011中，通过所述目标识别算法识别所述可见光图像中每个第一图像区域属于每个物体种类的置信度；
在步骤S2012中，根据所述置信度确定每个第一图像区域对应的物体种类。
在一个实施例中，目标识别算法例如可以是神经网络，所述神经网络可 以被训练用于识别图像中每个第一图像区域属于每个物体种类的置信度。例如可以采用如图4所示的神经网络。
由于对于某个第一图像区域，难以100％确定其所属的物体种类，而是可以确定其可以属于多个物体种类，以及属于每个物体种类的置信度，进而可以根据置信度确定第一图像区域到底应该对应哪个物体种类。例如可以将其中置信度最大的物体种类，作为第一图像区域对应的物体种类，例如第一图像区域20％的置信度属于柱子，30％的置信度属于植物，50％的置信度属于行人，其中最大的置信度为50％，对应的物体种类为行人，那么可以确定第一图像区域属于行人。
图11是根据本公开的实施例示出的又一种距离确定方法的示意流程图。如图11所示，在根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离之前，所述方法还包括：
在步骤S207中，根据所述深度图像中多个第二图像区域所对应的物体种类，确定多个所述第二图像区域中属于第二物体种类的第三图像区域；
其中，所述根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离包括：
在步骤S2042中，根据所述目标图像区域的深度信息与所述第三图像区域的深度信息确定属于第一物体种类的物体与所述可移动平台的距离。
可选地，属于第二物体种类的第一图像区域和属于第一物体种类的第一图像区域在所述可见光图像中相邻。
在实际情况中，第一物体种类的物体局部被第二物体种类的物体所遮挡，第一图像区域既包含属于第一物体种类的物体，又包含属于第二物体种类的物体，映射到深度图像中的第二图像区域，也既既包含属于第一物体种类的物体，又包含属于第二物体种类的物体。
例如第二图像区域中属于第二物体种类的物体为第三图像区域中，对第 二图像区域中属于第一物体种类的物体造成了遮挡，那么属于第一物体种类的物体与车辆的距离，不仅可以考虑目标图像区域的深度信息，还可以考虑第三图像区域的深度信息，也即根据根据目标图像区域的深度信息与第三图像区域的深度信息确定属于第一物体种类的物体与可移动平台的距离。
由于识别算法本身存在一定的不准确性，在确定第一图像区域中属于第一物体种类的物体和属于第二物体种类的物体时，针对部分像素可能会产生错误的判断结果，将实际上应该属于第二物体种类的物体的像素，确定为属于第一物体种类的物体，或者将实际上应该属于第一物体种类的物体的像素，确定为属于第二物体种类的物体，因此，本实施例综合考虑目标图像区域的深度信息以及第三图像区域的深度信息来确定属于第一物体种类的物体与车辆的距离，以便在对部分像素误判的情况下，也能够考虑到被误判像素的深度信息，有利于确保确定距离的准确性。
图12是根据本公开的实施例示出的又一种距离确定方法的示意流程图。如图12所示，所述根据所述目标图像区域的深度信息与所述第三图像区域的深度信息确定属于第一物体种类的物体与所述可移动平台的距离包括：
在步骤S20421中，通过第一权值对所述目标图像区域的深度信息进行加权，以及通过第二权值对所述第三图像区域的深度信息进行加权，计算加权后的两者之和获得所述属于第一物体种类的物体与所述可移动平台的距离。
在一个实施例中，根据目标图像区域的深度信息与第三图像区域的深度信息确定属于第一物体种类的物体与车辆的距离，具体可以通过第一权值对目标图像区域的深度信息进行加权，以及通过第二权值对第三图像区域的深度信息进行加权，计算加权后的两者之和获得属于第一物体种类的物体与车辆的距离。其中，第一权值和第二权值可以根据需要设定，一般情况下，可以设定第一权值大于第二权值。
可选地，所述目标识别算法为卷积神经网络。
可选地，所述卷积神经网络包括多组层结构，每组层结构包含卷积层、批量归一化层和线性整流层。
可选地，所述卷积神经网络包括残差网络。
本公开的实施例还提出了一种自动驾驶装置，应用于一车辆，所述车辆上搭载有摄像装置以及处理器，所述处理器用于执行如上述任一实施例所述距离确定方法中的步骤。
本公开的实施例还提出了一种距离确定装置，应用于一可移动平台，所述可移动平台上搭载有摄像装置以及处理器，所述处理器用于执行如上述任一实施例所述距离确定方法中的步骤。
本公开的实施例还提出了一种可移动平台，包括：
机体；
动力系统，设于所述机体，所述动力系统用于为所述可移动平台提供动力；
摄像装置，设于所述机体，所述摄像装置用于获取可见光图像；
以及一个或多个处理器，用于执行上述任一实施例所述距离确定方法中的步骤。
在一实施例中，所述可移动平台为无人机、自动驾驶汽车等。
本公开的实施例还提出了一种机器可读存储介质，适用于可移动平台，所述机器可读存储介质上存储有若干计算机指令，所述计算机指令被被配置为执行上述任一实施例所述距离确定方法中的步骤。
上述实施例阐明的系统、装置、模块或单元，具体可以由计算机芯片或实体实现，或者由具有某种功能的产品来实现。为了描述的方便，描述以上装置时以功能分为各种单元分别描述。当然，在实施本申请时可以把各单元的功能在同一个或多个软件和/或硬件中实现。本领域内的技术人员应明白， 本发明的实施例可提供为方法、系统、或计算机程序产品。因此，本发明可采用完全硬件实施例、完全软件实施例、或结合软件和硬件方面的实施例的形式。而且，本发明可采用在一个或多个其中包含有计算机可用程序代码的计算机可用存储介质(包括但不限于磁盘存储器、CD-ROM、光学存储器等)上实施的计算机程序产品的形式。
本说明书中的各个实施例均采用递进的方式描述，各个实施例之间相同相似的部分互相参见即可，每个实施例重点说明的都是与其他实施例的不同之处。尤其，对于系统实施例而言，由于其基本相似于方法实施例，所以描述的比较简单，相关之处参见方法实施例的部分说明即可。
需要说明的是，在本文中，诸如第一和第二等之类的关系术语仅仅用来将一个实体或者操作与另一个实体或操作区分开来，而不一定要求或者暗示这些实体或操作之间存在任何这种实际的关系或者顺序。术语“包括”、“包含”或者其任何其他变体意在涵盖非排他性的包含，从而使得包括一系列要素的过程、方法、物品或者设备不仅包括那些要素，而且还包括没有明确列出的其他要素，或者是还包括为这种过程、方法、物品或者设备所固有的要素。在没有更多限制的情况下，由语句“包括一个……”限定的要素，并不排除在包括所述要素的过程、方法、物品或者设备中还存在另外的相同要素。
以上所述仅为本申请的实施例而已，并不用于限制本申请。对于本领域技术人员来说，本申请可以有各种更改和变化。凡在本申请的精神和原理之内所作的任何修改、等同替换、改进等，均应包含在本申请的权利要求范围之内。

Claims (27)

  1. 一种自动驾驶方法，其特征在于，应用于一车辆，所述车辆上搭载有摄像装置，所述摄像装置用于获取可见光图像，所述方法包括：
    通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类；所述物体种类包括第一物体种类和第二物体种类；
    将所述可见光图像映射到深度图像，并根据多个所述第一图像区域所对应的物体种类确定所述深度图像中多个第二图像区域所对应的物体种类；
    根据所述深度图像中多个第二图像区域所对应的物体种类，确定多个所述第二图像区域中属于第一物体种类物体的目标图像区域；
    根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离。
  2. 根据权利要求1所述的方法，其特征在于，所述深度图像由搭载在所述车辆上的双目摄像设备或者激光雷达获得。
  3. 根据权利要求1所述的方法，其特征在于，在根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离之前，所述方法还包括：
    根据所述深度图像中各个点云的深度信息，对所述深度图像中的点云进行聚类生成聚类结果；
    根据所述聚类结果对所述目标图像区域进行修正；
    其中，所述根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离包括：
    根据修正后的目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离。
  4. 根据权利要求1所述的方法，其特征在于，所述通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类包括：
    通过所述目标识别算法识别所述可见光图像中每个第一图像区域属于每个物体种类的置信度；
    根据所述置信度确定每个第一图像区域对应的物体种类。
  5. 根据权利要求1所述的方法，其特征在于，在根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离之前，所述方法还包括：
    根据所述深度图像中多个第二图像区域所对应的物体种类，确定多个所述第二图像区域中属于第二物体种类的物体的第三图像区域；
    其中，所述根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述车辆的距离包括：；
    根据所述目标图像区域的深度信息与所述第三图像区域的深度信息确定属于第一物体种类的物体与所述车辆的距离。
  6. 根据权利要求5所述的方法，其特征在于，属于第二物体种类的物体的第一图像区域和属于第一物体种类的物体的第一图像区域在所述可见光图像中相邻。
  7. 根据权利要求5所述的方法，其特征在于，所述根据所述目标图像区域的深度信息与所述第三图像区域的深度信息确定属于第一物体种类的物体与所述车辆的距离包括：
    通过第一权值对所述目标图像区域的深度信息进行加权，以及通过第二权值对所述第三图像区域的深度信息进行加权，计算加权后的两者之和获得所述属于第一物体种类的物体与所述车辆的距离。
  8. 根据权利要求1至7中任一项所述的方法，其特征在于，所述目标识别算法为卷积神经网络。
  9. 根据权利要求8所述的方法，其特征在于，所述卷积神经网络包括多组层结构，每组层结构包含卷积层、批量归一化层和线性整流层。
  10. 根据权利要求9所述的方法，其特征在于，所述卷积神经网络包括残差网络。
  11. 一种距离确定方法，其特征在于，应用于一可移动平台，所述可移动平台上搭载有摄像装置，包括：
    通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类；所述可见光图像由所述摄像装置获取；
    将所述可见光图像映射到深度图像，并根据多个所述第一图像区域所对应的物体种类确定所述深度图像中多个第二图像区域所对应的物体种类；
    根据所述深度图像中多个第二图像区域所对应的物体种类，确定多个所述第二图像区域中属于第一物体种类的目标图像区域；
    根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离。
  12. 根据权利要求11所述的方法，其特征在于，所述可移动平台为汽车。
  13. 根据权利要求11所述的方法，其特征在于，所述第一物体种类为动态物体。
  14. 根据权利要求11所述的方法，其特征在于，所述第一物体种类包括汽车。
  15. 根据权利要求11所述的方法，其特征在于，所述深度图像由搭载在所述可移动平台上的双目摄像设备或者激光雷达获得。
  16. 根据权利要求11所述的方法，其特征在于，在根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离之前，所述方法还包括：
    根据所述深度图像中各个点云的深度信息，对所述深度图像中的点云进行聚类生成聚类结果；
    根据所述聚类结果对所述目标图像区域进行修正；
    其中，所述根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离包括：
    根据修正后的目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离。
  17. 根据权利要求11所述的方法，其特征在于，所述通过预设的目标识别算法识别可见光图像中多个第一图像区域所对应的物体种类包括：
    通过所述目标识别算法识别所述可见光图像中每个第一图像区域属于每个物体种类的置信度；
    根据所述置信度确定每个第一图像区域对应的物体种类。
  18. 根据权利要求11所述的方法，其特征在于，在根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离之前，所述方法还包括：
    根据所述深度图像中多个第二图像区域所对应的物体种类，确定多个所述第二图像区域中属于第二物体种类的第三图像区域；
    其中，所述根据所述目标图像区域的深度信息，确定属于第一物体种类的物体与所述可移动平台的距离包括：
    根据所述目标图像区域的深度信息与所述第三图像区域的深度信息确定属于第一物体种类的物体与所述可移动平台的距离。
  19. 根据权利要求18所述的方法，其特征在于，属于第二物体种类的第一图像区域和属于第一物体种类的第一图像区域在所述可见光图像中相邻。
  20. 根据权利要求18所述的方法，其特征在于，所述根据所述目标图像区域的深度信息与所述第三图像区域的深度信息确定属于第一物体种类的物体与所述可移动平台的距离包括：
    通过第一权值对所述目标图像区域的深度信息进行加权，以及通过第二权值对所述第三图像区域的深度信息进行加权，计算加权后的两者之和获得所述属于第一物体种类的物体与所述可移动平台的距离。
  21. 根据权利要求11至20中任一项所述的方法，其特征在于，所述目标识别算法为卷积神经网络。
  22. 根据权利要求11所述的方法，其特征在于，所述卷积神经网络包括多组层结构，每组层结构包含卷积层、批量归一化层和线性整流层。
  23. 根据权利要求12所述的方法，其特征在于，所述卷积神经网络包括残差网络。
  24. 一种自动驾驶装置，其特征在于，应用于一车辆，所述车辆上搭载 有摄像装置以及处理器，所述处理器用于执行权利要求11至23中任一项所述距离确定方法中的步骤。
  25. 一种距离确定装置，其特征在于，应用于一可移动平台，所述可移动平台上搭载有摄像装置以及处理器，所述处理器用于执行权利要求11至23中任一项所述距离确定方法中的步骤。
  26. 一种可移动平台，其特征在于，包括：
    机体；
    动力系统，设于所述机体，所述动力系统用于为所述可移动平台提供动力；
    摄像装置，设于所述机体，所述摄像装置用于获取可见光图像；
    以及一个或多个处理器，用于执行权利要求11至23中任一项所述距离确定方法中的步骤。
  27. 一种机器可读存储介质，其特征在于，适用于可移动平台，所述机器可读存储介质上存储有若干计算机指令，所述计算机指令被被配置为执行权利要求11至23中任一项所述距离确定方法中的步骤。
PCT/CN2020/093521 2020-05-29 2020-05-29 自动驾驶方法和装置、距离确定方法和装置 WO2021237738A1 (zh)

Priority Applications (2)

Application Number Priority Date Filing Date Title
PCT/CN2020/093521 WO2021237738A1 (zh) 2020-05-29 2020-05-29 自动驾驶方法和装置、距离确定方法和装置
CN202080005812.9A CN112912892A (zh) 2020-05-29 2020-05-29 自动驾驶方法和装置、距离确定方法和装置

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/CN2020/093521 WO2021237738A1 (zh) 2020-05-29 2020-05-29 自动驾驶方法和装置、距离确定方法和装置

Publications (1)

Publication Number Publication Date
WO2021237738A1 true WO2021237738A1 (zh) 2021-12-02

Family

ID=76112922

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/093521 WO2021237738A1 (zh) 2020-05-29 2020-05-29 自动驾驶方法和装置、距离确定方法和装置

Country Status (2)

Country Link
CN (1) CN112912892A (zh)
WO (1) WO2021237738A1 (zh)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113449646A (zh) * 2021-06-29 2021-09-28 长沙理工大学 一种具有安全距离提示的抬头显示系统

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103390164A (zh) * 2012-05-10 2013-11-13 南京理工大学 基于深度图像的对象检测方法及其实现装置
CN104952254A (zh) * 2014-03-31 2015-09-30 比亚迪股份有限公司 车辆识别方法、装置和车辆
CN108725318A (zh) * 2018-07-28 2018-11-02 惠州华阳通用电子有限公司 汽车安全预警方法、装置及计算机可读存储介质
CN109740443A (zh) * 2018-12-12 2019-05-10 歌尔股份有限公司 检测障碍物的方法、装置及运动设备
CN110930427A (zh) * 2018-09-20 2020-03-27 银河水滴科技（北京）有限公司 一种基于语义轮廓信息的图像分割方法、设备和存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103390164A (zh) * 2012-05-10 2013-11-13 南京理工大学 基于深度图像的对象检测方法及其实现装置
CN104952254A (zh) * 2014-03-31 2015-09-30 比亚迪股份有限公司 车辆识别方法、装置和车辆
CN108725318A (zh) * 2018-07-28 2018-11-02 惠州华阳通用电子有限公司 汽车安全预警方法、装置及计算机可读存储介质
CN110930427A (zh) * 2018-09-20 2020-03-27 银河水滴科技（北京）有限公司 一种基于语义轮廓信息的图像分割方法、设备和存储介质
CN109740443A (zh) * 2018-12-12 2019-05-10 歌尔股份有限公司 检测障碍物的方法、装置及运动设备

Also Published As

Publication number Publication date
CN112912892A (zh) 2021-06-04

Similar Documents

Publication Publication Date Title
US20210365610A1 (en) Procedural world generation using tertiary data
CN107031656B (zh) 用于车轮止动器检测的虚拟传感器数据生成
Possatti et al. Traffic light recognition using deep learning and prior maps for autonomous cars
WO2022141910A1 (zh) 一种基于行车安全风险场的车路激光雷达点云动态分割及融合方法
WO2021057134A1 (zh) 场景识别的方法与计算设备
CN112069643B (zh) 一种自动驾驶仿真场景生成方法及装置
CN103522970B (zh) 基于机器视觉的车辆行驶安全检测方法和系统
US9336630B2 (en) Method and apparatus for providing augmented reality
CN113196291A (zh) 自动选择用于注释的数据样本
CN111695546A (zh) 用于无人车的交通信号灯识别方法和装置
CN108594244B (zh) 基于立体视觉和激光雷达的障碍物识别迁移学习方法
JP6700373B2 (ja) ビデオ動画の人工知能のための学習対象イメージパッケージング装置及び方法
CN111931683B (zh) 图像识别方法、装置及计算机可读存储介质
CN112967283A (zh) 基于双目摄像头的目标识别方法、系统、设备及存储介质
Lee et al. Accurate ego-lane recognition utilizing multiple road characteristics in a Bayesian network framework
CN111860219B (zh) 一种高速占道判断方法、装置及电子设备
CN110751012A (zh) 目标检测评估方法、装置、电子设备及存储介质
WO2021237738A1 (zh) 自动驾驶方法和装置、距离确定方法和装置
CN103577790B (zh) 道路转弯类型检测方法和装置
CN114550116A (zh) 一种对象识别方法和装置
CN113963061A (zh) 路沿分布信息获取方法、装置、电子设备和存储介质
CN110727269B (zh) 车辆控制方法及相关产品
WO2022145054A1 (ja) 画像処理装置、画像処理方法、及び記録媒体
CN114972731A (zh) 交通灯检测识别方法及装置、移动工具、存储介质
CN113989774A (zh) 一种交通灯检测方法、装置、车辆和可读存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20937752

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20937752

Country of ref document: EP

Kind code of ref document: A1