WO2020228694A1 - 一种相机姿态信息检测方法、装置以及相应的智能驾驶设备 - Google Patents

一种相机姿态信息检测方法、装置以及相应的智能驾驶设备 Download PDF

Info

Publication number
WO2020228694A1
WO2020228694A1 PCT/CN2020/089765 CN2020089765W WO2020228694A1 WO 2020228694 A1 WO2020228694 A1 WO 2020228694A1 CN 2020089765 W CN2020089765 W CN 2020089765W WO 2020228694 A1 WO2020228694 A1 WO 2020228694A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
sub
camera
road surface
images
Prior art date
Application number
PCT/CN2020/089765
Other languages
English (en)
French (fr)
Inventor
胡荣东
唐铭希
谢林江
彭清
Original Assignee
长沙智能驾驶研究院有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 长沙智能驾驶研究院有限公司 filed Critical 长沙智能驾驶研究院有限公司
Priority to US17/610,514 priority Critical patent/US20220215573A1/en
Publication of WO2020228694A1 publication Critical patent/WO2020228694A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/70—Determining position or orientation of objects or cameras
    • G06T7/73—Determining position or orientation of objects or cameras using feature-based methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/56—Context or environment of the image exterior to a vehicle by using sensors mounted on the vehicle
    • G06V20/588—Recognition of the road, e.g. of lane markings; Recognition of the vehicle driving pattern in relation to the road
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N17/00—Diagnosis, testing or measuring for television systems or their details
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30244—Camera pose
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30248—Vehicle exterior or interior
    • G06T2207/30252—Vehicle exterior; Vicinity of vehicle

Definitions

  • This application relates to the field of intelligent driving, and in particular to a camera attitude detection method and device, and corresponding intelligent driving equipment.
  • the detection of camera attitude angle is of great significance in fields such as unmanned driving and intelligent robots.
  • the traditional monocular vision-based camera attitude angle calculation method usually requires the use of a ranging sensor to obtain the distance of a number of specific positions, combined with the coordinates of the specific position in the image coordinate system, and finally obtain the camera's position by solving the PNP (Perspective-n-Point) algorithm Attitude angle. Therefore, this method needs to be solved before using the camera, and cannot realize the real-time calculation of the camera's attitude angle.
  • PNP Perspective-n-Point
  • this application proposes a camera attitude information detection method, which includes collecting multiple images through a multi-lens camera, and determining a road surface area image based on the multiple images; and combining the road surface area image The point of is projected to the coordinate plane of the camera coordinate system to obtain at least the pitch angle and/or roll angle in the camera attitude information.
  • the method further includes obtaining points on the lane line, and projecting the points on the lane line to the coordinate plane of the camera coordinate system to obtain at least the yaw angle in the camera posture information.
  • determining the road surface area image based on the multiple images includes selecting one of the multiple images as the original image, and combining other images in the multiple images to generate a depth map of the original image; using a search box The original image is traversed, and multiple sub-images corresponding to each search box are obtained; and the road surface confidence of the sub-images is calculated, and the sub-image with the largest road surface confidence is selected as the road area image.
  • determining the road surface area image based on the multiple images further includes filtering the multiple sub-images based on a preset depth threshold, and only retaining the sub-images whose center point depth value is less than the depth threshold; The retained sub-images calculate the road surface confidence, and the sub-image with the highest confidence is used as the road surface area image.
  • calculating the road surface confidence of the sub-image includes calculating the information entropy of the sub-image using the following formula, and the reciprocal of the information entropy is used as the road surface confidence
  • E is the image information entropy
  • g represents the gray value
  • P(g) is the probability that the current gray value g appears in the sub-image
  • m is a natural number greater than or equal to 1.
  • using the search box to traverse the original image includes determining the size of the search box according to the following formula: the width of the search box Search box height Where W is the original image width, H is the original image height, ⁇ is the original image aspect ratio W/H, It is a round up symbol, n is a search scale parameter, and n is a natural number.
  • the value range of the depth threshold is at least 15 to 25 meters.
  • the points in the road surface area image are projected onto the coordinate plane of the camera coordinate system to obtain the camera posture information
  • the points on the lane line are projected onto the coordinate plane of the camera coordinate system to obtain the camera posture information, including the position of the projected point Fitting is performed in the coordinate plane where it is located, and the posture information of the camera is obtained based on the angle formed by the fitted line and the coordinate axis of the coordinate plane.
  • the present application also provides a camera attitude detection device, including an image acquisition module configured to acquire multiple images; a road surface area determination module configured to determine a road surface area image based on the multiple images; and a attitude determination module configured to be based on The projection of each point in the road surface area image on the coordinate plane of the camera coordinate system at least determines the pitch angle and/or the roll angle in the posture information of the camera.
  • the image acquisition module is further configured to acquire pixels on the lane line; the posture determination module is also configured to determine at least the camera's position based on the projection of the pixel points on the lane line on the coordinate plane of the camera coordinate system The yaw angle in the attitude information.
  • the image acquisition module is configured to select one of the multiple images as the original image, and generate a depth map of the original image according to other images in the multiple images; wherein the road area is determined
  • the module includes a search and traversal sub-module configured to use a search box to traverse the original image and obtain multiple sub-images corresponding to each search box; and a road surface confidence filtering sub-module configured to calculate the sub-image Road surface confidence, select the sub-image with the highest road surface confidence as the road surface area image.
  • the road surface area determination module further includes a depth value filtering sub-module, configured to filter the multiple sub-images based on a preset depth threshold, and only retain the sub-images whose center point depth value is less than the depth threshold
  • the road surface confidence filtering sub-module is configured to calculate the road surface confidence of the retained sub-images, and select the sub-image with the maximum road surface confidence as the road surface area image.
  • the road surface confidence filtering sub-module is configured to calculate the information entropy of the sub-image using the following formula, and use the reciprocal of the information entropy as the road surface confidence
  • E is the image information entropy
  • g represents the gray value
  • P(g) is the probability that the current gray value g appears in the sub-image
  • m is a natural number greater than 1.
  • the search and traversal submodule is configured to determine the size of the search box and the width of the search box according to the following formula Search box height
  • W is the original image width
  • H is the original image height
  • ⁇ is the original image aspect ratio W/H
  • It is a round up symbol
  • n is a search scale parameter
  • n is a natural number.
  • the value range of the depth threshold is at least 15 to 25 meters.
  • the posture determination module is configured to fit the points in the road surface area image or the points on the lane line projected on the coordinate plane of the camera coordinate system in the corresponding coordinate plane, and based on the fitting The angle formed by the subsequent line and the coordinate axis of the coordinate plane on which it is located obtains the posture information of the camera.
  • the present application also provides an intelligent driving device, which includes a processor, and a memory and a network interface coupled with the processor; the vehicle sensor unit includes a multi-lens camera configured to acquire multiple images; wherein the processor is configured to Perform any of the aforementioned methods.
  • This application provides a solution for the relative displacement of the posture between the camera and the body of the intelligent driving device.
  • the posture information of the camera coordinate system is obtained by using the road surface as a reference object, so as to correct the camera coordinate system to obtain the body coordinate system. This avoids the negative impact on automatic driving caused by the offset of the camera relative to the vehicle body, and avoids potential safety hazards.
  • Fig. 1 shows a schematic flowchart of a camera pose detection method according to an embodiment of the present application
  • FIG. 2 shows a schematic flowchart of a camera pose detection method according to another embodiment of the present application
  • Fig. 3 is a schematic diagram of correcting the camera in the pitch direction according to an embodiment of the present application.
  • Fig. 4 shows an exemplary schematic diagram of a camera posture detection device according to an embodiment of the present application. ;as well as
  • Fig. 5 is a schematic structural diagram of an intelligent driving device according to an embodiment of the present application.
  • the camera posture may undergo undesirable changes.
  • the attitude angle of the camera may change relative to the attitude angle of the vehicle body (or the initial attitude angle of the camera). That is to say, in this case, the camera coordinate system cannot be equal to the body coordinate system. Therefore, if an automatic driving command is generated based on the camera's captured image without correcting it, it may cause the command to be inconsistent with the actual situation of the vehicle, and cause a major safety hazard.
  • the present application proposes the following solutions.
  • the camera's line of sight should be parallel to the road surface near the vehicle.
  • the line of sight of the camera (the light-in axis of the camera) is no longer parallel to the road surface near the vehicle. Therefore, the solution provided by this application is to find the projection of the road surface near the vehicle in the camera coordinate system to obtain the angle of the camera's line of sight relative to the road surface near the vehicle, and then use these angles to correct the image data obtained by the camera to obtain the correct Body coordinate system to overcome the adverse effects caused by the deflection of the camera relative to the body.
  • Fig. 1 shows a schematic flow chart of a method for detecting a camera pose according to an embodiment of the present application.
  • Step 101 Collect an original image and obtain a corresponding depth map.
  • the so-called depth map refers to an image with the distance (depth) from the camera to each point in the scene as the pixel value, which can directly reflect the geometric shape of the visible surface of the scene.
  • the binocular or multi-eye stereo camera can shoot the same target separately to obtain two or more images. You can use one of the images as the original image and use other images for correction and matching to obtain a depth map.
  • Step 102 Define a search box, and use the search box to traverse the original image.
  • the traversal can start from the upper left corner (or any position) of the original image, for example, the traversal step in the width direction of the original image can be, for example, W s /2, and the traversal step in the height direction of the original image can be For example, H s /2, start to traverse the entire original image.
  • the traversal stride can also be set to other values as needed.
  • multiple sub-images with depth values corresponding to the size of the search box are generated.
  • Step 103 Determine whether the depth value D of the center point of each sub-image is less than a preset depth threshold D T.
  • step 104 If the depth value D of the center point of the current sub-image is less than the set depth threshold D T , then the sub-image is retained and jump to step 105; otherwise, continue to step 104.
  • the value range of the depth threshold D T may be 15 meters ⁇ D T ⁇ 25 meters. This is because the road area near the vehicle is considered parallel to the driving direction of the vehicle, and this neighborhood can be defined as the distance from the vehicle less than D T. Therefore, to obtain the road surface near the vehicle, it is necessary to filter the depth value D of the center point of the sub-image first, and only the area meeting this distance requirement may be the road surface area near the vehicle.
  • the operations of steps 102 and 103 may be to use the depth value of the center point to filter after obtaining the sub-image corresponding to each search box, and skip the search box if it is greater than D T , To obtain the sub-image corresponding to the next search box until the traversal of the entire original image is completed.
  • all the sub-images can be obtained first, and the depth threshold can be used for filtering.
  • Step 104 Abandon the sub-images whose central point depth value is greater than or equal to D T.
  • Step 105 Calculate the road surface confidence of the retained sub-image.
  • the so-called road surface confidence is used to determine which sub-image can be used as an index of the road area image.
  • the road surface confidence of the sub-image can be determined through image information entropy.
  • the corresponding image information entropy may be calculated according to the gray value of the pixel point of each reserved sub-image.
  • the following formula (1) may be used to calculate the image information entropy of the sub-image:
  • P(g) can be the probability that the gray value g appears in the corresponding area of the search box, and g can represent one of a variety of gray values.
  • the number of g The value can be 2 n , where m is a natural number greater than or equal to 1.
  • the road surface confidence of the sub-image may be calculated according to the image information entropy of the sub-image. For example, the following formula (2) is used to determine the road surface confidence of the corresponding area:
  • ⁇ is the road surface confidence.
  • Step 106 Use the sub-image corresponding to the maximum road surface confidence level as the road surface area image.
  • the color or gray level of the road area should be basically the same or change little. Therefore, it can be considered that the area with the smallest information entropy is the area with the highest road surface confidence, so it can be regarded as the road area.
  • the accuracy of this method may have a certain gap with the above method, because if there is a scene with relatively small information entropy in the distance, it may have a certain impact on the determination of the road area.
  • the points in the road surface area image can be projected into the coordinate plane of the camera coordinate system to obtain the angle of the camera deflection relative to the road surface and obtain the vehicle coordinate system.
  • Step 107 Project all pixels in the image of the road area onto the YOZ plane of the camera coordinate system, and calculate the pitch angle ⁇ pitch of the camera.
  • the Z axis represents an axis parallel to the camera's line of sight
  • the X axis and the Y axis together form a plane perpendicular to the camera's line of sight.
  • the probability of the camera deviating in the YOZ plane or the camera's pitch attitude change is higher, and the magnitude of the deviation may also be greater, so you can first calculate the point in the road area image in the camera coordinate system YOZ plane Projection in.
  • the coordinates of the road area in the image is assumed that the pixels in the camera coordinate system is (X i, Y i, Z i), wherein at any point, in the range of the search area image Pavement i represents the i
  • the frame size is related to the camera resolution.
  • the coordinates of the pixel points of the road surface area image are all projected to the YOZ plane of the camera coordinate system.
  • the coordinates are (Z i , Y i ).
  • the least squares method can be used to perform straight line fitting on all points projected on the YOZ plane of the road area to obtain the straight line equation of the fitted straight line:
  • the angle between the fitted straight line and the Z axis can be used as the pitch angle ⁇ pitch of the camera, and the calculation formula is as follows:
  • Step 108 Project the pixels in the image of the road area onto the coordinate plane of the camera coordinate system, and calculate the roll angle ⁇ roll of the camera.
  • the road surface area pixels can be projected to the XOY plane in the uncorrected camera coordinate system, or projected to the first intermediate camera coordinate system X corrected by the pitch angle ⁇ pitch.
  • the second method is introduced below.
  • Fig. 3 shows a schematic diagram of correcting the camera in the pitch direction according to an embodiment of the present application.
  • the camera coordinate system can be rotated counterclockwise by ⁇ pitch as shown in FIG. 3 to obtain the first intermediate camera coordinate system.
  • the camera coordinate system can be rotated clockwise by ⁇ pitch to obtain the first intermediate camera coordinate system.
  • coordinate point In the camera coordinate system of the road surface area coordinate point may be expressed as (X i, Y i, Z i), using the pitch angle ⁇ pitch correction (assuming a top shift occurs) in the region of an image point of the road surface in the first intermediate camera
  • the coordinates in the coordinate system can be expressed as (X i ', Y i ', Z i '), and the conversion formula is as follows:
  • all pixels in the road surface area image can be projected to the X'OY' plane of the first intermediate camera coordinate system, and the projected coordinates are (X i ', Y i '), using, for example, the least squares method Fit a straight line to all points after projection to get the straight line equation of the straight line:
  • Step 109 Obtain pixels on the lane line.
  • the point on the lane line may be a part of the road surface area image.
  • the pixel data on the lane line can be extracted from the road surface area image data in the process of using the pixel point data on the lane line to obtain the yaw angle.
  • the points of the lane line may not be in the road surface area image, for example, it can be obtained through GPS in conjunction with data pre-stored in the database.
  • the data of the pixels on the lane line can be obtained by any method, which is not limited here.
  • step 110 the pixel points on the lane line are projected into the coordinate plane of the camera coordinate system to calculate, for example, the camera yaw angle ⁇ yaw .
  • the points on the lane line can be projected into the XOZ plane of the camera coordinate system, and fitting is performed to obtain the yaw angle ⁇ yaw .
  • the points on the lane line can also be projected into the coordinate plane of the intermediate camera coordinate system. The process is similar to the method for obtaining the roll angle described above, and will not be repeated here.
  • the coordinates of the pixels on the lane line in the camera coordinate system can be expressed as (X i road , Y i road , Z i road ), and the coordinates of the pixels on the lane line in the first intermediate camera coordinate system using the pitch angle correction
  • the coordinates can be expressed as (X i road ', Y i road ', Z i road '). Project all the pixels on the lane line to the X'OZ' plane of the first intermediate camera coordinate system, and the coordinates of the points after projection are (X i road ', Z i road ').
  • the least square method can be used to fit a straight line to the point after the projection to obtain the straight line equation:
  • the vehicle coordinate system can be obtained based on the pitch angle, roll angle, and yaw angle of the camera, and the image data captured by the camera can be corrected.
  • both the uncorrected initial camera coordinate system and the partially corrected intermediate camera coordinate system may be referred to as camera coordinate systems.
  • the camera attitude information or camera attitude angle may include one or more of a pitch angle, a roll angle, or a yaw angle.
  • the processor that runs the camera posture detection method of the embodiments of this application can be located on the terminal side of the camera, or on the server side, and can also be implemented by the terminal side and the server in cooperation. limit.
  • Fig. 2 is a schematic flowchart of a method for detecting a camera pose according to another embodiment of the present application.
  • step 201 pixels on the lane line can be acquired.
  • the points on the lane line can be projected on the XOZ plane of the camera coordinate system, and the projected points can be fitted, and the offset is obtained based on the angle between the fitted line and the X axis and Z axis of the camera coordinate system.
  • the flight angle ⁇ yaw can refer to the embodiment shown in FIG. 1.
  • the points on the lane line can be projected on the YOZ plane of the camera coordinate system, and the projected points can be fitted, based on the fitted line and the Y axis and Z axis of the camera coordinate system Obtain the yaw angle ⁇ pitch .
  • the points on the lane line can be projected on the XOY plane of the camera coordinate system, and the projected points can be fitted, based on the fitted line and the X axis and Y axis of the camera coordinate system Obtain the yaw angle ⁇ roll .
  • Fig. 4 shows an exemplary schematic diagram of a camera posture detection device according to an embodiment of the present application.
  • the camera posture detection device of the present application may include an acquisition module 401, a road area determination module 402, and a posture determination module 403.
  • the acquiring module 401 may be configured to acquire multiple images and corresponding depth maps.
  • the acquisition module 401 may include a binocular stereo camera or other number of multi-eye stereo cameras, configured to acquire two or more images, and use one of them as the original image, and use the other images for correction and Cooperate to obtain a depth map.
  • the acquiring module 401 may also be configured to acquire pixels on the lane line.
  • the road surface area determination module 402 is configured to receive image data from the acquisition module 301 and determine an image of a road surface area near the vehicle.
  • the road surface area determination module may include a search and traversal submodule 4021, a depth value filtering submodule 4022, and a road surface confidence filtering submodule 4023.
  • the search and traversal submodule 4021 may be configured to determine the size of the search box for traversing the original image, and use the search box to traverse the original image.
  • the search and traversal sub-module 4021 may be configured to traverse from, for example, the upper left corner (or any position) of the original image to generate multiple sub-images.
  • the traversal step in the width direction of the original image may be, for example, W s /2
  • the traversal step in the height direction of the original image may be, for example, H s /2, starting to traverse the entire original image.
  • the traversal stride can also be set to other values as needed.
  • the depth value filtering module 4022 may be configured to filter according to the depth maps of all sub-images obtained by traversing, and if the depth value D of the center point of the current sub-image is less than the set depth threshold D T , the sub-image is retained, Otherwise, discard the sub-image.
  • the value range of the depth threshold D T may be 15 meters ⁇ D T ⁇ 25 meters. This is because the road area near the vehicle is considered parallel to the driving direction of the vehicle, and this neighborhood can be defined as the distance from the vehicle less than D T. Therefore, to obtain the road surface near the vehicle, it is necessary to filter the depth value D of the center point of the sub-image first, and only the area meeting this distance requirement may be the road surface area near the vehicle.
  • the road surface confidence filtering sub-module 4023 may be configured to determine the road surface confidence of the retained sub-image, and use the area with the highest road surface confidence as the road surface area.
  • the road surface confidence filtering sub-module 4023 may be configured to calculate the corresponding image information entropy according to the gray value of the pixel point of each reserved sub-image.
  • the following formula (12) may be used to calculate the image information entropy of the sub-image:
  • P(g) can be the probability that the gray value g appears in the corresponding area of the search box, and g can represent one of a variety of gray values.
  • the number of g The value can be 2 n , where n is a natural number greater than or equal to 1.
  • the road surface confidence of the sub-image may be calculated according to the image information entropy of the sub-image. For example, the following formula (13) is used to determine the road surface confidence of the corresponding area:
  • ⁇ is the road surface confidence.
  • the road surface confidence filtering module 4023 may be configured to determine the road surface with the highest confidence level among the retained sub-images, and use the sub-image with the maximum level of confidence as the road surface area image.
  • the road surface area determination module 402 may also only include a road surface confidence filtering sub-module.
  • the attitude determination module 403 may include a pitch angle determination sub-module 4031, a roll angle determination sub-module 4032, and a yaw angle determination sub-module 4033.
  • the Z axis represents an axis parallel to the camera's line of sight
  • the X axis and the Y axis together form a plane parallel to the camera's line of sight
  • the Y axis is a coordinate axis perpendicular to the plane.
  • the pitch angle determination sub-module 4031 may be configured to all pixels in the road surface are projected area of the image to the camera coordinate system YOZ plane, the coordinates of the projection (Z i, Y i).
  • the least squares method can be used to fit all the projected points to a straight line to obtain a straight line equation to obtain the pitch angle ⁇ pitch .
  • the roll angle determination sub-module 4032 may be configured to XOY plane area images pavement all pixels are projected to the camera coordinate system, the coordinates of the projection of (X i, Y i) according to.
  • the least square method can be used to fit the points projected on the XOY plane to a straight line equation to obtain the roll angle ⁇ roll .
  • the first intermediate camera coordinate system is a camera coordinate system obtained by correcting the obtained pitch angle ⁇ pitch .
  • the yaw angle determination sub-module 4033 may be configured to project all pixels to the XOZ plane of the camera coordinate system, and the projected coordinates are (X i , Z i ).
  • the least squares method can be used to fit all points projected on the XOZ plane to a straight line equation to obtain the yaw angle ⁇ yaw .
  • the pixel points on the lane line can also be projected on the X'OZ' plane of the first intermediate camera coordinate system, and the yaw angle ⁇ yaw can be obtained by fitting the projected points.
  • the first intermediate camera coordinate system is a camera coordinate system obtained by correcting the obtained pitch angle ⁇ pitch .
  • the camera coordinate system can be corrected based on the posture information of the camera to obtain the vehicle coordinate system.
  • the order and manner of obtaining the aforementioned attitude angles may be different.
  • the pitch angle ⁇ pitch may be calculated first, and the camera coordinate system may be corrected using the pitch angle ⁇ pitch to obtain the first intermediate coordinate system, and the roll angle and/or yaw are calculated in the first intermediate coordinate system angle.
  • Fig. 5 is a schematic structural diagram of an intelligent driving device according to an embodiment of the present application.
  • the intelligent driving device includes a processor 501 and a memory 502 for storing a computer program that can run on the processor 501, where the processor 501 is used to execute the computer program described in any of the embodiments of the present application. Provide method steps.
  • the processor 501 and the memory 502 do not mean that the corresponding number is one, but may be one or more.
  • the intelligent driving device may further include a memory 503, a network interface 504, and a system bus 505 connecting the memory 503, the network interface 504, the processor 501 and the memory 502.
  • the operating system and the data processing device provided in the embodiments of the present application are stored in the memory.
  • the processor 501 is used to support the operation of the entire intelligent driving device.
  • the memory 503 may be used to provide an environment for the running of the computer program in the memory 502.
  • the network interface 504 may be used for network communication with external server devices, terminal devices, etc., to receive or send data, such as obtaining driving control instructions input by the user.
  • the smart driving device may also include a GPS unit 506 configured to obtain location information of the driving device.
  • the sensor unit 507 may include a binocular or multi-lens camera, configured to obtain multiple images, and cooperate with the processor 501 and the memory 502 to obtain a depth map.
  • the processor 501 is configured to execute the method shown in FIG. 1 based on the information obtained by each unit to obtain camera posture information.
  • the embodiment of the present application also provides a computer storage medium, for example, including a memory storing a computer program.
  • the computer program can be executed by a processor to complete the steps of the camera posture information detection method provided in any embodiment of the present application.
  • the computer storage medium may be FRAM, ROM, PROM, EPROM, EEPROM, Flash Memory, magnetic surface memory, optical disk, or CD-ROM, etc.; it may also be various devices including one or any combination of the foregoing memories.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Multimedia (AREA)
  • Health & Medical Sciences (AREA)
  • Biomedical Technology (AREA)
  • General Health & Medical Sciences (AREA)
  • Signal Processing (AREA)
  • Image Analysis (AREA)
  • Image Processing (AREA)
  • Traffic Control Systems (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

一种相机姿态信息检测方法，包括通过多目相机采集多幅图像，并基于所述多幅图像确定路面区域图像；以及将路面区域图像中的点投影到相机坐标系中从而获得相机姿态信息。

Description

一种相机姿态信息检测方法、装置以及相应的智能驾驶设备 技术领域
本申请涉及智能驾驶领域，具体涉及一种相机姿态检测方法、装置、以及相应的智能驾驶设备。
背景技术
相机姿态角的检测，在无人驾驶、智能机器人等领域具有重要意义。传统基于单目视觉的相机姿态角计算方法，通常需要利用测距传感器获取若干特定位置的距离，结合特定位置在图像坐标系的坐标，最后通过求解PNP(Perspective-n-Point)算法得到相机的姿态角。因此，该方法需要在使用相机之前进行求解，无法实现相机姿态角的实时计算。
发明内容
针对现有技术中存在的技术问题，本申请提出了一种相机姿态信息检测方法，包括通过多目相机采集多幅图像，并基于所述多幅图像确定路面区域图像；以及将路面区域图像中的点投影到相机坐标系的坐标平面从而至少获得相机姿态信息中的俯仰角和/或翻滚角。
特别的，所述方法还包括获取车道线上的点，并且将车道线上的点投影到所述相机坐标系的坐标平面从而至少获得所述相机姿态信息中的偏航角。
特别的，基于所述多幅图像确定路面区域图像包括选择所述多幅图像中的一幅作为原始图像，并结合所述多幅图像中其他图像生成所述原始图像的深度图； 利用搜索框对所述原始图像进行遍历，并获得多个与各搜索框对应的子图像；以及计算所述子图像的路面置信度，选择路面置信度最大的子图像作为路面区域图像。
特别的，基于所述多幅图像确定路面区域图像还包括基于预设的深度阈值对所述多幅子图像进行过滤，并且只保留中心点深度值小于所述深度阈值的子图像；针对所述被保留下来的子图像计算路面置信度，并将置信度最高的子图像作为路面区域图像。
特别的，计算所述子图像的路面置信度包括利用以下公式计算所述子图像的信息熵，并将信息熵的倒数作为路面置信度
Figure PCTCN2020089765-appb-000001
其中E为图像信息熵，g代表灰度值，P(g)为当前灰度值g在所述子图像中出现的概率,m为大于等于1的自然数。
特别的，利用搜索框对所述原始图像进行遍历包括，根据以下公式确定所述搜索框尺寸搜索框的宽度
Figure PCTCN2020089765-appb-000002
搜索框高度
Figure PCTCN2020089765-appb-000003
其中，W为所述原始图像宽度，H为所述原始图像高度，δ为所述原始图像宽高比W/H，
Figure PCTCN2020089765-appb-000004
为向上取整符号，n为搜索比例参数，n取值为自然数。
特别的，所述深度阈值的取值范围至少是15至25米。
特别的，将路面区域图像中的点投影到相机坐标系中的坐标平面获得相机姿态信息，将车道线上的点投影到相机坐标系的坐标平面获得相机姿态信息，包括对经投影的点在其所在的坐标平面内进行拟合，，并基于拟合后的线条与该坐标平面的坐标轴形成的夹角获得所述相机的姿态信息。
本申请还提供了一种相机姿态检测装置，包括图像获取模块，配置为获得多 幅图像；路面区域确定模块，配置为基于所述多幅图像确定路面区域图像；以及姿态确定模块，配置为基于所述路面区域图像中各点在相机坐标系的坐标平面中的投影至少确定相机的姿态信息中的俯仰角和/或翻滚角。
特别的，所述图像获取模块还配置为获取车道线上的像素点；所述姿态确定模块还配置为基于所述车道线上的像素点在相机坐标系的坐标平面中的投影至少确定相机的姿态信息中的偏航角。
特别的，所述图像获取模块配置为选择所述多幅图像中的一幅作为原始图像，并根据所述多幅图像中其他图像生成所述原始图像的深度图；其中，所述路面区域确定模块包括搜索和遍历子模块，配置为利用搜索框对所述原始图像进行遍历，并获得多个与各搜索框对应的子图像；以及路面置信度过滤子模块，配置为计算所述子图像的路面置信度，选择路面置信度最大的子图像作为路面区域图像。
特别的，所述路面区域确定模块还包括深度值过滤子模块，配置为基于预设的深度阈值对所述多幅子图像进行过滤，并且只保留中心点深度值小于所述深度阈值的子图像；所述路面置信度过滤子模块配置为计算保留下来的子图像的路面置信度，选择路面置信度最大的子图像作为路面区域图像。
特别的，所述路面置信度过滤子模块配置为利用以下公式计算所述子图像的信息熵，并将信息熵的倒数作为路面置信度
Figure PCTCN2020089765-appb-000005
其中E为图像信息熵，g代表灰度值，P(g)为当前灰度值g在所述子图像中出现的概率，m为大于1的自然数。
特别的，所述搜索和遍历子模块，配置为根据以下公式确定所述搜索框寸，搜索框的宽度
Figure PCTCN2020089765-appb-000006
搜索框高度
Figure PCTCN2020089765-appb-000007
其中，W为所述原始图像宽 度，H为所述原始图像高度，δ为所述原始图像宽高比W/H，
Figure PCTCN2020089765-appb-000008
为向上取整符号，n为搜索比例参数，n取值为自然数。
特别的，所述深度阈值的取值范围至少是15至25米。
特别的，所述姿态确定模块配置为将投影在所述相机坐标系坐标平面中的所述路面区域图像中的点或车道线上的点在相应的坐标平面中进行拟合，并基于拟合后的线条与其所在的坐标平面的坐标轴形成的夹角获得相机的姿态信息。
本申请还提供了一种智能驾驶设备，包括处理器，以及与所述处理器耦合的存储器和网络接口；车辆传感器单元包括多目相机，配置为获取多幅图像；其中所述处理器配置为执行前述任一所述的方法。
本申请所为相机与智能驾驶设备的车身之间发生姿态的相对偏移时提供了解决方案，通过利用路面作为参照物来获得相机坐标系的姿态信息，从而对相机坐标系进行校正获得车身坐标系。由此避免了由于相机相对于车身的偏移而对自动驾驶造成的负面影响，避免了安全隐患。
附图说明
下面，将结合附图对本申请的实施方式进行进一步详细的说明，其中：
图1所示为根据本申请一个实施例的相机姿态检测方法的流程示意图；
图2所示为根据本申请另一个实施例的相机姿态检测方法的流程示意图；
图3所示为根据本申请一个实施例中对相机在俯仰方向进行校正的示意图；
图4所示为根据本申请一个实施例的相机姿态检测装置的示例性示意图。；以及
图5所示为根据本申请一个实施例的智能驾驶设备结构示意图。
具体实施方式
为使本申请实施例的目的、技术方案和优点更加清楚，下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本申请一部分实施例，而不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
在以下的详细描述中，可以参看作为本申请一部分用来说明本申请的特定实施例的各个说明书附图。在附图中，相似的附图标记在不同图式中描述大体上类似的组件。本申请的各个特定实施例在以下进行了足够详细的描述，使得具备本领域相关知识和技术的普通技术人员能够实施本申请的技术方案。
在一些实际应用场景中，由于抖动等因素，相机姿态可能会发生不希望的变化。例如，在无人驾驶场景，由于路面颠簸等原因，相机的姿态角可能与车身姿态角(或相机的初始姿态角)发生相对的变化。也就是说在这种情况下，相机坐标系不能等同于车身坐标系。那么如果基于相机捕获图像而不对其进行校正来产生自动驾驶指令，可能会导致指令与车辆实际情况不符的，进而造成重大安全隐患。
本申请为了解决前述的相机姿态相对车辆姿态发生变化的情况，提出了下列的解决方案。理想情况下，相机的视线应该是平行于车辆附近的路面的。然而，当相机的姿态角与车身姿态角发生的相对变化以后，相机的视线(相机的进光轴)就不再平行于车辆附近的路面。因此，本申请所提供的方案是通过找到车辆附近路面在相机坐标系中的投影，获得相机视线相对于车辆附近路面的夹角，进而利用这些角度对相机所获得的图像数据进行矫正，得到正确的车身坐标系，以克服相机相对于车身的偏转所带来的不良影响。
图1示出了根据本申请一个实施例的相机姿态检测方法的流程示意图。
步骤101，采集原始图像，并获得所对应的深度图。
所谓深度图是指将从相机到场景中各点的距离(深度)作为像素值的图像，能直接反映景物可见表面的几何形状。双目或多目立体相机可以分别对同一目标进行拍摄，得到两幅或多幅图像。可以将其中一幅图像作为原始图像并利用其他图像进行矫正和匹配，从而获得深度图。
步骤102，定义搜索框，利用搜索框遍历原始图像。
在一些实施例中，搜索框的尺寸可以预先指定，或者可以根据原始图像的参数确定。假设原始图像宽度为W，高度为H，宽高比δ＝W/H，则搜索框的宽度可以为
Figure PCTCN2020089765-appb-000009
高度可以为
Figure PCTCN2020089765-appb-000010
其中，
Figure PCTCN2020089765-appb-000011
为向上取整符号，n为搜索比例参数，可以取5～8或者其他的自然数。
根据一个实施例，可以从原始图像的例如左上角(或者任何一个位置)开始遍历，在原始图像的宽度方向遍历步幅可以为例如W s/2，在原始图像的高度方向遍历步幅可以为例如H s/2，开始遍历整幅原始图像。当然，遍历步幅也可以根据需要设置成为其他的值。
根据一个实施例，经过遍历后会生成多幅尺寸与搜索框尺寸对应的具有深度值的子图像。
步骤103，判断各子图像中心点深度值D是否小于预设的深度阈值D T。
如果当前子图像的中心点的深度值D小于设定深度阈值D T，则保留该子图像并跳转到步骤105，否则继续到步骤104。
根据一个实施例，深度阈值D T的取值范围可以为15米<D T<25米。这样做是 认为车辆附近的路面区域与车辆的行驶方向平行的，而这个附近就可以定义为与车辆距离小于D T的距离。因此，要获得车辆附近的路面，需要先对子图像中心点的深度值D进行筛选，只有满足这个距离要求的区域才有可能是车辆附近的路面区域。
可选择的，根据一个实施例，步骤102和103的操作可以是在获取到每个搜索框对应的子图象后利用其中心点的深度值进行过滤，如果大于D T则跳过该搜索框，获取下一搜索框对应的子图像，直到完成对整个原始图像的遍历。根据另一个实施例，可以先获得全部的子图像，并利用深度阈值来进行过滤。
步骤104，放弃中心点深度值大于等于D T的子图像。
步骤105，针对所保留的子图像计算其路面置信度。
所谓路面置信度就是用来确定哪个子图像可以被用来作为路面区域图像的指标。根据不同的实施例可以有不同的衡量指标。根据本申请的一个实施例，可以通过图像信息熵来确定子图像的路面置信度。
具体地，可以根据每个被保留的子图像的像素点的灰度值来计算对应的图像信息熵。
在一些实施例中，可以采用如下公式(1)计算子图像的图像信息熵：
Figure PCTCN2020089765-appb-000012
其中，E为图像信息熵，P(g)可以为灰度值g在搜索框对应区域中出现的概 率,g可以代表多种灰度值中的一个，在这个实施例中例如g的个数可以取值为2 n，其中m为大于等于1的自然数。
根据一个实施例，可以根据子图像的图像信息熵来计算子图像的路面置信度。例如采用如下公式(2)确定对应区域的路面置信度：
σ＝1/E     (2)
其中，σ为路面置信度。
步骤106，将路面置信度最大值对应的子图像作为路面区域图像。
由上可知，路面置信度越高的区域，信息熵越小，信息熵越小说明该区域的颜色或者灰度越统一。理论上说，路面区域的颜色或者说灰度应该是基本一致或者变化很小的。因此，可以认为信息熵最小的区域就是路面置信度最高的区域，因此也就可以被作为路面区域。
根据不同的实施例，也可以不进行深度值的过滤，直接计算各子图像的路面置信度，并进行下面的操作。但是这样做准确度可能与上述方法有一定差距，因为如果远处有信息熵比较小的景物的话，可能会对路面区域的确定造成一定的影响。
在获得了车辆附近的路面区域的图像数据以后，就可以将该路面区域图像中的点投影到相机坐标系的坐标平面中从而获得相机相对于路面偏转的角度，并获得车辆坐标系。
进行投影并获得相机姿态角的计算方法有很多可能性，例如，可以分别在相机坐标系下找到车辆附近路面区域的投影获得相机的三个姿态角，并依据这三 个姿态角对相机坐标系的三个坐标轴加以校正，最终获得车辆坐标系。或者，也可以先获得相机的一个姿态角，然后获得一个中间的坐标系，在这个坐标系下获得下一个姿态角，以此类推，最终获得车辆坐标系。
以下只是提出一个实施例作为说明。在该实施例的基础上调整或改变获得各姿态角的顺序获得相机姿态角的各种方法都属于本申请所保护的范围。
步骤107，将路面区域图像中所有像素点投影到相机坐标系的YOZ平面，并计算相机的俯仰角θ pitch。
假设在相机坐标系中，Z轴代表平行于相机视线的轴，X轴与Y轴一起构成垂直于相机视线的平面。
由于根据实际观察，相机在例如YOZ平面内发生偏离或者说相机发生俯仰姿态变化的概率更高，且偏离的幅度也可能更大，因此可以首先计算路面区域图像中的点在相机坐标系YOZ平面中的投影。
在一些实施例中，假设路面区域图像中像素点在相机坐标系下的坐标为(X i，Y i，Z i)，其中i代表路面区域图像中的任何一点，i的取值范围与搜索框尺寸和相机分辨率有关。路面区域图像的像素点的坐标都投影到相机坐标系的YOZ平面后的坐标为(Z i，Y i)。
根据一个实施例，可以利用最小二乘法对路面区域在YOZ平面投影的所有点进行直线拟合，得到拟合直线的直线方程：
Y＝a 1*Z+b 1     (3)
可以将拟合出来的直线与Z轴的夹角作为相机的俯仰角θ pitch，并有计算公 式如下：
θ pitch＝arctan(-a 1)    (4)
根据其他的情况，如果车辆附近的路面不是平面的话，可以拟合获得曲线方程。由于做出这样的变化不需要付出创造性劳动，因此仍然属于本申请保护的范围。
步骤108，将路面区域图像中像素点投影到相机坐标系的坐标平面中，计算相机的翻滚角θ roll。在计算翻滚角的时候可以有多种实施方式，可以将路面区域像素投影到未经校正的相机坐标系中的XOY平面，或者投影到利用俯仰角θ pitch校正后的第一中间相机坐标系X’Y’Z’中的平面。
以下对第二种方式进行介绍。
图3所示为根据本申请一个实施例中对相机在俯仰方向进行校正的示意图。在相机发生俯视偏移的情况下，可以如图3所示将相机坐标系沿逆时针方向旋转θ pitch，得到第一中间相机坐标系。当然，如果在相机发生仰视偏移的情况下，可以将将相机坐标系沿顺时针方向旋转θ pitch，得到第一中间相机坐标系。
在相机坐标系下路面区域的点的坐标可以表示为(X i，Y i，Z i)，利用俯仰角θ pitch校正后(假设发生俯视偏移)路面区域图像中的点在第一中间相机坐标系下的坐标可以表示为(X i’，Y i’，Z i’)，转换公式如下：
X i’＝X i     (5)
Y i’＝cos(θ pitch)*Y i+sin(θ pitch)*Z i   (6)
Z i’＝cos(θ pitch)*Z i-sin(θ pitch)*Y i    (7)
根据一个实施例，可以将路面区域图像中所有像素点投影至第一中间相机坐标系的X’OY’平面，经投影后的坐标为(X i’，Y i’)，利用例如最小二乘法对投影之后所有点进行直线拟合，得到直线的直线方程：
Y’＝a 2*X’+b 2    (8)
则相机的翻滚角计算公式如下：
θ roll＝arctan(-a 2)   (9)
步骤109，获取车道线上的像素点。根据一些实施例，车道线上的点可以是路面区域图像中的一部分。在这种情况下，在利用车道线上的像素点数据获得偏航角的过程中车道线上像素点的数据可以从路面区域图像数据中提取。根据其他实施例，车道线的点可能并不在路面区域图像中，例如可以通过GPS配合数据库中预存的数据获得。总之，根据不同的实施例，可以通过任何方法来获得车道线上像素点的数据，在此不做限制。
由于偏航角更适合利用车道线的投影来确定，因此在步骤110，将车道线上的像素点投影到相机坐标系的坐标平面中，计算例如相机偏航角θ yaw。
根据一个实施例，可以将车道线上的点投影到相机坐标系的XOZ平面中，并进行拟合获得偏航角θ yaw。根据另一个实施例，也可以将车道线上的点投影到中间相机坐标系的坐标平面中。过程与上述获得翻滚角的方式类似，在此不再赘述。
车道线上的像素点在相机坐标系下的坐标可以表示为(X i road，Y i road， Z i road)，在利用俯仰角校正的第一中间相机坐标系下的车道线上像素点的坐标可以表示为(X i road’，Y i road’，Z i road’)。将车道线上的所有像素点投影到第一中间相机坐标系的X’OZ’平面，投影后点的坐标为(X i road’，Z i road’)。例如可以利用最小二乘法对投影之后的点进行直线拟合，得到直线方程：
Z i road’＝a 3*X i road’+b 3     (10)
则相机的偏航角计算公式如下：
θ yaw＝π/2-arctan(a 3)    (11)
根据一个实施例，可以基于相机的俯仰角、翻滚角和偏航角获得车辆坐标系，并对相机捕获的图像数据进行矫正。
以上介绍中都是对经投影的路面区域图像中的点进行直线拟合。本领域技术人员知晓的是，在其他的情况下，例如路面不是完全平坦的情况下，可以对利用其他方程对这些经投影的点进行例如曲线拟合。但是无论如何，这些都属于本申请的保护范围。
在本申请中，未经校正的最初的相机坐标系和经过部分校正的中间相机坐标系都可以被称为相机坐标系。
在本申请中，相机姿态信息或者相机姿态角可以包括俯仰角，翻滚角或偏航角中的一个或多个。
应当理解，在具体实施时，运行本申请实施例的相机姿态检测方法的处理器可以位于相机的终端侧，也可以处于服务器侧，还可以由终端侧和服务器配合实现，本申请对此均不作限制。
图2所示为根据本申请另一个实施例的相机姿态检测方法的流程示意图。
在步骤201，可以获取车道线上的像素点。
在步骤202，可以将车道线上的点投影在相机坐标系的XOZ平面，并对经投影的点进行拟合，基于拟合后的线条与相机坐标系X轴和Z轴的夹角获取偏航角θ yaw。根据一个实施例，对经投影的车到线上的点在XOZ平面的拟合以及对偏航角的计算方法可以参考图1所示的实施例。
可选择的，可以在步骤203，可以将车道线上的点投影在相机坐标系的YOZ平面，并对经投影的点进行拟合，基于拟合后的线条与相机坐标系Y轴和Z轴的夹角获取偏航角θ pitch。
可选择的，可以在步骤204，可以将车道线上的点投影在相机坐标系的XOY平面，并对经投影的点进行拟合，基于拟合后的线条与相机坐标系X轴和Y轴的夹角获取偏航角θ roll。
图4所示为根据本申请一个实施例的相机姿态检测装置的示例性示意图。如图4所示，本申请的相机姿态检测装置可以包括获取模块401，路面区域确定模块402，以及姿态确定模块403。
在一些实施例中，获取模块401可以配置为获取多幅图像，以及所对应的深度图。根据一个实施例，获取模块401可以包括双目立体相机或其他数量的多目立体相机，配置为获取两幅或多幅图像，并利用其中的一幅作为原始图像，利用其他的图像进行校正和配合从而获得深度图。根据一个实施例，获取模块401还可以配置为获取车道线上的像素点。
在一些实施例中，路面区域确定模块402，配置为从获取模块301接收图像数据，并从中确定车辆附近的路面区域图像。
根据一个实施例，路面区域确定模块可以包括搜索和遍历子模块4021，深度值过滤子模块4022以及路面置信度过滤子模块4023。
根据一个实施例，搜索和遍历子模块4021可以配置为确定对原始图像进行遍历的搜索框尺寸，并利用该搜索框对原始图像进行遍历。
在一些实施例中，搜索框的尺寸可以预先指定，或者可以根据原始图像的参数确定。假设原始图像宽度为W，高度为H，宽高比δ＝W/H，则搜索框的宽度可以为
Figure PCTCN2020089765-appb-000013
高度可以为
Figure PCTCN2020089765-appb-000014
其中，
Figure PCTCN2020089765-appb-000015
为向上取整符号，n为搜索比例参数，可以取5～8或者其他的自然数。
根据一个实施例，搜索和遍历子模块4021可以配置为从原始图像的例如左上角(或者任何一个位置)开始遍历，从而生成多幅子图像。在原始图像的宽度方向遍历步幅可以为例如W s/2，在原始图像的高度方向遍历步幅可以为例如H s/2，开始遍历整幅原始图像。当然，遍历步幅也可以根据需要设置成为其他的值。
根据一个实施例，深度值过滤模块4022可以配置为根据遍历获得的所有子图像的深度图进行过滤，如果当前子图像中心点的深度值D小于设定深度阈值D T，则保留该子图像，否则放弃该子图像。
根据一个实施例，深度阈值D T的取值范围可以为15米<D T<25米。这样做是认为车辆附近的路面区域与车辆的行驶方向平行的，而这个附近就可以定义为 与车辆距离小于D T的距离。因此，要获得车辆附近的路面，需要先对子图像中心点的深度值D进行筛选，只有满足这个距离要求的区域才有可能是车辆附近的路面区域。
根据一个实施例，路面置信度过滤子模块4023可以配置为确定所保留的子图像的路面置信度，并采用路面置信度最大的区域作为路面区域。
根据一个实施例，路面置信度过滤子模块4023可以配置根据每个被保留的子图像的像素点的灰度值来计算对应的图像信息熵。
在一些实施例中，可以采用如下公式(12)计算子图像的图像信息熵：
Figure PCTCN2020089765-appb-000016
其中，E为图像信息熵，P(g)可以为灰度值g在搜索框对应区域中出现的概率,g可以代表多种灰度值中的一个，在这个实施例中例如g的个数可以取值为2 n，其中n为大于等于1的自然数。
根据一个实施例，可以根据子图像的图像信息熵来计算子图像的路面置信度。例如采用如下公式(13)确定对应区域的路面置信度：
σ＝1/E    (13)
其中，σ为路面置信度。
根据一个实施例，路面置信度过滤自模块4023可以配置为确定所保留的各子图像中路面置信度最大者，并将具有置信度最大值的子图像作为路面区域图 像。
根据不同的实施例，路面区域确定模块402也可以仅包括路面置信度过滤子模块。
根据一个实施例，姿态确定模块403可以包括俯仰角确定子模块4031，翻滚角确定子模块4032，以及偏航角确定子模块4033。
假设在相机坐标系中，Z轴代表平行于相机视线的轴，X轴与Y轴一起构成平行于相机视线的平面，Y轴是与该平面垂直的坐标轴。
根据一个实施例，俯仰角确定子模块4031可以配置为将路面区域图像的所有像素点都投影到相机坐标系的YOZ平面，投影后的坐标为(Z i，Y i)。可以利用最小二乘法对所有经投影点进行直线拟合，得到直线方程，从而获得俯仰角θ pitch。
根据一个实施例，根据一个实施例，翻滚角确定子模块4032可以配置为将路面区域图像中所有像素点都投影到相机坐标系的XOY平面，投影后的坐标为(X i，Y i)。可以利用最小二乘法对在XOY平面投影的点进行直线拟合，得到直线方程，从而获得翻滚角θ roll。当然，也可以将路面区域图像中的点投影到第一中间相机坐标系的X’OY’平面，通过对经投影的点进行拟合，从而获得翻滚角θ roll。第一中间相机坐标系是利用所获得的俯仰角θ pitch经过校正得到的相机坐标系。
根据一个实施例，根据一个实施例，偏航角确定子模块4033可以配置为像素点的都投影到相机坐标系的XOZ平面，投影后的坐标为(X i，Z i)。可以利用 最小二乘法对在XOZ平面投影的所有点进行直线拟合，得到直线方程，从而获得偏航角θ yaw。根据其他的实施例，也可以将车道线上的像素点投影到第一中间相机坐标系的X’OZ’平面上，通过对经投影的点进行拟合，获得偏航角θ yaw。第一中间相机坐标系是利用所获得的俯仰角θ pitch经过校正得到的相机坐标系。
由此可以基于相机的姿态信息对相机坐标系进行校正从而获得车辆坐标系。
根据不同的实施例，获得上述姿态角的顺序和方式可能有所不同。根据一个实施例，可以先计算出俯仰角θ pitch，并且利用俯仰角θ pitch对相机坐标系进行校正，得到第一中间坐标系，并且在第一中间坐标系下计算翻滚角和/或偏航角。但是无论如何变化，都没有超出本申请所要求保护的范围。
图5所示为根据本申请一个实施例的智能驾驶设备结构示意图。该智能驾驶设备包括处理器501以及用于存储能够在处理器501上运行的计算机程序的存储器502，其中，所述处理器501用于运行所述计算机程序时，执行本申请任一实施例所提供的方法步骤。这里处理器501和存储器502并非指代对应的数量为一个，而是可以为一个或者多个。该智能驾驶设备还可包括内存503、网络接口504、以及将内存503、网络接口504、处理器501和存储器502连接的系统总线505。存储器中存储有操作系统及本申请实施例所提供的数据处理装置。处理器501用于支持整个智能驾驶设备的操作。内存503可以用于为存储器502中的计算机程序的运行提供环境。网络接口504可以用于外部服务器设备、终端设备等进行网络通信，接收或发送数据，如获取用户输入的驾驶控制指令等。其中该智能驾驶设备还可以包括GPS单元506配置为获取驾驶设备所在位置信息。传感器单元507可以包括双目或多目相机，配置为获取多幅图像，并配合处 理器501和存储器502获得深度图。其中，处理器501配置为基于各单元获取的信息执行图1所示方法从而获得相机姿态信息。
本申请实施例还提供了一种计算机存储介质，例如包括存储有计算机程序的存储器，该计算机程序可以由处理器执行，以完成本申请任一实施例所提供的相机姿态信息检测的方法步骤。该计算机存储介质可以是FRAM、ROM、PROM、EPROM、EEPROM、Flash Memory、磁表面存储器、光盘、或CD-ROM等存储器；也可以是包括上述存储器之一或任意组合的各种设备。
以上所述实施例仅表达了本申请的几种实施方式，其描述较为具体和详细，但并不能因此而理解为对申请专利范围的限制。应当指出的是，对于本领域的普通技术人员来说，在不脱离本申请构思的前提下，还可以做出若干变形和改进，这些都属于本申请的保护范围。因此，本申请专利的保护范围应以所附权利要求为准。

Claims (17)

  1. 一种相机姿态信息检测方法，包括：
    通过多目相机采集多幅图像，并基于所述多幅图像确定路面区域图像；以及
    将路面区域图像中的点投影到相机坐标系的坐标平面从而至少获得相机姿态信息中的俯仰角和/或翻滚角。
  2. 如权利要求1所述的方法，还包括
    获取车道线上的点，并且将车道线上的点投影到所述相机坐标系的坐标平面从而至少获得所述相机姿态信息中的偏航角。
  3. 如权利要求1所述的方法，其中基于所述多幅图像确定路面区域图像包括：
    选择所述多幅图像中的一幅作为原始图像，并结合所述多幅图像中其他图像生成所述原始图像的深度图；
    利用搜索框对所述原始图像进行遍历，并获得多个与各搜索框对应的子图像；以及
    计算所述子图像的路面置信度，选择路面置信度最大的子图像作为路面区域图像。
  4. 如权利要求3所述的方法，其中基于所述多幅图像确定路面区域图像还包括：
    基于预设的深度阈值对所述多幅子图像进行过滤，并且只保留中心点深度值小于所述深度阈值的子图像；
    针对所述被保留下来的子图像计算路面置信度，并将置信度最高的子图像作为路面区域图像。
  5. 如权利要求3或4所述的方法，其中计算所述子图像的路面置信度包括利用以下公式计算所述子图像的信息熵，并将信息熵的倒数作为路面置信度，
    Figure PCTCN2020089765-appb-100001
    其中E为图像信息熵，g代表灰度值，P(g)为当前灰度值g在所述子图像中出现的概率,m为大于等于1的自然数。
  6. 如权利要求3所述的方法，其中利用搜索框对所述原始图像进行遍历包括，根据以下公式确定所述搜索框尺寸
    搜索框的宽度W s＝|W/(n*δ)|，
    搜索框高度H s＝|H/n|；
    其中，W为所述原始图像宽度，H为所述原始图像高度，δ为所述原始图像宽高比W/H，
    Figure PCTCN2020089765-appb-100002
    为向上取整符号，n为搜索比例参数，n取值为自然数。
  7. 如权利要求4所述的方法，其中所述深度阈值的取值范围至少是15至25米。
  8. 根据权利要求1或2所述的方法，其中将路面区域图像中的点投影到相机坐标系中的坐标平面获得相机姿态信息，或将车道线上的点投影到相机坐标系的坐标平面获得相机姿态信息，包括
    对经投影的点在其所在的坐标平面内进行拟合，并基于拟合后的线条与该坐标平面的坐标轴形成的夹角获得所述相机姿态信息。
  9. 一种相机姿态检测装置，包括
    图像获取模块，配置为获得多幅图像；
    路面区域确定模块，配置为基于所述多幅图像确定路面区域图像；以及
    姿态确定模块，配置为基于所述路面区域图像中各点在相机坐标系的坐标平面中的投影至少确定相机的姿态信息中的俯仰角和/或翻滚角。
  10. 如权利要求9所述的相机姿态检测装置，其中所述图像获取模块还配置为获取车道线上的像素点；所述姿态确定模块还配置为基于所述车道线上的像素点在相机坐标系的坐标平面中的投影至少确定相机的姿态信息中的偏航角。
  11. 如权利要求9所述的装置，其中所述图像获取模块配置为选择所述多幅图像中的一幅作为原始图像，并根据所述多幅图像中其他图像生成所述原始图像的深度图；
    其中，所述路面区域确定模块包括
    搜索和遍历子模块，配置为利用搜索框对所述原始图像进行遍历，并获得多个与各搜索框对应的子图像；以及
    路面置信度过滤子模块，配置为计算所述子图像的路面置信度，选择路面置信度最大的子图像作为路面区域图像。
  12. 如权利要求11所述的装置，其中所述路面区域确定模块还包括
    深度值过滤子模块，配置为基于预设的深度阈值对所述多幅子图像进行过滤，并且只保留中心点深度值小于所述深度阈值的子图像；
    所述路面置信度过滤子模块配置为计算保留下来的子图像的路面置信度，选择路面置信度最大的子图像作为路面区域图像。
  13. 如权利要求11或12所述的装置，其中所述路面置信度过滤子模块配置为利用以下公式计算所述子图像的信息熵，并将信息熵的倒数作为路面置信度，
    Figure PCTCN2020089765-appb-100003
    其中E为图像信息熵，g代表灰度值，P(g)为当前灰度值g在所述子图像中出现的概率，m为大于1的自然数。
  14. 如权利要求11所述的装置，其中所述搜索和遍历子模块，配置为根据以下公式确定所述搜索框尺寸
    搜索框的宽度W s＝|W/(n*δ)|，
    搜索框高度H s＝|H/n|；
    其中，W为所述原始图像宽度，H为所述原始图像高度，δ为所述原始图像宽高比W/H，
    Figure PCTCN2020089765-appb-100004
    为向上取整符号，n为搜索比例参数，n取值为自然数。
  15. 如权利要求12所述的装置，其中所述深度阈值的取值范围至少是15至25米。
  16. 如权利要求9或10所述的装置，其中所述姿态确定模块配置为将投影在所述相机坐标系坐标平面中的所述路面区域图像中的点或车道线上的点在相应的坐标平面中进行拟合，并基于拟合后的线条与其所在的坐标平面的坐标轴形成的夹角获得相机的姿态信息。
  17. 一种智能驾驶设备，包括
    处理器，以及与所述处理器耦合的存储器和网络接口；
    车辆传感器单元包括多目相机，配置为获取多幅图像；
    其中所述处理器配置为执行权利要求1-8中任一所述的方法。
PCT/CN2020/089765 2019-05-13 2020-05-12 一种相机姿态信息检测方法、装置以及相应的智能驾驶设备 WO2020228694A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US17/610,514 US20220215573A1 (en) 2019-05-13 2020-05-12 Camera pose information detection method and apparatus, and corresponding intelligent driving device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910394067.0A CN109903346B (zh) 2019-05-13 2019-05-13 相机姿态检测方法、装置、设备及存储介质
CN201910394067.0 2019-05-13

Publications (1)

Publication Number Publication Date
WO2020228694A1 true WO2020228694A1 (zh) 2020-11-19

Family

ID=66956591

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/089765 WO2020228694A1 (zh) 2019-05-13 2020-05-12 一种相机姿态信息检测方法、装置以及相应的智能驾驶设备

Country Status (3)

Country Link
US (1) US20220215573A1 (zh)
CN (1) CN109903346B (zh)
WO (1) WO2020228694A1 (zh)

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113639643A (zh) * 2021-06-24 2021-11-12 河南农业大学 基于rgb-d深度相机的作物苗期高度检测方法
CN114463683A (zh) * 2022-02-12 2022-05-10 河南城建学院 基于人工智能和大数据的变电设备智能监控系统和方法

Families Citing this family (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN109903346B (zh) * 2019-05-13 2019-09-06 长沙智能驾驶研究院有限公司 相机姿态检测方法、装置、设备及存储介质
CN110221328A (zh) * 2019-07-23 2019-09-10 广州小鹏汽车科技有限公司 一种组合导航方法和装置
CN114264310B (zh) * 2020-09-14 2024-06-21 阿里巴巴集团控股有限公司 定位及导航方法、装置、电子设备、计算机存储介质
CN112489131B (zh) * 2020-12-26 2024-04-05 上海有个机器人有限公司 基于路面检测构建代价地图的方法、装置、介质和机器人
CN113091889A (zh) * 2021-02-20 2021-07-09 周春伟 一种测量道路亮度的方法及装置
CN115460320A (zh) * 2021-06-09 2022-12-09 阿里巴巴新加坡控股有限公司 导航方法、装置、计算机存储介质及计算机程序产品
CN115937321B (zh) * 2022-09-27 2023-09-22 荣耀终端有限公司 一种电子设备的姿态检测方法和装置

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100295948A1 (en) * 2009-05-21 2010-11-25 Vimicro Corporation Method and device for camera calibration
CN103745452A (zh) * 2013-11-26 2014-04-23 理光软件研究所（北京）有限公司 相机外参评估方法、装置、相机外参标定方法和装置
CN105164726A (zh) * 2013-01-24 2015-12-16 微软技术许可有限责任公司 用于3d重构的相机姿态估计
CN105469386A (zh) * 2015-11-11 2016-04-06 浙江大华技术股份有限公司 一种确定立体相机高度与俯仰角的方法及装置
CN105469076A (zh) * 2015-12-29 2016-04-06 东莞市微模式软件有限公司 基于多示例学习的人脸比对验证方法
CN109903346A (zh) * 2019-05-13 2019-06-18 长沙智能驾驶研究院有限公司 相机姿态检测方法、装置、设备及存储介质

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP3668769B2 (ja) * 2000-06-26 2005-07-06 独立行政法人産業技術総合研究所 対象物体の位置・姿勢算出方法及び観測カメラの位置・姿勢算出方法
US8452053B2 (en) * 2008-04-24 2013-05-28 GM Global Technology Operations LLC Pixel-based texture-rich clear path detection
DE102009055776A1 (de) * 2009-11-25 2011-05-26 Conti Temic Microelectronic Gmbh Verfahren zur Schätzung des Rollwinkels in einem fahrenden Fahrzeug
JP5923422B2 (ja) * 2012-09-24 2016-05-24 クラリオン株式会社 カメラのキャリブレーション方法及び装置
JP2014112055A (ja) * 2012-12-05 2014-06-19 Denso It Laboratory Inc カメラ姿勢の推定方法およびカメラ姿勢の推定システム
ES2834997T3 (es) * 2014-04-14 2021-06-21 Vricon Systems Ab Un procedimiento y sistema para estimar información relacionada con el ángulo de cabeceo y/o alabeo de un vehículo
US10290119B2 (en) * 2016-09-15 2019-05-14 Sportsmedia Technology Corporation Multi view camera registration
CN108022264B (zh) * 2016-11-01 2023-06-13 北京墨土科技有限公司 相机位姿确定方法及设备

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100295948A1 (en) * 2009-05-21 2010-11-25 Vimicro Corporation Method and device for camera calibration
CN105164726A (zh) * 2013-01-24 2015-12-16 微软技术许可有限责任公司 用于3d重构的相机姿态估计
CN103745452A (zh) * 2013-11-26 2014-04-23 理光软件研究所（北京）有限公司 相机外参评估方法、装置、相机外参标定方法和装置
CN105469386A (zh) * 2015-11-11 2016-04-06 浙江大华技术股份有限公司 一种确定立体相机高度与俯仰角的方法及装置
CN105469076A (zh) * 2015-12-29 2016-04-06 东莞市微模式软件有限公司 基于多示例学习的人脸比对验证方法
CN109903346A (zh) * 2019-05-13 2019-06-18 长沙智能驾驶研究院有限公司 相机姿态检测方法、装置、设备及存储介质

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN113639643A (zh) * 2021-06-24 2021-11-12 河南农业大学 基于rgb-d深度相机的作物苗期高度检测方法
CN113639643B (zh) * 2021-06-24 2023-12-22 河南农业大学 基于rgb-d深度相机的作物苗期高度检测方法
CN114463683A (zh) * 2022-02-12 2022-05-10 河南城建学院 基于人工智能和大数据的变电设备智能监控系统和方法

Also Published As

Publication number Publication date
CN109903346B (zh) 2019-09-06
CN109903346A (zh) 2019-06-18
US20220215573A1 (en) 2022-07-07

Similar Documents

Publication Publication Date Title
WO2020228694A1 (zh) 一种相机姿态信息检测方法、装置以及相应的智能驾驶设备
CN110567469B (zh) 视觉定位方法、装置、电子设备及系统
US10165262B2 (en) Image processing device and markers
JP6112221B2 (ja) 移動体位置推定装置および移動体位置推定方法
CN106529587B (zh) 基于目标点识别的视觉航向识别方法
CN108335337B (zh) 一种正射影像图的生成方法及装置
CN108805934A (zh) 一种车载摄像机的外部参数标定方法及装置
WO2022078074A1 (zh) 车辆与车道线的位置关系检测方法、系统和存储介质
WO2013133129A1 (ja) 移動物体位置姿勢推定装置及び移動物体位置姿勢推定方法
CN112556685B (zh) 导航路线的显示方法、装置和存储介质及电子设备
CN112700486B (zh) 对图像中路面车道线的深度进行估计的方法及装置
KR20210095913A (ko) 맵 작성 방법, 장치, 및 시스템, 및 저장 매체
CN109727285B (zh) 使用边缘图像的位置和姿态确定方法和系统
WO2021016803A1 (zh) 高精度地图定位方法、系统、平台及计算机可读存储介质
CN112017238A (zh) 一种线状物体的空间位置信息确定方法及装置
CN112967344A (zh) 相机外参标定的方法、设备、存储介质及程序产品
CN112489106A (zh) 基于视频的车辆尺寸测量方法、装置、终端及存储介质
JP2006234703A (ja) 画像処理装置及び三次元計測装置並びに画像処理装置用プログラム
CN113034347A (zh) 倾斜摄影图像处理方法、装置、处理设备及存储介质
CN111105467A (zh) 一种图像标定方法、装置及电子设备
CN109708655A (zh) 导航方法、装置、车辆及计算机可读存储介质
CN111260538B (zh) 基于长基线双目鱼眼相机的定位及车载终端
CN113011212B (zh) 图像识别方法、装置及车辆
CN111260698B (zh) 双目图像特征匹配方法及车载终端
CN112598736A (zh) 一种基于地图构建的视觉定位方法及装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20805104

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20805104

Country of ref document: EP

Kind code of ref document: A1