TWI838022B - Ground plane fitting method, vehicle-mounted device, and storage medium - Google Patents

Abstract

The present application provides a ground plane fitting method, a vehicle-mounted device, and a storage medium. The method includes: obtaining point clouds and a target image towards a travelling direction of a vehicle, based on the point clouds and the target image; determining a geographic point cloud set corresponding to the target image; correcting a plurality of normal vectors of a camera that obtains the target image, and obtaining a plurality of ground normal vectors; according to the geographic point cloud set and the ground normal vectors, fitting a ground plane of the travelling direction of the vehicle and obtaining a fitting ground plane. This application can improve the fitting accuracy of the ground plane based on the image processing, and realize the safe driving of auxiliary autonomous vehicle.

Description

Ground plane fitting method, vehicle-mounted device and storage medium

The present invention relates to the field of safe driving technology, and in particular to a ground plane fitting method, a vehicle-mounted device, and a storage medium.

In the autonomous driving technology, plane fitting of the ground in front of the vehicle is an indispensable part. The driving of the autonomous vehicle can be controlled according to the fitted ground plane. For example, when the ground plane indicates that the ground in front is downhill and the slope is large, the autonomous vehicle is controlled to slow down. The existing ground plane fitting method has problems such as low fitting accuracy.

In view of the above, it is necessary to provide a ground plane fitting method, a vehicle-mounted device and a storage medium that can effectively improve the accuracy of fitting the ground plane and assist the vehicle in safe driving.

The ground plane fitting method includes: determining a ground point cloud set corresponding to the target image based on the point cloud and target image obtained in the direction of vehicle travel; correcting multiple camera normal vectors of the camera that obtains the target image to obtain multiple ground normal vectors; fitting the ground plane in the direction of vehicle travel according to the ground point cloud set and the multiple ground normal vectors to obtain a fitting ground plane.

Optionally, the method of determining a ground point cloud set corresponding to the target image based on the obtained point cloud of the vehicle's travel direction and the target image includes: determining a ground area in the target image; projecting the point cloud to the target image, and taking a set of point clouds corresponding to the ground area as the ground point cloud set.

Optionally, determining the ground area in the target image includes: identifying the ground area in the target image using a semantic segmentation algorithm.

Optionally, the step of correcting multiple camera normal vectors of a camera that obtains the target image to obtain multiple ground normal vectors includes: obtaining multiple camera normal vectors of the camera including a timestamp; obtaining motion data of the camera, and correcting the multiple camera normal vectors using the motion data to obtain the multiple ground normal vectors.

Optionally, the step of obtaining multiple camera normal vectors containing time stamps of the camera includes: using the coordinate axis pointing to the sky in the camera coordinate system of the camera as the camera normal vector.

Optionally, the method further includes: obtaining multiple images taken by the camera, the multiple images including the target image, and each image in the multiple images corresponds to a timestamp; obtaining a camera coordinate system corresponding to each image.

Optionally, the movement data of the camera includes gravity acceleration; the use of the movement data to correct the multiple camera normal vectors to obtain the multiple ground normal vectors includes: using the direction of the gravity acceleration to correct the rotation matrix in the camera posture of the camera, and correcting the camera normal vector to the ground normal vector.

Optionally, fitting the ground plane in the direction of travel of the vehicle according to the ground point cloud set and the multiple ground normal vectors to obtain a fitted ground plane includes: using the ground point cloud set and the multiple ground normal vectors as constraints for fitting a ground plane equation, and fitting the ground plane corresponding to the ground using the least squares method based on the constraints.

The computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by the processor, the ground plane fitting method or the ground plane fitting method is implemented.

The vehicle-mounted device includes a memory and at least one processor, wherein at least one instruction is stored in the memory, and when the at least one instruction is executed by the at least one processor, the ground plane fitting method is implemented.

Compared with the prior art, the ground plane fitting method provided in the embodiment of the present application determines the ground point cloud set corresponding to the target image according to the point cloud of the vehicle's traveling direction and the target image; calibrates multiple camera normal vectors of the camera that obtains the target image to obtain multiple ground normal vectors; and fits the ground plane of the vehicle's traveling direction according to the ground point cloud set and the multiple ground normal vectors to obtain a fitted ground plane, which can improve the accuracy of the obtained ground normal vector, thereby effectively improving the accuracy of the fitted ground plane and assisting the safe driving of the autonomous vehicle.

3: Vehicle-mounted device

30: Ground plane fitting system

31: Storage

32: Processor

33: Radar device

34: Camera

S1~S3: Steps

S11~S12: Steps

S21~S22: Steps

S211~S212: Steps

In order to more clearly illustrate the technical solutions in the embodiments of this application or the known technology, the drawings required for the embodiments or the known technology description will be briefly introduced below. Obviously, the drawings described below are only embodiments of this application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative labor.

Figure 1 is a flow chart of the ground plane fitting method provided by the embodiment of this application.

Figure 2 is a flow chart of determining the ground point cloud set corresponding to the target image provided by the embodiment of this application.

Figure 3 is a flowchart of obtaining multiple ground normal vectors provided by the embodiment of this application.

Figure 4 is a flow chart of obtaining the camera coordinate system provided by the embodiment of this application.

Figure 5 is a structural diagram of the vehicle-mounted device provided in the embodiment of this application.

In order to more clearly understand the above-mentioned purpose, features and advantages of this application, the following is a detailed description of this application in conjunction with the attached drawings and specific embodiments. It should be noted that the embodiments of this application and the features in the embodiments can be combined with each other without conflict.

In the following description, many specific details are explained to facilitate a full understanding of this application. The embodiments described are only part of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without creative labor are within the scope of protection of this application.

Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by technicians in the technical field of this application. The terms used in this specification of this application are only for the purpose of describing specific embodiments and are not intended to limit this application.

In one embodiment, in the automatic driving technology, plane fitting of the ground in front of the vehicle is an indispensable part. The driving of the automatic driving vehicle can be controlled according to the ground plane obtained by fitting. For example, when the ground plane indicates that the ground in front is downhill and the slope is large, the automatic driving vehicle is controlled to slow down. The existing ground plane fitting method has problems such as low fitting accuracy.

In order to solve the above problems, the ground plane fitting method provided in the embodiment of the present application determines the ground point cloud set corresponding to the target image according to the point cloud in the direction of vehicle travel and the target image; calibrates the multiple camera normal vectors of the camera that obtains the target image to obtain multiple ground normal vectors; and fits the ground plane in the direction of vehicle travel according to the ground point cloud set and the multiple ground normal vectors to obtain a fitted ground plane, which can improve the accuracy of the obtained ground normal vector, thereby effectively improving the accuracy of the fitted ground plane and assisting the safe driving of the autonomous vehicle.

Refer to Figure 1, which is a flow chart of the ground plane fitting method of the preferred embodiment of this application.

In this embodiment, the ground plane fitting method can be applied to a vehicle-mounted device installed in a vehicle (e.g., the vehicle-mounted device 3 shown in FIG. 5 ). For a vehicle-mounted device that needs to perform ground plane fitting, the ground plane fitting function provided by the method of the embodiment of the present application can be directly integrated on the vehicle-mounted device, or run on the vehicle-mounted device in the form of a software development kit (SDK).

As shown in Figure 1, the ground plane fitting method specifically includes the following steps. According to different requirements, the order of the steps in the flowchart can be changed, and some steps can be omitted.

Step S1, based on the obtained point cloud of the vehicle's travel direction and the target image, determine the ground point cloud set corresponding to the target image.

In one embodiment, the vehicle-mounted device may include multiple sensor devices, such as a point cloud acquisition device and an image acquisition device. Specifically, the point cloud acquisition device may include a radar device (such as a millimeter wave radar), and the image acquisition device may include a camera (such as a monocular camera). In the embodiment of the present application, the vehicle-mounted device may be a device configured in the vehicle, installed in the vehicle and having a corresponding software system to execute various instructions; or it may be an independent electronic device, for example, an external device (such as a mobile phone, computer, tablet computer, etc.) that can be connected to the vehicle through communication to achieve the acquisition of vehicle data and control of the vehicle.

The radar device can be installed in the vehicle, such as the front windshield of the vehicle, to obtain a point cloud (e.g., a three-dimensional point cloud) in the direction of the vehicle's travel; the camera can be installed in the front windshield of the vehicle, to obtain an image (e.g., a two-dimensional image) in the direction of the vehicle's travel. The camera can be a dash cam installed in the vehicle, or one or more cameras, which can be installed on the vehicle, or an independent device connected to the vehicle-mounted device through a network or other means. The actual application is not limited to the above examples.

In one embodiment, each image obtained by the camera contains a corresponding timestamp according to the frame rate of the camera. For example, if the frame rate of the camera is 30 frames per second, then the time interval between two timestamps corresponding to two adjacent images obtained by the camera is 1/30 second.

In one embodiment, the target image may represent any two-dimensional image of the vehicle's traveling direction obtained by the camera, and the timestamp corresponding to the target image may be represented as the target timestamp. The ground plane fitting method provided in the embodiment of the present application is used to fit the ground plane of the vehicle's traveling direction corresponding to the target timestamp, thereby controlling the driving of the autonomous vehicle according to the fitted ground plane.

In one embodiment, step S1 is described in detail in conjunction with the flowchart shown in FIG2. FIG2 is a flowchart for determining the ground point cloud set corresponding to the target image provided in the embodiment of the present application, which specifically includes the following steps:

Step S11, determine the ground area in the target image.

In one embodiment, there are many image recognition methods for determining the ground area in the target image, including but not limited to: using a semantic segmentation algorithm to identify the ground area in the target image.

In one embodiment, the semantic segmentation algorithm can classify the primitive points in the target image point by point according to the different semantic categories (e.g., ground, vehicle) expressed by the primitive points in the target image, thereby grouping or segmenting the primitive points in the target image to obtain the regions where objects of different types are located in the target image (e.g., ground regions).

In one embodiment, a training sample can also be obtained to pre-train a semantic segmentation model, so that the target image is directly input into the semantic segmentation model to obtain the ground region of the target image. The semantic segmentation model can include a dilated convolution semantic segmentation model based on full convolution, for example, a RefineNet model.

Specifically, the RefineNet model network consists of independent RefineNet modules, each of which includes: residual convolutional unit (RCU) block, multi-resolution fusion (MRF) layer and chained residual pooling (CRP) layer. Among them, the RCU block consists of a self-adjusting block to form a convolution set, which is used to segment the image based on the ResNet weight; the MRF layer fuses different activations to use convolution and upsampling layers to improve the resolution of the image; the CRP layer uses kernels of various sizes in the pool to obtain the global receptive field from a larger image area.

Step S12, projecting the point cloud to the target image, and taking the set of point clouds corresponding to the ground area as the ground point cloud set.

In one embodiment, projecting the point cloud onto the target image includes: jointly calibrating a radar device for acquiring the point cloud and a camera for acquiring the target image; and fusing the point cloud with the target image.

In the multi-sensor detection system composed of the radar device and the camera, the two sensors need to be jointly calibrated before the point cloud and the target image are fused, so as to obtain the correspondence between the point of the point cloud and the pixel of the target image, and project the three-dimensional point cloud into the two-dimensional image to complete the fusion of radar information and image information.

In one embodiment, the basic principle of jointly calibrating and fusing radar information and image information includes: obtaining external parameters (rotation matrix, translation vector, etc.) of the radar device and the camera respectively, and obtaining a transformation matrix between the world coordinate system of the radar device and the camera coordinate system of the camera based on the external parameters (for example, a coordinate transformation relationship matrix calculated based on the Perspective-n-Point algorithm); projecting points in the three-dimensional point cloud coordinate system into the camera coordinate system based on the transformation matrix; obtaining the intrinsic parameters of the camera (focal length, principal point, tilt coefficient, distortion coefficient, etc.) by calibrating the camera, eliminating the distortion effect of the convex lens of the camera based on the intrinsic parameters, and projecting the points in the three-dimensional coordinate system where the camera is located into a two-dimensional image. Specifically, a variety of calibration tools can be used to implement the above process, such as APOLLO's sensor equipment calibration tool, AUTOWARE's CalibrationTookit module, etc.

In one embodiment, after the point cloud is projected onto the target image, the set of point clouds corresponding to the ground area of the target image is the ground point cloud set in the point cloud.

Step S2, calibrating multiple camera normal vectors of the camera that obtains the target image to obtain multiple ground normal vectors.

In one embodiment, since the ground on which the autonomous vehicle is driving may have a slope, when the vehicle is tilted due to the ground slope, the deflection angle of the world coordinate system referenced by the camera installed in the vehicle will change. At this time, if the camera posture of the camera is always determined according to the original world coordinate system, and the camera normal vector corresponding to the camera posture is used as the ground normal vector, the fitted ground plane will have a large error, so the camera normal vector needs to be corrected.

In one embodiment, the detailed process of step S2 is described in conjunction with FIG3. FIG3 is a flowchart of obtaining multiple ground normal vectors provided in the embodiment of the present application, which specifically includes the following steps:

Step S21, obtaining multiple camera normal vectors of the camera including time stamps.

In one embodiment, the deflection angle of the camera normal vector of the camera can be used as the deflection angle of the world coordinate system referenced by the camera. When determining the camera normal vector, the camera coordinate system of the camera can be determined first.

Among them, in an embodiment of the present application, the flowchart for obtaining the camera coordinate system can refer to the process shown in Figure 4, which specifically includes the following steps:

Step S211, obtaining multiple images taken by the camera, wherein the multiple images include the target image, and each of the multiple images corresponds to a timestamp.

In one embodiment, the multiple images may be multiple continuous images with the target image as the last image. For example, after determining the target image, three continuous images are obtained by inferring from the target image, wherein the target image is the last image of the three continuous images. For the description of the timestamp, reference may be made to the corresponding record in step S1.

Since the frame rate of the camera is relatively large, the actual time span of the multiple images is very small, and can be regarded as a set of images taken by the vehicle of the same ground. For example, when the multiple images are 3, and the frame rate of the camera is 30 frames/second, the actual corresponding time span is only (3-1)×1/30=1/15 seconds. The distance traveled by the vehicle in 1/15 seconds is very small, so the multiple images can be regarded as a set of images taken by the vehicle of the same ground.

Step S212, obtain the camera coordinate system corresponding to each image.

In one embodiment, due to the change of the ground slope, the camera coordinate system corresponding to each image may change slightly, so the camera coordinate system corresponding to each image can be obtained. Specifically, the rotation matrix and translation matrix between the camera and the referenced world coordinate system can be determined by referring to the intrinsic and extrinsic parameters of the camera in step S1 to obtain the camera coordinate system. The method for determining the camera coordinate system is a commonly used technical means in this field and will not be described in detail.

In one embodiment, after obtaining the camera coordinate system, obtaining multiple camera normal vectors containing time stamps of the camera includes: using the coordinate axis pointing to the sky in the camera coordinate system of the camera as the camera normal vector. For example, when the camera coordinate system is an OXYZ coordinate system determined according to the right-hand rule, the camera normal vector represents the Z axis pointing to the sky.

In one embodiment, multiple camera normal vectors corresponding to multiple images are obtained in the above embodiment, which can improve the accuracy of the corrected ground normal vector in the subsequent steps.

Step S22, obtaining the motion data of the camera, and using the motion data to calibrate the multiple camera normal vectors to obtain the multiple ground normal vectors.

In one embodiment, in order to correct the normal vector of the camera, the vehicle-mounted device further includes an inertial motion unit (IMU) sensor. The IMU sensor can be installed inside or outside the camera. The IMU sensor includes an accelerometer and a gyroscope sensor, which can be used to detect the movement data of the camera at all times (such as acceleration and angular velocity, etc.), thereby obtaining the movement data including a timestamp.

Furthermore, the time accuracy of the IMU sensor is greater than the time accuracy corresponding to the frame rate of the camera. For example, the camera acquires a picture every 1/30 second, and the IMU sensor acquires the motion data every 1/60 second. Therefore, the timestamp corresponding to each image can correspond to the motion data of the same timestamp, so that each camera normal vector corresponding to each image can correspond to the motion data of the same timestamp, wherein the motion data includes gravity acceleration.

In one embodiment, the method of correcting the multiple camera normal vectors using the movement data to obtain the multiple ground normal vectors includes: correcting the rotation matrix in the camera posture of the camera using the direction of the gravity acceleration to correct the camera normal vector to the ground normal vector.

In one embodiment, the direction of gravity acceleration is toward the center of the earth and will not change in a short time. Therefore, the directions of gravity acceleration corresponding to the multiple camera normal vectors are unified, so the direction of gravity acceleration can be used to correct the rotation matrix in the camera posture of the camera. The camera posture represents the position and posture of the camera relative to the world coordinate system. The camera posture includes the rotation matrix of the camera coordinate system and the world coordinate system. The rotation matrix includes three matrices, one of which is the normal vector rotation matrix corresponding to the deflection angle of the camera normal vector and the longitudinal axis of the world coordinate system pointing to the sky.

Specifically, using the direction of the gravitational acceleration to correct the rotation matrix in the camera posture of the camera includes: determining the deflection angle between the opposite direction of the gravitational acceleration and the direction of the longitudinal axis of the world coordinate system pointing to the sky, determining the deflection matrix between the opposite direction and the direction of the longitudinal axis of the world coordinate system pointing to the sky according to the deflection angle, and multiplying the deflection matrix with the normal vector rotation matrix in the camera posture to obtain the corrected normal vector rotation matrix, thereby obtaining the corrected rotation matrix of the camera posture composed of the corrected normal vector rotation matrix. Among them, the method of determining the rotation matrix of the world coordinate system according to the deflection angle or the rotation angle can adopt the common method in the technical field, which will not be described again.

In one embodiment, after obtaining the corrected rotation matrix of the camera posture, the corrected camera coordinate system of the camera can be obtained, thereby obtaining the corrected camera normal vector as the ground normal vector.

Step S3, fitting the ground plane in the direction of travel of the vehicle according to the ground point cloud set and the multiple ground normal vectors to obtain a fitted ground plane.

In one embodiment, the ground plane in the direction of travel of the vehicle is fitted according to the ground point cloud set and the multiple ground normal vectors to obtain a fitted ground plane, including: using the ground point cloud set and the multiple ground normal vectors as constraints for fitting a ground plane equation, and fitting the ground plane corresponding to the ground using the least squares method based on the constraints.

In one embodiment, a ground normal vector and a ground point cloud set are usually used as a constraint condition or only a ground point cloud set containing depth information is used as a constraint condition, and the ground plane is fitted using the least squares method, but the accuracy of the fitted ground plane is usually difficult to guarantee. Therefore, in the embodiment of the present application, multiple ground normal vectors obtained in the above embodiment are used to obtain a more accurate ground plane.

In one embodiment, the method of least squares fitting plane equations is a common technique in this field, and its basic principle includes minimizing the sum of squares of errors between multiple data (for example, multiple ground normal vectors or ground point cloud sets) to find the best function match for multiple data, which will not be described further.

In one embodiment, after obtaining the ground plane, the automatic driving vehicle can be controlled to drive according to the ground plane. For example, when the ground plane indicates that the ground ahead is an uphill slope with a large slope, the speed of the automatic driving vehicle is controlled to increase.

In addition, in the above embodiment, the ground area can be divided into multiple sub-areas, and multiple ground planes can be obtained by fitting multiple ground point cloud sets corresponding to the multiple sub-areas, so as to further improve the accuracy of the fitted front ground, indicate a more accurate direction of travel for the self-driving vehicle, and improve the riding experience for the users in the car. For example, if the ground plane corresponding to a certain sub-area indicates collapse, the vehicle can be controlled to avoid the collapsed ground to prevent the passengers in the car from feeling bumpy.

In one embodiment, the ground plane fitting method provided by the present application determines a ground point cloud set corresponding to a target image according to a point cloud in the direction of vehicle travel and a target image; calibrates multiple camera normal vectors of a camera that obtains the target image to obtain multiple ground normal vectors; and fits the ground plane in the direction of vehicle travel according to the ground point cloud set and the multiple ground normal vectors to obtain a fitted ground plane, which can improve the accuracy of the obtained ground normal vector, thereby effectively improving the accuracy of the fitted ground plane and assisting the safe driving of autonomous vehicles.

The above FIG1 introduces the ground plane fitting method of the present application in detail. The following introduces the functional modules of the software system for implementing the ground plane fitting method and the hardware device architecture for implementing the ground plane fitting method in conjunction with FIG5.

It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

See Figure 5, which is a schematic diagram of the structure of the vehicle-mounted device provided in the preferred embodiment of this application.

In the preferred embodiment of the present application, the vehicle-mounted device 3 includes a storage device 31, at least one processor 32, at least one radar device 33, and at least one camera 34. Those skilled in the art should understand that the structure of the vehicle-mounted device shown in FIG. 5 does not constitute a limitation of the present application embodiment, and it can be a bus structure or a star structure. The vehicle-mounted device 3 can also include more or less other hardware or software than shown in the figure, or different component configurations.

In some embodiments, the vehicle-mounted device 3 includes a terminal capable of automatically performing numerical calculations and/or information processing according to pre-set or stored instructions, and its hardware includes but is not limited to a microprocessor, a dedicated integrated circuit, a programmable logic gate array, a digital signal processor, and an embedded device, etc.

It should be noted that the vehicle-mounted device 3 is only an example. Other existing or future electronic products that are suitable for this application should also be included in the protection scope of this application and included here by reference.

In some embodiments, the memory 31 is used to store program codes and various data. For example, the memory 31 can be used to store the ground plane simulation system 30 installed in the vehicle-mounted device 3, and realize high-speed and automatic access to programs or data during the operation of the vehicle-mounted device 3. The storage device 31 includes a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), a one-time programmable read-only memory (OTPROM), an electronically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, a magnetic disk storage, a magnetic tape storage, or any other computer-readable storage medium that can be used to carry or store data.

In some embodiments, the at least one processor 32 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units, microprocessors, digital signal processing chips, graphics processors, and combinations of various control chips. The at least one processor 32 is the control core (Control Unit) of the vehicle-mounted device 3, and uses various interfaces and lines to connect various components of the entire vehicle-mounted device 3, and executes or runs programs or modules stored in the memory 31, and calls data stored in the memory 31 to execute various functions of the vehicle-mounted device 3 and process data, such as executing the ground plane fitting function shown in Figure 1.

In some embodiments, the ground plane fitting system 30 runs in the vehicle-mounted device 3. The ground plane fitting system 30 may include a plurality of functional modules composed of program code segments. The program code of each program segment in the ground plane fitting system 30 may be stored in the memory 31 of the vehicle-mounted device 3 and executed by at least one processor 32 to realize the ground plane fitting function shown in FIG. 1.

In this embodiment, the ground plane simulation system 30 can be divided into multiple functional modules according to the functions it performs. The module referred to in this application refers to a series of computer program segments that can be executed by at least one processor and can complete fixed functions, which are stored in a memory.

Although not shown, the vehicle-mounted device 3 may also include a power source (such as a battery) for supplying power to various components. Preferably, the power source may be logically connected to the at least one processor 32 through a power management device, so as to manage charging, discharging, and power consumption through the power management device. The power source may also include one or more DC or AC power sources, recharging devices, power fault test circuits, power converters or inverters, power status indicators, and other arbitrary components. The vehicle-mounted device 3 may also include a variety of sensors, Bluetooth modules, Wi-Fi modules, etc., which will not be elaborated here.

It should be understood that the embodiments described are for illustrative purposes only and are not limited to this structure in the scope of the patent application.

The above-mentioned integrated unit implemented in the form of a software function module can be stored in a computer-readable storage medium. The above-mentioned software function module is stored in a storage medium, including several instructions for enabling a vehicle-mounted device (which can be a server, a personal computer, etc.) or a processor to execute a part of the method described in each embodiment of the present application.

The memory 31 stores program codes, and the at least one processor 32 can call the program codes stored in the memory 31 to execute related functions. The program codes stored in the memory 31 can be executed by the at least one processor 32, thereby realizing the functions of each module to achieve the purpose of ground plane fitting.

In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division, and there may be other division methods in actual implementation.

The modules described as separate components may or may not be physically separated, and the components shown as modules may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of this embodiment.

In addition, each functional module in each embodiment of the present application can be integrated into a processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional modules.

It is obvious to those skilled in the art that the present application is not limited to the details of the above exemplary embodiments and that the present application can be implemented in other specific forms without departing from the spirit or basic characteristics of the present application. Therefore, no matter from which point of view, the embodiments should be regarded as exemplary and non-restrictive, and the scope of the present application is defined by the attached claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims are included in the present application. Any figure marks in the claims should not be regarded as limiting the claims involved. In addition, it is obvious that the word "including" does not exclude other units or, and the singular does not exclude the plural. Multiple units or devices stated in the device claim may also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, not to indicate any particular order.

Finally, it should be noted that the above embodiments are only used to illustrate the technical solution of this application and are not limiting. Although this application is described in detail with reference to the above preferred embodiments, ordinary technicians in this field should understand that the technical solution of this application can be modified or replaced by equivalents without departing from the spirit and scope of the technical solution of this application.

S1~S3: Steps

Claims (10)

A ground plane fitting method is applied to a vehicle-mounted device, wherein the method includes: determining a ground point cloud set corresponding to the target image based on a point cloud and a target image obtained in the direction of vehicle travel; correcting multiple camera normal vectors of a camera that obtains the target image to obtain multiple ground normal vectors; fitting the ground plane in the direction of vehicle travel according to the ground point cloud set and the multiple ground normal vectors to obtain a fitted ground plane. The ground plane fitting method as described in claim 1, wherein the method of determining the ground point cloud set corresponding to the target image based on the obtained point cloud of the vehicle's travel direction and the target image comprises: determining the ground area in the target image; projecting the point cloud to the target image, and taking the set of point clouds corresponding to the ground area as the ground point cloud set. The ground plane fitting method as described in claim 2, wherein determining the ground area in the target image includes: identifying the ground area in the target image using a semantic segmentation algorithm. As described in claim 1, the method for fitting a ground plane, wherein the correction of multiple camera normal vectors of a camera that obtains the target image to obtain multiple ground normal vectors includes: obtaining multiple camera normal vectors containing time stamps of the camera; obtaining motion data of the camera, and correcting the multiple camera normal vectors using the motion data to obtain the multiple ground normal vectors. The ground plane fitting method as described in claim 4, wherein the step of obtaining multiple camera normal vectors of the camera including a timestamp includes: using the coordinate axis pointing to the sky in the camera coordinate system of the camera as the camera normal vector. The ground plane fitting method as described in claim 5, wherein the method further comprises: obtaining multiple images taken by the camera, the multiple images including the target image, and each of the multiple images corresponds to a timestamp; obtaining the camera coordinate system corresponding to each image. The ground plane fitting method as described in claim 4, wherein the movement data of the camera includes gravity acceleration; the use of the movement data to correct the multiple camera normal vectors to obtain the multiple ground normal vectors includes: using the direction of the gravity acceleration to correct the rotation matrix in the camera posture of the camera, and correcting the camera normal vector to the ground normal vector. The ground plane fitting method as described in claim 1, wherein the ground plane in the direction of vehicle travel is fitted according to the ground point cloud set and the multiple ground normal vectors to obtain the fitted ground plane, including: using the ground point cloud set and the multiple ground normal vectors as constraints for fitting the ground plane equation, and fitting the ground plane corresponding to the ground using the least squares method based on the constraints. A computer-readable storage medium, wherein the computer-readable storage medium stores at least one instruction, and when the at least one instruction is executed by a processor, the ground plane fitting method as described in any one of claims 1 to 8 is implemented. A vehicle-mounted device, wherein the vehicle-mounted device includes a memory and at least one processor, wherein at least one instruction is stored in the memory, and when the at least one instruction is executed by the at least one processor, the ground plane fitting method as described in any one of claims 1 to 8 is implemented.

Images