WO2020133172A1

WO2020133172A1 - Image processing method, apparatus, and computer readable storage medium

Info

Publication number: WO2020133172A1
Application number: PCT/CN2018/124726
Authority: WO
Inventors: 崔健
Original assignee: 深圳市大疆创新科技有限公司
Priority date: 2018-12-28
Filing date: 2018-12-28
Publication date: 2020-07-02
Also published as: CN111279354A

Abstract

An image processing method, an apparatus, and a computer readable storage medium. Said method comprises: acquiring, by means of a photographing device, a plan view image containing a target object; determining a spatial plane corresponding to the target object; determining the relative posture of the spatial plane and the photographing device; and converting, according to the relative posture, the plan view image into a top view image. The application of the embodiments of the present invention improves the accuracy of the detection of a lane line, and accurately positions a relationship between the lane line and the actual position of a vehicle.

Description

Image processing method, device and computer readable storage medium

Technical field

Embodiments of the present invention relate to the field of image processing technology, and in particular, to an image processing method, device, and computer-readable storage medium.

Background technique

In areas such as autonomous driving and ADAS (Advanced Driver Assistance Systems), lane line algorithms play an important role. The accuracy of lane line algorithms will directly affect the performance and reliability of the system. The lane line algorithm is automatic driving An important prerequisite for car control.

The lane line algorithm is divided into two levels, one is the detection of the lane line, and the other is the positioning of the lane line, which is to calculate the actual positional relationship between the lane line and the car. The traditional lane line detection algorithm can collect the head-up image through the shooting device, and use the head-up image to detect the lane line. The traditional lane line positioning algorithm can collect the head-up image through the shooting device, and use the head-up image to locate the lane line.

When using the head-up image to detect the lane line, there is a problem that the detection result is inaccurate. For example, the size and nature of the lane line in the head-up image are all through perspective projection, which has the effect of "near big and far small", resulting in distant Some pavement markers are distorted in shape and cannot be detected correctly. When using the head-up image to locate the lane line, there is a problem that the positioning result is not accurate. For example, the shape and size of the road surface marker in the head-up image are coupled with the positional relationship between the internal parameters of the camera, the camera and the road surface, It is impossible to directly know the actual position of the lane line by looking directly at the position in the image.

Summary of the invention

The invention provides an image processing method, device and computer-readable storage medium, which can improve the accuracy of detection of lane lines and accurately locate the actual positional relationship between lane lines and vehicles.

According to a first aspect of the present invention, there is provided a driving assistance device including at least one photographing device, a processor, and a memory; the driving assistance device is provided on a vehicle and communicates with the vehicle; the memory, For storing computer instructions executable by the processor;

The photographing device is configured to collect a head-up image including a target object, and send the head-up image including the target object to the processor;

The processor is configured to read computer instructions from the memory to implement:

Acquiring a head-up image containing a target object from the shooting device;

Determine the space plane corresponding to the target object;

Determine the relative posture of the space plane and the shooting device;

The head-up image is converted into a top-down image according to the relative posture.

According to a second aspect of the embodiments of the present invention, there is provided a vehicle equipped with a driving assistance system. The vehicle includes at least one camera, a processor, and a memory. The memory is used to store computer instructions executable by the processor. The shooting device is used to collect a head-up image containing a target object, and send the head-up image containing a target object to the processor;

Acquiring a head-up image containing a target object from the shooting device;

Determine the space plane corresponding to the target object;

Determine the relative posture of the space plane and the shooting device;

According to a third aspect of the embodiments of the present invention, an image processing method is provided, which is applied to a driving assistance system. The driving assistance system includes at least one photographing device. The method includes:

Acquiring the head-up image containing the target object through the shooting device;

Determine the space plane corresponding to the target object;

Determine the relative posture of the space plane and the shooting device;

According to a fourth aspect of the embodiments of the present invention, a computer-readable storage medium is provided. Computer instructions are stored on the computer-readable storage medium. When the computer instructions are executed, the above method is implemented.

Based on the above technical solution, in the embodiments of the present invention, the accuracy of the lane line detection can be improved, and the actual positional relationship between the lane line and the vehicle can be accurately located. Specifically, the head-up image can be converted into a bird's-eye view image, and the bird's-eye view image can be used to detect the lane line, thereby improving the accuracy of the lane line detection result. The head-up image can be converted into a bird's-eye view image, and the bird's-eye view image is used to locate the lane line, thereby improving the accuracy of the lane line positioning result and accurately knowing the actual position of the lane line.

BRIEF DESCRIPTION

In order to more clearly explain the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings used in the embodiments of the present invention or the description of the prior art. Obviously, the drawings in the following description It is only some of the embodiments described in the present invention. For those of ordinary skill in the art, other drawings can also be obtained from these drawings of the embodiments of the present invention.

FIG. 1 is a schematic diagram of an example of an image processing method in an embodiment;

2 is a schematic diagram of an example of an image processing method in another embodiment;

3 is a schematic diagram of an example of an image processing method in another embodiment;

4A is a schematic diagram of a head-up image and a top-down image of an image processing method in an embodiment;

4B is a schematic diagram of the relationship between the target object, the space plane and the camera in an embodiment;

5 is a block diagram of an example of a driving assistance device in an embodiment.

detailed description

The technical solutions in the embodiments of the present invention will be described clearly and completely in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present invention. In addition, in the case of no conflict, the following embodiments and the features in the embodiments can be combined with each other.

The terminology used in the present invention is for the purpose of describing specific embodiments only, and does not limit the present invention. The singular forms "a", "said" and "the" used in the present invention and claims are also intended to include the majority forms unless the context clearly indicates other meanings. It should be understood that the term "and/or" as used herein refers to any or all possible combinations including one or more associated listed items.

Although the terms first, second, third, etc. may be used to describe various information in the present invention, the information should not be limited to these terms. These terms are used to distinguish the same type of information from each other. For example, without departing from the scope of the present invention, the first information may also be referred to as second information, and similarly, the second information may also be referred to as first information. Depending on the context, in addition, the word "if" can be interpreted as "when", or "when", or "in response to a determination".

Example 1:

An embodiment of the present invention proposes an image processing method, which can be applied to a driving assistance system, and the driving assistance system may include at least one photographing device. Wherein, the driving assistance system may be mounted on a mobile platform (such as unmanned vehicles, ordinary vehicles, etc.), or the driving assistance system may also be mounted on driving assistance equipment (such as ADAS equipment, etc.), and the driving assistance equipment It is installed on a mobile platform (such as unmanned vehicles, ordinary vehicles, etc.). Of course, the above is only an example of two application scenarios, and the driving assistance system can also be carried on other vehicles, which is not limited.

Referring to FIG. 1, which is a schematic flowchart of an image processing method, the method may include:

Step 101: Obtain a head-up image containing a target object through a camera.

Specifically, if the driving assistance system is mounted on a mobile platform, the at least one shooting device is installed on the mobile platform, and at least one of the front, rear, left, or right of the mobile platform can be acquired through the shooting device The head-up image of the direction, the head-up image contains the target object.

If the driving assistance system is mounted on the driving assistance device, the at least one imaging device is provided in the driving assistance device, and at least one of the front, rear, left, or right directions of the driving assistance device can be acquired through the imaging device The head-up image contains the target object.

Step 102: Determine a space plane corresponding to the target object.

Specifically, if the driving assistance system is mounted on a mobile platform, the first posture information of the mobile platform (that is, the current posture information of the mobile platform) may be acquired, and the space plane may be determined according to the first posture information. The space plane refers to the position plane of the target object (such as road surface or ground) in the world coordinate system, that is, the position of the space plane in the world coordinate system.

If the driving assistance system is mounted on the driving assistance device, the second posture information of the driving assistance device (that is, the current posture information of the driving assistance device) may be acquired, and the space plane may be determined according to the second posture information. The space plane refers to the position plane of the target object (such as road surface or ground) in the world coordinate system, that is, the position of the space plane in the world coordinate system.

Step 103: Determine the relative posture of the space plane and the shooting device.

In one example, the relative posture refers to the relative posture of the shooting device relative to the space plane (such as road surface or ground), and can also be understood as the external parameter (that is, positional relationship) of the shooting device relative to the space plane . For example, the relative posture may include, but is not limited to: a pitch angle of the camera relative to the space plane, a roll angle of the camera relative to the space plane, and the camera relative to space The yaw of the plane, the height of the camera relative to the space plane, and the translation parameter of the camera relative to the space plane.

Step 104: Convert the head-up image to the top-down image according to the relative posture.

Specifically, the projection matrix corresponding to the head-up image can be obtained according to the relative pose; for example, the target rotation matrix can be determined according to the relative pose, the target rotation parameter can be obtained according to the target rotation matrix, and the relative pose and the target rotation parameter can be obtained The projection matrix corresponding to the head-up image. Then, the head-up image can be converted into a bird's-eye view image according to the projection matrix.

Wherein, the relative attitude includes the rotation angle of the camera on the pitch axis (that is, the pitch angle of the camera relative to the space plane), the rotation angle on the roll axis (that is, the roll angle of the camera relative to the space plane), and The rotation angle of the yaw axis (that is, the yaw angle of the camera relative to the space plane); based on this, the target rotation matrix is determined according to the relative attitude, which may include, but is not limited to: determining the first angle according to the rotation angle of the camera on the pitch axis A rotation matrix; determine the second rotation matrix according to the rotation angle of the camera on the roll axis; determine the third rotation matrix according to the rotation angle of the camera on the yaw axis; according to the first rotation matrix, the second rotation matrix, and the third rotation The matrix determines the target rotation matrix.

The target rotation matrix may include three column vectors, and the target rotation parameters may be obtained according to the target rotation matrix, which may include but not limited to: determine the first column vector in the target rotation matrix as the first rotation parameter, and determine The second column vector in the target rotation matrix is determined as the second rotation parameter; the first rotation parameter and the second rotation parameter are determined as the target rotation parameter.

Wherein, the relative posture also includes a translation parameter of the space plane and the shooting device (that is, a translation parameter of the shooting device relative to the space plane), and obtaining a projection matrix according to the relative posture and the target rotation parameter may include but not limited to: The target rotation parameter, the normalization coefficient, the internal parameter matrix of the shooting device, the space plane, and the translation parameter of the shooting device are obtained, and the projection matrix is obtained.

In the above embodiment, converting the head-up image into a bird's-eye view image according to the projection matrix may include, but is not limited to: for each first pixel in the head-up image, the The position information is converted into position information of the second pixel in the overhead image; based on this, the overhead image can be obtained according to the position information of each second pixel.

Wherein, the position information of the first pixel point is converted into the position information of the second pixel point in the bird's-eye view image according to the projection matrix, which may include but is not limited to: obtaining the inverse matrix corresponding to the projection matrix, and according to the The inverse matrix converts the position information of the first pixel point to the position information of the second pixel point in the bird's-eye view image, that is, each first pixel point corresponds to a second pixel point.

In one example, after the head-up image is converted into a bird's-eye image according to the relative posture, if the target object is a lane line, the lane line can be detected based on the bird's-eye image.

In one example, after converting the head-up image into a bird's-eye view image according to the relative posture, if the target object is a lane line, the lane line can be positioned according to the bird's-eye view image.

In summary, the lane line detection can be performed based on the top view image (not the lane line detection based on the head-up image) to improve the accuracy of the lane line detection. And/or, the lane line positioning is performed based on the top view image (not the lane line positioning based on the head-up image) to improve the accuracy of the lane line positioning.

Based on the above technical solution, in the embodiments of the present invention, the accuracy of the lane line detection can be improved, and the actual positional relationship between the lane line and the vehicle can be accurately located. Specifically, the head-up image can be converted into a top-down image, and the top-down image can be used to detect the lane line, thereby improving the accuracy of the lane-line detection result. The head-up image can be converted into a bird's-eye view image, and the bird's-eye view image is used to locate the lane line, thereby improving the accuracy of the lane line positioning result and accurately knowing the actual position of the lane line.

Example 2:

An embodiment of the present invention proposes an image processing method, which can be applied to a driving assistance system, and the driving assistance system may include at least one photographing device. Wherein, the driving assistance system can be mounted on a mobile platform (such as unmanned vehicles, ordinary vehicles, etc.). Of course, the above is only an example of the application scenario of the present invention, and the driving assistance system can also be mounted on other vehicles. limit.

Referring to FIG. 2, which is a schematic flowchart of an image processing method, the method may include:

Step 201: Obtain a head-up image containing a target object through a camera.

Specifically, the head-up image in at least one direction of the front, back, left, or right direction of the mobile platform may be acquired by the shooting device, and the head-up image includes a target object.

Step 202: Determine the space plane corresponding to the target object according to the first pose information of the mobile platform.

Specifically, the first posture information of the mobile platform may be obtained, and the space plane may be determined according to the first posture information. The space plane refers to the position plane of the target object (such as road surface or ground) in the world coordinate system, that is, the position of the space plane in the world coordinate system.

In one example, the process of acquiring the first posture information of the mobile platform, the mobile platform may include a posture sensor, the posture sensor collects the first posture information of the mobile platform, and provides the first posture information to the driving assistance system to enable driving assistance The system acquires the first posture information of the mobile platform. Of course, the first posture information of the mobile platform can also be obtained in other ways, which is not limited.

Among them, the attitude sensor is a high-performance three-dimensional motion attitude measurement system, which can include three-axis gyroscope, three-axis accelerometer (ie IMU), three-axis electronic compass and other auxiliary motion sensors, and output calibration through the embedded processor Sensor data such as angular velocity, acceleration, magnetic data, etc., and then, posture information can be measured based on the sensor data, and there is no restriction on the manner of acquiring posture information.

In one example, the process of determining the space plane corresponding to the target object according to the first pose information, after obtaining the first pose information of the mobile platform, the space plane can be determined according to the first pose information. I will not repeat them here.

Step 203: Determine the relative posture of the space plane and the shooting device.

In an example, the relative posture refers to the relative posture of the camera relative to the space plane, and can also be understood as the external parameter (ie, positional relationship) of the camera relative to the space plane. For example, the relative attitude may include but is not limited to: the pitch angle of the camera relative to the space plane, the roll angle of the camera relative to the space plane, the yaw angle of the camera relative to the space plane, and the height of the camera relative to the space plane , The translation of the camera relative to the space plane.

Step 204: Acquire a projection matrix corresponding to the head-up image according to the relative posture.

Specifically, a target rotation matrix may be determined according to the relative pose, a target rotation parameter may be obtained according to the target rotation matrix, and a projection matrix corresponding to the head-up image may be obtained according to the relative pose and the target rotation parameter. The process of obtaining the projection matrix will be described in detail in the subsequent embodiment 4.

Step 205: Convert the head-up image to the top-down image according to the projection matrix.

Specifically, for each first pixel in the head-up image, the position information of the first pixel is converted into the position information of the second pixel in the overhead image according to the projection matrix; based on this, it can be The top view image is obtained according to the position information of each second pixel.

Example 3:

An embodiment of the present invention proposes an image processing method, which can be applied to a driving assistance system, and the driving assistance system may include at least one photographing device. Wherein, the driving assistance system can also be equipped with driving assistance equipment (such as ADAS equipment, etc.), and the driving assistance equipment is installed on a mobile platform (such as unmanned vehicles, ordinary vehicles, etc.), of course, the above is only the application of the present invention For an example of a scenario, the driving assistance system can also be mounted on other vehicles, and there is no restriction on this.

Referring to FIG. 3, which is a schematic flowchart of an image processing method, the method may include:

Step 301: Obtain a head-up image containing a target object through a camera.

Specifically, the head-up image in at least one direction of the front, rear, left, or right direction of the driving assistance device may be acquired by the shooting device, and the head-up image includes a target object.

Step 302: Determine a space plane corresponding to the target object according to the second posture information of the driving assistance device. The space plane refers to the target object, that is, the position plane of the road surface or the ground in the world coordinate system.

Specifically, the second posture information of the driving assistance device may be acquired, and the space plane may be determined according to the second posture information. Wherein, the driving assistance device may include a posture sensor, and this posture sensor is used to collect the second posture information of the driving assistance device and provide the second posture information to the driving assistance system, so that the driving assistance system acquires the second posture of the driving assistance device information. Alternatively, the mobile platform may include an attitude sensor, the attitude sensor collects the first attitude information of the mobile platform, and provides the first attitude information to the driving assistance system. The driving assistance system may use the first attitude information of the mobile platform as the first position of the driving assistance device. The second posture information is the second posture information of the driving assistance device. Of course, the second posture information can also be obtained in other ways, which is not limited.

Step 303: Determine the relative posture of the space plane and the shooting device.

Step 304: Obtain a projection matrix corresponding to the head-up image according to the relative posture.

Step 305: Convert the head-up image to the top-down image according to the projection matrix.

Embodiment 4: A subsequent description will be made by taking a mobile platform as a vehicle and a shooting device as a camera as an example.

The traditional lane line algorithm can collect the head-up image through the camera, and use the head-up image to detect and locate the lane line. As shown in FIG. 4A, the image on the left is a schematic diagram of the head-up image. The road surface arrow and the lane line are twisted, and the shape is related to the position of the vehicle. Obviously, the lane line cannot be correctly performed based on the left head-up image in FIG. 4A. Detection and positioning. Different from the above method, in this embodiment, the head-up image is converted into a bird's-eye image, and the bird's-eye image is used to detect and locate the lane line. As shown in FIG. 4A, the image on the right is a schematic diagram of a top-down image. The arrows of the road surface markers and the lane lines are restored to true scales. The positions of the points on the road surface directly correspond to the real positions, and the positional relationship between a certain point and the vehicle can be directly obtained. It can meet the requirements of ADAS function and automatic driving function. Obviously, the detection and positioning of the lane line can be correctly performed based on the top view image on the right side of FIG. 4A.

Moreover, by converting the head-up image into a bird's-eye view image, the accuracy of road surface marker recognition can be improved, and a method for locating road surface markers (including lane lines) can be provided to assist in positioning.

In one example, in order to convert the head-up image into a top-down image, it can be implemented based on the geometric knowledge of computer vision, that is, convert the head-up image into a top-down image based on homography. Specifically, assuming that the head-up image is an image of a space plane and the top-down image is an image plane, the shape of the top-down image depends on the true shape of the head-up image of the space plane, the internal parameters of the camera, and the external parameters of the camera (that is, the camera relative to the space plane Position relationship), therefore, the pixels in the head-up image can be directly mapped to the top-down image according to the internal parameters of the camera and the external parameters of the camera, so as to correspond to the true scale of the spatial plane, improve the accuracy of lane line recognition, and provide Accurate positioning method of lane line.

Referring to FIG. 4B, it is a schematic diagram of the relationship between the target object, the space plane and the camera. The space plane is a plane including the target object, and the plane where the camera is located may be different from the space plane. For example, the target object may be a road (pavement or ground) containing lane lines as shown in the figure, and the spatial plane may be the plane where the target object is the road surface. The actual shooting screen of the camera is shown in the lower right corner of FIG. 4B, which corresponds to the head-up image on the left side of FIG. 4A.

In one example, homography can be expressed by the following formula, (u,v) is the pixel in the head-up image, that is, the pixel in the spatial plane, s is the normalization coefficient, M is the camera internal parameter, [r ₁ r ₂ r ₃ t] is the external parameter of the camera to the space plane, that is, the positional relationship, r ₁ is a 3*1 column vector, r ₂ is a 3*1 column vector, and r ₃ is a 3*1 column vector, and r ₁ , r ₂ and r ₃ form a rotation matrix, and t is a column vector of 3*1, which represents the translation of the camera to the object plane, that is, r ₁ , r ₂ and r ₃ form the rotation matrix and the translation t constitutes the camera pair The external parameters of the space plane, (X, Y) are the pixels in the overhead image, that is, the pixels in the image coordinate system.

In the above formula, the pixels in the overhead image can be (X, Y, Z), but considering that the target object is in a plane, that is, Z is 0, therefore, the product of r ₃ and Z is 0, that is to say After converting the homography formula, r ₃ and Z can be eliminated from the formula, and finally the following formula can be obtained.

In the above formula, assuming H=sM[r ₁ r ₂ t], the above formula can be converted into the following conversion matrix:

Further, by multiplying both sides of the formula by the inverse matrix of H at the same time, the following conversion matrix can be obtained:

It can be seen from the above formula that (X, Y) can be obtained when H and (u, v) are known.

In the above application scenario, the image processing method in the embodiment of the present invention may include:

Step a1: Obtain a head-up image containing a target object through a camera. Each pixel in the head-up image is called a first pixel, and each first pixel can be the above (u, v).

Step a2: Determine the space plane corresponding to the target object. The spatial plane refers to the position plane of the target object, that is, the road surface or ground on which it is located in the world coordinate system.

Step a3: Determine the relative posture of the space plane and the camera.

The relative posture can be the external parameter of the camera relative to the space plane (that is, the positional relationship), such as the pitch angle of the camera relative to the space plane, the roll angle of the camera relative to the space plane, and the camera relative to the space The yaw angle of the plane, the height of the camera relative to the space plane, and the translation parameter of the camera relative to the space plane, namely t in the above formula.

Step a4: Determine the target rotation matrix according to the relative posture.

For example, based on the above-mentioned relative attitude, a pitch angle of the camera relative to the space plane, a roll angle of the camera relative to the space plane, and a yaw angle of the camera relative to the space plane can be determined. Further, the first rotation matrix R _x can be determined according to the following formula based on the camera rotation angle on the pitch axis; the second rotation can be determined based on the camera rotation angle on the roll axis, and the second rotation can be determined based on the following formula Matrix R _y ; it can be based on the rotation angle (yaw) of the camera on the yaw axis, and the third rotation matrix R _z can be determined according to the following formula.

After obtaining the first rotation matrix, the second rotation matrix, and the third rotation matrix, based on the first rotation matrix, the second rotation matrix, and the third rotation matrix, the target rotation matrix R may be determined according to the following formula.

Step a5: Obtain the target rotation parameter according to the target rotation matrix.

For example, the first column vector in the target rotation matrix R can be determined as the first rotation parameter, and the second column vector in the target rotation matrix R can be determined as the second rotation parameter, and the first rotation parameter and the first The second rotation parameter is determined as the target rotation parameter. The first rotation parameter is r ₁ in the above formula, r ₁ is a column vector of 3*1, and the second rotation parameter is r ₂ in the above formula, r ₂ is a column vector of 3*1.

Step a6: Obtain a projection matrix according to the target rotation parameters r ₁ and r ₂ , the normalization coefficient, the camera's internal parameter matrix, and the translation parameter t. The projection matrix may be H in the above formula.

Among them, the normalization coefficient can be s in the above formula, the camera's internal parameter matrix can be M in the above formula, see the above formula H = sM [r ₁ r ₂ t], the target rotation parameters r ₁ and r ₂ , When the normalization coefficient s, the camera's internal parameter matrix M, and the translation parameter t are all known, the projection matrix H can be determined.

In the above formula, the camera's internal parameter matrix M can be

In the aforementioned internal reference matrix M, f _x , f _y can represent the focal length of the camera, c _x , c _y can represent the position of the camera lens optical axis through the imaging sensor f _x , f _y , c _x , c _y is a known value, there is no restriction on this.

In step a7, the head-up image can be converted into a bird's-eye view image according to the projection matrix.

Specifically, for each first pixel (u, v) in the head-up image, the position information of the first pixel can be converted to the position of the second pixel (X, Y) in the bird's-eye view image according to the projection matrix H Information, and obtain a bird's-eye view image according to the position information of each second pixel (X, Y), that is, the second pixel constitutes a bird's-eye view image. For example, based on the inverse matrix of the projection matrix H, you can refer to the above formula to convert the position information of the first pixel (u, v) to the position information of the second pixel (X, Y), and details are not described herein again.

Example 5:

Based on the same concept as the above method, referring to FIG. 5, an embodiment of the present invention also provides a driving assistance device 50 that includes at least one photographing device 51, a processor 52, and a memory 53; the driving The auxiliary device 50 is provided on the vehicle and communicates with the vehicle; the memory 53 is used to store computer instructions executable by the processor;

The shooting device 51 is configured to collect a head-up image including a target object, and send the head-up image including the target object to the processor 52;

The processor 52 is configured to read computer instructions from the memory 53 to implement:

Obtain a head-up image containing the target object from the shooting device 51;

Determine the space plane corresponding to the target object;

Determine the relative posture of the space plane and the shooting device;

The imaging device 51 is configured to acquire the head-up image in at least one direction of the front, rear, left, or right of the driving assistance device.

When the processor 52 determines the space plane corresponding to the target object, it is specifically used to:

Acquiring second posture information of the driving assistance device;

The spatial plane is determined according to the second posture information.

When the processor 52 converts the head-up image into a top-down image according to the relative posture, it is specifically used to: obtain a projection matrix corresponding to the head-up image according to the relative posture;

The head-up image is converted into a top-down image according to the projection matrix.

When the processor 52 obtains the projection matrix corresponding to the head-up image according to the relative posture, it is specifically used to: determine the target rotation matrix according to the relative posture;

Acquiring target rotation parameters according to the target rotation matrix;

Obtain the projection matrix according to the relative pose and the target rotation parameter.

The relative attitude includes the rotation angle of the shooting device on the pitch axis, the rotation angle on the roll axis, and the rotation angle on the yaw axis; the processor 52 is specifically used when determining the target rotation matrix according to the relative attitude : Determine the first rotation matrix according to the rotation angle of the shooting device on the pitch axis;

Determine the second rotation matrix according to the rotation angle of the shooting device on the roll axis;

Determine a third rotation matrix according to the rotation angle of the shooting device on the yaw axis;

The target rotation matrix is determined according to the first rotation matrix, the second rotation matrix, and the third rotation matrix.

When the processor 52 obtains a target rotation parameter according to the target rotation matrix, it is specifically used to:

Determine the first column vector in the target rotation matrix as the first rotation parameter;

Determine the second column vector in the target rotation matrix as the second rotation parameter;

The first rotation parameter and the second rotation parameter are determined as target rotation parameters.

The relative posture also includes translation parameters of the spatial plane and the shooting device; the processor 52 is specifically used when acquiring the projection matrix according to the relative posture and the target rotation parameter: according to the target rotation The parameters, the normalization coefficient, the internal parameter matrix of the photographing device, the spatial plane and the translation parameters of the photographing device are used to obtain the projection matrix.

When the processor 52 converts the head-up image into a top-down image according to the projection matrix, it is specifically used to: for each first pixel in the head-up image, convert the first pixel according to the projection matrix The position information of is converted into the position information of the second pixel in the overhead image;

The top view image is obtained according to the position information of each second pixel.

When the processor 52 converts the position information of the first pixel into the position information of the second pixel in the bird's-eye view image according to the projection matrix, it is specifically used to:

Acquiring an inverse matrix corresponding to the projection matrix, and converting the position information of the first pixel point to the position information of the second pixel point in the bird's-eye view image according to the inverse matrix.

Example 6:

Based on the same concept as the above method, an embodiment of the present invention also provides a vehicle equipped with a driving assistance system. The vehicle includes at least one camera, a processor, and a memory. The memory is used to store the processor. Computer instructions executed; the shooting device is used to collect a head-up image containing a target object, and send the head-up image containing the target object to the processor;

Acquiring a head-up image containing a target object from the shooting device;

Determine the space plane corresponding to the target object;

Determine the relative posture of the space plane and the shooting device;

The photographing device is configured to acquire the head-up image in at least one direction of the front, rear, left, or right of the vehicle.

When the processor determines the space plane corresponding to the target object, it is specifically used to: obtain first pose information of the vehicle; and determine the space plane according to the first pose information.

When the processor converts the head-up image into a bird's-eye view image according to the relative posture, it is specifically used to: obtain a projection matrix corresponding to the head-up image according to the relative posture;

When the processor obtains the projection matrix corresponding to the head-up image according to the relative posture, it is specifically used to: determine a target rotation matrix according to the relative posture;

Acquiring target rotation parameters according to the target rotation matrix;

The relative attitude includes the rotation angle of the shooting device on the pitch axis, the rotation angle on the roll axis, and the rotation angle on the yaw axis; when the processor determines the target rotation matrix according to the relative attitude, it is specifically used to: Determine the first rotation matrix according to the rotation angle of the shooting device on the pitch axis;

When the processor obtains the target rotation parameter according to the target rotation matrix, it is specifically used to:

The relative posture also includes translation parameters of the spatial plane and the shooting device; the processor is specifically used when acquiring the projection matrix according to the relative posture and the target rotation parameter:

The projection matrix is obtained according to the target rotation parameter, the normalization coefficient, the internal parameter matrix of the shooting device, the space plane, and the translation parameter of the shooting device.

When the processor converts the head-up image into a top-down image according to the projection matrix, it is specifically used to: for each first pixel in the head-up image, convert the first pixel according to the projection matrix. The position information is converted into the position information of the second pixel in the overhead image;

When the processor converts the position information of the first pixel point into the position information of the second pixel point in the bird's-eye view image according to the projection matrix, it is specifically used to:

Example 7:

An embodiment of the present invention further provides a computer-readable storage medium, on which computer instructions are stored, and when the computer instructions are executed, the above image processing method is implemented.

The system, device, module or unit explained in the above embodiments may be implemented by a computer chip or entity, or by a product with a certain function. A typical implementation device is a computer, and the specific form of the computer may be a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email sending and receiving device, and a game control Desk, tablet computer, wearable device, or any combination of these devices.

For the convenience of description, when describing the above device, the functions are divided into various units and described separately. Of course, when implementing the present invention, the functions of each unit can be implemented in one or more software and/or hardware.

Those skilled in the art should understand that the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Moreover, embodiments of the present invention may take the form of computer program products implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

The present invention is described with reference to flowcharts and/or block diagrams of methods, devices (systems), and computer program products according to embodiments of the present invention. It should be understood that each flow and/or block in the flowchart and/or block diagram and a combination of the flow and/or block in the flowchart and/or block diagram may be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing device to produce a machine that allows instructions generated by the processor of the computer or other programmable data processing device to be used A device for realizing the functions specified in one block or multiple blocks of one flow or multiple flows of a flowchart and/or one block or multiple blocks of a block diagram.

Moreover, these computer program instructions can also be stored in a computer readable memory that can guide a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce a manufactured product including an instruction device, The instruction device implements the functions specified in one block or multiple blocks of one flow or multiple blocks in the flowchart and/or block diagram.

These computer program instructions can also be loaded into a computer or other programmable data processing device so that a series of operating steps are performed on the computer or other programmable device to generate computer-implemented processing, thereby executing instructions on the computer or other programmable device Provides steps for implementing the functions specified in the flowchart flow one flow or flows and/or the block diagram one block or multiple blocks.

The above are only the embodiments of the present invention, and are not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and changes. Any modification, equivalent replacement, and improvement made within the spirit and principle of the present invention shall be included in the scope of the claims of the present invention.

Claims

A driving assistance device, characterized in that the driving assistance device includes at least one photographing device, a processor, and a memory; the driving assistance device is provided on a vehicle and communicates with the vehicle; and the memory is used to store Computer instructions executable by the processor;

The photographing device is configured to collect a head-up image including a target object, and send the head-up image including the target object to the processor;

The processor is configured to read computer instructions from the memory to implement:

Acquiring a head-up image containing a target object from the shooting device;

Determine the space plane corresponding to the target object;

Determine the relative posture of the space plane and the shooting device;

The head-up image is converted into a top-down image according to the relative posture.
The device according to claim 1, characterized in that

The shooting device is configured to acquire the head-up image in at least one direction of the front, rear, left, or right of the driving assistance device.
The device according to claim 1, characterized in that

When the processor determines the space plane corresponding to the target object, it is specifically used for:

Acquiring second posture information of the driving assistance device;

The spatial plane is determined according to the second posture information.
The device according to claim 1, wherein the processor is specifically used when the processor converts the head-up image into a top-down image according to the relative posture:

Obtaining a projection matrix corresponding to the head-up image according to the relative posture;

The head-up image is converted into a top-down image according to the projection matrix.
The device according to claim 4, wherein the processor is specifically used when acquiring the projection matrix corresponding to the head-up image according to the relative posture:

Determine the target rotation matrix according to the relative attitude;

Acquiring target rotation parameters according to the target rotation matrix;

Obtain the projection matrix according to the relative pose and the target rotation parameter.
The apparatus according to claim 5, wherein the relative posture includes a rotation angle of the shooting device on a pitch axis, a rotation angle on a roll axis, and a rotation angle on a yaw axis; When the relative pose determines the target rotation matrix, it is specifically used to:

Determine the first rotation matrix according to the rotation angle of the shooting device on the pitch axis;

Determine the second rotation matrix according to the rotation angle of the shooting device on the roll axis;

Determine a third rotation matrix according to the rotation angle of the shooting device on the yaw axis;

The target rotation matrix is determined according to the first rotation matrix, the second rotation matrix, and the third rotation matrix.
The device according to claim 5, characterized in that

When the processor obtains the target rotation parameter according to the target rotation matrix, it is specifically used to:

Determine the first column vector in the target rotation matrix as the first rotation parameter;

Determine the second column vector in the target rotation matrix as the second rotation parameter;

The first rotation parameter and the second rotation parameter are determined as target rotation parameters.
The apparatus according to claim 5, wherein the relative posture further includes a translation parameter of the space plane and the shooting device; the processor obtains the target according to the relative posture and the target rotation parameter The projection matrix is specifically used for:

The projection matrix is obtained according to the target rotation parameter, the normalization coefficient, the internal parameter matrix of the shooting device, the space plane, and the translation parameter of the shooting device.
The device according to claim 4, wherein the processor is specifically used when converting the head-up image into a bird's-eye image according to the projection matrix:

For each first pixel in the head-up image, the position information of the first pixel is converted into the position information of the second pixel in the overhead image according to the projection matrix;

The top view image is obtained according to the position information of each second pixel.
The device according to claim 9, characterized in that

When the processor converts the position information of the first pixel point into the position information of the second pixel point in the bird's-eye view image according to the projection matrix, it is specifically used to:

Acquiring an inverse matrix corresponding to the projection matrix, and converting the position information of the first pixel point to the position information of the second pixel point in the bird's-eye view image according to the inverse matrix.
A vehicle equipped with a driving assistance system, characterized in that the vehicle includes at least one photographing device, a processor and a memory, the memory is used to store computer instructions executable by the processor; the photographing device is used for To collect a head-up image containing the target object, and send the head-up image containing the target object to the processor;

The processor is configured to read computer instructions from the memory to implement:

Acquiring a head-up image containing a target object from the shooting device;

Determine the space plane corresponding to the target object;

Determine the relative posture of the space plane and the shooting device;

The head-up image is converted into a top-down image according to the relative posture.
The vehicle according to claim 11, wherein the shooting device is configured to acquire the head-up image in at least one direction of the front, rear, left, or right of the vehicle.
The vehicle according to claim 11, characterized in that

When the processor determines the space plane corresponding to the target object, it is specifically used for:

Acquiring the first posture information of the vehicle;

The space plane is determined according to the first posture information.
The vehicle according to claim 11, wherein the processor is specifically used when the processor converts the head-up image into a top-down image according to the relative posture:

Obtaining a projection matrix corresponding to the head-up image according to the relative posture;

The head-up image is converted into a top-down image according to the projection matrix.
The vehicle according to claim 14, wherein when the processor obtains the projection matrix corresponding to the head-up image according to the relative posture, it is specifically used to:

Determine the target rotation matrix according to the relative attitude;

Acquiring target rotation parameters according to the target rotation matrix;

Obtain the projection matrix according to the relative pose and the target rotation parameter.
The vehicle according to claim 15, wherein the relative posture includes a rotation angle of the shooting device on a pitch axis, a rotation angle on a roll axis, and a rotation angle on a yaw axis; When the relative pose determines the target rotation matrix, it is specifically used to:

Determine the first rotation matrix according to the rotation angle of the shooting device on the pitch axis;

Determine the second rotation matrix according to the rotation angle of the shooting device on the roll axis;

Determine a third rotation matrix according to the rotation angle of the shooting device on the yaw axis;

The target rotation matrix is determined according to the first rotation matrix, the second rotation matrix, and the third rotation matrix.
The vehicle according to claim 15, characterized in that

When the processor obtains the target rotation parameter according to the target rotation matrix, it is specifically used to:

Determine the first column vector in the target rotation matrix as the first rotation parameter;

Determine the second column vector in the target rotation matrix as the second rotation parameter;

The first rotation parameter and the second rotation parameter are determined as target rotation parameters.
The vehicle according to claim 15, wherein the relative posture further includes a translation parameter of the space plane and the shooting device; the processor acquires the relative posture and the target rotation parameter The projection matrix is specifically used for:

The projection matrix is obtained according to the target rotation parameter, the normalization coefficient, the internal parameter matrix of the shooting device, the space plane, and the translation parameter of the shooting device.
The vehicle according to claim 14, wherein the processor is specifically used when converting the head-up image into a top-down image according to the projection matrix:

For each first pixel in the head-up image, the position information of the first pixel is converted into the position information of the second pixel in the overhead image according to the projection matrix;

The top view image is obtained according to the position information of each second pixel.
The vehicle according to claim 19, wherein

When the processor converts the position information of the first pixel point into the position information of the second pixel point in the bird's-eye view image according to the projection matrix, it is specifically used to:

Acquiring an inverse matrix corresponding to the projection matrix, and converting the position information of the first pixel point to the position information of the second pixel point in the bird's-eye view image according to the inverse matrix.
An image processing method characterized by being applied to a driving assistance system, where the driving assistance system includes at least one photographing device, and the method includes:

Acquiring the head-up image containing the target object through the shooting device;

Determine the space plane corresponding to the target object;

Determine the relative posture of the space plane and the shooting device;

The head-up image is converted into a top-down image according to the relative posture.
The method according to claim 21, characterized in that

The driving assistance system is carried on a mobile platform;

The at least one photographing device is provided on the mobile platform and used to acquire the head-up image in at least one direction of the front, rear, left, or right of the mobile platform.
The method according to claim 22, wherein

The determining the space plane corresponding to the target object includes:

Acquiring the first posture information of the mobile platform;

The space plane is determined according to the first posture information.
The method according to claim 21, characterized in that

The driving assistance system is carried in driving assistance equipment;

The at least one shooting device is provided in the driving assistance device, and is configured to acquire the head-up image in at least one direction of front, rear, left, or right of the driving assistance device.
The method according to claim 24, characterized in that

The determining the space plane corresponding to the target object further includes:

Acquiring second posture information of the driving assistance device;

The spatial plane is determined according to the second posture information.
The method according to claim 21, characterized in that

The converting the head-up image into a top-down image according to the relative posture includes:

Obtaining a projection matrix corresponding to the head-up image according to the relative posture;

The head-up image is converted into a top-down image according to the projection matrix.
The method of claim 26, wherein

The obtaining the projection matrix corresponding to the head-up image according to the relative posture includes:

Determine the target rotation matrix according to the relative attitude;

Acquiring target rotation parameters according to the target rotation matrix;

Obtain the projection matrix according to the relative pose and the target rotation parameter.
The method according to claim 27, wherein the relative posture includes a rotation angle of the shooting device on a pitch axis, a rotation angle on a roll axis, and a rotation angle on a yaw axis;

The determining the target rotation matrix according to the relative posture includes:

Determine the first rotation matrix according to the rotation angle of the shooting device on the pitch axis;

Determine the second rotation matrix according to the rotation angle of the shooting device on the roll axis;

Determine a third rotation matrix according to the rotation angle of the shooting device on the yaw axis;

The target rotation matrix is determined according to the first rotation matrix, the second rotation matrix, and the third rotation matrix.
The method according to claim 27, characterized in that

The acquiring the target rotation parameter according to the target rotation matrix includes:

Determine the first column vector in the target rotation matrix as the first rotation parameter;

Determine the second column vector in the target rotation matrix as the second rotation parameter;

The first rotation parameter and the second rotation parameter are determined as target rotation parameters.
The method according to claim 27, characterized in that

The relative posture also includes translation parameters of the space plane and the shooting device;

Obtaining the projection matrix according to the relative pose and the target rotation parameter includes:

The projection matrix is obtained according to the target rotation parameter, the normalization coefficient, the internal parameter matrix of the shooting device, the space plane, and the translation parameter of the shooting device.
The method of claim 26, wherein

The converting the head-up image into a top-down image according to the projection matrix includes:

For each first pixel in the head-up image, the position information of the first pixel is converted into the position information of the second pixel in the overhead image according to the projection matrix;

The top view image is obtained according to the position information of each second pixel.
The method according to claim 31, wherein converting the position information of the first pixel point to the position information of the second pixel point in the bird's-eye view image according to the projection matrix includes:

Acquiring an inverse matrix corresponding to the projection matrix, and converting the position information of the first pixel point to the position information of the second pixel point in the bird's-eye view image according to the inverse matrix.
The method according to claim 21, characterized in that

After converting the head-up image into a top-down image according to the relative posture, the method further includes:

If the target object is a lane line, then the lane line is detected according to the overhead image.
The method according to claim 21, characterized in that

After converting the head-up image into a top-down image according to the relative posture, the method further includes:

If the target object is a lane line, the lane line is positioned according to the overhead image.
A computer-readable storage medium, characterized in that computer instructions are stored on the computer-readable storage medium, and when the computer instructions are executed, the method of claims 21-34 is implemented.