WO2022222121A1

WO2022222121A1 - Panoramic image generation method, vehicle-mounted image processing apparatus, and vehicle

Info

Publication number: WO2022222121A1
Application number: PCT/CN2021/089132
Authority: WO
Inventors: 陈晓丽; 张峻豪; 黄为; 王笑悦
Original assignee: 华为技术有限公司
Priority date: 2021-04-23
Filing date: 2021-04-23
Publication date: 2022-10-27
Also published as: CN113302648A; CN113302648B

Abstract

A panoramic image generation method, a vehicle-mounted image processing apparatus, and a vehicle. The method in the embodiments of the present application comprises: acquiring image information and depth information of an object around a vehicle, wherein the depth information is used for indicating coordinate point information of each point on the object around the vehicle; acquiring an initial model; according to the depth information, adjusting a first coordinate point on the initial model to a second coordinate point, so as to generate a first model; and generating a panoramic image according to the image information and the first model. In the present embodiment, the position of a first coordinate point is adjusted in real time according to depth information of an object around a vehicle; and since the position of the first coordinate point on the initial model is adjusted in real time, that is, the first model is a model obtained according to the actual distance between the object around the vehicle and the vehicle, a splicing ghost and dislocation in a panoramic image are eliminated, so as to obtain a panoramic image which is consistent with the actual environment around the vehicle.

Description

A method for generating a panoramic image, a vehicle-mounted image processing device, and a vehicle

technical field

The present application relates to the technical field of vehicle-mounted surround view, and in particular, to a method for generating panoramic images, a vehicle-mounted image processing device, and a vehicle.

Background technique

Generally speaking, traditional image-based reversing camera systems only install cameras at the rear of the vehicle, and the cameras can only cover a limited area around the rear of the vehicle, and the blind spots around the vehicle and the front of the vehicle increase the hidden danger of safe driving. In order to expand the driver's field of vision and improve driving safety, the vehicle surround view system came into being.

The vehicle surround view system uses cameras installed around the vehicle to reconstruct the vehicle and surrounding scenes, and performs perspective transformation and image stitching on the captured images to generate 3D panoramic images. In the current technology, the vehicle-mounted surround view system first calibrates the installed 4-channel fisheye camera, and after obtaining the internal and external parameters of the camera, a three-dimensional bowl-shaped fixed model (as shown in Figure 1A) is constructed. Mapping the internal and external parameters to obtain pixel coordinates, and then map the corresponding pixels in the fisheye image to the bowl-shaped fixed model to obtain a 3D panoramic image.

The current 3D panoramic image is generated based on a 3D fixed model, which is a simulated embodiment of real objects around the vehicle. When the distance between the 3D fixed model and the virtual vehicle is not equal to the distance between the vehicle and the actual object, there will be stitching ghosting and dislocation problems in the stitching area of the two images, thereby reducing the detection of obstacles around the vehicle. rate or generate detection blind spots, reducing driving safety.

SUMMARY OF THE INVENTION

Embodiments of the present application provide a method for generating a panoramic image, a vehicle-mounted image processing device, and a vehicle, which are used to eliminate stitching ghosts and dislocations in the panoramic image, and obtain a panoramic image consistent with the actual environment around the vehicle.

In a first aspect, an embodiment of the present application provides a method for generating a panoramic image, which is applied to a vehicle-mounted image processing device, and the method includes: the vehicle-mounted image processing device obtains image information and depth information of objects around the vehicle, and the depth information is used to indicate the vehicle. Coordinate point information of each point on the surrounding object; obtain the initial model, for example, the initial model is a bowl-shaped 3D model, or the initial model is a cylindrical 3D model; the vehicle-mounted image processing device converts the first on the initial model according to the depth information. The coordinate point is adjusted to the second coordinate point, and the first model is generated; the vehicle-mounted image processing device adjusts the position of the first coordinate point in real time according to the depth information of the objects around the vehicle. Since the position of the first coordinate point on the initial model is adjusted in real time, the The first model is a model obtained according to the actual distance of objects around the vehicle from the vehicle. The shape of the first model may be an irregular shape, and the shape of the first model varies with the distance between the vehicle and objects around the vehicle. Alternatively, the in-vehicle image processing apparatus acquires a panoramic image based on the image information and the first model. In the embodiment of the present application, the vehicle-mounted image processing device introduces depth information when creating a 3D model, and the vehicle-mounted image processing device adjusts the coordinate points on the initial model according to the depth information to obtain the first model. The in-vehicle image processing device generates a virtual first model according to the objects around the vehicle in the real world, and generates a virtual vehicle model based on the real-world vehicle correspondence, that is, when the distance between the objects around the vehicle and the vehicle changes, the vehicle model and the first The distance of the coordinate points corresponding to the objects on a model will also change, eliminating stitching ghosts and dislocations in the panoramic image, so as to obtain a 3D panoramic image that is consistent with the actual environment around the vehicle.

In an optional implementation manner, the adjusting the first coordinate point on the initial model to the second coordinate point according to the depth information, and generating the first model may include: the vehicle-mounted image processing device firstly adjusts the pixel point and the second coordinate point according to the image information. The depth information corresponding to the pixel points, convert the pixel points in the image information into the first point cloud in the camera coordinate system; then, the vehicle-mounted image processing device converts the first point cloud in the camera coordinate system into the second point cloud in the world coordinate system. point cloud; finally, the vehicle-mounted image processing device adjusts the first coordinate point to the second coordinate point through the coordinate point in the second point cloud to generate the first model, and the second coordinate point is obtained according to the coordinate point in the second point cloud.

In an optional implementation manner, the image information includes images collected by multiple image sensors, and after converting the first point cloud in the camera coordinate system to the second point cloud in the world coordinate system, the method further includes: on-board image processing The device splices multiple images collected by multiple image sensors in multiple second point clouds in the world coordinate system to obtain the target point cloud; the coordinate points on the target point cloud correspond to the actual distance between the objects around the vehicle and the image sensor. distance, the vehicle-mounted image processing apparatus adjusting the first coordinate point to the second coordinate point through the coordinate point in the second point cloud may include: first, the vehicle-mounted image processing apparatus determines a plurality of third coordinates within the neighborhood range of the first coordinate point point, the third coordinate point is the coordinate point on the target point cloud, and then, the vehicle-mounted image processing device determines the second coordinate point according to a plurality of third coordinate points; wherein, the second coordinate point is determined by the neighborhood range of the first coordinate point obtained from multiple third coordinate points within. Finally, the vehicle-mounted image processing device adjusts the first coordinate point to the second coordinate point. In this embodiment, the target point cloud is a point cloud obtained based on the actual distance between the objects around the vehicle and the vehicle, and the vehicle-mounted image processing device determines the first coordinate point neighborhood on the initial model among a large number of scattered points in the target point cloud Multiple third coordinate points within the range, and then determine the second coordinate point according to the multiple third coordinate points, and then adjust the points on the initial model to the second coordinate point. The actual distance to accurately reconstruct the first model.

In an optional implementation manner, splicing multiple second point clouds of multiple images collected by multiple image sensors in the world coordinate system to obtain the target point cloud may include: first, the on-board image processing device performs two The overlapping areas of the first image and the second image collected by the adjacent image sensors are matched to obtain a rotation matrix and a translation matrix for transformation between the point cloud of the first image and the point cloud of the second image; then, the vehicle-mounted image processing device The point cloud of the second image is transformed by using the rotation matrix and the translation matrix, and the transformed point cloud of the second image and the point cloud of the first image are spliced. In this embodiment, since the poses of the two adjacent cameras are different, the images collected by the two image sensors will have slight differences in angle and orientation, and the overlapping area of the images collected by the two image sensors (the same scene ) to match, so that the difference in angle and orientation of the images collected by the two image sensors can be found, that is, the difference can be balanced by rotation and translation, and then the point clouds of the images collected by multiple image sensors can be spliced to obtain a whole The target point cloud of the slice is adjusted by adjusting the first coordinate point on the initial model through the points on the target point cloud, so that the 3D model can be accurately reconstructed.

In an optional implementation manner, after the first model is generated, the method further includes: the vehicle-mounted image processing device performs interpolation and smoothing processing on the first model to obtain the second model, and further, the vehicle-mounted image processing device is based on the image information. Perform texture mapping on the second model to generate a panoramic image. In this embodiment, the second model after interpolation processing is a 3D model with a smooth surface, and the vehicle-mounted image processing device performs texture mapping on the second model with a smooth surface, thereby improving the rendering effect of the first model.

In an optional implementation manner, in the real world, the objects around the vehicle include a first object and a second object, when the distance between the vehicle and the first object is the first distance, and the distance between the vehicle and the second object is the first object When the distance is two, the position of the vehicle is mapped to the first position in the panoramic image, the position of the first object is mapped to the second position of the panoramic image, and the position of the second object is mapped to the third position in the panoramic image. When the distance is greater than the second distance, the method further includes: the vehicle-mounted image processing device displays a panoramic image, and in the panoramic image, the distance between the first position and the second position is greater than the distance between the first position and the third position. In this embodiment, when the distance between the objects around the vehicle and the vehicle changes, the distance between the vehicle model and the coordinate points corresponding to the objects on the first model also changes, so that a panoramic image consistent with the actual environment around the vehicle is obtained, Eliminate stitching ghosts and dislocations, and improve the detection accuracy and driver experience in the stitched area.

In a second aspect, an embodiment of the present application provides a vehicle-mounted surround view device, including: an acquisition module for acquiring image information and depth information of objects around the vehicle, where the depth information is used to indicate coordinate point information of each point on the objects around the vehicle; The processing module is used for acquiring the initial model; adjusting the first coordinate point on the initial model to the second coordinate point according to the depth information to generate the first model; and acquiring the panoramic image based on the image information and the first model.

In an optional implementation manner, the processing module is further specifically configured to: convert the pixels in the image information into the first point cloud in the camera coordinate system according to the pixels in the image information and the depth information corresponding to the pixels; Convert the first point cloud in the camera coordinate system to the second point cloud in world coordinates; adjust the first coordinate point to the second coordinate point through the coordinate points in the second point cloud to generate the first model, the second coordinate point is obtained from the coordinate points in the second point cloud.

In an optional implementation manner, the image information includes images collected by multiple image sensors, and the processing module is further specifically configured to: process multiple images collected by multiple image sensors on multiple second point clouds in the world coordinate system splicing to obtain the target point cloud; determining multiple third coordinate points within the neighborhood of the first coordinate point, the third coordinate point being the coordinate point on the target point cloud; determining the second coordinate point according to the multiple third coordinate points ; Adjust the first coordinate point to the second coordinate point.

In an optional implementation manner, the processing module is further specifically configured to: match the overlapping area of the first image and the second image collected by two adjacent image sensors to obtain the point cloud of the first image and the point cloud of the second image. A rotation matrix and a translation matrix for transforming between point clouds; using the rotation matrix and the translation matrix to transform the point cloud of the second image, and stitching the transformed point cloud of the second image and the point cloud of the first image.

In an optional implementation manner, the processing module is further specifically configured to: perform interpolation and smoothing processing on the first model to obtain a second model; and perform texture mapping on the second model according to image information to generate a panoramic image.

In an optional implementation manner, the objects around the vehicle include a first object and a second object. When the distance between the vehicle and the first object is the first distance, and the distance between the vehicle and the second object is the second distance, the vehicle The position of the object is mapped to the first position in the panoramic image, the position of the first object is mapped to the second position of the panoramic image, and the position of the second object is mapped to the third position in the panoramic image. When the first distance is greater than the second distance , the device further includes a display module; the display module is further used to display a panoramic image, in which the distance between the first position and the second position is greater than the distance between the first position and the third position.

In a third aspect, an embodiment of the present application provides an in-vehicle image processing device, including a processor, the processor is coupled with a memory, and the memory is used to store programs or instructions, when the programs or instructions are executed by the processor, the in-vehicle image processing device is The method as described in the first aspect above is performed.

In a fourth aspect, an embodiment of the present application provides a vehicle-mounted surround view system, including a sensor, a vehicle-mounted display, and the vehicle-mounted image processing device described in the third aspect above, wherein the sensor and the vehicle-mounted display are both connected to the vehicle-mounted image processor, wherein , the sensor is used to collect image information and depth information, and the vehicle-mounted display is used to display panoramic images.

In a fifth aspect, an embodiment of the present application provides a vehicle, including the vehicle-mounted surround view system as described in the fourth aspect.

In a sixth aspect, an embodiment of the present application provides a computer program product, the computer program product includes computer program code, and when the computer program code is executed by a computer, enables the computer to implement any one of the above-mentioned first aspects. method.

In a seventh aspect, an embodiment of the present application provides a computer-readable storage medium for storing a computer program or instruction, and when the computer program or instruction is executed, the computer executes the method described in any one of the above-mentioned first aspect.

In an eighth aspect, an embodiment of the present application provides a chip, including a processor and a communication interface, where the processor is configured to read an instruction to execute the method described in any one of the foregoing first aspects.

Description of drawings

Fig. 1A is a three-dimensional schematic diagram of a 3D model;

1B is a schematic side view of a 3D model;

2A is a schematic diagram of ghosting in a panoramic image in a conventional method;

2B is a schematic diagram of a stitching dislocation in a panoramic image in a traditional method;

3 is a schematic structural diagram of a vehicle-mounted surround view system in an embodiment of the application;

4 is a schematic diagram of a world coordinate system, a camera coordinate system, an image coordinate system and a pixel coordinate system in an embodiment of the application;

FIG. 5 is a schematic flowchart of steps of a method for generating a panoramic image according to an embodiment of the present application;

6 is a schematic diagram of the visualization effect of converting an image with depth information into a point cloud in an embodiment of the present application;

7 is a schematic diagram of matching an overlapping area in a first image and an overlapping area in a second image in an embodiment of the present application;

8 is a schematic diagram of splicing images collected by adjacent camera sensors in an embodiment of the present application;

FIG. 9A is a three-dimensional schematic diagram of a third coordinate point within a neighborhood range of a first coordinate point on an initial model in an embodiment of the present application;

9B is a schematic top view of a third coordinate point within the neighborhood of the first coordinate point on the initial model in the embodiment of the present application;

9C and 9D are schematic top views of the first model obtained by adjusting the first coordinate point on the initial model in the embodiment of the present application;

10 is a schematic diagram of a scene of a vehicle and surrounding objects in the real world and a vehicle and surrounding objects in a panoramic image according to an embodiment of the application;

11 is a schematic diagram of performing interpolation processing on a first model in an embodiment of the present application;

12 is a schematic structural diagram of an embodiment of an in-vehicle image processing apparatus in an embodiment of the application;

FIG. 13 is a schematic structural diagram of another embodiment of an in-vehicle image processing apparatus in an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be described below with reference to the accompanying drawings in the embodiments of the present application. The terms "first", "second" and the like in the description and claims of the present application and the drawings are used to distinguish objects, and are not necessarily used to describe a specific order or sequence. It is to be understood that the terms so used are interchangeable under appropriate circumstances.

The vehicle-mounted surround view system can acquire images captured by multiple cameras installed on the vehicle body, and perform perspective transformation and image stitching on the acquired images to generate panoramic images. Exemplarily, the panoramic image is a 360-degree panoramic image around the vehicle, or the panoramic image is a 720-degree panoramic image. Usually, cameras are installed around the car body, and the image information of objects around the car body is collected by the cameras installed in the front, rear, left and right directions of the car body, and the images collected by each two adjacent cameras are spliced and mapped to On the 3D model that has been constructed with a fixed size, the 3D model is a simulated embodiment of the real objects around the vehicle. Please refer to FIG. 1B , XOZ represents the plane coordinate system, Z=0 represents the ground plane, and R represents the radius from the Z axis to the wall of the 3D model. Since the distance between the vehicle and surrounding objects changes during the driving process of the vehicle, and the size of the 3D model is fixed, when the actual distance between the vehicle and surrounding objects is less than R, two adjacent cameras collect The obtained images will overlap, resulting in ghosting in the image splicing area (as shown in Figure 2A). This leads to the dislocation of the images in the stitched area (as shown in Figure 2B). It should be understood that a camera may also be installed on the vehicle body for capturing images of the vehicle bottom. For example, a camera installed on the chassis, or a camera installed around the vehicle, the camera's angle of view can capture the position where the vehicle bottom will pass.

Based on the above problem, an embodiment of the present application provides a method for generating a panoramic image, and the method is applied to a vehicle-mounted surround view system. 3 , the vehicle surround view system includes a sensor 301 , a vehicle image processing device 302 and a vehicle display 303 , wherein the sensor 301 is connected to the vehicle image processing device 302 , and the vehicle image processing device 302 is connected to the vehicle display 303 . The sensor 301 is used to collect image information and depth information of objects around the vehicle. The in-vehicle image processing device 302 first obtains the depth information of objects around the vehicle, and creates an initial model, and then adjusts the position of the first coordinate point on the initial model according to the depth information to generate the first model; finally, the in-vehicle image processing device 302 The image information performs texture mapping on the first model to generate a panoramic image. The vehicle-mounted image processing device 302 outputs the panoramic image to the vehicle-mounted display 303, and the vehicle-mounted display 303 is used for displaying the panoramic image. In the embodiment of the present application, when creating a 3D model, depth information is introduced, the first coordinate point on the initial model is adjusted according to the depth information, and the first coordinate point is adjusted to the second coordinate point to obtain the first model, so that the surrounding area of the vehicle is The distance between the object (corresponding to the first model) and the vehicle (corresponding to the vehicle model) is basically equal to the distance between the vehicle model and the first model. Since the first model is accurately reconstructed according to the depth information, the stitching ghosts and images in the panoramic image are eliminated. dislocation, so as to obtain a 3D panoramic image that is consistent with the actual environment around the vehicle, and improve the detection accuracy of the patchwork area and the driver experience effect.

For a better understanding of the present application, the words and expressions involved in the present application are exemplified.

The depth information can be used to indicate the three-dimensional coordinate information of each point on the detected object. Depth information is usually also called depth. For the convenience of calculation, depth in the field of machine vision refers to the distance of each point in space relative to the camera sensor. In the embodiment of the present application, the depth information of objects around the vehicle refers to the distance between the three-dimensional coordinate point on the object around the vehicle and the camera sensor, that is, the distance between the three-dimensional coordinate point on the object around the vehicle and the position of the sensor on the vehicle body.

Please refer to Figure 4, which shows the world coordinate system (O _w -X _w Y _w Z _w ), the camera coordinate system (O _c -X _c Y _c Z _c ), the image coordinate system (o-xy ) and the pixel coordinate system (uv). Among them, point P is a point in the world coordinate system, that is, a point in the real environment. Point p is the imaging point of point P, the coordinates of point p in the image coordinate system are (x, y), and the coordinates in the pixel coordinate system are (u, v). The origin (o) of the image coordinate system is located on the Z-axis of the camera coordinate system, and the distance between the origin (o) of the image coordinate system and the origin (O _c ) of the camera coordinate system is f, where f is the camera focal length. The four coordinate systems shown in FIG. 4 will be described below.

The world coordinate system, also known as the measurement coordinate system, is a three-dimensional coordinate system. For example, the three-dimensional coordinate system can be a three-dimensional orthogonal coordinate system, a cylindrical coordinate system, a spherical coordinate system, and the like. In this embodiment of the present application, the world coordinate system may use a three-dimensional orthogonal coordinate system (X _w , Y _w , Z _w ). The spatial position of the camera, the vehicle, and objects around the vehicle can be described in the world coordinate system. The position of the world coordinate system is determined by itself according to the actual situation. In this application, for the convenience of processing, the world coordinate system is selected to be centered on the vehicle (the vehicle position is located at the origin position O _w of the world coordinate system), the Z _w axis is perpendicular to the ground, the X _w axis represents the direction of the vehicle, and the coordinate system The marker changes with the movement of the vehicle. The unit of the world coordinate system can be meters (m).

The camera coordinate system is a three-dimensional rectangular coordinate system (X _c , Y _c , Z _c ). The origin of the camera coordinate system is the optical center of the lens, the X _c and Y _c axes are respectively parallel to both sides of the image plane, and the Z _c axis is the optical axis of the lens, which is perpendicular to the image plane. The unit of the camera coordinate system can be meters (m).

Pixel coordinate system, transform the coordinates of the camera coordinate system once to obtain the commonly used plane pixel coordinates (z=1 at this time). The pixel coordinate system is in pixels, and the coordinate origin is at the upper left corner of the camera coordinate system.

The relationship between the image coordinate system and the pixel coordinate system may be: the origin of the image coordinate system is the midpoint of the pixel coordinate system. The units of the image coordinate system may be millimeters (mm).

The transformation between the world coordinate system and the camera coordinate system is a rigid transformation, that is, only the spatial position (translation) and orientation (rotation) of the object are changed, but the shape of the object is not changed. This transformation can be represented by rotation and translation. There is no rotation between the image coordinate system and the pixel coordinate system, but the coordinate origin of the image coordinate system and the pixel coordinate system are different.

Homogeneous coordinates, which represent an n-dimensional vector with a (n+1)-dimensional vector, refer to a coordinate system used in projection geometry, just like Cartesian coordinates used in Euclidean geometry. For example, the homogeneous coordinates of a two-dimensional point (x, y) are represented as (hx, hy, h). The homogeneous representation of a vector is not unique, and different values of h in homogeneous coordinates represent the same point. For example, the homogeneous coordinates (8, 4, 2) and (4, 2, 1) represent the two-dimensional point (4, 2). The purpose of introducing homogeneous coordinates is mainly to combine multiplication and addition in matrix operations. Homogeneous coordinates provide a method for transforming a set of points in two-, three-, and even higher-dimensional spaces from one coordinate system to another using matrix operations. The purpose of introducing homogeneous coordinates is to facilitate computer graphics to perform affine geometric transformations. It can be understood that the introduction of homogeneous coordinates can use a matrix to describe rotation and translation at the same time, and use matrix multiplication to express the rotation and translation of an object.

The internal parameters of the camera may be parameters related to the characteristics of the camera itself, such as the focal length of the camera, the pixel size, and the like.

The external parameters of the camera can be parameters in the world coordinate system, such as the position of the camera, the direction of rotation, etc.

A point cloud can be a collection of massive points that express the spatial distribution of the target and the characteristics of the target surface under the same spatial reference system. After obtaining the spatial coordinates of each sampling point on the surface of the object, a collection of points is obtained, which is called "point cloud". ” (point cloud).

Texture can be pixel feature information on a two-dimensional image. When a texture is mapped to the surface of an object in a specific way, it is also called a texture map.

Referring to FIG. 5 , an embodiment of the present application provides a method for generating a panoramic image. The execution body of the method is the vehicle-mounted surround view system in FIG. 3 , or the execution body of the method is the vehicle-mounted image in FIG. 3 . The processing device, or the execution body of the method is a processor or chip in the vehicle image processing device. In the embodiments of the present application, the execution body of the method is described by taking the vehicle surround view system as an example.

Step 501: The vehicle-mounted surround view system acquires image information and depth information of objects around the vehicle.

Multiple cameras are arranged around the vehicle, for example, the camera is a wide-angle camera (such as a fisheye camera), and at least one fisheye camera is arranged in each of the four directions of the front, rear, left, and right directions of the vehicle. The fisheye camera is used to collect the image information around the vehicle in real time. Because the fisheye camera is a special ultra-wide-angle lens, its structure is imitated by the fish eye imaging, which can independently realize large-angle shooting, and the angle of view of the fisheye camera can even reach 180° , capable of monitoring objects in a wide range of scenes. A fisheye camera is set in each direction around the vehicle, and four fisheye cameras can collect panoramic images around the vehicle, thereby saving the number of cameras and reducing costs. Of course, in the embodiment of the present application, the camera is not limited to a wide-angle camera, and the camera can also be an ordinary camera. The viewing angle of the camera is increased by increasing the number of ordinary cameras. For example, two or three cameras are set in each direction of the vehicle circumference. A camera is used to collect panoramic images around the vehicle by using ordinary cameras. Although the number of cameras is increased, the image information collected by ordinary cameras is not deformed, and the collected images have better effect. In the embodiment of the present application, the camera is taken as an example of a fisheye camera, and the number of fisheye cameras is taken as an example, that is, one fisheye camera is provided in each direction of the vehicle, front, rear, left, right, and right.

The in-vehicle surround view system obtains the depth information of objects around the vehicle, including the following two implementations. In a first implementation manner, the sensor 301 is an image sensor in a camera. The image sensor can collect the image information of the objects around the vehicle, and transmit the image information to the vehicle-mounted image processing device, and the vehicle-mounted image processing device can obtain the depth information of the objects around the vehicle according to the image information. Exemplarily, the vehicle-mounted image processing device performs depth estimation on the image information collected by the fisheye camera to obtain depth information, and the methods for obtaining the depth information by the vehicle-mounted image processing device include but are not limited to methods such as monocular depth estimation and binocular depth estimation. For example, the monocular depth estimation method may use the image data of a single view as the input of the trained depth model, and use the depth model to output the depth corresponding to each pixel in the image. That is, the monocular estimation method based on deep learning reflects the depth relationship according to the pixel value relationship, and maps the image information into a depth map. The method of binocular depth estimation is to use a binocular camera to shoot left and right viewpoint images of the same scene, and use a stereo matching algorithm to obtain a disparity map, and then obtain a depth map. A depth map, also known as a distance image, refers to an image in which the distance (depth) from the image sensor (or camera) to each point in the scene is used as a pixel value. Colder means higher depth values. In the first implementation, only the image sensor can be used to obtain both the scene image around the vehicle and the depth information of the objects around the vehicle, without adding other relevant components for obtaining depth information, thus saving costs.

In the second implementation manner, the sensor 301 further includes a depth sensor, and the depth sensor is used to collect depth information of objects around the vehicle. Depth sensors include, but are not limited to, millimeter-wave radar and lidar. For example, the method of obtaining the depth information of objects around the vehicle through lidar is that the lidar emits laser light into space at certain time intervals, and records the signal of each scanning point from the radar to the object in the scene under test, and then reflects back through the object. The interval time to the lidar, the distance between the surface of the object and the radar is calculated according to the interval time, that is, the depth information of the objects around the vehicle is obtained. For another example, the method of obtaining depth information through millimeter-wave radar is that the millimeter-wave radar sends out a high-frequency continuous signal, the signal is reflected after encountering objects around the vehicle, and the receiver receives the reflected signal reflected by the object. The radar has a certain time interval t from transmitting the signal to receiving the reflected signal. t=2d/c, where d is the distance from the radar to the object, and c is the speed of light. In this embodiment of the present application, the above two methods for acquiring depth information are only exemplary descriptions, and do not limit specific methods for acquiring depth information. In the second implementation manner, the depth sensor collects the depth information of objects around the vehicle and transmits the depth information to the vehicle-mounted image processing device, which can save the steps of depth estimation according to the image information, thereby saving the calculation of the vehicle-mounted image processing device. force. In the embodiment of the present application, the method for acquiring depth information in the above-mentioned first implementation manner is used as an example for description.

Step 502, the vehicle-mounted surround view system acquires an initial model.

Exemplarily, the vehicle-mounted image processing device generates an initial model, and the initial model is a bowl-shaped 3D model, or the initial model is a cylindrical 3D model, and the like. Alternatively, the initial model can be a smooth solid model, or the initial model can be a scatter model.

It should be noted that there is no timing limitation between the

above steps

501 and 502, and step 502 may be executed before step 501, or step 502 may be executed after step 501, and steps 502 and 501 may also be executed synchronously.

Step 503: The vehicle-mounted surround view system adjusts the first coordinate point on the initial model to the second coordinate point according to the depth information to obtain the first model.

First, please refer to FIG. 6 , which is a schematic diagram of a point cloud. Based on the acquired depth information, the vehicle-mounted image processing device converts the pixel coordinates in the image information into the first point cloud coordinates in the camera coordinate system according to the pixel coordinates in the image information and the depth information corresponding to the pixel coordinates. The image information of the vehicle surrounding scene obtained in the above step 501 includes a plurality of pixel points, for example, a first pixel point (eg, _ui , v _i ) is included in the plurality of pixel points. The depth information of the scene around the vehicle obtained in the above step 501 . The vehicle-mounted image processing device converts the point under the pixel coordinate system to the point cloud coordinate under the camera coordinate system according to the first pixel point and the depth information corresponding to the first pixel point, and the point cloud coordinate (x _i , y under the camera coordinate system) _i , z _i ) are represented by the following formula (1).

Among them, u ₀ , v ₀ represent the coordinates of the optical center in the image coordinate system, f _x represents the focal length in the horizontal direction, and f _y represents the focal length in the vertical direction.

Then, the vehicle-mounted image processing device converts the first point cloud coordinates (x _i , y _i , z _i ) in the camera coordinate system to the second point cloud coordinates (X _i , Y _i , Z _i ) in the world coordinate system, It should be noted that, in this embodiment, in order to distinguish the point cloud coordinates in the camera coordinate system from the point cloud coordinates in the world coordinate system, the point cloud coordinates in the camera coordinate system are called "first point cloud coordinates", and the world coordinate system is called "first point cloud coordinates". The point cloud coordinates in the coordinate system are called "second point cloud coordinates". Specifically, for converting the coordinates of the first point cloud into the coordinates of the second point cloud, please refer to the following formulas (2) and (3).

The formula (2) can be expressed as the following formula (3) by homogeneous coordinates.

Among them, in the above formulas (2) and (3), r _j represents the rotation matrix of the coordinate system transformation corresponding to the jth camera, for example, the value of j is 1, 2, 3, 4, such as the first camera is the front image sensor, the second camera is the left image sensor, the third camera is the rear image sensor, and the fourth camera is the right image sensor; t _j represents the translation matrix of the coordinate system transformation corresponding to the jth camera; (* ) ^-1 means matrix inversion;

Indicates the external parameters of the jth camera; (X _i , Y _i , Z _i ) are the coordinates of the three-dimensional coordinate point corresponding to the pixel point (u _i , v _i ) in the world coordinate system.

Then, the vehicle-mounted image processing device splices (or connects) the image information collected by the four image sensors in the four point clouds in the world coordinate system to obtain a complete whole panoramic point cloud (ie, the target point cloud). Exemplarily, please refer to the following steps (a) and (b) for description.

(a), the vehicle-mounted image processing device matches the point clouds of the overlapping regions of the images (the first image and the second image) collected by two adjacent image sensors, so as to obtain a rotation matrix for transforming the first image and the second image (denoted by "R") and translation matrix (denoted by "T"). It should be understood that, referring to FIG. 7 , the first image sensor and the second image sensor are adjacent image sensors (such as the front camera and the left camera, the left camera and the rear camera, the rear camera and the right camera, and the right camera and the front camera. ) will capture images with overlapping areas. The first image sensor is used for capturing the first image, and the second image sensor is used for capturing the second image. The point cloud data of the overlapping area in the first image is marked as "P", and the point cloud data of the overlapping area in the second image is marked as "Q". A set of rotation matrices R and translation matrices T are found using the objective function of the following equation (4). It should be understood that due to the different poses of the two cameras, the images collected by the two image sensors will have slight differences in angle and orientation, and the overlapping area (same scene) of the images collected by the two image sensors can be found. This difference, thereby eliminating calculation errors.

Among them, f(q _i ) represents the three-dimensional coordinates of the i-th point in the point cloud data Q, f( _pi ) represents the three-dimensional coordinates of the i-th point in the point cloud data P; R _h represents the rotation matrix of the point cloud data P for transformation , T _h represents the translation matrix transformed by the point cloud data P; h=j-1; j is the number of image sensors. For example, the number of image sensors is 4, and the value of h is 1, 2, 3. Four image sensors are adjacent to each other, and a total of 3 sets of R and T need to be found. The 3 groups R and T are "R ₁ and T ₁ ", "R ₂ and T ₂ " and "R ₃ and T ₃ ", respectively. It can be seen from the above formula (4) that R×f( _pi )+T is f( _pi ) which is the three-dimensional coordinates of the point cloud of the image B after rotation and translation, and a set of R is found through the above formula (4). _h and _Th , so that the objective function has a minimum value, that is, the point cloud data P has a minimum value between the point cloud data P and the point cloud data Q after rotation and translation, so that the point cloud data Q and the point cloud data P in the two overlapping areas match .

It should be noted that, in order to distinguish the point cloud of the original image from the point cloud transformed by rotation and translation, in this embodiment, the point cloud of the original image is called "point cloud A", and the transformed point cloud is called "point cloud A". is "point cloud B". In order to distinguish the images collected by different image sensors, the image collected by the front image sensor is called "image A", the image collected by the left image sensor is called "image B", the image collected by the rear image sensor is called "image C", and the image collected by the rear image sensor is called "image C". The image captured by the image sensor is called "Image D".

8 , the front image sensor collects image A, the left image sensor collects image B, and the overlapping area of image A and image B is the image of the object in front of the left of the vehicle. In the above formula (4), f(q _i ) represents the point cloud data of the overlapping area in the point cloud A of the image A, and f( _pi ) represents the point cloud data of the overlapping area in the point cloud A of the image B. According to the above Formula (4) yields R ₂ and T ₂ .

The vehicle-mounted image processing device needs to match the overlapping area in image A with the overlapping area in image B. When the above formula (4) has a minimum value, the overlapping area in image A matches the overlapping area in image B. The purpose of matching the overlapping area in image A with the overlapping area in image B is to find a set of R ₁ and T ₁ , so that the point cloud of image A or the point cloud of image B can be processed through R ₁ and T ₁ . Transform, and then stitch the point clouds of the images collected by the two adjacent image sensors.

(b) After obtaining R _h and T _h by the above formula (4), the vehicle-mounted image processing device splices the point clouds of the images collected by each two adjacent image sensors to obtain a complete whole panoramic point cloud. Exemplarily, the vehicle-mounted image processing apparatus uses the above formula (4) to match the overlapping area of point cloud A of image A with the overlapping area of point cloud A of image B, and find a set to obtain R ₁ and T ₁ . After the vehicle-mounted image processing device obtains R ₁ and T ₁ , the vehicle-mounted image processing device may first fix the point cloud A of the image A collected by the front image sensor. Then, using R ₁ and T ₁ to rotate and translate the point cloud A of the image B, the whole point cloud data of the point cloud A of the image B collected by the left image sensor in the world coordinate system can be rotated and translated After that, the point cloud B of the image B is obtained, and the point cloud B of the image B is spliced with the point cloud A of the image A.

In the same way, the rear image sensor collects the image C, and the vehicle-mounted image processing device finds a set of R ₂ according to the overlapping area in the point cloud B of the image B and the point cloud of the overlapping area in the point cloud A of the image C through the above formula (4). and T ₂ . Among them, f(q _i ) represents the point cloud data of the overlapping area in the point cloud B of the image B, and f( _pi ) represents the point cloud data of the overlapping area in the point cloud A of the image C. The vehicle-mounted image processing device uses the above formula (4) to match the overlapping area of point cloud B of image B with the overlapping area of point cloud A of image C, and find a set to obtain R ₂ and T ₂ . Then the vehicle-mounted image processing device fixes the point cloud B of the second image, and uses R ₂ and T ₂ to perform rotation and translation transformation on the point cloud A of the image C, so that the whole image C collected by the rear image sensor can be adjusted in the world coordinate system. After the point cloud is rotated and translated, point cloud B of image C is obtained. Further, the vehicle-mounted image processing device splices the point cloud B of the image C and the point cloud B of the image B. Similarly, the right image sensor collects the image D, and the vehicle-mounted image processing device finds a set of R ₃ and T ₃ according to the overlapping area in the point cloud B of the image C and the overlapping area in the point cloud A of the image D through the above formula (4). . Among them, f(q _i ) represents the point cloud data of the overlapping area in the point cloud B of the image C, and f( _pi ) represents the point cloud data of the overlapping area in the point cloud A of the image D. The in-vehicle image processing apparatus obtains R ₃ and T ₃ by using the above formula (4). Then, the vehicle-mounted image processing device fixes the point cloud B of the image C, and uses R ₃ and T ₃ to rotate and translate the point cloud A of the image D, so that the entire image D collected by the right camera in the world coordinate system can be transformed. After the point cloud data is rotated and translated, the point cloud B of the image D is obtained. Further, the vehicle-mounted image processing device splices the point cloud B of the image D and the point cloud B of the image C. The vehicle-mounted image processing device then splices the point cloud B of the image D and the point cloud A of the image A to obtain the target point cloud. The target point cloud is the entire point cloud after the vehicle-mounted image processing device splices point cloud A of image A, point cloud B of image B, point cloud C of image C, and point cloud B of image D. In this embodiment, the overlapping area (same scene) of the images collected by the two image sensors is matched, so that the difference in angle and orientation of the images collected by the two image sensors can be found, that is, the rotation and translation can balance this Therefore, a complete image can be obtained by stitching the point clouds of the images collected by multiple image sensors, and the 3D model can be reconstructed by adjusting the first coordinate point on the initial model through the points on the target point cloud.

Finally, the vehicle-mounted surround view system adjusts the first coordinate point on the initial model, adjusts the first coordinate point to the second coordinate point, and generates a first model. Wherein, the second coordinate point is obtained from a plurality of third coordinate points within the neighborhood of the first coordinate point, and the third coordinate point is a coordinate point on the target point cloud. Referring to Figure 9A, in the world coordinate system, the Z axis is perpendicular to the ground, then for a first coordinate point (X _a , Y _a , Z _a ) on the initial model in the world coordinate system, Z _a represents the distance from the ground Height, the vehicle-mounted image processing device determines a plurality of third coordinate points within the neighborhood of the first coordinate point among a large number of scattered points in the target point cloud, for example, the third coordinate point is a three-dimensional coordinate point with a height of Z _a , the set of three-dimensional coordinate points whose height is Z _a is M. For example, M includes three-dimensional coordinate points (X ₁ , Y ₁ , Z ₁ ), (X ₂ , Y ₂ , Z ₂ ) and (X ₃ , Y ₃ , Z ₃ ), etc., where Z ₁ , Z The values of ₂ and Z ₃ are both equal to Z _a , and the vehicle-mounted image processing device is based on the X values (such as X ₁ , X ₂ and X ₃ ) and Y values (such as Y ₁ , Y ₂ and Y ₃ of each three-dimensional point in the M set) ) to adjust the values of X _a and Y _a . It should be understood that the X value and the Y value of each coordinate point in M can represent the actual distance between the vehicle and the scene around the vehicle. And the three-dimensional coordinate points in the M set are coordinate points within the neighborhood range of the first coordinate point (X _a , Y _a , Z _a ), please refer to FIG. 9B for understanding, the “neighborhood range” of the first coordinate point refers to The intersection of "First Range" and "Second Range". The "first range" and the "second range" are exemplified. The "first range" can be understood as the lateral range, that is, the range between two rays passing through the center point with the center point of the initial model as the center of the circle. For example, the ray a from the center point to the first coordinate point, the ray a rotates α degrees (°) counterclockwise around the center point to obtain the ray b, and the ray a rotates α° clockwise around the center point to obtain the ray c, the ray b and the ray The range between c is the first range. For example, α is an angle less than or equal to 10°, and the size of α can be set according to actual needs. The "second range" can be understood as a longitudinal range. For example, the center point of the initial model is taken as the center of the circle, and the distance between the first coordinate point and the center of the circle is R1, then the second range is: the radius is R2 and the radius is R3. The range covered by a ring composed of concentric circles. Among them, R2 is less than R1 and R3 is greater than R1, R1-R2=g1, R3-R1=g2, g1 and g2 can be equal or unequal, and g1 and g2 can be set according to empirical or experimental values. Further, the vehicle-mounted image processing device determines the second coordinate point according to the plurality of third coordinate points. Exemplarily, the vehicle-mounted image processing apparatus adjusts X _a according to the X value of each three-dimensional coordinate point in the set M, and adjusts Y _a according to the Y value of each three-dimensional coordinate point in the set M. The point to be adjusted is the first coordinate point (X _a , Y _a , Z _a ), the adjusted second coordinate point is (X _a ′, Y _a ′, Z _a ), the adjusted X _a ′ and Y _a ' is represented by the following formulas (5) and (6).

Among them, X _a ′ is the adjusted X value, X _b belongs to the points in the neighborhood of X _a , n is the number of points in the neighborhood, δ(*) represents the neighborhood, δ(X _a ) represents the neighborhood of X _a , That is, the set of X values in the three-dimensional coordinate points in the M set, for example, δ(X _a ) includes X ₁ , X ₂ and X ₃ .

Among them, Y _a ′ is the adjusted Y value, Y _b belongs to the points in the neighborhood of Y _a , n is the number of points in the neighborhood, δ(*) represents the neighborhood, δ(Y _a ) represents the neighborhood of Y _a , That is, the set of Y values in the three-dimensional coordinate points in the M set, for example, δ(Y _a ) includes Y ₁ , Y ₂ and Y ₃ .

Please refer to FIGS. 9C and 9D , the three-dimensional coordinate point on the initial model is (X _a , Y _a , Z _a ), and the position of the three-dimensional coordinate point is adjusted. According to the above formulas (5) and (6), the (X _a , Y _a , Z _a ) are adjusted to (X _a ′, Y _a ′, Z _a ′) to obtain the first model. It should be understood that, in this embodiment, only one three-dimensional point (X _a , Y _a , Z _a ) on the initial model is adjusted as an example for description, and the adjustment of other first coordinate points can be obtained by the same method as above. This will not be described one by one, and in practical applications, the number of the first coordinate points is not limited, the position of the first coordinate point is not limited, and the position of the adjusted second coordinate point is not limited. For example, please refer to As shown in Figure 9C, the adjusted second coordinate points (X _a ', Y _a ', Z _a ') may be located "outside" the original model. Referring to Figure 9D, the adjusted second coordinate points (X _a ', Y _a ', Z _a ') may be located "inside" the original model. The vehicle-mounted image processing device adjusts the position of the first coordinate point in real time according to the depth information of the objects around the vehicle. Since the position of the first coordinate point on the initial model is adjusted in real time, the shape of the initial model will change, that is, the first model is based on A model derived from the actual distance of objects around the vehicle from the vehicle. The shape of the first model may be an irregular shape, and the shape of the first model varies with the distance of the vehicle from objects around the vehicle. In this embodiment, the target point cloud is a point cloud obtained based on the actual distance between the objects around the vehicle and the vehicle, and the vehicle-mounted image processing device determines the first coordinate point neighborhood on the initial model among a large number of scattered points in the target point cloud Multiple third coordinate points within the range, and then determine the second coordinate point according to the multiple third coordinate points, and then adjust the points on the initial model to the second coordinate point. The actual distance to accurately reconstruct the first model.

Step 504: The vehicle-mounted surround view system acquires a panoramic image based on the image information and the first model.

The vehicle-mounted surround view system performs texture mapping on the first model according to the image information to generate a 3D panoramic image (or called a "panoramic image").

In step 501 , four image sensors collect images in four directions around the vehicle, and the image sensors transmit the images in the four directions to the vehicle-mounted image processing device, which may use texture mapping to map. Exemplarily, the in-vehicle image processing device obtains the internal and external parameters of the camera through pre-calibration, performs external and internal parameter mapping on the three-dimensional coordinates of the optimized first model, and obtains two-dimensional pixel coordinates, and then obtains the two-dimensional pixel coordinates from the image (or also from the image collected by the fisheye camera). (called "texture image") to obtain the corresponding texture pixels, and correspond the coordinate points in the image to the surface of the first model, that is, which point on the first model corresponds to each pixel on the texture image for rendering and coloring , so that the entire texture image is covered on the first model, thereby obtaining a 3D panoramic image.

Step 505 , the vehicle-mounted surround view system outputs a panoramic image.

The vehicle-mounted image processing device outputs the 3D panoramic image to the vehicle-mounted display, and the vehicle-mounted display displays the 3D panoramic image.

In the embodiment of the present application, the vehicle-mounted image processing device introduces depth information when creating a 3D model, and the vehicle-mounted image processing device adjusts the coordinate points on the initial model according to the depth information to obtain the first model. The in-vehicle image processing device generates a virtual first model corresponding to the objects around the vehicle in the real world, and generates a virtual vehicle model based on the corresponding vehicle in the real world. For example, please refer to Figure 10. In the real world, the objects around the vehicle include a first object and a second object, the distance between the vehicle and the first object is the first distance, and the distance between the vehicle and the second object is the second distance. The position of the vehicle is mapped to the first position in the panoramic image, the position of the first object is mapped to the second position of the panoramic image, and the position of the second object is mapped to the third position in the panoramic image. Location. When the first distance is greater than the second distance, in the panoramic image, the distance between the first position and the second position is greater than the distance between the first position and the third position. That is, when the distance between the objects around the vehicle and the vehicle changes, the distance between the vehicle model and the coordinate points corresponding to the objects on the first model will also change, so that a 3D panoramic image consistent with the actual environment around the vehicle can be obtained, eliminating the need for stitching. It can improve the detection accuracy and driver experience effect of the seam area.

Optionally, after step 503 and before step 504, it further includes the following steps: the vehicle-mounted image processing device performs interpolation and smoothing processing on the scattered points on the first model to obtain a second model, and the second model is the smoothed model. model; in step 504, the vehicle-mounted image processing device performs texture mapping on the second model according to the image information to generate a 3D panoramic image.

The first model is composed of a large number of scatter points. The on-board image processing device directly connects the scatter points, and the obtained model may be bumpy, and then texture mapping the bumpy 3D model will result in poor visual effect of the generated panoramic image. . Therefore, the vehicle-mounted image processing device performs interpolation processing on the first model. Please refer to FIG. 11 . FIG. 11 is a schematic diagram of the interpolation processing. The continuous function is interpolated on the basis of the 3D scatter model by the interpolation method, so that the 3D model The overall surface of the surface passes through the scatter points on the 3D model. The second model after interpolation processing is a 3D model with a smooth surface, and the vehicle-mounted image processing device performs texture mapping on the second model with a smooth surface, thereby improving the rendering effect of the 3D model. The interpolation method in the embodiment of the present application may adopt interpolation methods such as spline interpolation, bicubic interpolation, and discrete smooth interpolation method. In the embodiment of the present application, the surface of the second model may be made smooth by the interpolation method, and the specific interpolation method is not limited.

Referring to FIG. 12 , an embodiment of the present application provides an in-vehicle image processing apparatus, and the in-vehicle image processing apparatus is configured to execute the method performed by the in-vehicle image processing apparatus in the above method embodiments. The in-vehicle image processing apparatus 1200 includes an acquisition module 1201 and a processing module 1202 , and optionally, the in-vehicle image processing apparatus further includes a display module 1203 .

an acquisition module 1201, configured to acquire image information and depth information of objects around the vehicle, where the depth information is used to indicate coordinate point information of each point on the objects around the vehicle;

The processing module 1202 is used to obtain an initial model; adjust the first coordinate point on the initial model to a second coordinate point according to the depth information, and generate a first model; obtain based on the image information and the first model Panoramic image.

Optionally, the processing module 1202 is a processor, and the processor is a general-purpose processor or a special-purpose processor or the like. Optionally, the processor includes a transceiver unit for implementing receiving and transmitting functions. For example, the transceiver unit is a transceiver circuit, or an interface, or an interface circuit. Transceiver circuits, interfaces, or interface circuits for implementing receiving and transmitting functions are deployed separately, or optionally, integrated together. The above-mentioned transceiver circuit, interface or interface circuit is used for reading and writing code or data, or the above-mentioned transceiver circuit, interface or interface circuit is used for signal transmission or transmission.

Optionally, the acquisition module 1201 can be replaced by a transceiver module, optionally, the transceiver module is a communication interface. Optionally, the communication interface is an input-output interface or a transceiver circuit. The input and output interface includes an input interface and an output interface. The transceiver circuit includes an input interface circuit and an output interface circuit.

Optionally, the transceiver module is configured to receive image information and depth information of objects around the vehicle from the sensor.

Optionally, the acquisition module 1201 can also be replaced by the processing module 1202 .

Further, the obtaining module 1201 is configured to perform step 501 in the embodiment corresponding to FIG. 5 . The processing module 1202 is configured to execute step 502 , step 503 , step 504 and step 505 in the embodiment corresponding to FIG. 5 . Optionally, the display module 1203 is configured to perform step 505 in the embodiment corresponding to FIG. 5 , and the display module 1203 is configured to display a panoramic image.

Specifically, in an optional implementation manner, the processing module 1202 is further specifically configured to: convert the pixels in the image information into the camera coordinate system according to the pixels in the image information and the depth information corresponding to the pixels the first point cloud under the camera coordinate system; convert the first point cloud under the camera coordinate system to the second point cloud under world coordinates; adjust the first coordinate point to the second point cloud through the coordinate points in the second point cloud Two coordinate points are used to generate the first model, and the second coordinate points are obtained according to the coordinate points in the second point cloud.

In an optional implementation manner, the image information includes images collected by multiple image sensors, and the processing module 1202 is further specifically configured to: place multiple images collected by the multiple image sensors in the world coordinate system The plurality of second point clouds obtained are spliced to obtain the target point cloud; determine a plurality of third coordinate points within the neighborhood of the first coordinate point, and the third coordinate point is the coordinate point on the target point cloud; A second coordinate point is determined according to the plurality of third coordinate points; and the first coordinate point is adjusted to the second coordinate point.

In an optional implementation manner, the processing module 1202 is further specifically configured to: match the overlapping area of the first image and the second image collected by two adjacent image sensors to obtain the point of the first image A rotation matrix and a translation matrix for transforming between the cloud and the point cloud of the second image; using the rotation matrix and the translation matrix to transform the point cloud of the second image, and transforming the point cloud of the second image after the transformation The cloud is stitched with the point cloud of the first image.

In an optional implementation manner, the processing module 1202 is further specifically configured to: perform interpolation and smoothing processing on the first model to obtain a second model; and texture the second model according to the image information map to generate a panoramic image.

In an optional implementation manner, the objects around the vehicle include a first object and a second object, and when the distance between the vehicle and the first object is the first distance, the distance between the vehicle and the second object is the first object. When the distance is two, the position of the vehicle is mapped to the first position in the panoramic image, the position of the first object is mapped to the second position of the panoramic image, and the position of the second object is mapped to the panoramic image the third position in The distance between the second positions is greater than the distance between the first position and the third position.

Referring to FIG. 13 , an in-vehicle image processing apparatus in an embodiment of the present application is used to perform steps 501 to 505 in the method embodiment corresponding to FIG. 5 . For details, refer to the descriptions in the foregoing method embodiments. The in-vehicle image processing apparatus may include one or more processors 1301, and the processors 1301 may also be referred to as processing units, which may implement certain control functions. The processor 1301 may be a general-purpose processor or a special-purpose processor, for example, the processor 1301 is a graphics processor (graphics processing unit, GPU). The central processing unit can be used to control the vehicle-mounted image processing device, execute software programs, and process data of the software programs.

In an optional design, the processor 1301 may also store instructions 1303, and the instructions 1303 may be executed by the processor, so that the in-vehicle image processing apparatus 1300 executes the methods described in the above method embodiments.

In another optional design, the processor 1301 may include a transceiver unit for implementing receiving and transmitting functions. For example, the transceiver unit may be a transceiver circuit, or an interface, or an interface circuit. Transceiver circuits, interfaces or interface circuits used to implement receiving and transmitting functions may be separate or integrated. The above-mentioned transceiver circuit, interface or interface circuit can be used for reading and writing code/data, or the above-mentioned transceiver circuit, interface or interface circuit can be used for signal transmission or transmission.

The in-vehicle image processing apparatus 1300 may include one or more memories 1302, and instructions 1304 may be stored thereon, and the instructions may be executed on the processor, so that the in-vehicle image processing apparatus 1300 executes the above method implementation. method described in the example. Optionally, data may also be stored in the memory. Optionally, instructions and/or data may also be stored in the processor. The processor and the memory can be provided separately or integrated together.

Optionally, the in-vehicle image processing apparatus 1300 may further include a transceiver 1305 . The processor 1301 may be referred to as a processing unit, and controls the in-vehicle image processing apparatus 1300 . The transceiver 1305 may be referred to as a transceiver unit, a transceiver, a transceiver circuit, a transceiver device, or a transceiver module, etc., and is used to implement a transceiver function.

There is also a vehicle in the embodiment of the present application, and the vehicle includes the vehicle-mounted surround view system shown in FIG. 3 . The in-vehicle surround view system includes the in-vehicle image processing device shown in FIG. 12 , or the in-vehicle surround view system includes the in-vehicle image processing device shown in FIG. 13 , and the in-vehicle image processing device is used to perform step 501 in the above-mentioned embodiment corresponding to FIG. 5 . Go to step 505 .

The embodiments of the present application further provide a computer program product, the computer program product includes computer program code, when the computer program code is executed by the computer, the computer is made to realize the vehicle-mounted image processing device (or vehicle-mounted image processing device) in the above method embodiments. Look around system) method.

Embodiments of the present application further provide a computer-readable storage medium for storing computer programs or instructions, and when the computer programs or instructions are executed, cause the computer to execute the vehicle-mounted image processing device (or vehicle-mounted surround view system) in the above method embodiments. method of execution.

An embodiment of the present application provides a chip including a processor and a communication interface, where the processor is configured to read an instruction to execute the method performed by the vehicle-mounted image processing device (or vehicle-mounted surround view system) in the above method embodiments.

The integrated unit, if implemented in the form of a software functional unit and sold or used as an independent product, may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application can be embodied in the form of software products in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, and the computer software products are stored in a storage medium , including several instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, Read-Only Memory (ROM, Read-Only Memory), Random Access Memory (RAM, Random Access Memory), magnetic disk or optical disk and other media that can store program codes .

As mentioned above, the above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those of ordinary skill in the art should understand: The technical solutions recorded in the embodiments are modified, or some technical features thereof are equivalently replaced; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions in the embodiments of the present application.

Claims

A method for generating panoramic images, comprising:

Obtain image information and depth information of objects around the vehicle, where the depth information is used to indicate coordinate point information of each point on the objects around the vehicle;

get the initial model;

Adjusting the first coordinate point on the initial model to the second coordinate point according to the depth information to generate a first model;

A panoramic image is acquired based on the image information and the first model.
The method according to claim 1, wherein the generating the first model by adjusting the first coordinate point on the initial model to the second coordinate point according to the depth information comprises:

Convert the pixels in the image information to the first point cloud in the camera coordinate system according to the pixels in the image information and the depth information corresponding to the pixels;

Converting the first point cloud in the camera coordinate system to the second point cloud in world coordinates;

The first model is generated by adjusting the first coordinate point to the second coordinate point through the coordinate point in the second point cloud, and the second coordinate point is obtained according to the coordinate point in the second point cloud of.
The method according to claim 2, wherein the image information comprises images collected by a plurality of image sensors, and the converting the first point cloud in the camera coordinate system into the second point cloud in world coordinates Afterwards, the method further includes:

splicing multiple images collected by the multiple image sensors on multiple second point clouds in the world coordinate system to obtain a target point cloud;

The adjusting from the first coordinate point to the second coordinate point through the coordinate points in the second point cloud includes:

determining a plurality of third coordinate points within the neighborhood of the first coordinate point, where the third coordinate points are coordinate points on the target point cloud;

determining a second coordinate point according to the plurality of third coordinate points;

Adjust the first coordinate point to the second coordinate point.
The method according to claim 3, characterized in that, splicing multiple second point clouds of multiple images collected by the multiple image sensors in the world coordinate system to obtain the target point cloud, comprising:

Match the overlapping areas of the first image and the second image collected by two adjacent image sensors to obtain a rotation matrix and a translation matrix for transformation between the point cloud of the first image and the point cloud of the second image ;

The point cloud of the second image is transformed by using the rotation matrix and the translation matrix, and the transformed point cloud of the second image and the point cloud of the first image are stitched together.
The method according to any one of claims 1-4, wherein after the generating the first model, the method further comprises:

Interpolating and smoothing the first model to obtain a second model;

The obtaining a panoramic image based on the image information and the first model includes:

Perform texture mapping on the second model according to the image information to generate a panoramic image.
The method according to claim 1, wherein the objects around the vehicle include a first object and a second object, and when the distance between the vehicle and the first object is the first distance, the distance between the vehicle and the second object is the first distance. When the distance is the second distance, the position of the vehicle is mapped to the first position in the panoramic image, the position of the first object is mapped to the second position of the panoramic image, and the position of the second object is mapped to a third position in the panoramic image, when the first distance is greater than the second distance, the method further includes:

The panoramic image is displayed, in which the distance between the first position and the second position is greater than the distance between the first position and the third position.
A vehicle-mounted surround-view device, comprising:

an acquisition module for acquiring image information and depth information of objects around the vehicle, where the depth information is used to indicate coordinate point information of each point on the objects around the vehicle;

a processing module for acquiring an initial model; adjusting the first coordinate point on the initial model to a second coordinate point according to the depth information to generate a first model; obtaining a panorama based on the image information and the first model image.
The device according to claim 7, wherein the processing module is further specifically configured to:

Convert the pixels in the image information to the first point cloud in the camera coordinate system according to the pixels in the image information and the depth information corresponding to the pixels;

Converting the first point cloud in the camera coordinate system to the second point cloud in world coordinates;

The first model is generated by adjusting the first coordinate point to the second coordinate point through the coordinate point in the second point cloud, and the second coordinate point is obtained according to the coordinate point in the second point cloud of.
The device according to claim 8, wherein the image information comprises images collected by a plurality of image sensors, and the processing module is further specifically configured to:

splicing multiple images collected by the multiple image sensors on multiple second point clouds in the world coordinate system to obtain a target point cloud;

determining a plurality of third coordinate points within the neighborhood of the first coordinate point, where the third coordinate points are coordinate points on the target point cloud;

determining a second coordinate point according to the plurality of third coordinate points;

Adjust the first coordinate point to the second coordinate point.
The device according to claim 9, wherein the processing module is further specifically configured to:

Match the overlapping areas of the first image and the second image collected by two adjacent image sensors to obtain a rotation matrix and a translation matrix for transformation between the point cloud of the first image and the point cloud of the second image ;

The point cloud of the second image is transformed by using the rotation matrix and the translation matrix, and the transformed point cloud of the second image and the point cloud of the first image are stitched together.
The device according to any one of claims 7-10, wherein the processing module is further specifically configured to:

Interpolating and smoothing the first model to obtain a second model;

Perform texture mapping on the second model according to the image information to generate a panoramic image.
The device according to claim 7, wherein the objects around the vehicle include a first object and a second object, and when the distance between the vehicle and the first object is the first distance, the distance between the vehicle and the second object is the first distance. When the distance is the second distance, the position of the vehicle is mapped to the first position in the panoramic image, the position of the first object is mapped to the second position of the panoramic image, and the position of the second object is mapped to a third position in the panoramic image, when the first distance is greater than the second distance, the device further includes a display module; the display module is further configured to display the panoramic image, where the panoramic image , the distance between the first position and the second position is greater than the distance between the first position and the third position.
An in-vehicle image processing device, characterized in that it includes a processor, the processor is coupled with a memory, and the memory is used to store programs or instructions, and when the programs or instructions are executed by the processor, the The in-vehicle image processing device executes the method according to any one of claims 1 to 6 .
An in-vehicle surround view system, characterized in that it includes a sensor, the in-vehicle image processing device and the in-vehicle display as claimed in claim 13, wherein the sensor and the in-vehicle display are both connected to the in-vehicle image processor, wherein the The sensor is used to collect image information and depth information, and the vehicle-mounted display is used to display panoramic images.
A vehicle, characterized by comprising the vehicle-mounted surround view system as claimed in claim 14 .
A computer program product, comprising computer program code, characterized in that, when the computer program code is executed by a computer, the computer is made to implement the method according to any one of claims 1 to 6 .
A computer-readable storage medium, characterized in that it is used for storing computer programs or instructions, which, when executed, cause a computer to perform the method according to any one of claims 1 to 6 .
A chip, characterized by comprising a processor and a communication interface, wherein the processor is configured to read instructions to execute the method of any one of claims 1 to 6 .