WO2019192359A1 - Vehicle panoramic video display system and method, and vehicle controller - Google Patents

Abstract

A vehicle panoramic video display system and method and a vehicle controller, the system comprising: a vehicle video collector (1010), a vehicle controller (1020) and a vehicle display (1030), wherein the vehicle video collector (1010) comprises a plurality of cameras; each camera separately collects original videos in a coverage area, and the total coverage area of all of the cameras in the vehicle video collector (1010) is greater than or equal to 360 degrees; and the original videos collected by the each camera are sent to the vehicle controller (1020). The vehicle controller (1020) is used to search for a pre-established panoramic synthesis mapping table and to respectively generate a video to be spliced that corresponds to each original video and a splicing relationship between each video to be spliced; the videos to be spliced are spliced according to the splicing relationship so as to generate a panoramic video; and the panoramic video is sent to the vehicle display (1030). The vehicle display (1030) is used to display the panoramic video by means of a preset display policy. By means of the present solution, the safety of driving a vehicle and the display effect of a captured video may be improved.

Description

Vehicle panoramic video display system, method and vehicle controller

This application claims priority from the Chinese Patent Application entitled "A Car Panoramic Video Display System, Method, and Vehicle Controller" by the Chinese Patent Office on April 2, 2018, the application number is 201101828336.3, the entire contents of which are incorporated by reference. Combined in this application.

Technical field

The present application relates to the field of intelligent assisted driving technology, and in particular, to a vehicle panoramic video display system, method, and vehicle controller.

Background technique

The intelligent assisted driving technology of the vehicle plays an important role in driving safety and efficient driving efficiency. The use of intelligent assisted driving technology can largely avoid traffic accidents caused by driving errors of the driver of the vehicle.

In the intelligent assisted driving technology, there is an intelligent assisted driving method based on video information. In the intelligent assisted driving mode, the driving recorder is used to capture the road condition ahead of the vehicle, and based on the captured video, it is identified whether there is an obstacle target ahead of the vehicle, and if the obstacle target is identified, an early warning signal is generated, and according to the The warning signal prompts the driver to avoid obstacles. However, the factors affecting the driving safety of the vehicle include not only the obstacles directly in front of the vehicle, but also the driving state of the driver affecting the driving safety. For example, the driver is in a fatigue driving state, and the probability of a traffic accident is also increased.

In order to cope with the above problem, a camera is added to the driving recorder for collecting the driver's eye information, and by analyzing the driver's eye information, if it is determined that the driver is in a fatigue driving state, or based on the above-mentioned shooting road conditions. The video recognizes that there is an obstacle target in front of the vehicle, and generates an early warning signal, and according to the warning signal, prompts the driver to have a safety hazard. However, based on the above scheme, the driving recorder can only capture the road condition directly in front of the vehicle and the driver's video, and it is impossible to photograph the side of the vehicle, and the safety of the side of the vehicle is difficult to be secured; and the driving recorder is performing When the video is displayed, the in-vehicle video or the off-camera video is displayed independently, and the driver cannot observe the overall situation of the vehicle in a macroscopic view, and the display effect is poor.

Summary of the invention

The purpose of the embodiments of the present application is to provide a vehicle panoramic video display system, method, and vehicle controller to improve the safety of driving the vehicle and the display effect of the captured video. The specific technical solutions are as follows:

In a first aspect, an embodiment of the present application provides a vehicle panoramic video display system, where the system includes an in-vehicle video collector, a vehicle controller, and an in-vehicle display;

The vehicle video capture device includes a plurality of cameras, and each camera captures original video in the coverage area, and the total coverage area of all the cameras in the vehicle video collector is greater than or equal to 360 degrees; the original video collected by each camera is sent to The vehicle controller;

The vehicle controller is configured to receive the original video collected by each camera sent by the vehicle video capture device; and search for a pre-established panoramic synthesis mapping table, respectively generate a video to be spliced corresponding to each original video and a video to be spliced a splicing relationship, the panorama synthesis mapping table includes a correspondence between pixel coordinates of the video to be spliced and pixel coordinates of the original video; and splicing the video to be spliced according to the splicing relationship to generate a panoramic video; Sended to the on-board display;

The on-board display is configured to receive the panoramic video sent by the onboard controller; and display the panoramic video by using a preset display policy.

In a second aspect, an embodiment of the present application provides a vehicle panoramic video display method, which is applied to an in-vehicle controller, and the method includes:

Obtaining the original video collected by each camera in the car video collector;

Searching for a pre-established panoramic synthesis mapping table, respectively, to generate a splicing relationship between the video to be spliced corresponding to each original video and each video to be spliced, wherein the panoramic compositing mapping table includes pixel coordinates of the video to be spliced and pixel coordinates of the original video. Correspondence relationship

According to the splicing relationship, the video to be spliced is spliced to generate a panoramic video;

The panoramic video is sent to the onboard display.

In a third aspect, an embodiment of the present application provides an onboard controller, where the onboard controller includes a processor and a memory;

The memory is configured to store a computer program;

The processor, when executing the program stored on the memory, implements the following steps:

Obtaining the original video collected by each camera in the car video collector;

Searching for a pre-established panoramic synthesis mapping table, respectively, to generate a splicing relationship between the video to be spliced corresponding to each original video and each video to be spliced, wherein the panoramic compositing mapping table includes pixel coordinates of the video to be spliced and pixel coordinates of the original video. Correspondence relationship

According to the splicing relationship, the video to be spliced is spliced to generate a panoramic video;

The panoramic video is sent to the onboard display.

In a fourth aspect, the embodiment of the present application provides a computer readable storage medium, where the computer readable storage medium stores a computer program, and when the computer program is executed by the processor, the following steps are implemented:

Obtaining the original video collected by each camera in the car video collector;

Searching for a pre-established panoramic synthesis mapping table, respectively, to generate a splicing relationship between the video to be spliced corresponding to each original video and each video to be spliced, wherein the panoramic compositing mapping table includes pixel coordinates of the video to be spliced and pixel coordinates of the original video. Correspondence relationship

According to the splicing relationship, the video to be spliced is spliced to generate a panoramic video;

The panoramic video is sent to the onboard display.

In summary, in the solution provided by the embodiment of the present application, the vehicle panoramic video display system includes an in-vehicle video collector, a vehicle controller, and an in-vehicle display. The vehicle video capture device includes a plurality of cameras, each camera respectively collects original video in the coverage area, and sends the original video collected by each camera to the vehicle controller; the vehicle controller is used to find a pre-established panoramic synthesis mapping table. Generating a splicing relationship between the video to be spliced corresponding to each original video and each video to be spliced respectively; according to the splicing relationship, splicing each video to be spliced to generate a panoramic video, and transmitting the generated panoramic video to the vehicle display; Used to display the received panoramic video through a preset display policy. Since the total coverage area of all the cameras in the vehicle video capture device is greater than or equal to 360 degrees, during the running of the vehicle, not only the road ahead of the vehicle is photographed, but also the side of the vehicle and the interior of the vehicle are photographed, and the 360 degree has no dead angle. Panoramic shooting of the surrounding conditions of the vehicle to effectively improve the safety of the vehicle; and the vehicle controller synthesizes the panoramic video by splicing the video to be spliced corresponding to the original video collected by each camera, and using the vehicle display to view the panorama The display of the video effectively improves the display effect of the video, and enables the driver to more intuitively observe the surroundings of the vehicle from the display content, thereby improving the driver's interactive experience.

DRAWINGS

1 is a schematic structural diagram of a vehicle panoramic video display system according to an embodiment of the present application;

2 is a schematic flowchart of a method for displaying a vehicle panoramic video according to an embodiment of the present application;

FIG. 3a is a schematic structural diagram of a corresponding fisheye device including a video processing chip; FIG.

FIG. 3b is a schematic structural diagram of a corresponding fisheye device without a video processing chip; FIG.

4 is a schematic flowchart of a working mechanism of an on-board controller according to an embodiment of the present application;

FIG. 5 is a schematic flowchart of a process for generating a panoramic synthesis mapping table according to an embodiment of the present application;

FIG. 6 is a schematic structural diagram of a panoramic synthesis mapping table according to an embodiment of the present application;

FIG. 7 is a schematic flowchart of a panoramic video generation process according to an embodiment of the present application;

FIG. 8 is a schematic diagram of adjusting a panoramic video when a driver's posture changes according to an embodiment of the present application; FIG.

FIG. 9 is a schematic diagram of adjusting a panoramic video when an object of an obstacle outside the vehicle is recognized according to an embodiment of the present application;

FIG. 10 is a schematic structural diagram of a vehicle panoramic video display system according to another embodiment of the present application; FIG.

FIG. 11 is a schematic diagram of adjustment of panoramic video when a steering wheel of a vehicle rotates according to an embodiment of the present application; FIG.

FIG. 12 is a schematic diagram showing the display of a panoramic video when the steering wheel angle is β _{0 according to} an embodiment of the present application; FIG.

12b is a schematic diagram showing the display of a panoramic video when the steering wheel angle is β _{1 according to} an embodiment of the present application;

FIG. 13 is a schematic structural diagram of an on-board controller according to an embodiment of the present application.

detailed description

In order to make the objects, technical solutions, and advantages of the present application more comprehensible, the present application will be further described in detail below with reference to the accompanying drawings. It is apparent that the described embodiments are only a part of the embodiments of the present application, and not all of them. All other embodiments obtained by a person of ordinary skill in the art based on the embodiments of the present application without departing from the inventive scope are the scope of the present application.

In order to improve the safety of driving the vehicle and the display effect of the captured video, the embodiment of the present application provides a vehicle panoramic video display system, method, and vehicle controller, which are respectively described in detail below.

The embodiment of the present application provides a vehicle panoramic video display system, as shown in FIG. 1 , which is a schematic structural diagram of a vehicle panoramic video display system, including: a vehicle video capture device 110 , a vehicle controller 120 , and a vehicle Display 130.

In the following, the vehicle panoramic video display method provided by the embodiment of the present application is introduced from the perspective of the interaction of the devices in the vehicle panoramic video display system as shown in FIG. 1. As shown in FIG. 2, the vehicle panoramic video display method can be Including the following steps.

In S201, each camera in the vehicle video capture device separately collects the original video in the coverage area.

The car video capture device is mounted on the vehicle and can be located in the front of the vehicle. The vehicle video capture device includes a plurality of cameras, and the total coverage area of all the cameras is greater than or equal to 360 degrees. For example, the in-vehicle video collector can include four cameras, each of which covers a coverage area of 90 degrees, and the total coverage area of the four cameras is a 360-degree coverage area; for example, in a car video collector. It can include 3 cameras, each of which has a coverage area of 120 degrees, and the total coverage area of the three cameras is a 360-degree coverage area. The coverage area of each camera in the vehicle video capture device may not be equal. For example, the vehicle video capture device may include five cameras, the coverage area of two cameras is a coverage area of 90 degrees, and the coverage areas of the other three cameras are A coverage area of 60 degrees; for example, the vehicle video capture device may include five cameras, and the coverage areas of the cameras are 45 degrees, 60 degrees, 65 degrees, 90 degrees, and 100 degrees, respectively. The coverage area of the camera is the area covered by the camera's field of view.

In a possible implementation manner of the present application, the in-vehicle video collector may be a fisheye device.

As shown in Figures 3a and 3b, for two typical fisheye devices, the fisheye device shown in Figure 3a and the fisheye device shown in Figure 3b are both back-to-back fisheye cameras (first fisheye) The camera 301 and the second fisheye camera 302) are composed. The fisheye device shown in Fig. 3a includes a video processing chip 303, and the fisheye device shown in Fig. 3b does not include a video processing chip. The difference between the two types of fisheye equipment is whether it has video processing capability, and the video processing chip can realize functions such as panoramic video synthesis and intelligent detection. However, since the fisheye lens device shown in Fig. 3a includes the video processing chip 303, the size and weight of the fisheye device are significantly heavier than the binocular device shown in Fig. 3b, and the cost is higher, and assembly is difficult. Therefore, in this embodiment, the fisheye device shown in FIG. 3b, the fisheye device without the video processing chip, the panoramic video synthesis function, and only the video capture can be used, and the fisheye device can complete the image filtering. Basic image processing functions such as noise and white balance. In this way, the system cost can be effectively controlled, the miniaturization of the fisheye device can be facilitated, and the baseline convergence between the cameras can be reduced.

Each camera in the car video capture device may not have consistent key parameters and installation parameters, including but not limited to camera internal parameters, field of view angle, distortion, focal length, etc.; installation parameters include but are not limited to installation position, installation angle Wait. As long as the total coverage area of all cameras satisfies greater than or equal to 360 degrees.

S202. The vehicle video collector sends the original video collected by each camera to the vehicle controller.

The vehicle controller is a device installed in the central control position of the vehicle, communicating with the vehicle host and the communication bus according to a prescribed protocol, and completing the collection, storage and transmission of information. The vehicle video capture device sends the original video collected by each camera to the vehicle controller, so that the vehicle controller can complete functions such as panoramic video generation and intelligent detection.

In a possible implementation manner of the present application, between the vehicle video capture device and the vehicle controller, the original video collected by each camera may be transmitted through the interaction module.

The original video captured by each camera can be transmitted between the vehicle video capture device and the vehicle controller through an interactive module. The interaction module includes a communication bus between the vehicle video capture device and the vehicle controller, various data protocols, and a vehicle video capture device. Communication interface between communication interface and vehicle controller. The communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus.

The original video captured by each camera can also be transmitted by wireless transmission between the vehicle video capture device and the vehicle controller, and will not be described here.

S203. The in-vehicle controller searches for a pre-established panoramic synthesis mapping table, and respectively generates a stitching relationship between the video to be stitched corresponding to each original video and each video to be stitched.

The panoramic composition map includes the correspondence between the pixel coordinates of the video to be stitched and the pixel coordinates of the original video. The working mechanism of the vehicle controller is as shown in FIG. 4, and the vehicle controller stores a panoramic synthesis mapping table and an intelligent detection model file. The function of the on-board controller is to complete, including but not limited to, panoramic video generation, intelligent detection, etc., output panoramic video and detection information, and can perform viewpoint and perspective transformation on the panoramic video according to the control information input to realize panoramic roaming.

The panoramic synthesis map is pre-established and stored in the onboard controller, and the panoramic synthesis map is used to synthesize panoramic video. The panoramic synthesis mapping table describes a one-to-one correspondence between the pixels of the video to be spliced and the pixels of the original video. The panoramic synthesis mapping table stores the pixel coordinates of the video and the splicing relationship between the videos to be spliced.

In a possible implementation manner of the present application, the method for establishing the panoramic synthesis mapping table may include the following steps: acquiring sample images collected by each camera in the vehicle video capture device, and participating in each camera of the vehicle video collector calibration Camera external reference; according to each camera internal reference, each sample image is spherically projected to generate a spherical projection image corresponding to each sample image; according to each camera external parameter, the relative positional relationship between each spherical projection image is obtained; according to the relative positional relationship Convert and crop each spherical projection image to obtain a panoramic composition map.

The sample image is a pre-acquired image when the vehicle video capture device establishes the panoramic synthesis mapping table, and the sample image may be a pre-photographed image, or may be any video frame in the pre-acquired video sample, or multiple video frames; The sample image may also be any video frame or multiple video frames extracted from the original video before the panoramic video is generated for the original video captured by each camera. In the sample image, the camera internal parameter or the camera external parameter is often pre-calibrated. Therefore, each camera internal parameter or each camera external parameter can be obtained directly through each sample image.

Taking the vehicle video recorder as a fisheye device as an example, as shown in FIG. 5, a process of generating a panoramic synthesis map is provided. Obtaining the sample images collected by the two fisheye cameras separately, and expanding the two sample images through spherical projection to generate an undistorted spherical projection image, and then obtaining a relative positional relationship between the two images according to the external parameters of the camera, according to the relative position Relationship, transforming and cropping two images to obtain a panoramic composition map. The transformation of the two images may be: using one of the images as a reference, and transforming the other image to achieve seamless mosaic of the panorama.

Since the relative positional relationship of each camera in the vehicle video capture device is fixed, the internal parameters and external parameters of the camera can be obtained by offline calibration, and the relative positional relationship between the images/videos collected by each camera is also fixed, and the coordinate image can be The coordinates of the spherical projection are expanded and transformed, and after cropping, a panoramic composite mapping table can be obtained. The panorama synthesis map is only relevant to the resolution of the input image. With off-line calibration, only one calibration is required for each car video collector, which can quickly generate panoramic video.

The process of generating the panoramic synthesis mapping table may also be generated according to an inverse process, establishing an empty panoramic mapping table, and establishing a correspondence between the panoramic composite mapping table and the original image through inverse transformation and pseudo projection expansion.

In a possible implementation manner of the present application, the method for establishing the panoramic synthesis mapping table may include the following steps: acquiring sample images collected by each camera in the vehicle video capture device, and participating in each camera of the vehicle video collector calibration a camera external parameter; establishing a first panoramic mapping table, the content of the first panoramic mapping table is empty; according to each camera external parameter, obtaining a positional relationship between each preset area in the first panoramic mapping table; according to the positional relationship, each The preset area is inversely transformed to obtain an inverse transformation area of each preset area; according to each camera internal parameter, spherical image projection is performed on each sample image to generate a spherical projection image corresponding to each sample image; according to each spherical projection image and each inverse transformation The correspondence between the regions determines the panoramic composition map.

The panoramic synthesis map can be generated by the above positive process or the inverse process, and the positive process generates a panoramic synthesis map with less time, and the inverse process generates a panoramic synthesis map with higher accuracy.

The external parameters of each camera of the car video capture device are often fixed. After the car video capture device is installed, the external position of the camera installation position and installation angle are fixed, but the camera internal reference and the camera external reference need to be pre-independent. The auxiliary calibration equipment is calibrated. In order to reduce the number of auxiliary calibration equipment, in order to improve the efficiency of internal reference and external reference calibration, the external reference of the camera in the image can be calibrated by the external reference to calculate the internal reference of the camera, that is, only the external parameters of each camera need to be calibrated. The internal parameters of each camera can be obtained by the preset internal parameter detection algorithm, and the internal reference parameters of each camera are not required to be calibrated by a separate internal reference calibration auxiliary device.

In a possible implementation manner of the present application, the manner of establishing the panoramic synthesis mapping table may further include: acquiring an external reference calibration image collected by each camera in the vehicle video capture device, and a calibration camera outside the calibration image of each external reference. Detecting; detecting the outer edge of the effective region of each external reference calibration image; fitting the outer edge of the effective region of each external reference calibration image by curve fitting, obtaining the outer contour of the effective region of each external reference calibration image; calibrating based on each external parameter The outer contour of the effective area of the image determines the camera internal parameters for collecting the calibration images of the external parameters; according to the internal parameters of each camera, the spherical images of the external reference calibration images are respectively spherically generated, and the spherical projection images corresponding to the external reference calibration images are generated; according to the respective cameras The outer parameter obtains a relative positional relationship between the spherical projection images; and according to the relative positional relationship, each spherical projection image is transformed and cropped to obtain a panoramic composite mapping table.

The external reference calibration image is an image of the vehicle-mounted video collector that is pre-acquired based on the external parameters of each camera when establishing the panoramic composite mapping table. The external reference calibration image may be a pre-photographed image, or may be a video pre-acquired based on each camera external parameter. Any video frame in the sample, or multiple video frames; the external reference calibration image may also be any video frame or multiple video frames extracted from the original video before the panoramic video is generated for the original video captured by each camera. . The camera external parameter is pre-calibrated in the external reference calibration image. Therefore, the calibration external camera parameters can be obtained directly from each external reference calibration image.

The method of detecting the outer edge of the effective area of each external reference calibration image may detect the outer edge of the effective area by recognizing the transition edge of the black and white area of the image, or search for the four farthest points of the outer edge of the effective area by means of a straight line search. The outer edge of the effective area is obtained by connecting the four farthest points. The manner of detecting the outer edge of the effective area belongs to the protection scope of the embodiment, and details are not described herein again.

The panoramic synthesis map may also be generated through the inverse process by using the camera external parameters and the calculated camera internal parameters based on the calibration of the external reference calibration image.

In a possible implementation manner of the present application, the manner of establishing the panoramic synthesis mapping table may further include: acquiring an external reference calibration image collected by each camera in the vehicle video capture device, and a calibration camera outside the calibration image of each external reference. Detecting; detecting the outer edge of the effective region of each external reference calibration image; fitting the outer edge of the effective region of each external reference calibration image by curve fitting, obtaining the outer contour of the effective region of each external reference calibration image; calibrating based on each external parameter An outer contour of the effective area of the image, determining a camera internal parameter for collecting the calibration image of each foreign parameter; establishing a first panoramic mapping table, the content of the first panoramic mapping table is empty; according to each camera external parameter, obtaining each of the first panoramic mapping table Positional relationship between preset areas; inversely transforming each preset area according to the positional relationship, and obtaining an inverse transformation area of each preset area; performing spherical projection on each of the external reference calibration images according to each camera internal parameter, generating each a spherical projection image corresponding to the external reference calibration image; according to the correspondence between each spherical projection image and each inverse transformation region, Panorama predetermined mapping table.

In a possible implementation manner of the present application, the panoramic synthesis mapping table includes respective mapping tables of the cameras in the in-vehicle video collector, and there is an overlapping area between the two mapping tables.

The content of the overlapping image of the sample images acquired by two adjacent cameras is the same. Taking the vehicle video recorder as a fisheye device as an example, as shown in the panoramic composite mapping table structure shown in FIG. 6, the panoramic synthesis mapping table is divided into two parts, a first fisheye camera mapping table 601 and a second fisheye camera mapping table. 602, which respectively represents the mapping relationship between the images acquired by the two fisheye cameras in the plane panorama. The first fisheye camera mapping table 601 and the second fisheye camera mapping table 602 each have an overlapping area representing the overlapping area of the two fisheye cameras on the field of view, and the contents in the two overlapping areas are the same. The splicing relationship between the video to be spliced and the spliced video to be spliced corresponding to the original video may be generated for each original video by using a pre-established panoramic compositing mapping table.

S204: The vehicle controller splices the video to be spliced according to the splicing relationship to generate a panoramic video.

Through off-line calibration, for a car video collector with a fixed relative position of each camera, only the panoramic synthesis mapping table needs to be stored in the vehicle controller, and when the vehicle video collector collects the video, the panoramic synthesis mapping table is used. Looking up the table, the video to be spliced is obtained, and the panoramic video can be generated by splicing and merging the videos to be spliced. The splicing relationship is given in the splicing relationship, for example, the first column of pixels of the second video to be spliced overlaps with the twentieth column of the first video to be spliced, and the second video to be spliced is flipped 45 After the degree overlaps with the first video to be stitched and so on.

Since the panoramic synthesis mapping table is generated based on the sample image, when the video is spliced, the video frames may be spliced, and then the spliced video frames are video-combined to obtain a panoramic video.

In a possible implementation manner of the present application, the S203 may include: searching for a pre-established panoramic synthesis mapping table, respectively generating a video frame to be spliced corresponding to each video frame in each original video and a video frame to be spliced Splicing relationship; for each video frame to be spliced generated by the same original video, using a preset video generation technology to generate a video to be spliced.

The S204 may include: splicing corresponding video frames to be spliced in each video to be spliced according to a splicing relationship between the video frames to be spliced, to obtain multiple panoramic images; and using preset video generation technology based on each panoramic image , generate a panoramic video.

The splicing relationship between the video frames to be spliced and the video frames to be spliced corresponding to each video frame in the original video can be obtained based on the panoramic compositing mapping table. Since the video is generated by the video frame, the preset video generation technology can be used to generate the splicing relationship. For the video to be spliced, the preset video generation technology specifies the order of the video frames in the video and the playback speed when the video is generated. Based on the splicing relationship between the video frames to be spliced, the corresponding video frames to be spliced may be spliced to obtain multiple panoramic images, and the panoramic video may be generated by using a preset video generation technology.

The panoramic video can be a two-dimensional video or a three-dimensional video. Since the three-dimensional video can more clearly display the spatial positional relationship between obstacles, vehicles, and drivers in the three-dimensional space, the panoramic video is mainly three-dimensional video.

In a possible implementation manner of the present application, the S204 may be specifically: splicing each video to be spliced according to a splicing relationship to generate a two-dimensional video; and performing three-dimensional projection on the two-dimensional video according to a preset three-dimensional projection manner, Panoramic video.

Taking the vehicle video capture device as the fisheye device as an example, when the fisheye device collects the video, the video collected by the two fisheye cameras is used to look up the table by using the panoramic synthesis mapping table, and two videos to be stitched are obtained, according to the stitching relationship. The two videos to be stitched are stitched together to generate a two-dimensional video, and a three-dimensional spherical panoramic video can be generated by performing spherical rendering on the two-dimensional video.

As shown in FIG. 7 , when the fisheye device collects the video, the video frames collected by the two fisheye cameras can also be extracted, and the panoramic synthesis map is used to perform the table lookup, and two subgraphs to be spliced are obtained, and two are to be spliced. The sub-picture is spliced by the image to obtain a two-dimensional panoramic image. After the video generation technology, the two-dimensional panoramic video is generated, and the two-dimensional panoramic video is spherically rendered to generate a three-dimensional spherical panoramic video. The 2D panoramic video can generate corresponding 3D spherical panoramic video through cylindrical projection, cube projection and other projection methods. Alternatively, the two-dimensional panoramic image may be spherically projected to obtain a three-dimensional panoramic image, and a three-dimensional spherical panoramic video is generated by a video generation technology, which is not specifically limited herein.

S205. The vehicle controller sends the panoramic video to the vehicle display.

The in-vehicle display is a display device for displaying panoramic video installed in a centrally controlled position of the vehicle.

In a possible implementation manner of the present application, the panoramic video can be transmitted through the interaction module between the onboard controller and the onboard display.

The panoramic video can be transmitted between the vehicle controller and the vehicle display through the interaction module. The interaction module includes a communication bus between the vehicle controller and the vehicle display, various data protocols, and a communication interface of the vehicle controller and a communication interface of the vehicle display. The communication bus may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus.

The panoramic video can also be transmitted between the on-board controller and the on-board display by wireless transmission, and will not be described here.

S206. The on-board display displays a panoramic video by using a preset display strategy.

The preset display strategy may be to clear the content that has been displayed on the in-vehicle display, and then display the panoramic video; for the panoramic video of the three-dimensional spherical surface, the preset display strategy may also be a three-dimensional display mode, which is not described here.

Vehicle conditions in the interior of the vehicle and outside the vehicle are constantly changing, such as steering wheel rotation, obstacles in front of the vehicle, changes in driver attitude, and the like. When the vehicle status changes, the panoramic video needs to be adjusted so that the on-board display can display the adjusted panoramic video to provide the driver with a more intuitive vehicle scene.

In a possible implementation manner of the present application, after S204, the method further includes the following steps: the vehicle controller acquires vehicle state information; the vehicle controller determines the to-be-processed information of the panoramic video according to the vehicle state information; and the vehicle controller is configured according to the vehicle The information to be processed is processed by the panoramic video to obtain a processed panoramic video.

Then, S205 may specifically be: the vehicle controller sends the processed panoramic video to the on-board display.

The S206 may specifically be: the on-vehicle display displays the processed panoramic video after a preset display strategy.

The vehicle state information may include steering wheel rotation information, driver attitude change information, vehicle exterior obstacle target information, and the like, corresponding to the in-vehicle device state, the driver state, and the vehicle exterior state, respectively. The processing of the panoramic video may be an adjustment of parameters such as a viewing angle and a viewpoint.

In a possible implementation of the present application, the vehicle status information includes driver attitude change information.

The vehicle video capture device can also be used to identify driver attitude change information and transmit driver attitude change information to the vehicle controller.

The step of acquiring the vehicle state information by the vehicle controller may be specifically: the vehicle controller acquires driver posture change information sent by the vehicle video collector.

The step of determining, by the in-vehicle controller, the information to be processed of the panoramic video according to the vehicle state information may be specifically: the vehicle controller determines the viewpoint conversion parameter of the panoramic video by the viewpoint conversion relationship according to the driver posture change information.

The step of the vehicle controller processing the panoramic video according to the to-be-processed information to obtain the processed panoramic video may be: the vehicle controller performs the viewpoint update on the panoramic video according to the viewpoint conversion parameter, and obtains the panoramic video after the viewpoint update.

The step of the vehicle controller transmitting the processed panoramic video to the in-vehicle display may be: the vehicle controller sends the updated panoramic video of the viewpoint to the on-board display.

The step of displaying the processed panoramic video through the preset display strategy of the on-vehicle display may be: the on-board display passes the preset display strategy, and displays the panoramic video after the viewpoint update.

The driver's posture has changed. For example, the driver's body leans back and causes the head to move. At this time, the camera in the car video collector that recognizes the inside of the car recognizes the change of the human body posture, and transmits the collected human body posture change video to the vehicle. The controller detects that the driver's head has a displacement of (x _m , y _m , z _m ), and the vehicle controller adjusts the viewpoint of the panoramic video by the parameter, and the viewpoint is (x ₀ , y ₀ , z ₀ ) The change is (x ₁ , y ₁ , z ₁ ), and the viewpoint transformation parameter can be described as:

k _x , k _y and k _z are viewpoint control parameters and can be set in advance. As shown in FIG. 8, when the driver's posture changes, the camera facing the interior of the vehicle can recognize the video capture and posture change of the driver inside the vehicle, the vehicle controller performs motion analysis and tracking, and uses the motion information to the panoramic view. The viewpoint is updated to establish an interaction between the in-vehicle personnel and the vehicle panoramic video display system.

In a possible implementation manner of the present application, the vehicle status information may include: an off-vehicle obstacle target information.

The vehicle video collector can also be used to identify the obstacle information of the vehicle outside the vehicle and send the obstacle information of the vehicle to the vehicle controller.

The step of acquiring the vehicle state information by the vehicle controller may be specifically: the vehicle controller acquires the vehicle obstacle target information identified by the vehicle video collector.

The step of determining, by the vehicle controller, the information to be processed of the panoramic video according to the vehicle status information may specifically be: the vehicle controller determines the category information of the obstacle outside the vehicle according to the target information of the obstacle outside the vehicle, and the obstacle video in the panoramic video Location information in .

The step of the vehicle controller processing the panoramic video according to the to-be-processed information to obtain the processed panoramic video may specifically be: the vehicle controller superimposes the category information and the location information on the panoramic video to obtain the panoramic video after the superimposed information.

The step of the vehicle controller transmitting the processed panoramic video to the vehicle display may be: the vehicle controller sends the panoramic video after the superimposed information to the on-board display.

The step of displaying the processed panoramic video through the preset display strategy of the on-vehicle display may specifically be: the on-vehicle display passes the preset display strategy, and displays the panoramic video after the superimposed information.

The camera facing the outside of the car in the car video capture device can capture the obstacle target that affects the vehicle outside the vehicle. Through analysis, the category information of the obstacle outside the vehicle and the position information of the obstacle outside the vehicle in the panoramic video can be obtained. The category information and location information of the external obstacle target are superimposed on the panoramic video and displayed on the on-board display, and the warning purpose of the obstacle target can be achieved, as shown in FIG.

With the embodiment, the vehicle panoramic video display system includes an in-vehicle video collector, a vehicle controller, and an in-vehicle display. The vehicle video capture device includes a plurality of cameras, each camera respectively collects original video in the coverage area, and sends the original video collected by each camera to the vehicle controller; the vehicle controller is used to find a pre-established panoramic synthesis mapping table. Generating a splicing relationship between the video to be spliced corresponding to each original video and each video to be spliced respectively; according to the splicing relationship, splicing each video to be spliced to generate a panoramic video, and transmitting the generated panoramic video to the vehicle display; Used to display the received panoramic video through a preset display policy. Since the total coverage area of all the cameras in the vehicle video capture device is greater than or equal to 360 degrees, during the running of the vehicle, not only the road ahead of the vehicle is photographed, but also the side of the vehicle and the interior of the vehicle are photographed, and the 360 degree has no dead angle. Panoramic shooting of the surrounding conditions of the vehicle to effectively improve the safety of the vehicle; and the vehicle controller synthesizes the panoramic video by splicing the video to be spliced corresponding to the original video collected by each camera, and using the vehicle display to view the panorama The display of the video effectively improves the display effect of the video, and enables the driver to more intuitively observe the surroundings of the vehicle from the display content, thereby improving the driver's interactive experience. In the vehicle panoramic video display system, a closed loop is formed between the vehicle video capture device, the vehicle controller and the vehicle display, which can provide useful support for assisted driving. Moreover, migrating the video splicing function from the vehicle video capture device to the vehicle controller not only saves the purchase cost of the display processing device, but also contributes to the miniaturization of the vehicle video capture device, reduces the baseline convergence between the cameras, and reduces the parallax. Improve the quality of the stitching.

Based on the vehicle panoramic video display system shown in FIG. 1 , as shown in FIG. 10 , it is a schematic structural diagram of another vehicle panoramic video display system, which includes the following devices: an in-vehicle video collector 1010 and an on-board controller. 1020. On-board display 1030 and body sensor 1040.

The interaction between the in-vehicle video capture device 1010, the in-vehicle controller 1020, and the in-vehicle display 1030 is as shown in FIG. 2, and details are not described herein again.

The body sensor 1040 is configured to collect steering wheel rotation information and transmit steering wheel rotation information to the onboard controller 1020.

The vehicle controller 1020 can also be configured to receive the steering wheel rotation information sent by the vehicle body sensor 1040; according to the steering wheel rotation information, obtain the rotation angle information of the panoramic video by the rotation angle conversion; rotate the information according to the rotation angle of the viewing angle, and obtain the rotation conversion of the panoramic video. The converted panoramic video; the converted panoramic video is transmitted to the in-vehicle display 1030.

The in-vehicle display 1030 is further configured to receive the converted panoramic video sent by the on-board controller 1020; and display the converted panoramic video through a preset display policy.

When the steering wheel of the vehicle rotates, the driver's field of vision changes, and the direction of change is related to the direction of rotation of the steering wheel. After the body sensor collects the steering wheel rotation information, the steering wheel rotation information is sent to the vehicle controller, and the vehicle controller recalculates the angle of view. To generate a panoramic video of the corresponding perspective, as shown in FIG. 11, the panoramic video after changing the viewing angle is displayed by the in-vehicle display.

As shown in Fig. 12a and Fig. 12b, when the steering wheel angle is changed from β ₀ as shown in Fig. 12a to β ₁ as shown in Fig. 12b, the deflection angle of the panoramic video is changed from β ₀ to β ₁ , and the view appears horizontally. Rotate.

In this embodiment, the vehicle panoramic video display system further includes a vehicle body sensor for collecting steering wheel rotation information. When the vehicle body sensor rotates, the vehicle body sensor transmits the panoramic video after the angle of view rotation to the vehicle display, and the display of the vehicle display is performed. Displaying the scene after the vehicle is deflected facilitates the driver's intuitive observation of the driving environment and has a high intelligent assisted driving effect.

The embodiment of the present invention further provides an onboard controller, as shown in FIG. 13, which may include a processor 1301 and a memory 1302. The memory 1302 is configured to store a computer program. The processor 1301 is configured to execute the The following steps are implemented in the program stored on the memory 1302.

Acquiring the original video collected by each camera in the vehicle video capture device; searching for a pre-established panoramic synthesis mapping table, respectively generating a splicing relationship between the video to be spliced corresponding to each original video and the video to be spliced, wherein the panoramic composite mapping table includes Corresponding relationship between the pixel coordinates of the video to be stitched and the pixel coordinates of the original video; according to the stitching relationship, the video to be stitched is stitched to generate a panoramic video; and the panoramic video is sent to the in-vehicle display.

In a possible implementation manner of the present application, the processor 1301 may further implement the following steps: acquiring a sample image collected by each camera in the in-vehicle video collector, and each camera internal reference of the vehicle video collector calibration And each camera external parameter; performing spherical projection on each sample image according to each camera internal parameter to generate a spherical projection image corresponding to each sample image; and obtaining a spherical projection image according to each camera external parameter a relative positional relationship; transforming and cropping the spherical projection images according to the relative positional relationship to obtain a panoramic composition mapping table.

In a possible implementation manner of the present application, the processor 1301 may further implement the following steps: acquiring a sample image collected by each camera in the in-vehicle video collector, and each camera internal reference of the vehicle video collector calibration And establishing a first panoramic mapping table, where the content of the first panoramic mapping table is empty; according to the external parameters of the cameras, obtaining a position between each preset area in the first panoramic mapping table And performing inverse transformation on the preset regions according to the positional relationship to obtain an inverse transform region of each preset region; performing spherical projection on each sample image according to each camera internal parameter, generating the A spherical projection image corresponding to each sample image; and a panoramic composition mapping table is determined according to a correspondence relationship between each spherical projection image and each inverse transformation region.

In a possible implementation manner of the present application, the processor 1301 may further implement the following steps: acquiring an external reference calibration image collected by each camera in the vehicle video capture device, and a calibration camera in each external reference calibration image. External parameter; detecting an outer edge of the effective region of the external reference calibration image; fitting the outer edge of the effective region of the external reference calibration image by curve fitting to obtain an effective region of the external reference calibration image a contour; determining, according to an outer contour of the effective region of the external reference calibration image, a camera internal reference for acquiring the calibration image of each external reference; and performing spherical projection on each of the external reference calibration images according to each camera internal reference, generating a a spherical projection image corresponding to each external reference calibration image; obtaining a relative positional relationship between each spherical projection image according to each camera external parameter; and transforming and cutting the spherical projection images according to the relative positional relationship Panorama synthesis map.

In a possible implementation manner of the present application, the processor 1301 may further implement the following steps: acquiring an external reference calibration image collected by each camera in the vehicle video capture device, and a calibration camera in each external reference calibration image. External parameter; detecting an outer edge of the effective region of the external reference calibration image; fitting the outer edge of the effective region of the external reference calibration image by curve fitting to obtain an effective region of the external reference calibration image a contour; determining, according to an outer contour of the effective area of the external reference calibration image, a camera internal parameter for collecting the external reference calibration image; establishing a first panoramic mapping table, wherein the content of the first panoramic mapping table is empty; Each camera external parameter obtains a positional relationship between each preset area in the first panoramic mapping table; and inversely transforms each preset area according to the positional relationship to obtain an inverse of each preset area Converting a region; performing spherical projection on each of the external reference calibration images according to each camera internal parameter, and generating a spherical projection image corresponding to each external reference calibration image; The panoramic composition map is determined according to the correspondence between each spherical projection image and each inverse transform region.

In a possible implementation manner of the present application, the panoramic synthesis mapping table includes a mapping table of each camera in the in-vehicle video collector, and there is an overlapping area between the two mapping tables.

In a possible implementation manner of the present application, the processor 1301 is configured to implement the pre-established panoramic synthesis mapping table, and respectively generate a splicing relationship between the video to be spliced corresponding to each original video and each video to be spliced. In the step, the following steps may be implemented: searching for a pre-established panoramic synthesis mapping table, respectively generating a splicing relationship between the video frames to be spliced corresponding to each video frame in each original video and the video frames to be spliced; The generated video frames to be spliced are generated by using a preset video generation technology to generate a video to be spliced.

When the processor 1301 performs the step of splicing the video to be spliced according to the splicing relationship to generate a panoramic video, the following steps may be specifically implemented: according to the splicing relationship between the video frames to be spliced, Each of the corresponding video frames to be spliced in each video to be spliced is spliced to obtain a plurality of panoramic images; and based on each panoramic image, a panoramic video is generated by using a preset video generation technology.

In a possible implementation manner of the present application, when the processor 1301 implements the step of splicing each video to be spliced according to the splicing relationship to generate a panoramic video, the following steps may be specifically implemented: The splicing relationship is performed, and the video to be spliced is spliced to generate a two-dimensional video; according to the preset three-dimensional projection mode, the two-dimensional video is three-dimensionally projected to obtain a panoramic video.

In a possible implementation manner of the present application, the processor 1301 may further implement the following steps: acquiring vehicle state information; determining, according to the vehicle state information, pending information of the panoramic video; according to the to-be-processed Information, processing the panoramic video to obtain a processed panoramic video.

When the processor 1301 implements the step of transmitting the panoramic video to the in-vehicle display, specifically, the processor 1301 may be configured to: send the processed panoramic video to the in-vehicle display.

In a possible implementation manner of the present application, the vehicle status information includes steering wheel rotation information.

When the step of acquiring the vehicle state information is implemented, the processor 1301 may specifically implement the following steps: acquiring steering wheel rotation information collected by the vehicle body sensor.

When the processor 1301 determines the step of determining the to-be-processed information of the panoramic video according to the vehicle state information, the processor 1301 may specifically implement the following steps: according to the steering wheel rotation information, the rotation angle is converted to obtain the The perspective of the panoramic video rotates the information.

When the processor 1301 performs the step of processing the panoramic video according to the to-be-processed information to obtain the processed panoramic video, specifically, the following steps may be implemented: rotating the information according to the viewing angle, The panoramic video is rotated and transformed to obtain a transformed panoramic video.

The processor 1301 may specifically implement the step of transmitting the converted panoramic video to the in-vehicle display when the step of transmitting the processed panoramic video to the in-vehicle display is implemented.

In a possible implementation manner of the present application, the vehicle state information includes driver posture change information.

When the step of acquiring the vehicle state information is implemented, the processor 1301 may specifically implement the following steps: acquiring driver posture change information recognized by the in-vehicle video collector.

The processor 1301, when implementing the step of determining the to-be-processed information of the panoramic video according to the vehicle state information, may specifically implement the following steps: determining, according to the driver attitude change information, a viewpoint transformation relationship The viewpoint conversion parameter of the panoramic video.

When the processor 1301 performs the step of processing the panoramic video according to the to-be-processed information to obtain the processed panoramic video, specifically, the following steps may be implemented: according to the viewpoint transformation parameter, The panoramic video is updated by the viewpoint to obtain a panoramic video after the viewpoint is updated.

When the processor 1301 implements the step of transmitting the processed panoramic video to the in-vehicle display, specifically, the processor 1301 may be configured to: send the panoramic video after the view update to the on-board display.

In a possible implementation manner of the present application, the vehicle status information includes: an off-vehicle obstacle target information.

When the step of acquiring the vehicle state information is implemented, the processor 1301 may specifically implement the following steps: acquiring the vehicle exterior obstacle target information identified by the vehicle video capture device.

The processor 1301, when implementing the step of determining the to-be-processed information of the panoramic video according to the vehicle state information, may specifically implement the following steps: determining an obstacle outside the vehicle according to the target information of the vehicle exterior obstacle Category information, and location information of the outer obstacle in the panoramic video.

When the processor 1301 performs the step of processing the panoramic video according to the to-be-processed information to obtain the processed panoramic video, the following steps may be specifically implemented: the category information and the location information. Superimposed on the panoramic video to obtain a panoramic video after superimposing information.

When the processor 1301 is configured to send the processed panoramic video to the in-vehicle display, the processor 1301 may specifically implement the step of: transmitting the panoramic video after the superimposed information to the in-vehicle display.

In a possible implementation manner of the application, the in-vehicle video collector is a fisheye device.

When the processor 1301 implements the step of acquiring the original video collected by each camera in the vehicle video capture device, the following steps may be specifically implemented: acquiring the original video separately collected by the two fisheye cameras in the fisheye device.

Data transmission between the memory 1302 and the processor 1301 may be performed by means of a wired connection or a wireless connection, and the computer device may communicate with other devices through a wired communication interface or a wireless communication interface. It should be noted that only an example of transmitting data between the processor 1301 and the memory 1302 via the bus is shown in FIG. 13, and is not limited to a specific transmission mode.

The above memory may include a RAM (Random Access Memory), and may also include NVM (Non-Volatile Memory), such as at least one disk storage. In a possible implementation manner of the present application, the memory may also be at least one storage device located away from the processor.

The processor may be a general-purpose processor, including a CPU (Central Processing Unit), an NP (Network Processor), or the like; or a DSP (Digital Signal Processor) or an ASIC (Application) Specific Integrated Circuit, FPGA (Field-Programmable Gate Array) or other programmable logic device, discrete gate or transistor logic device, discrete hardware components.

In addition, the embodiment of the present application provides a computer readable storage medium, where the computer readable storage medium stores a computer program, a computer program, corresponding to the vehicle-mounted panoramic video display method applied to the vehicle-mounted controller provided by the above embodiment. All steps of the in-vehicle panoramic video display method applied to the on-vehicle controller of the embodiment of the present application are implemented when executed by the processor.

Through the above-mentioned on-board controller, it can be realized that since the total coverage area of all the cameras in the vehicle video capture device is greater than or equal to 360 degrees, during the running of the vehicle, not only the road ahead of the vehicle is photographed, but also the side of the vehicle and The inside of the vehicle is photographed, and the surrounding conditions of the vehicle are panoramicly photographed without a dead angle at 360 degrees, thereby effectively improving the safety of the vehicle traveling; and the vehicle controller splicing and synthesizing the video to be spliced corresponding to the original video collected by each camera The panoramic video is then displayed by the on-board display, which effectively improves the display effect of the video, enables the driver to more intuitively observe the surroundings of the vehicle from the display content, and improves the driver's interactive experience.

The computer readable storage medium stores an application program that executes the on-vehicle panoramic video display method applied to the on-vehicle controller provided by the embodiment of the present application at runtime, and thus can realize: the total coverage area of all the cameras in the vehicle video capture device It is greater than or equal to 360 degrees. During the running of the vehicle, not only the road ahead of the vehicle is photographed, but also the side of the vehicle and the inside of the vehicle are photographed. 360 degrees without panoramic view, the panoramic view of the surrounding conditions of the vehicle is effectively improved. The safety of the vehicle travels; and the vehicle controller synthesizes the panoramic video by splicing the video to be spliced corresponding to the original video collected by each camera, and then displaying the panoramic video by using the vehicle display, thereby effectively improving the display effect of the video. It can make the driver more intuitively observe the situation around the vehicle from the display content, and improve the driver's interactive experience.

In addition, corresponding to the vehicular panoramic video display method provided by the foregoing embodiment, the embodiment of the present application provides an application program for performing at runtime: the vehicle panoramic video image applied to the vehicle controller provided by the embodiment of the present application Display method.

In this embodiment, the application performs the on-vehicle panoramic video display method applied to the on-board controller provided by the embodiment of the present application during operation, so that the total coverage area of all the cameras in the on-board video collector is greater than or equal to 360. In the process of driving the vehicle, not only the road ahead of the vehicle is photographed, but also the side of the vehicle and the inside of the vehicle are photographed, and 360-degree panoramic view of the surroundings of the vehicle is performed without dead angle, thereby effectively improving the safety of the vehicle. And the vehicle controller synthesizes the panoramic video by splicing the video to be spliced corresponding to the original video collected by each camera, and then displaying the panoramic video by using the vehicle display, thereby effectively improving the display effect of the video and enabling driving The staff intuitively observed the surrounding conditions of the vehicle from the display content and improved the driver's interactive experience.

For the on-board controller, the computer readable storage medium, and the application embodiment, since the method content involved is basically similar to the foregoing method embodiment, the description is relatively simple, and the relevant parts refer to the description of the method embodiment. Just fine.

It should be noted that, in this context, relational terms such as first and second are used merely to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply such entities or operations. There is any such actual relationship or order between them. Furthermore, the term "comprises" or "comprises" or "comprises" or any other variations thereof is intended to encompass a non-exclusive inclusion, such that a process, method, article, or device that comprises a plurality of elements includes not only those elements but also Other elements, or elements that are inherent to such a process, method, item, or device. An element that is defined by the phrase "comprising a ..." does not exclude the presence of additional equivalent elements in the process, method, item, or device that comprises the element.

The various embodiments in the present specification are described in a related manner, and the same or similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from the other embodiments. In particular, for the device, the on-board controller, the computer readable storage medium, and the application embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and the relevant portions can be referred to the description of the method embodiment.

The above is only the preferred embodiment of the present application, and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc., which are made within the spirit and principles of the present application, should be included in the present application. Within the scope of protection.

Claims (21)

1. A vehicle panoramic video display system, characterized in that the system comprises an in-vehicle video collector, a vehicle controller and an on-board display;

   The vehicle video capture device includes a plurality of cameras, and each camera captures original video in the coverage area, and the total coverage area of all the cameras in the vehicle video collector is greater than or equal to 360 degrees; the original video collected by each camera is sent to The vehicle controller;

   The vehicle controller is configured to receive the original video collected by each camera sent by the vehicle video capture device; and search for a pre-established panoramic synthesis mapping table, respectively generate a video to be spliced corresponding to each original video and a video to be spliced a splicing relationship, the panorama synthesis mapping table includes a correspondence between pixel coordinates of the video to be spliced and pixel coordinates of the original video; and splicing the video to be spliced according to the splicing relationship to generate a panoramic video; Sended to the on-board display;

   The on-board display is configured to receive the panoramic video sent by the onboard controller; and display the panoramic video by using a preset display policy.
2. The system of claim 1 wherein said in-vehicle video capture device is a binocular device.
3. The system according to claim 1, wherein the original video captured by each camera is transmitted between the in-vehicle video collector and the on-board controller through an information interaction module;

   The panoramic video is transmitted between the onboard controller and the onboard display through an information interaction module.
4. The system of claim 1 wherein said system further comprises: a body sensor;

   The vehicle body sensor is configured to collect steering wheel rotation information; and send the steering wheel rotation information to the vehicle controller;

   The vehicle controller is further configured to receive the steering wheel rotation information sent by the vehicle body sensor; according to the steering wheel rotation information, obtain the perspective rotation information of the panoramic video by rotating angle conversion; and rotate the information according to the perspective Rotating and transforming the panoramic video to obtain a transformed panoramic video; and transmitting the converted panoramic video to the on-board display;

   The on-board display is further configured to receive the converted panoramic video sent by the onboard controller; and display the converted panoramic video by using a preset display policy.
5. The system of claim 1 wherein:

   The vehicle video capture device is further configured to identify driver attitude change information; and send the driver posture change information to the onboard controller;

   The vehicle controller is further configured to receive the driver attitude change information sent by the vehicle video capture device, and determine a viewpoint transformation parameter of the panoramic video by using a viewpoint transformation relationship according to the driver posture change information; And performing a viewpoint update on the panoramic video according to the viewpoint conversion parameter to obtain a panoramic video after the viewpoint update; and transmitting the panoramic video after the viewpoint update to the on-board display;

   The on-board display is further configured to receive the panoramic video after the view update sent by the onboard controller, and display the updated panoramic video of the view by using a preset display policy.
6. The system of claim 1 wherein:

   The in-vehicle video collector is further configured to identify an off-vehicle obstacle target information; and send the off-vehicle obstacle target information to the on-board controller;

   The vehicle controller is further configured to receive the vehicle exterior obstacle target information sent by the vehicle video capture device; determine, according to the vehicle exterior obstacle object information, category information of the vehicle exterior obstacle, and the vehicle Location information of the external obstacle in the panoramic video; superimposing the category information and the location information on the panoramic video to obtain a panoramic video after superimposing information; sending the panoramic video after the superimposed information to The vehicle display;

   The in-vehicle display is further configured to receive the panoramic video after the superimposed information sent by the on-board controller; and display the panoramic video after the superimposed information by using a preset display strategy.
7. The invention relates to a vehicle panoramic video display method, which is characterized in that it is applied to an on-board controller, and the method comprises:

   Obtaining the original video collected by each camera in the car video collector;

   Searching for a pre-established panoramic synthesis mapping table, respectively, to generate a splicing relationship between the video to be spliced corresponding to each original video and each video to be spliced, wherein the panoramic compositing mapping table includes pixel coordinates of the video to be spliced and pixel coordinates of the original video. Correspondence relationship

   According to the splicing relationship, the video to be spliced is spliced to generate a panoramic video;

   The panoramic video is sent to the onboard display.
8. The method according to claim 7, wherein the manner in which the panoramic synthesis mapping table is established includes:

   Acquiring a sample image collected by each camera in the vehicle video capture device, and participating in each camera of the camera calibration by the on-board video collector;

   Performing a spherical projection on each sample image according to each of the camera internal parameters to generate a spherical projection image corresponding to each sample image;

   Obtaining a relative positional relationship between the spherical projection images according to the external parameters of the cameras;

   The respective spherical projection images are transformed and cropped according to the relative positional relationship to obtain a panoramic composition mapping table.
9. The method according to claim 7, wherein the manner in which the panoramic synthesis mapping table is established includes:

   Acquiring a sample image collected by each camera in the vehicle video capture device, and participating in each camera of the camera calibration by the on-board video collector;

   Establishing a first panoramic mapping table, where the content of the first panoramic mapping table is empty;

   Obtaining a positional relationship between each preset area in the first panoramic mapping table according to the external parameters of the cameras;

   Performing inverse transform on each preset area according to the positional relationship to obtain an inverse transform area of each preset area;

   Performing a spherical projection on each sample image according to each of the camera internal parameters to generate a spherical projection image corresponding to each sample image;

   A panoramic composition map is determined based on the correspondence between each spherical projection image and each inverse transform region.
10. The method according to claim 7, wherein the manner in which the panoramic synthesis mapping table is established includes:

    Obtaining an external reference calibration image acquired by each camera in the vehicle video capture device, and a camera external reference calibrated in each external reference calibration image;

    Detecting an outer edge of an effective area of the external reference calibration image;

    Using the curve fitting, fitting the outer edge of the effective region of the external reference calibration image to obtain the outer contour of the effective region of the external reference calibration image;

    Determining, according to the outer contour of the effective area of the external reference calibration image, the camera internal reference for collecting the external reference calibration image;

    Performing a spherical projection on each of the external reference calibration images according to each camera internal reference to generate a spherical projection image corresponding to each of the external reference calibration images;

    Obtaining a relative positional relationship between each spherical projection image according to each camera external parameter;

    The respective spherical projection images are transformed and cropped according to the relative positional relationship to obtain a panoramic composition mapping table.
11. The method according to claim 7, wherein the manner in which the panoramic synthesis mapping table is established includes:

    Obtaining an external reference calibration image acquired by each camera in the vehicle video capture device, and a camera external reference calibrated in each external reference calibration image;

    Detecting an outer edge of an effective area of the external reference calibration image;

    Using the curve fitting, fitting the outer edge of the effective region of the external reference calibration image to obtain the outer contour of the effective region of the external reference calibration image;

    Determining, according to the outer contour of the effective area of the external reference calibration image, the camera internal reference for collecting the external reference calibration image;

    Establishing a first panoramic mapping table, where the content of the first panoramic mapping table is empty;

    Obtaining a positional relationship between each preset area in the first panoramic mapping table according to each camera external parameter;

    Performing inverse transform on each preset area according to the positional relationship to obtain an inverse transform area of each preset area;

    Performing a spherical projection on each of the external reference calibration images according to each camera internal reference to generate a spherical projection image corresponding to each of the external reference calibration images;

    A panoramic synthetic map is determined based on the correspondence between each spherical projection image and each inverse transform region.
12. The method according to any one of claims 8 to 11, wherein the panoramic composition mapping table comprises a mapping table of each camera in the in-vehicle video collector, and there is an overlapping area between the two mapping tables.
13. The method according to any one of claims 8 to 11, wherein the searching for a pre-established panoramic synthesis mapping table respectively generates a splicing relationship between the video to be spliced corresponding to each original video and the video to be spliced, including:

    Searching for a pre-established panoramic synthesis mapping table, respectively generating a splicing relationship between the video frames to be spliced corresponding to each video frame in each original video and the video frames to be spliced;

    The video to be spliced is generated by using a preset video generation technology for each video frame to be spliced generated by the same original video;

    According to the splicing relationship, the video to be spliced is spliced to generate a panoramic video, including:

    And splicing, according to the splicing relationship between the video frames to be spliced, all the video frames to be spliced in the video to be spliced, to obtain a plurality of panoramic images;

    Based on each panoramic image, a panoramic video is generated using a preset video generation technique.
14. The method according to claim 7, wherein the splicing the video to be spliced according to the splicing relationship to generate a panoramic video comprises:

    According to the splicing relationship, the video to be spliced is spliced to generate a two-dimensional video;

    The two-dimensional video is three-dimensionally projected according to a preset three-dimensional projection manner to obtain a panoramic video.
15. The method according to claim 7, wherein after the splicing of the video to be spliced according to the splicing relationship to generate a panoramic video, the method further includes:

    Obtaining vehicle status information;

    Determining, to be processed, the information of the panoramic video according to the vehicle status information;

    And processing the panoramic video according to the to-be-processed information to obtain a processed panoramic video;

    The transmitting the panoramic video to the on-board display comprises:

    The processed panoramic video is transmitted to the in-vehicle display.
16. The method of claim 15 wherein said vehicle status information comprises steering wheel rotation information;

    The obtaining vehicle status information includes:

    Obtain steering wheel rotation information collected by the body sensor;

    Determining the to-be-processed information of the panoramic video according to the vehicle status information, including:

    Obtaining rotation angle information of the panoramic video according to the rotation information of the steering wheel;

    The processing, according to the to-be-processed information, processing the panoramic video to obtain a processed panoramic video, including:

    Rotating the panoramic video according to the rotation information of the viewing angle to obtain a transformed panoramic video;

    The transmitting the processed panoramic video to the on-board display comprises:

    The transformed panoramic video is transmitted to the in-vehicle display.
17. The method of claim 15 wherein said vehicle status information comprises driver attitude change information;

    The obtaining vehicle status information includes:

    Obtaining driver posture change information recognized by the in-vehicle video collector;

    Determining the to-be-processed information of the panoramic video according to the vehicle status information, including:

    Determining, according to the driver posture change information, a viewpoint conversion parameter of the panoramic video by a viewpoint conversion relationship;

    The processing, according to the to-be-processed information, processing the panoramic video to obtain a processed panoramic video, including:

    Performing a viewpoint update on the panoramic video according to the viewpoint transformation parameter to obtain a panoramic video after the viewpoint update;

    The transmitting the processed panoramic video to the on-board display comprises:

    The panoramic video after the viewpoint update is sent to the in-vehicle display.
18. The method according to claim 15, wherein the vehicle status information comprises: an off-vehicle obstacle target information;

    The obtaining vehicle status information includes:

    Acquiring the target information of the vehicle exterior obstacle identified by the vehicle video capture device;

    Determining the to-be-processed information of the panoramic video according to the vehicle status information, including:

    Determining, according to the vehicle exterior obstacle target information, category information of an obstacle outside the vehicle, and location information of the exterior obstacle in the panoramic video;

    The processing, according to the to-be-processed information, processing the panoramic video to obtain a processed panoramic video, including:

    Superimposing the category information and the location information on the panoramic video to obtain a panoramic video after superimposing information;

    The transmitting the processed panoramic video to the on-board display comprises:

    The panoramic video after the superimposed information is sent to the on-board display.
19. The method according to claim 7, wherein the in-vehicle video collector is a fisheye device;

    The acquiring the original video collected by each camera in the vehicle video capture device includes:

    Obtain the original video captured by the two fisheye cameras in the fisheye device.
20. An on-board controller, characterized in that the on-board controller includes a processor and a memory;

    The memory is configured to store a computer program;

    The processor, when executing the program stored on the memory, implements the following steps:

    Obtaining the original video collected by each camera in the car video collector;

    Searching for a pre-established panoramic synthesis mapping table, respectively, to generate a splicing relationship between the video to be spliced corresponding to each original video and each video to be spliced, wherein the panoramic compositing mapping table includes pixel coordinates of the video to be spliced and pixel coordinates of the original video. Correspondence relationship

    According to the splicing relationship, the video to be spliced is spliced to generate a panoramic video;

    The panoramic video is sent to the onboard display.
21. A computer readable storage medium, wherein the computer readable storage medium stores a computer program, and when the computer program is executed by the processor, the following steps are implemented:

    Obtaining the original video collected by each camera in the car video collector;

    Searching for a pre-established panoramic synthesis mapping table, respectively, to generate a splicing relationship between the video to be spliced corresponding to each original video and each video to be spliced, wherein the panoramic compositing mapping table includes pixel coordinates of the video to be spliced and pixel coordinates of the original video. Correspondence relationship

    According to the splicing relationship, the video to be spliced is spliced to generate a panoramic video;

    The panoramic video is sent to the onboard display.

Images