WO2023024421A1

WO2023024421A1 - Method and system for splicing multiple channels of images, and readable storage medium and unmanned vehicle

Info

Publication number: WO2023024421A1
Application number: PCT/CN2022/071845
Authority: WO
Inventors: 柴双林
Original assignee: 北京三快在线科技有限公司
Priority date: 2021-08-25
Filing date: 2022-01-13
Publication date: 2023-03-02
Also published as: CN113781302B; CN113781302A

Abstract

The present disclosure relates to a method and system for splicing multiple channels of images, and a readable storage medium and an unmanned vehicle. The method is applied to a field programmable gate array platform, and the method comprises: for each of multiple channels of images to be spliced, executing the following processing to obtain multiple rows of row buffer data of each of said multiple channels of images: repeatedly executing the process of acquiring row data, and on the basis of a preset mapping table, writing pixel data comprised in the row data into a row buffer area corresponding to said channel of image to obtain row buffer data until a first preset condition is met, so as to obtain the multiple rows of row buffer data of said channel of image; writing the multiple rows of row buffer data of each of said multiple channels of images into a frame buffer area, and executing filling processing on the multiple rows of row buffer data of each said channel of image, so as to obtain spliced frame buffer data; and preprocessing the spliced frame buffer data, so as to obtain a spliced image.

Description

Multi-channel image stitching method, system, readable storage medium, and unmanned vehicle

Cross References to Related Applications

This disclosure claims the priority of the Chinese patent application with application number 202110984382.6 and titled "multi-channel image stitching method, system, readable storage medium, and unmanned vehicle" submitted to the China Patent Office on August 25, 2021. The entire contents are incorporated by reference in this disclosure.

technical field

The present disclosure relates to the technical field of vehicles, and in particular, relates to a multi-channel image stitching method and system, a readable storage medium, and an unmanned vehicle.

Background technique

At present, in the field of unmanned vehicles, the vehicle surround view system is widely used in automatic driving and remote control driving to assist the driver in driving safely. For the automatic driving system, the surround-view perspective can improve the full-coverage obstacle detection capability. For remote remote control driving, the surround-view perspective can maximize the assistance of remote control driving to remove people from the scene and minimize the ratio of people to vehicles.

In the related technologies, most of the vehicle surround view systems use pure software algorithms to realize multi-channel image stitching, and there is a bottleneck in stitching performance. It is impossible to achieve high frame rate and high-channel image stitching, and the stitched images are distorted and the stitching delay is high. To this end, the present disclosure proposes a multi-channel image stitching method, system, readable storage medium, and unmanned vehicle.

Contents of the invention

The purpose of the present disclosure is to provide a multi-channel image stitching method, system, readable storage medium, and unmanned vehicle, so as to at least partially solve the above-mentioned problems in the related art.

In order to achieve the above purpose, according to the first aspect of the embodiments of the present disclosure, a multi-channel image stitching method is provided, which is characterized in that it is applied to a field programmable gate array platform, and the method includes:

For each path in the multiple paths of images to be spliced, perform the following processing to obtain multiple rows of line buffer data for each path in the multiple paths of images to be spliced:

Repeatedly performing the process of obtaining row data, and writing pixel data included in the row data into the row buffer corresponding to the image to be spliced based on the preset mapping table, to obtain the row buffer data, until the first preset condition is met, Obtaining the multiple lines of line buffer data of the path of images to be spliced;

Wherein, the preset mapping table is obtained based on the mapping relationship of the pixel data between the historical image to be stitched and the historical stitched image;

Writing the multiple lines of line buffer data of each of the multiple lines of images to be spliced into a frame buffer, and performing filling processing on the multiple lines of line buffer data of each of the multiple lines of images to be spliced to obtain a spliced frame buffer data;

Perform preprocessing on the spliced frame buffer data to obtain a spliced image.

Optionally, writing the multiple lines of line buffer data of each of the multiple lines of images to be spliced into a frame buffer, and performing filling of the multiple lines of line buffer data of each of the multiple lines of images to be spliced Processing to get spliced framebuffer data, including:

Write the multiple lines of line buffer data of each line of the multiple lines of images to be spliced into the frame buffer in a line writing unit, and write the line buffer data to the frame buffer during the writing process of each line of the line buffer data The filling process is performed on the line buffer data to obtain the spliced frame buffer data.

Optionally, writing the multiple lines of line buffer data of each line of the multiple lines of images to be spliced into the frame buffer in a row writing unit includes:

Write the multiple rows of line buffer data of each channel in the multiple channels of images to be spliced into the frame buffer based on the writing unit of the row and based on a polling arbitration mode.

Optionally, the obtaining row data includes:

Decoding the preset interface signal output by the target camera device received based on the preset interface to obtain decoded information; wherein, the decoded information includes at least line and field synchronization signals and pixel data of the image to be spliced; the target camera device It is the camera equipment of the image to be spliced;

After the row data in the initial format is obtained from the pixel data of the image to be spliced based on the row and field synchronous signals, format conversion is performed on the row data to obtain row data in a target format.

Optionally, for each of the multiple paths of images to be spliced, the following processing is performed to obtain multiple rows of line buffer data for each of the multiple paths of images to be spliced, including:

For each path of the multiple paths of images to be spliced, the processing is executed in parallel to obtain the multiple lines of line buffer data of each path of the multiple paths of images to be spliced.

Optionally, performing preprocessing on the spliced frame buffer data to obtain a spliced image includes:

After writing the line buffer data of a preset line each time, performing encoding processing on the line buffer data of the preset line until a second preset condition is met to obtain an encoded image;

The coded image is decoded to obtain a spliced image.

Optionally, after each writing of the line buffer data of a preset line, encoding processing is performed on the line buffer data of the preset line until a second preset condition is met to obtain an encoded image;

Using the built-in encoder of the field programmable gate array system, after writing the row buffer data of the preset row each time, the encoding process is performed on the row buffer data of the preset row until the row buffer data of the preset row is satisfied. The second preset condition is to obtain the coded image.

According to the second aspect of the embodiments of the present disclosure, a multi-channel image stitching system is provided, which is applied to a field programmable gate array platform, and the system includes:

The multi-line buffer module is used to repeatedly execute the acquisition of line data for each of the multiple lines of images to be spliced, and based on the preset mapping table, write the pixel data included in the line data into the line corresponding to the line of images to be spliced Buffering, the process of obtaining line buffer data until the first preset condition is met, so as to obtain the multiple lines of line buffer data of the image to be spliced; wherein, the preset mapping table is based on historical images to be spliced and The mapping relationship of the pixel data between historical mosaic images is obtained;

A frame buffer writing module, configured to write the multiple lines of line buffer data of each of the multiple channels of images to be spliced into a frame buffer, and execute Filling processing to obtain spliced frame buffer data;

The splicing processing module is configured to perform preprocessing on the spliced frame buffer data to obtain a spliced image.

Optionally, the frame buffer writing module includes:

The parallel filling sub-module is used to write the multi-line line buffer data of each line in the multiple lines of images to be spliced into the frame buffer in units of line writing, and write the line buffer data of each line into the frame buffer. During the writing process, the filling process is performed on the line buffer data to obtain the spliced frame buffer data.

Optionally, the parallel filling submodule includes:

The arbitration writing subunit is used to write the multiple rows of line buffer data of each channel in the multiple channels of images to be spliced into the frame buffer based on the polling arbitration mode in the writing unit of the behavior .

Optionally, the multiline buffer module includes:

The decoding sub-module is used to decode the preset interface signal output by the target camera device received based on the preset interface to obtain decoding information; wherein, the decoding information includes at least line and field synchronization signals and pixel data of the image to be spliced ; The target imaging device is the imaging device of the image to be stitched;

A format conversion sub-module, configured to convert the format of the row data to obtain the row data of the target format after obtaining the row data in the initial format from the pixel data of the image to be spliced based on the row and field synchronization signals data.

Optionally, the multiline buffer module includes:

The parallel row buffering submodule is configured to execute the processing in parallel for each of the multiple paths of images to be spliced, so as to obtain the multiple rows of row buffer data for each path of the multiple paths of images to be spliced.

Optionally, the splicing processing module includes:

The encoding submodule is configured to perform encoding processing on the line buffer data of the preset line after writing the line buffer data of the preset line each time, until the second preset condition is met, and obtain an encoded image;

The decoding submodule is used to decode the coded image to obtain a spliced image.

Optionally, the encoding submodule includes:

The encoding subunit is used to use the built-in encoder of the field programmable gate array system to perform the following operation on the line buffer data of the preset line after writing the line buffer data of the preset line each time. The encoding process is performed until the second preset condition is satisfied, and the encoded image is obtained.

According to a third aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium, on which a computer program is stored, and when the program is executed by a processor, the steps of any one of the methods described in the first aspect are implemented.

According to a fourth aspect of the embodiments of the present disclosure, there is provided an unmanned vehicle, including the multi-channel image stitching system described in the second aspect.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a computing processing device, including: a memory, in which computer-readable codes are stored; and one or more processors, when the computer-readable codes are stored by the one or more When executed by two processors, the computing processing device executes the multi-channel image stitching method as described in the first aspect above.

According to a sixth aspect of the embodiments of the present disclosure, there is provided a computer program, including computer readable code, which, when the computer readable code is run on a computing processing device, causes the computing processing device to execute the computer program according to the above first aspect. The multi-channel image stitching method described above.

Through the above technical solution, the pixel data of the row data is written into the row buffer based on the preset mapping table, and the pixels included in the row data can be accurately written to the corresponding mapping position of the row buffer. Avoiding the writing error caused by de-distortion through the fitting algorithm, and then solving the problem of image distortion and nonlinear distortion caused by the fitting algorithm to achieve image stitching, and can realize seamless alignment on the stitching hypotenuse; and the multi-channel image stitching method disclosed in the present disclosure Realized by the field programmable gate array platform, it is hardware improvement, the algorithm is simple, and it occupies less logic resources.

Other features and advantages of the present disclosure will be described in detail in the detailed description that follows.

The above description is only an overview of the technical solution of the present disclosure. In order to better understand the technical means of the present disclosure, it can be implemented according to the contents of the description, and in order to make the above and other purposes, features and advantages of the present disclosure more obvious and understandable , the specific embodiments of the present disclosure are enumerated below.

Description of drawings

In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or related technologies, the following will briefly introduce the drawings that need to be used in the descriptions of the embodiments or related technologies. Obviously, the drawings in the following description are the For some disclosed embodiments, those skilled in the art can also obtain other drawings based on these drawings without any creative work.

The accompanying drawings are used to provide a further understanding of the present disclosure, and constitute a part of the description, together with the following specific embodiments, are used to explain the present disclosure, but do not constitute a limitation to the present disclosure. In the attached picture:

FIG. 1 is a flow chart of a multi-channel image stitching method shown in an exemplary embodiment of the present disclosure;

Fig. 2 is an exemplary schematic diagram of a process of writing pixel data of row-by-row data shown in an exemplary embodiment of the present disclosure;

Fig. 3 is a flowchart of obtaining row data shown in an exemplary embodiment of the present disclosure;

Fig. 4 is a block diagram of a multi-channel image stitching system shown in an exemplary embodiment of the present disclosure;

Figure 5 schematically illustrates a block diagram of a computing processing device for performing a method according to the present disclosure; and

Fig. 6 schematically shows a storage unit for holding or carrying program codes implementing the method according to the present disclosure.

specific embodiment

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments It is a part of the embodiments of the present disclosure, but not all of them. Based on the embodiments in the present disclosure, all other embodiments obtained by persons of ordinary skill in the art without creative efforts fall within the protection scope of the present disclosure.

Specific embodiments of the present disclosure will be described in detail below in conjunction with the accompanying drawings. It should be understood that the specific embodiments described here are only used to illustrate and explain the present disclosure, and are not intended to limit the present disclosure.

Before introducing the multi-channel image stitching method, system, readable storage medium, and unmanned vehicle of the present disclosure, the application scenarios of the present disclosure are firstly introduced. The various embodiments provided in the present disclosure can be used in a surround-view stitching scene of multiple images collected by multiple cameras of an unmanned vehicle. In some embodiments, unmanned vehicles may at least include self-driving vehicles and remote-controlled vehicles. It is worth noting that the various embodiments provided in the present disclosure can also be used in other scenarios where surround-view stitched images need to be obtained. For example, a video surveillance scenario.

In related technologies, the look-around mosaic of multiple images can be realized by a pure software algorithm, but at least one of the following defects exists in this method: (1) the algorithm deployed on the field programmable gate array (Field Programmable Gate Array, FPGA) platform side It is complex and requires a lot of logic resources such as Digital Signal Processing (DSP), block RAM (BRAM, BRAM) and look-up tables (LUTs) when deployed on the FPGA platform, which is not conducive to cost reduction; (2) through the algorithm Distortion correction is completed in a combined way. After correction, the image is generally distorted and deformed, resulting in distortion and distortion of the spliced image; (3) The delay is large, and it is only suitable for low-speed parking and other application scenarios; (4) Multi-channel video data access, splicing The reading and writing of data requires a large amount of storage bandwidth; (5) There is a bottleneck in the splicing performance of pure software splicing, which cannot meet the data access bandwidth requirements of high frame rate, high number of channels, low distortion, and high viewing distance cameras.

To this end, the present disclosure provides a multi-channel image stitching method. FIG. 1 is a flow chart of a multi-channel image stitching method shown in an exemplary embodiment of the present disclosure, and the method can be used on a Field Programmable Gate Array (Field Programmable Gate Array, FPGA) platform. The FPGA platform can be a semi-custom circuit in an ASIC, a programmable logic array. In some embodiments, the FPGA platform can be deployed in the unmanned vehicle to display stitched images on the display screen of the unmanned vehicle to assist the safe driving of the unmanned vehicle. The methods include:

Step 102, for each of the multiple channels of images to be spliced, perform the following processing to obtain multiple rows of line buffer data for each of the multiple channels of images to be spliced: repeatedly execute the acquisition of row data, and based on the preset mapping table, The process of writing the pixel data included in the line data into the line buffer corresponding to the image to be spliced to obtain the line buffer data, until the first preset condition is met, and obtaining the multiple lines of the image to be spliced Row buffered data.

In some embodiments, the image to be spliced may be an image that requires surround-view stitching. Surround view splicing may refer to splicing images collected in multiple directions (ie, multi-channel images) to obtain a panoramic view image. The multiple images to be stitched can be acquired by multiple camera devices. For example, corresponding image frames are respectively extracted from video data collected by multiple cameras. In an implementable scenario, the multi-camera equipment may be the camera equipment installed on the unmanned vehicle, which is used to collect images from different directions of the vehicle.

In some embodiments, the number of multi-camera devices can be specifically set according to actual conditions, for example, four cameras, six cameras, and so on. Taking the four-way camera equipment as an example, the camera equipment can be installed at the front, rear, left, and right positions of the vehicle to collect images in four directions. In some embodiments, the type of camera device may at least include a fisheye camera and/or a wide-angle camera. It is worth noting that the respective installation positions and types of the multi-channel cameras can be flexibly set according to actual conditions, and this disclosure does not impose any limitation on this.

In some embodiments, the multiple lines of line buffer data may be buffered data obtained after all line data of the image to be spliced is written into the line buffer. In some embodiments, multiple paths of images to be spliced may correspond to different line buffers, and correspondingly, row data of each path of images to be spliced may be written into corresponding line buffers. Specifically, the FPGA platform can perform the following processing to obtain multiple lines of line buffer data for each image to be spliced: For each of the multiple lines of images to be spliced, repeatedly execute the acquisition of line data, and based on the preset mapping table, the line data includes The process of writing the pixel data of the image into the line buffer corresponding to the image to be spliced to obtain the line buffer data, until the preset conditions are met, and obtaining the multiple lines of the line buffer data of the image to be spliced.

In some embodiments, the images to be stitched may have different resolutions. For example, 1920*1080 pixels, 2560*1440 pixels. Generally, the resolution of the image to be spliced may be the same as that of the display screen used to display the spliced image. For example, the pixel resolution of a 1080P display is 1920*1080 pixels. For another example, the pixel resolution of a 2K display screen is 2560*1440 pixels. Based on the resolution of the image to be spliced, the image can be divided into multiple rows of data, and each row of data includes corresponding pixel data. For example, taking the resolution of 1920*1080 pixels as an example, the image to be stitched can be divided into 1920 lines, each line has 1080 pixels, and the 1080 pixels are the pixel data corresponding to the line data of the line.

In some embodiments, the FPGA platform can acquire row data through a preset interface, and perform format conversion on the row data. Details about obtaining row data can be found in FIG. 3 and related descriptions, and will not be repeated here.

In some embodiments, the preset mapping table may be obtained based on a mapping relationship of pixel data between historical images to be stitched and historical stitched images. Based on the historical image to be stitched and the historical stitched image, the coordinate transformation relationship between the corresponding pixels of the two can be obtained, so as to obtain the mapping relationship between the corresponding pixel points, and a preset mapping table can be constructed based on the mapping relationship. Therefore, the preset mapping table can reflect the mapping relationship between the pixel points at each position in the historical image to be spliced and the coordinate value of the pixel point at each position in the historical spliced image. For example, the preset mapping table reflects the first pixel of the first row of data in the historical image to be spliced, and there is a corresponding mapping relationship with the first pixel of the first row of data in the historical image to be spliced. Write the first pixel of the first row data of the image to the first pixel of the first row of the line buffer.

In some embodiments, the preset mapping table may be pre-stored in the memory area. For example, it is stored in a double-rate SDRAM (Double Data Rate SDRAM, DDR) parameter block. By storing the preset mapping table in the memory area, when writing the pixel data of the row data, the preset mapping table can be directly read from the memory to execute the write operation, which reduces the read and write bandwidth.

In some embodiments, the first preset condition may be that the writing of pixel data of all lines of image data to be spliced is completed, thus, multiple rows of line buffer data of the path of image to be spliced may be obtained. Referring to FIG. 2 , FIG. 2 is an exemplary schematic diagram of a process of writing pixel data of row-by-row data according to an exemplary embodiment of the present disclosure. As shown in Figure 2, assuming that the first line of data T1 in the image to be spliced contains 10 pixels, the first pixel in T1 can be mapped to the first line T' of the line buffer according to the preset mapping table For the first pixel of ₁ , map the second pixel in T1 to the second pixel of the second row T _′ of the line buffer, and map the third pixel in T1 to the line buffer The 6th pixel point of the 100th line T′ ₁₀₀ of the 100th line, in order to avoid redundant description, the mapping process of the 4th-10th pixel point in T1 is omitted. Assuming that the pixel points included in the row data of the image to be spliced can only be mapped to T′ ₁₀₀ at most, it can be seen that by performing the writing process of the pixel data of the row data in the example in Figure 2 for each row of row data of each image to be spliced , you can get multiple lines of line buffer data, that is, 100 lines of line buffer data.

In some embodiments, the de-distortion and affine transformation process of the image to be stitched may be reflected as remapping of pixel coordinate values on the image to be stitched and the stitched image. In the embodiment of this specification, a preset mapping table reflecting the mapping relationship of pixel data is pre-constructed through the historical image to be stitched and the historical stitching image, which is a point-to-point mapping relationship. Query the mapping position of the corresponding pixel in the preset mapping table for precise writing. It avoids the error caused by de-distortion through the fitting algorithm, and then solves the problem of image distortion and nonlinear distortion caused by the fitting algorithm to achieve image stitching, and can seamlessly align the stitching hypotenuse.

In some embodiments, based on the actual distortion of each camera device, a preset memory space can be reserved in the line buffer corresponding to each image to be spliced for the pixel data of the line data of each image to be spliced. write. Still taking the above example as an example, assuming that the camera device is a fisheye camera, and the actual distortion of the fisheye camera needs to write 100 lines of data, 100 lines of preset memory space can be reserved. In some embodiments, an imaging device with less actual distortion may be selected to reduce the number of times of writing to the line buffer, thereby reducing splicing delay.

In some embodiments, for each path of the multiple paths of images to be spliced, the above processing of obtaining multiple lines of line buffer data may be performed in parallel, so as to obtain multiple lines of line buffer data of each path of the multiple paths of images to be spliced. Through the parallel processing of multiple images to be stitched, the delay in the image stitching process can be reduced.

Step 104, write the multiple lines of line buffer data of each of the multiple lines of images to be spliced into the frame buffer, and perform filling processing on the multiple lines of line buffer data of each of the multiple lines of the images to be spliced, to obtain Stitching framebuffer data.

Due to the distortion of the image to be spliced, when the pixel data of the line data of the image to be spliced is written into the line buffer, the original pixel data is thinned out, and then the obtained multi-line line buffer data has missing data, that is, there are missing pixels value. In some embodiments, filling processing may be performed on multiple rows of line buffer data, so as to fill in missing data. In some embodiments, padding may include weighted reconstruction padding, eg, linear interpolation. For the filling process, reference may be made to descriptions in related technologies, which will not be described in detail in this disclosure.

In some embodiments, stitching the framebuffer data may include writing linebuffer data to the framebuffer. In some embodiments, in the process of writing the multiple lines of line buffer data of each line of multiple lines of images to be spliced into the frame buffer, the filling process can be performed on the multiple lines of line buffer data of each line of images to be spliced to obtain the splicing Framebuffer data. By performing filling processing in the process of writing to the frame buffer, it is equivalent to performing parallel execution of frame buffer writing and filling processing, thereby further reducing the delay of image splicing.

In some embodiments, the FPGA platform may write multiple lines of line buffer data of each line of the multiple lines of images to be spliced into the frame buffer in a line write unit. For example, taking the first image to be spliced as an example, one line of line buffer data can be written each time. By using the line write unit, the delay is the write delay of the row data, and the frame buffer data is written as the unit, and the delay is the write delay of the frame data. The write delay of the row data is significantly lower than that of the frame data. Input delay, it can be seen that the writing delay can be reduced in line writing units, which in turn can reduce the delay of image splicing. At the same time, the bandwidth occupation caused by repeated reading in the process of writing row data can be reduced by writing in row units. In some embodiments, the FPGA platform may perform filling processing on each row of row buffer data during the writing process of the row buffer data. By performing the padding process in parallel during the writing process of each row of buffer data, the delay of image mosaic can be further reduced.

In some embodiments, the FPGA platform writes multiple lines of line-buffered data for each line of the multiple lines of images to be spliced into the frame buffer based on the polling arbitration mode in line write units. In some embodiments, the polling arbitration mode can be used to dynamically adjust the order in which each line of the multiple lines of line buffer data of multiple images to be spliced is written into the frame buffer. In some embodiments, the polling arbitration mode can be used to write multiple lines of line buffer data of multiple lines of images to be spliced into the frame based on the read time sequence of each line of line buffer data of multiple lines of images to be spliced by the FPGA platform buffer. For example, assume that the chronological order of reading the line buffer data is: the first line buffer data of the first image to be stitched, the first line buffer data of the third image to be stitched, the first line of the second image to be stitched For the line buffer data of the first line of the fourth image to be spliced, based on the polling arbitration mode, the aforementioned line buffer data of each line can be sequentially written to the frame buffer. By using the polling arbitration mode to write the multi-line buffer data of multiple images to be spliced into the frame buffer, the dynamic writing of the multi-line buffer data of multiple images to be spliced can be realized, avoiding the same image to be spliced It is always processed, and other images to be stitched are not processed, thereby further reducing the delay of image stitching.

In some embodiments, the multi-channel image stitching method provided by the embodiments of the present disclosure may be applied to an AXI (Advanced eXtensible Interface) bus system. In some embodiments, the FPGA platform can write multiple lines of line buffer data of each line of the multiple lines of images to be spliced into the frame buffer through the AXI4 bus.

Step 106, perform preprocessing on the spliced frame buffer data to obtain a spliced image.

In some embodiments, preprocessing may include encoding processing and decoding processing. In some embodiments, the FPGA platform can perform encoding processing on the line buffer data of the preset line after writing the line buffer data of the preset line each time, until the second preset condition is met, and an encoded image is obtained; for the encoded image Decoding is performed to obtain a spliced image. In some embodiments, the preset row can be specifically set according to actual conditions. The second preset condition may include that the multi-line buffer data of the multiple images to be spliced is completely written. By performing encoding processing every time the preset line buffer data is written, the encoding delay is the delay of the preset line data instead of the delay of a whole frame of data, which can greatly reduce the encoding delay, thereby reducing the delay of image splicing.

In some embodiments, the built-in encoder of the field programmable gate array platform can be used to perform the above-mentioned operation on the row buffer data of the preset row after writing the row buffer data of the preset row each time. Encoding processing, until the second preset condition is met, to obtain the encoded image. In some embodiments, the encoder may be a hardcore H265 encoder, in which case the preset lines may be 16 lines. Based on its own performance, the H265 encoder can encode the written 16 lines of line buffer data after every 16 lines of line buffer data are written into the frame buffer, thereby reducing the encoding delay. In addition, the encoding delay can be reduced through the built-in encoder of the field programmable gate array platform, hardware devices can be reduced, and the design complexity of the field programmable gate array platform can be reduced.

In the embodiment of this specification, the multi-channel image mosaic method of the present disclosure is implemented through an FPGA platform, only uses storage resources inside the FPGA, has a simple algorithm, and occupies less storage and logic resources. And FPGA is a hardware platform, which can be applied to high frame rate (for example, 1080P60), high number of channels (for example, 6 channels or above), low distortion, and high line-of-sight camera equipment data access bandwidth requirements.

Fig. 3 is a flow chart of obtaining row data shown in an exemplary embodiment of the present disclosure. As shown in Figure 3, the process can include:

Step 302, decoding the preset interface signal output by the target camera device received based on the preset interface, to obtain decoded information; wherein, the decoded information includes at least line and field synchronization signals and pixel data of the image to be spliced; the The target imaging device is the imaging device of the path of images to be spliced.

In some embodiments, the preset interface may be an interface for the FPGA platform to receive a signal output by the camera device. In some embodiments, the preset interface may be specifically set according to actual conditions, for example, a MIPI interface. Correspondingly, the preset interface signal is a MIPI signal, and by decoding the MIPI signal, decoding information including a line and field synchronization signal and pixel data of an image to be spliced can be obtained.

Step 304: After obtaining the line data in the original format from the pixel data of the image to be spliced based on the line and field synchronization signals, perform format conversion on the line data to obtain line data in the target format.

Line and field synchronization signals can reflect the information of the end of a line (field). Therefore, the line data of each line can be obtained from the pixel data of the image to be spliced through the line and field synchronization information. Generally, the initial format of the pixel data transmitted through the MIPI interface is the YUV422 format, that is, the initial format of the line data is the YUV422 format. The format required for processing the pixel data in the row data is RGB format or NV12 format, therefore, in some embodiments, the row data can be format-converted to obtain row data in RGB format or NV12 format, the RGB format Or NV12 format which is the target format. In some embodiments, obtaining row data in the original format and format transcoding can be performed in parallel, and the delay of image splicing can be further reduced through the parallel execution.

Based on the same inventive concept, the present disclosure also provides a multi-channel image mosaic system, which is applied to a field programmable gate array platform. FIG. 4 is a block diagram of a multi-channel image stitching system shown in the present disclosure. As shown in FIG. 4, the institute system 400 includes:

The multi-line buffering module 402 is configured to repeatedly execute the acquisition of line data for each of the multiple lines of images to be spliced, and based on the preset mapping table, write the pixel data included in the line data into the image corresponding to the line of images to be spliced. Line buffer, the process of obtaining line buffer data until the first preset condition is met, so as to obtain the multiple lines of line buffer data of the image to be spliced; wherein, the preset mapping table is based on historical images to be spliced The mapping relationship between the pixel data and the historical mosaic image is obtained;

A frame buffer writing module 404, configured to write the multiple lines of line buffer data of each of the multiple lines of images to be spliced into the frame buffer, and write the multiple lines of line buffer data of each of the multiple lines of the image to be spliced Perform filling processing to obtain spliced frame buffer data;

The splicing processing module 406 is configured to perform preprocessing on the spliced frame buffer data to obtain a spliced image.

Optionally, the frame buffer writing module 404 includes:

Optionally, the parallel filling submodule includes:

Optionally, the multiline buffering module 402 includes:

Optionally, the splicing processing module 406 includes:

Optionally, the encoding submodule includes:

Regarding the system in the above embodiment, the specific manner in which each module executes operations has been described in detail in the embodiment of the method, and will not be described in detail here.

In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, and when the program instructions are executed by a processor, the steps of the above-mentioned multi-channel image stitching method are implemented.

In another exemplary embodiment, an unmanned vehicle is also provided, including the multi-channel image stitching system provided in the present disclosure. Based on the multi-channel image stitching system, the unmanned vehicle can stitch the multi-channel images collected by the unmanned vehicle and display it on the display screen of the unmanned vehicle, which can assist the safe driving of the unmanned vehicle. Parking improves the safety performance of unmanned vehicles.

The preferred embodiments of the present disclosure have been described in detail above in conjunction with the accompanying drawings. However, the present disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of the present disclosure, various simple modifications can be made to the technical solutions of the present disclosure. These simple modifications all belong to the protection scope of the present disclosure.

In addition, it should be noted that the various specific technical features described in the above specific implementation manners may be combined in any suitable manner if there is no contradiction. In order to avoid unnecessary repetition, various possible combinations are not further described in this disclosure.

The device embodiments described above are only illustrative, and the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in One place, or it can be distributed to multiple network elements. Part or all of the modules can be selected according to actual needs to achieve the purpose of the solution of this embodiment. It can be understood and implemented by those skilled in the art without any creative efforts.

The various component embodiments of the present disclosure may be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. Those skilled in the art should understand that a microprocessor or a digital signal processor (DSP) may be used in practice to implement some or all functions of some or all components in the computing processing device according to the embodiments of the present disclosure. The present disclosure can also be implemented as an apparatus or apparatus program (eg, computer program and computer program product) for performing a part or all of the methods described herein. Such a program realizing the present disclosure may be stored on a computer-readable medium, or may have the form of one or more signals. Such a signal may be downloaded from an Internet site, or provided on a carrier signal, or provided in any other form.

For example, FIG. 5 illustrates a computing processing device that may implement methods according to the present disclosure. The computing processing device conventionally includes a processor 510 and a computer program product or computer readable medium in the form of memory 520 . Memory 520 may be an electronic memory such as flash memory, EEPROM (Electrically Erasable Programmable Read Only Memory), EPROM, hard disk, or ROM. The memory 520 has a storage space 530 for program code 531 for performing any method steps in the methods described above. For example, the storage space 530 for program codes may include respective program codes 531 for respectively implementing various steps in the above methods. These program codes can be read from or written into one or more computer program products. These computer program products comprise program code carriers such as hard disks, compact disks (CDs), memory cards or floppy disks. Such a computer program product is typically a portable or fixed storage unit as described with reference to FIG. 6 . The storage unit may have storage segments, storage spaces, etc. arranged similarly to the memory 520 in the computing processing device of FIG. 5 . The program code can eg be compressed in a suitable form. Typically, the storage unit includes computer readable code 531', i.e. code readable by, for example, a processor such as 510, which code, when executed by a computing processing device, causes the computing processing device to perform the above-described method. each step.

Reference herein to "one embodiment," "an embodiment," or "one or more embodiments" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the present disclosure. Additionally, please note that examples of the word "in one embodiment" herein do not necessarily all refer to the same embodiment.

In the description provided herein, numerous specific details are set forth. However, it is understood that embodiments of the present disclosure may be practiced without these specific details. In some instances, well-known methods, structures and techniques have not been shown in detail in order not to obscure the understanding of this description.

In the claims, any reference signs placed between parentheses shall not be construed as limiting the claim. The word "comprising" does not exclude the presence of elements or steps not listed in a claim. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. The disclosure can be implemented by means of hardware comprising several distinct elements, and by means of a suitably programmed computer. In a unit claim enumerating several means, several of these means can be embodied by one and the same item of hardware. The use of the words first, second, and third, etc. does not indicate any order. These words can be interpreted as names.

In addition, any combination of various implementations of the present disclosure can also be made, as long as they do not violate the idea of the present disclosure, they should also be regarded as the content disclosed in the present disclosure.

Claims

A multi-channel image stitching method is characterized in that it is applied to a field programmable gate array platform, and the method includes:

For each path in the multiple paths of images to be spliced, perform the following processing to obtain multiple rows of line buffer data for each path in the multiple paths of images to be spliced:

Repeatedly performing the process of obtaining row data, and writing the pixel data included in the row data into the row buffer corresponding to the image to be spliced based on the preset mapping table, to obtain the row buffer data, until the first preset condition is met, Obtaining the multiple lines of line buffer data of the path of images to be spliced;

Wherein, the preset mapping table is obtained based on the mapping relationship of the pixel data between the historical image to be stitched and the historical stitched image;

Writing the multiple lines of line buffer data of each of the multiple lines of images to be spliced into a frame buffer, and performing filling processing on the multiple lines of line buffer data of each of the multiple lines of images to be spliced to obtain a spliced frame buffer data;

Perform preprocessing on the spliced frame buffer data to obtain a spliced image.
The method according to claim 1, wherein the multi-line buffer data of each path in the multiple paths of images to be spliced is written into a frame buffer, and the data of each path of the image to be spliced is The multiple rows of line buffer data are filled to obtain spliced frame buffer data, including:

Write the multiple lines of line buffer data of each line of the multiple lines of images to be spliced into the frame buffer in a line write unit, and write the line buffer data to the The filling process is performed on the line buffer data to obtain the spliced frame buffer data.
The method according to claim 2, wherein writing the multiple lines of line buffer data of each line of the multiple lines of images to be spliced into the frame buffer in units of line writing comprises:

Write the multiple rows of line buffer data of each channel in the multiple channels of images to be spliced into the frame buffer based on the writing unit of the row and based on a polling arbitration mode.
The method according to claim 1, wherein said acquiring row data comprises:

Decoding the preset interface signal output by the target camera device received based on the preset interface to obtain decoded information; wherein, the decoded information includes at least line and field synchronization signals and pixel data of the image to be spliced; the target camera device It is the camera equipment of the image to be spliced;

After the row data in the initial format is obtained from the pixel data of the image to be spliced based on the row and field synchronous signals, format conversion is performed on the row data to obtain row data in a target format.
The method according to claim 1, wherein, for each of the multiple paths of images to be spliced, the following processing is performed to obtain multiple lines of line buffer data of each path of the multiple paths of images to be spliced, including :

For each path of the multiple paths of images to be spliced, the processing is executed in parallel to obtain the multiple lines of line buffer data of each path of the multiple paths of images to be spliced.
The method according to claim 2, wherein the performing preprocessing on the spliced frame buffer data to obtain a spliced image comprises:

After writing the line buffer data of a preset line each time, performing encoding processing on the line buffer data of the preset line until a second preset condition is met to obtain an encoded image;

The coded image is decoded to obtain a spliced image.
The method according to claim 6, characterized in that, after writing the line buffer data of the preset line each time, the line buffer data of the preset line is encoded until the second predetermined line is satisfied. Set the condition to get the coded image;

Using the built-in encoder of the field programmable gate array system, after writing the row buffer data of the preset row each time, the encoding process is performed on the row buffer data of the preset row until the row buffer data of the preset row is satisfied. The second preset condition is to obtain the coded image.
A multi-channel image stitching system is characterized in that it is applied to a field programmable gate array platform, and the system includes:

The multi-line buffer module is used to repeatedly execute the acquisition of line data for each of the multiple lines of images to be spliced, and based on the preset mapping table, write the pixel data included in the line data into the line corresponding to the line of images to be spliced Buffering, the process of obtaining line buffer data until the first preset condition is met, so as to obtain the multiple lines of line buffer data of the image to be spliced; wherein, the preset mapping table is based on historical images to be spliced and The mapping relationship of the pixel data between historical mosaic images is obtained;

A frame buffer writing module, configured to write the multiple lines of line buffer data of each of the multiple channels of images to be spliced into a frame buffer, and execute Filling processing to obtain spliced frame buffer data;

A processing module, configured to perform preprocessing on the spliced frame buffer data to obtain a spliced image.
A computer-readable storage medium, on which a computer program is stored, characterized in that, when the program is executed by a processor, the steps of the method in any one of claims 1-7 are implemented.
An unmanned vehicle is characterized in that comprising the multi-channel image stitching system as claimed in claim 8.
A computing processing device, characterized in that it includes:

a memory having computer readable code stored therein; and

One or more processors, when the computer readable code is executed by the one or more processors, the computing processing device executes the multi-channel image stitching method according to any one of claims 1-7 .
A computer program, comprising computer-readable codes, when the computer-readable codes are run on a computing processing device, causing the computing processing device to perform the multi-channel image mosaic according to any one of claims 1-7 method.