CN114662592B

CN114662592B - Vehicle travel control method, device, storage medium, electronic device, and vehicle

Info

Publication number: CN114662592B
Application number: CN202210289296.8A
Authority: CN
Inventors: 黄彬
Original assignee: Xiaomi Automobile Technology Co Ltd
Current assignee: Xiaomi Automobile Technology Co Ltd
Priority date: 2022-03-22
Filing date: 2022-03-22
Publication date: 2023-04-07
Anticipated expiration: 2042-03-22
Also published as: CN114662592A

Abstract

The disclosure relates to a vehicle driving control method, a device, a storage medium, an electronic device and a vehicle, relating to the technical field of image recognition, wherein the method comprises the following steps: acquiring a first Bayer image output by a shooting device on a vehicle; inputting the first Bayer image into an image perception model to obtain a target detection result output by the image perception model; and controlling the vehicle to run according to the target detection result. Therefore, the target detection result is obtained by inputting the first Bayer image into the image perception model, compared with the method using the RGB image, the loss of image information can be reduced, the accuracy of image target detection is improved, the step of converting the first Bayer image into the RGB image is omitted, and the speed of image target detection is improved.

Description

Vehicle travel control method, device, storage medium, electronic device, and vehicle

Technical Field

The present disclosure relates to the field of image recognition technologies, and in particular, to a vehicle driving control method and apparatus, a storage medium, an electronic device, and a vehicle.

Background

Vision-based autopilot schemes require recognition of the captured image, including identifying the size, type, and location of objects in the image. Among them, the object recognition result is an important factor for determining the vehicle on which the vehicle is traveling, and therefore, how to accurately recognize an object in an image is an important issue in a vision-based automatic driving scheme.

Disclosure of Invention

To overcome the problems in the related art, the present disclosure provides a vehicle travel control method, apparatus, storage medium, electronic device, and vehicle.

According to a first aspect of an embodiment of the present disclosure, there is provided a vehicle travel control method including:

acquiring a first Bayer image output by a shooting device on a vehicle;

inputting the first Bayer image into an image perception model to obtain a target detection result output by the image perception model;

and controlling the vehicle to run according to the target detection result.

Optionally, the image perception model is obtained by:

the method comprises the steps of obtaining all-round-looking Bayer image samples around a vehicle, wherein each all-round-looking Bayer image sample is marked with a first target detection frame and a category to which the first target detection frame belongs;

obtaining a target splicing image under a bird's-eye view according to the Bayer image sample viewed all around;

and training a machine learning model based on the target spliced image to obtain a trained image perception model.

Optionally, the obtaining a target stitched image under a bird's eye view according to the bayer image sample for looking around includes:

constructing a pre-spliced image in a Bayer format under a bird's-eye view based on camera parameters corresponding to the shooting device;

determining green color values of pixels belonging to other color channels in the look-around Bayer image sample based on pixel values of a green color channel in the look-around Bayer image sample;

for each first target pixel belonging to a green channel in the pre-spliced image, determining a first mapping pixel of the first target pixel, which is mapped on the look-around Bayer image sample; and

determining a green color value corresponding to the first target pixel according to the green color values of pixels adjacent to the first mapped pixel;

for each second target pixel belonging to a red channel and a blue channel in the pre-spliced image, determining coordinate information of a second mapping pixel of the second target pixel mapped on the look-around Bayer image sample; and

determining a third target pixel in the looking-around Bayer image sample according to the coordinate information of the second mapping pixel;

determining a color value corresponding to the color channel to which the second target pixel belongs according to the color value of the third target pixel;

determining the pre-stitched image to the color value of each pixel as the target stitched image.

Optionally, the determining, according to the coordinate information of the second mapped pixel, a third target pixel in the bayer image sample for looking around includes:

when the coordinate information of the second mapping pixel represents that the second mapping pixel is positioned in the middle of four first pixels which belong to the same color channel as the second target pixel, determining the four first pixels as the third target pixel;

determining a color value corresponding to the color channel to which the second target pixel belongs according to the color value of the third target pixel includes:

and determining a color value corresponding to the color channel to which the second target pixel belongs based on a difference value between the color value corresponding to the color channel to which the four first pixels belong and the green color value of the first pixel.

when the coordinate information of the second mapping pixel represents that the second mapping pixel is located in the middle of two second pixels which belong to the same color channel as the second target pixel, determining the two second pixels as the third target pixel;

and determining the color value corresponding to the color channel to which the second target pixel belongs based on the difference value between the color value corresponding to the color channel to which the two second pixels belong and the green color value of the second pixel.

Optionally, the determining, according to the coordinate information of the second mapping pixel, a third target pixel in the looking-around bayer image sample includes:

when the coordinate information of the second mapping pixel represents that the second mapping pixel is located on a third pixel which belongs to the same color channel as the second target pixel, determining the third pixel and a fourth pixel as the third target pixel, wherein the fourth pixel is a pixel which has a distance with the third pixel within a preset distance threshold value and belongs to the same color channel as the third pixel;

determining, according to the color value of the third target pixel, a color value corresponding to a color channel to which the second target pixel belongs, including:

and determining a color value corresponding to the color channel to which the second target pixel belongs based on the third pixel and the fourth pixel.

Optionally, the training samples of the image perception model include those obtained by:

acquiring a shot image marked with a second target detection frame and a category to which the second target detection frame belongs;

and converting the shot image into a second Bayer image according to the optical characteristic parameters of the shooting device, and taking the second Bayer image as a training sample of the image perception model.

Optionally, the training sample of the image perception model further includes a sample obtained by:

acquiring a third Bayer image, wherein the third Bayer image is a Bayer image which is output by the shooting device in a real environment and marked with a third target detection frame and a category to which the third target detection frame belongs;

the method further comprises the following steps:

and adjusting an image perception model obtained by training the second Bayer image based on the third Bayer image to obtain an adjusted image perception model.

Optionally, the adjusting, based on the third bayer image, the image perception model obtained by training using the second bayer image to obtain an adjusted image perception model includes:

processing the third Bayer image based on an image processing technology to obtain a fourth Bayer image, wherein the image processing technology comprises one of image overturning, image enlarging and image rotating;

correcting the coordinates of a third target detection frame in the fourth Bayer image to obtain a corrected fourth Bayer image;

and adjusting an image perception model obtained by training the second Bayer image based on the third Bayer image and the corrected fourth Bayer image to obtain an adjusted image perception model.

Optionally, the processing the third bayer image based on an image processing technique to obtain a fourth bayer image includes:

adding at least one column and at least one row of pixels at a boundary of the third bayer image when the image processing technique is the image flipping; turning over the third Bayer image with the pixels increased according to a preset turning angle, and cutting the increased pixel rows and pixel columns at the corresponding boundary of the turned over third Bayer image to obtain a fourth Bayer image;

alternatively, the first and second electrodes may be,

adding at least one column and at least one row of pixels at a boundary of the third bayer image when the image processing technique is the image rotation; rotating the third Bayer image with the added pixels according to a preset rotation angle, and cutting the added pixel rows and pixel columns at the corresponding boundary of the rotated third Bayer image to obtain a fourth Bayer image;

alternatively, the first and second electrodes may be,

when the image processing technology is the image amplification, performing image enhancement processing on the third Bayer image to enhance the third Bayer image into four channels; and amplifying the third Bayer image enhanced into the four channels to obtain the fourth Bayer image.

Optionally, the correcting the coordinates of the third target detection frame in the fourth bayer image to obtain a corrected fourth bayer image includes:

when the fourth bayer image is obtained by performing the image inversion on the third bayer image, correcting coordinates of a third target detection frame in the fourth bayer image according to the preset inversion angle and a first coordinate correction value corresponding to the preset inversion angle to obtain a corrected fourth bayer image;

and when the fourth bayer image is obtained by performing the image rotation on the third bayer image, correcting the coordinates of a third target detection frame in the fourth bayer image according to the preset rotation angle and a second coordinate correction value corresponding to the preset rotation angle, so as to obtain the corrected fourth bayer image.

According to a second aspect of the embodiments of the present disclosure, there is provided a vehicle travel control apparatus including:

an acquisition module configured to acquire a first bayer image output by a camera on a vehicle;

the input module is configured to input the first Bayer image into an image perception model to obtain a target detection result output by the image perception model;

a control module configured to control the vehicle to travel according to the target detection result.

According to a third aspect of the embodiments of the present disclosure, there is provided a computer-readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the vehicle travel control method provided by the first aspect of the present disclosure.

According to a fourth aspect of the embodiments of the present disclosure, there is provided an electronic apparatus including:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to execute executable instructions stored in the memory to implement the vehicle travel control method of the first aspect.

According to a fifth aspect of the embodiments of the present disclosure, there is provided a vehicle including the vehicle running control apparatus of the second aspect.

The technical scheme provided by the embodiment of the disclosure can have the following beneficial effects: because the first bayer image is the original image data of the digital signal converted from the captured light source signal by the image sensing module of the shooting device, and the target detection result is obtained by inputting the first bayer image into the image sensing model, compared with the method using an RGB image, the method not only can reduce the loss of image information and improve the accuracy of image target detection, but also omits the step of converting the first bayer image into an RGB image and improves the speed of image target detection.

It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and together with the description, serve to explain the principles of the disclosure.

Fig. 1 is a flowchart illustrating a vehicle travel control method according to an exemplary embodiment.

FIG. 2 is a flow diagram illustrating obtaining an image perception model according to an exemplary embodiment.

Fig. 3 is a detailed flowchart of step 220 shown in fig. 2.

Fig. 4 is a schematic diagram illustrating a look-around bayer image sample, according to an example embodiment.

FIG. 5 is a schematic diagram illustrating a pixel coordinate mapping relationship in accordance with an exemplary embodiment.

FIG. 6 is a schematic diagram illustrating a third target pixel, according to an example embodiment.

Fig. 7 is a schematic diagram illustrating a third target pixel according to another exemplary embodiment.

Fig. 8 is a schematic diagram illustrating a third target pixel according to yet another exemplary embodiment.

Fig. 9 is a schematic flowchart illustrating an adjustment of an image perception model by using a third bayer image according to an exemplary embodiment.

Fig. 10 is a schematic diagram illustrating a fourth bayer image obtained through horizontal flipping according to an exemplary embodiment.

Fig. 11 is a schematic diagram illustrating a fourth bayer image obtained through vertical flipping according to an exemplary embodiment.

Fig. 12 is a schematic diagram illustrating a fourth bayer image obtained through 90 ° rotation, according to an example embodiment.

Fig. 13 is a schematic diagram illustrating a fourth bayer image obtained through 180 ° rotation, according to an example embodiment.

Fig. 14 is a schematic diagram illustrating a fourth bayer image obtained through 270 ° rotation, according to an example embodiment.

Fig. 15 is a schematic diagram illustrating a third bayer image for four channels, according to an example embodiment.

Fig. 16 is a block diagram showing a vehicle travel control apparatus according to an example embodiment.

FIG. 17 is a block diagram illustrating an electronic device in accordance with an example embodiment.

Detailed Description

Reference will now be made in detail to the exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, like numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the exemplary embodiments below are not intended to represent all implementations consistent with the present disclosure. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present disclosure, as detailed in the appended claims.

It should be noted that all actions of acquiring signals, information or data in the present application are performed under the premise of complying with the corresponding data protection regulation policy of the country of the location and obtaining the authorization given by the owner of the corresponding device.

Fig. 1 is a flowchart illustrating a vehicle running control method, which may be used in a vehicle, as shown in fig. 1, according to an exemplary embodiment, including the following steps.

In step 110, a first bayer image output by a camera on the vehicle is acquired.

Here, the photographing device is provided on the vehicle to capture an environmental image around the vehicle. For example, the photographing means may be a looking-around fisheye camera disposed in four directions of the front, rear, right, and left of the vehicle. It should be understood that the number of cameras arranged in each direction may be selected according to the actual situation. For example, one imaging device may be provided in both the front and rear directions, or two imaging devices may be provided.

The first bayer image is an image file in RAW format (one image file format) output by the photographing device. The first bayer image is actually raw image data of the image sensing module of the camera converting the captured light source signal into a digital signal, and is essentially a bayer array.

In step 120, the first bayer image is input into an image sensing model, and a target detection result output by the image sensing model is obtained.

Here, the image perception model is obtained by training a machine learning model using training samples, and is used to identify objects such as pedestrians, vehicles, lane lines, traffic signs, and the like from the first bayer image.

As an example, the training samples of the image perception model may be obtained by: acquiring a shot image marked with a second target detection frame and a category to which the second target detection frame belongs; and converting the shot image into a second Bayer image according to the optical characteristic parameters of the shooting device, and taking the second Bayer image as a training sample of the image perception model.

The captured image may be an RGB (one color mode) image downloaded from an open source database supporting visual perception of automatic driving, among others. The captured image is marked with a second target detection frame and a category to which the second target detection frame belongs. For example, the second object detection frame is marked for an image area of a pedestrian, a vehicle, a lane line, a traffic sign, or the like in the captured image, and the category of the image area is marked. It should be appreciated that the second target detection box is actually a GT box (true).

The optical characteristic parameters of the camera may include CCM matrix (color Correction matrix) of the camera, LSC matrix (Lens Shading Correction), noise characteristic, blur characteristic, and the like. The optical characteristic parameters of the shooting devices can be obtained by a calibration method for different types of shooting devices.

The specific process of converting the captured image into the second bayer image is as follows:

the operations of inverse Tone Mapping (inverse Tone Mapping), inverse gamma (inverse gamma), inverse color correction using a CCM matrix, mosaic (Mosaic), and inverse white balance are sequentially performed on the photographed image in RGB format, obtaining a simulated bayer image. It should be understood that obtaining a simulated bayer Image is actually performing an inverse ISP (Image Signal Processing) operation on a captured Image in RGB format. After the simulated bayer image is obtained, noise is added to the simulated bayer image according to noise characteristics of the photographing device, and a second bayer image is obtained.

In some embodiments, noise may be added on the simulated bayer image according to a noise model. Wherein the noise model is:

wherein σ _noise (I) Representing the calibrated noise standard deviation, K representing the correction parameters of the noise model, I representing the current pixel intensity, σ _dark Representing the standard deviation of the gaussian component of the noise.

It should be appreciated that the noise model described above is effectively a poisson gaussian model, which effectively represents the relationship between the camera's noise strength and the camera gain effects and signal strength values.

In this way, the trained image sensing model is obtained by training the image sensing model using the second bayer image generated by simulation as a training sample. And subsequently, inputting the first Bayer image output by the shooting device on the vehicle in real time into the image perception model, so as to obtain a target detection result output by the image perception model. Here, the target detection result refers to category information of the target detected in the first bayer image, position information of the target, size information of the target, and the like.

Therefore, the simulated second Bayer image is generated according to the optical characteristic parameters of the shooting device, the generated second Bayer image is close to the RAW image acquired by the shooting device, and data acquisition and labeling do not need to be carried out again, so that the cost for acquiring the training sample is saved.

It is to be noted that, in the above-described embodiment, the second bayer image generated by the simulation may be used as the training sample in the case where the amount of samples of the bayer image output by the camera in the real environment is insufficient. Of course, in the case where the amount of samples of the bayer image output by the imaging device in the real environment is sufficient, the bayer image output by the imaging device in the real environment may be directly used as the training sample.

It should be understood that, since the first bayer image is raw image data directly output by the camera, without being subjected to image signal processing, and the image data contained in the first bayer image has a linear relationship with the signal intensity of the environment, the target detection result obtained from the first bayer image is more accurate.

In step 130, the vehicle is controlled to run according to the target detection result.

Here, after the target detection result is obtained, a strategy for controlling the vehicle to travel is determined according to the target detection result, and the vehicle is controlled to travel according to the strategy. For example, after target detection results of pedestrians, vehicles, lane lines, traffic signs, and the like are obtained, a corresponding driving strategy may be determined according to the driving strategy model.

Therefore, the first Bayer image is the original image data obtained by converting the captured light source signal into the digital signal through the image sensing module of the shooting device, and the target detection result is obtained by inputting the first Bayer image into the image sensing model.

In some embodiments, the images in the input image perception model may be stitched images under Bird Eye View (BEV) according to different image recognition requirements. The stitched image is obtained by stitching a first Bayer image output by an imaging device on the vehicle. Therefore, the training sample of the image perception model also needs to be a stitched image under the bird's eye view.

Fig. 2 is a flow diagram illustrating obtaining an image perception model according to an example embodiment, which may be obtained by the following steps in some embodiments, as shown in fig. 2.

In step 210, looking around bayer image samples around the vehicle are obtained, and each looking around bayer image sample is marked with a first target detection frame and a category to which the first target detection frame belongs.

Here, the bayer image samples for looking around the vehicle may be obtained by cameras disposed in four directions of the front, rear, right, and left of the vehicle. It should be understood that the looking around bayer image sample may be obtained by converting a captured image, on which a first target detection frame and a category to which the first target detection frame belongs, into a bayer image in RAW format according to optical characteristic parameters of a capturing device.

It should be noted that the related meanings of the first target detection frame and the category to which the first target detection frame belongs are the same as the meanings of the second target detection frame and the category to which the second target detection frame belongs, and are not repeated herein.

In step 220, a target stitching image under the bird's eye view is obtained according to the circular Bayer image sample.

Here, after the round-looking bayer image samples are obtained, a pixel coordinate mapping relationship between the target stitched image and the round-looking bayer image samples may be calculated according to the camera internal and external parameters of the photographing device, and the round-looking bayer image samples may be stitched into the target stitched image in the bird's eye view according to the pixel coordinate mapping relationship.

In step 230, a machine learning model is trained based on the target stitched image, and a trained image perception model is obtained.

Here, after the target stitched image in the bird's eye view is obtained, the machine learning model is trained using the target stitched image as a training sample, and a trained image perception model is obtained. Wherein the machine learning model may be a neural network model.

It is worth noting that after the trained image perception model is obtained, the image perception model may be deployed on the vehicle. In the practical application process, a first Bayer image output by a shooting device on a vehicle is obtained, a spliced image under a bird's-eye view is obtained according to the first Bayer image, and then the spliced image is used as the input of a trained image perception model to obtain a target detection result.

Therefore, the machine learning model is trained by using the target stitching image under the aerial view to obtain the image perception model, so that the image perception model is also suitable for target detection under the aerial view.

FIG. 3 is a detailed flow chart of step 220 shown in FIG. 2. As shown in FIG. 3, in some embodiments, the target stitched image may be obtained by the following steps.

In step 221, a pre-stitched image in a bayer format under a bird's eye view is constructed based on the camera parameters corresponding to the photographing devices.

Here, the camera parameters corresponding to the image capturing device refer to an image capturing distance and an image capturing range of the image capturing device, and a bayer-format pre-stitched image in a bird's eye view is constructed based on the image capturing distance and the image capturing range in combination with the length and the width of the vehicle. For example, the photographing devices provided in the front and rear directions on the vehicle respectively capture images within a range of 25m, and the photographing devices provided in the left and right directions on the vehicle respectively capture images within a range of 10 m. The pre-stitched image is of size 20m × 50m.

It should be understood that the bayer format of the pre-stitched image is consistent with the bayer format of the training samples of the image perception model. For example, the bayer format of the training samples of the image perception model is the RGGB format, and then the bayer format of the pre-stitched image is also the RGGB format.

In step 222, green color values of pixels belonging to other color channels in the look-around bayer image sample are determined based on the pixel values of the green color channel in the look-around bayer image sample.

Here, the bayer image sample for looking around may be a bayer array of RGGB, and each pixel on the bayer image sample for looking around belongs to one color channel, which is a green channel (G channel), a red channel (R channel), and a blue channel (B channel). And calculating to obtain green color values corresponding to the pixels belonging to the R channel and the B channel according to the pixel values of the pixels belonging to the G channel in the ring-view Bayer image sample aiming at the pixels belonging to the R channel and the B channel in the ring-view Bayer image sample.

Fig. 4 is a schematic diagram illustrating a looking around bayer image sample according to an exemplary embodiment, as shown in fig. 4, G1, G2, G3, G4, G5, G6, G7, G8, G9, G10, G11, G12 are pixels belonging to a G channel in the looking around bayer image sample, and G1, G2, G3, G4, G5, G6, G7, G8, G9, G10, G11, G12, G13 are pixels belonging to an R channel or a B channel in the looking around bayer image sample. In general, G1, G2, G3, G4, G5, G6, G7, G8, G9, G10, G11, G12, and G13 include the pixel value of the color channel corresponding to the pixel, and do not include the green color value belonging to the G channel.

For pixels belonging to an R channel and a B channel in a bayer image sample for looking around, interpolation calculation may be performed according to green color values of pixels belonging to a G channel adjacent to the pixel, so as to obtain a green color value corresponding to the pixel.

For example, for the pixel G4, the green color value corresponding to the pixel G4 can be calculated from the green color values corresponding to G1, G3, G4, and G6. Specifically, it can be obtained by calculation using the following calculation formula.

Where G4 denotes the green color value of the pixel G4, and G1, G3, G4, G6 denote the green color values of pixels belonging to the G channel adjacent to the pixel G4.

In step 223, for each first target pixel belonging to the green channel in the pre-stitched image, a first mapping pixel of the first target pixel mapped on the look-around bayer image sample is determined.

Here, since the pre-stitched image is also a bayer image belonging to the RGGB format, pixels on the pre-stitched image also belong to the G channel, the R channel, and the B channel, respectively, but the color values of the color channels corresponding to the pixels on the pre-stitched image are indeterminate, and therefore, it is necessary to stitch the color values of the look-around bayer image samples to the pixels corresponding to the pre-stitched image.

Specifically, for each first target pixel belonging to the G channel in the pre-stitched image, a first mapping pixel of the first target pixel mapped on the looking-around bayer image sample may be determined according to the pixel coordinate mapping relationship.

The pixel coordinate mapping relation is a mapping relation between pixel coordinates of the pre-spliced image obtained according to the internal and external parameters of the camera of the shooting device and pixel coordinates of the look-around Bayer image sample. FIG. 5 is a schematic illustration of a pixel coordinate mapping relationship shown in accordance with an exemplary embodiment. As shown in fig. 5, when the image capturing device installed on the vehicle includes image capturing devices in four directions, front, rear, left, and right, the pixel coordinate mapping relationship refers to pixels in the coordinate system of the pre-stitched image, and the pixels mapped on the camera coordinate system of the front view image capturing device or the camera coordinate system of the rear view image capturing device or the camera coordinate system of the left view image capturing device or the camera coordinate system of the right view image capturing device are obtained through conversion under the world coordinate system of the bird's eye view image and the world coordinate system of the vehicle.

In step 224, a green color value corresponding to the first target pixel is determined based on the green color values of the pixels adjacent to the first mapped pixel.

Here, after determining the coordinate information of the first mapped pixel on the all-round bayer image sample, the green color value corresponding to the first target pixel is determined from the green color values of the pixels adjacent to the first mapped pixel.

Wherein the green color value of the pixel adjacent to the first mapped pixel is determined even if the pixel adjacent to the first mapped pixel does not belong to the G channel, since the green color value of each pixel on the looking-around bayer image sample has been calculated in step 222.

In some embodiments, the green color value corresponding to the first target pixel may be calculated by the following calculation formula.

/>

Wherein the content of the first and second substances,

representing a coordinate pick-up in the pre-stitched image>

(x, y) represents coordinate values of the first mapped pixel on the lookaround bayer image sample, G (x, y-1), G (x, y + 1), G (x-1,y) and G (x +1,y) respectively represent green color values of pixels adjacent to the first mapped pixel, and remap represents a pixel coordinate mapping relationship.

It should be understood that the above calculation formula is actually to obtain the green color value corresponding to the first target pixel belonging to the G channel by interpolation using the green color values of the pixels adjacent to the first mapped pixel.

In step 225, for each second target pixel belonging to the red channel and the blue channel in the pre-stitched image, determining coordinate information of a second mapped pixel of the second target pixel mapped on the look-around bayer image sample.

Here, for each second target pixel belonging to the R channel and the B channel in the pre-stitched image, the coordinate information of the second mapped pixel, on which the second target pixel is mapped on the looking-around bayer image sample, is determined through the above-mentioned pixel coordinate mapping relationship.

It should be understood that the steps performed by the second target pixel are identical regardless of whether the second target pixel is a pixel belonging to the R channel or the B channel.

In step 226, a third target pixel is determined in the look-around bayer image sample based on the coordinate information of the second mapped pixel.

Here, after determining the coordinate information to the third mapped pixel, the third target pixel is determined in the look-around bayer image sample based on the coordinate information. It should be appreciated that different third target pixels may be determined based on different coordinate information.

As an example, when the coordinate information of the second mapping pixel indicates that the second mapping pixel is located in the middle of two second pixels belonging to the same color channel as the second target pixel, two of the second pixels are determined as the third target pixel.

FIG. 6 is a schematic diagram illustrating a third target pixel, according to an example embodiment. As shown in fig. 6, sub-graph (a) is a partial schematic diagram of the pre-stitched image, and sub-graph (b) is a partial schematic diagram of a round-looking bayer image sample. In the pre-stitched image, according to the pixel coordinate mapping relationship, the pixel of the second target pixel 60 belonging to the R channel mapped on the look-around bayer image sample is the second mapped pixel 61. Since the second mapped pixel 61 is located in the middle of the first pixels R1, R2, R3 and R4 belonging to the same R channel as the second target pixel. Therefore, four first pixels of R1, R2, R3, and R4 are determined as the third target pixel.

It should be understood that, in the above example, the second target pixel belonging to the R channel is exemplified, and when the second target pixel is a pixel of the B channel, the method for determining the third target pixel is consistent, and is not described herein again.

As another example, when the coordinate information of the second mapping pixel indicates that the second mapping pixel is located in the middle of two second pixels belonging to the same color channel as the second target pixel, the two second pixels are determined as the third target pixel.

Fig. 7 is a schematic diagram illustrating a third target pixel according to another exemplary embodiment. As shown in fig. 7, sub-graph (a) is a partial schematic diagram of the pre-stitched image, and sub-graph (b) is a partial schematic diagram of a round-looking bayer image sample. In the pre-stitched image, according to the pixel coordinate mapping relationship, the pixel of the second target pixel 70 belonging to the R channel mapped on the look-around bayer image sample is the second mapped pixel 71. Since the second mapped pixel 71 is located in the middle of the second pixels R2 and R4 belonging to the same R channel as the second target pixel. Therefore, two second pixels of R2 and R4 are determined as the third target pixel.

It should be understood that, in the above example, the second target pixel belonging to the R channel is taken as an example, and when the second target pixel is a pixel of the B channel, the method for determining the third target pixel is consistent, and will not be described herein again.

As a further example, when the coordinate information of the second mapping pixel indicates that the second mapping pixel is located on a third pixel belonging to the same color channel as the second target pixel, the third pixel and a fourth pixel are determined as the third target pixel, where the fourth pixel is a pixel which is within a preset distance threshold from the third pixel and belongs to the same color channel as the third pixel.

Fig. 8 is a schematic diagram illustrating a third target pixel according to yet another exemplary embodiment. As shown in fig. 8, sub-graph (a) is a partial schematic diagram of the pre-stitched image, and sub-graph (b) is a partial schematic diagram of a round-looking bayer image sample. In the pre-stitched image, according to the pixel coordinate mapping relationship, the pixel of the second target pixel 80 belonging to the R channel mapped on the look-around bayer image sample is the second mapped pixel 81. Since the second mapped pixel 81 is the third pixel R2 belonging to the same R channel as the second target pixel 80. Therefore, the third pixel R2 and the fourth pixels R1, R3, and R4 belonging to the R channel together with the third pixel R2 and having a distance from the third pixel within a preset distance threshold are determined as the third target pixel.

The preset distance threshold may be determined according to the position information of the second mapping pixel in the pre-stitched image. For example, when the second mapped pixel is located at the left half of the pre-stitched image, a fourth pixel which is located at the left half of the third pixel, is separated from the third pixel by a pixel distance, and belongs to the same color channel as the third pixel, and is determined as the third target pixel. It is noted that the predetermined distance threshold may have a positive or negative score to reflect that the fourth pixel is to the left or right of the third pixel.

In step 227, according to the color value of the third target pixel, a color value corresponding to the color channel to which the second target pixel belongs is determined.

Here, after the third target pixel is obtained, a color value corresponding to the color channel to which the second target pixel belongs on the pre-stitched image is calculated and obtained according to the color value of the third target pixel in the looking-around bayer image sample.

For example, when the second target pixel is a pixel of the R channel, interpolation calculation is performed based on the color value of the third target pixel in the ring-view bayer image sample, and the red color value of the R channel of the second target pixel is obtained. And when the second target pixel is a pixel of the B channel, performing interpolation calculation according to the color value of the third target pixel in the ring-vision Bayer image sample to obtain a second target pixel red color value belonging to the B channel.

As an example, when the third target pixel is four first pixels, the color value corresponding to the color channel to which the second target pixel belongs is determined based on a difference between the color value corresponding to the color channel to which the four first pixels belong and the green color value of the first pixel.

The following description will be made in terms of calculating the red color value of the third target pixel belonging to the R channel, and the calculation method is consistent with that for the blue color value of the third target pixel belonging to the B channel. In some examples, the red color value of the third target pixel belonging to the R channel may be calculated by the following calculation formula.

/>

Wherein the content of the first and second substances,

indicating a coordinate of pickin the pre-stitched image>

The red color value of the second target pixel belonging to the R channel, remap represents the pixel coordinate mapping relationship, (x, y) represents the coordinate information of the second mapped pixel in the looking around bayer image sample, G (floor (x), floor (y)) represents the green color value of the pixel with coordinates (floor (x), floor (y)) in the looking around bayer image sample, G (floor (x) -1, floor (y) -1, floor (x) -1, floor (y) + 1), G (floor (x) +1, floor-1, floor +1, respectively represent the green color value of the corresponding first pixel, R (floor (x) -1, floor (y) -1, R (floor (x) -1, floor (y) +1, floor (x) +1, floor (y) + 1), R (floor (x) +1, floor (y) +1, and R (floor (x) +1, floor +1, respectively represent the red color value of the first pixel.

As another example, when the third target pixel is two second pixels, the color value corresponding to the color channel to which the second target pixel belongs is determined based on a difference between the color value corresponding to the color channel to which the two second pixels belong and the green color value of the second pixel.

Wherein the content of the first and second substances,

indicating a coordinate of pickin the pre-stitched image>

The remap represents a pixel coordinate mapping relationship, (x, y) represents coordinate information of the second mapped pixel in the looking-around bayer image sample, G (floor (x), floor (y)) represents a green color value of a pixel having coordinates (floor (x), floor (y)) in the looking-around bayer image sample, G (floor (x), floor (y) -1) and G (floor (x), floor (y) -1) respectively represent a green color value of a corresponding second pixel, and R (floor (x), floor (y) -1) and R (floor (x) respectively represent a red color value of a second pixel belonging to the R channel.

As another example, when the third target pixel is the third pixel and the fourth pixel, the color value corresponding to the color channel to which the second target pixel belongs is determined based on the third pixel and the fourth pixel.

As shown in fig. 8, the third pixel is R2, and the fourth pixel is R1, R3, and R4. The red color value of the R channel to which the second target pixel belongs can be calculated by the following calculation formula.

Wherein the content of the first and second substances,

indicating a coordinate of pickin the pre-stitched image>

Of a second target pixel belonging to the R channel, R ₂ The red color value of the third pixel, R1, R3 and R4 are the red color values of the corresponding fourth pixel, and w1, w2, w3 and w4 are weight values. Wherein w1, w2, w3 and w4 are constants, and the values of w1, w2, w3 and w4 can be set according to actual conditions.

It should be understood that, in the above example, the calculation of the red color value of the third target pixel belonging to the R channel is exemplified, and the calculation method is consistent for the blue color value of the third target pixel belonging to the B channel.

In step 228, the pre-stitched image determined to the color value of each pixel is determined to be the target stitched image.

Here, after the corresponding color value of each pixel in the pre-stitched image is calculated, the pre-stitched image is determined as the target stitched image.

It should be understood that the

above steps

223 and 225 may be performed simultaneously, that is, the calculation of the green color value of the pixel belonging to the G channel in the pre-stitched image and the calculation of the red color value of the pixel belonging to the R channel in the pre-stitched image and the calculation of the blue color value of the pixel belonging to the B channel in the pre-stitched image may be performed simultaneously.

Therefore, the color values of the R-channel and B-channel pixels on the target stitched image can be accurately mapped in the target stitched image by assisting the calculation of the color values of the R-channel and B-channel pixels on the target stitched image according to the green color values of the pixels in the ring-view Bayer image samples.

It is to be noted that the method of obtaining the stitched image in the bird's eye view from the first bayer image is consistent with the method of obtaining the target stitched image from the looking-around bayer image sample, and will not be described in detail here.

In some embodiments, the image perception model trained using the second bayer image is actually a preliminarily trained image perception model, so as to further improve the accuracy of the image perception model. A third bayer image may be acquired, where the third bayer image is a bayer image that is output by the imaging device in a real environment and that is marked with a third target detection frame and a category to which the third target detection frame belongs; and adjusting the image perception model obtained by training the second Bayer image based on the third Bayer image to obtain an adjusted image perception model.

Here, the third bayer image may be a RAW image output by the photographing device in a real environment. Since the third bayer image is a bayer array, labeling is inconvenient. The third bayer image may be converted into a visualized RGB image by a local image signal processing tool. And marking the image position of the target in each image area by using a third target detection frame in the RGB image and marking the category to which the third detection frame belongs. Since the pixels of the third bayer image have a corresponding relationship with the pixels of the RGB image, the third target detection frame marked on the RGB image and the class to which the third target detection frame belongs may be mapped on the third bayer image, and the third bayer image having the labeling information may be obtained.

Wherein, converting the third bayer image into a visualized RGB image is actually performing ISP processing on the third bayer image. The specific process can be as follows: and sequentially performing linearization, black level correction, dark angle removal, digital gain, denoising, white balance, color interpolation, color correction, gamma processing, tone mapping, denoising and sharpening on the third Bayer image, and converting the third Bayer image into a visual RGB image.

It should be appreciated that converting the third bayer image to a visualized RGB image may select different ISP processing flows depending on the image perception modules of the different types of cameras.

And after the third Bayer image is obtained, adjusting parameters in the image perception model obtained by training the second Bayer image by taking the third Bayer image as a training sample to obtain an adjusted image perception model.

Therefore, the target detection accuracy of the image perception model adjusted through the third Bayer image is higher.

Fig. 9 is a schematic flowchart illustrating an adjustment of an image perception model by using a third bayer image according to an exemplary embodiment. As shown in fig. 9, in some embodiments, the image perception model may be adjusted by the following steps.

In step 910, the third bayer image is processed based on an image processing technique to obtain a fourth bayer image, wherein the image processing technique includes one of image flipping, image enlarging, and image rotating.

Here, the processing of the third bayer image using the image processing technique is actually converting the third bayer image into a fourth bayer image to increase the sample amount of training samples used for training the image perception model. Wherein the image is one of image flipping, image magnification, and image rotation. For example, the third bayer image may be subjected to image inversion processing to obtain a fourth bayer image. For another example, the third bayer image may be subjected to image enlargement processing to obtain a fourth bayer image. Also for example, the third bayer image may be subjected to image rotation processing to obtain a fourth bayer image.

It should be understood that, by processing the third bayer image using an image processing technique and changing the format of the image, the fourth bayer image and the third bayer image both belong to an image in a RAW format.

As an example, when the image processing technique is the image flipping, at least one column and at least one row of pixels are added at the boundary of the third bayer image; and turning over the third Bayer image with the increased pixels according to a preset turning angle, and cutting the increased pixel rows and pixel columns at the corresponding boundary of the turned over third Bayer image to obtain the fourth Bayer image.

Fig. 10 is a schematic diagram illustrating a fourth bayer image obtained through horizontal flipping according to an exemplary embodiment. As shown in fig. 10, sub-diagram (a) in fig. 10 is a third bayer image, sub-diagram (b) is a third bayer image in which at least one column and at least one row of pixels are added, sub-diagram (c) is a third bayer image after horizontal inversion, and sub-diagram (d) is a fourth bayer image.

Fig. 11 is a schematic diagram illustrating a fourth bayer image obtained through vertical flipping according to an exemplary embodiment. As shown in fig. 11, sub-diagram (a) in fig. 11 is a third bayer image, sub-diagram (b) is a third bayer image in which at least one column and at least one row of pixels are added, sub-diagram (c) is a third bayer image after vertical inversion, and sub-diagram (d) is a fourth bayer image.

Wherein adding at least one column and at least one row of pixels at the boundary of the third bayer image may be adding at least one column and at least one row of pixels at a right boundary and a bottom edge of the original third bayer image, so that the rows and columns of the added pixels of the third bayer image become odd rows and odd columns. It should be understood that the format of the flipped fourth bayer image may be made consistent with the format of the third bayer image by increasing the pixel rows and pixel columns. For example, the format of the third bayer image is RGGB format, and by adding pixel rows and pixel columns, the format of the inverted third bayer image is also RGGB format.

And after the third Bayer image with the added pixels is turned over, cutting the added pixel rows and pixel columns at the corresponding boundary of the turned-over third Bayer image to obtain the fourth Bayer image. For example, if at least one column and at least one row of pixels are added to the right boundary and the bottom edge of the original third bayer image, at least one column and at least one row of pixels are cut on the right boundary and the bottom edge corresponding to the inverted third bayer image.

It is worth noting that the added rows and columns of pixels may be obtained by mirroring the rows and columns of pixels on opposite boundaries.

As yet another example, when the image processing technique is the image rotation, at least one column and at least one row of pixels are added at a boundary of the third bayer image; and rotating the third Bayer image with the added pixels according to a preset rotation angle, and cutting the added pixel rows and pixel columns at the corresponding boundary of the rotated third Bayer image to obtain the fourth Bayer image.

Fig. 12 is a schematic diagram illustrating a fourth bayer image obtained through 90 ° rotation, according to an example embodiment. As shown in fig. 12, a sub-image (a) in fig. 12 is a third bayer image, a sub-image (b) is a third bayer image added with at least one column and at least one row of pixels, a sub-image (c) is a third bayer image rotated by 90 °, and a sub-image (d) is a fourth bayer image.

Fig. 13 is a schematic diagram illustrating a fourth bayer image obtained through 180 ° rotation, according to an example embodiment. As shown in fig. 13, a sub-image (a) in fig. 13 is a third bayer image, a sub-image (b) is a third bayer image added with at least one column and at least one row of pixels, a sub-image (c) is a third bayer image rotated by 180 °, and a sub-image (d) is a fourth bayer image.

Fig. 14 is a schematic diagram illustrating a fourth bayer image obtained through 270 ° rotation, according to an example embodiment. As shown in fig. 14, a sub-diagram (a) in fig. 14 is a third bayer image, a sub-diagram (b) is a third bayer image added with at least one column and at least one row of pixels, a sub-diagram (c) is a third bayer image rotated by 270 °, and a sub-diagram (d) is a fourth bayer image.

Wherein adding at least one column and at least one row of pixels at the boundary of the third bayer image may be adding at least one column and at least one row of pixels at a right boundary and a bottom edge of the original third bayer image, so that the rows and columns of the added pixels of the third bayer image become odd rows and odd columns. It should be appreciated that the format of the rotated fourth bayer image may be made consistent with the format of the third bayer image by increasing the number of rows and columns of pixels. For example, the format of the third bayer image is an RGGB format, and by increasing the pixel rows and the pixel columns, the format of the rotated third bayer image is also the RGGB format. If the pixel rows are not increased, the third bayer image in RGGB format will be changed to GRBG format after 90 ° rotation.

And after the third Bayer image with the added pixels is rotated, cutting the added pixel rows and pixel columns at the corresponding boundary of the rotated third Bayer image to obtain the fourth Bayer image. For example, if at least one column and at least one row of pixels are added to the right boundary and the bottom edge of the original third bayer image, at least one column and at least one row of pixels are cut from the right boundary and the bottom edge corresponding to the rotated third bayer image.

As another example, when the image processing technique is the image enlargement, the third bayer image is subjected to image enhancement processing to enhance the third bayer image into four channels; and amplifying the third Bayer image enhanced into the four channels to obtain the fourth Bayer image.

Fig. 15 is a schematic diagram illustrating a third bayer image for four channels, according to an example embodiment. As shown in fig. 15, sub-diagram (a) in fig. 15 is a third bayer image, and sub-diagram (b) is a partial schematic diagram of the third bayer image enhanced to four channels.

And performing image enhancement processing on the third Bayer image to enhance the third Bayer image into four channels, and performing lossless amplification on the third Bayer image enhanced into four channels. It is possible to avoid directly confusing signal values of different colors in the third bayer image.

In step 920, coordinates of the third target detection frame in the fourth bayer image are corrected, so as to obtain a corrected fourth bayer image.

Here, since the fourth bayer image is processed by an image processing technique, the position of the third target detection frame in the fourth bayer image may change, and therefore, the coordinates of the third target detection frame in the fourth bayer image need to be corrected.

As an example, when the fourth bayer image is obtained by performing the image reversal on the third bayer image, the coordinates of the third target detection frame in the fourth bayer image are corrected according to the preset reversal angle and the first coordinate correction value corresponding to the preset reversal angle, so as to obtain the corrected fourth bayer image.

As shown in fig. 10, when the third target detection frame is labeled in a coordinate form, the coordinates of the third target detection frame in the fourth bayer image obtained through horizontal inversion can be calculated by the following calculation formula:

wherein the content of the first and second substances,

coordinates representing a third target detection frame in the fourth bayer image, hflip representing a horizontal inversion, (x, y) representing coordinates of the third target detection frame in the third bayer image, and (1,0) representing a first coordinate correction value corresponding to the horizontal inversion.

As shown in fig. 10, when the third target detection frame is labeled in the form of a pixel region (for example, labeled by a mask image labeling tool), the coordinates of the third target detection frame in the fourth bayer image obtained through horizontal inversion can be calculated by the following calculation formula:

wherein the content of the first and second substances,

pixel region coordinates representing a third target detection frame in the fourth bayer image, hflip representing horizontal inversion, (mask) pixel region coordinates representing a third target detection frame in the third bayer image, and (x-1,y) a first coordinate correction value corresponding to the horizontal inversion.

As shown in fig. 11, when the third target detection frame is labeled in a coordinate form, the coordinates of the third target detection frame in the fourth bayer image obtained through vertical inversion can be calculated by the following calculation formula:

wherein the content of the first and second substances,

coordinates of a third target detection frame in the fourth bayer image are represented, vflip represents vertical inversion, (x, y) coordinates of the third target detection frame in the third bayer image are represented, and (0,1) a first coordinate correction value corresponding to the vertical inversion is represented.

As shown in fig. 11, when the third target detection frame is labeled in the form of a pixel region (for example, labeled by a mask image labeling tool), the coordinates of the third target detection frame in the fourth bayer image obtained by vertical inversion can be calculated by the following calculation formula:

wherein the content of the first and second substances,

pixel region coordinates representing a third target detection frame in the fourth bayer image, vflip representing a vertical inversion, (mask) pixel region coordinates representing a third target detection frame in the third bayer image, and (x, y-1) a first coordinate correction value corresponding to the vertical inversion.

As another example, when the fourth bayer image is obtained by performing the image rotation on the third bayer image, the coordinates of the third target detection frame in the fourth bayer image are corrected according to the preset rotation angle and the second coordinate correction value corresponding to the preset rotation angle, so as to obtain the corrected fourth bayer image.

As shown in fig. 12, when the third target detection frame is labeled in the form of coordinates, the coordinates of the third target detection frame in the fourth bayer image obtained through 90 ° rotation may be calculated by the following calculation formula:

wherein, the first and the second end of the pipe are connected with each other,

coordinates of a third target detection frame in the fourth bayer image are indicated, rot90 indicates 90 ° rotation, (x, y) indicates coordinates of the third target detection frame in the third bayer image, and (1,0) indicates a second coordinate correction value corresponding to 90 ° rotation.

As shown in fig. 12, when the third target detection frame is labeled in the form of a pixel region (for example, labeled by a mask image labeling tool), the coordinates of the third target detection frame in the fourth bayer image obtained through 90 ° rotation can be calculated by the following calculation formula:

wherein the content of the first and second substances,

pixel region coordinates representing a third target detection frame in the fourth bayer image, rot90 representing 90 ° rotation, (mask) representing pixel region coordinates of the third target detection frame in the third bayer image, and (x-1,y) representing a second coordinate correction value corresponding to 90 ° rotation.

As shown in fig. 13, when the third target detection frame is labeled in the form of coordinates, the coordinates of the third target detection frame in the fourth bayer image obtained through 180 ° rotation may be calculated by the following calculation formula:

/>

wherein the content of the first and second substances,

coordinates representing a third target detection frame in the fourth bayer image, rot180 representing a 180 ° rotation, (x, y) representing coordinates of the third target detection frame in the third bayer image, and (1,1) representing a second coordinate correction value corresponding to the 180 ° rotation.

As shown in fig. 13, when the third target detection frame is labeled in the form of a pixel region (for example, labeled by a mask image labeling tool), the coordinates of the third target detection frame in the fourth bayer image obtained through 180 ° rotation can be calculated by the following calculation formula:

wherein the content of the first and second substances,

pixel region coordinates representing a third target detection frame in the fourth bayer image, rot180 representing a 180 ° rotation, (mask) representing pixel region coordinates of the third target detection frame in the third bayer image, and (x-1,y-1) representing a second coordinate correction value corresponding to the 180 ° rotation.

As shown in fig. 14, when the third target detection frame is labeled in the form of coordinates, the coordinates of the third target detection frame in the fourth bayer image obtained through 180 ° rotation may be calculated by the following calculation formula:

wherein the content of the first and second substances,

coordinates of a third target detection frame in the fourth bayer image are represented, rot270 represents 270 ° rotation, (x, y) represents coordinates of the third target detection frame in the third bayer image, and (0,1) represents a second coordinate correction value corresponding to 270 ° rotation.

As shown in fig. 14, when the third target detection frame is labeled in the form of a pixel region (for example, labeled by a mask image labeling tool), the coordinates of the third target detection frame in the fourth bayer image obtained through 270 ° rotation can be calculated by the following calculation formula:

wherein the content of the first and second substances,

pixel region coordinates representing a third target detection frame in the fourth bayer image, rot270 representing a 270 ° rotation, (mask) representing pixel region coordinates of the third target detection frame in the third bayer image, and (x, y-1) representing a second coordinate correction value corresponding to the 270 ° rotation.

It should be noted that, when the fourth bayer image is obtained by image amplification, coordinate correction is not required, and the fourth bayer image of four channels may be directly subjected to lossless amplification, so that the position of the third target detection frame does not shift.

In step 930, an image perceptual model obtained by training the second bayer image is adjusted based on the third bayer image and the modified fourth bayer image, so as to obtain an adjusted image perceptual model.

After obtaining the corrected fourth bayer image, the image sensing model obtained by training the second bayer image is adjusted by using the third bayer image and the corrected fourth bayer image as training samples, so as to obtain an adjusted image sensing model. Wherein the adjusted image perception model is deployed on a vehicle for target detection of an automatic driving scheme based on visual perception.

Therefore, the third Bayer image is processed through the image processing technology, more training samples for training the image perception model can be obtained, and the cost for obtaining the training samples again is reduced.

Fig. 16 is a block diagram showing a vehicle travel control apparatus according to an exemplary embodiment. Referring to fig. 16, the apparatus 1600 includes:

an acquisition module 1601 configured to acquire a first bayer image output by a camera on a vehicle;

an input module 1602, configured to input the first bayer image into an image sensing model, to obtain a target detection result output by the image sensing model;

a control module 1603 configured to control the vehicle to run according to the target detection result.

Optionally, the input module 1602 includes:

the system comprises a sample acquisition unit, a detection unit and a processing unit, wherein the sample acquisition unit is configured to acquire all-around-view Bayer image samples around a vehicle, and each all-around-view Bayer image sample is marked with a first target detection frame and a category to which the first target detection frame belongs;

the image splicing unit is configured to obtain a target spliced image under a bird's-eye view according to the ring-view Bayer image sample;

and the training unit is configured to train a machine learning model based on the target spliced image to obtain a trained image perception model.

Optionally, the image stitching unit includes:

the image construction unit is configured to construct a pre-spliced image in a Bayer format under a bird's-eye view based on the camera parameters corresponding to the shooting device;

a first calculation unit configured to determine, based on pixel values of a green channel in the looking-around bayer image sample, green color values of pixels belonging to other color channels in the looking-around bayer image sample;

a first coordinate determination unit, configured to determine, for each first target pixel belonging to a green channel in the pre-stitched image, a first mapped pixel of the first target pixel mapped on the look-around bayer image sample; and

a second calculation unit configured to determine a green color value corresponding to the first target pixel from green color values of pixels adjacent to the first mapped pixel;

a second coordinate determination unit configured to determine, for each second target pixel belonging to a red channel and a blue channel in the pre-stitched image, coordinate information of a second mapped pixel of the second target pixel mapped on the look-around bayer image sample; and

a target pixel determination unit configured to determine a third target pixel in the look-around bayer image sample according to the coordinate information of the second mapped pixel;

the third calculation unit is configured to determine a color value corresponding to a color channel to which the second target pixel belongs according to the color value of the third target pixel;

an image determining unit configured to determine the pre-stitched image determined to the color value of each pixel as the target stitched image.

Optionally, the target pixel determination unit is specifically configured to:

the third computing unit is specifically configured to:

and determining the color value corresponding to the color channel to which the second target pixel belongs based on the difference value between the color value corresponding to the color channel to which the four first pixels belong and the green color value of the first pixel.

Optionally, the target pixel determination unit is specifically configured to:

the third computing unit is specifically configured to:

and determining a color value corresponding to the color channel to which the second target pixel belongs based on a difference value between the color value corresponding to the color channel to which the two second pixels belong and the green color value of the second pixel.

Optionally, the target pixel determination unit is specifically configured to:

the third computing unit is specifically configured to:

Optionally, the input module 1602 includes:

a first image acquisition unit configured to acquire a captured image marked with a second target detection frame and a category to which the second target detection frame belongs;

and the conversion unit is configured to convert the shot image into a second Bayer image according to the optical characteristic parameters of the shooting device, and the second Bayer image is used as a training sample of the image perception model.

Optionally, the input module 1602 further includes:

a second image acquisition unit configured to acquire a third bayer image, wherein the third bayer image is a bayer image that is output by the imaging apparatus in a real environment and is marked with a third target detection frame and a category to which the third target detection frame belongs;

the apparatus 1600 further comprises:

and the adjusting module is configured to adjust the image perception model obtained by training the second Bayer image based on the third Bayer image to obtain an adjusted image perception model.

Optionally, the adjusting module includes:

an image processing unit configured to process the third bayer image based on an image processing technique to obtain a fourth bayer image, wherein the image processing technique includes one of image flipping, image magnification, and image rotation;

a correction unit configured to correct coordinates of a third target detection frame in the fourth bayer image, to obtain a corrected fourth bayer image;

and the model adjusting unit is configured to adjust an image perception model obtained by training the second bayer image based on the third bayer image and the corrected fourth bayer image, so as to obtain an adjusted image perception model.

Optionally, the image processing unit is specifically configured to:

adding at least one column and at least one row of pixels at a boundary of the third bayer image when the image processing technique is the image flipping; turning over the third Bayer image with the added pixels according to a preset turning angle, and cutting the added pixel rows and pixel columns at the boundary corresponding to the turned over third Bayer image to obtain a fourth Bayer image;

alternatively, the first and second electrodes may be,

Optionally, the modification unit is specifically configured to:

and when the fourth Bayer image is obtained by performing the image rotation on the third Bayer image, correcting the coordinates of a third target detection frame in the fourth Bayer image according to the preset rotation angle and a second coordinate correction value corresponding to the preset rotation angle to obtain the corrected fourth Bayer image.

With respect to the apparatus 1600 in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment related to the method, and will not be elaborated here.

The present disclosure also provides a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, implement the steps of the vehicle travel control method provided by the present disclosure.

The present disclosure also provides a vehicle including the vehicle travel control apparatus described above.

FIG. 17 is a block diagram of an electronic device shown in accordance with an example embodiment. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, an exercise device, a personal digital assistant, and the like.

Referring to fig. 17, electronic device 800 may include one or more of the following components: processing component 802, memory 804, power component 806, multimedia component 808, audio component 810, input/output (I/O) interface 812, sensor component 814, communication component 816, and a camera (not shown in fig. 17).

The processing component 802 generally controls overall operation of the electronic device 800, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing components 802 may include one or more processors 820 that execute instructions to perform all or a portion of the steps of the vehicle travel control method described above. Further, the processing component 802 can include one or more modules that facilitate interaction between the processing component 802 and other components. For example, the processing component 802 can include a multimedia module to facilitate interaction between the multimedia component 808 and the processing component 802.

The memory 804 is configured to store various types of data to support operations at the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, image samples, and so forth. The memory 804 may be implemented by any type or combination of volatile or non-volatile memory devices, such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disks.

The power components 806 provide power to the various components of the electronic device 800. Power components 806 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for electronic device 800.

The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and a user. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundary of a touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front facing camera and/or a rear facing camera. The front camera and/or the rear camera may receive external multimedia data when the electronic device 800 is in an operation mode, such as a shooting mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

The audio component 810 is configured to output and/or input audio signals. For example, the audio component 810 includes a Microphone (MIC) configured to receive external audio signals when the electronic device 800 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal may further be stored in the memory 804 or transmitted via the communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

The I/O interface 812 provides an interface between the processing component 802 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to: a home button, a volume button, a start button, and a lock button.

The sensor assembly 814 includes one or more sensors for providing various aspects of state assessment for the electronic device 800. For example, the sensor assembly 814 may detect an open/closed state of the electronic device 800, the relative positioning of components, such as a display and keypad of the electronic device 800, the sensor assembly 814 may also detect a change in the position of the electronic device 800 or a component of the electronic device 800, the presence or absence of user contact with the electronic device 800, orientation or acceleration/deceleration of the electronic device 800, and a change in the temperature of the electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 further includes a Near Field Communication (NFC) module to facilitate short-range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the electronic device 800 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, micro-controllers, microprocessors or other electronic components for performing the above-described vehicle travel control method.

In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions, such as the memory 804 including instructions, executable by the processor 820 of the electronic device 800 to perform the vehicle travel control method described above is also provided. For example, the non-transitory computer readable storage medium may be a ROM, a Random Access Memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like.

In another exemplary embodiment, a computer program product is also provided, which comprises a computer program executable by a programmable device, the computer program having code portions for performing the vehicle travel control method described above when executed by the programmable device.

Other embodiments of the disclosure will be apparent to those skilled in the art from consideration of the specification and practice of the disclosure. This application is intended to cover any variations, uses, or adaptations of the disclosure following, in general, the principles of the disclosure and including such departures from the present disclosure as come within known or customary practice in the art to which the disclosure pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the disclosure being indicated by the following claims.

It will be understood that the present disclosure is not limited to the precise arrangements that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the appended claims.

Claims

1. A vehicle travel control method characterized by comprising:

acquiring a first Bayer image output by a shooting device on a vehicle;

controlling the vehicle to run according to the target detection result;

the image perception model is obtained by the following steps:

training a machine learning model based on the target spliced image to obtain a trained image perception model;

the obtaining of the target mosaic image under the aerial view according to the Bayer image sample around the view comprises:

constructing a pre-spliced image in a Bayer format under an aerial view based on camera parameters corresponding to the shooting device;

determining green color values of pixels belonging to other color channels in the looking-around Bayer image sample based on pixel values of a green color channel in the looking-around Bayer image sample;

for each first target pixel belonging to a green channel in the pre-spliced image, determining a first mapping pixel of the first target pixel, which is mapped on the looking-around Bayer image sample; and

2. The vehicle travel control method according to claim 1, wherein the determining a third target pixel in the looking-around bayer image sample based on the coordinate information of the second mapped pixel includes:

when the coordinate information of the second mapping pixel represents that the second mapping pixel is located in the middle of four first pixels which belong to the same color channel as the second target pixel, determining the four first pixels as the third target pixel;

3. The vehicle travel control method according to claim 1, wherein the determining a third target pixel in the looking-around bayer image sample based on the coordinate information of the second mapped pixel includes:

4. The vehicle travel control method according to claim 1, wherein the determining a third target pixel in the looking-around bayer image sample based on the coordinate information of the second mapped pixel includes:

when the coordinate information of the second mapping pixel represents that the second mapping pixel is located on a third pixel which belongs to the same color channel as the second target pixel, determining the third pixel and a fourth pixel as the third target pixel, wherein the fourth pixel is a pixel which has a distance from the third pixel within a preset distance threshold value and belongs to the same color channel as the third pixel;

5. The vehicle travel control method according to claim 1, wherein the training samples of the image perception model include samples obtained by:

6. The vehicle travel control method according to claim 5, wherein the training samples of the image perception model further include one obtained by:

the method further comprises the following steps:

7. The vehicle travel control method according to claim 6, wherein the adjusting an image perception model trained using the second bayer image based on the third bayer image to obtain an adjusted image perception model includes:

8. The vehicle travel control method according to claim 7, wherein the processing the third bayer image based on an image processing technique to obtain a fourth bayer image includes:

when the image processing technique is the image flipping, adding at least one column and at least one row of pixels at the boundary of the third bayer image; turning over the third Bayer image with the added pixels according to a preset turning angle, and cutting the added pixel rows and pixel columns at the boundary corresponding to the turned over third Bayer image to obtain a fourth Bayer image;

alternatively, the first and second electrodes may be,

9. The vehicle travel control method according to claim 8, wherein the correcting coordinates of a third target detection frame in the fourth bayer image to obtain a corrected fourth bayer image includes:

10. A vehicle travel control device characterized by comprising:

a control module configured to control the vehicle to travel according to the target detection result;

the input module includes:

the system comprises a sample acquisition unit, a detection unit and a display unit, wherein the sample acquisition unit is configured to acquire all-round-looking Bayer image samples around a vehicle, and each all-round-looking Bayer image sample is marked with a first target detection frame and a category to which the first target detection frame belongs;

the training unit is configured to train a machine learning model based on the target spliced image to obtain a trained image perception model;

the image stitching unit includes:

a first calculation unit configured to determine, based on pixel values of a green channel in the look-around bayer image sample, green color values of pixels belonging to other color channels in the look-around bayer image sample;

a first coordinate determination unit, configured to determine, for each first target pixel belonging to a green channel in the pre-stitched image, a first mapping pixel of the first target pixel mapped on the looking-around bayer image sample;

a second coordinate determination unit configured to determine, for each second target pixel belonging to a red channel and a blue channel in the pre-stitched image, coordinate information of a second mapped pixel of the second target pixel mapped on the look-around bayer image sample;

11. An electronic device, comprising:

a processor;

a memory for storing processor-executable instructions;

wherein the processor is configured to execute executable instructions stored in the memory to implement the vehicle travel control method of any one of claims 1-9.

12. A computer-readable storage medium, on which computer program instructions are stored, characterized in that the program instructions, when executed by a processor, implement the steps of the vehicle travel control method according to any one of claims 1-9.

13. A vehicle characterized by comprising the vehicle travel control apparatus according to claim 10.