CN116777752A

CN116777752A - Image correction method, image correction device, electronic device, storage medium, and vehicle

Info

Publication number: CN116777752A
Application number: CN202210226782.5A
Authority: CN
Inventors: 刘锋; 勾晓菲; 李涛
Original assignee: Beijing Co Wheels Technology Co Ltd
Current assignee: Beijing Co Wheels Technology Co Ltd
Priority date: 2022-03-09
Filing date: 2022-03-09
Publication date: 2023-09-19

Abstract

The invention provides an image correction method, an image correction device, electronic equipment, a storage medium and a vehicle, wherein the method comprises the following steps: acquiring an original image; the original image is used as a texture map and is attached to a set arc-shaped surface; the method comprises the steps of setting a plane in which straight lines at two side edges in an arc-shaped surface are located to be parallel to an image plane in which an original image is located, and setting an intersection line between the arc-shaped surface and a tangent plane perpendicular to the image plane to be an arc; and projecting the mapped set arc-shaped surface according to the virtual camera parameters and the target projection parameters to obtain a corrected target image. Therefore, the method does not need to calibrate each camera, the video content of each frame is used as a texture map to be attached to the set arc surface, and then the corrected target image can be obtained by projecting the mapped set arc surface, so that the workload can be reduced, the efficiency can be improved, the display of a near object is relatively normal under the condition of keeping a large field angle, and the discomfort of watching by a user is reduced.

Description

Image correction method, image correction device, electronic device, storage medium, and vehicle

Technical Field

The present invention relates to the field of image processing technologies, and in particular, to an image correction method, an image correction device, an electronic device, a storage medium, and a vehicle.

Background

The angle of view of the conventional camera is relatively small, so that a large blind area inevitably exists, and practical requirements are difficult to meet. The fisheye camera is a lens with a field angle larger than 120 degrees, and the characteristics of large field of view can be widely applied to many fields in life, but people can feel uncomfortable due to serious distortion of the photographed image. Therefore, it is necessary to restore the image captured by the fisheye camera to an image that is suitable for human viewing, that is, to correct distortion of the fisheye image.

In the related art, when correcting the distortion of the fisheye image, the fisheye cameras are often required to be calibrated in advance, and if each fisheye camera is calibrated, the workload is large and the efficiency is low. In addition, according to the distortion correction effect of the related art, on the premise of maintaining a large field of view, an object near the vehicle appears to be abnormally large, and a user feels uncomfortable to watch.

Disclosure of Invention

The present invention aims to solve at least one of the technical problems in the related art to some extent.

Therefore, a first object of the present invention is to provide an image correction method, which does not need to calibrate a fisheye camera in advance, only needs to paste each frame of video content as a texture map on a set arc surface, and projects the mapped set arc surface according to a target virtual camera parameter and a target projection parameter to obtain a corrected target image, so that the workload can be reduced, the efficiency can be improved, the display of a near object is relatively normal under the condition of keeping a large viewing angle, and the discomfort of watching by a user is reduced.

A second object of the present invention is to provide an image correction apparatus.

A third object of the present invention is to propose an electronic device.

A fourth object of the present invention is to propose a non-transitory computer readable storage medium.

A fifth object of the present invention is to propose a vehicle.

A sixth object of the invention is to propose a computer programme product.

To achieve the above object, an embodiment of a first aspect of the present invention provides a method, including: acquiring an original image; the original image is used as a texture map and is attached to a set arc-shaped surface; the plane of the straight line at the edges of the two sides of the set arc-shaped surface is parallel to the image plane of the original image, and the intersection line between the set arc-shaped surface and the tangent plane perpendicular to the image plane is arc-shaped; and projecting the mapped set arc-shaped surface according to the target virtual camera parameters and the target projection parameters to obtain a corrected target image.

According to the image correction method provided by the embodiment of the invention, an original image is firstly obtained, the original image is used as a texture map and is attached to a set arc surface, wherein the plane of a straight line positioned at two side edges in the set arc surface is parallel to the image plane of the original image, the intersection line between the arc surface and a tangent plane perpendicular to the image plane is set to be an arc, and then the set arc surface after mapping is projected according to the target virtual camera parameters and the target projection parameters so as to obtain a corrected target image. Therefore, the method does not need to calibrate the camera in advance, only needs to paste each frame of video content as texture mapping on the set arc surface, and projects the mapped set arc surface according to the target virtual camera parameters and the target projection parameters, so that corrected target images can be obtained, the workload can be reduced, the efficiency can be improved, the display of near objects is relatively normal under the condition of keeping a large viewing angle, and the discomfort of watching by users is reduced.

In addition, the image correction method provided by the embodiment of the first aspect of the present invention may further have the following additional technical features:

according to one embodiment of the present invention, after the original image to be corrected is acquired, the method further includes: determining the radius of a visual area in the original image; and determining the radius of the set arc-shaped surface according to the radius of the visual area.

According to one embodiment of the present invention, the pasting the original image as a texture map on a set arc surface includes: determining the origin of a three-dimensional coordinate system where the set arc-shaped surface is located according to the center of a visible area in the original image; wherein, the X axis in the three-dimensional coordinate system is parallel to the image plane, and the Y axis is parallel to and perpendicular to the X axis and coincides with the straight line of the set arc surface; determining texture coordinates corresponding to the target point according to X-axis and Y-axis coordinates of the target point on the set arc surface in the three-dimensional coordinate system; and according to the texture coordinates corresponding to the target points, attaching the pixel units with the texture coordinates in the texture coordinate system of the original image to the target points on the set arc-shaped surface.

According to one embodiment of the present invention, the determining the corresponding texture coordinates according to the X-axis and Y-axis coordinates of the target point on the set arc surface in the three-dimensional coordinate system includes: according to the coordinates (x ₀ ，y ₀ ) Translating X-axis and Y-axis coordinates (X, Y) of the target point on the set arc surface in the three-dimensional coordinate system; and normalizing the coordinate obtained after translation according to the resolution of the original image to obtain the texture coordinate.

According to an embodiment of the present invention, the projecting the mapped set arc surface according to the target virtual camera parameter and the target projection parameter to obtain a corrected target image includes: determining a target optical axis direction of the virtual camera according to the direction parameter in the target virtual camera parameters; determining a target field angle of the virtual camera according to the angle parameter in the target projection parameters; and based on the target field angle along the target optical axis direction, projecting the mapped set arc-shaped surface to the virtual camera so as to obtain a corrected target image.

According to one embodiment of the present invention, the projecting the mapped set arc surface to the virtual camera based on the target field angle along the target optical axis direction to obtain the corrected target image includes: determining a virtual camera matrix according to the target optical axis direction of the virtual camera; determining a projection matrix between an imaging coordinate system of the virtual camera and a three-dimensional coordinate system where the arc-shaped surface is positioned according to a target field angle of the virtual camera; and based on the virtual camera matrix and the projection matrix, projecting the mapped set arc-shaped surface to the virtual camera along the optical axis direction so as to obtain a corrected target image.

According to an embodiment of the present invention, before projecting the mapped set arc surface according to the target virtual camera parameter and the target projection parameter to obtain the corrected target image, the method further includes: and determining the target virtual camera parameters and the target projection parameters according to the model of the fisheye camera of the acquired original image.

According to an embodiment of the present invention, before projecting the mapped set arc surface according to the target virtual camera parameter and the target projection parameter to obtain the corrected target image, the method further includes: projecting the set arc-shaped surface after mapping according to the multiple candidate virtual camera parameters and the multiple candidate projection parameters to obtain a reference image; determining a selected image from each of the candidate virtual camera parameters and reference images corresponding to each of the candidate projection parameters in response to a user operation; and respectively taking the candidate virtual camera parameters and the candidate projection parameters corresponding to the selected image as the corresponding target virtual camera parameters and the target projection parameters.

To achieve the above object, a second aspect of the present invention provides an image correction apparatus, comprising: the acquisition module is used for acquiring an original image; the mapping module is used for mapping the original image serving as a texture mapping to a set arc surface; the plane of the straight line at the edges of the two sides of the set arc-shaped surface is parallel to the image plane of the original image, and the intersection line between the set arc-shaped surface and the tangent plane perpendicular to the image plane is arc-shaped; the first projection module is used for projecting the set arc-shaped surface after mapping according to the target virtual camera parameters and the target projection parameters so as to obtain a corrected target image.

According to the image correction device provided by the embodiment of the invention, an original image is acquired through the acquisition module, the original image is used as a texture map through the map pasting module, and is pasted to a set arc-shaped surface, wherein the plane of a straight line positioned at two side edges in the set arc-shaped surface is parallel to the image plane of the original image, the intersection line between the set arc-shaped surface and a tangent plane perpendicular to the image plane is arc-shaped, and the set arc-shaped surface after mapping is projected through the first projection module according to the target virtual camera parameters and the target projection parameters so as to obtain a corrected target image. Therefore, the device does not need to calibrate the camera in advance, only needs to paste each frame of video content as texture mapping on the set arc surface, projects the mapped set arc surface according to the target virtual camera parameters and the target projection parameters, and can obtain corrected target images.

In addition, the image correction device according to the second aspect of the present invention may further have the following additional technical features:

According to an embodiment of the present invention, the image correction apparatus further includes: the first determining module is used for determining the radius of a visual area in an original image after the original image to be corrected is acquired; and the second determining module is used for determining the radius of the set arc-shaped surface according to the radius of the visual area.

According to one embodiment of the invention, the mapping module comprises: the first determining unit is used for determining the origin of the three-dimensional coordinate system where the set arc-shaped surface is located according to the center of the visible area in the original image; wherein, the X axis in the three-dimensional coordinate system is parallel to the image plane, and the Y axis is parallel to and perpendicular to the X axis and coincides with the straight line of the set arc surface; the second determining unit is used for determining corresponding texture coordinates according to X-axis coordinates and Y-axis coordinates of the target point on the set arc surface in the three-dimensional coordinate system; and the mapping unit is used for attaching the pixel units with the texture coordinates in the texture coordinate system where the original image is positioned to the target point according to the texture coordinates.

According to an embodiment of the present invention, the second determining unit includes: a translation subunit for translating the image according to the coordinates (x ₀ ，y ₀ ) Translating X-axis and Y-axis coordinates (X, Y) of the target point in the three-dimensional coordinate system; and the processing subunit is used for normalizing the coordinates obtained after translation according to the resolution of the original image so as to obtain texture coordinates corresponding to the target point.

According to one embodiment of the invention, the first projection module comprises: a third determining unit, configured to determine a target optical axis direction of the virtual camera according to a direction parameter in the target virtual camera parameters; a fourth determining unit, configured to determine a target field angle of the virtual camera according to an angle parameter in the target projection parameters; and the first projection unit is used for projecting the mapped set arc-shaped surface to the virtual camera based on the target view angle along the target optical axis direction so as to obtain a corrected target image.

According to an embodiment of the present invention, the first projection unit includes: a first determining subunit, configured to determine a virtual camera matrix according to a target optical axis direction of the virtual camera; the second determining subunit is used for determining a projection matrix between an imaging coordinate system of the virtual camera and a three-dimensional coordinate system where the arc-shaped surface is located according to the target field angle of the virtual camera; and the projection subunit is used for projecting the mapped set arc-shaped surface to the virtual camera along the direction of the target optical axis based on the virtual camera matrix and the projection matrix so as to obtain a corrected target image.

According to an embodiment of the present invention, the image correction apparatus further includes: and the third determining module is used for determining the target virtual camera parameters and the target projection parameters according to the model of the fisheye camera of the acquired original image.

According to an embodiment of the present invention, the image correction apparatus further includes: the second projection module is also used for projecting the set arc-shaped surface after mapping according to the plurality of candidate virtual camera parameters and the plurality of candidate projection parameters so as to obtain a reference image; a fourth determining module, configured to determine, in response to a user operation, a selected image from each of the candidate virtual camera parameters and reference images corresponding to each of the candidate projection parameters; and the setting module is used for respectively taking the candidate virtual camera parameters and the candidate projection parameters corresponding to the selected image as the corresponding target virtual camera parameters and the target projection parameters.

To achieve the above object, an embodiment of a third aspect of the present invention provides an electronic device, including: a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, for implementing the above-described image correction method.

According to the electronic equipment provided by the embodiment of the invention, by executing the image correction method, a camera does not need to be calibrated in advance, each frame of video content is only required to be used as a texture map to be attached to the set arc surface, and the corrected target image can be obtained by projecting the set arc surface after mapping according to the target virtual camera parameters and the target projection parameters, so that the workload can be reduced, the efficiency can be improved, the display of a near object is relatively normal under the condition of keeping a large viewing angle, and the discomfort of watching by a user is reduced.

To achieve the above object, a fourth aspect of the present invention provides a non-transitory computer-readable storage medium having stored thereon a computer program which, when executed by a processor, implements the above-described image correction method.

According to the non-transitory computer readable storage medium, by executing the image correction method, a camera is not required to be calibrated in advance, each frame of video content is only required to be used as a texture map, the texture map is attached to a set arc surface, and the corrected target image can be obtained by projecting the set arc surface after mapping according to target virtual camera parameters and target projection parameters, so that the workload can be reduced, the efficiency can be improved, the display of a near object is relatively normal under the condition of keeping a large visual angle, and the discomfort of watching by a user is reduced.

To achieve the above object, a fifth aspect of the present invention provides a vehicle including the above electronic device.

When the fisheye camera is used, the vehicle provided by the embodiment of the invention does not need to calibrate the fisheye camera in advance, only needs to paste each frame of video content as a texture map on the set arc surface, and projects the mapped set arc surface according to the target virtual camera parameters and the target projection parameters, so that a corrected target image can be obtained, the workload can be reduced, the efficiency can be improved, the display of a near object is relatively normal under the condition of keeping a large visual angle, and the discomfort of watching by a user is reduced.

To achieve the above object, an embodiment of a sixth aspect of the present invention provides a computer program product, which when executed by an instruction processor in the computer program product, performs the above-mentioned image correction method.

According to the computer program product of the embodiment of the invention, by executing the image correction method, a camera does not need to be calibrated in advance, each frame of video content is only required to be used as a texture map to be attached to a set arc surface, and the corrected target image can be obtained by projecting the set arc surface after mapping according to the target virtual camera parameters and the target projection parameters, so that the workload can be reduced, the efficiency can be improved, the display of a near object is relatively normal under the condition of keeping a large viewing angle, and the discomfort of watching by a user is reduced.

Additional aspects and advantages of the invention will be set forth in part in the description which follows and, in part, will be obvious from the description, or may be learned by practice of the invention.

Drawings

The foregoing and/or additional aspects and advantages of the invention will become apparent and readily appreciated from the following description of the embodiments, taken in conjunction with the accompanying drawings, in which:

fig. 1 is a flowchart of an image correction method according to an embodiment of the present invention;

FIG. 2 is a schematic diagram of image correction according to one embodiment of the invention;

FIG. 3 is a flow chart of an image correction method according to one embodiment of the invention;

FIG. 4 is a flow chart of an image correction method according to another embodiment of the present invention;

FIG. 5 is a flow chart of an image correction method according to yet another embodiment of the present invention;

fig. 6 is a block schematic diagram of an image correction apparatus according to an embodiment of the present invention.

Detailed Description

Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein like or similar reference numerals refer to like or similar elements or elements having like or similar functions throughout. The embodiments described below by referring to the drawings are illustrative and intended to explain the present invention and should not be construed as limiting the invention.

An image correction method, an image correction apparatus, an electronic device, and a non-transitory computer-readable storage medium according to an embodiment of the present invention are described below with reference to the accompanying drawings.

The angle of view of the traditional camera is smaller, and a large blind area inevitably exists when the traditional camera is used in a vehicle-mounted vision system, so that the actual requirement is difficult to meet. The fisheye camera is a lens with the field angle larger than 120 degrees, wherein the camera with the field angle of about 180 degrees can meet the field angle requirement of the vehicle-mounted vision system. The large field of view of the fisheye camera makes it widely applicable in many fields of life, but it is uncomfortable for people to look at due to the severe distortion of the captured image. In case we need to use the distorted image contents, we need to correct the images to be in line with people's habit. The image shot by the fisheye lens is restored to be an image which accords with human watching, and the image is called distortion correction.

In the related art, the calculation amount of fisheye image correction is large, and the real-time video stream distortion correction needs to occupy serious resources. For embedded devices, achieving real-time performance requirements is costly because it is limited by the processor's main frequency. The processing of the open graphics library (Open Graphics Library) by the GPU (Graphics Processing Unit) can break through the limitation of the main frequency of the processor and realize high-performance computing. At present, most video correction algorithms based on a CPU (Central Processing Unit ) are used for processing, and often cannot meet the requirement of correcting fish-eye video in real time or have extremely high CPU occupancy rate, so that the operation of the whole system is greatly influenced. The GPU has a massive parallel throughput system structure with multiple concurrent threads, has strong flexibility, and can meet the application requirements of fisheye video correction. In the related art, distortion correction of a fisheye image often needs to calibrate a fisheye camera in advance to determine internal parameters and distortion parameters of the fisheye camera. If each fisheye camera needs to be calibrated, the workload is large and the efficiency is low. In addition, according to the distortion correction effect of the related art, on the premise of maintaining a large field of view, an object near the vehicle appears to be abnormally large, and a user feels uncomfortable to watch.

Therefore, the embodiment of the invention provides an image correction method, which is capable of obtaining corrected images by sticking each frame of video content on a set arc surface as a texture map without calibrating a fisheye camera in advance and projecting the mapped set arc surface according to target virtual camera parameters and target projection parameters, so that the workload can be reduced, the efficiency can be improved, the display of nearby objects is relatively normal under the condition of keeping a large viewing angle, and the discomfort of watching by a user is reduced.

Fig. 1 is a schematic diagram of an image correction method according to an embodiment of the present invention.

As shown in fig. 1, the image correction method according to the embodiment of the present invention includes the following steps:

s101, acquiring an original image.

In this step, the original image is an original image to be corrected, which is directly photographed using a fisheye camera, and the shape of the original image to be corrected may be circular, as shown in fig. 2.

S102, taking an original image as a texture map, and attaching the texture map to a set arc surface; the plane of the straight line at the edges of the two sides of the arc-shaped surface is set to be parallel to the image plane of the original image, and the intersection line between the arc-shaped surface and the tangent plane perpendicular to the image plane is set to be arc-shaped.

It should be noted that the set arc surface may be a semi-cylindrical arc surface, a hemispherical arc surface, a semi-elliptical arc surface, etc.

For example, when the set arc surface is a semi-cylindrical arc surface, a cylindrical model or a tubular model is first built, the cylindrical model or the tubular model is cut in the length direction, and the cut section is parallel to the image plane where the original image is located, so as to form the set arc surface in the application. In the application, each frame of video content shot by the fisheye camera is used as a texture map to be attached to the set arc-shaped surface, as shown by a dotted arrow shown in fig. 2.

And S103, projecting the set arc-shaped surface after mapping according to the target virtual camera parameters and the target projection parameters so as to obtain a corrected target image.

In this step, the target virtual camera parameters may include a position of the virtual camera, an orientation of the virtual camera head, and a direction of the virtual camera line of sight; the target projection parameters may include an angle of view of the view volume, an aspect ratio of the view volume, a distance of a nearest arced face of the view volume to the observer, and a distance of a farthest arced face of the view volume to the observer. When the fisheye camera imaging of the model is corrected for the first time, the set arc surface after mapping can be projected according to a plurality of candidate virtual camera parameters and a plurality of candidate projection parameters to obtain a reference image, a user selects a proper image from the plurality of reference images, and at the moment, the candidate camera parameters and the candidate projection parameters corresponding to the image selected by the user are respectively used as a target virtual camera parameter and a target projection parameter; when the fisheye camera imaging of the model is not corrected for the first time, the model of the fisheye camera, a prestored corresponding relation table of the model of the fisheye camera, the target virtual camera parameters and the target projection parameters are acquired, and the target virtual camera parameters and the target projection parameters can be directly determined according to the model of the fisheye camera through table lookup.

After the target virtual camera parameters and the target projection parameters are acquired, the mapped set arc-shaped surface is rendered on a screen according to the target virtual camera parameters and the target projection parameters, so that a corrected target image is obtained, and a user can watch corrected video on the screen.

Therefore, the image correction method of the embodiment of the invention directly establishes the set arc surface model without calibrating the fisheye camera in advance, uses each frame of video content as the texture map, sticks the texture map to the set arc surface, and projects the set arc surface after each frame of map by changing the line-of-sight direction and the angle of view of the virtual camera at the center of the set arc surface so as to obtain the target video with the de-distortion effect. Therefore, under the condition of keeping a large visual angle, the display of a near object is relatively normal, and the discomfort of watching by a user is reduced.

Fig. 3 is a flow chart of an image correction method according to an embodiment of the present invention.

As shown in fig. 3, the image correction method according to the embodiment of the present invention includes the following steps:

s301, acquiring an original image.

In this step, the original image is an original image to be corrected, which is directly photographed using a fisheye camera, and the edge of the original image to be corrected may be black, as shown in fig. 2.

S302, determining the radius of the visual area for the visual area in the original image.

In this step, the visible area is a closed loop area including a black edge, which is visible to the human eye, the closed loop area being circular, and the radius r of the visible area being the radius r of the circular area.

S303, determining the radius of the set arc-shaped surface according to the radius of the visible area.

In this step, a radius R of the arc surface is defined as a radius R of the visible region.

S304, determining an origin of a three-dimensional coordinate system where the set arc-shaped surface is located according to the center of a visible area in the original image; wherein, X axis in the three-dimensional coordinate system is parallel to the image plane, and Y axis is parallel to and perpendicular to X axis with the straight line that sets for the arcwall face coincidence.

In this step, the pixel coordinates that locate the center of the original image are (x 0, y0, 0), the radius is r, and the units are pixels.

Establishing a three-dimensional coordinate system where a set cambered surface is located and setting the cambered surface: the center of the original image is taken as the origin of coordinates, the right side of the original image is taken as the positive direction of the X axis, the lower side of the original image is taken as the positive direction of the Y axis, the inward direction of the original image is taken as the positive direction of the Z axis, and a three-dimensional coordinate system is established, wherein the unit is 1 pixel. Taking the center of an original image as the center of a semi-cylinder, taking the Y-axis direction as the center axis, and establishing a semi-cylinder with the radius r, wherein the arc surface of the semi-cylinder is a set arc surface.

S305, determining corresponding texture coordinates according to X-axis coordinates and Y-axis coordinates of the target point on the set arc surface in the three-dimensional coordinate system.

In this step, the coordinates (x) in the texture coordinate system in which the original image is located (the texture coordinate system is the coordinate system in which the x-axis and the y-axis as indicated in fig. 2 are located) can be determined based on the center of the visible region ₀ ，y ₀ ) And translating X-axis and Y-axis coordinates (X, Y) of the target point on the set arc surface in the three-dimensional coordinate system, wherein the coordinates obtained after translation are corresponding texture coordinates of the X-axis and Y-axis coordinates of the target point on each set arc surface in the three-dimensional coordinate system.

In order to enhance the texture of the target image, the coordinate values obtained after translation can be normalized according to the resolution of the original image. For example, w and h are resolutions of the original image in the X-axis and the Y-axis, respectively, and the coordinate values obtained after translation are normalized, that is, the value of the X-axis of the coordinate values obtained after translation is divided by the resolution w of the original image in the X-axis, and the value of the Y-axis of the coordinate values obtained after translation is divided by the resolution h of the original image in the Y-axis.

S306, sticking the pixel units with texture coordinates in the texture coordinate system of the original image to the target point according to the texture coordinates.

That is, the pixel unit corresponding to each texture coordinate in the texture coordinate system in the visible area of the original image is attached to the corresponding position of the set arc surface.

S307, determining the target optical axis direction of the virtual camera according to the direction parameters in the target virtual camera parameters.

The target virtual camera parameters may include, among other things, the position of the virtual camera, the orientation of the virtual camera head, and the direction of the virtual camera's line of sight. A virtual camera matrix, namely a filling function gluLookAt (eye, at, up), can be constructed according to target virtual camera parameters, wherein the position eye of the fisheye camera is the origin of a virtual camera imaging coordinate system, namely the position of the virtual camera; the sight line at of the fish-eye camera is the direction of the optical axis of the virtual camera, namely the direction of the sight line of the virtual camera, the included angle between the direction of the sight line at of the fish-eye camera and the X axis is a, and the included angle between the direction of the sight line at of the fish-eye camera and the Z axis is b; the head up of the fish-eye camera faces in the negative direction of the Y-axis, i.e. the direction of the virtual camera head (i.e. the direction of the virtual camera shutter).

After the pixel unit having the texture coordinates in the texture coordinate system where the original image is located is attached to the target point, the blank area of the image with the arc-shaped surface set after the mapping is filled by using a filling function.

S308, determining the target field angle of the virtual camera according to the angle parameter in the target projection parameters.

The target projection parameters may include, among other things, an angle of view of the view volume, an aspect ratio of the view volume, a distance of a nearest arced face of the view volume to the observer, and a distance of a farthest arced face of the view volume to the observer. A projection matrix gluPerrective (fovy, aspect, zNear, zFar) can be established through the projection parameters, wherein fovy is the angle of the view body; aspect is the aspect ratio of the view volume; zNear is the distance of the observer to the nearest arc surface of the view volume; zFar is the distance of the viewer to the furthest arc of the view volume. The target field angle of the virtual camera may be determined from the angle fovy of the field of view of the view volume.

S309, projecting the mapped set arc surface to the virtual camera based on the target view angle along the target optical axis direction so as to obtain a corrected target image.

In this step, after the target view angle and the target optical axis direction are acquired, a virtual camera matrix gluLookAt (eye, at, up) is determined according to the target optical axis direction of the virtual camera, and a projection matrix gluperselect (fovy, aspect, zNear, zFar) between an imaging coordinate system of the virtual camera and a three-dimensional coordinate system where the arc surface is located is determined according to the target view angle of the virtual camera, and then the mapped set arc surface is projected to the virtual camera along the target optical axis direction of the virtual camera based on the virtual camera matrix and the projection matrix, so as to obtain a corrected target image.

Therefore, the image correction method of the embodiment of the invention can directly establish the set arc surface model without calibrating the fisheye camera in advance, uses each frame of video content as the texture map, pastes the texture map on the set arc surface, and projects the set arc surface after each frame of map by changing the line of sight direction and the angle of view of the virtual camera at the center of the set arc surface so as to obtain the target video with the de-distortion effect. Therefore, under the condition of keeping a large visual angle, the display of a near object is relatively normal, and the discomfort of watching by a user is reduced.

Fig. 4 is a flowchart of an image correction method according to another embodiment of the present invention.

As shown in fig. 4, the image correction method according to the embodiment of the present invention includes the following steps:

s401, acquiring an original image.

S402, taking an original image as a texture map, and attaching the texture map to a set arc surface; the plane of the straight line at the edges of the two sides of the arc-shaped surface is set to be parallel to the image plane of the original image, and the intersection line between the arc-shaped surface and the tangent plane perpendicular to the image plane is set to be arc-shaped.

S403, determining target virtual camera parameters and target projection parameters according to the model of the fisheye camera of the acquired original image.

In this embodiment, the corresponding relation tables of the fisheye cameras with different models, the target virtual camera parameters and the target projection parameters need to be stored in advance, so that when the fisheye camera imaging with the model is not corrected for the first time, the relation table is directly called, and the target virtual camera parameters and the target projection parameters are obtained according to the model of the fisheye camera.

S404, projecting the set arc-shaped surface after mapping according to the target virtual camera parameters and the target projection parameters so as to obtain a corrected target image.

Note that, the contents of the steps S401, S402, and S404 are described in the steps S101 to S103.

Fig. 5 is a flowchart of an image correction method according to still another embodiment of the present invention.

As shown in fig. 5, the image correction method according to the embodiment of the present invention includes the steps of:

s501, acquiring an original image.

S502, taking an original image as a texture map, and attaching the texture map to a set arc surface; the plane of the straight line at the edges of the two sides of the arc-shaped surface is set to be parallel to the image plane of the original image, and the intersection line between the arc-shaped surface and the tangent plane perpendicular to the image plane is set to be arc-shaped.

S503, projecting the set arc-shaped surface after mapping according to the plurality of candidate virtual camera parameters and the plurality of candidate projection parameters to obtain a reference image.

S504, determining a selected image from the reference images corresponding to each candidate virtual camera parameter and each candidate projection parameter in response to the user operation.

S505, the candidate virtual camera parameters and the candidate projection parameters corresponding to the selected image are respectively used as the corresponding target virtual camera parameters and the target projection parameters.

That is, the user may use a, b in different virtual camera parameters and fovy in projection parameters, render the texture map with the set arc surface on the screen by using OpenGL, so as to obtain a corresponding reference image, determine whether the reference image is satisfied, and if one of the reference images is satisfied, namely, the reference image is used as a selected image, respectively using candidate virtual camera parameters and candidate projection parameters corresponding to the selected image as corresponding target virtual camera parameters and target projection parameters; if the reference image does not have the image satisfactory to the user, the user can use a, b in different virtual camera parameters and the fovy in the projection parameters to continue projecting the original image until a user-satisfactory image is projected, and at this time, the candidate virtual camera parameter and the candidate projection parameter corresponding to the image can be respectively used as the corresponding target virtual camera parameter and the target projection parameter.

S506, projecting the set arc-shaped surface after mapping according to the target virtual camera parameters and the target projection parameters so as to obtain a corrected target image.

Note that, the content of the steps S501, S502, and S506 is described in the steps S101 to S103.

Therefore, the fisheye video is directly mapped to the half-cylinder surface with the same pixel resolution and the half-cylinder surface is rendered in an opengl mode. Different distortion correction effects are obtained by modifying the direction of the visual line of the virtual camera, and different distortion correction effects can be realized by selecting different directions of the visual line of the virtual camera and virtual visual angle sizes according to different scenes. Thus, each fish-eye camera does not need to be calibrated in advance; the method can be completed only by using opengl without relying on third party library functions such as opencv and the like; video data can be processed in real time; on the premise of keeping a large visual field, objects close to the vehicle can be normal.

In summary, according to the image correction method of the embodiment of the present invention, an original image is obtained first, and the original image is used as a texture map and is attached to a set arc surface, wherein a plane where straight lines at two side edges in the set arc surface are located is parallel to an image plane where the original image is located, an intersection line between the set arc surface and a tangential plane perpendicular to the image plane is arc, and then the set arc surface after mapping is projected according to a target virtual camera parameter and a target projection parameter, so as to obtain a corrected target image. Therefore, the method does not need to calibrate the camera in advance, only needs to paste each frame of video content as texture mapping on the set arc surface, and projects the mapped set arc surface according to the target virtual camera parameters and the target projection parameters, so that corrected target images can be obtained, the workload can be reduced, the efficiency can be improved, the display of near objects is relatively normal under the condition of keeping a large viewing angle, and the discomfort of watching by users is reduced.

As shown in fig. 6, an image correction apparatus 600 according to an embodiment of the present invention includes: an acquisition module 601, a mapping module 602 and a first projection module 603.

The acquiring module 601 is configured to acquire an original image. The mapping module 602 is configured to map the original image to a set arc surface as a texture mapping; the plane of the straight line at the edges of the two sides of the arc-shaped surface is set to be parallel to the image plane of the original image, and the intersection line between the arc-shaped surface and the tangent plane perpendicular to the image plane is set to be arc-shaped. The first projection module 603 is configured to project the mapped set arc surface according to the target virtual camera parameter and the target projection parameter, so as to obtain a corrected target image.

According to an embodiment of the present invention, the image correction apparatus further includes: a first determination module and a second determination module. The first determining module is used for determining the radius of the visual area in the original image after the original image to be corrected is acquired. The second determining module is used for determining the radius of the set arc-shaped surface according to the radius of the visible area.

According to one embodiment of the invention, the mapping module 602 includes: the device comprises a first determining unit, a second determining unit and a mapping unit. The first determining unit is used for determining an origin of a three-dimensional coordinate system where the set arc-shaped surface is located according to the center of a visible area in the original image; the X axis in the three-dimensional coordinate system is parallel to the image plane, and the Y axis is parallel to and perpendicular to the straight line overlapped with the set arc-shaped surface. The second determining unit is used for determining corresponding texture coordinates according to X-axis coordinates and Y-axis coordinates of the target point on the set arc-shaped surface in the three-dimensional coordinate system. And the mapping unit is used for pasting the pixel units with the texture coordinates in the texture coordinate system where the original image is positioned to the target point according to the texture coordinates.

According to an embodiment of the present invention, the second determining unit includes: a translation subunit and a processing subunit. Wherein the translation subunit is configured to translate the coordinates (x ₀ ，y ₀ ) The X-axis and Y-axis coordinates (X, Y) of the target point in the three-dimensional coordinate system are translated. The processing subunit is used for normalizing the coordinate obtained after translation according to the resolution of the original image so as to obtain the texture coordinate corresponding to the target point.

According to one embodiment of the invention, the first projection module 603 comprises: the third determining unit, the fourth determining unit and the first projection unit. The third determining unit is used for determining the target optical axis direction of the virtual camera according to the direction parameters in the target virtual camera parameters. The fourth determining unit is used for determining the target field angle of the virtual camera according to the angle parameter in the target projection parameters. The first projection unit is used for projecting the mapped set arc-shaped surface to the virtual camera along the optical axis direction based on the target field angle so as to obtain a corrected target image.

According to an embodiment of the present invention, a first projection unit includes: the first determining subunit and the second determining subunit and the projection subunit. The first determining subunit is configured to determine a virtual camera matrix according to an optical axis direction of the virtual camera. And the second determination subunit is used for determining a projection matrix between the imaging coordinate system of the virtual camera and the three-dimensional coordinate system where the arc-shaped surface is positioned according to the target field angle of the virtual camera. And the projection subunit is used for projecting the mapped set arc-shaped surface to the virtual camera along the optical axis direction based on the virtual camera matrix and the projection matrix so as to obtain a corrected target image.

According to an embodiment of the present invention, the image correction apparatus further includes: and a third determination module. The third determining module is used for determining target virtual camera parameters and target projection parameters according to the model of the fisheye camera of the acquired original image.

According to an embodiment of the present invention, the image correction apparatus further includes: the device comprises a second projection module, a fourth determination module and a setting module. The second projection module is further configured to project the mapped set arc surface according to the plurality of candidate virtual camera parameters and the plurality of candidate projection parameters, so as to obtain a reference image. The fourth determining module is used for determining selected images from the candidate virtual camera parameters and the reference images corresponding to the candidate projection parameters in response to user operation. The setting module is used for respectively taking the candidate virtual camera parameters and the candidate projection parameters corresponding to the selected image as corresponding target virtual camera parameters and target projection parameters.

It should be noted that, details not disclosed in the image correction device in the embodiment of the present invention are referred to in the image correction method in the embodiment of the present invention, and details thereof are not described herein.

Based on the embodiment, the invention further provides electronic equipment.

The electronic equipment of the embodiment of the invention comprises: a processor and a memory; wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory, for implementing the above-described image correction method.

Based on the above embodiments, the present invention also proposes a non-transitory computer-readable storage medium.

The non-transitory computer-readable storage medium of the embodiment of the present invention stores thereon a computer program that, when executed by a processor, implements the above-described image correction method.

Based on the embodiment, the invention further provides a vehicle.

The vehicle provided by the embodiment of the invention comprises the electronic equipment.

Based on the above embodiments, the present invention also proposes a computer program product.

The computer program product of the embodiment of the present invention, when executed by an instruction processor in the computer program product, performs the above-described image correction method.

In the description of the present specification, a description referring to terms "one embodiment," "some embodiments," "examples," "specific examples," or "some examples," etc., means that a particular feature, structure, material, or characteristic described in connection with the embodiment or example is included in at least one embodiment or example of the present invention. In this specification, schematic representations of the above terms are not necessarily directed to the same embodiment or example. Furthermore, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. Furthermore, the different embodiments or examples described in this specification and the features of the different embodiments or examples may be combined and combined by those skilled in the art without contradiction.

Furthermore, the terms "first," "second," and the like, are used for descriptive purposes only and are not to be construed as indicating or implying a relative importance or implicitly indicating the number of technical features indicated. Thus, a feature defining "a first" or "a second" may explicitly or implicitly include at least one such feature. In the description of the present invention, the meaning of "plurality" means at least two, for example, two, three, etc., unless specifically defined otherwise.

Any process or method descriptions in flow charts or otherwise described herein may be understood as representing modules, segments, or portions of code which include one or more executable instructions for implementing specific logical functions or steps of the process, and additional implementations are included within the scope of the preferred embodiment of the present invention in which functions may be executed out of order from that shown or discussed, including substantially concurrently or in reverse order from that shown or discussed, depending on the functionality involved, as would be understood by those reasonably skilled in the art of the embodiments of the present invention.

Logic and/or steps represented in the flowcharts or otherwise described herein, e.g., a ordered listing of executable instructions for implementing logical functions, can be embodied in any computer-readable medium for use by or in connection with an instruction execution system, apparatus, or device, such as a computer-based system, processor-containing system, or other system that can fetch the instructions from the instruction execution system, apparatus, or device and execute the instructions. For the purposes of this description, a "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transport the program for use by or in connection with the instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of the computer-readable medium would include the following: an electrical connection (electronic device) having one or more wires, a portable computer diskette (magnetic device), a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber device, and a portable compact disc read-only memory (CDROM). Additionally, the computer-readable medium may even be paper or other suitable medium upon which the program is printed, as the program may be electronically captured, via, for instance, optical scanning of the paper or other medium, then compiled, interpreted or otherwise processed in a suitable manner, if necessary, and then stored in a computer memory.

It is to be understood that portions of the present invention may be implemented in hardware, software, firmware, or a combination thereof. In the above-described embodiments, the various steps or methods may be implemented in software or firmware stored in a memory and executed by a suitable instruction execution system. As with the other embodiments, if implemented in hardware, may be implemented using any one or combination of the following techniques, as is well known in the art: discrete logic circuits having logic gates for implementing logic functions on data signals, application specific integrated circuits having suitable combinational logic gates, programmable Gate Arrays (PGAs), field Programmable Gate Arrays (FPGAs), and the like.

Those of ordinary skill in the art will appreciate that all or part of the steps carried out in the method of the above-described embodiments may be implemented by a program to instruct related hardware, and the program may be stored in a computer readable storage medium, where the program when executed includes one or a combination of the steps of the method embodiments.

In addition, each functional unit in the embodiments of the present invention may be integrated in one processing module, or each unit may exist alone physically, or two or more units may be integrated in one module. The integrated modules may be implemented in hardware or in software functional modules. The integrated modules may also be stored in a computer readable storage medium if implemented as software functional modules and sold or used as a stand-alone product.

The above-mentioned storage medium may be a read-only memory, a magnetic disk or an optical disk, or the like. While embodiments of the present invention have been shown and described above, it will be understood that the above embodiments are illustrative and not to be construed as limiting the invention, and that variations, modifications, alternatives and variations may be made to the above embodiments by one of ordinary skill in the art within the scope of the invention.

Claims

1. An image correction method, characterized by comprising the steps of:

acquiring an original image;

the original image is used as a texture map and is attached to a set arc-shaped surface; the plane of the straight line at the edges of the two sides of the set arc-shaped surface is parallel to the image plane of the original image, and the intersection line between the set arc-shaped surface and the tangent plane perpendicular to the image plane is arc-shaped;

and projecting the mapped set arc-shaped surface according to the target virtual camera parameters and the target projection parameters to obtain a corrected target image.

2. The method of claim 1, wherein after the capturing the original image, further comprising:

determining the radius of a visual area according to the visual area in the original image;

And determining the radius of the set arc-shaped surface according to the radius of the visual area.

3. The method of claim 1, wherein said attaching the original image as a texture map to a set arced surface comprises:

determining the origin of a three-dimensional coordinate system where the set arc-shaped surface is located according to the center of the visible area in the original image; wherein, the X axis in the three-dimensional coordinate system is parallel to the image plane, and the Y axis is parallel to and perpendicular to the X axis and coincides with the straight line of the set arc surface;

determining corresponding texture coordinates according to X-axis coordinates and Y-axis coordinates of the target point on the set arc surface in the three-dimensional coordinate system;

and according to the texture coordinates, attaching the pixel units with the texture coordinates in the texture coordinate system where the original image is positioned to the target point.

4. A method according to claim 3, wherein said determining corresponding texture coordinates from X-axis and Y-axis coordinates of the target point on the set arc surface in the three-dimensional coordinate system comprises:

according to the coordinates (x ₀ ，y ₀ ) Translating X-axis and Y-axis coordinates (X, Y) of the target point in the three-dimensional coordinate system;

and normalizing the coordinate obtained after translation according to the resolution of the original image to obtain the texture coordinate.

5. The method according to any one of claims 1-4, wherein projecting the mapped set arc surface according to the target virtual camera parameters and the target projection parameters to obtain the corrected target image includes:

determining a target optical axis direction of the virtual camera according to the direction parameter in the target virtual camera parameters;

determining a target field angle of the virtual camera according to the angle parameter in the target projection parameters;

and based on the target field angle along the target optical axis direction, projecting the mapped set arc-shaped surface to the virtual camera so as to obtain a corrected target image.

6. The method of claim 5, wherein projecting the mapped set arc surface to the virtual camera based on the target field angle along the target optical axis direction to obtain the corrected target image, comprises:

determining a virtual camera matrix according to the target optical axis direction of the virtual camera;

Determining a projection matrix between an imaging coordinate system of the virtual camera and a three-dimensional coordinate system where the arc-shaped surface is positioned according to a target field angle of the virtual camera;

and based on the virtual camera matrix and the projection matrix, projecting the mapped set arc-shaped surface to the virtual camera along the direction of the target optical axis so as to obtain a corrected target image.

7. The method according to any one of claims 1-4, wherein the projecting the mapped set arc surface according to the target virtual camera parameters and the target projection parameters to obtain the corrected target image further comprises:

and determining the target virtual camera parameters and the target projection parameters according to the model of the fisheye camera of the acquired original image.

8. The method according to any one of claims 1-4, wherein the projecting the mapped set arc surface according to the target virtual camera parameters and the target projection parameters to obtain the corrected target image further comprises:

projecting the set arc-shaped surface after mapping according to the multiple candidate virtual camera parameters and the multiple candidate projection parameters to obtain a reference image;

Determining a selected image from each of the candidate virtual camera parameters and reference images corresponding to each of the candidate projection parameters in response to a user operation;

and respectively taking the candidate virtual camera parameters and the candidate projection parameters corresponding to the selected image as the corresponding target virtual camera parameters and the target projection parameters.

9. An image correction apparatus, comprising:

the acquisition module is used for acquiring an original image;

the mapping module is used for mapping the original image serving as a texture mapping to a set arc surface; the plane of the straight line at the edges of the two sides of the set arc-shaped surface is parallel to the image plane of the original image, and the intersection line between the set arc-shaped surface and the tangent plane perpendicular to the image plane is arc-shaped;

the first projection module is used for projecting the set arc-shaped surface after mapping according to the target virtual camera parameters and the target projection parameters so as to obtain a corrected target image.

10. An electronic device, comprising:

a processor and a memory;

wherein the processor runs a program corresponding to the executable program code by reading the executable program code stored in the memory for implementing the image correction method according to any one of claims 1 to 8.

11. A non-transitory computer readable storage medium having stored thereon a computer program, which when executed by a processor implements the image correction method according to any of claims 1-8.

12. A vehicle, characterized by comprising: the electronic device of claim 10.