CN114299159A

CN114299159A - Simulation image generation method and device and electronic equipment

Info

Publication number: CN114299159A
Application number: CN202111624717.XA
Authority: CN
Inventors: 张安春; 李涵
Original assignee: Beijing Wuyi Vision Digital Twin Technology Co ltd
Current assignee: Beijing Wuyi Vision Digital Twin Technology Co ltd
Priority date: 2021-12-28
Filing date: 2021-12-28
Publication date: 2022-04-08

Abstract

The disclosure relates to a simulation image generation method and device and electronic equipment. The simulation image generation method comprises the following steps: calibrating a virtual calibration environment according to basic parameters of the simulation camera equipment to obtain a first calibration result; calibrating the defect parameters of the simulation camera equipment according to the first calibration result to obtain a second calibration result; calibrating the brightness parameter of the simulation camera equipment according to the second calibration result to obtain a third calibration result; calibrating the high-frequency parameters of the simulation camera equipment according to the third calibration result to obtain a fourth calibration result; calibrating the chromaticity parameters of the simulation camera equipment according to the fourth calibration result to obtain a fifth calibration result; and rendering the target object according to the first calibration result, the second calibration result, the third calibration result, the fourth calibration result and the fifth calibration result to generate a simulation image. The method and the device realize the calibration of the parameters of the simulation camera equipment and improve the quality of the simulation image.

Description

Simulation image generation method and device and electronic equipment

Technical Field

The disclosure belongs to the technical field of computer information processing, and particularly relates to a method, a device and a system for generating a simulation image and electronic equipment.

Background

The development of the current computer vision mainly depends on the improvement of a deep learning algorithm, and the deep learning algorithm needs a large amount of data to train a model; however, the real data acquisition and marking cost is high, and data under various special weather (such as rain, snow, fog and the like), special illumination and extreme working conditions are more difficult to acquire; therefore, simulation data becomes an important source of deep learning training and testing data.

The simulation of the camera device can be used for obtaining a large amount of simulated image data, the simulation of the camera device generally constructs a virtual three-dimensional model of a real world through a graphics technology, adds color and optical attributes to the three-dimensional model according to the real material and texture of an object, and renders the three-dimensional world from the view angle of the camera device by adopting a rendering and illumination technology based on physics.

In this way, the simulation scene has different Image quality results due to different lens characteristics, photoelectric conversion process of the Image sensor and processing process of the ISP (Image Signal Processor) in the viewing angles of different brands and models of Image capturing devices; the computer vision depth learning algorithm extracts target features depending on image quality details, so that the difference between the simulated camera device image set and the real camera device image set is large, and the simulated images are not real enough.

Disclosure of Invention

The embodiment of the disclosure aims to provide a method, a device and a system for generating a simulation image and an electronic device, which realize the calibration of parameters of a simulation camera device and improve the quality of the simulation image.

In a first aspect, an embodiment of the present disclosure provides a method for generating a simulation image, where the method includes:

calibrating a virtual calibration environment according to basic parameters of the simulation camera equipment to obtain a first calibration result;

calibrating the defect parameters of the simulation camera equipment according to the first calibration result to obtain a second calibration result;

calibrating the brightness parameter of the simulation camera equipment according to the second calibration result to obtain a third calibration result;

calibrating the high-frequency parameters of the simulation camera equipment according to the third calibration result to obtain a fourth calibration result;

calibrating the chromaticity parameters of the simulation camera equipment according to the fourth calibration result to obtain a fifth calibration result;

and rendering the target object according to the first calibration result, the second calibration result, the third calibration result, the fourth calibration result and the fifth calibration result to generate a simulation image.

Optionally, the basic parameters include: resolution and field angle; calibrating basic parameters of the simulation camera equipment in the virtual calibration environment to obtain a first calibration result, wherein the calibration result comprises the following steps:

determining the resolution of the simulated camera equipment according to the resolution of the real camera equipment;

determining the field angle of the simulation camera equipment according to the field angle of the real camera equipment;

determining a first target distance between the simulation camera shooting equipment and the virtual calibration board in the virtual calibration environment according to the field angle;

under the condition that the light intensity of the central point of the virtual calibration plate is the same as the real light intensity, obtaining the light intensity of the virtual point light source according to the light intensity of the central point of the virtual calibration plate and a second target distance between the virtual point light source and the virtual calibration plate, wherein the second target distance is a preset distance;

and determining the first target distance, the second target distance and the light intensity of the virtual point light source as a first calibration result.

Optionally, the simulated camera device is located on a connecting line between the virtual point light source and the center of the virtual calibration board, and the first target distance is calculated according to the field angle and all images taken by the simulated camera device to the virtual calibration board.

Optionally, the defect parameters include: a distortion parameter and a brightness uniformity parameter; according to the first calibration result, calibrating the defect parameters of the simulation camera equipment to obtain a second calibration result, which comprises the following steps:

under the condition of obtaining a first calibration result, calibrating the distortion parameter of the simulation camera equipment according to a preset distortion parameter calibration algorithm to obtain a radial distortion parameter and a tangential distortion parameter of the simulation camera equipment;

acquiring a brightness uniformity parameter of the simulation camera equipment according to a brightness variation gradient of a first target image obtained by shooting a pure white calibration board in a virtual calibration environment by the simulation camera equipment;

and determining the radial distortion parameter, the tangential distortion parameter and the brightness uniformity parameter of the simulation camera equipment as a second calibration result.

Optionally, the luminance parameters include: exposure values and tone response curves;

according to the second calibration result, calibrating the brightness parameter of the simulation camera device to obtain a third calibration result, including:

under the condition of obtaining a second calibration result, shooting an exposure value of a second target image obtained by a first test card according to the simulated camera equipment in the virtual calibration environment, obtaining the exposure value of the simulated camera equipment and obtaining a tone response curve of the simulated camera equipment according to the gradient of the tone response curve of the second target image;

and determining the exposure value and the tone response curve of the simulation camera equipment as a third calibration result.

Optionally, the high frequency parameters include: sharpness, noise and dispersion purple edge parameters; according to the third calibration result, calibrating the high-frequency parameters of the simulation camera equipment to obtain a fourth calibration result, which comprises the following steps:

under the condition of obtaining a third calibration result, according to the definition of a third target image obtained by shooting a second test card by the simulation camera equipment in a virtual calibration environment, obtaining the definition of the simulation camera equipment and according to the dispersion purple boundary of the third target image, obtaining the dispersion purple boundary parameter of the simulation camera equipment;

shooting the noise of a fourth target image obtained by a third test card according to the fact that the simulation camera shooting device is in the virtual calibration environment, and obtaining the noise of the simulation camera shooting device;

and determining the definition, noise and dispersion purple boundary parameters of the simulation camera equipment as a fourth calibration result.

Optionally, the chrominance parameters include: a white balance parameter and a color reduction parameter; according to the fourth calibration result, calibrating the chromaticity parameter of the simulation camera device to obtain a fifth calibration result, including:

under the condition of obtaining a fourth calibration result, according to a white balance parameter and a color restoration parameter of a fifth target image obtained by shooting a fourth test card by the simulation camera device in a virtual calibration environment, obtaining the white balance parameter and the color restoration parameter of the simulation camera device;

and determining the white balance parameter and the color restoration parameter of the simulation camera equipment as a fifth calibration result.

In a second aspect, an embodiment of the present disclosure provides a simulation image generation apparatus, including:

the first obtaining module is used for calibrating the virtual calibration environment according to basic parameters of the simulation camera equipment to obtain a first calibration result;

the second obtaining module is used for calibrating the defect parameters of the simulation camera equipment according to the first calibration result to obtain a second calibration result;

the third obtaining module is used for calibrating the brightness parameter of the simulation camera equipment according to the second calibration result to obtain a third calibration result;

the fourth obtaining module is used for calibrating the high-frequency parameters of the simulation camera equipment according to the third calibration result to obtain a fourth calibration result;

a fifth obtaining module, configured to calibrate the chromaticity parameter of the simulated image capturing apparatus according to the fourth calibration result, so as to obtain a fifth calibration result;

and the image generation module is used for rendering the target object according to the first calibration result, the second calibration result, the third calibration result, the fourth calibration result and the fifth calibration result to generate a simulation image.

Optionally, the basic parameters include: resolution and field angle; the first obtaining module includes:

the first obtaining submodule is used for determining the resolution of the simulation camera shooting equipment according to the resolution of the real camera shooting equipment;

the second obtaining submodule is used for determining the field angle of the simulation camera shooting equipment according to the field angle of the real camera shooting equipment;

the third obtaining submodule is used for determining a first target distance between the simulation camera shooting equipment and the virtual calibration plate in the virtual calibration environment according to the field angle;

the fourth obtaining submodule is used for obtaining the light intensity of the virtual point light source according to the light intensity of the central point of the virtual calibration plate and a second target distance between the virtual point light source and the virtual calibration plate under the condition that the light intensity of the central point of the virtual calibration plate is the same as the real light intensity, wherein the second target distance is a preset distance;

and the fifth obtaining submodule is used for determining the first target distance, the second target distance and the light intensity of the virtual point light source as a first calibration result.

Optionally, the defect parameters include: a distortion parameter and a brightness uniformity parameter; the second obtaining module includes:

the sixth obtaining submodule is used for calibrating the distortion parameter of the simulated camera equipment according to a preset distortion parameter calibration algorithm under the condition of obtaining the first calibration result, so as to obtain the radial distortion parameter and the tangential distortion parameter of the simulated camera equipment;

the seventh obtaining submodule is used for obtaining a brightness uniformity parameter of the simulation camera shooting equipment according to a brightness change gradient of a first target image obtained by shooting the pure white calibration plate in the virtual calibration environment of the simulation camera shooting equipment;

and the eighth obtaining submodule is used for determining the radial distortion parameter, the tangential distortion parameter and the brightness uniformity parameter of the simulation camera equipment as a second calibration result.

Optionally, the luminance parameters include: exposure values and tone response curves; the third obtaining module includes:

the ninth obtaining submodule is used for obtaining an exposure value of a second target image obtained by shooting the first test card according to the simulation camera shooting equipment in the virtual calibration environment under the condition of obtaining the second calibration result, obtaining the exposure value of the simulation camera shooting equipment and obtaining a tone response curve of the simulation camera shooting equipment according to the gradient of the tone response curve of the second target image;

and the tenth obtaining submodule is used for determining the exposure value and the tone response curve of the simulation camera equipment as a third calibration result.

Optionally, the high frequency parameters include: sharpness, noise and dispersion purple edge parameters; the fourth obtaining module includes:

the eleventh obtaining submodule is used for obtaining the definition of a third target image obtained by shooting the second test card according to the simulation camera shooting equipment in the virtual calibration environment under the condition of obtaining the third calibration result, obtaining the definition of the simulation camera shooting equipment and obtaining the chromatic dispersion purple boundary parameter of the simulation camera shooting equipment according to the chromatic dispersion purple boundary of the third target image;

the twelfth obtaining submodule is used for obtaining the noise of the simulated camera equipment by shooting the noise of the fourth target image obtained by the third test card according to the simulated camera equipment in the virtual calibration environment;

and the thirteenth obtaining submodule is used for determining the definition, noise and dispersion purple fringe parameters of the simulation camera equipment as a fourth calibration result.

Optionally, the chrominance parameters include: a white balance parameter and a color reduction parameter; the fifth obtaining module includes:

a fourteenth obtaining submodule, configured to, when a fourth calibration result is obtained, obtain a white balance parameter and a color restoration parameter of the simulated imaging device according to a white balance parameter and a color restoration parameter of a fifth target image obtained by shooting the fourth test card by the simulated imaging device in the virtual calibration environment;

and a fifteenth obtaining submodule for determining the white balance parameter and the color restoration parameter of the simulation image pickup apparatus as a fifth calibration result.

In a third aspect, the disclosed embodiments provide an electronic device comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, the program or instructions, when executed by the processor, implementing the steps of the simulation image generation method according to the first aspect.

In a fourth aspect, the disclosed embodiments provide a readable storage medium on which a program or instructions are stored, which when executed by a processor, implement the steps of the simulation image generation method according to the first aspect.

In the embodiment of the disclosure, the simulation camera device is calibrated based on the real camera device, the full process from the calibration to the generation of the simulation image of the image quality of the simulation camera device is established, the simulation image which is closer to the image detail effect obtained by the real camera device is simulated by the simulation camera device, and the quality of the simulation image is improved.

Drawings

FIG. 1 is a flow chart of a method for generating a simulation image according to an embodiment of the present disclosure;

FIG. 2 is a schematic layout diagram of a virtual calibration environment provided by an embodiment of the present disclosure;

fig. 3 is a schematic diagram of a calibration flow of a simulation image capturing apparatus in a virtual calibration environment according to an embodiment of the present disclosure;

FIG. 4 is a schematic structural diagram of a simulation image generation apparatus provided in an embodiment of the present disclosure;

fig. 5 is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure;

fig. 6 is a schematic hardware structure diagram of an electronic device provided by an embodiment of the disclosure.

Detailed Description

Technical solutions in the embodiments of the present disclosure will be clearly described below with reference to the drawings in the embodiments of the present disclosure, and it is obvious that the described embodiments are some embodiments of the present disclosure, but not all embodiments. All other embodiments derived by one of ordinary skill in the art from the embodiments disclosed herein are intended to be within the scope of the present disclosure.

The terms first, second and the like in the description and in the claims of the present disclosure are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It will be appreciated that the data so used may be interchanged under appropriate circumstances such that embodiments of the disclosure may be practiced other than those illustrated or described herein, and that the objects identified as "first," "second," etc. are generally a class of objects and do not limit the number of objects, e.g., a first object may be one or more. In addition, "and/or" in the specification and claims means at least one of connected objects, a character "/" generally means that a preceding and succeeding related objects are in an "or" relationship.

The disclosed embodiment mainly relates to generation of a simulation image of simulation camera equipment, based on real camera equipment, the simulation camera equipment is calibrated from the image quality of the simulation image, wherein the image quality comprises distortion, white balance, color restoration, dynamic range, signal-to-noise ratio, definition and the like of the simulation image, through calibration of the simulation camera equipment, the image quality difference of the simulation camera equipment and the real camera equipment at a microscopic pixel level is reduced, and the simulation image obtained through the calibrated simulation camera equipment is closer to the real image obtained by the real camera equipment;

the virtual calibration environment where the simulation camera shooting device is located and the scene rendered by the target object can be used for simulating a three-dimensional virtual space and can also be used for simulating a two-dimensional virtual space, and the three-dimensional virtual space or the two-dimensional virtual space can be an open space. The virtual calibration environment and the scene rendered by the target object can be used for simulating a real environment in reality, and certainly, in a possible implementation manner, the virtual calibration environment and the scene rendered by the target object can also correspond to a real scene, the scene rendered by the target object in the virtual calibration environment can also support time control, can be freely adjusted at night and in the daytime, and can also support a weather system, and can be set in sunny days, rainy days, snowy days, foggy days and the like.

It should be noted that, the scene rendered by the virtual calibration environment and the target object may include a plurality of virtual objects, where the virtual object may be an avatar in the virtual scene that is used to represent a user or other avatars, and the avatar may be in any form, for example, an artificial camera device, a virtual calibration plate, a virtual point light source, a virtual calibration dark box, and the like, which is not limited in this disclosure. The scene rendered by the virtual calibration environment and the target object can comprise a plurality of virtual objects, and each virtual object has a shape, a structure and a volume in the virtual scene and occupies a part of the space in the virtual scene.

The method, the apparatus, the system and the electronic device for generating a simulation image according to the embodiments of the present disclosure are described in detail below with reference to the accompanying drawings.

Fig. 1 is a flowchart of a method for generating a simulation image according to an embodiment of the present disclosure, and referring to fig. 1, the method may include the following steps:

step 101, calibrating a virtual calibration environment according to basic parameters of a simulation camera device to obtain a first calibration result;

the basic parameters here include resolution and field angle; the resolution is used for determining the fineness of the details of the simulation image, and the higher the resolution of the simulation image is, the more pixels are contained, and the clearer the simulation image is; the field angle is used to determine the field range of an optical instrument (image pickup apparatus), and the larger the field angle, the larger the field of view, and the smaller the optical magnification, that is, when a target object exceeds the field angle of the optical instrument (image pickup apparatus), the optical instrument (image pickup apparatus) does not pick up the target object.

In an optional embodiment of the present disclosure, step 101 includes:

step 1011, determining the resolution of the simulated camera equipment according to the resolution of the real camera equipment; here, preferably, the resolution of the real image pickup apparatus may be determined as the resolution of the simulated image pickup apparatus;

step 1012, determining the field angle of the simulated camera equipment according to the field angle of the real camera equipment; here, preferably, the field angle of the real image pickup apparatus may be determined as the field angle of the simulated image pickup apparatus; for example, when the real image of the real image capturing apparatus is 1920 × 1080 pixels and the horizontal angle of view is 60 °, it is preferable to determine that the resolution of the simulation image capturing apparatus is 1920 × 1080 and the horizontal angle of view is 60 °;

step 1013, determining a first target distance between the simulation camera device and the virtual calibration board in the virtual calibration environment according to the field angle;

here, a description is given of a virtual calibration environment in which the simulation imaging apparatus is located:

fig. 2 is a schematic layout diagram of a virtual calibration environment provided in the embodiment of the present disclosure, referring to fig. 2, the virtual calibration environment includes a virtual calibration camera bellows, a virtual calibration plate disposed inside the virtual calibration camera bellows, a virtual point light source, and a simulation camera device;

a virtual calibration dark box is built in the virtual calibration environment, and the virtual calibration dark box is preferably made of absolute black body physical materials and is an ideal box body which is completely black and does not reflect any light; the virtual calibration plate arranged in the virtual calibration camera bellows cannot be interfered by illumination of the external environment of the virtual calibration camera bellows, namely the virtual calibration camera bellows can completely isolate any illumination interference outside the box body;

a virtual calibration plate is arranged in the virtual calibration dark box, the virtual calibration plate is preferably made of a standard lambertian physical material, and when light irradiates the virtual calibration plate, the virtual calibration plate can be completely diffusely reflected; in addition, in order to not generate specular reflection highlight which influences the shooting effect on the simulation camera equipment, the virtual calibration plate adopts a reflection type, and the reflectivity of each block of the virtual calibration plate to different wavelengths corresponding to three primary colors red, green and blue is consistent with that of the real calibration plate, so that the finally presented color is ensured to be consistent with that of the real calibration plate; in addition, the size of the virtual calibration plate is preferably 90cm by 60cm, and the size of the black and white grid on each virtual calibration plate is preferably 6cm by 6 cm;

a virtual point light source is arranged at a position which is a certain distance away from the virtual calibration plate; the virtual point light source preferably adopts a white point light source which meets the power spectrum distribution of D65, namely a white standard light source specified by the International Commission on illumination and an ideal light source with the color temperature of 6500K.

In an optional embodiment of the present disclosure, the virtual point light source is located on a connecting line between the virtual point light source and the center of the virtual calibration board, and the first target distance is calculated according to the angle of view and all images captured by the virtual calibration board by the virtual camera. Here, the field angle of the pseudo image pickup device is determined to be the same as the field angle of the real image pickup device, and the first target distance between the pseudo image pickup device and the virtual calibration board can be calculated by setting the pseudo field angle at a position where all images of the virtual calibration board can be photographed.

1014, under the condition that the light intensity of the central point of the virtual calibration plate is the same as the real light intensity, obtaining the light intensity of the virtual point light source according to the light intensity of the central point of the virtual calibration plate and a second target distance between the virtual point light source and the virtual calibration plate, wherein the second target distance is a preset distance;

in the embodiment, the illumination intensity of any point in the virtual calibration dark box is in inverse proportion to the square of the distance between the point and the virtual point light source; setting the light intensity of the central point of the virtual calibration plate to be the same as the real light intensity based on the inverse proportion relation and the preset second target distance, and calculating to obtain the illumination intensity (namely the light intensity) of the virtual point light source;

in addition, it should be noted that, because the virtual calibration environment is a virtual space and the simulated camera device has no entity, the simulated camera device in the virtual calibration environment has no entity, and cannot cause illumination shielding to light rays generated by the virtual point light source, and the imaging plane of the simulated camera device is parallel to the plane of the virtual calibration plate;

here, the actual light intensity is preferably obtained by: in the real world environment, the illuminometer is arranged at the central point of the real calibration plate, and the real light intensity is obtained through calibration.

In one implementation example, the light intensity of the virtual point light source is obtained by the light intensity of the central point of the virtual calibration plate and the second target distance from the virtual point light source to the virtual calibration plate, and the method specifically includes:

the virtual point light source is fixedly arranged at a position 3.162 meters away from the center of the virtual calibration plate, namely the preset second target distance is 3.162 meters; when the real light intensity is 4w candela (cd), then the virtual light intensity isThe luminous intensity of the point light source should also be 4w candela (cd), since the solid angle of the whole point light source of the virtual point light source is 4 pi steradians (sr), the luminous flux is about 50w lumens (lm), wherein 1 lumen (lm) is equal to the luminous flux emitted by one virtual point light source with 1 candela (cd) uniform luminous intensity in 1 steradian (sr) unit solid angle, i.e. 1lm ═ 1cd 1 sr; as can be seen, the intensity of the light at 1 meter from the virtual point source is 4 Wlux (cd sr/m)²) (ii) a From the spherical surface area, the spherical surface area corresponding to 1 steradian at S meter is S²(m²) Then, the illumination intensity of the virtual calibration plate at a distance of 3.162 meters (the second target distance) from the virtual point light source is about 4000 lux (cd × sr/m)²)。

Step 1015, determining the first target distance, the second target distance and the light intensity of the virtual point light source as a first calibration result.

In this embodiment, the first target distance, the second target distance, and the light intensity of the virtual point light source are determined as a first calibration result, in the subsequent calibration process, the first target distance, the second target distance, and the light intensity of the virtual point light source are all fixed and unchangeable values, and similarly, the resolution and the field angle determined in steps 1011 and 1012 are also fixed and unchangeable basic parameters in the subsequent calibration process.

Step 102, calibrating defect parameters of the simulation camera equipment according to the first calibration result to obtain a second calibration result;

the defect parameters are preferably distortion parameters and brightness uniformity parameters; the distortion parameter is used for determining the position of each block of the virtual calibration plate in the simulated image; the distortion parameters include radial distortion coefficients (k1, k2, and k3) and tangential distortion coefficients (p1 and p 2); the radial distortion generated by the radial distortion coefficient generally occurs in the process of converting a camera coordinate system into an image physical coordinate system, and the tangential distortion generated by the tangential distortion coefficient generally occurs in the process of manufacturing the video camera due to the fact that the plane of the photosensitive element is not parallel to the lens; the brightness uniformity parameter can reflect the vignetting and vignetting conditions of the simulation image and is used for determining the brightness conditions of each block in the simulation image; when the uniformity degree of the brightness uniformity parameter is within the target limit range, the light sensing of the simulation image is uniform, and the quality of the generated simulation image is improved.

In an optional embodiment of the present disclosure, step 102 includes:

step 1021, under the condition that the first calibration result is obtained, calibrating the distortion parameter of the simulation camera equipment according to a preset distortion parameter calibration algorithm to obtain a radial distortion parameter and a tangential distortion parameter of the simulation camera equipment;

the preset distortion parameter calibration algorithm may be an algorithm for calibrating the distortion parameter through a geometric distortion measurement standard, may also be a zhang chessboard calibration method, and may also be other modes capable of calibrating the distortion parameter of the simulation camera device, which is not limited in this application.

When the error between the radial distortion parameter of the real camera shooting equipment and the radial distortion parameter of the simulation camera shooting equipment is within a first preset range, and the error between the tangential distortion parameter of the real camera shooting equipment and the tangential distortion parameter of the simulation camera shooting equipment is within a second preset range, it can be determined that the bright distortion parameter simulation effect of the simulation camera shooting equipment at the moment is good.

Step 1022, obtaining a brightness uniformity parameter of the simulated camera device according to a brightness variation gradient of a first target image obtained by shooting a pure white calibration board by the simulated camera device in a virtual calibration environment;

here, the calibrating the luminance uniformity parameter may specifically include: shooting a pure white calibration board through simulation camera equipment to obtain a first target image; and calculating the brightness change gradient according to the center of the first target image to obtain the brightness uniformity parameter of the simulation camera equipment.

When the error between the brightness uniformity parameter of the real camera device and the brightness uniformity parameter of the simulated camera device is within the third preset range, it can be determined that the simulation effect of the brightness uniformity parameter of the simulated camera device at the moment is good.

And step 1023, determining the radial distortion parameter, the tangential distortion parameter and the brightness uniformity parameter of the simulation camera equipment as a second calibration result.

In this embodiment, the calibrated radial distortion parameter, tangential distortion parameter and luminance uniformity parameter are determined as the second calibration result, and in the subsequent calibration process, on the basis that the first target distance, the second target distance, the light intensity of the virtual point light source and the basic parameter in step 101 are fixed and unchanged, the radial distortion parameter, the tangential distortion parameter and the luminance uniformity parameter are also fixed and unchanged values.

103, calibrating the brightness parameter of the simulation camera equipment according to the second calibration result to obtain a third calibration result;

here, the luminance parameter of the simulation imaging apparatus preferably includes an exposure value and a tone response curve; the exposure value and the tone response curve reflect the brightness of the simulated image generated by the simulated camera equipment, and influence the overall brightness, contrast and dynamic range value of the simulated image;

the exposure value EV is a base 2 logarithmic scale system in the camera, and is generally controlled by the shutter, aperture and sensitivity of the camera, and can be represented by the formula:

calculating to obtain;

where N is the aperture (f-number) of the simulation imaging apparatus; t is an exposure time (shutter) of the simulation imaging apparatus in seconds; as can be seen from the above formula, as the exposure value increases, the shutter speed will be faster or the f-number of the aperture will be larger; the first-gear exposure of the simulation camera equipment is changed every time the exposure value is increased by 1, namely the exposure amount is halved;

the contrast response curve is used for describing the process of photoelectrically converting and processing an illumination value of an object on a sensor plane into a brightness value of each pixel by the simulation camera shooting equipment; the tone response curve is preferably an s-curve, the tone response curve in the form of an s-curve being insensitive to lower and higher numerical responses and sensitive to intermediate numerical responses.

The exposure value and the tone response curve need to be calibrated simultaneously, and the exposure value can influence the shape of the tone response curve, for example, when the exposure value is in an underexposed state, the value of the tone response curve can be located in a dark part of the curve.

In an optional embodiment of the present disclosure, step 103 includes:

step 1031, under the condition that the second calibration result is obtained, obtaining an exposure value of the second target image obtained by shooting the first test card (such as a virtual gray scale card) according to the simulation camera shooting equipment in the virtual calibration environment, obtaining the exposure value of the simulation camera shooting equipment and obtaining a tone response curve of the simulation camera shooting equipment according to the gradient of the tone response curve of the second target image;

here, based on the second calibration result, the basic parameters and the defect parameters of the simulation camera device are set; the test of the exposure value and the tone response curve of the simulation camera equipment belongs to the test of the photoelectric conversion function, and in a virtual calibration environment, the test process of the photoelectric conversion function of the simulation camera equipment is a black box test process, so that the calibration of the exposure value and the tone response curve is realized by simulating a second target image with the same exposure value and the same tone response curve slope as the real camera equipment; when the virtual gray scale card is shot to obtain an exposure value of a second target image, the second target image is preferably a gray scale image;

step 1032 determines the exposure value and the tone response curve of the simulation imaging apparatus as a third calibration result.

In this embodiment, the calibrated exposure value and the tone response curve are determined as the third calibration result, and in the subsequent calibration process, the exposure value and the tone response curve are also fixed and unchangeable values on the basis that the basic parameter and the defect parameter are fixed and unchangeable.

Step 104, calibrating the high-frequency parameters of the simulation camera equipment according to the third calibration result to obtain a fourth calibration result;

here, the high-frequency parameters of the simulation imaging apparatus preferably include sharpness, noise, and dispersion purple-fringing parameters;

the definition is used for reflecting the edge quality of the simulation Image, the definition can be influenced by a lens, a sensor and an ISP (Image Signal Processor) algorithm of the simulation camera equipment, and the definition is preferably calibrated through a virtual e-SFR (Small form-factor pluggable) graphic card;

the generation of noise is generated in the photoelectric conversion process;

the chromatic dispersion purple boundary parameter is related to chromatic dispersion generated by refraction of light rays with different wavelengths after passing through the lens, and the chromatic dispersion purple boundary parameter is a chromatic aberration purple boundary phenomenon generated at a high-contrast edge of an image after color interpolation of a sensor mask filter;

it should be noted that the above-mentioned sharpness, noise and dispersion purple fringing parameters affect each other, and the three should be calibrated at the same time, but not according to a certain sequence.

In an optional embodiment of the present disclosure, step 104 includes:

step 1041, under the condition of obtaining the third calibration result, according to the definition of the third target image obtained by shooting the second test card (such as the virtual e-SFR graphic card) by the simulation camera device in the virtual calibration environment, obtaining the definition of the simulation camera device and according to the chromatic dispersion purple boundary of the third target image, obtaining the chromatic dispersion purple boundary parameter of the simulation camera device;

the definition is calibrated through the virtual e-SFR graphic card under the condition of calibrating the basic parameters, the defect parameters and the high-frequency parameters, and the dispersion purple edge parameters of the simulation camera equipment can be obtained by combining the definition calibration, namely calibrating the color difference condition of the edge of the simulation image through the virtual e-SFR graphic card.

Step 1042, according to the noise of the fourth target image obtained by shooting the third test card (such as a virtual test card) in the virtual calibration environment of the simulation camera device, obtaining the noise of the simulation camera device;

the virtual test card can be used for calibrating noise and the signal-to-noise ratio of the noise, and the signal-to-noise ratio can be calculated according to the formula db-10 lg (s/n) of the noise, wherein db is the decibel number of the noise, and s/n is the signal-to-noise ratio of the noise.

And 1043, determining the definition, noise and dispersion purple fringe parameters of the simulation camera equipment as a fourth calibration result.

In this embodiment, the calibrated sharpness, noise and dispersion purple fringing parameters are the fourth calibration result, and in the subsequent calibration process, on the basis that the basic parameters, the defect parameters and the brightness parameters are fixed and unchanged, the sharpness, noise and dispersion purple fringing parameters are also fixed and unchanged values.

It should be noted that the method may further include:

and calibrating the dynamic range according to the parameters of definition, noise and dispersion purple boundary.

105, calibrating the chromaticity parameters of the simulation camera equipment according to the fourth calibration result to obtain a fifth calibration result;

here, the chromaticity parameters of the simulation image pickup apparatus preferably include a white balance parameter and a color restoration parameter; the white balance parameter is used for describing an index of white accuracy after the three primary colors of red, green and blue are mixed and generated in the display; the color reduction parameter refers to the degree of color reduction of the simulated image to the actual picture shot.

In an optional embodiment of the present disclosure, step 105 includes:

step 1051, under the condition of obtaining the fourth calibration result, obtaining a white balance parameter and a color restoration parameter of the simulated camera device according to a white balance parameter and a color restoration parameter of a fifth target image obtained by shooting a fourth test card (such as a virtual 24-color standard color card) in a virtual calibration environment by the simulated camera device;

in this embodiment, based on the fourth calibration result, a virtual 24-color standard color card is shot in a virtual calibration dark box in the virtual calibration environment to calibrate the white balance parameter and the color reduction parameter, so that each color block in a picture obtained by shooting the virtual 24-color standard color card in the virtual calibration environment is close to the color block in the virtual 24-color standard color card.

Step 1052, determining the white balance parameter and the color restoration parameter of the simulation image pickup apparatus as a fifth calibration result.

In this embodiment, the calibrated white balance parameter and color reduction parameter are the fifth calibration result, and in the subsequent calibration process, on the basis that the basic parameter, the defect parameter, the luminance parameter, and the high-frequency parameter are fixed and unchanged, the white balance parameter and the color reduction parameter are also fixed and unchanged values.

And 106, rendering the target object according to the first calibration result, the second calibration result, the third calibration result, the fourth calibration result and the fifth calibration result to generate a simulation image.

In this embodiment, the basic parameters, the defect parameters, the luminance parameters, the high-frequency parameters, and the chrominance parameters are sequentially calibrated in sequence, and parameters of glare, delay of the camera device, and a frame rate can be further calibrated on the basis, so that the target object is rendered according to the first calibration result, the second calibration result, the third calibration result, the fourth calibration result, and the fifth calibration result, and a simulation image is generated; the generated simulation image is closer to a real image; furthermore, because the scene rendered by the target object is a virtual space, the rendering result under any virtual scene is obtained according to the calibrated simulation camera device, and the acquisition cost of data acquisition is greatly reduced.

It should be noted that the simulation image generation method may further include:

calibrating parameters of glare, camera delay and frame rate of the simulation camera based on the fifth calibration result to obtain a sixth calibration result;

and rendering the target object according to the first calibration result, the second calibration result, the third calibration result, the fourth calibration result, the fifth calibration result and the sixth calibration result to generate the simulation video.

Parameters of glare, delay of the camera shooting equipment and a frame rate are calibrated, so that the simulated video shot by the simulated camera shooting equipment is closer to the real video of the real camera shooting equipment.

Fig. 3 is a schematic diagram of a calibration flow of a simulated image capturing apparatus in a virtual calibration environment according to an embodiment of the present disclosure, and referring to fig. 3, in yet another embodiment, calibrating the simulated image capturing apparatus in the virtual calibration environment includes:

step 31, calibrating the field angle and the image resolution of the simulation camera equipment according to the real camera equipment; wherein, the field angle and the image resolution are basic parameters;

step 32, calibrating the distortion parameter and the brightness uniformity on the basis of calibrated basic parameters; wherein, the distortion parameter and the brightness uniformity are defect parameters;

step 33, calibrating the exposure value and the contrast response curve on the basis of calibrated basic parameters and calibrated defect parameters; wherein, the exposure value and the tone response curve are brightness parameters;

step 34, calibrating the definition, the noise and the dispersion purple fringe on the basis of calibrating the basic parameters, the defect parameters and the brightness parameters; wherein, definition, noise and dispersion purple edge are high-frequency parameters;

step 35, calibrating the white balance and the color restoration on the basis that the basic parameters, the defect parameters, the brightness parameters and the high-frequency parameters are calibrated; wherein, white balance and color are reduced into chromaticity parameters;

in the method for generating a simulation image, in step 35, the method may further include: and calibrating the glare.

Through the simulated camera device calibrated in the steps 31 to 35, the generated simulated image or simulated video is closer to the real image or real video obtained by the real camera device.

According to the simulation image generation method in the embodiment of the disclosure, the simulation camera is calibrated based on the real camera, the full process from calibration to generation of the simulation image of the image quality of the simulation camera is established, the simulation image which is closer to the image detail effect obtained by the real camera is simulated by the simulation camera, and the quality of the simulation image is improved.

All the above optional technical solutions can be combined at will to form optional embodiments of the present disclosure, and are not described herein again.

Fig. 4 is a schematic structural diagram of a simulation image generation apparatus provided in an embodiment of the present disclosure, and referring to fig. 4, the apparatus 400 includes:

a first obtaining module 401, configured to calibrate a virtual calibration environment according to basic parameters of the simulated camera device, and obtain a first calibration result;

a second obtaining module 402, configured to calibrate a defect parameter of the simulated image capturing apparatus according to the first calibration result, and obtain a second calibration result;

a third obtaining module 403, configured to calibrate a brightness parameter of the simulated image capturing apparatus according to the second calibration result, so as to obtain a third calibration result;

a fourth obtaining module 404, configured to calibrate the high-frequency parameter of the simulated image capturing apparatus according to the third calibration result, so as to obtain a fourth calibration result;

a fifth obtaining module 405, configured to calibrate the chromaticity parameter of the simulated image capturing apparatus according to the fourth calibration result, so as to obtain a fifth calibration result;

and the image generation module 406 is configured to render the target object according to the first calibration result, the second calibration result, the third calibration result, the fourth calibration result, and the fifth calibration result, so as to generate a simulation image.

Optionally, the basic parameters include: resolution and field angle; the first obtaining module 401 includes:

Optionally, the defect parameters include: a distortion parameter and a brightness uniformity parameter; the second obtaining module 402 includes:

Optionally, the luminance parameters include: exposure values and tone response curves; the third obtaining module 403 includes:

Optionally, the high frequency parameters include: sharpness, noise and dispersion purple edge parameters; the fourth obtaining module 404 includes:

Optionally, the chrominance parameters include: a white balance parameter and a color reduction parameter; the fifth obtaining module 405 includes:

According to the device provided by the embodiment of the disclosure, the simulation camera device is calibrated according to the sequence of the basic parameter, the defect parameter, the brightness parameter, the high-frequency parameter and the chromaticity parameter in the virtual calibration environment, and then the simulation image is generated according to the simulation camera device, the simulation image generated by the calibrated simulation camera device is closer to the true image shot by the real camera device, and the quality of the simulation image is improved.

It should be noted that: in the simulation process, the simulation image generating apparatus provided in the above embodiment is only illustrated by the division of the above functional modules, and in practical applications, the above function distribution may be completed by different functional modules according to needs, that is, the internal structure of the apparatus may be divided into different functional modules to complete all or part of the above described functions. In addition, the simulation image generation apparatus and the simulation image generation method provided by the above embodiments belong to the same concept, and specific implementation processes thereof are described in the method embodiments and are not described herein again.

The simulation image generation device in the embodiment of the present disclosure may be a virtual device, or may be a component, an integrated circuit, or a chip in a server or a terminal. The device can be mobile electronic equipment or non-mobile electronic equipment. By way of example, the mobile electronic device may be a mobile phone, a tablet computer, a notebook computer, a palm top computer, a vehicle-mounted electronic device, a wearable device, an ultra-mobile personal computer (UMPC), a netbook or a Personal Digital Assistant (PDA), and the like, and the non-mobile electronic device may be a server, a Network Attached Storage (NAS), a Personal Computer (PC), a Television (TV), a teller machine or a self-service machine, and the like, and the disclosed embodiments are not limited in particular.

The simulation image generation apparatus in the embodiment of the present disclosure may be an apparatus having an operating system. The operating system may be an Android (Android) operating system, an ios operating system, or other possible operating systems, and the embodiment of the present disclosure is not particularly limited.

The simulation image generation device provided by the embodiment of the present disclosure can implement each process implemented by the method embodiments of fig. 1 to fig. 3, and is not described here again to avoid repetition.

Optionally, as shown in fig. 5, an electronic device 500 is further provided in an embodiment of the present disclosure, and includes a processor 501, a memory 502, and a program or an instruction stored in the memory 502 and executable on the processor 501, where the program or the instruction is executed by the processor 501 to implement each process of the above-described simulation image generation method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here. It should be noted that the electronic devices in the embodiments of the present disclosure include the mobile electronic device and the non-mobile electronic device described above.

Fig. 6 is a schematic diagram of a hardware structure of an electronic device implementing an embodiment of the present disclosure.

The electronic device 600 includes, but is not limited to: a radio frequency unit 601, a network module 602, an audio output unit 603, an input unit 604, a sensor 605, a display unit 606, a user input unit 607, an interface unit 608, a memory 609, a processor 610, and the like.

Those skilled in the art will appreciate that the electronic device 600 may further comprise a power source (e.g., a battery) for supplying power to the various components, and the power source may be logically connected to the processor 610 through a power management system, so as to implement functions of managing charging, discharging, and power consumption through the power management system. The electronic device structure shown in fig. 6 does not constitute a limitation of the electronic device, and the electronic device may include more or less components than those shown, or combine some components, or arrange different components, and thus, the description is omitted here.

It is to be understood that, in the embodiment of the present disclosure, the input Unit 604 may include a Graphics Processing Unit (GPU) 6041 and a microphone 6042, and the Graphics processor 6041 processes image data of a still picture or a video obtained by an image capturing apparatus (such as a camera device) in a video capture mode or an image capture mode. The display unit 606 may include a display panel 6061, and the display panel 6061 may be configured in the form of a liquid crystal display, an organic light emitting diode, or the like. The user input unit 607 includes a touch panel 6071 and other input devices 6072. A touch panel 6071, also referred to as a touch screen. The touch panel 6071 may include two parts of a touch detection device and a touch controller. Other input devices 6072 may include, but are not limited to, a physical keyboard, function keys (e.g., volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick, which are not described in detail herein. The memory 609 may be used to store software programs as well as various data including, but not limited to, application programs and an operating system. The processor 610 may integrate an application processor, which primarily handles operating systems, user interfaces, applications, etc., and a modem processor, which primarily handles wireless communications. It will be appreciated that the modem processor described above may not be integrated into the processor 610.

The embodiment of the present disclosure further provides a readable storage medium, where a program or an instruction is stored on the readable storage medium, and when the program or the instruction is executed by a processor, the program or the instruction implements each process of the above-mentioned simulation image generation method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.

The processor is the processor in the electronic device in the above embodiment. Readable storage media, including computer-readable storage media, such as Read-Only Memory (ROM), Random Access Memory (RAM), magnetic or optical disks, etc.

The embodiment of the present disclosure further provides a chip, where the chip includes a processor and a communication interface, the communication interface is coupled to the processor, and the processor is configured to run a program or an instruction to implement each process of the above-mentioned simulation image generation method embodiment, and can achieve the same technical effect, and in order to avoid repetition, details are not repeated here.

It should be understood that the chips mentioned in the embodiments of the present disclosure may also be referred to as system-on-chip, system-on-chip or system-on-chip, etc.

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or apparatus that comprises the element. Further, it is noted that the scope of the methods and apparatus in the embodiments of the present disclosure is not limited to performing functions in the order shown or discussed, but may include performing functions in a substantially simultaneous manner or in a reverse order based on the functions involved, e.g., the methods described may be performed in an order different than that described, and various steps may be added, omitted, or combined. In addition, features described with reference to certain examples may be combined in other examples.

Through the above description of the embodiments, those skilled in the art will clearly understand that the method of the above embodiments can be implemented by software plus a necessary general hardware platform, and certainly can also be implemented by hardware, but in many cases, the former is a better implementation manner. Based on such understanding, the technical solutions of the present disclosure may be embodied in the form of a computer software product, which is stored in a storage medium (e.g., ROM/RAM, magnetic disk, optical disk) and includes instructions for enabling a terminal (e.g., a mobile phone, a computer, a server, or a network device) to execute the method of the embodiments of the present disclosure.

While the present disclosure has been described with reference to the embodiments illustrated in the drawings, which are intended to be illustrative rather than restrictive, it will be apparent to those of ordinary skill in the art in light of the present disclosure that many more modifications may be made without departing from the spirit of the disclosure and the scope of the appended claims.

Claims

1. A method of generating a simulated image, the method comprising:

2. The method of generating a simulation image according to claim 1, wherein the basic parameters include: resolution and field angle; calibrating a virtual calibration environment according to basic parameters of the simulation camera equipment to obtain a first calibration result, which comprises the following steps:

determining the resolution of the simulation camera equipment according to the resolution of the real camera equipment;

determining a first target distance between the simulation camera equipment and a virtual calibration board in a virtual calibration environment according to the field angle;

and determining the first target distance, the second target distance and the light intensity of the virtual point light source as the first calibration result.

3. The method according to claim 2, wherein the virtual point light source is located on a line connecting centers of the virtual calibration plate and the virtual camera, and the first target distance is calculated from the angle of view and an entire image taken by the virtual calibration plate by the virtual camera.

4. The method of generating a simulation image according to claim 1, wherein the defect parameters include: a distortion parameter and a brightness uniformity parameter; calibrating the defect parameters of the simulation camera equipment according to the first calibration result to obtain a second calibration result, wherein the second calibration result comprises the following steps:

acquiring a brightness uniformity parameter of the simulation camera equipment according to a brightness change gradient of a first target image obtained by shooting a pure white calibration board in a virtual calibration environment of the simulation camera equipment;

and determining the radial distortion parameter, the tangential distortion parameter and the brightness uniformity parameter of the simulation camera equipment as the second calibration result.

5. The simulation image generation method according to claim 1, wherein the luminance parameter includes: exposure values and tone response curves;

calibrating the brightness parameter of the simulation camera device according to the second calibration result to obtain a third calibration result, including:

under the condition of obtaining a second calibration result, according to an exposure value of a second target image obtained by shooting a first test card by the simulation camera equipment in a virtual calibration environment, obtaining the exposure value of the simulation camera equipment and according to a gradient of a tone response curve of the second target image, obtaining the tone response curve of the simulation camera equipment;

and determining the exposure value and the tone response curve of the simulation camera equipment as the third calibration result.

6. The simulation image generation method according to claim 1, wherein the high-frequency parameter includes: sharpness, noise and dispersion purple edge parameters; according to the third calibration result, calibrating the high-frequency parameters of the simulation camera device to obtain a fourth calibration result, including:

under the condition of obtaining a third calibration result, according to the definition of a third target image obtained by shooting a second test card by the simulation camera equipment in a virtual calibration environment, obtaining the definition of the simulation camera equipment and according to the chromatic dispersion purple boundary of the third target image, obtaining the chromatic dispersion purple boundary parameter of the simulation camera equipment;

shooting the noise of a fourth target image obtained by a third test card according to the simulation camera equipment in a virtual calibration environment to obtain the noise of the simulation camera equipment;

and determining the definition, noise and dispersion purple fringe parameters of the simulation camera equipment as the fourth calibration result.

7. The simulation image generation method according to claim 1, wherein the chromaticity parameters include: a white balance parameter and a color reduction parameter; calibrating the chromaticity parameter of the simulation camera device according to the fourth calibration result to obtain a fifth calibration result, including:

and determining the white balance parameter and the color restoration parameter of the simulation camera equipment as the fifth calibration result.

8. A simulation image generation apparatus, comprising:

a third obtaining module, configured to calibrate a brightness parameter of the simulated shooting device according to the second calibration result, so as to obtain a third calibration result;

a fourth obtaining module, configured to calibrate the high-frequency parameter of the simulated shooting device according to the third calibration result, so as to obtain a fourth calibration result;

9. An electronic device comprising a processor, a memory, and a program or instructions stored on the memory and executable on the processor, the program or instructions when executed by the processor implementing the steps of the method of generating a simulated image as claimed in any of claims 1 to 7.

10. A readable storage medium, on which a program or instructions are stored, which when executed by a processor, carry out the steps of the simulation image generation method according to any of claims 1 to 7.