CN114387378A - Image generation method and device based on digital twin rendering engine and electronic equipment - Google Patents

Abstract

The embodiments of the present application provide an image generation method, device, and electronic device based on a digital twin rendering engine. If the object to be displayed is in a far distance range, the resolution of the target object model of the object to be displayed is low. If the object is in the shorter distance range, the resolution of the target object model of the object to be displayed is higher, that is, the resolution of the object to be displayed that is farther from the viewer is lower, and the texture is more single, and the object to be displayed that is closer to the viewer is to be displayed. The resolution of the object is higher and the texture is richer, so that in the generated image, the situation of rendering multiple different colors in one pixel can be reduced, and the moiré phenomenon in the image can be reduced; The image is lossless and will not be blurred after processing.

Description

Image generation method, device and electronic device based on digital twin rendering engine

technical field

The present application relates to the technical field of image processing, and in particular, to an image generation method, apparatus and electronic device based on a digital twin rendering engine.

Background technique

Digital twin is a multi-disciplinary, multi-physical, multi-scale, multi-probability simulation process that uses physical models, sensor data, operational history data and other data to complete the mapping in virtual space, thereby reflecting the full life cycle of the corresponding physical equipment. process. The real-time rendering part of the digital twin 3D visualization engine needs to have image data acquisition and image data rendering processing capabilities. Visually presented sensory experience.

The main reasons for generating moiré patterns include the large color difference between adjacent pixels and the rendering of multiple different colors for one pixel. In a digital twin 3D visualization engine, when patterns such as repeating lines, circles or dots overlap with imperfect alignment, a new, irregular dynamic pattern emerges, a phenomenon known as moiré. Moiré appears in 3D models and map textures in 3D scenes, changing the shape and frequency of its elements as the two original patterns move relative to each other.

In the prior art, the moiré solution in the real-time rendering engine mainly includes control filtering, upsampling interpolation, etc. to adjust the space and frequency. The main purpose of upsampling or image interpolation is to enlarge the original image, so that it can be displayed in a higher resolution. rate on the display device. The magnifying operation of the image cannot bring more detailed information about the image, so the quality of the image will inevitably be affected, and the problem of moiré flicker still exists in the image.

SUMMARY OF THE INVENTION

The purpose of the embodiments of the present application is to provide an image generation method, device and electronic device based on a digital twin rendering engine, so as to reduce the moiré phenomenon in the image. The specific technical solutions are as follows:

In a first aspect, an embodiment of the present application provides an image generation method based on a digital twin rendering engine, the method comprising:

Obtain the target distance of each object to be displayed in the 3D scene from the rendering camera;

For each object to be displayed, determine the distance interval to which the target distance of the object to be displayed belongs within a plurality of preset distance intervals, as the target distance interval of the object to be displayed;

For each object to be displayed, the object model of the level of detail corresponding to the target distance interval of the object to be displayed is determined as the target object model of the object to be displayed; wherein, different distance intervals correspond to different levels of detail, for any distance interval, the longer the distance represented by the distance interval, the lower the level of detail corresponding to the distance interval; for any detail level, the lower the detail level, the lower the resolution of the object model under the detail level;

An image to be displayed is generated based on the respective target object models of the objects to be displayed.

In a possible implementation manner, the generating the image to be displayed based on the respective target object models of the objects to be displayed includes:

acquiring the pose of each object to be displayed in the image to be displayed;

For each object to be displayed, according to the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, the target object model of the object to be displayed is rendered to generate the object to be displayed in the to-be-displayed object model the pixel area in the image;

The to-be-displayed image is generated based on the pixel area of each of the to-be-displayed objects in the to-be-displayed image.

In a possible implementation, the method further includes:

acquiring image texture maps of each object in the three-dimensional scene collected at different distances, collection distances of the image texture maps of each object, and preset distance intervals;

For each image texture map, divide the image texture map into the image set corresponding to the distance interval to which the acquisition distance of the image texture map belongs;

For each distance interval, according to the image texture maps of each object in the image set corresponding to the distance interval, an object model of each object at the level of detail corresponding to the distance interval is established.

In a possible implementation manner, for each distance interval, according to the image texture map of each object in the image set corresponding to the distance interval, the object model of each object under the level of detail corresponding to the distance interval is established, including :

For each distance interval, select multiple batches of sample data from the image texture maps of each object in the distance interval, and use the sample data of each batch to determine the amount of each object at the level of detail corresponding to the distance interval. a primary object model;

For each object, model mixing is performed on each primary object model of the object at the level of detail corresponding to the same distance interval, and the object model of the object at the level of detail corresponding to each distance interval is obtained.

In a possible implementation, the method further includes:

For the moiré area in the image texture map with moiré, calculating the color difference between adjacent pixels in the moiré area;

For adjacent pixels with a color difference greater than a preset difference threshold, a difference color is calculated according to the colors of two pixels in the adjacent pixels, and a pixel of the difference color is inserted between the adjacent pixels.

In a second aspect, an embodiment of the present application provides an image generation device based on a digital twin rendering engine, the device comprising:

A distance acquisition module, used to acquire the target distance of each object to be displayed in the 3D scene from the rendering camera;

an interval determination module for determining, for each object to be displayed, the distance interval to which the target distance of the object to be displayed belongs within a plurality of preset distance intervals, as the target distance interval of the object to be displayed;

The model determination module is used for determining the object model of the detail level corresponding to the target distance interval of the to-be-displayed object for each to-be-displayed object as the target object model of the to-be-displayed object; wherein, different distance intervals correspond to different details Level, for any distance interval, the longer the distance represented by the distance interval, the lower the level of detail corresponding to the distance interval; for any level of detail, the lower the level of detail, the lower the resolution of the object model under the level of detail ;

The image generation module is configured to generate an image to be displayed based on the respective target object models of the objects to be displayed.

In a possible implementation manner, the image generation module is specifically used for:

acquiring the pose of each object to be displayed in the image to be displayed;

For each object to be displayed, according to the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, the target object model of the object to be displayed is rendered to generate the object to be displayed in the to-be-displayed object model the pixel area in the image;

The to-be-displayed image is generated based on the pixel area of each of the to-be-displayed objects in the to-be-displayed image.

In a possible implementation, the device further includes:

a map acquisition module, configured to acquire image texture maps of each object in the three-dimensional scene collected at different distances, a collection distance of the image texture maps of each object, and preset distance intervals;

A map dividing module, for dividing the image texture map into the image set corresponding to the distance interval to which the acquisition distance of the image texture map belongs for each image texture map;

The model building module is used for, for each distance interval, to establish an object model of each object under the level of detail corresponding to the distance interval according to the image texture maps of each object in the image set corresponding to the distance interval.

In a possible implementation manner, the model establishment module is specifically used for:

For each distance interval, select multiple batches of sample data from the image texture maps of each object in the distance interval, and use the sample data of each batch to determine the amount of each object at the level of detail corresponding to the distance interval. a primary object model;

For each object, model mixing is performed on each primary object model of the object at the level of detail corresponding to the same distance interval, and the object model of the object at the level of detail corresponding to each distance interval is obtained.

In a possible implementation manner, the device further includes a color compensation module for:

For the moiré area in the image texture map with moiré, calculating the color difference between adjacent pixels in the moiré area;

For adjacent pixels with a color difference greater than a preset difference threshold, a difference color is calculated according to the colors of two pixels in the adjacent pixels, and a pixel of the difference color is inserted between the adjacent pixels.

In a third aspect, an embodiment of the present application provides an electronic device, including a processor and a memory;

the memory for storing computer programs;

The processor is configured to implement any of the methods described in this application when executing the program stored in the memory.

In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any one of the methods described in the present application is implemented .

In a fifth aspect, the embodiments of the present application provide the embodiments of the present application and also provide a computer program product including instructions, which, when run on a computer, cause the computer to execute any one of the methods described in the present application.

Beneficial effects of the embodiments of the present application:

The image generation method, device, and electronic device based on a digital twin rendering engine provided by the embodiments of the present application acquire the target distance of each object to be displayed in the three-dimensional scene from the rendering camera; for each object to be displayed, at a plurality of preset distances In the interval, the distance interval to which the target distance of the object to be displayed belongs is determined as the target distance interval of the object to be displayed; for each object to be displayed, the object model of the level of detail corresponding to the target distance interval of the object to be displayed is determined as the target distance interval of the object to be displayed. The target object model of the object to be displayed; wherein, different distance intervals correspond to different detail levels, and for any distance interval, the longer the distance represented by the distance interval, the lower the detail level corresponding to the distance interval; for any detail level level, the lower the level of detail, the lower the resolution of the object model under the level of detail; the image to be displayed is generated based on the respective target object models of the objects to be displayed. If the object to be displayed is in a far distance interval, the resolution of the target object model of the object to be displayed is lower, and if the object to be displayed is in a short distance interval, the resolution of the target object model of the object to be displayed is higher High, that is, the object to be displayed that is far from the viewer has a lower resolution and a more monolithic texture, and the object to be displayed that is closer to the viewer has a higher resolution and richer texture, so that one pixel can be reduced in the generated image. In the case of rendering multiple different colors, the moiré phenomenon in the image can be reduced; and the image after the moiré removal is generated in the embodiment of the present application is a lossless image, and will not become blurred after processing.

Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time.

Description of drawings

In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the following briefly introduces the accompanying drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are only These are some embodiments of the present application, and for those of ordinary skill in the art, other drawings can also be obtained according to these drawings.

1 is a schematic diagram of an image generation method based on a digital twin rendering engine according to an embodiment of the application;

FIG. 2 is a schematic diagram of a possible implementation of step S104 in the embodiment of the present application;

3 is a schematic diagram of a method for establishing an object model according to an embodiment of the present application;

4 is another schematic diagram of the method for establishing an object model according to an embodiment of the present application;

5 is a schematic diagram of an image generation device based on a digital twin rendering engine according to an embodiment of the application;

FIG. 6 is a schematic diagram of an electronic device according to an embodiment of the present application.

Detailed ways

The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art based on the present application fall within the protection scope of the present application.

In order to reduce the moiré phenomenon in the image, an embodiment of the present application provides an image generation method based on a digital twin rendering engine, see FIG. 1 , the method includes:

S101: Acquire a target distance of each object to be displayed in a three-dimensional scene from a rendering camera.

The image generation method based on the digital twin rendering engine in the embodiment of the present application may be implemented by an electronic device, and may be applied to scenarios such as a digital twin engine, three-dimensional scene modeling, and image data rendering. In an example, the above electronic device may be a personal computer, a server, a smart phone, or a VR (Virtual Reality, virtual reality) device, or the like.

Here, the three-dimensional scene is a three-dimensional scene that needs to be displayed, and the rendering camera is a virtual camera, which can be understood as a viewpoint for viewing the three-dimensional scene, rather than a real camera device. To facilitate understanding, a simple example is given below. For example, the 3D scene is a classroom 3D scene. If the user wants to see the entire classroom from the center of the podium, the rendering camera is considered to be in the center of the podium, that is, the rendering camera is the user's viewpoint. The image captured by the rendering camera is the image to be displayed, that is, the image that the user sees.

In one example, the target distance between the object to be displayed and the rendering camera can be obtained through a front-end perception algorithm. The front-end perception algorithm refers to the algorithm of the distance of the objects in the 3D scene of the digital twin engine from the rendering camera and the relative size of the device display resolution.

In a possible implementation manner, the acquiring the target distance of each object to be displayed in the three-dimensional scene from the rendering camera includes: acquiring the pose of the rendering camera in the three-dimensional scene; determining the pose of the rendering camera according to the pose of the rendering camera. Each object to be displayed in the three-dimensional scene and the target distance from the rendering camera to each of the objects to be displayed.

The pose includes two aspects: position and attitude. The pose of the rendering camera represents the shooting direction of the rendering camera. Combined with the position of the rendering camera, which objects in the 3D scene can be captured by the rendering camera, the 3D scene captured by the rendering camera can be determined. The object in is called the object to be displayed. According to the pose of the object to be displayed in the 3D scene, the target distance between the rendering camera and each object to be displayed can also be obtained.

For the acquisition method of the pose of the rendering camera in the three-dimensional scene, reference may be made to the prior art. In an example, taking a VR device as an example, the VR device in the world coordinate system can be obtained according to a gyroscope, a locator, etc. in the VR device. The pose, combined with the resolution of the display and the conversion relationship between the world coordinate system and the three-dimensional scene coordinate system, can obtain the pose of the rendering camera in the three-dimensional scene.

S102 , for each object to be displayed, determine a distance interval to which the target distance of the object to be displayed belongs within a plurality of preset distance intervals as the target distance interval of the object to be displayed.

The distance interval is preset. The number of distance intervals and the distance span of each distance interval can be customized according to the actual situation. The distance spans of different distance intervals can be the same or different. In an example, take three distance intervals as an example, the distance interval 1 is 0 to 10 meters, the distance interval 2 is 10 to 50 meters, and the distance interval 3 is more than 50 meters; The target distance is 5 meters, and 5 meters belongs to the distance interval 1 from 0 meters to 10 meters, then the distance interval 1 is the target distance interval of the object 1 to be displayed; if the target distance of the object 2 to be displayed is 15 meters, 15 meters belong to 10 If the distance interval 2 is between meters and 50 meters, the distance interval 2 is the target distance interval of the object 2 to be displayed.

S103, for each object to be displayed, determine the object model of the level of detail corresponding to the target distance interval of the object to be displayed, as the target object model of the object to be displayed; wherein, different distance intervals correspond to different levels of detail, for any A distance interval, the longer the distance represented by the distance interval, the lower the level of detail corresponding to the distance interval; for any detail level, the lower the detail level, the lower the resolution of the object model under the detail level.

Different distance intervals correspond to different levels of detail, and the object models of the same object at different levels of detail have different resolutions. The longer the distance represented by the distance interval, the lower the level of detail corresponding to the distance interval, and the lower the level of detail, the lower the resolution of the object model under the level of detail. The resolution of the lower object model at different levels of detail can be set according to the resolution size of the display.

In order to facilitate understanding, a simple example is given below. Taking three distance intervals as an example, distance interval 1 is 0 to 10 meters, corresponding to detail level 3; distance interval 2 is 10 meters to 50 meters, corresponding to detail level 2; distance The interval 3 is more than 50 meters, corresponding to the detail level 1; then the object model of object 1 at detail level 3 can be represented by 1000 pixels, the object model of object 1 at detail level 2 can be represented by 100 pixels, and the object model of object 1 at detail level 2 can be represented by 100 pixels. Object models at detail level 1 can be represented by 10 pixels. It can be understood that the numerical values here are only examples, and in actual scenarios, custom settings can be made according to the principles in this application.

In one example, the level of detail in the embodiment of the present application may be LOD (Levels of Detail), and LOD refers to determining the allocation of rendering resources in the image to be displayed according to the position and importance of key nodes of the object in the display environment. , reduce the accuracy and detail of non-important objects, so as to obtain high-efficiency calculation results. In the embodiment of the present application, each object is displayed separately according to different visual distances (ie, the target distance of the object to be displayed).

The z buffer is used to represent the target distance of the object to be displayed from the rendering camera. In the case of spanning distance intervals, the larger the zbuffer value is, the lower the LOD level of the object to be displayed. The resolution is also lower. According to the change of the z buffer value, different LOD levels are automatically calculated and switched, and the object model under the LOD corresponding to each object to be displayed is obtained.

S104, based on the respective target object models of the objects to be displayed, generate an image to be displayed.

The target object model of each object to be displayed is rendered to obtain the image to be displayed. In one example, based on the target object model of each object to be displayed, a multi-scale algorithm can be used to fuse the image texture data and the feature map of the three-dimensional model data to restore the image after the moiré has been eliminated.

According to the target distance (z buffer) from the rendering camera of the digital twin rendering engine to the object to be displayed, the LOD level of the object to be displayed is determined, and the display resolution of the object to be displayed in the display is finally determined. The smaller the zbuffer value of the object to be displayed is , the higher the LOD level of its engine resolution. The larger the Z buffer value of the object to be displayed, the smaller the LOD level displayed by its engine resolution, thereby reducing the moiré flickering phenomenon.

In the embodiment of the present application, if the object to be displayed is in a far distance interval, the resolution of the target object model of the object to be displayed is lower, and if the object to be displayed is in a relatively short distance interval, the resolution of the object to be displayed is The resolution of the target object model is higher, that is, the object to be displayed that is farther from the viewer has a lower resolution and a simpler texture, and the object to be displayed that is closer to the viewer has a higher resolution and richer texture. In the image, the situation of rendering multiple different colors in one pixel can be reduced, and the moiré phenomenon in the image can be reduced; and the image after the moiré removal generated in the embodiment of the present application is a lossless image, and will not become blurred after processing.

In a possible implementation manner, referring to FIG. 2 , the generation of the image to be displayed based on the respective target object models of the objects to be displayed includes:

S1041: Acquire the pose of each object to be displayed in the image to be displayed.

In one example, the pose of each object to be displayed in the image to be displayed may be determined according to the pose of the rendering camera and the pose of each object to be displayed in the three-dimensional scene.

S1042, for each to-be-displayed object, according to the pose of the to-be-displayed object in the to-be-displayed image and the target distance of the to-be-displayed object, render the target object model of the to-be-displayed object, and generate the to-be-displayed object at The area of pixels in the image to be displayed.

For each object to be displayed, according to the target distance of the object to be displayed, the size of the target object model of the object to be displayed in the image to be displayed can be obtained. Combined with the pose of the object to be displayed in the image to be displayed , the target object model of the object to be displayed is rendered, and the pixel area of the object to be displayed in the image to be displayed can be generated.

In one example, the object models at the same level of detail can also be classified into different levels of finesse. In the case where the resolution of the display is fixed, assuming that the target distance D of the object to be displayed is 1, the fineness ratio of the object to be displayed is 1, then the fineness of the object to be displayed can be expressed as:

L=|1/D|

L represents the fineness of the model. When D is less than 1, as D decreases, L is infinitely close to positive infinity, and the highest-definition object model under the corresponding LOD is applied; when D is greater than 1, as D decreases If L grows, L approaches 0 infinitely, and the object model with the lowest accuracy under the corresponding LOD is applied.

S1043 , generating an image to be displayed based on the pixel area of each of the objects to be displayed in the image to be displayed.

In the embodiment of the present application, the target object model of each object to be displayed is rendered, the pixel area of each object to be displayed in the to-be-displayed image is obtained, and the to-be-displayed image is further obtained.

The establishment process of the object model will be described below. In a possible implementation manner, referring to FIG. 3 , the method further includes:

S301: Acquire image texture maps of each object in the three-dimensional scene collected at different distances, acquisition distances of the image texture maps of each object, and preset distance intervals.

In one example, the collected image texture maps can be preprocessed, and the digital twin rendering engine is used to process image texture maps of different levels of detail. The size of the image texture map needs to be a power of 2, where the power of 2 refers to The table has 2, 4, 8, 16, 32, 64, 128, 256, 512, 1024, 2048 size pictures.

S302 , for each image texture map, divide the image texture map into image sets corresponding to the distance intervals to which the acquisition distance of the image texture map belongs.

For each image texture map, the detail level corresponding to the image texture map is determined according to the distance from the rendering camera to the acquisition of the image texture map. Different detail levels correspond to different distance intervals, and the image texture map is divided into the image texture map. In the image set corresponding to the distance interval to which the collection distance belongs. The smaller the acquisition distance of the image texture map, the higher the level of detail corresponding to the image texture map. The larger the collection distance of the image texture map, the lower the level of detail corresponding to the image texture map, which can reduce the moiré flickering phenomenon caused by the image texture map factor.

S303 , for each distance interval, establish an object model of each object at the level of detail corresponding to the distance interval according to the image texture maps of each object in the image set corresponding to the distance interval.

In one example, objects at different levels of detail can be modeled separately by combining image texture map data, 3D model data of the object, and multi-resolution parameter data.

In a possible implementation manner, for each distance interval, according to the image texture map of each object in the image set corresponding to the distance interval, the object model of each object under the level of detail corresponding to the distance interval is established, including :

Step 1: For each distance interval, select multiple batches of sample data from the image texture maps of each object in the distance interval, and use the sample data of each batch to determine the level of detail corresponding to each object in the distance interval. Multiple primary object models below.

Step 2: For each object, model mixing is performed on each primary object model of the object at the level of detail corresponding to the same distance interval, to obtain the object model of the object at the level of detail corresponding to each distance interval.

In one example, the multi-data model mixing algorithm can be used to realize model mixing. The multi-data model mixing algorithm refers to the mixing and fusion of data source models of multiple dimensions, which can achieve accurate analysis and prediction ability, and multi-data model mixing can be achieved. The model shuffling algorithm is to shuffle multiple weak models of different types into a strong model. For each distance interval, automatic sampling of multiple batches of sample data is performed from the image texture map of each object in the distance interval. The sampling data of each batch contains multiple sample data, with a total of N batches. N primary object models are obtained by training with N batches of sample data respectively, and then the N primary object model results of the same object are fused and mixed to obtain the object model of the object at the level of detail corresponding to the distance interval.

The model mixing algorithm can use the voting method and the mean method respectively. Among them, the voting method means that if it is a classification model, each model will give a category prediction result. Through voting, a new prediction result is obtained by fusion according to the principle of minority obeying the majority. The mean method means that if it is a regression model, the prediction results given by each model are numerical. At this time, we can calculate the mean of the prediction results of all sub-models as the final fusion result. In the model mixing algorithm, there is no mutual connection between the primary object models. It is a parallel fusion method. It can process N primary object models in parallel at the same time, which can greatly improve the execution efficiency of the algorithm. The process can be shown in Figure 4, where, In addition to image texture maps, LOD data can also include 3D model data of objects and the like.

In the embodiment of the present application, the data source used for establishing the object model is simple and convenient to obtain, the implementation cost is low, and the model is mixed to obtain the object model, and the algorithm is simple and does not require a large amount of deep learning training; In the digital twin project of data collection by computer, the implementation cost can be greatly reduced.

In order to further reduce the phenomenon of moiré flickering, in a possible implementation manner, the method further includes:

Step A, for the moiré area in the image texture map where the moiré exists, calculate the color difference between adjacent pixels in the moiré area.

Step B: For adjacent pixels with a color difference greater than a preset difference threshold, calculate a difference color according to the colors of two pixels in the adjacent pixels, and insert a pixel of the difference color between the adjacent pixels.

The preset difference threshold can be customized according to the actual situation, which can be an experimental value or an empirical value. In one example, a weighted average of the colors of two adjacent pixels may be performed to obtain a difference color.

When moiré appears in the image texture map under a certain LOD level, the flickering phenomenon of moiré can be reduced by the auxiliary method of compensating the difference value. Compensation value means that when the color difference between adjacent pixels is too large, such as when the adjacent pixels are black and white, an interpolated color is added between black and white as an intermediate value, thereby further reducing the moiré flickering phenomenon. In addition to adding interpolated colors, the model structure of two adjacent pixels with large color differences can also be removed, or the model structure of pixels with similar colors can be adjusted.

In the embodiment of the present application, the color difference between adjacent pixels can be reduced by means of compensating the color difference, thereby further reducing the phenomenon of moiré flickering.

An embodiment of the present application provides an image generation device based on a digital twin rendering engine, referring to FIG. 5 , the device includes:

A distance obtaining module 501, configured to obtain the target distance of each object to be displayed in the three-dimensional scene from the rendering camera;

The interval determination module 502 is configured to, for each object to be displayed, determine the distance interval to which the target distance of the object to be displayed belongs within a plurality of preset distance intervals, as the target distance interval of the object to be displayed;

The model determination module 503 is used for, for each object to be displayed, to determine the object model of the level of detail corresponding to the target distance interval of the object to be displayed, as the target object model of the object to be displayed; wherein, different distance intervals correspond to different Detail level, for any distance interval, the longer the distance represented by the distance interval, the lower the detail level corresponding to the distance interval; for any detail level, the lower the detail level, the higher the resolution of the object model under the detail level. Low;

The image generation module 504 is configured to generate an image to be displayed based on the respective target object models of the objects to be displayed.

In a possible implementation manner, the image generation module is specifically used for:

acquiring the pose of each object to be displayed in the image to be displayed;

For each object to be displayed, according to the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, the target object model of the object to be displayed is rendered to generate the object to be displayed in the to-be-displayed object model the pixel area in the image;

The to-be-displayed image is generated based on the pixel area of each of the to-be-displayed objects in the to-be-displayed image.

In a possible implementation, the device further includes:

a map acquisition module, configured to acquire image texture maps of each object in the three-dimensional scene collected at different distances, a collection distance of the image texture maps of each object, and preset distance intervals;

a map dividing module, for dividing the image texture map into the image set corresponding to the distance interval to which the acquisition distance of the image texture map belongs for each image texture map;

The model building module is used for, for each distance interval, to establish an object model of each object under the level of detail corresponding to the distance interval according to the image texture maps of each object in the image set corresponding to the distance interval.

In a possible implementation manner, the model establishment module is specifically used for:

For each distance interval, select multiple batches of sample data from the image texture maps of each object in the distance interval, and use the sample data of each batch to determine the amount of each object at the level of detail corresponding to the distance interval. a primary object model;

For each object, model mixing is performed on each primary object model of the object at the level of detail corresponding to the same distance interval, and the object model of the object at the level of detail corresponding to each distance interval is obtained.

In a possible implementation manner, the device further includes a color compensation module for:

For the moiré area in the image texture map with moiré, calculating the color difference between adjacent pixels in the moiré area;

For adjacent pixels with a color difference greater than a preset difference threshold, a difference color is calculated according to the colors of two pixels in the adjacent pixels, and a pixel of the difference color is inserted between the adjacent pixels.

Embodiments of the present application also provide an electronic device, including: a processor and a memory;

The above-mentioned memory is used to store computer programs;

When the above-mentioned processor is configured to execute the computer program stored in the above-mentioned memory, any one of the image generation methods based on a digital twin rendering engine described in this application can be implemented.

Optionally, referring to FIG. 6 , the electronic device in the embodiment of the present application further includes a communication interface 602 and a communication bus 604 , wherein the processor 601 , the communication interface 602 , and the memory 603 communicate with each other through the communication bus 604 .

The communication bus mentioned in the above electronic device may be a PCI (Peripheral Component Interconnect, Peripheral Component Interconnect Standard) bus or an EISA (Extended Industry Standard Architecture, Extended Industry Standard Architecture) bus or the like. The communication bus can be divided into an address bus, a data bus, a control bus, and the like. For ease of presentation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

The communication interface is used for communication between the above electronic device and other devices.

The memory may include RAM (Random Access Memory, random access memory), or may include NVM (Non-Volatile Memory, non-volatile memory), for example, at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

The above-mentioned processor may be a general-purpose processor, including a CPU (Central Processing Unit, central processing unit), an NP (Network Processor, network processor), etc.; it may also be a DSP (Digital Signal Processing, digital signal processor), an ASIC ( Application Specific Integrated Circuit), FPGA (Field-Programmable GateArray, Field Programmable Gate Array) or other programmable logic devices, discrete gate or transistor logic devices, and discrete hardware components.

Embodiments of the present application further provide a computer-readable storage medium, where a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any one of the digital twin rendering engine-based rendering engine described in the present application is implemented. Image generation method.

In yet another embodiment provided by the present application, a computer program product including instructions is also provided, which, when run on a computer, enables the computer to execute any one of the digital twin rendering engine-based image generation described in the present application method.

In the above-mentioned embodiments, it may be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented in software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of the present application are generated. The computer may be a general purpose computer, special purpose computer, computer network, or other programmable device. The computer instructions may be stored in or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be downloaded from a website site, computer, server, or data center Transmission to another website site, computer, server, or data center by wire (eg, coaxial cable, optical fiber, digital subscriber line) or wireless (eg, infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server, a data center, or the like that includes an integration of one or more available media. The usable media may be magnetic media (eg, floppy disks, hard disks, magnetic tapes), optical media (eg, DVD), or semiconductor media (eg, Solid State Disk (SSD)), and the like.

It should be noted that, in this article, the technical features in each optional solution can be combined to form solutions as long as they are not contradictory, and these solutions are all within the scope of the disclosure of the present application. Relational terms such as first and second, etc. are only used to distinguish one entity or operation from another and do not necessarily require or imply any such actual relationship between these entities or operations or order. Moreover, the terms "comprising", "comprising" or any other variation thereof are intended to encompass non-exclusive inclusion such that a process, method, article or device comprising a list of elements includes not only those elements, but also includes not explicitly listed or other elements inherent to such a process, method, article or apparatus. Without further limitation, an element qualified by the phrase "comprising a..." does not preclude the presence of additional identical elements in a process, method, article or apparatus that includes the element.

Each embodiment in this specification is described in a related manner, and the same and similar parts between the various embodiments may be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the embodiments of the apparatus, electronic equipment, computer program product and storage medium, since they are basically similar to the method embodiments, the description is relatively simple, and reference may be made to some descriptions of the method embodiments for related parts.

The above descriptions are only preferred embodiments of the present application, and are not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of this application are included in the protection scope of this application.

Claims (12)

1. An image generation method based on a digital twin rendering engine, the method comprising:

acquiring a target distance between each object to be displayed in the three-dimensional scene and the rendering camera;

for each object to be displayed, determining a distance interval to which a target distance of the object to be displayed belongs within a plurality of preset distance intervals as a target distance interval of the object to be displayed;

determining an object model of a detail level corresponding to a target distance interval of each object to be displayed as a target object model of the object to be displayed; the different distance intervals correspond to different detail levels, and for any distance interval, the longer the distance represented by the distance interval is, the lower the detail level corresponding to the distance interval is; for any level of detail, the lower the resolution of the object model at that level of detail;

and generating an image to be displayed based on the respective target object model of each object to be displayed.

2. The method according to claim 1, wherein the generating an image to be displayed based on the target object model of each of the objects to be displayed comprises:

acquiring the pose of each object to be displayed in the image to be displayed;

aiming at each object to be displayed, rendering a target object model of the object to be displayed according to the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, and generating a pixel area of the object to be displayed in the image to be displayed;

and generating an image to be displayed based on the pixel area of each object to be displayed in the image to be displayed.

3. The method of claim 1, further comprising:

acquiring image texture maps of all objects in the three-dimensional scene acquired at different distances, acquiring distances of the image texture maps of all the objects and preset distance intervals;

for each image texture mapping, dividing the image texture mapping into an image set corresponding to a distance interval to which the acquisition distance of the image texture mapping belongs;

and aiming at each distance interval, establishing an object model of each object under the detail level corresponding to the distance interval according to the image texture mapping of each object in the image set corresponding to the distance interval.

4. The method according to claim 3, wherein for each distance interval, establishing an object model of each object at a level of detail corresponding to the distance interval according to the image texture map of each object in the image set corresponding to the distance interval comprises:

for each distance interval, selecting a plurality of batches of sample data from the image texture maps of the objects in the distance interval, and respectively determining a plurality of primary object models of each object under the detail level corresponding to the distance interval by using the batches of sample data;

and for each object, respectively performing model blending on each primary object model of the object under the detail level corresponding to the same distance interval to obtain an object model of the object under the detail level corresponding to each distance interval.

5. The method of claim 3, further comprising:

calculating color difference values between adjacent pixels in a moire pattern area aiming at the moire pattern area in the image texture mapping with the moire pattern;

and aiming at the adjacent pixels with the color difference value larger than a preset difference value threshold, calculating to obtain a difference color according to the colors of two pixels in the adjacent pixels, and inserting the pixels with the difference colors between the adjacent pixels.

6. An image generation apparatus based on a digital twin rendering engine, the apparatus comprising:

the distance acquisition module is used for acquiring the target distance between each object to be displayed in the three-dimensional scene and the rendering camera;

the interval determining module is used for determining a distance interval to which a target distance of each object to be displayed belongs within a plurality of preset distance intervals as a target distance interval of the object to be displayed;

the model determining module is used for determining an object model of a detail level corresponding to a target distance interval of each object to be displayed as a target object model of the object to be displayed; the different distance intervals correspond to different detail levels, and for any distance interval, the longer the distance represented by the distance interval is, the lower the detail level corresponding to the distance interval is; for any level of detail, the lower the resolution of the object model at that level of detail;

and the image generation module is used for generating the image to be displayed based on the target object model of each object to be displayed.

7. The apparatus of claim 6, wherein the image generation module is specifically configured to:

acquiring the pose of each object to be displayed in the image to be displayed;

aiming at each object to be displayed, rendering a target object model of the object to be displayed according to the pose of the object to be displayed in the image to be displayed and the target distance of the object to be displayed, and generating a pixel area of the object to be displayed in the image to be displayed;

and generating an image to be displayed based on the pixel area of each object to be displayed in the image to be displayed.

8. The apparatus of claim 6, further comprising:

the map acquiring module is used for acquiring image texture maps of all objects in the three-dimensional scene acquired at different distances, the acquisition distance of the image texture maps of all the objects and preset distance intervals;

the mapping dividing module is used for dividing each image texture mapping into an image set corresponding to a distance interval to which the acquisition distance of the image texture mapping belongs;

and the model establishing module is used for establishing an object model of each object under the detail level corresponding to each distance interval according to the image texture mapping of each object in the image set corresponding to the distance interval.

9. The apparatus of claim 8, wherein the model building module is specifically configured to:

for each distance interval, selecting a plurality of batches of sample data from the image texture maps of the objects in the distance interval, and respectively determining a plurality of primary object models of each object under the detail level corresponding to the distance interval by using the batches of sample data;

and for each object, respectively performing model blending on each primary object model of the object under the detail level corresponding to the same distance interval to obtain an object model of the object under the detail level corresponding to each distance interval.

10. The apparatus of claim 8, further comprising a color compensation module to:

calculating color difference values between adjacent pixels in a moire pattern area aiming at the moire pattern area in the image texture mapping with the moire pattern;

and aiming at the adjacent pixels with the color difference value larger than a preset difference value threshold, calculating to obtain a difference color according to the colors of two pixels in the adjacent pixels, and inserting the pixels with the difference colors between the adjacent pixels.

11. An electronic device comprising a processor and a memory;

the memory is used for storing a computer program;

the processor, when executing the program stored in the memory, implementing the method of any of claims 1-5.

12. A computer-readable storage medium, in which a computer program is stored which, when being executed by a processor, carries out the method of any one of claims 1 to 5.

Images