WO2023151271A1

WO2023151271A1 - Model presentation method and apparatus, and electronic device and storage medium

Info

Publication number: WO2023151271A1
Application number: PCT/CN2022/118201
Authority: WO
Inventors: 范涛; 周玉杰
Original assignee: 上海商汤智能科技有限公司
Priority date: 2022-02-10
Filing date: 2022-09-09
Publication date: 2023-08-17
Also published as: CN114463167A

Abstract

Provided in the present disclosure are a model presentation method and apparatus, and an electronic device and a storage medium. The method comprises: acquiring a projection image collected by a projection camera, and a thumbnail corresponding to each three-dimensional model comprised in the projection image; determining region information of an image superposition region, which is set in the projection image; on the basis of projection information of the three-dimensional models in the projection image and the region information, determining superposition information of the thumbnails corresponding to the three-dimensional models, wherein the superposition information comprises: superposition positions and/or presentation sizes on the image superposition region; and according to the superposition information of the thumbnails, superposing, onto the image superposition region of the projection image, the thumbnails corresponding to the three-dimensional models, so as to generate a target presented image.

Description

Model display method, device, electronic device and storage medium

This disclosure is required to be submitted to the China Patent Office on February 10, 2022. The application number is 202210125905.6, and the title of the invention is "model display method, device, electronic equipment, and storage medium." The entire content of which is incorporated by reference in this disclosure.

technical field

The present disclosure relates to the technical field of three-dimensional models, and in particular, to a model display method, device, electronic equipment, and storage medium.

Background technique

In virtual reality scenarios, digital twin projects and other scenarios, after the 3D model is established, each 3D model can be roamed. Among them, when roaming each 3D model, two virtual cameras can be set. The positions of the two virtual cameras are the same on the horizontal plane, and their positions are different in the direction vertical to the horizontal plane. The two virtual cameras move synchronously, and in the process of moving , using the first virtual camera to collect the perspective view of the 3D model, using the second virtual camera to collect the top view of the 3D model, and completing the roaming of each 3D model.

However, the method of using two virtual cameras to collect images of the 3D model has the problem of multi-camera synchronization, and the operation process is cumbersome and the efficiency is low.

Contents of the invention

In view of this, the present disclosure at least provides a model display method, device, electronic equipment and storage medium.

In a first aspect, the present disclosure provides a model presentation method, including:

Acquiring a projection image captured by the projection camera and a thumbnail corresponding to each 3D model included in the projection image;

determining the area information of the image superposition area set in the projected image;

Based on the projection information of the 3D model in the projected image and the area information, determine the overlay information of the thumbnail corresponding to the 3D model; wherein the overlay information includes: in the image overlay area The overlay position and/or display size on the ;

According to the superimposition information of the thumbnail, the thumbnail corresponding to each three-dimensional model is superimposed on the image superposition area of the projected image to generate a target display image.

In the above method, the projection image is collected by using the projection camera, and the thumbnail corresponding to each 3D model in the projection image is obtained; the thumbnail can represent the shape of the 3D model; The superimposition information corresponding to the thumbnail is to superimpose the thumbnail corresponding to each 3D model on the image superposition area of the projected image to generate a target display image. The target display image includes not only the projection image of the 3D model, but also the thumbnail representing the shape of the 3D model, which realizes the roaming of each 3D model with a virtual camera, alleviates the problem of multi-camera synchronization, and improves the efficiency of model roaming. At the same time, using one frame of target display image to display the projected image and thumbnail, alleviates the problem of waste of resources caused by generating multiple frames of images during the roaming process of the 3D model.

In a possible implementation manner, generating a thumbnail corresponding to each 3D model includes:

Determining the plane view of the three-dimensional model in the target collection direction;

Based on the plan view, a thumbnail corresponding to the three-dimensional model is generated.

Here, by determining the plan view of the 3D model in the direction of target acquisition, wherein the plan view of the 3D model in the direction of target acquisition can more accurately represent the shape of the 3D model, and then based on the plan view, after generating the corresponding thumbnail of the 3D model, The thumbnail can represent the 3D model more accurately. At the same time, by determining the thumbnail corresponding to the three-dimensional model based on the plan view, the process of determining the thumbnail does not need manual hand drawing, and the generation process of the thumbnail is relatively simple and efficient.

In a possible implementation manner, in the case where the superimposition information includes a superimposition position, determining the corresponding location of the 3D model based on the projection information of the 3D model in the projection image and the area information The overlay information of the thumbnail includes:

Based on the area size indicated by the area information and the image size corresponding to the projected image, determine the size ratio information between the image superposition area and the projected image;

Based on the size ratio information and the position information of the 3D model on the projection image indicated by the projection information corresponding to the 3D model, the superimposition position of the thumbnail image corresponding to the 3D model is determined.

Here, by determining the size ratio information between the image superposition area and the projected image, and then based on the position information of the three-dimensional model on the projected image, the superimposed position of the thumbnail corresponding to the three-dimensional model is determined, so that the multiple three-dimensional models on the projected image The relative positional relationship is matched with the relative positional relationship of the thumbnails respectively corresponding to the plurality of three-dimensional models on the image superposition area, thereby improving the accuracy of the determined superimposition position.

In a possible implementation manner, determining the position information of the three-dimensional model on the projection image includes:

Based on the projection area information of the three-dimensional model in the projection image, determine the center point of the projection area corresponding to the three-dimensional model, and determine the position information of the center point as the three-dimensional model in the projection image location information on ; or,

Obtain display position information of the 3D model in the set world coordinate system;

Using the determined coordinate system conversion matrix, the display position information is converted into the view coordinate system corresponding to the projection image, so as to obtain the position information of the three-dimensional model on the projection image.

Here, the position information of the 3D model on the projected image can be flexibly determined through the above two methods, so as to provide data support for subsequent determination of the superimposition position of the thumbnail corresponding to the 3D model on the image superimposition area.

In a possible implementation manner, based on the size ratio information and the position information of the 3D model on the projection image indicated by the projection information corresponding to the 3D model, it is determined that the 3D model corresponds to The overlay position of the thumbnail, including:

Determine the abscissa position in the superimposed position based on the abscissa information in the position information of the three-dimensional model on the projection image and the width ratio in the size ratio information;

Based on the ordinate information in the position information of the three-dimensional model on the projected image and the height ratio in the size ratio information, determine the ordinate position in the superposition position.

Here, by using the abscissa information in the position information of the 3D model on the projected image and the width ratio in the size ratio information, it is easier and more efficient to determine the lateral distance information of the thumbnail corresponding to the 3D model; The vertical coordinate information in the position information on the projection image and the height ratio in the size ratio information are relatively simple and efficient to determine the longitudinal distance information of the thumbnail corresponding to the 3D model; realize the use of the position of each 3D model in the projection image Information, more accurately determine the superposition position of the thumbnail corresponding to each 3D model.

In a possible implementation manner, in the case where the superimposition information includes a display size, the determination of the 3D model corresponding to the 3D model is based on the projection information of the 3D model in the projection image and the area information. The overlay information of the thumbnail includes:

Based on the size ratio information and the projection area information of the 3D model in the projection image indicated by the projection information, the display size of the thumbnail image corresponding to the 3D model is determined.

In the disclosed example, by determining the size ratio information between the image superposition area and the projected image, and then based on the projection area information of the 3D model indicated by the projection information in the projected image, the display size of the thumbnail image corresponding to the 3D model is determined, so that the 3D The ratio information between the display size of the thumbnail corresponding to the model and the image size of the three-dimensional model matches the size ratio information between the image superposition area and the projected image, thereby improving the display effect of the thumbnail.

In a possible implementation manner, the thumbnail corresponding to the 3D model is determined based on the size ratio information and the projection area information of the 3D model in the projection image indicated by the projection information. display dimensions, including:

determining the display width in the display size based on the width ratio indicated by the size ratio information and the width information in the projection area information;

Based on the height ratio indicated by the size ratio information and the height information in the projection area information, the display height in the display size is determined.

In the disclosed example, by determining the size ratio information between the image superposition area and the projection image, and then based on the projection area information of the 3D model indicated by the projection information in the projection image, the display size of the thumbnail image corresponding to the 3D model can be determined more accurately , so that the size of the thumbnail corresponding to the 3D model matches the size of the 3D model, and the size of the image superposition area matches the size of the projected image, and then the follow-up can be more accurate based on the display size The thumbnail corresponding to the 3D model is overlaid in the image overlay area.

For the description of the effects of the following devices, electronic equipment, etc., refer to the description of the above method, and will not be repeated here.

In a second aspect, the present disclosure provides a model display device, including:

An acquisition module, configured to acquire a projection image captured by the projection camera and a thumbnail corresponding to each 3D model included in the projection image;

A first determination module, configured to determine the area information of the image superposition area set in the projected image;

The second determination module is configured to determine the overlay information of the thumbnail image corresponding to the three-dimensional model based on the projection information of the three-dimensional model in the projected image and the area information; wherein the overlay information includes : the overlay position and/or display size on the image overlay area;

The first generating module is configured to superimpose the thumbnail corresponding to each 3D model on the image superposition area of the projected image according to the superimposition information of the thumbnail to generate a target display image.

In a third aspect, the present disclosure provides an electronic device, including: a processor, a memory, and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the processor and the The memory communicates with each other through a bus, and when the machine-readable instructions are executed by the processor, the steps of the model presentation method as described in the first aspect or any implementation manner are executed.

In a fourth aspect, the present disclosure provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the model as described in the above-mentioned first aspect or any implementation mode is executed. Show the steps of the method.

In order to make the above-mentioned objects, features and advantages of the present disclosure more comprehensible, preferred embodiments will be described in detail below together with the accompanying drawings.

Description of drawings

In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following will briefly introduce the drawings required in the embodiments.

Fig. 1 shows a schematic flowchart of a model presentation method provided by an embodiment of the present disclosure;

FIG. 2 shows a schematic diagram of multiple acquisition directions corresponding to a 3D model in a model presentation method provided by an embodiment of the present disclosure;

Fig. 3 shows a schematic diagram of a target display image provided by an embodiment of the present disclosure;

Fig. 4 shows a schematic structural diagram of a model display device provided by an embodiment of the present disclosure;

Fig. 5 shows a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure.

Detailed ways

In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure.

However, the method of using two virtual cameras to collect images of the 3D model has the problem of multi-camera synchronization, and the operation process is cumbersome and the efficiency is low. At the same time, using the first virtual camera to collect the perspective view of the 3D model and using the second virtual camera to collect the perspective view of the 3D model will generate multiple frames of images with the same resolution, resulting in a waste of resources.

In order to alleviate the above problems, embodiments of the present disclosure provide a model display method, device, electronic device, and storage medium.

It should be noted that like numerals and letters denote similar items in the following figures, therefore, once an item is defined in one figure, it does not require further definition and explanation in subsequent figures.

In order to facilitate the understanding of the embodiments of the present disclosure, a model display method disclosed in the embodiments of the present disclosure is first introduced in detail. The execution subject of the model presentation method provided by the embodiments of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a terminal device or a server; the server may be, for example, a local server, a cloud server, and the like. The terminal device may be, for example, a mobile phone, a tablet, an augmented reality (Augmented Reality, AR) glasses, a personal digital assistant (Personal Digital Assistant, PDA) and other devices. In some possible implementations, the model presentation method can be implemented by a processor calling computer-readable instructions stored in a memory.

Referring to FIG. 1 , it is a schematic flowchart of a model display method provided by an embodiment of the present disclosure. The method includes: S101-S104, specifically:

S101, acquiring a projection image collected by a projection camera and a thumbnail corresponding to each three-dimensional model included in the projection image;

S102. Determine area information of an image superposition area set in the projected image;

S103. Based on the projection information of the 3D model in the projection image and the area information, determine superimposition information of the thumbnail image corresponding to the 3D model; wherein the superposition information includes: Overlay position and/or display size on the overlay area;

S104. According to the superimposition information of the thumbnail, superimpose the thumbnail corresponding to each 3D model on the image superposition area of the projected image to generate a target display image.

S101-S104 will be specifically described below.

For S101:

The projection camera can be a virtual projection camera in the model rendering engine; or, the generated camera generation module can also be called to generate a virtual projection camera, and the virtual projection camera is used as a projection camera. Wherein, the projection camera may be a perspective projection camera, or an orthogonal projection camera. When the projection camera is a perspective projection camera, the projection image is a perspective projection image; when the projection camera is an orthographic projection camera, the projection image is an orthographic projection image.

The projection camera corresponds to pose data, the pose data includes camera position and camera pose, the camera position can be the coordinate information of the projection camera in the world coordinate system corresponding to the virtual scene, and the camera pose can be the Euler angle of the projection camera information.

During implementation, the projection camera may be used to collect a projection image of at least one three-dimensional model included in the virtual scene. After the projected image is obtained, a thumbnail corresponding to each three-dimensional model in the projected image is obtained. Wherein, the thumbnail corresponding to each three-dimensional model may be pre-generated before the projection image is collected, or may be generated in real time after the projection image is collected. Wherein, the thumbnail image corresponding to the 3D model may be an image capable of representing the shape of the 3D model. For example, the thumbnail of the 3D model may be a front view, a top view, etc. of the 3D model.

In a possible implementation manner, generating a thumbnail corresponding to each 3D model may include: determining a plan view of the 3D model in a target collection direction; and generating a thumbnail corresponding to the 3D model based on the plan view.

The target collection direction corresponding to the 3D model can be set as required, wherein different 3D models can correspond to one target collection direction, or different 3D models can also correspond to different target collection directions. For example, when different models correspond to a target acquisition direction, the target acquisition direction can be any one of the top view direction, the bottom view direction, the front view direction, the back view direction, the left view direction, and the right view direction. See Figure 2 for a schematic diagram of multiple acquisition directions.

When different models correspond to different target acquisition directions, the target acquisition direction corresponding to the three-dimensional model can be determined according to the structural characteristics of the three-dimensional model. For example, if the form displayed in the front view direction of 3D model 1 has iconic characteristics, then the target acquisition direction corresponding to 3D model 1 can be the front view direction; The acquisition direction of the target can be the overlooking direction and the like.

After the target collection direction of the 3D model is determined, the plane view of the 3D model in the target collection direction can be determined. For example, the newly generated virtual camera can be used to collect the plane view of the 3D model in the direction of the target collection. Alternatively, a projection camera may also be used to collect a plane view of the 3D model in the target collection direction.

During implementation, the plane view may be used as a thumbnail corresponding to the 3D model. Alternatively, pixel information such as size and brightness of the plan view may be adjusted, and the adjusted plan view may be used as a thumbnail corresponding to the 3D model.

For S102:

During implementation, an image overlay area can be set in the projected image, so that the adjusted thumbnail can be overlaid on the image overlay area later. The size of the image superposition area may be determined according to the size of the projected image; for example, the size of the image superposition area may be one-tenth, one-twentieth, etc. of the projected image. That is, if the size of the projected image is 510×510, the size of the image superposition area may be 51×51.

Alternatively, the proportional relationship between the width and the height of the projected image may also be determined according to the size information of the projected image; based on the proportional relationship, the size of the image superposition area is determined. For example, the size of the projected image is 1024×1024, the size of the image overlay area can be 128×128, 64×64, etc., the size of the projected image is 1024×512, and the size of the image overlay area can be 128×64, 64×32 wait.

The position of the image superposition area can be determined according to the position of the blank area included in the projected image. For example, if the blank area at the upper left of the projected image is relatively large, an image superposition area of a specific size may be set at the upper left of the projected image. Alternatively, one of the upper left position, lower left position, upper right position, and lower right position of the projected image may be arbitrarily selected as the position of the image superposition area.

After the image superposition area is set in the projected image, area information of the image superimposed area in the projected image can be determined. The area information may be position information, size information, etc. of the image superposition area in the projected image.

For S103:

After the thumbnail corresponding to each 3D model is generated, the thumbnail should be superimposed on the image superposition area of the projected image. When superimposing the thumbnail on the image superimposing area, it is necessary to determine the display size and/or superimposing position of the thumbnail on the image superimposing area.

During implementation, based on the projection information of the 3D model in the projection image and the area information of the image superposition region, the superimposition position and/or display size of the thumbnail corresponding to the 3D model on the image superposition region is determined. Wherein, the projection information may include: the image size of the 3D model in the projected image, the position information of the 3D model on the projected image, that is, the coordinate information in the view coordinate system corresponding to the projected image. The area information may include an area size of the image overlay area.

For example, the image proportion occupied by the three-dimensional model in the projected image can be determined according to the image size of the three-dimensional model in the projected image and the size information of the projected image; The area size of the overlay area determines the display size of the thumbnail corresponding to the 3D model, so that the image proportion occupied by the 3D model in the projected image matches the area proportion occupied by the thumbnail corresponding to the 3D model in the image overlay area .

For another example, according to the position information of the 3D model on the projected image and the size ratio between the projected image and the image superimposed area, the superimposed position of the thumbnail corresponding to the 3D model on the image projected area can be determined, so that the 3D model can be projected The position in the image, and the position of the thumbnail corresponding to the 3D model in the image overlay area match.

In a possible implementation manner, in the case where the superimposition information includes a superimposition position, the 3D model corresponding to the 3D model is determined based on the projection information of the 3D model in the projection image and the area information. The superimposed information of the above thumbnails may include:

Step A1, based on the area size indicated by the area information and the image size corresponding to the projection image, determine the size ratio information between the image superposition area and the projection image;

Step A2, based on the size ratio information and the position information of the 3D model on the projection image indicated by the projection information corresponding to the 3D model, determine the superimposition of the thumbnail image corresponding to the 3D model Location.

In step A1, based on the area size indicated by the area information of the image superposition area and the image size corresponding to the projected image, size ratio information between the image superimposed area and the projected image is determined. Wherein, the size ratio information may include: width ratio and height ratio. For example, it is possible to determine the first ratio between the width value in the area size of the image superposition area and the width value in the image size of the projected image, and determine the first ratio as the width ratio; and determine the area size of the image overlay area A second ratio between the height value in and the height value in the image size of the projected image is determined as the height ratio.

In step A2, the position information of the 3D model on the projected image can be determined first; based on the size ratio information and the position information of the 3D model on the projected image, the superimposed position of the thumbnail corresponding to the 3D model can be determined.

For example, the position information can be multiplied by the size ratio information, and the obtained product result can be used to determine the superposition position of the thumbnail corresponding to the 3D model, so that the distance between the position of the 3D model and the lower boundary of the projected image and the height of the projected image can be determined. The first ratio between matches the distance between the superimposed position of the thumbnail and the lower boundary of the image superimposed area, and the second ratio between the height of the image superimposed area, and the distance between the position of the 3D model and the left boundary of the projected image, and The third ratio between the width of the projected image matches the distance between the superimposed position of the thumbnail image and the left border of the image superimposed area, and the fourth ratio between the width of the image superimposed area.

Exemplarily, as shown in FIG. 3, the width value of the area size of the image superposition area is a1, the height value is b1, the width value of the image size corresponding to the projected image is a2, and the height value is b2. Determine the image superposition area and In the size ratio information between projection images, the width ratio is a1/a2, and the height ratio is b1/b2. Determine the position information of the three-dimensional model D on the projection image; based on the size ratio information and the position information of the three-dimensional model D on the projection image, determine the superposition position of the thumbnail D' corresponding to the three-dimensional model.

In one possible implementation manner, the position information of the three-dimensional model on the projection image may be determined according to the following two methods:

Mode 1: Based on the projection area information of the 3D model in the projection image, determine the center point of the projection area corresponding to the 3D model, and determine the position information of the center point as the 3D model in the projected image. location information on the projected image.

Method 2: Obtain the display position information of the 3D model in the set world coordinate system; use the determined coordinate system conversion matrix to convert the display position information into the view coordinate system corresponding to the projected image, and obtain the Position information of the 3D model on the projected image.

In the first way, the projection area information of the 3D model in the projected image is determined. The projected area can be the outline area of the 3D model on the projected image (the outline area can be the projected area), or the 3D model on the projected image. The detection frame corresponding to the contour area of the contour area (the area where the detection frame is located may be a projection area); that is, the projection area information may include the position information of the contour area on the projection image, or the position information of the detection frame corresponding to the contour area on the projection image location information, etc.

Then, the center point of the projection area corresponding to the three-dimensional model may be determined, and the position information of the center point may be determined as the position information of the three-dimensional model on the projection image.

In the second manner, generally, after the 3D model is constructed, the display pose of the 3D model in a real scene may be determined, and the display pose includes display position information and pose information. Therefore, the display position information of the 3D model in the set world coordinate system can be obtained. The world coordinate system may be a coordinate system constructed with the target scene position in the real scene as the origin.

During implementation, the determined first coordinate system conversion matrix can be used to convert the display position information of the 3D model in the world coordinate system to the camera coordinate system corresponding to the projection camera to obtain intermediate position information; and then use the second coordinate system conversion matrix , the intermediate display position information of the 3D model in the camera coordinate system corresponding to the projection camera is transformed into the view coordinate system corresponding to the projection image, and the position information of the 3D model on the projection image is obtained. Wherein, the conversion matrix of the first coordinate system may be determined according to information such as camera extrinsic parameters and the position of the projection camera. And the conversion matrix of the second coordinate system may be determined according to information such as camera internal parameters and the position of the projection camera.

In a possible implementation manner, in step A2, the 3D model is determined based on the size ratio information and the position information of the 3D model on the projection image indicated by the projection information corresponding to the 3D model The corresponding superposition position of the thumbnail may include:

During implementation, after determining the size ratio information between the image superposition area and the projected image, and the position information of the 3D model on the projected image, based on the abscissa information in the position information of the 3D model on the projected image, and the size ratio The width ratio in the information determines the distance between the thumbnail corresponding to the 3D model and the left boundary of the image superposition area, that is, the horizontal distance information. Then, the abscissa position in the superimposed position can be determined based on the lateral distance information and the area information.

And based on the ordinate information in the position information of the 3D model on the projected image and the height ratio in the size ratio information, determine the distance between the thumbnail corresponding to the 3D model and the lower boundary of the image superposition area, that is, the longitudinal distance information. Then, based on the longitudinal distance information and the area information, the ordinate position in the superimposed position can be determined.

Wherein, the abscissa position and the ordinate position may be coordinate information in a view coordinate system corresponding to the projected image. Alternatively, it may also be coordinate information in the view coordinate system corresponding to the image superposition area.

Exemplarily, as shown in FIG. 3, the position information of the three-dimensional model D in the view coordinate system corresponding to the projection image in the figure is (768, 128), that is, the abscissa information is 768, and the ordinate information is 128; in the size ratio information The width ratio is a1/a2, that is, the width ratio is 1/16; the height ratio in the size ratio information is b1/b2, that is, the height ratio is 1/16;

Multiply the width ratio 1/16 in the size ratio information with the abscissa information (768) in the position information of the three-dimensional model D, and the horizontal distance information is 48; use the height ratio 1/16 in the size ratio information and the three-dimensional model The ordinate information (128) in the position information of D is multiplied, and the longitudinal distance information is 8; then the abscissa position and the ordinate position of the superposition position can be determined according to the determined horizontal distance information, longitudinal distance information and area information. For example, in FIG. 3 , the superimposed position of the thumbnail corresponding to the three-dimensional model D may be (48, 488), and the superimposed position is position information in the view coordinate system corresponding to the projected image.

In a possible implementation manner, in the case where the superimposition information includes a display size, based on the projection information of the 3D model in the projection image and the area information, determine the corresponding 3D model. The superimposed information of the above thumbnails may include:

Step B1, based on the area size indicated by the area information and the image size corresponding to the projected image, determine the size ratio information between the image superposition area and the projected image;

Step B2, based on the size ratio information and the projection area information of the 3D model in the projection image indicated by the projection information, determine the display size of the thumbnail image corresponding to the 3D model.

In step B1, the size ratio information between the image superposition region and the projected image can be determined based on the region size indicated by the region information of the image superposition region and the image size corresponding to the projected image; wherein, the size ratio information includes: width ratio and height ratio. Specifically, for the determination process of the size ratio information, reference may be made to the description for step A1, which will not be described in detail here.

In step B2, based on the size ratio information and the projection area information of each 3D model in the projection image indicated by the projection information, the display size of the thumbnail image corresponding to each 3D model is determined. Wherein, the projection area information may include a detection frame corresponding to a contour area of the three-dimensional model on the projection image, and size information on the projection image.

Exemplarily, target detection may be performed on the projection image, and projection area information corresponding to each three-dimensional model in the projection image may be determined. For example, the trained neural network can be used to perform object detection on the projected image.

For example, the width value indicated by the projection area information can be multiplied by the width ratio in the size ratio information to obtain the width value in the display size; the height value indicated by the projection area information can be multiplied by the height ratio in the size ratio information , to get the height value in the display size.

Exemplarily, as shown in FIG. 3 , in the size ratio information between the image superposition area and the projection image in the figure, the width ratio is a1/a2, and the height ratio is b1/b2; the projection area information corresponding to the three-dimensional model D includes : The width is m2, the height is n2, and the display size of the thumbnail D' corresponding to the 3D model can be determined. That is, the display size can be width: a1/a2×m2, height: b1/b2×n2.

In a possible implementation manner, the determination of the thumbnail corresponding to the 3D model is based on the size ratio information and the projection area information of the 3D model in the projection image indicated by the projection information. Display dimensions, which can include:

Based on the width ratio indicated by the size ratio information and the width information in the projection area information, the display width in the display size is determined; for example, the display width can be obtained by multiplying the width information in the projection area information and the width ratio. And based on the height ratio indicated by the size ratio information and the height information in the projection area information, the display height in the display size is determined; for example, the height information in the projection area information can be multiplied by the height ratio to obtain the display height.

Exemplarily, refer to FIG. 3 , where the projection information indicates that the three-dimensional model D has a width of m2 and a height of n2 on the projection image; the width ratio in the size ratio information is a1/a2, and the height in the size ratio information is The ratio is b1/b2;

Multiply the width ratio a1/a2 in the size ratio information by the width m2 of the 3D model D, that is (1/16)×64, to obtain the display width of the thumbnail corresponding to the 3D model is 4; the height in the size ratio information The ratio b1/b2 is multiplied by the height n2 of the 3D model D, that is, (1/16)×16, and the display height of the thumbnail image corresponding to the 3D model is 1; that is, the display of the thumbnail image D' corresponding to the 3D model D can be determined Dimensions include: 4 for display width and 1 for display height.

For S104:

After determining the overlay position and/or display size of the thumbnail corresponding to each 3D model in the image overlay area, the thumbnail corresponding to each 3D model can be superimposed on the projection according to the overlay position and/or display size corresponding to the thumbnail On the image superposition area of the image, the target display image is obtained.

For example, when the overlay information includes the overlay position but does not include the display size, you can first adjust the thumbnail corresponding to the 3D model to the preset size to obtain the adjusted thumbnail; then overlay the adjusted thumbnail on the image overlay area At the superimposed position, a target display image is generated.

For another example, when the overlay information does not include the overlay position and display size, you can first adjust the size of the thumbnail corresponding to the 3D model to the display size to obtain the adjusted thumbnail; then overlay the adjusted thumbnail on the image overlay A target display image is generated at a preset position in the region. Wherein, the preset size and preset position can be set as required.

For another example, when the overlay information includes the overlay position and display size, you can first adjust the thumbnail corresponding to the 3D model to the display size to obtain the adjusted thumbnail; then overlay the adjusted thumbnail on the image overlay area position, generate a target display image.

During implementation, after the target display image is generated, the display screen of any device may be controlled to display the target display image. Wherein, the device may be a mobile phone, a computer, and the like.

Those skilled in the art can understand that in the above method of specific implementation, the writing order of each step does not mean a strict execution order and constitutes any limitation on the implementation process. The specific execution order of each step should be based on its function and possible The inner logic is OK.

Based on the same idea, the embodiment of the present disclosure also provides a model display device, as shown in FIG. The second determination module 403 and the first generation module 404, specifically:

An acquisition module 401, configured to acquire a projection image captured by the projection camera and a thumbnail corresponding to each 3D model included in the projection image;

The first determination module 402 is configured to determine the area information of the image superposition area set in the projected image;

The second determination module 403 is configured to determine, based on the projection information of the 3D model in the projection image and the area information, the overlay information of the thumbnail image corresponding to the 3D model; wherein the overlay information Including: the overlay position and/or display size on the image overlay area;

The first generation module 404 is configured to superimpose the thumbnail corresponding to each 3D model on the image superposition area of the projected image according to the superimposition information of the thumbnail to generate a target display image.

In a possible implementation manner, the device further includes: a second generation module 405, configured to generate a thumbnail corresponding to each 3D model according to the following steps:

In a possible implementation manner, the second determination module 403, if the superimposition information includes a superimposition position, the projection information based on the 3D model in the projection image and the area information , when determining the overlay information of the thumbnail corresponding to the 3D model, it is used for:

In a possible implementation manner, the second determination module 403, when determining the position information of the 3D model on the projection image, is configured to:

In a possible implementation manner, the second determination module 403, based on the size ratio information and indicated by the projection information corresponding to the 3D model, the 3D model on the projection image The position information is used to determine the overlay position of the thumbnail corresponding to the 3D model:

In a possible implementation manner, the second determination module 403, when the superimposition information includes display size, the projection information based on the 3D model in the projection image and the area information , when determining the overlay information of the thumbnail corresponding to the 3D model, it is used for:

In a possible implementation manner, the second determination module 403 determines the projection area information of the 3D model in the projection image based on the size ratio information and the projection information. When displaying the size of the thumbnail corresponding to the 3D model, it is used for:

In some embodiments, the functions of the device provided by the embodiments of the present disclosure or the included templates can be used to execute the methods described in the above method embodiments, and its specific implementation can refer to the description of the above method embodiments. For brevity, here No longer.

Based on the same technical idea, an embodiment of the present disclosure also provides an electronic device. Referring to FIG. 5 , it is a schematic structural diagram of an electronic device provided by an embodiment of the present disclosure, including a processor 501 , a memory 502 , and a bus 503 . Among them, the memory 502 is used to store execution instructions, including a memory 5021 and an external memory 5022; the memory 5021 here is also called an internal memory, and is used to temporarily store calculation data in the processor 501 and exchange data with an external memory 5022 such as a hard disk. The processor 501 exchanges data with the external memory 5022 through the memory 5021. When the electronic device 500 is running, the processor 501 communicates with the memory 502 through the bus 503, so that the processor 501 executes the following instructions:

Wherein, for the specific processing flow of the processor 501, reference may be made to the descriptions in the foregoing method embodiments, and details are not repeated here.

In addition, an embodiment of the present disclosure also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is run by a processor, the steps of the model demonstration method described in the above-mentioned method embodiments are executed. . Wherein, the storage medium may be a volatile or non-volatile computer-readable storage medium.

The embodiment of the present disclosure also provides a computer program product, the computer program product carries a program code, and the instructions included in the program code can be used to execute the steps of the model presentation method described in the above method embodiment, for details, please refer to the above method The embodiment will not be repeated here.

Wherein, the above-mentioned computer program product may be specifically realized by means of hardware, software or a combination thereof. In an optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (Software Development Kit, SDK) etc. wait.

The embodiments of the present disclosure relate to the field of augmented reality. By acquiring the image information of the target object in the real environment, and then using various visual correlation algorithms to detect or identify the relevant characteristics, states and attributes of the target object, and thus obtain the specific application Matching AR effect combining virtual and reality. Exemplarily, the target object may involve faces, limbs, gestures, actions, etc. related to the human body, or markers and markers related to objects, or sand tables, display areas or display items related to venues or places. Vision-related algorithms can involve visual positioning, SLAM, 3D reconstruction, image registration, background segmentation, object key point extraction and tracking, object pose or depth detection, etc. Specific applications can not only involve interactive scenes such as guided tours, navigation, explanations, reconstructions, virtual effect overlays and display related to real scenes or objects, but also special effects processing related to people, such as makeup beautification, body beautification, special effect display, virtual Interactive scenarios such as model display. The relevant features, states and attributes of the target object can be detected or identified through the convolutional neural network. The above-mentioned convolutional neural network is a network model obtained by performing model training based on a deep learning framework.

Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working process of the above-described system and device can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods may be implemented in other ways. The device embodiments described above are only illustrative. For example, the division of the units is only a logical function division. In actual implementation, there may be other division methods. For example, multiple units or components can be combined or May be integrated into another system, or some features may be ignored, or not implemented. In another point, the mutual coupling or direct coupling or communication connection shown or discussed may be through some communication interfaces, and the indirect coupling or communication connection of devices or units may be in electrical, mechanical or other forms.

The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, each unit may exist separately physically, or two or more units may be integrated into one unit.

If the functions are realized in the form of software function units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium executable by a processor. Based on this understanding, the technical solution of the present disclosure is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including Several instructions are used to make a computer device (which may be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in various embodiments of the present disclosure. The aforementioned storage media include: U disk, mobile hard disk, read-only memory (Read-Only Memory, ROM), random access memory (Random Access Memory, RAM), magnetic disk or optical disc and other media that can store program codes. .

The above is only the specific implementation of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Anyone skilled in the art can easily think of changes or substitutions within the technical scope of the present disclosure, which should be covered within the protection scope of the present disclosure. Therefore, the protection scope of the present disclosure should be determined by the protection scope of the claims.

Claims

A model display method, characterized by comprising:

Acquiring a projection image captured by the projection camera and a thumbnail corresponding to each 3D model included in the projection image;

determining the area information of the image superposition area set in the projected image;

Based on the projection information of the 3D model in the projected image and the area information, determine the overlay information of the thumbnail corresponding to the 3D model; wherein the overlay information includes: in the image overlay area The overlay position and/or display size on the ;

According to the superimposition information of the thumbnail, the thumbnail corresponding to each three-dimensional model is superimposed on the image superposition area of the projected image to generate a target display image.
The method according to claim 1, wherein generating a thumbnail corresponding to each 3D model comprises:

Determining the plane view of the three-dimensional model in the target collection direction;

Based on the plan view, a thumbnail corresponding to the three-dimensional model is generated.
The method according to claim 1 or 2, wherein, in the case where the superimposition information includes a superimposition position, the projection information based on the three-dimensional model in the projection image and the area information, Determining the overlay information of the thumbnail corresponding to the 3D model includes:

Based on the area size indicated by the area information and the image size corresponding to the projected image, determine the size ratio information between the image superposition area and the projected image;

Based on the size ratio information and the position information of the 3D model on the projection image indicated by the projection information corresponding to the 3D model, the superimposition position of the thumbnail image corresponding to the 3D model is determined.
The method according to claim 3, wherein determining the position information of the three-dimensional model on the projected image comprises:

Based on the projection area information of the three-dimensional model in the projection image, determine the center point of the projection area corresponding to the three-dimensional model, and determine the position information of the center point as the three-dimensional model in the projection image location information on ; or,

Obtain display position information of the 3D model in the set world coordinate system;

Using the determined coordinate system conversion matrix, the display position information is converted into the view coordinate system corresponding to the projection image, so as to obtain the position information of the three-dimensional model on the projection image.
The method according to claim 3 or 4, wherein the position information of the three-dimensional model on the projection image indicated by the projection information corresponding to the three-dimensional model based on the size ratio information , determining the overlay position of the thumbnail corresponding to the 3D model, including:

Determine the abscissa position in the superimposed position based on the abscissa information in the position information of the three-dimensional model on the projection image and the width ratio in the size ratio information;

Based on the ordinate information in the position information of the three-dimensional model on the projected image and the height ratio in the size ratio information, determine the ordinate position in the superposition position.
The method according to any one of claims 1 to 5, characterized in that, in the case where the overlay information includes a display size, the projection information based on the three-dimensional model in the projection image and the area information, to determine the overlay information of the thumbnail corresponding to the 3D model, including:

Based on the area size indicated by the area information and the image size corresponding to the projected image, determine the size ratio information between the image superposition area and the projected image;

Based on the size ratio information and the projection area information of the 3D model in the projection image indicated by the projection information, the display size of the thumbnail image corresponding to the 3D model is determined.
The method according to claim 6, characterized in that, based on the size ratio information and the projection area information of the three-dimensional model indicated by the projection information in the projection image, it is determined that the three-dimensional model corresponds to The display dimensions of said thumbnails, including:

determining the display width in the display size based on the width ratio indicated by the size ratio information and the width information in the projection area information;

Based on the height ratio indicated by the size ratio information and the height information in the projection area information, the display height in the display size is determined.
A model display device, characterized in that it comprises:

An acquisition module, configured to acquire a projection image captured by the projection camera and a thumbnail corresponding to each 3D model included in the projection image;

A first determination module, configured to determine the area information of the image superposition area set in the projected image;

The second determination module is configured to determine the overlay information of the thumbnail image corresponding to the three-dimensional model based on the projection information of the three-dimensional model in the projected image and the area information; wherein the overlay information includes : the overlay position and/or display size on the image overlay area;

The first generating module is configured to superimpose the thumbnail corresponding to each 3D model on the image superposition area of the projected image according to the superimposition information of the thumbnail to generate a target display image.
An electronic device, characterized in that it includes: a processor, a memory, and a bus, the memory stores machine-readable instructions executable by the processor, and when the electronic device is running, the connection between the processor and the memory The machine-readable instructions are executed by the processor through bus communication, and the steps of the model presentation method according to any one of claims 1 to 7 are executed.
A computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and when the computer program is run by a processor, the steps of the model presentation method according to any one of claims 1 to 7 are executed.