WO2020034738A1

WO2020034738A1 - Three-dimensional model processing method and apparatus, electronic device and readable storage medium

Info

Publication number: WO2020034738A1
Application number: PCT/CN2019/090557
Authority: WO
Inventors: 张弓
Original assignee: Oppo广东移动通信有限公司
Priority date: 2018-08-16
Filing date: 2019-06-10
Publication date: 2020-02-20
Also published as: CN109285214A

Abstract

A three-dimensional model processing method and apparatus, an electronic device, and a readable storage device. The method comprises: acquiring a three-dimensional model of a human body; the three-dimensional model comprising a plurality of key points, a model frame formed by the plurality of key points being connected, and texture information encompassing the model frame (101); acquiring environmental information concerning an environment in which the three-dimensional model is constructed (102); and adjusting the positions of part of the key points in the three-dimensional model according to the environmental information, and/or rendering the texture information of the three-dimensional model (103). The method enables a three-dimensional model of a human body to more accurately reflect the effect of an environment, improving the fidelity of the display effect of the three-dimensional model.

Description

Method, device, electronic device and readable storage medium for processing three-dimensional model

Cross-reference to related applications

This disclosure requires the priority of China Patent Application No. “201810934594.1”, submitted by OPPO Guangdong Mobile Communication Co., Ltd. on August 16, 2018, with the invention name “Processing Method, Device, Electronic Device, and Readable Storage Medium for Three-Dimensional Models” right.

Technical field

The present disclosure relates to the technical field of electronic devices, and in particular, to a method, an apparatus, an electronic device, and a readable storage medium for processing a three-dimensional model.

Background technique

3D model reconstruction is a mathematical model suitable for computer representation and processing. It is the basis for processing, manipulating, and analyzing its properties in a computer environment. It is also a key technology for establishing a virtual reality that expresses the objective world in a computer. Usually, the key points in the three-dimensional model are processed to realize the reconstruction of the model.

In actual operation, when the electronic device associated with the three-dimensional model shakes or the ambient light changes, the three-dimensional model also changes accordingly. For example, when the electronic device moves or is exposed to different light, the three-dimensional model of the face will also change accordingly. For example, bangs and facial muscles will swing at the same time as the electronic device shakes, and the light on the face will also change with the ambient light.

Summary of the Invention

The present disclosure aims to solve at least one of the technical problems in the related art.

For this reason, the present disclosure proposes a method for processing a three-dimensional model, so as to solve the current situation that the electronic device associated with the three-dimensional model shakes or the light in the environment changes, the three-dimensional model also changes accordingly, so that the three-dimensional model of the human body cannot be real Technical issues that respond to environmental effects.

The present disclosure proposes a processing device for a three-dimensional model.

The present disclosure proposes an electronic device.

The present disclosure proposes a computer-readable storage medium.

An embodiment of one aspect of the present disclosure provides a method for processing a three-dimensional model, including:

Acquiring a three-dimensional model of a human body; wherein the three-dimensional model includes a plurality of key points, a model frame formed by connecting the plurality of key points, and texture information covering the model frame;

Acquiring environmental information of an environment in which the three-dimensional model is constructed;

According to the environment information, position adjustments are performed on some key points in the three-dimensional model, and / or rendering the three-dimensional model texture information.

The method for processing a three-dimensional model in the embodiment of the present disclosure obtains a three-dimensional model of a human body; wherein the three-dimensional model includes a plurality of key points, a model frame formed by connecting the plurality of key points, and texture information covering the model frame; obtaining and constructing a three-dimensional model Environmental information of the environment in which the model is located; according to the environmental information, position adjustments of some key points in the 3D model, and / or rendering texture information of the 3D model. Therefore, by adjusting the position of some key points of the 3D human body model and rendering the texture information of the 3D human body model, the 3D human body model more realistically reflects the environment effect, and the fidelity of the 3D model display special effects is improved.

An embodiment of another aspect of the present disclosure provides a three-dimensional model processing device, including:

A first acquisition module for acquiring a three-dimensional model of a human body; wherein the three-dimensional model includes a plurality of key points, a model frame formed by a plurality of key points connected, and texture information covering the model frame;

A second acquisition module, configured to acquire environmental information of an environment in which the three-dimensional model is constructed;

A processing module, configured to adjust position of some key points in the three-dimensional model and / or render texture information of the three-dimensional model according to the environment information.

The apparatus for processing a three-dimensional model according to an embodiment of the present disclosure obtains a three-dimensional model of a human body; wherein the three-dimensional model includes multiple key points, a model frame formed by the connection of multiple key points, and texture information covering the model frame; Environmental information of the environment in which the model is located; according to the environmental information, position adjustments of some key points in the 3D model, and / or rendering texture information of the 3D model. Therefore, by adjusting the position of some key points of the 3D human body model and rendering the texture information of the 3D human body model, the 3D human body model more realistically reflects the environment effect, and the fidelity of the 3D model display special effects is improved.

An embodiment of another aspect of the present disclosure provides an electronic device, including: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the foregoing implementation is implemented. The processing method of the three-dimensional model described in the example.

An embodiment of another aspect of the present disclosure provides a computer-readable storage medium on which a computer program is stored, characterized in that, when the program is executed by a processor, the method for processing a three-dimensional model according to the foregoing embodiment is implemented.

Additional aspects and advantages of the present disclosure will be given in part in the following description, part of which will become apparent from the following description, or be learned through the practice of the present disclosure.

BRIEF DESCRIPTION OF THE DRAWINGS

The above and / or additional aspects and advantages of the present disclosure will become apparent and easily understood from the following description of the embodiments in conjunction with the accompanying drawings, in which:

FIG. 1 is a schematic flowchart of a three-dimensional model processing method according to an embodiment of the present disclosure;

2 is a schematic flowchart of adjusting a position of a target key point according to an embodiment of the present disclosure;

3 is a schematic structural diagram of a three-dimensional model processing apparatus according to an embodiment of the present disclosure;

4 is a schematic structural diagram of another three-dimensional model processing apparatus according to an embodiment of the present disclosure;

5 is a schematic diagram of an internal structure of an electronic device in an embodiment;

6 is a schematic diagram of an image processing circuit as a possible implementation manner;

FIG. 7 is a schematic diagram of an image processing circuit as another possible implementation manner.

detailed description

Hereinafter, embodiments of the present disclosure will be described in detail. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the drawings are exemplary, and are intended to explain the present disclosure, and should not be construed as limiting the present disclosure.

The three-dimensional model processing method and device according to the embodiments of the present disclosure are described below with reference to the drawings.

FIG. 1 is a schematic flowchart of a three-dimensional model processing method according to an embodiment of the present disclosure.

As shown in FIG. 1, the processing method of the three-dimensional model includes the following steps:

In the embodiment of the present disclosure, the electronic device may be a hardware device such as a mobile phone, a tablet computer, a personal digital assistant, or a wearable device, which has various operating systems, a touch screen, and / or a display screen.

Step 101: Obtain a three-dimensional model of the human body. The three-dimensional model includes a plurality of key points, a model frame formed by connecting the plurality of key points, and texture information covering the model frame.

The three-dimensional model of the human body obtained in this embodiment includes multiple key points and a model frame formed by the connection of multiple key points, and texture information covering the model frame. Among them, the key points and the model frame formed by the connection of multiple key points can be expressed in the form of three-dimensional coordinates.

Because different regions of the human body have different skin textures, different regions of the human 3D model have different texture information. For example, the skin texture of hair, eye, face, hand, etc. is different.

The acquisition of the three-dimensional model of the human body in this embodiment is obtained by performing three-dimensional reconstruction according to the depth information and the human body image, instead of simply acquiring RGB data and depth data.

As a possible implementation manner, the depth information and the color information corresponding to the two-dimensional image of the human body can be fused to obtain a three-dimensional model of the human body. Specifically, it is possible to extract the key points of the human body from the depth information and the key points of the human body from the color information based on the human key point detection technology, and then extract the key points from the depth information and the key points from the color information. , To perform registration and key point fusion processing, and finally generate a three-dimensional model of the human body based on the key points after fusion. The key point is a conspicuous point on the human body or a point on a key position. For example, the key points may be hair, eyes, nose, mouth, hands, and the like.

Further, based on human keypoint detection technology, keypoint recognition can be performed on a human image to obtain keypoints corresponding to the human image, so that multiple keypoints can be connected to form a three-dimensional model according to the relative position of each keypoint in three-dimensional space. Frame to obtain texture information covering the model frame.

Step 102: Obtain environmental information of an environment in which a three-dimensional model is constructed.

The environment information includes acceleration information and lighting information of an environment in which a three-dimensional model is constructed.

It should be noted that when constructing a three-dimensional model, acceleration information may be generated due to movements of the electronic device due to shaking, shaking, and the like. At the same time, the three-dimensional model will also change accordingly. For example, as the electronic device shakes, the hair or facial muscles in the three-dimensional model of the human body will also swing.

When the intensity of the ambient light in which the electronic device is located changes, the three-dimensional model of the human body also changes. For example, when the electronic device is illuminated with strong light, the three-dimensional model will also be illuminated by strong light.

In the embodiment of the present disclosure, a first acceleration vector is first measured by an acceleration sensor configured by an electronic device, and then a second acceleration vector is measured by a gravity sensor configured by the electronic device, and then the measured first acceleration vector and the second acceleration vector are obtained. Synthesis is performed to obtain an acceleration vector, and finally acceleration information in the environment information is determined according to the synthesized acceleration vector. The acceleration vector includes the magnitude and direction of the acceleration.

Similarly, the ambient light sensor in the electronic device associated with the three-dimensional model acquires illumination information in the environment where the electronic device is located. The lighting information includes the intensity of light in the environment and / or the angle information of the light illuminating the screen of the electronic device.

Step 103: Adjust position of some key points in the 3D model according to the environmental information, and / or render texture information of the 3D model.

In the embodiment of the present disclosure, a target key point to be adjusted is determined from a plurality of key points of the obtained three-dimensional model to adjust the position of the target key point.

When acceleration information is generated due to vibration or shaking of the electronic device associated with the three-dimensional model, the three-dimensional model also changes accordingly. Therefore, it is necessary to position the target key points to be adjusted according to the obtained acceleration information of the electronic device associated with the three-dimensional model. Adjust, further, render the texture information to the 3D model after the position adjustment process, so that the 3D model can show more realistic special effects.

Further, the light intensity and the light angle irradiated to the electronic device are determined according to the lighting information of the electronic device associated with the three-dimensional model, and the size of the highlight value at different positions of the three-dimensional model is adjusted according to the light intensity and the light angle. Furthermore, the 3D model after the position adjustment process is subjected to light effect rendering of the skin texture, so that the light intensity on the surface of the 3D model changes with the change of the lighting information.

As another possible implementation manner, the light intensity of the electronic device is determined according to the illumination information of the electronic device associated with the three-dimensional model, and the size of the highlight value at different positions of the three-dimensional model is adjusted according to the light intensity. For example, when irradiating an electronic device with strong light, the three-dimensional model will be as if it was illuminated by strong light. Therefore, it is necessary to adjust the size of the highlight value at different positions of the three-dimensional model, and then render the light effect of the skin texture on the three-dimensional model to make the three-dimensional model The light intensity of a surface changes with the intensity of the light.

As another possible implementation manner, when the electronic device flips, shakes, or the like, the angle at which the ambient light strikes the electronic device changes. When the ambient light of the same light intensity illuminates the electronic device from different angles, the The light intensity is different at different locations. According to the lighting information of the electronic device associated with the three-dimensional model, the angle of the light irradiated to the electronic device is determined, and the size of the highlight value at different positions of the three-dimensional model is adjusted according to the light intensity. The light intensity on the model surface changes with the angle of illumination.

The method for processing a three-dimensional model according to the embodiment of the present disclosure obtains a three-dimensional model of a human body; wherein the three-dimensional model includes a plurality of key points, a model frame formed by connecting the plurality of key points, and texture information covering the model frame; obtaining and constructing a three-dimensional model Environmental information of the environment in which the environment is located; according to the environmental information, position adjustment of some key points in the 3D model, and / or rendering texture information of the 3D model. In the present disclosure, by adjusting position of some key points of the three-dimensional human body model and rendering texture information of the three-dimensional human body model, the effect of the three-dimensional human body model more realistically reflects the environment, and the fidelity of the three-dimensional model display special effects is improved.

As a possible implementation manner, for the determined target key point of the three-dimensional model of the human body, the position of the target key point is adjusted according to the acceleration information. In order to accurately adjust the position of the target keypoint according to the acceleration information, in the embodiment of the present disclosure, according to the determined adjustment direction of the target keypoint, and then move the target keypoint along the adjustment direction, see FIG. 2. The specific steps are as follows: :

Step 201: Read a correspondence between a pre-stored target keypoint and a conversion coefficient.

Among them, the conversion coefficient refers to a conversion coefficient between acceleration and displacement corresponding to a target key point.

It should be noted that there are multiple target keypoints, and the value of the conversion coefficient corresponding to each target keypoint is determined according to the material of the three-dimensional model corresponding to each target keypoint and / or the relative position in the three-dimensional model. Therefore, the target keypoints at different positions correspond to different conversion coefficients.

As an example, when the target key point is a hair part in the 3D model, the target key point of the hair part in the 3D model and the displacement weight w of the target key point are set. For example, the weight of the hair root is small, and the weight of the hair tip is small. The weight is large, and the target keypoints are determined according to the material of the corresponding 3D model of the target keypoints, such as black material, and the relative position of the target keypoints in the 3D model. The value of the corresponding conversion coefficient.

Step 202: Determine the displacement value corresponding to the target key point according to the acceleration value indicated by the acceleration information and the conversion coefficient between acceleration and displacement corresponding to the target key point.

Specifically, according to the acceleration information measured by the acceleration sensor of the electronic device associated with the three-dimensional model, the magnitude of the acceleration is obtained, and the conversion coefficient corresponding to the target key point is determined according to the corresponding relationship between the pre-stored target key point and the conversion coefficient. value. Furthermore, according to the acceleration of the target keypoint and the corresponding conversion coefficient between acceleration and displacement, the displacement value corresponding to the target keypoint is determined.

As an example, according to the magnitude V of the acceleration measured by the acceleration sensor of the electronic device and the direction D of the acceleration, it is determined that the movement direction of the hair part in the three-dimensional model is -VD. Further, the gravity acceleration G of the electronic device is obtained according to the measurement of the gravity sensor, and the gravity acceleration G and the acceleration V are combined to obtain the movement direction of the hair in the three-dimensional model caused by the movement of the electronic device and the external force as G-VD. Furthermore, the displacement value S = w (G-VD) corresponding to each target key point of the hair part in the three-dimensional model is determined.

Step 203: Determine the adjustment direction of the target key point according to the acceleration direction indicated by the acceleration information.

Specifically, according to the acceleration information measured by the acceleration sensor of the electronic device associated with the three-dimensional model, the direction of acceleration is obtained, and then the adjustment direction of the target key point is determined.

In step 204, the target key point is moved along the adjustment direction, and the moving distance conforms to the displacement value.

In this embodiment, the position of the target key point is moved along the adjustment direction of the target key point determined by the acceleration direction, and the moving distance is a corresponding displacement value.

In the method for processing a three-dimensional model according to the embodiment of the present disclosure, by reading a pre-stored correspondence between a target keypoint and a conversion coefficient; an acceleration value indicated according to acceleration information, and a conversion coefficient between acceleration and displacement corresponding to the target keypoint , Determine the displacement value corresponding to the target key point; then determine the adjustment direction of the target key point according to the acceleration direction indicated by the acceleration information; finally, move the target key point along the adjustment direction, and the moving distance conforms to the displacement value. This method adjusts the position and direction of the key points of the target in the 3D model to make the 3D model of the human body more realistically reflect the effect of the environment and improve the fidelity of the 3D model display special effects.

In order to implement the above embodiments, the present disclosure also proposes a processing device for a three-dimensional model.

FIG. 3 is a schematic structural diagram of a three-dimensional model processing apparatus according to an embodiment of the present disclosure.

As shown in the figure, the three-dimensional model processing device 100 includes a first acquisition module 110, a second acquisition module 120, and a processing module 130.

The first obtaining module 110 is configured to obtain a three-dimensional model of the human body. The three-dimensional model includes multiple key points, a model frame formed by connecting the multiple key points, and texture information covering the model frame.

The second acquisition module 120 is configured to acquire environmental information of an environment in which a three-dimensional model is constructed.

The processing module 130 is configured to adjust position of some key points in the three-dimensional model and / or render texture information of the three-dimensional model according to environmental information.

As a possible implementation manner, referring to FIG. 4, the processing module 130 further includes:

A determining unit 131 is configured to determine a target key point to be adjusted from a plurality of key points of the three-dimensional model.

The adjusting unit 132 is configured to adjust the position of the target key point according to the acceleration information.

The rendering unit 133 is configured to perform skin texture rendering on the three-dimensional model after the position adjustment process.

As a possible implementation manner, the adjusting unit 132 is further configured to determine the displacement value corresponding to the target key point according to the acceleration value indicated by the acceleration information and the conversion coefficient between acceleration and displacement corresponding to the target key point;

Determine the adjustment direction of the target key point according to the acceleration direction indicated by the acceleration information;

Along the adjustment direction, move the key point of the target, and the moving distance conforms to the displacement value.

As a possible implementation manner, referring to FIG. 4, the second obtaining module 120 further includes:

The first measurement unit 121 is configured to measure a first acceleration vector through an acceleration sensor.

The second measurement unit 122 is configured to measure a second acceleration vector through a gravity sensor.

A combining unit 123 is configured to combine the first acceleration vector and the second acceleration vector.

The second determining unit 124 is configured to determine acceleration information in the environment information according to the synthesized acceleration vector.

As a possible implementation manner, referring to FIG. 4, the apparatus for processing a three-dimensional model further includes:

The reading module 140 is configured to read a correspondence between a pre-stored target key point and a conversion coefficient;

There are multiple target keypoints, and the value of the conversion coefficient corresponding to each target keypoint is determined according to the material of the three-dimensional model corresponding to each target keypoint and / or the relative position in the three-dimensional model.

As a possible implementation manner, the rendering unit 133 is further configured to determine a light intensity and / or a light angle according to the lighting information; and further, perform a light effect rendering on the skin texture of the three-dimensional model according to the light intensity and / or a light angle.

The three-dimensional model processing device according to the embodiment of the present disclosure obtains a three-dimensional model of a human body; wherein the three-dimensional model includes multiple key points, a model frame formed by the connection of multiple key points, and texture information covering the model frame; obtaining and constructing a three-dimensional model Environmental information of the environment in which the environment is located; according to the environmental information, position adjustment of some key points in the 3D model, and / or rendering texture information of the 3D model. In the present disclosure, by adjusting the position of some key points of the three-dimensional human body model and rendering the skin texture of the three-dimensional human body model, the effect of the three-dimensional human body model more realistically reflects the environment, and the fidelity of the three-dimensional model display special effects is improved.

It should be noted that the foregoing explanation of the embodiment of the three-dimensional model processing method is also applicable to the three-dimensional model processing apparatus of this embodiment, and details are not described herein again.

In order to implement the above embodiments, the present disclosure also provides an electronic device including: a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the implementation is as described above. The processing method of the three-dimensional model according to the embodiment.

FIG. 5 is a schematic diagram of the internal structure of the electronic device 200 in an embodiment. The electronic device 200 includes a processor 220, a memory 230, a display 240, and an input device 250 connected through a system bus 210. The memory 230 of the electronic device 200 stores an operating system and computer-readable instructions. The computer-readable instructions may be executed by the processor 220 to implement a face recognition method according to an embodiment of the present disclosure. The processor 220 is used to provide computing and control capabilities to support the operation of the entire electronic device 200. The display 240 of the electronic device 200 may be a liquid crystal display or an electronic ink display, and the input device 250 may be a touch layer covered on the display 240, or a button, a trackball, or a touchpad provided on the housing of the electronic device 200. It can also be an external keyboard, trackpad, or mouse. The electronic device 200 may be a mobile phone, a tablet computer, a notebook computer, a personal digital assistant, or a wearable device (e.g., a smart bracelet, a smart watch, a smart helmet, a smart glasses), and the like.

Those skilled in the art can understand that the structure shown in FIG. 5 is only a schematic diagram of a part of the structure related to the solution of the present disclosure, and does not constitute a limitation on the electronic device 200 to which the solution of the present disclosure is applied. The specific electronic device 200 may include more or fewer components than shown in the figure, or some components may be combined, or have different component arrangements.

As a possible implementation manner, referring to FIG. 6, an image processing circuit according to an embodiment of the present disclosure is provided. The image processing circuit may be implemented by using hardware and / or software components.

As shown in FIG. 6, the image processing circuit specifically includes an image unit 310, a depth information unit 320, and a processing unit 330. among them,

The image unit 310 is configured to output a two-dimensional human body image.

The depth information unit 320 is configured to output depth information.

In the embodiment of the present disclosure, a two-dimensional image may be acquired through the image unit 310, and depth information corresponding to the image may be acquired through the depth information unit 320.

The processing unit 330 is electrically connected to the image unit 310 and the depth information unit 320, respectively, and is configured to identify the target three-dimensional template matching the image according to the two-dimensional image obtained by the image unit and the corresponding depth information obtained by the depth information unit. , Output the information associated with the target 3D module.

In the embodiment of the present disclosure, the two-dimensional image obtained by the image unit 310 may be sent to the processing unit 330, and the depth information corresponding to the image obtained by the depth information unit 320 may be sent to the processing unit 330. The processing unit 330 may be based on the image and the depth information. Identify the matching target 3D template in the image and output the information associated with the target 3D module. For a specific implementation process, reference may be made to the explanation of the method for processing a three-dimensional model in the embodiments in FIG. 1 to FIG. 2 described above, and details are not described herein.

Further, as a possible implementation manner of the present disclosure, referring to FIG. 7, based on the embodiment shown in FIG. 6, the image processing circuit may further include:

As a possible implementation manner, the image unit 310 may specifically include: an electrically connected image sensor 311 and an image signal processing (Image Signal Processing, ISP) processor 312. among them,

The image sensor 311 is configured to output original image data.

The ISP processor 312 is configured to output an image according to the original image data.

In the embodiment of the present disclosure, the original image data captured by the image sensor 311 is first processed by the ISP processor 312. The ISP processor 312 analyzes the original image data to capture image statistics that can be used to determine one or more control parameters of the image sensor 311. Information, including images in YUV or RGB format. The image sensor 311 may include a color filter array (such as a Bayer filter), and a corresponding photosensitive unit. The image sensor 311 may obtain light intensity and wavelength information captured by each photosensitive unit, and provide information that can be processed by the ISP processor 312. A set of raw image data. After the ISP processor 312 processes the original image data, an image in a YUV format or an RGB format is obtained and sent to the processing unit 330.

Among them, when the ISP processor 312 processes the original image data, it can process the original image data pixel by pixel in a variety of formats. For example, each image pixel may have a bit depth of 8, 10, 12, or 14 bits, and the ISP processor 312 may perform one or more image processing operations on the original image data and collect statistical information about the image data. The image processing operations may be performed with the same or different bit depth accuracy.

As a possible implementation manner, the depth information unit 320 includes an electrically connected structured light sensor 321 and a depth map generation chip 322. among them,

The structured light sensor 321 is configured to generate an infrared speckle pattern.

The depth map generation chip 322 is configured to output depth information according to the infrared speckle map; the depth information includes a depth map.

In the embodiment of the present disclosure, the structured light sensor 321 projects speckle structured light onto a subject, obtains the structured light reflected by the subject, and images the structured light reflected by the subject to obtain an infrared speckle pattern. The structured light sensor 321 sends the infrared speckle pattern to the depth map generation chip 322, so that the depth map generation chip 322 determines the morphological change of the structured light according to the infrared speckle pattern, and then determines the depth of the object to obtain a depth map. (Depth map), the depth map indicates the depth of each pixel in the infrared speckle map. The depth map generation chip 322 sends the depth map to the processing unit 330.

As a possible implementation manner, the processing unit 330 includes a CPU 331 and a GPU (Graphics Processing Unit) 332 which are electrically connected. among them,

The CPU 331 is configured to align the image and the depth map according to the calibration data, and output a three-dimensional model according to the aligned image and the depth map.

The GPU 332 is configured to determine a matching target 3D template according to the 3D model, and output information related to the target 3D template.

In the embodiment of the present disclosure, the CPU 331 obtains the human body image from the ISP processor 312, and obtains the depth map from the depth map generation chip 322. In combination with the calibration data obtained in advance, the two-dimensional image can be aligned with the depth map, thereby determining the image. Depth information corresponding to each pixel. Furthermore, the CPU 331 performs three-dimensional reconstruction based on the depth information and the image to obtain a three-dimensional model.

The CPU 331 sends the three-dimensional model to the GPU 332, so that the GPU 332 executes the three-dimensional model processing method as described in the foregoing embodiment according to the three-dimensional model, implements position adjustment of some key points in the three-dimensional model, and / or renders skin texture of the three-dimensional model. .

Specifically, the GPU 332 may determine a matching target three-dimensional template according to the three-dimensional model, and then perform annotation in the image according to the information associated with the target three-dimensional template, and output an image with the labeled information.

Further, the image processing circuit may further include a display unit 340.

The display unit 340 is electrically connected to the GPU 332 and is configured to display an image with labeled information.

Specifically, the beautified image processed by the GPU 332 may be displayed by the display 340.

Optionally, the image processing circuit may further include: an encoder 350 and a memory 360.

In the embodiment of the present disclosure, the beautified image processed by the GPU 332 may also be encoded by the encoder 350 and stored in the memory 360, where the encoder 350 may be implemented by a coprocessor.

In one embodiment, the memory 360 may be multiple or divided into multiple storage spaces. The image data processed by the storage GPU312 may be stored in a dedicated memory, or a dedicated storage space, and may include DMA (Direct Memory Access, direct and direct). Memory access) feature. The memory 360 may be configured to implement one or more frame buffers.

The above process is described in detail below with reference to FIG. 7.

As shown in FIG. 7, the original image data captured by the image sensor 311 is first processed by the ISP processor 312. The ISP processor 312 analyzes the original image data to capture image statistics that can be used to determine one or more control parameters of the image sensor 311. The information, including images in YUV format or RGB format, is sent to the CPU 331.

As shown in FIG. 7, the structured light sensor 321 projects speckle structured light onto a subject, acquires the structured light reflected by the subject, and forms an infrared speckle pattern based on the reflected structured light. The structured light sensor 321 sends the infrared speckle pattern to the depth map generation chip 322, so that the depth map generation chip 322 determines the morphological change of the structured light according to the infrared speckle pattern, and then determines the depth of the object to obtain a depth map. (Depth Map). The depth map generation chip 322 sends the depth map to the CPU 331.

The CPU 331 obtains a two-dimensional human body image from the ISP processor 312, and obtains a depth map from the depth map generation chip 322. In combination with the calibration data obtained in advance, the face image can be aligned with the depth map, thereby determining the correspondence of each pixel in the image Depth information. Furthermore, the CPU 331 performs three-dimensional reconstruction based on the depth information and the two-dimensional image to obtain a reconstructed three-dimensional model.

The CPU 331 sends the three-dimensional model of the human body to the GPU 332, so that the GPU 332 performs the three-dimensional model processing method described in the foregoing embodiment according to the three-dimensional model of the human body, realizes position adjustment of some key points in the three-dimensional model, and / or renders skin texture of the three-dimensional model. . The processed three-dimensional model obtained by the processing by the GPU 332 may be displayed on the display 340, and / or stored in the memory 360 after being encoded by the encoder 350.

For example, the following are the steps for implementing the control method using the processor 220 in FIG. 5 or the image processing circuit (specifically, the CPU 331 and the GPU 332) in FIG.

Acquiring environmental information of an environment in which the electronic device associated with the three-dimensional model is located;

The CPU 331 obtains a two-dimensional human body image and depth information corresponding to the human body image; the CPU 331 performs three-dimensional reconstruction according to the depth information and the face image to obtain a three-dimensional model of the human body, wherein the three-dimensional model includes multiple Key points, and a model frame formed by the connection of multiple key points, and the skin texture covering the model frame; the CPU 331 further obtains environmental information of the environment in which the electronic device associated with the 3D model is located; the GPU 332 performs a Some key points in the position adjustment, and / or rendering of the skin texture of the 3D model.

In order to implement the above embodiments, the present disclosure also proposes a computer-readable storage medium on which a computer program is stored, characterized in that when instructions in the storage medium are executed by a processor, the implementation is as described in the foregoing embodiment. Processing method of 3D model.

In the description of this specification, the description with reference to the terms “one embodiment”, “some embodiments”, “examples”, “specific examples”, or “some examples” and the like means specific features described in conjunction with the embodiments or examples , Structure, material, or characteristic is included in at least one embodiment or example of the present disclosure. In this specification, the schematic expressions of the above terms are not necessarily directed to the same embodiment or example. Moreover, the particular features, structures, materials, or characteristics described may be combined in any suitable manner in any one or more embodiments or examples. In addition, without any contradiction, those skilled in the art may combine and combine different embodiments or examples and features of the different embodiments or examples described in this specification.

In addition, the terms "first" and "second" are used for descriptive purposes only, and cannot be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, the features defined as "first" and "second" may explicitly or implicitly include at least one of the features. In the description of the present disclosure, the meaning of "a plurality" is at least two, for example, two, three, etc., unless it is specifically and specifically defined otherwise.

Any process or method description in a flowchart or otherwise described herein can be understood as representing a module, fragment, or portion of code that includes one or more executable instructions for implementing steps of a custom logic function or process And, the scope of the preferred embodiments of the present disclosure includes additional implementations in which functions may be performed out of the order shown or discussed, including performing functions in a substantially simultaneous manner or in the reverse order according to the functions involved, which should It is understood by those skilled in the art to which the embodiments of the present disclosure belong.

Logic and / or steps represented in a flowchart or otherwise described herein, for example, a sequenced list of executable instructions that may be considered to implement a logical function, may be embodied in any computer-readable medium, For use by, or in combination with, an instruction execution system, device, or device (such as a computer-based system, a system that includes a processor, or another system that can fetch and execute instructions from an instruction execution system, device, or device) Or equipment. For the purposes of this specification, a "computer-readable medium" may be any device that can contain, store, communicate, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. More specific examples (non-exhaustive list) of computer-readable media include the following: electrical connections (electronic devices) with one or more wirings, portable computer disk cartridges (magnetic devices), random access memory (RAM), Read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disk read-only memory (CDROM). In addition, the computer-readable medium may even be paper or other suitable medium on which the program can be printed, because, for example, by optically scanning the paper or other medium, followed by editing, interpretation, or other suitable Processing to obtain the program electronically and then store it in computer memory.

It should be understood that portions of the present disclosure may be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods may be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. For example, if implemented in hardware as in another embodiment, it may be implemented using any one or a combination of the following techniques known in the art: Discrete logic circuits with logic gates for implementing logic functions on data signals Logic circuits, ASICs with suitable combinational logic gate circuits, programmable gate arrays (PGA), field programmable gate arrays (FPGA), etc.

A person of ordinary skill in the art can understand that all or part of the steps carried by the methods in the foregoing embodiments can be implemented by a program instructing related hardware. The program can be stored in a computer-readable storage medium. The program is When executed, one or a combination of the steps of the method embodiment is included.

In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing module, or each unit may exist separately physically, or two or more units may be integrated into one module. The above integrated modules may be implemented in the form of hardware or software functional modules. If the integrated module is implemented in the form of a software functional module and sold or used as an independent product, it may also be stored in a computer-readable storage medium.

The aforementioned storage medium may be a read-only memory, a magnetic disk, or an optical disk. Although the embodiments of the present disclosure have been shown and described above, it can be understood that the above embodiments are exemplary and should not be construed as limitations on the present disclosure. Those skilled in the art can understand the above within the scope of the present disclosure. Embodiments are subject to change, modification, substitution, and modification.

Claims

A method for processing a three-dimensional model, wherein the method includes the following steps:

Acquiring a three-dimensional model of a human body; wherein the three-dimensional model includes a plurality of key points, a model frame formed by connecting the plurality of key points, and texture information covering the model frame;

Acquiring environmental information of an environment in which the three-dimensional model is constructed;

According to the environment information, position adjustments are performed on some key points in the three-dimensional model, and / or rendering the three-dimensional model texture information.
The processing method according to claim 1, wherein the environment information includes acceleration information; and according to the environment information, a position adjustment is performed on some key points in the three-dimensional model, and / or the 3D model texture information for rendering, including:

Determining a target key point to be adjusted from a plurality of key points of the three-dimensional model;

Adjusting the position of the target key point according to the acceleration information;

Render the texture information to the 3D model after position adjustment.
The processing method according to claim 2, wherein the adjusting the position of the target keypoint according to the acceleration information comprises:

Determining the displacement value corresponding to the target key point according to the acceleration value indicated by the acceleration information and a conversion coefficient between acceleration and displacement corresponding to the target key point;

Determining an adjustment direction of the target key point according to an acceleration direction indicated by the acceleration information;

Moving the target key point along the adjustment direction, and the moving distance conforms to the displacement value.
The processing method according to any one of claims 1-3, wherein the acquiring environmental information of an environment in which the three-dimensional model is constructed comprises:

Measuring a first acceleration vector through an acceleration sensor;

A second acceleration vector measured by a gravity sensor;

Synthesizing the first acceleration vector and the second acceleration vector;

Acceleration information in the environment information is determined according to the acceleration vector obtained through synthesis.
The processing method according to claim 3 or 4, wherein the target key is determined according to an acceleration value indicated by the acceleration information and a conversion coefficient between acceleration and displacement corresponding to the target key point. The displacement value corresponding to the point also includes:

Reading a pre-stored correspondence between the target keypoint and the conversion coefficient;

There are multiple target keypoints, and the value of the conversion coefficient corresponding to each target keypoint is determined according to the material of the three-dimensional model corresponding to each target keypoint and / or the relative position in the three-dimensional model.
The processing method according to any one of claims 1 to 5, wherein the environment information includes lighting information; and rendering the texture information of the three-dimensional model includes:

Determining light intensity and / or light angle according to the illumination information;

Performing light effect rendering on the texture information of the three-dimensional model according to the light intensity and / or light angle.
The processing method according to claim 6, wherein performing light effect rendering on the texture information of the three-dimensional model according to the light intensity and / or light angle comprises:

Adjusting highlight values at different positions of the three-dimensional model according to the light intensity and / or light angle;

Light effect rendering is performed on the texture information of the three-dimensional model according to the highlight value.
The processing method according to any one of claims 1-7, wherein the acquiring a three-dimensional model of a human body comprises:

The depth information is fused with the color information corresponding to the two-dimensional image of the human body to obtain a three-dimensional model of the human body.
The processing method according to claim 8, wherein the fusing the depth information with the color information corresponding to the two-dimensional image of the human body to obtain a three-dimensional model of the human body comprises:

Extracting a first key point of the human body from the depth information, and extracting a second key point of the human body from the color information;

Performing registration and fusion processing on the first key point and the second key point to obtain a fused key point;

According to the fused key points, the three-dimensional model of the human body is generated.
A three-dimensional model processing device, wherein the device includes:

A first acquisition module for acquiring a three-dimensional model of a human body; wherein the three-dimensional model includes a plurality of key points, a model frame formed by a plurality of key points connected, and texture information covering the model frame;

A second acquisition module, configured to acquire environmental information of an environment in which the three-dimensional model is constructed;

A processing module, configured to adjust position of some key points in the three-dimensional model and / or render texture information of the three-dimensional model according to the environment information.
The processing device according to claim 10, wherein the environmental information includes acceleration information; and the processing module includes:

A determining unit, configured to determine a target key point to be adjusted from a plurality of key points of the three-dimensional model;

An adjustment unit, configured to adjust a position of the target key point according to the acceleration information;

The rendering unit is used to render texture information to the three-dimensional model after the position adjustment process.
The processing device according to claim 11, wherein the adjustment unit is configured to:

Determining the displacement value corresponding to the target key point according to the acceleration value indicated by the acceleration information and a conversion coefficient between acceleration and displacement corresponding to the target key point;

Determining an adjustment direction of the target key point according to an acceleration direction indicated by the acceleration information;

Moving the target key point along the adjustment direction, and the moving distance conforms to the displacement value.
The processing device according to any one of claims 10-12, wherein the second acquisition module comprises:

A first measurement unit, configured to measure a first acceleration vector through an acceleration sensor;

A second measurement unit, configured to measure a second acceleration vector through a gravity sensor;

A synthesis unit, configured to synthesize the first acceleration vector and the second acceleration vector;

The second determining unit is configured to determine acceleration information in the environment information according to the synthesized acceleration vector.
The processing device according to claim 12 or 13, wherein the processing device further comprises:

A reading module, configured to read a corresponding relationship between the target keypoint and the conversion coefficient that are stored in advance;

There are multiple target keypoints, and the value of the conversion coefficient corresponding to each target keypoint is determined according to the material of the three-dimensional model corresponding to each target keypoint and / or the relative position in the three-dimensional model.
The processing device according to any one of claims 10 to 14, wherein the environment information includes lighting information; and the rendering unit is configured to:

Determining light intensity and / or light angle according to the illumination information;

Performing light effect rendering on the texture information of the three-dimensional model according to the light intensity and / or light angle.
The processing device according to claim 15, wherein the rendering unit is further configured to:

Adjusting highlight values at different positions of the three-dimensional model according to the light intensity and / or light angle;

Light effect rendering is performed on the texture information of the three-dimensional model according to the highlight value.
The processing device according to any one of claims 10 to 16, wherein the acquisition module comprises:

A fusion unit is used to fuse the depth information with the color information corresponding to the two-dimensional image of the human body to obtain a three-dimensional model of the human body.
The processing device according to claim 17, wherein the fusion unit is configured to:

Extracting a first key point of the human body from the depth information, and extracting a second key point of the human body from the color information;

Performing registration and fusion processing on the first key point and the second key point to obtain a key point after fusion;

According to the fused key points, the three-dimensional model of the human body is generated.
An electronic device, comprising: a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the method according to any one of claims 1-9 is implemented. A method for processing a three-dimensional model.
A computer-readable storage medium having stored thereon a computer program, characterized in that when the program is executed by a processor, a method for processing a three-dimensional model according to any one of claims 1-9 is implemented.