CN117994348A

CN117994348A - Image processing method, related equipment and structured light system

Info

Publication number: CN117994348A
Application number: CN202211349694.0A
Authority: CN
Inventors: 宋钊; 曹军; 刘利刚
Original assignee: Huawei Technologies Co Ltd
Current assignee: Huawei Technologies Co Ltd
Priority date: 2022-10-31
Filing date: 2022-10-31
Publication date: 2024-05-07
Also published as: WO2024093282A1

Abstract

The application discloses an image processing method, related equipment and a structured light system, which can be applied to the structured light system. The method comprises the following steps: acquiring at least three image groups, wherein the at least three image groups are reflection images of the object surface under at least three visual angles aiming at the material coding pattern; at least three image groups are in one-to-one correspondence with at least three viewing angles; acquiring the initial depth of the object surface corresponding to any one of at least three image groups; parameter information of the object is generated based on the at least three image groups and the initial depth, wherein the parameter information comprises texture map parameters and/or geometric structure parameters. And generating parameter information based on the at least three image groups and the initial depth of the object surface, wherein the parameter information comprises texture mapping parameters and/or geometric structure parameters. Thereby realizing the acquisition of the parameters of the texture mapping.

Description

Image processing method, related equipment and structured light system

Technical Field

The present application relates to the field of image processing, and in particular, to an image processing method, a related device, and a structured light system.

Background

Structured light technology is a three-dimensional reconstruction technology based on triangulation, and a typical structured light system is composed of a camera and a projector. In the scanning process, the projector firstly projects a pattern with specific coding information onto the surface of a target scene, the industrial camera acquires the reflected coding information, then the correspondence between the projector and the camera pixels is established through decoding, and finally the depth information of the target scene is acquired based on the triangulation principle.

However, with the development of metauniverse and three-dimensional digital industry, three-dimensional reconstruction technology and system need to meet the requirements of high precision and high reality at the same time, namely high-precision geometry and texture mapping.

Therefore, how to obtain the texture map is a technical problem to be solved.

Disclosure of Invention

The embodiment of the application provides an image processing method, related equipment and a structured light system, which are used for obtaining a texture map.

An embodiment of the present application provides an image processing method, which may be applied to a structured light system. The method may be performed by an image processing apparatus or by a component of an image processing apparatus (e.g., a processor, a chip, or a system-on-chip, etc.). The method comprises the following steps: acquiring at least three image groups, wherein the at least three image groups are reflection images of the object surface under at least three visual angles aiming at the material coding pattern; the at least three image groups are in one-to-one correspondence with the at least three viewing angles; acquiring the initial depth of the object surface corresponding to any one of at least three image groups; parameter information of the object is generated based on the at least three image groups and the initial depth, wherein the parameter information comprises texture map parameters and/or geometric structure parameters.

In the embodiment of the application, at least three image groups obtained by reflecting the object surface under at least three visual angles with respect to the material coding pattern are used for generating parameter information based on the at least three image groups and the initial depth of the object surface, wherein the parameter information comprises material mapping parameters and/or geometric structure parameters. Thereby realizing the acquisition of the parameters of the texture mapping.

Optionally, in a possible implementation manner of the first aspect, the method is applied to a structured light system, the structured light system including a camera, a projection device, a rotation device, and an object connected to the rotation device, the rotation device being configured to rotate a plurality of times such that the object is located at least three viewing angles; acquiring at least three image groups, comprising: triggering the projection device to project a texture-encoding pattern onto the object at each of at least three viewing angles; triggering a camera to acquire image groups reflected by the object aiming at the material coding pattern under each view angle so as to acquire at least three image groups. The initial depth is obtained by means of the projection device projecting a structured light encoding pattern towards the object.

In this possible implementation, an additional light source and RGB camera are required to obtain multi-view RGB images compared to the existing structured light based material measurement scheme. According to the embodiment, the multi-view RGB image required by material modeling can be obtained by changing the coding strategy of the structured light without adding an additional light source and camera.

Optionally, in a possible implementation manner of the first aspect, the material coding pattern includes a full black pattern and a full white pattern. Each of the at least three image groups includes two reflection images.

In the possible implementation manner, through adding two material coding patterns of full white and full black, multi-view RGB images required by material modeling can be obtained without adding an additional light source and camera.

Optionally, in a possible implementation manner of the first aspect, the steps are as follows: generating parameter information of the object based on the at least three images and the initial depth, comprising: acquiring shielding information of space point clouds corresponding to the space point clouds on the surface of an object in a target image group in two image groups, wherein any one image group is the target image group, and the two image groups are two image groups except the target image group in at least three image groups; removing pixel values corresponding to shielding information in the two image groups to obtain a visual matrix; based on pose calibration information and initialization depth, obtaining an observation matrix of each point on the surface of the object under at least three visual angles, wherein the observation matrix comprises: the pixel observation values under at least three visual angles of incident light direction, reflected light direction and light source intensity, and the pose calibration information comprises: the projection equipment is used for projecting material coding patterns, and the camera is used for collecting at least three image groups; parameter information is determined based on the visualization matrix and the observation matrix.

In the possible implementation manner, the observation matrix and the visualization matrix can be obtained through pose calibration information and relative position relations, and further parameter information of the object is obtained according to the observation matrix and the visualization matrix.

Optionally, in a possible implementation manner of the first aspect, the steps are as follows: determining parameter information based on the visualization matrix and the observation matrix, comprising: constructing an energy function based on the visual matrix and the observation matrix, wherein the energy function is used for representing the difference between the estimated value and the observed value of each point on the surface of the object, the estimated value is related to the visual matrix, and the observed value is related to the observation matrix; the value of the energy function is minimized to obtain parameter information.

In this possible implementation, by constructing an energy function of the estimated value and the observed value based on the visualization matrix and the observation matrix, the texture map parameters and/or geometry parameters may be optimized in minimizing the energy function.

Optionally, in a possible implementation manner of the first aspect, the parameter information includes: texture mapping parameters and/or geometry parameters, the geometry parameters including optimized depth or initialization depth; the energy function is shown in equation one:

Equation one:

Wherein the texture mapping parameters include Is a diffuse reflection variable; /(I)Is a specular reflection variable; /(I)Is roughness; z ^* is the geometry parameter; /(I)The calculation mode of the estimated value is shown as a formula II, and I ⁱ is an observed value of the arbitrary point at any one view angle (also can be understood as a pixel difference value between a reflected image corresponding to a full black pattern and a reflected image corresponding to a full white pattern at any one view angle) obtained by the camera; i is the number of said different viewing angles; e is a regularization term;

Formula II:

Wherein E ⁱ is the light source intensity of the arbitrary point at any view angle, d is the distance between the arbitrary point and the projector, f () is a reflection characteristic function, where the reflection characteristic function is shown in formula three, n is the surface normal vector of the arbitrary point, l ⁱ is the incident light direction of the arbitrary point at any view angle, and v ⁱ is the reflected light direction of the arbitrary point at any view angle;

And (3) a formula III:

Wherein, Is an initial diffuse reflection variable; /(I)Is the initial specular reflection variable; r _s is the initial roughness; d () represents a microplane distribution function and G () represents a geometric attenuation coefficient.

A second aspect of an embodiment of the present application provides an image processing method, which may be applied to a structured light system, the structured light system including: camera, projection device, rotation device, and object. The method may be performed by an image processing apparatus or by a component of an image processing apparatus (e.g., a processor, a chip, or a system-on-chip, etc.). The method comprises the following steps: triggering/controlling the projection device to project a texture-coded pattern onto the object; triggering/controlling a camera to collect reflected images of an object under different visual angles aiming at a material coding pattern, wherein the reflected images are used for generating a material map of the object; the triggering/controlling rotation device rotates the object to achieve that the object is located at different viewing angles.

In this embodiment, compared to the existing structured light-based material measurement scheme, an additional light source and an RGB camera are required to obtain multi-view RGB images. According to the embodiment, the multi-view RGB image required by material modeling can be obtained by changing the coding strategy of the structured light without adding an additional light source and camera.

Optionally, in a possible implementation manner of the second aspect, the method further includes: and generating material mapping parameters of the object based on the reflected images at different viewing angles.

Optionally, in a possible implementation manner of the second aspect, the material coding pattern includes a full black pattern and a full white pattern. The number of reflected images at each of the different viewing angles is two.

A third aspect of the embodiments of the present application provides an image processing apparatus that can be applied to a structured light system. The image processing apparatus includes: the acquisition unit is used for acquiring at least three image groups, wherein the at least three image groups are reflection images of the object surface aiming at the material coding pattern under at least three visual angles; the acquisition unit is also used for acquiring the initial depth of the object surface corresponding to any one of the at least three image groups; and the generating unit is used for generating parameter information of the object based on at least three image groups and the initial depth, wherein the parameter information comprises texture mapping parameters and/or geometric structure parameters.

Optionally, in a possible implementation manner of the third aspect, the image processing device described above is applied to a structured light system, the structured light system including a camera, a projection device, a rotation device, and an object connected to the rotation device, the rotation device being configured to rotate a plurality of times such that the object is located at least three viewing angles; the acquisition unit is specifically used for triggering the projection equipment to project the material coding pattern to the object under each of at least three visual angles; the acquisition unit is specifically used for triggering the camera to acquire image groups of the object reflected by the material coding pattern under each view angle so as to acquire at least three image groups. The at least three image groups are in one-to-one correspondence with the at least three viewing angles; the initial depth is obtained by means of the projection device projecting a structured light encoding pattern towards the object.

Optionally, in a possible implementation manner of the third aspect, the material coding pattern includes a full black pattern and a full white pattern. The number of reflected images at each of the different viewing angles is two.

Optionally, in one possible implementation manner of the third aspect, the generating unit is specifically configured to obtain shielding information of a spatial point cloud of an object surface in a target image group corresponding to the spatial point cloud in two image groups, where any one image group is the target image group, and the two image groups are two image groups except for the target image group in at least three image groups; the generating unit is specifically used for eliminating pixel values corresponding to the shielding information in the two image groups so as to obtain a visual matrix; the generating unit is specifically configured to obtain, based on pose calibration information and initialization depth, an observation matrix of each point on the surface of the object under at least three viewing angles, where the observation matrix includes: the pixel observation values under at least three visual angles of incident light direction, reflected light direction and light source intensity, and the pose calibration information comprises: the projection equipment is used for projecting material coding patterns, and the camera is used for collecting at least three image groups; the generation unit is specifically used for determining parameter information based on the visualization matrix and the observation matrix.

Optionally, in a possible implementation manner of the third aspect, the generating unit is specifically configured to construct an energy function based on the visualization matrix and the observation matrix, where the energy function is used to represent a difference between an estimated value and an observed value of each point on the surface of the object, the estimated value is related to the visualization matrix, and the observed value is related to the observation matrix; and a generation unit, in particular for minimizing the value of the energy function to obtain the parameter information.

Optionally, in a possible implementation manner of the third aspect, the parameter information includes: texture mapping parameters and/or geometry parameters, the geometry parameters including optimized depth or initialization depth; the energy function is shown in equation one:

Equation one:

Formula II:

And (3) a formula III:

A fourth aspect of the embodiments of the present application provides an image processing apparatus that can be applied to a structured light system. The image processing apparatus includes: the control unit is used for triggering/controlling the projection equipment to project the material coding pattern to the object; the control unit is also used for triggering/controlling the camera to acquire reflected images of the object under different visual angles aiming at the material coding patterns, and the reflected images are used for generating material mapping parameters of the object; and the control unit is also used for triggering/controlling the rotating equipment to rotate the object so as to realize that the object is positioned at different visual angles.

Optionally, in a possible implementation manner of the fourth aspect, the image processing apparatus further includes a generating unit, configured to generate texture map parameters of the object based on the reflected images at different viewing angles.

Optionally, in a possible implementation manner of the fourth aspect, the texture coding pattern includes a full black pattern and a full white pattern. The number of reflected images at each of the different viewing angles is two.

A fifth aspect of an embodiment of the present application provides a structured light system, the structured light system comprising: camera, projection device, rotation device, and object; the camera is used for collecting reflected images of the object under different view angles aiming at material coding patterns, and the material coding patterns are used for obtaining material mapping parameters of the object; a projection device for projecting a texture-encoding pattern onto a surface of an object; and the rotating device is used for rotating the object so as to realize that the object is positioned at different visual angles.

A sixth aspect of an embodiment of the present application provides an image processing apparatus including: a processor coupled to the memory for storing a program or instructions that when executed by the processor cause the image processing apparatus to implement the method of the first aspect or any possible implementation of the first aspect or cause the image processing apparatus to implement the method of the second aspect or any possible implementation of the second aspect.

A seventh aspect of the embodiments of the present application provides a computer-readable storage medium storing one or more computer-executable instructions which, when executed by a processor, perform a method as described above for any one of the possible implementations of the first aspect or the first aspect, or perform a method as described above for any one of the possible implementations of the second aspect or the second aspect.

An eighth aspect of the embodiments of the present application provides a computer program product (or computer program) storing one or more computers, which when executed by the processor performs the method of any one of the possible implementations of the first aspect or the first aspect, or performs the method of any one of the possible implementations of the second aspect or the second aspect.

A ninth aspect of the embodiments of the present application provides a chip system comprising at least one processor for supporting an image processing apparatus to implement the functions as referred to in the first aspect or any one of the possible implementations of the first aspect or to implement the functions as referred to in the second aspect or any one of the possible implementations of the second aspect.

In one possible design, the system-on-chip may further include a memory to hold the necessary program instructions and data for the first communication device. The chip system can be composed of chips, and can also comprise chips and other discrete devices. Optionally, the chip system further comprises an interface circuit providing program instructions and/or data to the at least one processor.

From the above technical scheme, the application has the following advantages: and generating parameter information based on the at least three image groups and the initial depth of the object surface, wherein the parameter information comprises texture mapping parameters and/or geometric structure parameters. Thereby realizing the acquisition of the parameters of the texture mapping.

Drawings

Fig. 1 is a schematic structural diagram of an application scenario provided in an embodiment of the present application;

FIG. 2 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 3 is an exemplary diagram of a structured light encoding pattern and a texture encoding pattern according to an embodiment of the present application;

FIG. 4 is a schematic flow chart of a data processing method according to an embodiment of the present application;

FIG. 5 is a schematic diagram of another flow chart of a data processing method according to an embodiment of the present application;

FIG. 6 is an exemplary diagram of a vase and vase geometry provided in accordance with an embodiment of the present application;

FIG. 7 is an exemplary diagram of a texture map generated by the prior art and the results of texture map-based object reconstruction;

FIG. 8 is an exemplary diagram of a texture map and a texture map-based object reconstruction result according to an embodiment of the present application;

FIG. 9 is a schematic diagram of system hardware according to an embodiment of the present application;

fig. 10 is a schematic structural view of an image processing apparatus according to an embodiment of the present application;

fig. 11 is another schematic structural view of an image processing apparatus according to an embodiment of the present application.

Detailed Description

The application provides an image processing method, related equipment and a structured light system, which are used for achieving acquisition of a texture map.

At present, the structured light technology can be divided into three development stages, and a structured light system in a stage one only supports geometric output and has no texture and texture mapping output function; the structured light system in stage two supports texture map output on a stage one geometry basis by additionally adding a Red Green Blue (RGB) camera and a light source, but the texture map cannot correctly separate diffuse reflection and highlight components, has obvious highlight noise, and does not support physical-based rendering (Physical base rendering, PBR); in order to meet the requirements of high realism and high precision of three-dimensional modeling, a structured light system capable of supporting the output of PBR material mapping is a development trend of the next stage.

In one aspect, as described in the background art, the existing structured light technology can only acquire depth information of a target scene. The texture map cannot be obtained. On the other hand, in order to obtain multi-view images required by material measurement, on the basis of a structured light system, a light source and a camera are additionally added to the existing hardware system for supporting the material measurement of the spatial bidirectional reflection distribution function (SPATIALLY-Variant Bidirectional Reflectance Distribution Function, svBRDF). However, this approach often consists of multiple cameras, multiple projectors, and several tens of different directional high power white Light Emitting Diode (LED) area lights. The system integration complexity is increased, and the equipment is huge and inconvenient to use.

In order to solve the above-mentioned problems, an embodiment of the present application provides an image processing method, which generates parameter information including texture map parameters and/or geometric parameters based on at least three image groups obtained by reflecting an object surface with respect to a texture coding pattern under at least three viewing angles without adding additional light sources and cameras, and generates the parameter information based on the at least three image groups and an initial depth of the object surface. Thereby realizing the acquisition of the parameters of the texture mapping.

For ease of understanding, related terms and concepts primarily related to embodiments of the present application are described below.

1. Texture map

The texture map of the support svBRDF (i.e.) includes: diffuse reflectance mapping (diffuse map), specular reflectance mapping (specular map), roughness mapping (rouchness map), normal mapping (normal map).

Before describing the method provided by the embodiment of the present application, an application scenario to which the method provided by the embodiment of the present application is applicable is described. The scene to which the method provided by the embodiment of the application is applicable may be a structured light system as shown in fig. 1. The structured light system comprises: camera 101, projection device 102, rotation device 103, object 104.

Wherein the camera 101 is used for acquiring reflected images of the object 104 under different viewing angles for the coding pattern. The coding pattern comprises a texture coding pattern, and the texture coding pattern is used for obtaining texture mapping parameters.

A projection device 102 for projecting a coding pattern onto a surface of an object 104.

A rotation device 103 for rotating the object 104 to achieve that the object 104 is located at different viewing angles.

Object 104, which may be understood as an object to be scanned in three dimensions.

Optionally, the rotation device 103 is used for placing the object 104. It will be appreciated that the rotation device 13 may also comprise a plurality of holders to support or secure the object 104. So as to realize the rotation of the rotating device 103 and simultaneously drive the object 104 to rotate.

Optionally, the coding pattern may also comprise a structured light coding pattern for obtaining the machine geometry (e.g. depth) of the object.

During acquisition, the projection device 102 projects a specific coding pattern onto the object 104, and the camera 101 takes a picture to obtain a corresponding reflected image. After one scan is completed, the rotating device 103 rotates a specific angle, and the above image acquisition process is repeated for a plurality of times, which is often greater than 2 times, and the times can be set according to actual needs. In the calculation process, the image processing apparatus generates parameter information of the object 104 according to the reflected images acquired in the above-mentioned multiple acquisition processes, so as to implement three-dimensional reconstruction of the object 104.

The image processing device in the embodiment of the present application may be a server, a mobile phone, a tablet computer (pad), a portable game machine, a personal computer (PDA), a notebook computer, an ultra mobile personal computer (ultra mobile personal computer, UMPC), a handheld computer, a netbook, a vehicle-mounted media playing device, a wearable electronic device, a Virtual Reality (VR) terminal device, an augmented reality (augmented reality, AR) terminal device, or the like with sufficient computing power.

The image processing method provided by the embodiment of the application is described in detail below. The method may be performed by an image processing apparatus. Or by a component of the image processing device, such as a processor, chip, or system-on-chip, etc. The method may be applied to the structured light system shown in fig. 1, please refer to fig. 2, which is a flowchart of an image processing method according to an embodiment of the present application, and the method may include steps 201 to 207. Steps 201 to 207 are described in detail below.

Step 201, triggering the projector.

The image processing device sends a first trigger signal to the projector (i.e., the projection device in fig. 1). Accordingly, the projector receives a first trigger signal sent by the image processing device. The first trigger signal is used by the projector to project a structured light encoding pattern onto the object.

Step 202, projecting a structured light code.

After the projector receives the first trigger signal, the projector projects a structured light encoding pattern (or structured light encoding) onto the object. The structured light encoding pattern is used to obtain the depth of the object.

Illustratively, the structured light encoding pattern is as shown in fig. 3 (a). It will be appreciated that (a) in fig. 3 is merely an example of 8 coding patterns (8 rows respectively). In practical applications, there may also be a fewer (e.g., 4, 5, etc.) or greater (e.g., 16, 20, etc.) number of encoding patterns. The structured light encoding pattern may include a plurality of black-and-white patterns or a plurality of patterns corresponding to 0 to 255.

Step 203, triggering camera acquisition.

After the projector projects the structured light encoding pattern, the image processing device sends second trigger information to the camera to trigger the camera to acquire an image of the object surface reflected against the structured light encoding pattern. The second trigger information is used for the camera to collect a reflected image of the object surface.

Alternatively, the image may be used as an input for structured light decoding to obtain the initial depth of the object.

In step 204, a texture code is projected.

After the structured light code projection and collection is completed. The image processing device sends third trigger information to the projector to trigger the projector to project a texture-coded pattern (or texture code). The third trigger information is used for the projector to project a material coding pattern to the object. The material coding pattern comprises a full black pattern and a full white pattern.

Illustratively, the texture coding pattern is as shown in (b) of fig. 3.

Step 205, triggering camera acquisition.

After the projector projects the material coding pattern, the image processing device sends fourth trigger information to the camera to trigger the trigger camera to acquire the RGB image reflected by the object surface.

Alternatively, the RGB image may be input to photometric constraint modeling.

Step 206, the turntable is triggered.

The image processing device triggers the turntable (i.e. the rotating device in fig. 1) to rotate a specific angle, and obtains RGB images of different poses through the relative movement of the object and each device in the structured light acquisition.

In addition, based on the set turntable angle, space point cloud splicing and fusion of objects corresponding to RGB images of different poses are completed, and a scanning result is obtained.

Step 207, the determination is ended.

Judging whether the system completes the preset acquisition times or not; if yes, stopping triggering of the projector, and ending acquisition; if not, repeating steps 201-207 until the end.

It should be understood that the above process is exemplified by projecting the optical structure coding pattern and the material coding pattern at each view angle, and in practical application, the optical structure coding pattern may be projected at the main view angle, and the material coding pattern may be projected at the main view angle and other view angles, which is not limited herein.

In this embodiment, by providing an image acquisition manner, a multi-view RGB image required for material modeling can be obtained. And taking the multi-view RGB image as a material modeling input. Compared with the existing structured light-based material measurement scheme, the multi-view RGB image acquisition method has the advantages that a light source and an RGB camera are additionally arranged to acquire the multi-view RGB image. According to the embodiment, through changing the coding strategy of the structured light and adding the full-white and full-black material coding patterns, namely, the reflection image of the object aiming at the projection pattern is obtained in a mode of projecting the pattern by the projection equipment, and the multi-view RGB image required by material modeling can be obtained without adding a light source and a camera.

Referring to fig. 4, a flowchart of an image processing method according to an embodiment of the present application may be implemented by an image processing apparatus. Or by a component of the image processing device, such as a processor, chip, or system-on-chip, etc. The method may be applied to the structured light system shown in fig. 1, and the method may comprise steps 401 to 403. Steps 401 to 403 are described in detail below.

At step 401, at least three image groups are acquired.

At least three image groups in the embodiment of the application are reflection images of the object surface aiming at the material coding pattern under at least three visual angles. Wherein, at least three image groups are in one-to-one correspondence with at least three viewing angles.

Optionally, the material coding pattern includes a full black pattern and a full white pattern. In this case, at each viewing angle, a first reflected image of the object for the full black pattern and a second reflected image for the full white pattern are acquired. An image set may be acquired at a first viewing angle, the image set comprising a first reflected image and a second reflected image. I.e. the three image sets comprise six reflection images.

The image processing device in the embodiment of the present application may acquire at least three image groups in various manners, which may be a manner of receiving a transmission from another device, or may be a manner of selecting from a database, or may acquire at least three image groups in the manner of the embodiment shown in fig. 2, which is not limited herein.

Step 402, obtaining an initial depth of an object surface corresponding to any one of the at least three image groups.

The image processing device in the embodiment of the present application may acquire the initial depth in various manners, which may be a manner of receiving the transmission of other devices, or may be a manner selected from a database, or may acquire at least three image groups in the manner of the embodiment shown in fig. 2, which is not limited herein.

Optionally, at least one viewing angle (i.e. at least including the viewing angle corresponding to the subsequent set of target images), the projection device projects a structured light encoding pattern onto the object surface, the camera captures a reflected image of the object surface for the structured light encoding pattern, and the initial depth of the object at that viewing angle is obtained from the image.

It will be appreciated that at least three image sets of the preceding step 401 may also include a reflected image of the object surface for the structured light encoding pattern. For example, assuming a scene in which a structured light encoding pattern is projected at each viewing angle, each of the aforementioned at least three image groups includes a first reflected image and a second reflected image at a certain viewing angle. The first reflection image is a reflection image of the object surface aiming at the structured light coding pattern, and the second reflection image is a reflection image of the object surface aiming at the material coding pattern. In other words, the initial depth may be obtained by processing from at least three image groups, or may be obtained by processing from an image other than at least three image groups (i.e., a reflected image of the object surface for the structured light encoding pattern). It will be appreciated that, assuming a scene in which the structured-light encoding pattern is projected at one viewing angle (or referred to as the main viewing angle), the aforementioned at least three image groups may be four image groups, including: the object surface is directed to a reflected image of the material encoding pattern at three viewing angles and the object surface is directed to a reflected image of the structured light encoding pattern at the primary viewing angle. In addition, the number of the reflective images corresponding to the structured light coding pattern is not limited (for example, the description of (a) in fig. 3).

In step 403, parameter information of the object is generated based on the at least three image groups and the initial depth.

After the image processing device obtains the at least three image groups and the initial depth, parameter information of the object is generated based on the at least three image groups and the initial depth, wherein the parameter information comprises texture map parameters and/or geometric structure parameters.

The flow of this step 403 may refer to fig. 5. The fig. 5 includes steps 501 through 506. Steps 501 to 506 are described in detail below.

Step 501, key frames.

The image processing device takes an image group corresponding to any one of at least three views as a key frame. And taking the view corresponding to the key frame as a main view. The point cloud contained in the key frame is an optimization target (or understood as determining the range of the texture map), and the pixel coordinates corresponding to the point cloud in the key frame are image prospects.

Optionally, taking the view corresponding to the initial depth in step 402 as the main view, the image group corresponding to the initial depth is the target image group.

Step 502, determining adjacent frames, and completing image registration (or image calibration).

And selecting at least two adjacent frame images around the key frame as adjacent frames based on the known pose relation among different frames. The key frame point cloud coordinates are re-projected to pixel locations and RGB values in each adjacent frame.

Specifically, the image processing apparatus may acquire the relative positional relationship between at least three image groups and the object. And acquiring shielding information of the space point cloud of the object surface in the target image group corresponding to the space point cloud in the two image groups based on the relative position relation. The two image groups are two image groups other than the target image group among the at least three image groups. The method for obtaining the shielding information may be a method such as re-projection, which is not limited herein.

At step 503, an observation and visualization matrix is generated.

After the image processing device acquires the shielding information, eliminating pixel values corresponding to the shielding information in the two image groups to acquire a visual matrix. This process can be understood as being used to reduce noise from occlusion.

In addition, the image processing device can also acquire an observation matrix of each point on the surface of the object under at least three visual angles based on the pose calibration information and the initialization depth. The observation matrix includes: pixel observations at least three viewing angles of incident light direction, reflected light direction, light source intensity.

The pose calibration information comprises the following steps: the projection equipment is used for projecting material coding patterns, and the camera is used for collecting at least three image groups.

Alternatively, the observation matrix may beFor a point P on the object surface, L is the incident light direction of the point at the I-view angle (which may also be understood as the irradiation direction of the projection device, i.e. the irradiation direction of the light source), V is the reflected light direction of the point at the I-view angle (which may also be understood as the observation direction of the camera), and I is the observed value of the pixel at the point acquired by the camera at the I-view angle (which may also be understood as the pixel difference between the reflected image corresponding to the full black pattern and the reflected image corresponding to the full white pattern at the I-view angle). E is the intensity of the light source at that point at i viewing angle.

After the image processing apparatus acquires the visualization matrix and the observation matrix, parameter information may be determined based on the visualization matrix and the observation matrix (as shown in steps 504 to 506 below). The parameter information includes: texture map parameters and/or geometry parameters, including optimized depth or initialization depth. Where the geometry parameters include initialization depth, the present embodiment may be understood as acquiring texture map parameters. In case the geometry parameters comprise an optimized depth, the present embodiment may be understood as optimizing the geometry parameters of the object.

At step 504, an energy function is established.

The image processing device constructs an energy function based on the visualization matrix and the observation matrix, wherein the energy function is used for representing the difference between the estimated value and the observed value of each point on the surface of the object, the estimated value is related to the visualization matrix, and the observed value is related to the observation matrix.

Alternatively, the energy function is as shown in equation one:

Equation one:

Wherein the texture mapping parameters include The diffuse reflection variable after optimization; /(I)The optimized specular reflection variable is; /(I)The roughness after optimization; z ^* is the optimized geometry parameter; /(I)For the estimated value of any point on the surface of the object under different view angles, the calculation mode of the estimated value is shown as a formula II, and I ⁱ is the observed value of any point under any view angle, which is obtained by a camera; i is the number of different viewing angles; e is a regularization term.

Optionally, the regular term includes a regular term corresponding to a normal vector, a depth, a material, and the like, which is not limited herein.

Formula II:

Wherein E ⁱ is the light source intensity of any point at any view angle, d is the distance between any point and the projector, f () is the reflection characteristic function, the reflection characteristic function is shown in formula three, n is the normal vector of the surface of any point, l ⁱ is the incident light direction of any point at any view angle, v ⁱ is the reflected light direction of any point at any view angle;

And (3) a formula III:

Wherein, Is an initial diffuse reflection variable; /(I)Is the initial specular reflection variable; r _s is the initial roughness; d () represents a micro-plane distribution function for expressing a change in the slope of the micro-plane, and G () represents a geometric attenuation coefficient. Incident light on a micro-plane may be blocked by an adjacent micro-plane before reaching or after being reflected by the surface, which blocking may cause a slight darkening of the specular reflection, which effect may be measured by the geometric attenuation coefficient.

It should be understood that the above formula one, formula two and/or formula three are merely examples, and in practical application, the above formula one, formula two and/or formula three may also be other manners, and are not limited herein.

In step 505, the energy function is minimized.

After the image processing apparatus constructs the energy function, the value of the energy function may be minimized to acquire the parameter information of the object. The parameter information also includes: texture map parameters and/or geometry parameters, including optimized depth or initialization depth. Where the geometry parameters include initialization depth, the present embodiment may be understood as acquiring texture map parameters. In case the geometry parameters comprise an optimized depth, the present embodiment may be understood as optimizing the geometry parameters of the object.

In one possible implementation, the parameter information may include the aboveIn this case, Z ^* is the initial depth Z, that is, the initial depth is a constant value, and the texture map parameter is a parameter to be optimized.

In another possible implementation, the parameter information may include z ^* as described above. In this caseFor/>R _s is a preset value, that is, the texture map parameter is a fixed value, and the geometric structure parameter is a parameter to be optimized.

In another possible implementation, the parameter information may include z ^* as described above,In this case, it is understood that both the texture map parameters and the geometry parameters are parameters to be optimized.

Alternatively, the initial depth/optimized depth may be used to generate a normal map, the initial or optimized diffuse reflectance variable is used to generate a diffuse reflectance map, the initial or optimized specular reflectance variable is used to generate a specular reflectance map, and the initial or optimized roughness is used to generate a roughness map.

And step 506, convergence judgment.

In minimizing the value of the energy function, it is determined whether a convergence condition is satisfied. If yes, ending the minimized energy function and outputting the optimal solution. If not, repeating steps 502-506 until the convergence condition is satisfied. Wherein the convergence condition includes at least one of: the number of repetitions is a first preset threshold, the value of the energy function is less than a second preset threshold, etc.

It will be appreciated that the calculation of the parameter information for the object at the main viewing angle is completed. The above-described process may be repeatedly performed by reselecting one view as the main view. Finally, the parameter information of the object under a plurality of view angles is obtained. And then the generation of the object material map is realized according to the processing such as fusion splicing and the like.

In the embodiment of the application, at least three image groups obtained by reflecting the object surface under at least three visual angles with respect to the material coding pattern are used for generating parameter information based on the at least three image groups and the initial depth of the object surface, wherein the parameter information comprises material mapping parameters and/or geometric structure parameters. On the one hand, on the premise of not adding an additional light source, the projection equipment is changed into a light source by changing the coding strategy of the structured light, so that the PBR texture mapping output can be supported. In another aspect, the texture maps include diffuse reflection maps, specular reflection maps, roughness maps, and normal maps, supporting PBR rendering. On the other hand, the material modeling and solving algorithm can be used for measuring the material of the mobile phone terminal, a material modeling scheme is provided for the existing three-dimensional reconstruction algorithm based on the mobile phone terminal, and the material output function of the mobile phone terminal is supported.

In order to more intuitively see the beneficial effects of the image processing method provided by the embodiment of the application, a reconstructed vase is taken as an example, and the reconstruction result of the vase by using the prior art and the image processing method provided by the application is described in an exemplary way.

The object is a physical diagram of the vase as shown in fig. 6 (a), and fig. 6 (b) is a geometric structural diagram of the object. Fig. 7 (a) shows a texture map generated by the prior art, and fig. 7 (b) shows the reconstruction result of the texture map in the prior art. Fig. 8 (a) shows a texture map obtained by the method according to the embodiment of the present application, and fig. 8 (b) shows a reconstruction result of the texture map obtained by the method according to the embodiment of the present application. Through comparison, the existing texture-based reconstruction result cannot separate diffuse reflection and highlight components correctly, so that highlight noise exists in the texture-map-based reconstruction result, and PBR rendering is not supported. The image processing method provided by the embodiment of the application can effectively generate the PBR material map, acquire the reconstruction result with high sense of reality and reduce highlight noise.

In addition, the embodiment of the application also provides system hardware. The system hardware is as shown in fig. 9, and includes: the device comprises a turntable unit, a control unit, a polishing unit, a sensor unit, a storage unit and a calculation unit. In the acquisition process, a control unit firstly sends out a trigger signal to enable a projector to project a specific coding pattern, and a camera is triggered by the projector to shoot so as to acquire a corresponding image, and the corresponding image is uploaded to a storage unit. After one scanning is completed, the control unit controls the turntable to rotate by a specific angle, and the image acquisition process is repeated for a preset number of times; after the complete scan is completed, the calculation of the object parameter information (i.e., including the geometry and svBRDF) is completed by the calculation unit.

Wherein the turntable unit may include a turntable and a power supply. The control unit may include a central processing unit (central processing unit, CPU) and a cache. The light unit includes a power source and a projector. The sensor unit includes a camera and a transmission line. The storage unit comprises a cache and an external storage. The computing unit includes a CPU, a graphics processor (graphics processing unit, GPU), a cache, and a transmission line.

The image processing method in the embodiment of the present application is described above, and the image processing apparatus in the embodiment of the present application is described below with reference to fig. 10, where an embodiment of the image processing apparatus in the embodiment of the present application includes:

An obtaining unit 1001, configured to obtain at least three image groups, where the at least three image groups are reflection images of a surface of an object for a texture coding pattern under at least three viewing angles;

The acquiring unit 1001 is further configured to acquire an initial depth of an object surface corresponding to any one of the at least three image groups;

The generating unit 1002 is configured to generate parameter information of the object based on the at least three image groups and the initial depth, where the parameter information includes a texture map parameter and/or a geometry parameter.

Optionally, the image processing device is applied to a structured light system, the structured light system comprising a camera, a projection device, a rotation device and an object connected to the rotation device, the rotation device being adapted to rotate a plurality of times such that the object is located at least three viewing angles;

Optionally, the acquiring unit 1001 is specifically configured to trigger the projection device to project a texture coding pattern to the object at each of at least three viewing angles; the acquiring unit 1001 is specifically configured to trigger the camera to acquire, at each view angle, image groups of the object reflected by the texture coding pattern, so as to acquire at least three image groups.

Optionally, the texture coding pattern includes a full black pattern and a full white pattern.

Optionally, the generating unit 1002 is specifically configured to obtain shielding information of a spatial point cloud of an object surface in a target image group corresponding to the spatial point cloud in two image groups, where any one image group is the target image group, and the two image groups are two image groups except for the target image group in at least three image groups; the generating unit 1002 is specifically configured to reject pixel values corresponding to the occlusion information in the two image groups, so as to obtain a visualization matrix; the generating unit 1002 is specifically configured to obtain, based on pose calibration information and an initialization depth, an observation matrix of each point on the surface of the object under at least three viewing angles, where the observation matrix includes: the pixel observation values under at least three visual angles of incident light direction, reflected light direction and light source intensity, and the pose calibration information comprises: the projection equipment is used for projecting material coding patterns, and the camera is used for collecting at least three image groups; the generating unit 1002 is specifically configured to determine parameter information based on the visualization matrix and the observation matrix.

Optionally, the generating unit 1002 is specifically configured to construct an energy function based on the visualization matrix and the observation matrix, where the energy function is used to represent a difference between an estimated value and an observed value of each point on the surface of the object, the estimated value is related to the visualization matrix, and the observed value is related to the observation matrix;

Optionally, the generating unit 1002 is specifically configured to minimize a value of the energy function to obtain the parameter information.

In this embodiment, operations performed by each unit in the image processing apparatus are similar to those described in the embodiments shown in fig. 1 to 5, and are not described here again.

In this embodiment, on the one hand, on the premise of not adding an additional light source, the projection device is changed into a light source by changing the coding strategy of the structured light, so that the output of the PBR texture map can be supported. In another aspect, the texture maps include diffuse reflection maps, specular reflection maps, roughness maps, and normal maps, supporting PBR rendering. On the other hand, the material modeling and solving algorithm can be used for measuring the material of the mobile phone terminal, a material modeling scheme is provided for the existing three-dimensional reconstruction algorithm based on the mobile phone terminal, and the material output function of the mobile phone terminal is supported.

Referring to fig. 11, another image processing apparatus according to the present application is schematically shown. The image processing device may include a processor 1101, a memory 1102, and a communication port 1103. The processor 1101, memory 1102 and communication ports 1103 are interconnected by wires. Wherein program instructions and data are stored in memory 1102.

The memory 1102 stores therein program instructions and data corresponding to the steps executed by the image processing apparatus in the respective embodiments shown in fig. 1 to 5.

A processor 1101 for executing the steps executed by the image processing apparatus as shown in any of the embodiments shown in the foregoing fig. 1 to 5.

The communication port 1103 may be used for receiving and transmitting data, and for performing the steps related to acquiring, transmitting and receiving in any of the embodiments shown in fig. 1 to 5.

In one implementation, the image processing device may include more or less components relative to FIG. 11, which is only exemplary and not limiting.

Embodiments of the present application also provide a computer-readable storage medium storing one or more computer-executable instructions that, when executed by a processor, perform a method as described in a possible implementation of the image processing apparatus in the previous embodiments.

Embodiments of the present application also provide a computer program product (or computer program) storing one or more computers, which when executed by the processor performs a method as described above as a possible implementation of an image processing apparatus.

The embodiment of the application also provides a chip system which comprises at least one processor and is used for supporting the terminal equipment to realize the functions related in the possible realization modes of the image processing equipment. Optionally, the chip system further comprises an interface circuit providing program instructions and/or data to the at least one processor. In one possible design, the system-on-chip may further include a memory to hold program instructions and data necessary for the image processing device. The chip system can be composed of chips, and can also comprise chips and other discrete devices.

It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, which are not repeated herein.

In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.

The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

In addition, each functional unit in the embodiments of the present application may be integrated in one processing unit, or each unit may exist alone physically, or two or more units may be integrated in one unit. The integrated units may be implemented in hardware or in software functional units.

The integrated units, if implemented in the form of software functional units and sold or used as stand-alone products, may be stored in a computer readable storage medium. Based on such understanding, the technical solution of the present application may be embodied essentially or in part or all of the technical solution or in part in the form of a software product stored in a storage medium, including instructions for causing a computer device (which may be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method according to the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random-access memory (RAM, random access memory), a magnetic disk, or an optical disk, or other various media capable of storing program codes.

Claims

1. An image processing method, the method comprising:

Acquiring at least three image groups, wherein the at least three image groups are reflection images of the surface of the object under at least three visual angles, which aim at material coding patterns; the at least three image groups are in one-to-one correspondence with the at least three viewing angles;

Acquiring the initial depth of the object surface corresponding to any one image group in the at least three image groups;

parameter information of the object is generated based on the at least three image groups and the initial depth, wherein the parameter information comprises texture map parameters and/or geometric structure parameters.

2. The method according to claim 1, wherein the method is applied to a structured light system comprising a camera, a projection device, a rotation device and an object connected to the rotation device, the rotation device being adapted to be rotated a plurality of times such that the object is located at the at least three viewing angles;

The acquiring at least three image groups includes:

Triggering the projection device to project a texture-coded pattern onto the object at each of the at least three viewing angles;

triggering the camera to acquire image groups of the object reflected by the material coding pattern under each view angle so as to acquire at least three image groups;

the initial depth is obtained by means of the projection device projecting a structured light encoding pattern towards the object.

3. The method of claim 1 or 2, wherein the texture coding pattern comprises a full black pattern and a full white pattern, and each of the at least three image groups comprises two reflection images.

4. A method according to any one of claims 1 to 3, wherein the generating parameter information of the object based on the at least three images and the initial depth comprises:

Acquiring shielding information of space point clouds corresponding to the space point clouds of the object surface in two image groups, wherein any one image group is the target image group, and the two image groups are two image groups except the target image group in the at least three image groups;

Removing pixel values corresponding to the shielding information in the two image groups to obtain a visual matrix;

Obtaining an observation matrix of each point on the object surface under the at least three view angles based on pose calibration information and the initialization depth, wherein the observation matrix comprises: the pixel observation values under the at least three visual angles of the incident light direction, the reflected light direction and the light source intensity, and the pose calibration information comprises: an internal parameter of a projection device and a camera, an external parameter between the projection device and the camera, wherein the projection device is used for projecting the material coding pattern, and the camera is used for acquiring the at least three image groups;

the parameter information is determined based on the visualization matrix and the observation matrix.

5. The method of claim 4, wherein the determining the parameter information based on the visualization matrix and the observation matrix comprises:

Constructing an energy function based on the visualization matrix and the observation matrix, wherein the energy function is used for representing the difference between an estimated value and an observed value of each point on the surface of the object, the estimated value is related to the visualization matrix, and the observed value is related to the observation matrix;

the value of the energy function is minimized to obtain the parameter information.

6. The method of claim 5, wherein the parameter information comprises: the texture mapping parameters and/or the geometric parameters, wherein the geometric parameters comprise optimized depth or initialization depth; the energy function is shown in formula one:

Equation one:

Wherein the texture mapping parameters include Is a diffuse reflection variable; /(I)Is a specular reflection variable; /(I)Is roughness; z ^* is the geometry parameter; /(I)The calculation mode of the estimated value is shown as a formula II, and I ⁱ is an observed value of the arbitrary point at any one view angle, which is obtained by the camera; i is the number of said different viewing angles; e is a regularization term;

Formula II:

And (3) a formula III:

7. An image processing apparatus, characterized in that the image processing apparatus comprises:

the device comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring at least three image groups, wherein the at least three image groups are reflection images of the surface of an object aiming at a material coding pattern under at least three visual angles; the at least three image groups are in one-to-one correspondence with the at least three viewing angles;

The acquisition unit is further used for acquiring the initial depth of the object surface corresponding to any one of the at least three image groups;

And the generating unit is used for generating parameter information of the object based on the at least three image groups and the initial depth, wherein the parameter information comprises texture mapping parameters and/or geometric structure parameters.

8. The image processing apparatus according to claim 7, wherein the image processing apparatus is applied to a structured light system including a camera, a projection apparatus, a rotation apparatus for rotating a plurality of times so that the object is located at the at least three viewing angles, and an object connected to the rotation apparatus;

the acquisition unit is specifically configured to trigger the projection device to project a texture coding pattern to the object under each of the at least three viewing angles;

The acquisition unit is specifically configured to trigger the camera to acquire image groups of the object reflected by the material coding pattern under each view angle, so as to acquire the at least three image groups;

9. The image processing apparatus according to claim 7 or 8, wherein the texture coding pattern comprises a full black pattern and a full white pattern, and each of the at least three image groups comprises two reflection images.

10. The image processing apparatus according to any one of claims 7 to 9, wherein the generating unit is specifically configured to acquire occlusion information of a spatial point cloud of the object surface in a target image group in two image groups corresponding to the spatial point cloud, the arbitrary one image group being the target image group, the two image groups being two image groups other than the target image group of the at least three image groups;

The generating unit is specifically configured to reject pixel values corresponding to the shielding information in the two image groups, so as to obtain a visualization matrix;

The generating unit is specifically configured to obtain, based on pose calibration information and the initialization depth, an observation matrix of each point on the object surface under the at least three view angles, where the observation matrix includes: the pixel observation values under the at least three visual angles of the incident light direction, the reflected light direction and the light source intensity, and the pose calibration information comprises: an internal parameter of a projection device and a camera, an external parameter between the projection device and the camera, wherein the projection device is used for projecting the material coding pattern, and the camera is used for acquiring the at least three image groups;

The generating unit is specifically configured to determine the parameter information based on the visualization matrix and the observation matrix.

11. The image processing device according to claim 10, characterized in that the generating unit is configured in particular to construct an energy function based on the visualization matrix and the observation matrix, the energy function being configured to represent differences between estimated values of points on the object surface and observed values, the estimated values being related to the visualization matrix, the observed values being related to the observation matrix;

The generating unit is specifically configured to minimize a value of the energy function to obtain the parameter information.

12. The image processing apparatus according to claim 11, wherein the parameter information includes: the texture mapping parameters and/or the geometric parameters, wherein the geometric parameters comprise optimized depth or initialization depth; the energy function is shown in formula one:

Equation one:

Formula II:

And (3) a formula III:

13. A structured light system, the structured light system comprising: camera, projection device, rotation device, and object;

the camera is used for collecting reflection images of the object under different view angles aiming at material coding patterns, and the material coding patterns are used for obtaining material mapping parameters of the object;

the projection device is used for projecting the material coding pattern to the surface of the object;

The rotating device is used for rotating the object so as to enable the object to be located at the different view angles.

14. The structured light system of claim 13, wherein the texture coded pattern comprises a full black pattern and a full white pattern, and the number of reflected images at each of the different viewing angles is two.

15. An image processing apparatus, characterized by comprising: a processor coupled to a memory for storing a program or instructions that, when executed by the processor, cause the image processing apparatus to perform the method of any of claims 1 to 6.

16. A computer readable storage medium, characterized in that the medium stores instructions which, when executed by a computer, implement the method of any one of claims 1 to 6.

17. A computer program product comprising instructions which, when run on a computer, cause the computer to perform the method of any one of claims 1 to 6.