WO2024093282A1 - Procédé de traitement d'image, dispositif associé et système de lumière structurée - Google Patents

Procédé de traitement d'image, dispositif associé et système de lumière structurée Download PDF

Info

Publication number
WO2024093282A1
WO2024093282A1 PCT/CN2023/103013 CN2023103013W WO2024093282A1 WO 2024093282 A1 WO2024093282 A1 WO 2024093282A1 CN 2023103013 W CN2023103013 W CN 2023103013W WO 2024093282 A1 WO2024093282 A1 WO 2024093282A1
Authority
WO
WIPO (PCT)
Prior art keywords
image groups
image
point
viewing angles
coding pattern
Prior art date
Application number
PCT/CN2023/103013
Other languages
English (en)
Chinese (zh)
Inventor
宋钊
曹军
刘利刚
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2024093282A1 publication Critical patent/WO2024093282A1/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T15/00—3D [Three Dimensional] image rendering
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/80—Analysis of captured images to determine intrinsic or extrinsic camera parameters, i.e. camera calibration

Definitions

  • the present application relates to the field of image processing, and in particular to an image processing method, related equipment and a structured light system.
  • Structured light technology is a 3D reconstruction technology based on triangulation.
  • a typical structured light system consists of a camera and a projector. During the scanning process, the projector first projects a pattern with specific coded information onto the surface of the target scene. The industrial camera then obtains the reflected coded information, and then decodes to establish the correspondence between the projector and camera pixels. Finally, the depth information of the target scene is obtained based on the principle of triangulation.
  • the embodiments of the present application provide an image processing method, related equipment and a structured light system for acquiring material maps.
  • the first aspect of the embodiment of the present application provides an image processing method that can be applied to a structured light system.
  • the method can be performed by an image processing device, or by a component of the image processing device (such as a processor, a chip, or a chip system, etc.).
  • the method includes: obtaining at least three image groups, at least three image groups are reflection images of the object surface for a material coding pattern at at least three viewing angles; the at least three image groups correspond one-to-one to the at least three viewing angles; obtaining the initial depth of the object surface corresponding to any one of the at least three image groups; generating parameter information of the object based on the at least three image groups and the initial depth, the parameter information including material mapping parameters and/or geometric structure parameters.
  • At least three image groups are obtained by reflecting the material coding pattern from the object surface at least three viewing angles, and parameter information is generated based on the initial depth of the at least three image groups and the object surface, and the parameter information includes material mapping parameters and/or geometric structure parameters, thereby achieving the acquisition of material mapping parameters.
  • the method is applied to a structured light system, the structured light system comprising a camera, a projection device, a rotating device, and an object connected to the rotating device, the rotating device being used to rotate multiple times so that the object is located at at least three viewing angles; acquiring at least three image groups, comprising: triggering the projection device to project a material coding pattern onto the object at each of the at least three viewing angles; triggering the camera to collect an image group reflected by the object for the material coding pattern at each viewing angle, so as to acquire at least three image groups.
  • the initial depth is obtained by the projection device projecting the structured light coding pattern onto the object.
  • this embodiment changes the encoding strategy of structured light, and can obtain multi-view RGB images required for material modeling without additional light sources and cameras.
  • the material coding pattern includes a full black pattern and a full white pattern.
  • Each of the at least three image groups includes two reflection images.
  • the multi-view RGB images required for material modeling can be obtained without adding additional light sources and cameras.
  • the above-mentioned step: generating parameter information of the object based on at least three images and the initial depth includes: obtaining occlusion information of the spatial point cloud of the object surface in the target image group corresponding to the spatial point cloud in the two image groups, any one of the image groups is the target image group, and the two image groups are two image groups other than the target image group in the at least three image groups; eliminating pixel values corresponding to the occlusion information in the two image groups to obtain a visualization matrix; obtaining the image matrix of each point on the surface of the object based on the pose calibration information and the initialization depth.
  • the observation matrix under at least three viewing angles includes: the incident light direction, the reflected light direction, the pixel observation values under at least three viewing angles of the light source intensity;
  • the posture calibration information includes: the intrinsic parameters of the projection device and the camera, the extrinsic parameters between the projection device and the camera, the projection device is used to project the material coding pattern, and the camera is used to collect at least three image groups; the parameter information is determined based on the visualization matrix and the observation matrix.
  • the observation matrix and the visualization matrix can be obtained through the posture calibration information and the relative position relationships, and then the parameter information of the object can be obtained according to the observation matrix and the visualization matrix.
  • the above steps: determining parameter information based on the visualization matrix and the observation matrix include: constructing an energy function based on the visualization matrix and the observation matrix, the energy function being used to represent the difference between the estimated value and the observed value of each point on the surface of the object, the estimated value being related to the visualization matrix, and the observed value being related to the observation matrix; minimizing the value of the energy function to obtain parameter information.
  • the material mapping parameters and/or geometric structure parameters can be optimized in the process of minimizing the energy function.
  • the parameter information includes: material mapping parameters and/or geometric structure parameters, the geometric structure parameters include optimized depth or initialized depth; the energy function is shown in Formula 1:
  • the material mapping parameters include is the diffuse reflection variable; is the specular reflection variable; is the roughness; z * is the geometric structure parameter; is the estimated value of any point on the surface of the object at different viewing angles, and the calculation method of the estimated value is shown in Formula 2, where Ii is the observation value of any point at any viewing angle obtained by the camera (it can also be understood as the pixel difference between the reflection image corresponding to the all-black pattern and the reflection image corresponding to the all-white pattern at any viewing angle); i is the number of different viewing angles; E is the regularization term;
  • E i is the light source intensity of any point at any viewing angle
  • d is the distance between any point and the projector
  • f() is the reflection characteristic function
  • the reflection characteristic function is shown in Formula 3
  • n is the surface normal vector of any point
  • l i is the incident light direction of any point at any viewing angle
  • vi is the reflected light direction of any point at any viewing angle
  • the second aspect of the embodiment of the present application provides an image processing method, which can be applied to a structured light system, the structured light system comprising: a camera, a projection device, a rotating device and an object.
  • the method can be performed by an image processing device, or by a component of the image processing device (such as a processor, a chip, or a chip system, etc.).
  • the method comprises: triggering/controlling a projection device to project a material coding pattern onto an object; triggering/controlling a camera to collect a reflection image of the object at different viewing angles for the material coding pattern, the reflection image being used to generate a material map of the object; triggering/controlling a rotating device to rotate the object to achieve that the object is located at different viewing angles.
  • this embodiment compared with the existing material measurement scheme based on structured light, which requires additional light sources and RGB cameras to obtain multi-view RGB images, this embodiment changes the encoding strategy of structured light, and can obtain multi-view RGB images required for material modeling without additional light sources and cameras.
  • the method further includes: generating material mapping parameters of the object based on reflection images at different viewing angles.
  • the material coding pattern includes a full black pattern and a full white pattern.
  • the number of the reflected images at each of the different viewing angles is two.
  • the multi-view RGB images required for material modeling can be obtained without adding additional light sources and cameras.
  • the third aspect of the embodiment of the present application provides an image processing device that can be applied to a structured light system.
  • the image processing device includes: an acquisition unit for acquiring at least three image groups, the at least three image groups being reflection images of the object surface for the material coding pattern at at least three viewing angles; the acquisition unit is also used to acquire the initial depth of the object surface corresponding to any one of the at least three image groups; a generation unit is used to generate parameter information of the object based on the at least three image groups and the initial depth, the parameter information including material mapping parameters and/or geometric structure parameters. number.
  • the above-mentioned image processing device is applied to a structured light system
  • the structured light system includes a camera, a projection device, a rotating device, and an object connected to the rotating device, the rotating device is used to rotate multiple times so that the object is located at at least three viewing angles; an acquisition unit is specifically used to trigger the projection device to project a material coding pattern to the object at each of the at least three viewing angles; the acquisition unit is specifically used to trigger the camera to collect an image group reflected by the object for the material coding pattern at each viewing angle, so as to obtain at least three image groups.
  • the at least three image groups correspond to the at least three viewing angles one by one; the initial depth is obtained by the projection device projecting the structured light coding pattern to the object.
  • the material coding pattern includes a full black pattern and a full white pattern.
  • the number of the reflected images at each of the different viewing angles is two.
  • the above-mentioned generation unit is specifically used to obtain occlusion information of the spatial point cloud on the surface of the object in the target image group corresponding to the spatial point cloud in the two image groups, any one of the image groups is the target image group, and the two image groups are two image groups other than the target image group in at least three image groups;
  • the generation unit is specifically used to eliminate the pixel values corresponding to the occlusion information in the two image groups to obtain a visualization matrix;
  • the generation unit is specifically used to obtain the observation matrix of each point on the surface of the object under at least three viewing angles based on the pose calibration information and the initialization depth, the observation matrix including: pixel observation values under at least three viewing angles of incident light direction, reflected light direction, and light source intensity,
  • the pose calibration information including: intrinsic parameters of the projection device and the camera, extrinsic parameters between the projection device and the camera, the projection device is used to project material coding patterns, and the camera is used to collect at least three image groups;
  • the above-mentioned generation unit is specifically used to construct an energy function based on the visualization matrix and the observation matrix, the energy function is used to represent the difference between the estimated value and the observed value of each point on the surface of the object, the estimated value is related to the visualization matrix, and the observed value is related to the observation matrix; the generation unit is specifically used to minimize the value of the energy function to obtain parameter information.
  • the parameter information includes: material mapping parameters and/or geometric structure parameters, the geometric structure parameters include optimized depth or initialized depth; the energy function is shown in Formula 1:
  • the material mapping parameters include is the diffuse reflection variable; is the specular reflection variable; is the roughness; z * is the geometric structure parameter; is the estimated value of any point on the surface of the object at different viewing angles, and the calculation method of the estimated value is shown in Formula 2, where Ii is the observation value of any point at any viewing angle obtained by the camera (it can also be understood as the pixel difference between the reflection image corresponding to the all-black pattern and the reflection image corresponding to the all-white pattern at any viewing angle); i is the number of different viewing angles; E is the regularization term;
  • E i is the light source intensity of any point at any viewing angle
  • d is the distance between any point and the projector
  • f() is the reflection characteristic function
  • the reflection characteristic function is shown in Formula 3
  • n is the surface normal vector of any point
  • l i is the incident light direction of any point at any viewing angle
  • vi is the reflected light direction of any point at any viewing angle
  • the fourth aspect of the embodiment of the present application provides an image processing device that can be applied to a structured light system.
  • the image processing device includes: a control unit for triggering/controlling a projection device to project a material coding pattern onto an object; the control unit is also used to trigger/control a camera to collect reflection images of the object at different viewing angles for the material coding pattern, and the reflection images are used to generate material mapping parameters of the object; the control unit is also used to trigger/control a rotation device to rotate the object to achieve the object being located at different viewing angles.
  • the above-mentioned image processing device also includes a generation unit, which is used to generate material mapping parameters of the object based on the reflection images at different viewing angles.
  • the material coding pattern includes a full black pattern and a full white pattern.
  • the number of the reflected images at each of the different viewing angles is two.
  • a fifth aspect of an embodiment of the present application provides a structured light system, which includes: a camera, a projection device, a rotating device and an object; the camera is used to collect reflection images of the object at different viewing angles for a material coding pattern, and the material coding pattern is used to obtain material mapping parameters of the object; the projection device is used to project the material coding pattern onto the surface of the object; the rotating device is used to rotate the object to achieve that the object is located at different viewing angles.
  • this embodiment compared with the existing material measurement scheme based on structured light, which requires additional light sources and RGB cameras to obtain multi-view RGB images, this embodiment changes the encoding strategy of structured light, and can obtain multi-view RGB images required for material modeling without additional light sources and cameras.
  • a sixth aspect of an embodiment of the present application provides an image processing device, comprising: a processor, the processor being coupled to a memory, the memory being used to store programs or instructions, and when the programs or instructions are executed by the processor, the image processing device is enabled to implement the method in the above-mentioned first aspect or any possible implementation of the first aspect, or the image processing device is enabled to implement the above-mentioned second aspect or any possible implementation of the second aspect.
  • a seventh aspect of an embodiment of the present application provides a computer-readable storage medium storing one or more computer-executable instructions.
  • the processor executes the method described in the first aspect or any possible implementation of the first aspect, or executes the method described in the second aspect or any possible implementation of the second aspect.
  • An eighth aspect of an embodiment of the present application provides a computer program product (or computer program) storing one or more computers.
  • the processor executes the method of the first aspect or any possible implementation of the first aspect, or executes the method of the second aspect or any possible implementation of the second aspect.
  • a ninth aspect of an embodiment of the present application provides a chip system, which includes at least one processor for supporting an image processing device to implement the functions involved in the above-mentioned first aspect or any possible implementation of the first aspect, or to implement the functions involved in the above-mentioned second aspect or any possible implementation of the second aspect.
  • the chip system may also include a memory for storing program instructions and data necessary for the first communication device.
  • the chip system may be composed of a chip, or may include a chip and other discrete devices.
  • the chip system also includes an interface circuit, which provides program instructions and/or data for the at least one processor.
  • the present application has the following advantages: at least three image groups are obtained from the reflection of the material coding pattern on the object surface at at least three viewing angles, and parameter information is generated based on the at least three image groups and the initial depth of the object surface, and the parameter information includes material mapping parameters and/or geometric structure parameters. Thus, the acquisition of material mapping parameters is realized.
  • FIG1 is a schematic diagram of the structure of an application scenario provided by an embodiment of the present application.
  • FIG2 is a flow chart of a data processing method provided in an embodiment of the present application.
  • FIG3 is an example diagram of a structured light coding pattern and a material coding pattern provided in an embodiment of the present application
  • FIG4 is another schematic flow chart of a data processing method provided in an embodiment of the present application.
  • FIG5 is another schematic flow chart of a data processing method provided in an embodiment of the present application.
  • FIG. 6 is an example diagram of a vase and a vase geometric structure provided in an embodiment of the present application.
  • FIG. 7 is an example diagram of a texture map generated by the prior art and an object reconstruction result based on the texture map
  • FIG8 is an example diagram of a material map and an object reconstruction result based on the material map provided in an embodiment of the present application
  • FIG9 is a schematic diagram of the structure of the system hardware provided in an embodiment of the present application.
  • FIG10 is a schematic diagram of a structure of an image processing device provided in an embodiment of the present application.
  • FIG. 11 is another schematic diagram of the structure of the image processing device provided in an embodiment of the present application.
  • the present application provides an image processing method, related equipment and structured light system for acquiring material maps.
  • stage one the structured light system only supports geometric output, without texture and material stickers.
  • stage two supports texture map output on the basis of stage one geometry by adding additional red, green and blue (RGB) cameras and light sources, but the texture map cannot correctly separate the diffuse and highlight components, there is obvious highlight noise, and it does not support physical base rendering (PBR).
  • RGB red, green and blue
  • PBR physical base rendering
  • the existing structured light technology can only obtain the depth information of the target scene. It is impossible to obtain the material map.
  • the existing hardware systems that support the spatial bidirectional reflectance distribution function (svBRDF) material measurement need to add additional light sources and cameras.
  • this method often consists of multiple cameras, multiple projectors, and dozens of high-power white light emitting diode (LED) surface light sources in different directions. It increases the complexity of system integration, and the equipment is bulky and inconvenient to use.
  • the embodiment of the present application provides an image processing method, which generates parameter information based on the initial depth of the at least three image groups and the object surface, without adding additional light sources and cameras, by obtaining at least three image groups reflected by the material coding pattern from the object surface at at least three viewing angles, and the parameter information includes material mapping parameters and/or geometric structure parameters, thereby achieving the acquisition of material mapping parameters.
  • Material maps that support svBRDF include: diffuse map, specular map, roughness map, and normal map.
  • the scenario to which the method provided in the embodiment of the present application is applicable may be the structured light system shown in FIG1 .
  • the structured light system includes: a camera 101, a projection device 102, a rotating device 103, and an object 104.
  • the camera 101 is used to collect reflection images of the object 104 at different viewing angles with respect to the coding pattern.
  • the coding pattern includes a material coding pattern, and the material coding pattern is used to obtain material mapping parameters.
  • the projection device 102 is used to project a coded pattern onto the surface of the object 104 .
  • the rotating device 103 is used to rotate the object 104 so that the object 104 is located at different viewing angles.
  • the object 104 can be understood as an object to be three-dimensionally scanned.
  • the rotating device 103 is used to place the object 104. It is understandable that the rotating device 103 may also include a plurality of brackets to support or fix the object 104, so as to realize the rotation of the rotating device 103 and drive the object 104 to rotate at the same time.
  • the coding pattern may further include a structured light coding pattern, which is used to obtain the machine geometry (such as depth) of the object.
  • the projection device 102 projects a specific coded pattern onto the object 104, and the camera 101 takes a picture to obtain the corresponding reflected image.
  • the rotation device 103 rotates a specific angle and repeats the above image acquisition process multiple times, which is usually greater than 2 times and can be set according to actual needs.
  • the image processing device generates parameter information of the object 104 based on the reflected images obtained in the above multiple acquisition processes to achieve three-dimensional reconstruction of the object 104.
  • the image processing device in the embodiments of the present application can be a server, a mobile phone, a tablet computer (pad), a portable game console, a personal digital assistant (PDA), a laptop computer, an ultra mobile personal computer (UMPC), a handheld computer, a netbook, a car media player, a wearable electronic device, a virtual reality (VR) terminal device, an augmented reality (AR) terminal device, and other devices with sufficient computing power.
  • PDA personal digital assistant
  • UMPC ultra mobile personal computer
  • a handheld computer a netbook
  • a car media player a wearable electronic device
  • VR virtual reality
  • AR augmented reality
  • the method can be performed by an image processing device. It can also be performed by a component of the image processing device (such as a processor, a chip, or a chip system, etc.).
  • the method can be applied to the structured light system shown in FIG. 1. Please refer to FIG. 2, which is a flow chart of the image processing method provided by the embodiment of the present application.
  • the method may include steps 201 to step 202. Step 207. Steps 201 to 207 are described in detail below.
  • Step 201 triggering the projector.
  • the image processing device sends a first trigger signal to the projector (ie, the projection device in FIG1 ).
  • the projector receives the first trigger signal sent by the image processing device.
  • the first trigger signal is used by the projector to project a structured light coding pattern onto an object.
  • Step 202 Project structured light coding.
  • the projector After the projector receives the first trigger signal, the projector projects a structured light coding pattern (or structured light coding) onto the object.
  • the structured light coding pattern is used to obtain the depth of the object.
  • the structured light coding pattern is shown in (a) of FIG3. It is understandable that (a) of FIG3 is only an example of 8 coding patterns (corresponding to 8 rows respectively). In practical applications, there may be fewer (e.g., 4, 5, etc.) or more (e.g., 16, 20, etc.) coding patterns. In addition, the structured light coding pattern may include multiple black and white patterns or multiple patterns corresponding to 0-255.
  • Step 203 triggering camera acquisition.
  • the image processing device After the projector projects the structured light coding pattern, the image processing device sends second trigger information to the camera to trigger the camera to acquire the image reflected by the object surface for the structured light coding pattern.
  • the second trigger information is used for the camera to collect the reflected image of the object surface.
  • this image can be used as input to structured light decoding to obtain an initial depth of the object.
  • Step 204 projecting the material code.
  • the image processing device After completing the structured light coding projection and acquisition, the image processing device sends a third trigger message to the projector to trigger the projector to project a material coding pattern (or material coding).
  • the third trigger message is used for the projector to project a material coding pattern onto the object.
  • the material coding pattern includes a full black pattern and a full white pattern.
  • the material coding pattern is shown in (b) of FIG3 .
  • Step 205 triggering camera acquisition.
  • the image processing device After the projector projects the material coding pattern, the image processing device sends fourth trigger information to the camera to trigger the camera to acquire the RGB image reflected by the surface of the object.
  • this RGB image can be used as input for photometric constraint modeling.
  • Step 206 turntable triggering.
  • the image processing device triggers the turntable (i.e., the rotating device in FIG1 ) to rotate to a specific angle, and obtains RGB images of different positions and postures through the relative movement of the object and each device in the structured light acquisition.
  • the turntable i.e., the rotating device in FIG1
  • the spatial point cloud stitching and fusion of the objects corresponding to the RGB images in different postures are completed to obtain the scanning results.
  • Step 207 end the determination.
  • light structure coding patterns can be projected at the main viewing angle
  • material coding patterns can be projected at the main viewing angle and other viewing angles, etc., which are not limited here.
  • a multi-view RGB image required for material modeling can be obtained. And the multi-view RGB image is used as the input for material modeling.
  • by changing the coding strategy of structured light by adding two material coding patterns of all white and all black, that is, by projecting the pattern through a projection device, the reflected image of the object for the projection pattern is obtained, and the multi-view RGB image required for material modeling can be obtained without adding an additional light source and camera.
  • FIG. 4 is a flowchart of an image processing method provided in an embodiment of the present application.
  • the method can be performed by an image processing device. It can also be performed by a component of the image processing device (such as a processor, a chip, or a chip system, etc.).
  • the method can be applied to the structured light system shown in FIG. 1, and the method can include steps 401 to 403. Steps 401 to 403 are described in detail below.
  • Step 401 Acquire at least three image groups.
  • the at least three image groups in the embodiment of the present application are reflection images of the object surface with respect to the material coding pattern at at least three viewing angles, wherein the at least three image groups correspond one-to-one to the at least three viewing angles.
  • the material coding pattern includes an all-black pattern and an all-white pattern.
  • a first reflection image of the object for the all-black pattern and a second reflection image of the all-white pattern are obtained.
  • an image group can be obtained, and the image group includes the first reflection image and the second reflection image. That is, three image groups include six reflection images.
  • the image processing device can obtain at least three image groups in the embodiment of the present application. It can be by receiving images sent by other devices, by selecting from a database, or by obtaining at least three image groups through the method of the embodiment shown in Figure 2 above, etc. The specifics are not limited here.
  • Step 402 Obtain an initial depth of the object surface corresponding to any one of at least three image groups.
  • the image processing device can obtain the initial depth in the embodiment of the present application. It can be by receiving it from other devices, by selecting it from a database, or by obtaining at least three image groups through the method of the embodiment shown in Figure 2 above, etc. The specifics are not limited here.
  • the projection device projects a structured light coding pattern onto the surface of the object
  • the camera collects a reflection image of the object surface for the structured light coding pattern, and obtains the initial depth of the object at the viewing angle through the image.
  • the at least three image groups in the aforementioned step 401 may also include a reflection image of the object surface for the structured light coding pattern.
  • each of the aforementioned at least three image groups includes a first reflection image and a second reflection image at a certain viewing angle.
  • the first reflection image is a reflection image of the object surface for the structured light coding pattern
  • the second reflection image is a reflection image of the object surface for the material coding pattern.
  • the initial depth can be obtained by processing at least three image groups, or it can be obtained by processing images other than at least three image groups (i.e., reflection images of the object surface for the structured light coding pattern).
  • the aforementioned at least three image groups may be four image groups, and the four image groups include: reflection images of the object surface for the material coding pattern at three viewing angles and reflection images of the object surface for the structured light coding pattern at the main viewing angle.
  • the number of reflection images corresponding to the structured light coding pattern for example, the description of (a) in the aforementioned FIG. 3).
  • Step 403 Generate parameter information of the object based on at least three image groups and the initial depth.
  • the image processing device After acquiring at least three image groups and an initial depth, the image processing device generates parameter information of the object based on the at least three image groups and the initial depth, where the parameter information includes material mapping parameters and/or geometric structure parameters.
  • step 403 may refer to Fig. 5.
  • Fig. 5 includes steps 501 to 506. Steps 501 to 506 are described in detail below.
  • Step 501 key frame.
  • the image processing device uses the image group corresponding to any one of the at least three perspectives as a key frame.
  • the perspective corresponding to the key frame is used as the main perspective.
  • the point cloud contained in the key frame is the optimization target (or understood as determining the range of the material map), and the pixel coordinates corresponding to the point cloud in the key frame are the image foreground.
  • the viewing angle corresponding to the initial depth in the above step 402 is used as the main viewing angle
  • the image group corresponding to the initial depth is the target image group.
  • Step 502 adjacent frames are determined to complete image registration (or image calibration).
  • At least two frames of images adjacent to the key frame are selected as adjacent frames.
  • the point cloud coordinates in the key frame are reprojected to the pixel positions and RGB values in each adjacent frame.
  • the image processing device can obtain the relative position relationship between at least three image groups and the object. And based on the relative position relationship, the occlusion information of the spatial point cloud of the object surface in the target image group corresponding to the spatial point cloud in the two image groups is obtained.
  • the two image groups are two image groups other than the target image group in the at least three image groups.
  • the occlusion information can be obtained by using methods such as reprojection, which are not specifically limited here.
  • Step 503 generate observation and visualization matrices.
  • the image processing device After the image processing device obtains the occlusion information, the pixel values corresponding to the occlusion information in the two image groups are removed to obtain a visualization matrix. This process can be understood as reducing the noise caused by occlusion.
  • the image processing device can also obtain the view of each point on the surface of the object from at least three perspectives based on the pose calibration information and the initialization depth.
  • the measurement matrix includes: pixel observation values under at least three viewing angles: incident light direction, reflected light direction, and light source intensity.
  • the pose calibration information includes: internal parameters of the projection device and the camera, and external parameters between the projection device and the camera.
  • the projection device is used to project the material coding pattern
  • the camera is used to collect at least three image groups.
  • the observation matrix can be Among them, for a point P on the surface of the object, L is the incident light direction of the point at the i viewing angle (it can also be understood as the irradiation direction of the projection device, that is, the irradiation direction of the light source), V is the reflected light direction of the point at the i viewing angle (it can also be understood as the observation direction of the camera), I is the pixel observation value at the point obtained by the camera at the i viewing angle (it can also be understood as the pixel difference between the reflected image corresponding to the all-black pattern and the reflected image corresponding to the all-white pattern at the i viewing angle). E is the light source intensity at the point at the i viewing angle.
  • the parameter information can be determined based on the visualization matrix and the observation matrix (as shown in steps 504 to 506 below).
  • the parameter information includes: material mapping parameters and/or geometric structure parameters, and the geometric structure parameters include optimized depth or initialized depth. Wherein, in the case where the geometric structure parameters include the initialized depth, this embodiment can be understood as obtaining material mapping parameters. In the case where the geometric structure parameters include the optimized depth, this embodiment can be understood as optimizing the geometric structure parameters of the object.
  • Step 504 Establish an energy function.
  • the image processing device constructs an energy function based on the visualization matrix and the observation matrix.
  • the energy function is used to represent the difference between the estimated value and the observed value of each point on the surface of the object.
  • the estimated value is related to the visualization matrix, and the observed value is related to the observation matrix.
  • the energy function is as shown in Formula 1:
  • the material mapping parameters include is the optimized diffuse reflection variable; is the optimized specular reflection variable; is the optimized roughness; z * is the optimized geometric structure parameter; is the estimated value of any point on the surface of the object at different viewing angles.
  • the calculation method of the estimated value is shown in Formula 2, where I i is the observation value of any point obtained by the camera at any viewing angle; i is the number of different viewing angles; and E is the regularization term.
  • the regularization term includes regularization terms corresponding to normal vectors, depths, materials, etc., which are not specifically limited here.
  • E i is the light source intensity at any point at any viewing angle
  • d is the distance between any point and the projector
  • f() is the reflection characteristic function
  • the reflection characteristic function is shown in Formula 3
  • n is the surface normal vector of any point
  • l i is the incident light direction at any point at any viewing angle
  • vi is the reflected light direction at any point at any viewing angle
  • the initial diffuse reflectance variable is the initial specular reflection variable; rs is the initial roughness;
  • D() represents the microfacet distribution function, which is used to express the change of microfacet slope;
  • G() represents the geometric attenuation coefficient.
  • the incident light on the microfacet may be blocked by the adjacent microfacets before reaching a surface or after being reflected by the surface. This blocking will cause a slight dimming of the specular reflection, and the geometric attenuation coefficient can be used to measure this effect.
  • Step 505 Minimize the energy function.
  • the parameter information also includes: material mapping parameters and/or geometric structure parameters, and the geometric structure parameters include optimized depth or initialized depth. Among them, when the geometric structure parameters include the initialized depth, this embodiment can be understood as obtaining material mapping parameters. When the geometric structure parameters include the optimized depth, this embodiment can be understood as optimizing the geometric structure parameters of the object.
  • the parameter information may include the above
  • z * is the initial depth Z, i.e.
  • the initial depth is a fixed value
  • the material mapping parameters are the parameters to be optimized.
  • the parameter information may include the above-mentioned z * . for r s is the preset value, that is, the material mapping parameters are fixed values, and the geometric structure parameters are the parameters to be optimized.
  • the parameter information may include the above z * , In this case, it can be understood that both the material mapping parameters and the geometric structure parameters are parameters to be optimized.
  • the initial depth/optimized depth can be used to generate a normal map
  • the initial or optimized diffuse variables can be used to generate a diffuse map
  • the initial or optimized specular variables can be used to generate a specular map
  • the initial or optimized roughness can be used to generate a roughness map.
  • Step 506 Convergence determination.
  • the convergence condition includes at least one of the following: the number of repetitions is a first preset threshold, the value of the energy function is less than a second preset threshold, etc.
  • a new viewing angle can be selected as the main viewing angle to repeat the above process, and finally the parameter information of the object under multiple viewing angles is obtained. Then, the object material map is generated according to the fusion splicing and other processing.
  • At least three image groups are obtained by reflecting the material coding pattern on the surface of an object at least three viewing angles, and parameter information is generated based on the initial depth of the at least three image groups and the surface of the object, and the parameter information includes material mapping parameters and/or geometric structure parameters.
  • the projection device is changed into a light source, which can support PBR material mapping output.
  • the material map includes diffuse reflection map, specular reflection map, roughness map and normal map, which supports PBR rendering.
  • the above-mentioned material modeling and solution algorithm can be used for mobile phone material measurement, providing a material modeling solution for the existing mobile phone-based three-dimensional reconstruction algorithm, and supporting the material output function of the mobile phone.
  • the reconstruction of a vase is taken as an example below to exemplarily describe the reconstruction results of the vase using the prior art and the image processing method provided in the present application.
  • the object is a physical picture of a vase as shown in Figure 6 (a), and Figure 6 (b) is a geometric structure diagram of the object.
  • Figure 7 (a) is a texture map generated by the prior art
  • Figure 7 (b) is the reconstruction result of the texture map in the prior art.
  • Figure 8 (a) is a material map obtained by the method of the embodiment of the present application
  • Figure 8 (b) is the reconstruction result of the material map obtained by the method of the embodiment of the present application.
  • the embodiment of the present application also provides a system hardware.
  • the system hardware is shown in Figure 9, and the system hardware includes: a turntable unit, a control unit, a lighting unit, a sensor unit, a storage unit and a computing unit.
  • the control unit first sends a trigger signal to enable the projector to project a specific coded pattern, and the projector triggers the camera to take pictures to obtain the corresponding image, and upload it to the storage unit.
  • the control unit controls the turntable to rotate to a specific angle, and repeats the above image acquisition process to a preset number of times; after the complete scan is completed, the computing unit completes the calculation of the object parameter information (i.e., including the geometric structure and svBRDF).
  • the turntable unit may include a turntable and a power supply.
  • the control unit may include a central processing unit (CPU) and a cache.
  • the lighting unit includes a power supply and a projector.
  • the sensor unit includes a camera and a transmission line.
  • the storage unit includes a cache and an external storage.
  • the computing unit includes a CPU, a graphics processing unit (GPU), a cache and a transmission line.
  • An embodiment of the image processing device in the embodiment of the present application includes:
  • An acquisition unit 1001 is used to acquire at least three image groups, where the at least three image groups are reflection images of the object surface at at least three viewing angles with respect to the material coding pattern;
  • the acquisition unit 1001 is further used to acquire an initial depth of the object surface corresponding to any one of the at least three image groups;
  • a generating unit 1002 is used to generate parameter information of an object based on at least three image groups and an initial depth, wherein the parameter information includes material map parameters. number and/or geometric parameters.
  • the image processing device is applied to a structured light system, the structured light system comprising a camera, a projection device, a rotating device, and an object connected to the rotating device, the rotating device being used to rotate multiple times so that the object is located at at least three viewing angles;
  • the acquisition unit 1001 is specifically used to trigger the projection device to project a material coding pattern onto the object at each of at least three viewing angles; the acquisition unit 1001 is specifically used to trigger the camera to capture an image group reflected by the object for the material coding pattern at each viewing angle to obtain at least three image groups.
  • the material coding pattern includes an all-black pattern and an all-white pattern.
  • the generation unit 1002 is specifically used to obtain the occlusion information of the spatial point cloud on the surface of the object in the target image group corresponding to the spatial point cloud in the two image groups, any one of the image groups is the target image group, and the two image groups are two image groups other than the target image group in at least three image groups; the generation unit 1002 is specifically used to eliminate the pixel values corresponding to the occlusion information in the two image groups to obtain a visualization matrix; the generation unit 1002 is specifically used to obtain the observation matrix of each point on the surface of the object under at least three viewing angles based on the pose calibration information and the initialization depth, the observation matrix including: the pixel observation values under at least three viewing angles of the incident light direction, the reflected light direction, and the light source intensity, the pose calibration information including: the intrinsic parameters of the projection device and the camera, the extrinsic parameters between the projection device and the camera, the projection device is used to project the material coding pattern, and the camera is used to collect at least three image groups; the generation unit 1002
  • the generating unit 1002 is specifically used to construct an energy function based on the visualization matrix and the observation matrix, the energy function is used to represent the difference between the estimated value and the observed value of each point on the surface of the object, the estimated value is related to the visualization matrix, and the observed value is related to the observation matrix;
  • the generating unit 1002 is specifically configured to minimize the value of the energy function to obtain parameter information.
  • the projection device is turned into a light source, which can support PBR material map output.
  • the material map includes diffuse reflection map, specular reflection map, roughness map and normal map, which supports PBR rendering.
  • the above-mentioned material modeling and solution algorithm can be used for mobile phone material measurement, providing a material modeling solution for the existing mobile phone-based 3D reconstruction algorithm, and supporting the material output function of the mobile phone.
  • the image processing device may include a processor 1101, a memory 1102, and a communication port 1103.
  • the processor 1101, the memory 1102, and the communication port 1103 are interconnected via a line.
  • the memory 1102 stores program instructions and data.
  • the memory 1102 stores program instructions and data corresponding to the steps executed by the image processing device in the corresponding implementation modes shown in the aforementioned FIGS. 1 to 5 .
  • the processor 1101 is used to execute the steps performed by the image processing device shown in any of the embodiments shown in Figures 1 to 5 above.
  • the communication port 1103 can be used to receive and send data, and to execute the steps related to acquisition, sending, and receiving in any of the embodiments shown in FIG. 1 to FIG. 5 .
  • the image processing device may include more or fewer components than those in FIG. 11 , and this application is merely an illustrative description and is not intended to be limiting.
  • An embodiment of the present application further provides a computer-readable storage medium storing one or more computer-executable instructions.
  • the processor executes the method described in the possible implementation manner of the image processing device in the aforementioned embodiment.
  • An embodiment of the present application also provides a computer program product (or computer program) storing one or more computers.
  • the processor executes the method of the possible implementation mode of the above-mentioned image processing device.
  • the embodiment of the present application also provides a chip system, which includes at least one processor for supporting a terminal device to implement the functions involved in the possible implementation of the above-mentioned image processing device.
  • the chip system also includes an interface circuit, which provides program instructions and/or data for the at least one processor.
  • the chip system may also include a memory, which is used to store the necessary program instructions and data for the image processing device.
  • the chip system may be composed of chips, or may include chips and other discrete devices.
  • the disclosed systems, devices and methods can be implemented in other ways.
  • the device embodiments described above are only schematic.
  • the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed.
  • Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.
  • the units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
  • each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
  • the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.
  • the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions to enable a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application.
  • the aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Computer Graphics (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • Length Measuring Devices By Optical Means (AREA)

Abstract

La présente demande divulgue un procédé de traitement d'image, un dispositif associé et un système de lumière structurée, qui peuvent être appliqués à un système de lumière structurée. Le procédé consiste à : acquérir au moins trois groupes d'images, les au moins trois groupes d'images étant des images de réflexion d'une surface d'un objet pour un motif de codage de matériau à au moins trois angles de vue et les au moins trois groupes d'images étant en correspondance biunivoque avec les au moins trois angles de vue ; acquérir une profondeur initiale de la surface de l'objet correspondant à l'un quelconque des au moins trois groupes d'images ; et générer des informations de paramètres de l'objet sur la base des au moins trois groupes d'images et de la profondeur initiale, les informations de paramètres comprenant un paramètre de carte de matériau et/ou un paramètre de structure géométrique. Les informations de paramètres sont générées sur la base des au moins trois groupes d'images et de la profondeur initiale de la surface de l'objet au moyen des trois groupes d'images, de la surface de l'objet, qui sont obtenus par réflexion du motif de codage de matériau aux au moins trois angles de vue, et les informations de paramètres comprennent le paramètre de carte de matériau et/ou le paramètre de structure géométrique. Par conséquent, le paramètre de carte de matériau est acquis.
PCT/CN2023/103013 2022-10-31 2023-06-28 Procédé de traitement d'image, dispositif associé et système de lumière structurée WO2024093282A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN202211349694.0A CN117994348A (zh) 2022-10-31 2022-10-31 一种图像处理方法、相关设备及结构光系统
CN202211349694.0 2022-10-31

Publications (1)

Publication Number Publication Date
WO2024093282A1 true WO2024093282A1 (fr) 2024-05-10

Family

ID=90893118

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2023/103013 WO2024093282A1 (fr) 2022-10-31 2023-06-28 Procédé de traitement d'image, dispositif associé et système de lumière structurée

Country Status (2)

Country Link
CN (1) CN117994348A (fr)
WO (1) WO2024093282A1 (fr)

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100315490A1 (en) * 2009-06-15 2010-12-16 Electronics And Telecommunications Research Institute Apparatus and method for generating depth information
US20120237112A1 (en) * 2011-03-15 2012-09-20 Ashok Veeraraghavan Structured Light for 3D Shape Reconstruction Subject to Global Illumination
CN108985333A (zh) * 2018-06-15 2018-12-11 浙江大学 一种基于神经网络的材质获取方法和系统
CN110570503A (zh) * 2019-09-03 2019-12-13 浙江大学 基于神经网络的三维物体法向量、几何及材质获取方法
US20200074698A1 (en) * 2018-08-30 2020-03-05 Cognex Corporation Methods and apparatus for generating a three-dimensional reconstruction of an object with reduced distortion
WO2022021680A1 (fr) * 2020-07-28 2022-02-03 中国科学院深圳先进技术研究院 Procédé de reconstruction d'un objet tridimensionnel par fusion de lumière structurée et photométrie, et équipement terminal

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20100315490A1 (en) * 2009-06-15 2010-12-16 Electronics And Telecommunications Research Institute Apparatus and method for generating depth information
US20120237112A1 (en) * 2011-03-15 2012-09-20 Ashok Veeraraghavan Structured Light for 3D Shape Reconstruction Subject to Global Illumination
CN108985333A (zh) * 2018-06-15 2018-12-11 浙江大学 一种基于神经网络的材质获取方法和系统
US20200074698A1 (en) * 2018-08-30 2020-03-05 Cognex Corporation Methods and apparatus for generating a three-dimensional reconstruction of an object with reduced distortion
CN110570503A (zh) * 2019-09-03 2019-12-13 浙江大学 基于神经网络的三维物体法向量、几何及材质获取方法
WO2022021680A1 (fr) * 2020-07-28 2022-02-03 中国科学院深圳先进技术研究院 Procédé de reconstruction d'un objet tridimensionnel par fusion de lumière structurée et photométrie, et équipement terminal

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
张启灿 QICAN ZHANG: "Three-dimensional imaging technique based on Gray-coded structured illumination", INFRARED AND LASER ENGINEERING, CN, vol. 49, no. 3, 1 January 2020 (2020-01-01), CN , pages 303004 - 303004, XP093168868, ISSN: 1007-2276, DOI: 10.3788/IRLA202049.0303004 *

Also Published As

Publication number Publication date
CN117994348A (zh) 2024-05-07

Similar Documents

Publication Publication Date Title
CA3074547C (fr) Capture et transfert de representation faciale en direct vers un modele tridimensionnel genere par ordinateur
US11115633B2 (en) Method and system for projector calibration
US8218003B2 (en) Optimization strategies for GPU view projection matrix implementation
CN108876926B (zh) 一种全景场景中的导航方法及系统、ar/vr客户端设备
US7901095B2 (en) Resolution scalable view projection
CN112116692B (zh) 模型渲染方法、装置、设备
JP2011147125A (ja) プロジェクターシステムを較正する方法、プログラム、コンピューターシステムおよびプロジェクターシステム
US7528831B2 (en) Generation of texture maps for use in 3D computer graphics
US9509905B2 (en) Extraction and representation of three-dimensional (3D) and bidirectional reflectance distribution function (BRDF) parameters from lighted image sequences
JP2003202216A (ja) 三次元画像処理方法、三次元画像処理装置、三次元画像処理システムおよび三次元画像処理プログラム
Nomoto et al. Dynamic multi-projection mapping based on parallel intensity control
CN116664752B (zh) 基于图案化光照实现全景显示的方法、系统及存储介质
Law et al. Projector placement planning for high quality visualizations on real-world colored objects
WO2024093282A1 (fr) Procédé de traitement d'image, dispositif associé et système de lumière structurée
CN116485969A (zh) 体素对象生成方法、装置和计算机可读存储介质
JP2001118074A (ja) ３次元画像作成方法、３次元画像作成装置及びプログラム記録媒体
US20230206567A1 (en) Geometry-aware augmented reality effects with real-time depth map
GB2584192A (en) On-set facial performance capture and transfer to a three-dimensional computer-generated model
Scopigno et al. Tutorial T1: 3D data acquisition
Bimber Multi-projector techniques for real-time visualizations in everyday environments
AU2020449562B2 (en) Geometry-aware augmented reality effects with a real-time depth map
US20230090732A1 (en) System and method for real-time ray tracing in a 3d environment
WO2023197689A1 (fr) Procédé, système et dispositif de traitement de données
Bimber Projector-based augmentation
WO2024072698A1 (fr) Systèmes et procédés de représentation numérique d'une scène avec des primitives à facettes multiples

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23884221

Country of ref document: EP

Kind code of ref document: A1