WO2020192706A1 - Système et procédé de reconstruction de modèle tridimensionnel d'objet - Google Patents

Système et procédé de reconstruction de modèle tridimensionnel d'objet Download PDF

Info

Publication number
WO2020192706A1
WO2020192706A1 PCT/CN2020/081222 CN2020081222W WO2020192706A1 WO 2020192706 A1 WO2020192706 A1 WO 2020192706A1 CN 2020081222 W CN2020081222 W CN 2020081222W WO 2020192706 A1 WO2020192706 A1 WO 2020192706A1
Authority
WO
WIPO (PCT)
Prior art keywords
depth
depth image
image
color image
pixel
Prior art date
Application number
PCT/CN2020/081222
Other languages
English (en)
Chinese (zh)
Inventor
李江
张朋
柳跃天
徐紫雅
苏文丹
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2020192706A1 publication Critical patent/WO2020192706A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T17/00—Three dimensional [3D] modelling, e.g. data description of 3D objects

Definitions

  • This application relates to the field of computer vision, and in particular to a method and device for reconstructing a three-dimensional model of an object.
  • Computer vision is an inseparable part of various intelligent systems in various application fields, such as manufacturing, inspection, document analysis, medical diagnosis, and military. It is about how to use cameras/video cameras and computers to obtain what we need Yes, the knowledge of the subject’s data and information. Vividly speaking, it is to install eyes (camera/camcorder) and brain (algorithm) on the computer to replace the human eye to identify, track and measure the target, so that the computer can perceive the environment. Because perception can be seen as extracting information from sensory signals, computer vision can also be seen as a science that studies how to make artificial systems "perceive" from images or multi-dimensional data.
  • computer vision is to use various imaging systems instead of visual organs to obtain input images, and then the computer replaces the brain to complete the processing and interpretation of these input images.
  • the ultimate research goal of computer vision is to enable computers to observe and understand the world through vision like humans, and have the ability to adapt to the environment autonomously.
  • the object 3D model reconstruction technology is to truly reconstruct the 3D virtual model of the surface of the object in the computer, and build a complete 3D model of the object.
  • Object 3D model reconstruction technology has many applications in the fields of computer graphics and computer vision, such as movie special effects, 3D stereo image games, virtual reality, and human-computer interaction.
  • three-dimensional object model reconstruction technology is increasingly used in terminal devices.
  • the terminal device uses the front structured light to scan the object to extract the three-dimensional information of the object, build a three-dimensional model of the object based on the three-dimensional information, and finally move the object.
  • a commonly used method of rebuilding a 3D model of an object is to place the object on the desktop, and the user holds the mobile phone horizontally and at 45 degrees to the desktop.
  • the mobile phone recognizes the object to be scanned, the mobile phone starts to scan the object , And build a three-dimensional model of the object based on the scan results.
  • the mobile phone will incorrectly recognize the hand as a part of the object, which will cause the failure of the three-dimensional model reconstruction of the object.
  • the user needs to adjust the shooting angle or the position of the object multiple times to scan the panorama of the object, which is complicated and takes a long time. Therefore, it is necessary to study a method for reconstructing a three-dimensional object model with simple operation and shorter time.
  • the embodiments of the present application provide a method and device for reconstructing a three-dimensional model of an object.
  • a user can directly hold an object for scanning, and then establish a three-dimensional model of the object, with simple operation, high scanning efficiency, and improved user experience.
  • an embodiment of the present application provides an object reconstruction method, which includes: determining a first color image and a first depth image of a target object, wherein the target object includes a target object and an interference object;
  • the pixel points in a color image correspond to the pixels in the first depth image one-to-one; determine the color interference pixel points corresponding to the interference object in the first color image; adjust the depth in the first depth image
  • the pixel value of the interference pixel is obtained to obtain a processed first depth image, where the depth interference pixel is a pixel in the first depth image that corresponds to the color interference pixel one-to-one; according to the first color
  • the image and the processed first depth image construct a three-dimensional model of the target object.
  • the first color image may be a color image obtained by down-sampling or up-sampling the original color image obtained by scanning the target object.
  • the first depth image obtained by shooting the target object and the interfering object is processed, and the processed first depth image only contains the depth information of the target object.
  • the processed first depth image and the depth image Sequence corresponding color images to construct a three-dimensional model of the target object; the user can hold the target object for scanning or place the target object on a selective rotating device for scanning, which has high scanning efficiency and simple operation.
  • the method before the determining the first color image and the first depth image of the target object, the method further includes: acquiring a color image sequence and a depth image sequence; the color image sequence includes the Multi-frame color images of the target object in multiple poses, and the depth image sequence includes multiple-frame depth images of the target object in the multiple poses; wherein, the first depth image is the multiple Any one of the frame depth images, the first color image is an image corresponding to the first depth image among the multi-frame color images; correspondingly, the first color image is based on the first color image and the Constructing a three-dimensional model of the target object with the processed first depth image includes: constructing a three-dimensional model of the target object according to multiple frames of the first color image and multiple frames of the processed first depth image.
  • the depth image sequence obtained by shooting the target object and the interfering object is processed to obtain a depth image sequence containing only the depth information of the target object, according to the processed depth image sequence and the color corresponding to the depth image sequence
  • the image sequence is fused to obtain a three-dimensional model of the target object; the user can hold the target object for scanning or place the target object on a selective rotating device for scanning, and the three-dimensional model of the target object can be constructed accurately.
  • the method before the acquiring the color image sequence and the depth image sequence, further includes: scanning the target object to obtain a front color image sequence and a front depth image sequence;
  • the front depth image in the depth image sequence corresponds to the front color image in the front color image sequence, and the pixels in the front depth image correspond to the front depth image corresponding to the front depth image.
  • the pixels in the color image correspond one-to-one; in determining that the ratio of the area of the image of the target object in each frame of the front color image to the area of the front color image is in the target interval, and the If the image of the target object in the front color image is in the target area of the front color image, and the displacement of the target object is determined to be less than the first threshold according to the front depth image sequence, scan the target Object to obtain the color image sequence and the depth image sequence; the distance between the center point of the target area and the center point of the front color image is less than a distance threshold.
  • the target area is a rectangular area or a regular polygonal area.
  • the color image obtained by scanning the target object can be down-sampled to obtain a color image with the same resolution as the depth image.
  • the scanning the target object to obtain the front color image sequence and the front depth image sequence may be: scanning the target object to obtain the original front color image sequence and the front depth image sequence; Each frame of the original front color image in the image sequence is down-sampled to obtain the front color image sequence.
  • the scanning the target object to obtain the color image sequence and the depth image sequence may be: scanning the target object to obtain the original color image sequence and the depth image sequence; and for each frame in the original color image sequence The original color image is down-sampled to obtain the color image sequence.
  • the ratio of the area of the target object in each frame of the front color image to the area of each frame of the front color image is in the target interval, and the image of the target object in each frame of the front color image is in the front of each frame
  • the target area of the color image, and the displacement of the target object determined according to the front depth image sequence is less than the first threshold, it indicates that the size and position of the target object in each frame of the front color image meet the requirements and the target object The position is basically unchanged.
  • the ratio of the area of the image of the target object in each frame of the color image in the color image sequence obtained by scanning the target object to the area of each frame of the color image is in the target interval, And the image of the target object in each frame of color image is in the target area of the front color image, so as to construct a three-dimensional model of the target object.
  • the user after determining that the front depth image sequence and the front color image sequence meet certain requirements, the user is instructed to adjust the posture of the target object, and then scan the target object to obtain the depth image sequence and color image used to construct the target object Sequence, the user can be notified in time to adjust the posture of the target object.
  • the adjusting the pixel value of the depth interference pixel in the first depth image includes: setting the pixel value of the depth interference pixel in the first depth image to zero.
  • the pixel value of the interfering pixel in each frame of the depth image in the depth image sequence is set to zero, which can effectively filter the depth information of the interfering object in the depth image.
  • the method before the constructing a three-dimensional model of the target object according to the color image sequence and the processed depth image sequence, the method further includes: determining the front color image The pixel points corresponding to the interfering object in the last frame of the front color image in the sequence; the front color images of each frame in the front color image sequence are sorted according to the sequence obtained by scanning; The pixel value of the first pixel in the last frame of the front depth image in the depth image sequence is set to zero to obtain a reference front depth image; the last frame of front depth image and the last frame of front color Corresponding to the image, the first pixel includes the pixel corresponding to the pixel corresponding to the interfering object in the last frame of the pre-color image in the last frame of the pre-depth image, and the last Pixel points in the pre-frame depth image with pixel values greater than a second threshold and pixels in the last frame pre-depth image with pixel values less than a third threshold, the second threshold being greater than the third threshold; The pre-depth images of each frame in
  • the interference pixel in the first depth image is a pixel corresponding to the interference object in the color image corresponding to the first depth image. Since it is difficult to accurately determine each pixel corresponding to the interfering object in the color image corresponding to the first depth image, the interfering pixel in the first depth image may only be the value of the pixel corresponding to the interfering object in the first depth image. Part.
  • the coordinates of the pixel points corresponding to the interfering object in two adjacent depth images are likely to be the same, and the first depth image and the last frame of the previous depth image are two adjacent depth images.
  • the first depth image is further adjusted by using the adjusted depth image of the last frame of the front depth image, and the pixel value of the pixel corresponding to the interfering object in the first depth image can be further set to zero.
  • the method before the constructing a three-dimensional model of the target object according to the first color image and the processed first depth image, the method further includes: If the depth image is not the first depth image in the depth image sequence, determine the normal vector of each pixel in the depth image of the previous frame of the first depth image; The normal vectors of the pixels corresponding to the interfering object are all zero, and the normal vectors of the pixels in the previous frame of depth image except for the pixels corresponding to the interfering object are all non-zero, and in the depth image sequence
  • the depth images of each frame are sorted according to the sequence obtained by scanning; the pixel value of the third pixel in the first depth image is set to zero; the pixel in the first depth image is the same as the previous one
  • the pixels in the depth image of the frame have a one-to-one correspondence, and the normal vector of the corresponding pixel of the third pixel in the depth image of the previous frame is zero.
  • the coordinates of the pixels corresponding to the interfering objects in two adjacent depth images are likely to be the same.
  • the pixel corresponding to the interfering object in the previous depth image is determined by calculating the normal vector of each pixel in the previous depth image of the first depth image.
  • the pixel corresponding to the pixel corresponding to the interfering object in the previous depth image is set to zero, and the depth information corresponding to the interfering object in the first depth image can be further filtered out.
  • the method before the determining that the displacement of the target object is less than a first threshold according to the pre-depth image sequence, the method further includes: determining the pre-color image in each frame Pixel corresponding to the interfering object; respectively set the pixel value of the reference pixel in the pre-depth image of each frame to zero to obtain the processed pre-depth image of each frame; the reference pixel includes the pre-depth image Pixels in the depth image corresponding to the pixels corresponding to the interference object in the pre-color image corresponding to the pre-depth image, and pixels in the pre-depth image whose pixel value is greater than the fourth threshold And pixels in the pre-depth image whose pixel value is less than a fifth threshold, and the fourth threshold is greater than the fifth threshold; according to the processed frames of each frame of the pre-depth image Depth image, determine a bounding box of the target object to obtain a bounding box sequence; the first bounding box to the last bounding box in the bounding box sequence are sequentially based on the processed pre-depth image sequence From the first
  • the pre-depth image sequence After scanning the target object to obtain the pre-depth image sequence and the pre-color image sequence, process each frame of the pre-depth image in the pre-depth image sequence, and determine the target according to the adjusted pre-depth image of each frame A bounding box of the object to obtain a bounding box sequence.
  • process the frame of pre-depth image after scanning the target object to obtain a frame of pre-color image and a frame of pre-depth image corresponding to the frame of pre-color image, process the frame of pre-depth image, and process the frame of pre-depth image according to the processed pre-depth image
  • the image determines a bounding box. That is, every time a frame of front color image is obtained by scanning the target object and a frame of front color image corresponding to the frame of front color image, a bounding box is determined.
  • the distance between the center point of the bounding box and the center point of the previous bounding box is based on the front depth image of the frame
  • a bounding box is determined by the depth image processed by the previous frame depth image.
  • the method before the scanning the target object to obtain the color image sequence and the depth image sequence, the method further includes: determining the second-to-last frame before the front color image Set a rectangular area of the image of the target object in the color image to obtain the first area; according to the reference front depth image, determine where the image of the target object in the last frame of the front color image is located The second area; the pixels in the reference pre-depth image correspond to the pixels in the last frame of the pre-color image, and the second area is the pixel value in the reference pre-depth image.
  • the zero pixels are in the area formed by the corresponding pixels in the last frame of the front color image; the scanning the target object to obtain the color image sequence and the depth image sequence includes: determining the When the second area is included in the third area in the last frame of the pre-color image, scan the target object to obtain the color image sequence and the depth image sequence; the third area is the The pixels in the first area are an area composed of corresponding pixels in the last frame of the front color image.
  • the second area of the target object in the last frame of the front color image is included in the third area of the last frame of the front color image, it indicates that the target object is scanned to obtain the last frame of the front color image
  • the posture of the image is basically the same as the posture when the front-end color image of the penultimate frame is obtained by scanning.
  • the target object is scanned to obtain the color image sequence and the depth image sequence; this can be determined quickly and accurately
  • the pose of the target object is basically unchanged, so that the depth image sequence and color image sequence that meet the requirements can be obtained by scanning.
  • the interfering object is a user's hand.
  • the scanning the target object to obtain the front color image sequence and the front depth image sequence may be scanning the target object and the hand of the user who directly contacts the target object to obtain the front color image Sequence and the pre-depth image sequence.
  • the user can directly hold the target object for scanning to obtain the front color image sequence and the front depth image sequence.
  • the user adjusting the posture of the target object may be that the user directly holds the target object to adjust the posture of the target object.
  • the scanning of the target object to obtain the color image sequence and the depth image sequence may be scanning the target object and the hand of the user who directly contacts the target object to obtain the color image sequence and the depth image sequence.
  • the depth image sequence In this implementation manner, the user can directly hold the target object for scanning, which has simple operation and high scanning efficiency.
  • the method before the scanning the target object to obtain the color image sequence and the depth image sequence, the method further includes: instructing a user to adjust the posture of the target object multiple times;
  • the scanning the target object to obtain the color image sequence and the depth image sequence includes: scanning the target object to obtain the color image sequence when the user adjusts the posture of the target object multiple times And the depth image sequence.
  • the user can be notified in time to adjust the posture of the target object, so as to perform a panoramic scan of the target object.
  • the method further includes: setting the area of the image of the target object in any frame of the front color image in the front color image to the same The ratio of the area of the set color image is not in the target interval, or the image of the target object in any frame of the front color image is not in the target area of any frame of the front color image, or In the case that the displacement is not less than the first threshold, the user is instructed to adjust the position of the target object.
  • the user can be notified in time to adjust the position of the target object, so as to scan the target object to obtain an image that meets the requirements.
  • the method further includes: in a case where it is determined that the second area is not included in the third area, instructing the user to keep the pose of the target object unchanged.
  • the second area of the target object in the last frame of the front color image is not included in the third area of the last frame of the front color image, it indicates that the target object is scanned to get the last frame of front
  • the pose of the color image is different from the pose of the penultimate frame of the front color image being scanned.
  • the user is instructed to maintain the posture of the target object, so as to obtain the depth image sequence and color image sequence that meet the requirements by scanning.
  • an embodiment of the present application provides a device for reconstructing a three-dimensional model of an object.
  • the device includes a determining module for determining a first color image and a first depth image of a target object, and transmitting the first color image to A model reconstruction module, and transmitting the first depth image to a depth map processing module; wherein, the target object includes a target object and an interfering object; the pixels in the first color image and the first depth image The pixel points of the one-to-one correspondence; the determining module is configured to determine the color interference pixel points corresponding to the interference object in the first color image, and send the first description information to the depth map processing module; the first color image A description information is used to describe the coordinates of the color interference pixel points in the first color image; the depth map processing module is used to adjust the depth interference pixel points in the first depth image according to the first description information To obtain the processed first depth image, and the depth interference pixel points are the pixels in the first depth image that correspond to the color interference pixels one-to-one
  • the device further includes: an acquisition module, specifically configured to acquire a color image sequence and a depth image sequence, and transmit the color image sequence to the determination module and the model reconstruction module, and The depth image sequence is transmitted to the depth map processing module; the color image sequence includes multi-frame color images of the target object in multiple poses, and the depth image sequence includes the target object in the Multi-frame depth images in various poses; wherein, the first depth image is any one of the multi-frame depth images, and the first color image is the same as the one among the multi-frame color images.
  • the determining module is specifically configured to determine the color interference pixel corresponding to the interference object in each frame of the color image, and send the second description information to the depth map processing module;
  • the second description information is used to describe the coordinates of the pixel points corresponding to the interfering object in the color image of each frame;
  • the depth map processing module is used to adjust the depth image of each frame according to the second description information.
  • the pixel value of the mid-depth interference pixel is obtained to obtain the processed depth image sequence; the pixel corresponding to the interference object in the color image is the corresponding pixel in the depth image corresponding to the color image.
  • the model reconstruction module is specifically configured to construct a three-dimensional model of the target object according to multiple frames of the first color image and multiple frames of the processed first depth image.
  • the device further includes: a scanning module for scanning the target object to obtain a front color image sequence and a front depth image sequence, and combining the front depth color image sequence and the The front depth image sequence is transmitted to the determining module; the front depth image in the front depth image sequence corresponds to the front color image in the front color image sequence in a one-to-one correspondence, and the front depth image The pixel points in the front depth image correspond to the pixels in the front color image corresponding to the front depth image; the scanning module is further configured to determine each frame of the front color image by the determining module The ratio of the area of the area where the image of the target object is located to the area of the front color image is in a target interval, and the image of the target object in the front color image is in the target area of the front color image , And when it is determined according to the pre-depth image sequence that the displacement of the target object is less than the first threshold, the target object is scanned to obtain the color image sequence and the depth image sequence; the target area includes The area within the center point
  • the depth map processing module is specifically configured to set the pixel value of the depth interference pixel in the first depth image to zero according to the first description information.
  • the determining module is further configured to determine the pixel corresponding to the interfering object in the last frame of the front color image in the front color image sequence, and transmit the third description information
  • the third description information is used to describe the coordinates of the pixel points corresponding to the interfering object in the last frame of color image, and each frame in the front color image sequence is The set color images are sorted according to the sequence obtained by scanning;
  • the depth map processing module is further configured to, according to the third description information, combine the first frame of the last pre-depth image in the pre-depth image sequence
  • the pixel value of the pixel is set to zero to obtain the reference front depth image;
  • the last frame of the front depth image corresponds to the last frame of the front color image, and the first pixel includes the last frame Pixels in the pre-depth image corresponding to the pixels corresponding to the interference object in the last frame of pre-color image, pixels in the last frame of pre-depth image with pixel values greater than a second threshold, and
  • the determining module is further configured to determine the value of the first depth image when the first depth image is not the first depth image in the depth image sequence
  • the normal vector of each pixel in the depth image of the previous frame is transmitted to the depth map processing module; the fourth description information is used to describe the normal vector of each pixel in the depth image of the previous frame;
  • the normal vectors of the pixels corresponding to the interfering object in the previous frame of depth image are all zero, and the normal vectors of the pixels other than the pixels corresponding to the interfering object in the previous depth image are not Zero, the depth images of each frame in the depth image sequence are sorted according to the sequence obtained by scanning;
  • the depth map processing module is further configured to combine the depth images in the first depth image according to the fourth description information
  • the pixel value of the third pixel is set to zero; the pixel in the first depth image corresponds to the pixel in the depth image of the previous frame one-to-one, and the third pixel is in the depth of the previous frame.
  • the determining module is further configured to determine pixels corresponding to the interfering objects in the front color image of each frame, and send fifth description information to the depth map processing module;
  • the fifth description information is used to describe the coordinates of the pixel points corresponding to the interfering object in the front color image of each frame;
  • the depth map processing module is also used to separately compare each
  • the pixel value of the reference pixel in the pre-depth image of the frame is set to zero to obtain the pre-depth image of each frame after processing; the reference pixel includes the pre-depth image and the pre-depth image.
  • the pixel corresponding to the pixel corresponding to the interference object in the front color image corresponding to the image, the pixel with the pixel value greater than the fourth threshold in the front depth image, and the pixel value in the front depth image For pixels smaller than the fifth threshold, the fourth threshold is greater than the fifth threshold; according to each frame of the pre-depth image in the pre-depth images of the processed frames, one of the target objects is determined Bounding boxes to obtain a bounding box sequence; the first bounding box to the last bounding box in the bounding box sequence are based on the first frame of the processed pre-depth image sequence to the last A frame of pre-depth image is determined; the determining module is also used to determine when the distance between the center points of any two adjacent bounding boxes in the bounding box sequence is less than the first threshold The displacement is less than the first threshold.
  • the determining module is further configured to determine a rectangular area of the image of the target object in the second-to-last frame of the front color image in the front color image to obtain the first area According to the reference front depth image, determine the second area where the image of the target object in the last frame of the front color image is located; the pixel points in the reference front depth image and the last The pixel points in the pre-frame color image correspond one-to-one, and the second area is the pixel point corresponding to the pixel point in the reference pre-depth image whose pixel value is not zero in the last frame pre-color image
  • the scanning module is specifically configured to scan the target object when the determining module determines that the second area is included in the third area in the last frame of the pre-color image
  • the color image sequence and the depth image sequence; the third area is an area composed of pixels in the first area corresponding to pixels in the last frame of the front color image.
  • the interfering object is a user's hand.
  • the device further includes: an indication module for determining, in the determining module, where the image of the target object is located in any frame of the front color image in the front color image The ratio of the area of the region to the area of the front color image of any frame is not in the target interval, or the image of the target object in the front color image of any frame is not in the front of any frame In the case where the target area of the color image or the displacement is not less than the first threshold, the user is instructed to adjust the position of the target object.
  • the indication module is further configured to determine, in the determining module, the area of the region where the image of the target object is located in the front color image of each frame and the front color image The ratio of the area of is in the target interval, and the image of the target object in the front color image is in the target area of the front color image, and it is determined according to the front depth image sequence that the displacement of the target object is less than In the case of the first threshold, instruct the user to adjust the posture of the target object multiple times; the scanning module is specifically configured to scan the target when the user adjusts the posture of the target object multiple times Object to obtain the color image sequence and the depth image sequence.
  • the indicating module is further configured to instruct the user to keep the target object in a case where the determining module determines that the second area is not included in the third area. The pose remains unchanged.
  • an embodiment of the present application provides another object 3D model reconstruction device, which includes a processor, a bus, a depth sensor module, and a color camera; the depth sensor module, the color camera, and the processor pass through The bus is connected; the depth sensor module and the color camera are used to perform a panoramic scan of the target object under the control of the processor; the processor is used to control the object three-dimensional model reconstruction device to perform the above-mentioned One aspect and any one of the optional implementation methods of the foregoing first aspect.
  • the embodiments of the present application provide a mobile terminal, which includes a memory, a processor, a bus, a depth sensor module, and a color camera; the color camera and the depth sensor module are located in the mobile terminal
  • the memory, the depth sensor module, the color camera, and the processor are connected by the bus; the depth sensor module and the color camera are used for pairing under the control of the processor
  • the target object performs a panoramic scan; the memory is used to store computer programs and instructions; the processor is used to call the computer programs and instructions stored in the memory to make the mobile terminal execute the above-mentioned first aspect and the above-mentioned first aspect
  • any optional implementation method is used to call the computer programs and instructions stored in the memory to make the mobile terminal execute the above-mentioned first aspect and the above-mentioned first aspect
  • an embodiment of the present application provides a computer-readable storage medium, the computer storage medium stores a computer program, and the computer program includes program instructions that, when executed by a processor, cause the processing
  • the device executes the foregoing first aspect and any one of the optional implementation methods of the foregoing first aspect.
  • FIG. 1 is a schematic diagram of the hardware structure of a terminal provided by an embodiment of the application.
  • 2A is a flowchart of a method for reconstructing a three-dimensional model of an object according to an embodiment of the application
  • 2B is a flowchart of another method for reconstructing a three-dimensional model of an object according to an embodiment of the application
  • 3A is a schematic diagram of a reconstruction process of a three-dimensional object model provided by an embodiment of the application.
  • FIG. 3B is a schematic diagram of a process of adjusting a depth image according to an embodiment of the application.
  • FIG. 5 is a schematic diagram of structured light according to an embodiment of the application.
  • FIG. 6 is a schematic diagram of a TOF in an embodiment of the application.
  • FIG. 7 is a flowchart of a method of gridding + texture mapping in an embodiment of the application.
  • FIG. 8 is a flowchart of a specific grid implementation scheme in an embodiment of the present invention.
  • FIG. 9 is a flowchart of a specific implementation scheme of texture mapping in an embodiment of the present invention.
  • Fig. 10 is a specific example of gridding + texture mapping in the embodiment of the present invention.
  • FIG. 11 is a schematic structural diagram of an apparatus for reconstructing a three-dimensional model of an object according to an embodiment of the application.
  • FIG. 12 is a schematic structural diagram of another object three-dimensional model reconstruction device provided by an embodiment of the application.
  • FIG. 13 is a schematic structural diagram of a mobile terminal provided by an embodiment of this application.
  • FIG. 1 shows a schematic diagram of an optional hardware structure of the terminal 100.
  • the apparatus for reconstructing a three-dimensional model of an object in the embodiment of the present application may be the terminal 100.
  • the terminal 100 may include a radio frequency (RF) unit 110, a memory 120, an input unit 130, a display unit 140, a photographing unit 150, an audio circuit 160, a speaker 161, a microphone 162, a processor 170, and external Port 180, power supply 190 and other components.
  • RF radio frequency
  • the radio frequency unit 110 can be used to receive and send signals during the process of sending and receiving information or talking. In particular, after receiving the downlink information of the base station, it is transmitted to the processor 170 for processing; in addition, the uplink data is sent to the base station.
  • the RF unit (also referred to as an RF circuit) 110 includes, but is not limited to, an antenna, at least one amplifier, a transceiver, a coupler, a Low Noise Amplifier (LNA), a duplexer, and the like.
  • the radio frequency unit 110 may also communicate with network devices and other devices through wireless communication.
  • the wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile Communication (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (Code Division Multiple Access). Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Long Term Evolution (LTE), Email, Short Messaging Service (SMS), etc.
  • GSM Global System of Mobile Communication
  • GPRS General Packet Radio Service
  • CDMA Code Division Multiple Access
  • WCDMA Wideband Code Division Multiple Access
  • LTE Long Term Evolution
  • Email Short Messaging Service
  • the memory 120 may be used to store instructions (for example, software programs) and data.
  • the processor 170 executes various functional applications and data processing of the terminal 100 by running the software programs stored in the memory 120.
  • the memory 120 may mainly include a storage instruction area and a storage data area.
  • the storage data area may store data created according to the use of the terminal (such as audio data, phone books, color images, depth images), etc.;
  • the storage instruction area may store operating systems, Software units such as applications, instructions required for at least one function, or their subsets or extensions.
  • the memory 120 may include a double-rate synchronous dynamic random access memory (Synchronous Dynamic Random Access Memory, SDRAM), of course, may also include a high-speed random access memory, or include other storage units such as non-volatile memory, such as at least one disk storage device, Flash memory devices, or other volatile solid-state storage devices, etc.
  • SDRAM Synchronous Dynamic Random Access Memory
  • non-volatile memory such as at least one disk storage device, Flash memory devices, or other volatile solid-state storage devices, etc.
  • the input unit 130 may be used to receive inputted number or character information, and generate key signal input related to user settings and function control of the terminal.
  • the input unit 130 may include a touch screen 131 and other input devices 132.
  • the touch screen 131 can collect the user's touch operations on or near it (for example, the user uses any suitable objects such as fingers, joints, stylus, etc. to operate on the touch screen or near the touch screen), and drive the corresponding operation according to a preset program ⁇ Connection device.
  • the touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 170, and can receive and execute the command sent by the processor 170; the touch signal includes at least a touch Point coordinate information.
  • the touch screen 131 may provide an input interface and an output interface between the terminal 100 and the user.
  • touch screens can be implemented in multiple types such as resistive, capacitive, infrared, and surface acoustic waves.
  • the input unit 130 may also include other input devices.
  • the other input device 132 may include, but is not limited to, one or more of a physical keyboard, function keys (such as a volume control button 132, a switch button 133, etc.), a trackball, a mouse, and a joystick.
  • the touch screen 131 can cover the display panel 141.
  • the touch screen 131 detects a touch operation on or near it, it transmits it to the processor 170 to determine the type of the touch event, and then the processor 170 displays it on the display panel according to the type of the touch event.
  • the touch screen and the display unit can be integrated into one component to realize the input, output, and display functions of the terminal 100; for ease of description, the embodiment of the present invention uses a touch screen to represent the set of functions of the touch screen and the display unit; In some embodiments, the touch screen and the display unit can also be used as two independent components.
  • the display unit 140 may be used to display information input by the user or information provided to the user and various menus of the terminal 100.
  • the display unit is also used to display color images acquired by the device using the camera 150, which may include preview images in certain shooting modes, initial images captured, and images processed by certain algorithms after shooting, such as According to the color image obtained by shooting the target object, a three-dimensional model of the target object is reconstructed.
  • the photographing unit 150 is used to collect images or videos, and can be triggered to be turned on by instructions of an application program to realize a photographing or video recording function.
  • the photographing unit may include imaging lenses, filters, image sensors and other components. The light emitted or reflected by the object enters the imaging lens, passes through the filter, and finally converges on the image sensor.
  • the imaging lens is mainly used to converge and image the light emitted or reflected by the object (also known as the object to be photographed or the target object) in the camera angle of view; the filter is mainly used to capture the excess light waves in the light (for example, in addition to visible light)
  • the image sensor is mainly used to photoelectrically convert the received light signal, convert it into an electrical signal, and input it to the processing 170 for subsequent processing.
  • the photographing unit 150 may further include a color camera 151 and a depth camera 152; the color camera is used to collect color images of the target object, including a color camera commonly used in popular terminal products.
  • the depth camera is used to obtain the depth information of the target object.
  • the depth camera can be implemented through Time of Flight (TOF) technology and structured light technology.
  • TOF Time of Flight
  • TOF technology is that a sensor (such as a depth sensor module) emits modulated near-infrared light and reflects after encountering an object.
  • the sensor calculates the time difference or phase difference between light emission and reflection to convert the distance of the photographed scene to generate depth information .
  • the three-dimensional outline of the object can be presented in a topographic map with different colors representing different distances.
  • structured light is a group of system structures composed of projection elements and cameras.
  • the projection element is used to project specific light information (such as diffraction by grating) onto the surface of the object and the background, and then it is collected by the camera.
  • specific light information such as diffraction by grating
  • the camera According to the change of the light signal caused by the object (such as the change and displacement of light thickness), the position and depth of the object are calculated; and then the entire three-dimensional space is restored.
  • the audio circuit 160, the speaker 161, and the microphone 162 may provide an audio interface between the user and the terminal 100.
  • the audio circuit 160 can transmit the electrical signal converted from the received audio data to the speaker 161, which is converted into a sound signal for output by the speaker 161; on the other hand, the microphone 162 is used to collect sound signals and can also convert the collected sound signals It is an electrical signal, which is received by the audio circuit 160 and converted into audio data, and then processed by the audio data output processor 170, and sent to, for example, another terminal via the radio frequency unit 110, or the audio data is output to the memory 120 for further processing,
  • the audio circuit may also include an earphone jack 163 for providing a connection interface between the audio circuit and the earphone.
  • the processor 170 is the control center of the terminal 100. It uses various interfaces and lines to connect various parts of the entire terminal. It executes various functions of the terminal 100 by running or executing instructions stored in the memory 120 and calling data stored in the memory 120. Function and process data to control the terminal as a whole.
  • the processor 170 may include one or more processing units; preferably, the processor 170 may integrate an application processor and a modem processor, where the application processor mainly processes the operating system, user interface, and application programs, etc. , The modem processor mainly deals with wireless communication. It can be understood that the foregoing modem processor may not be integrated into the processor 170.
  • the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented on separate chips.
  • the processor 170 can also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing equipment, read and process data in the software, especially read and process data and programs in the memory 120, so that Each functional module executes the corresponding function, thereby controlling the corresponding component to act as required by the instruction.
  • the terminal 100 also includes an external interface 180.
  • the above-mentioned external interface can be a standard Micro USB interface or a multi-pin connector, which can be used to connect the terminal 100 to communicate with other devices, or to connect a charger to charge the terminal 100. .
  • the terminal 100 further includes a power source 190 (such as a battery) for supplying power to various components.
  • a power source 190 such as a battery
  • the power source may be logically connected to the processor 170 through a power management system, so that functions such as charging, discharging, and power management are realized through the power management system.
  • the terminal 100 may also include a flash, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be repeated here. All the methods described below can be applied to the terminal shown in FIG. 1.
  • FIG. 1 is only an example of an object 3D model reconstruction device, and does not constitute a limitation on the object 3D model reconstruction device. It may include more or less components than shown in the figure, or combine some Parts, or different parts.
  • Object 3D model reconstruction is to use a color camera and a depth camera to perform a panoramic scan of an object, that is, scan the object to obtain a color image sequence and a depth image sequence.
  • the color image sequence includes the color image of the object in multiple poses.
  • the depth image sequence Including depth images of the object in multiple poses; using the depth image sequence and the color image sequence to fuse to obtain a textured mesh model, that is, a three-dimensional (3D) model of the target object.
  • the method for reconstructing a three-dimensional object model provided by the embodiments of the present application can be applied to a scene for reconstructing a three-dimensional object model. The following is a brief introduction to the reconstruction scene of the 3D model of the object.
  • Object 3D model reconstruction scene In this scene, the object 3D model reconstruction process is divided into three stages, namely preview stage, online scanning stage and post-processing stage. Only after the preview phase is completed, will the online scanning phase be entered.
  • the preview stage the mobile phone is placed flat on the desktop or fixed on the stand, and the user directly holds the object to scan. During the scanning process, the user can adjust the angle at which the mobile phone scans the object and the distance between the mobile phone and the object.
  • the mobile phone instructs the user to adjust the posture of the object and enter the online scanning phase; in the online scanning phase, the user In the process of adjusting the posture of the object, scan the object to obtain a color image sequence and a depth image sequence, and process each frame depth image in the depth image sequence to eliminate the depth information of the hand in each frame depth image; In the processing stage, the color image sequence and the processed depth image sequence are used to fuse to obtain a textured mesh model, that is, a 3D model of the object.
  • the user adjusts the position of the object, and keeps the position of the object unchanged when the position of the object is appropriate, so that the color image and/or depth image obtained by the mobile phone scanning the object meet the requirements. That is to say, when the position of the object is basically unchanged and the multiple frames of continuous color images recently scanned by the mobile phone meet the requirements, the 3D model reconstruction process of the object enters the online scanning stage.
  • the user adjusts the posture of the object while keeping the position of the object basically unchanged so that the mobile phone can perform a panoramic scan of the object. For example, in the online scanning stage, the user can hold the object to rotate and keep the position of the object unchanged, so that the mobile phone can scan to obtain the color image and depth image of the object in different poses.
  • the same user holds a mobile phone in one hand and an object in the other hand to achieve a panoramic scan of the object.
  • one user holds a mobile phone and another user holds an object to realize a panoramic scan of the object.
  • enter the post-processing stage after completing the meshing and texture mapping, a three-dimensional model of the object with texture can be obtained.
  • the user directly holds the object and scans the object by rotating the object to obtain the color image and depth image of the object in different postures, thereby realizing a panoramic scan of the object, and the operation is simple.
  • the following describes the operations performed by the object 3D reconstruction device in the online scanning stage and the post-processing stage.
  • the embodiment of the present application provides a method for reconstructing a three-dimensional model of an object. As shown in FIG. 2A, the method may include:
  • the device for reconstructing a three-dimensional model of an object determines a first color image and a first depth image of the target object.
  • the target objects include target objects and interference objects.
  • the pixels in the first color image correspond to the pixels with the same coordinates in the first depth image in a one-to-one correspondence.
  • the first color image may be a color image obtained by down-sampling an original color image obtained by scanning the target object.
  • the size of the first color image and the first depth push are the same, that is, the resolution is the same.
  • the device for reconstructing the three-dimensional model of the object determines the color interference pixel points corresponding to the interference object in the first color image.
  • the aforementioned color interference pixel points refer to the pixel points corresponding to the interference object in the first color image. The following will describe in detail how to determine the pixel points corresponding to the interfering objects in the color image, which will not be described in detail here.
  • the device for reconstructing the three-dimensional model of the object adjusts the pixel value of the depth interference pixel in the first depth image to obtain a processed first depth image.
  • the depth interference pixel points are pixels in the first depth image that correspond to the color interference pixel points one-to-one.
  • the device for reconstructing the three-dimensional object model constructs the three-dimensional model of the target object according to the first color image and the processed first depth image.
  • the first depth image obtained by shooting the target object and the interfering object is processed, and the processed first depth image only contains the depth information of the target object.
  • the processed first depth image and the depth image Sequence the corresponding color images to construct a three-dimensional model of the target object; the user can hold the target object for scanning or place the target object on a selective rotating device for scanning, and the operation is simple.
  • the embodiment of the present application provides another method for reconstructing a three-dimensional model of an object. As shown in FIG. 2B, the method may include:
  • the device for reconstructing the three-dimensional model of the object acquires a color image sequence and a depth image sequence.
  • the object 3D model reconstruction device can be a terminal, such as a mobile phone, a tablet computer, a notebook computer, a camera, etc., or a server, etc.
  • the object three-dimensional model reconstruction device obtains the color image sequence and the depth image sequence.
  • the object three-dimensional model reconstruction device can obtain the color image sequence and the depth image sequence from its own storage device, or it can be obtained from other equipment (such as a server).
  • the color image sequence and the depth image sequence includes multiple frames of color images of the target object in multiple poses
  • the depth image sequence includes multiple frames of depth images of the target object in multiple poses
  • the target object includes target objects and interfering objects.
  • the multi-frame depth images in multiple poses are multi-frame depth images obtained by performing a panoramic scan of the target object.
  • the multi-frame depth image at least contains the depth information of each point on the surface of the target object, and the number of depth images contained in the depth image sequence is not limited.
  • the first depth image is any one of the multiple frames of depth images
  • the first color image is an image corresponding to the first depth image in the multiple color images.
  • the first color image and the first depth image are obtained by synchronous scanning. That is, the time interval between the time point when the first color image is scanned and the time when the first depth image is scanned is less than a time threshold, such as 1 ms, 5 ms, 10 ms, and so on.
  • the interfering object is an object directly in contact with the target object, and may be a user's hand or other objects used to adjust the position and/or posture of the target object.
  • the depth images in the depth image sequence correspond to the color images in the color image sequence one to one.
  • the pixel point in any depth image in the depth image sequence corresponds to the pixel point in the color image corresponding to the any depth image in a one-to-one correspondence. That is to say, the resolution of any depth image and the color image corresponding to the any depth image are the same, and the pixel points in the Pth row and Qth column in the any depth image are the same as the Pth row and Qth column in the color image.
  • P and Q are integers greater than 0.
  • the device for reconstructing the three-dimensional model of the object determines the pixel points corresponding to the interfering object in each frame of the color image included in the color image sequence.
  • the device for reconstructing the three-dimensional model of the object can determine the pixel point corresponding to the interfering object in each frame of color image to obtain the coordinates of the pixel point corresponding to the interfering object in each frame of color image.
  • the following method is used to determine the pixel corresponding to the interfering object in any frame of the color image in the color image sequence: input any color image to the deep convolutional neural network for processing to obtain the first feature map, and the depth volume
  • the product neural network is obtained by pre-training; the first feature map is up-sampled to obtain a second feature map with the same resolution as that of any color image.
  • the pixels in the second feature map are the same as the any color image.
  • the pixel point corresponding to the pixel point is used as the pixel point corresponding to the interference object in any color image
  • the calibration pixel point is the pixel point whose probability value of the pixel point corresponding to the interference object in the second feature map is greater than the target threshold.
  • the target threshold can be 0.4, 0.5, 0.6, 0.8, etc.
  • Deep convolutional neural networks can also be replaced by other networks.
  • the device for reconstructing the three-dimensional model of the object can use different types of neural networks to process the color images of each frame to determine the pixels corresponding to the interfering objects in the color images of each frame.
  • the device for reconstructing the three-dimensional model of the object may also use other methods to determine the pixel points corresponding to the interfering object in each frame of the color image, which is not limited in this application.
  • the object three-dimensional model reconstruction device can sequentially determine the pixel points corresponding to the interference object in each frame of the color image in the color image sequence, that is, sequentially determine the coordinates of the pixel points corresponding to the interference object in each frame of the color image.
  • the device for reconstructing the three-dimensional model of the object adjusts the pixel value of the interfering pixel in the depth image of each frame to obtain a processed depth image sequence.
  • the pixel point corresponding to the interference object in any color image in the color image sequence in the depth image corresponding to the any color image is the interference pixel point in the depth image.
  • the pixels from the first pixel position to the tenth pixel position in the color image corresponding to any depth image are the pixels corresponding to the interfering object
  • the interfering pixels in any depth image are the pixels in any depth image
  • the pixel value of the interfering pixel in each frame of the depth image is set to zero to obtain the processed depth image sequence.
  • the device for reconstructing the three-dimensional model of the object determines the three-dimensional model of the target object according to the color image sequence and the processed depth image sequence.
  • Figure 3A is the main process from scanning an object to realizing a three-dimensional model reconstruction of the object.
  • First scan the target object scan through the depth camera to obtain the depth image sequence, scan through the color camera to obtain the color image sequence; process the depth image sequence; use the processed depth image sequence and the color image sequence to fuse to obtain
  • the mesh model of the texture is the 3D model of the target object.
  • the depth image sequence obtained by shooting the target object and the interference object is processed to obtain a depth image sequence containing only the depth information of the target object, and the processed depth image sequence and the color corresponding to the depth image sequence are used.
  • the image sequence is fused to obtain a three-dimensional model of the target object; the user can hold the target object for scanning or place the target object on a selective rotating device for scanning, and the operation is simple.
  • the device for reconstructing a three-dimensional object model Before the device for reconstructing a three-dimensional object model executes 221, the device for reconstructing a three-dimensional object model needs to perform a panoramic scan of the target object to obtain the aforementioned depth image sequence and the aforementioned color image sequence.
  • the object three-dimensional model reconstruction device may perform the following operations:
  • the object 3D model reconstruction device scans the target object.
  • the target objects include target objects and interference objects.
  • the device for reconstructing the three-dimensional model of the object can use a depth camera to scan the target object to obtain multiple frames of front depth images, and use a color camera to scan the target object to obtain multiple frames of front color images.
  • the multi-frame front depth images are sorted according to the sequence obtained by scanning, and the multi-frame front color images are sorted according to the sequence obtained by scanning.
  • the front color image of each frame obtained by scanning the target object can be down-sampled, so that the front color image of each frame is different from the front color image.
  • the resolution of the depth images is the same.
  • the front color image sequence may be a color image sequence obtained by scanning the target object, or may be obtained by down-sampling the original front color image sequence obtained by scanning the target object. It can be understood that the device for reconstructing the three-dimensional model of the object can down-sample each frame of the front color image obtained by scanning the target object to obtain a front color image with the same resolution as the front depth image.
  • the object 3D model reconstruction device judges whether the front color image sequence and the front depth image sequence meet the target conditions.
  • the front color image sequence is the last F frame front color image in the multi-frame front color image
  • the front depth image sequence is the last F frame front depth image in the multi-frame front depth image
  • F is An integer greater than 1. That is, the front depth image sequence includes F frames of front depth images obtained by scanning the target object most recently, and the front color image sequence includes F frames of front color images obtained by scanning the target object most recently.
  • the front depth image in the front depth image sequence corresponds to the front color image in the front color image sequence one-to-one, and the pixel points in any front depth image in the front depth image sequence correspond to the any The pixels in the front color image corresponding to the front depth image correspond one-to-one.
  • the condition that the pre-color image sequence and the pre-depth image sequence meet the target condition may be that the area of the image of the target object in each frame of the pre-color image sequence in the pre-color image sequence and the area of each frame before The ratio of the area of the set color image is in the target interval, and the image of the target object in each frame of the front color image is in the target area of each frame of the front color image, and the displacement of the target object determined according to the front depth image sequence is less than The first threshold.
  • the target interval can be 0.1 to 0.8, 0.2 to 0.6, and so on.
  • the first threshold may be 0.5 cm, 1 cm, 2 cm, etc.
  • the distance between the center point of the target area of each frame front color image and the center point of each frame front color image is less than the distance threshold, which can be one-tenth or five times the length of each frame front color image One-of-a-kind.
  • the object three-dimensional model reconstruction device instructs the user to adjust the posture of the target object multiple times.
  • Object 3D model reconstruction can output voice to instruct the user to adjust the posture of the target object multiple times, or output corresponding information or interface to instruct the user to adjust the posture of the target object multiple times, or instruct the user to adjust the posture of the target object multiple times in other ways.
  • the posture of the target object is not limited in this application.
  • the device for reconstructing the three-dimensional model of the object determines that the pre-color image sequence and the pre-depth image sequence satisfy the target condition to ensure that the user can adjust the posture of the target object.
  • the depth image sequence and the color image sequence obtained by scanning the target object can successfully construct a three-dimensional model of the target object.
  • the object 3D model reconstruction device instructs the user to adjust the position of the target object, and returns to step 2211.
  • the device for reconstructing the three-dimensional model of the object instructs the user to adjust the position of the target object when the pre-color image sequence and the pre-depth image sequence do not meet the target condition, and scan the target object to obtain a new pre-color image ( The F frame color image obtained from the most recent scan of the target object) and a new front depth image (the F frame depth image obtained from the most recent scan of the target object).
  • the device for reconstructing a three-dimensional object model scans the target object to obtain a color image sequence and a depth image sequence when the user adjusts the posture of the target object.
  • the scanning of the target object to obtain the color image sequence and the depth image sequence may be: scanning the target object to obtain the original color image sequence and the depth image sequence; down-sampling each frame of the original color image in the original color image sequence to obtain the foregoing Color image sequence.
  • the color image sequence may be obtained by scanning the target object, or it may be obtained by down-sampling the original front color image sequence obtained by scanning the target object.
  • the user is instructed to adjust the pose of the target object, and then the target object is scanned to obtain the depth image sequence and the color image sequence used to construct the target object. Simple to implement.
  • the device for reconstructing the three-dimensional model of the object can adjust the pixel value of the interfering pixel in the depth image of each frame in the same manner.
  • the operation of the object 3D model reconstruction device to adjust the pixel value of the interfering pixel in any depth image may be as follows: determine the pixel corresponding to the interfering object in the color image corresponding to any depth image to obtain a description of the interfering object in the color image
  • the coordinate information of the coordinates of the corresponding pixel point, the pixel point in any depth image corresponds to the pixel point in the color image one-to-one; according to the coordinate information, the pixel point corresponding to the interference object in the color image is in the
  • the pixel value of the corresponding pixel in any depth image is adjusted to zero.
  • the coordinate information is used to describe the coordinates of one or more pixels. It can be understood that the pixel corresponding to the interfering object in the color image corresponds to the pixel in any depth image as the interfering pixel in any depth image
  • the operation of the object 3D model reconstruction device to adjust the pixel value of the interference pixel in any depth image may also be as follows: determine the pixel corresponding to the interference object in the color image corresponding to any depth image to obtain the first binary image, the color The pixel points in the image correspond to the pixels in the first binary image one-to-one, and the pixel points corresponding to the interfering objects in the color image have the first value in the first binary image.
  • the values of other pixels in the second binary image are all second values; the value of the corresponding pixel in the first binary image in any depth image is the pixel value of the pixel with the first value Set to zero, the pixel points in the any depth image correspond to the pixels in the first binary image one-to-one.
  • the first value may be 255, and the second value may be zero.
  • the values of the pixels in the color image except for the pixels corresponding to the interfering object in the first binary image correspond to the second value. It can be understood that, in any depth image, a pixel with a corresponding pixel value in the first binary image having the first value is an interference pixel in any depth image.
  • depth(i,j) represents the pixel value of the pixel in the i-th row and j-th column in any depth image, that is, the depth value; handmask(i,j) represents the i-th row in the first binary image The value of the pixel in column j.
  • the above two methods are only the two methods provided by this application for setting the interfering pixels in the depth image to zero.
  • the object 3D model reconstruction device can also use other methods to set the interfering pixels in the depth image to zero. List them again.
  • the device for reconstructing the three-dimensional model of the object can also adjust the depth image of each frame by using the following operations.
  • the operation performed here can be performed after setting the value of the interfering pixel in each frame of depth image to zero, or before setting the value of the interfering pixel in each frame of depth image to zero.
  • the first depth image currently to be processed by the device for reconstructing the three-dimensional object model is the first depth image in the depth image sequence
  • each frame depth image in the depth image sequence is sorted according to the order obtained by scanning
  • each frame front color image in the front color image sequence is sorted according to the order obtained by scanning
  • the preceding depth image sequence is sorted
  • the pixel value of the first pixel in the last frame of the pre-depth image in is set to zero to obtain the reference pre-depth image
  • each frame of the pre-depth image in the pre-depth image sequence is sorted in the order of scanning.
  • the last frame of the front depth image corresponds to the last frame of the front color image
  • the first pixel includes the last frame of the front depth image corresponding to the interference object in the last frame of the front color image
  • the second The threshold is greater than the third threshold; the pixel value of the second pixel in the first depth image is set to zero; the pixel in the first depth image corresponds to the pixel in the reference pre-depth image one-to-one, The pixel value of the corresponding pixel of the second pixel in the reference pre-depth image is zero.
  • the second threshold may be 80cm
  • the third threshold may be 10cm, and so on.
  • the device for reconstructing the three-dimensional model of the object may obtain the reference front depth image by using the following operations: determining the pixel points corresponding to the interfering object in the last frame of the front color image to obtain the second binary image;
  • the pixels in the last frame of the front color image correspond to the pixels in the second binary image one-to-one, and the pixels corresponding to the interfering objects in the last frame of the front color image are in the second binary image
  • the values of the corresponding pixels are all the third value, and the values of other pixels in the second binary image are all the fourth values; the corresponding pixel in the second binary image in the previous frame of the preceding depth image
  • the pixel value of the pixel with the third value is set to zero to obtain the intermediate front depth image, and the pixel in the last frame of the front depth image is the same as the pixel in the second binary image.
  • the pixel values of pixels in the intermediate front depth image whose pixel values are greater than the second threshold and the pixel values less than the third threshold are set to zero to obtain the reference front depth image.
  • the third value may be 255
  • the fourth value may be 0.
  • the value of the pixel points corresponding to the second binary image in the pixel points corresponding to the second binary image of the pixels other than the pixels corresponding to the interfering object in the last frame of the front color image is the fourth value.
  • the above setting of the pixel value of the second pixel in the first depth image to zero may be: determining a third binary image according to the reference pre-depth image, and the pixel in the reference pre-depth image and the second pixel The pixel points in the three-binary image correspond one-to-one, and the pixel with a pixel value of zero in the reference pre-depth image has the fifth value in the third binary image.
  • the reference pre-depth The pixel value of the pixel in the image whose pixel value is not zero is the sixth value in the third binary image; the pixel value corresponding to the third binary image in the first depth image is the The pixel value of the pixel of the fifth value is set to zero, and the pixel in the first depth image corresponds to the pixel in the third binary image one-to-one.
  • the fifth value may be 0, and the sixth value may be 255.
  • the device for reconstructing the three-dimensional object model determines each pixel in the depth image of the previous frame of the first depth image in the depth image sequence
  • the normal vector of the point, the depth images in the depth image sequence are sorted according to the sequence obtained by scanning; the pixel value of the third pixel in the first depth image is set to zero, and the pixels in the first depth image
  • the points correspond one-to-one with the pixels in the depth image of the previous frame, and the normal vector of the third pixel corresponding to the pixel in the depth image of the previous frame is zero.
  • the device for reconstructing the three-dimensional model of the object calculates a normal vector map according to the previous depth image, and the pixels in the normal vector map correspond to the pixels in the previous depth image one-to-one; according to the normal vector map , Determine the fourth binary image, the pixels in the normal vector map correspond to the pixels in the fourth binary image one-to-one, and the pixels with zero normal in the normal vector map are in the fourth binary image
  • the values of the corresponding pixels in the normal vector diagram are all seventh values, and the pixels whose normal vectors are not zero in the normal vector diagram have the eighth value in the fourth binary diagram;
  • the pixel value of the corresponding pixel in the fourth binary image in the depth image is set to zero.
  • the pixel value of the pixel in the fourth binary image is set to zero.
  • Pixels correspond one to one.
  • the seventh value may be 0, and the eighth value may be 255.
  • the calculation of the normal vector map may be to convert the depth image of the previous frame into a point cloud, and calculate the normal vector map of the point cloud.
  • the following example illustrates how to calculate the normal vector of the point cloud: corresponding to each pixel on a certain frame of depth image, assuming that the second pixel on the right corresponds to point A in the world coordinate system, similarly, the second pixel below it A pixel corresponds to point B in the world coordinate system, the second pixel to the left corresponds to point C in the world coordinate system, and the second pixel above it corresponds to point D in the world coordinate system, then the pixel
  • the normal vector of the point is: Is the vector corresponding to point A and point B, Is the vector corresponding to point C and point D.
  • the device for reconstructing the three-dimensional model of the object can also use other methods to calculate the normal vector of each pixel in the previous depth image, which is not limited in this application.
  • the device for reconstructing the three-dimensional model of the object may use the following formula to set the pixel value of the pixel with the seventh value corresponding to the fourth binary image in the first depth image to zero:
  • depth(i,j) represents the pixel value of the pixel in the i-th row and j-th column in the first depth image, that is, the depth value; r(i,j) represents the i-th row in the fourth binary image The value of the pixel in column j. It can be seen from formula (2) that the pixel value of the corresponding pixel in the fourth binary image in the first depth image is 255 (that is, the eighth value) remains unchanged, and the first In the depth image, the pixel value of the pixel with the value of 0 (that is, the seventh value) corresponding to the fourth binary image is set to zero.
  • the object 3D model reconstruction device can use the above operations to adjust each frame of the depth image in the depth image sequence, so that each frame of the depth image in the adjusted depth image sequence only retains the depth information of the target object, that is, In each depth image sequence, the pixel values of pixels other than the pixel corresponding to the target object are all zero.
  • FIG. 3B shows a schematic diagram of the process of adjusting the depth image.
  • the target color image corresponds to the first depth image
  • the first binary image is a binary image obtained according to the pixel points corresponding to the interference object in the target color image.
  • the device for reconstructing the three-dimensional model of the object can use the first binary image to adjust the first depth image using formula (1), and the third binary image or the fourth binary image to further adjust the first depth image using formula (2).
  • Depth image It can be seen from FIG. 3B that the pixel value corresponding to the interfering object in the adjusted first depth image is adjusted to zero.
  • the adjusted first depth image basically only retains the depth information of the target object, and can be directly used to construct a three-dimensional model of the target object.
  • the device for reconstructing the three-dimensional model of the object separately processes the depth images of each frame in the same manner, so as to construct a three-dimensional model of the target object using the processed depth images of each frame.
  • steps 221 to 223 correspond to the online scanning stage of the 3D object model reconstruction process
  • step 224 corresponds to the post-processing stage of the 3D object model reconstruction process.
  • the 3D object model reconstruction device has completed the preview stage before performing step 221.
  • the following describes the detailed operation of the 3D object model reconstruction device in the preview stage, which corresponds to step 2212.
  • Fig. 4 is an image preprocessing method provided by an embodiment of the application. As shown in FIG. 4, before performing step 221, the device for reconstructing a three-dimensional model of an object needs to perform the following operations:
  • the device for reconstructing the three-dimensional model of the object determines whether the identification position is the first reference value.
  • Figure 4 describes the process of processing the target front depth image and the target front color image corresponding to the target front depth image.
  • the target front depth image is any frame of the front depth image in the preceding depth image sequence.
  • the target front color image is included in the foregoing front color image sequence.
  • the device for reconstructing a three-dimensional model of an object can use the method in FIG. 4 to sequentially process each frame of the front depth image in the front depth image sequence and each frame of the front color image in the front color image sequence.
  • the above-mentioned first reference value may be true, or 1, or other values, which is not limited in this application. If yes, go to 402; if not, go to 403.
  • the initial value (ie, the initialized value) of the identification bit is the second reference value, and the second reference value may be false, or 0, or other values, which is not limited in this application.
  • the flag bit indicates that the 3D model reconstruction of the object has determined the previous frame depth image based on the previous frame depth image of the target front depth image in the preceding depth image sequence.
  • the image includes a matrix area of the image of the target object; when the flag is false, the flag indicates that a matrix area of the image of the target object is not determined in the previous frame of the front color image.
  • the rectangular area can be understood as the object in the previous frame of the pre-color image determined according to the previous frame of the pre-depth image of the target in the pre-depth image sequence before step 402 is performed by the object 3D model reconstruction device
  • the Region of Interest (ROI) of the object How to determine how to obtain the matrix area will be described in detail below, and will not be described in detail here.
  • the pixel points in the target front color image correspond one-to-one with the pixels in the previous frame front color image.
  • the device for reconstructing the three-dimensional model of the object may use the reference region corresponding to the ROI in the previous frame of the front color image in the target front color image as the ROI in the target front color image.
  • the pixels in the ROI in the previous frame of the front color image correspond to the pixels in the reference area one-to-one.
  • the device for reconstructing the three-dimensional model of the object determines the pixel points corresponding to the interfering object in the reference area in the target front color image to obtain a fifth binary image.
  • the pixel points in the target front color image correspond to the pixel points in the fifth binary image one-to-one.
  • the pixels in the target front color image can be divided into the first type of pixels and the second type of pixels; the first type of pixels are the pixels corresponding to the interfering objects in the reference area, and the second type of pixels is Pixels in the target front color image except for the first type of pixels.
  • the pixel points in the target front color image correspond to the pixels in the fifth binary image one-to-one, and the first type of pixels in the target front color image correspond to the pixels in the fifth binary image.
  • the values of are all the ninth value, and the values of the corresponding pixels of the second type of pixel in the fifth binary image are all the tenth value.
  • Step 402 is adaptive ROI segmentation, that is, determining the pixel points corresponding to the interfering object in the ROI (that is, the aforementioned reference region) in the target front color graphic.
  • the position of the target object is basically unchanged, the area where the image of the target object is located in the two adjacent pre-color images obtained by scanning the target object is roughly the same, so only the interference object corresponding to the aforementioned reference area can be determined pixel.
  • the identification bit is the first reference value, it indicates that the ratio of the area of the image of the target object in the preceding frame of the preceding frame to the area of the preceding frame of preceding color image is in the aforementioned target interval , And the image of the target object in the front color image of the previous frame is in the target area of the front color image of the previous frame.
  • the target area may be a rectangular area or a regular polygon area, and the distance between the center point of the calibration area and the center point of the previous frame of the front color image is less than a distance threshold.
  • the distance threshold may be one-tenth, one-fifth, etc. of the length of the preceding color image of the previous frame.
  • the object three-dimensional model reconstruction device can extract the image of the reference area in the target front color image, and determine the pixel points corresponding to the interference object in the image of the reference area. It can be understood that when the identification bit is the first reference value, the reconstruction of the three-dimensional object model only needs to determine the pixel points corresponding to the interfering object in the reference area in the target front color image, instead of determining the entire target front color image. The pixels corresponding to the interference object can effectively reduce the amount of calculation.
  • the device for reconstructing the three-dimensional model of the object determines the pixel points corresponding to the interfering object in the target front color image to obtain a sixth binary image.
  • the pixel points in the target front color image correspond to the pixel points in the sixth binary image one-to-one.
  • the pixel point corresponding to the interference object in the target front color image has the eleventh value in the sixth binary image.
  • the target front color image except for the pixel point corresponding to the interference object The corresponding pixel values of the pixels in the sixth binary image are all the twelfth value.
  • the eleventh value may be 255, and the twelfth value may be 0.
  • Step 403 is full image segmentation, that is, the pixels corresponding to the interfering objects in the entire target front color image are determined.
  • the device for reconstructing the three-dimensional model of the object adjusts the pixel values of the interfering pixels in the target front depth image to obtain an adjusted target front depth image.
  • the device for reconstructing the object 3D model adjusting the pixel value of the interfering pixel in the target pre-depth image may include: corresponding to the target pre-depth image in the fifth second value map.
  • the pixel value of the pixel with the above ninth value is set to zero, and the pixel in the target front depth image corresponds to the pixel in the fifth binary image one-to-one; or, the target front Set the pixel value of the corresponding pixel in the above-mentioned sixth and second value map in the set depth image.
  • the pixel value of the above-mentioned eleventh value pixel in the set depth image is set to zero.
  • the pixels in the value map correspond one to one.
  • depth(i,j) represents the pixel value of the pixel in the i-th row and j-th column in the target pre-depth image, that is, the depth value; handmask(i,j) the fifth binary image above or the 62nd above The value of the pixel in the i-th row and j-th column of the value map.
  • the device for reconstructing the three-dimensional model of the object may further set the pixel values in the target front depth image that are greater than the second threshold and less than the third threshold to zero.
  • the first threshold may be 10 cm
  • the second threshold may be 80 cm.
  • the following formula is used to process the target front depth image:
  • the purpose of adjusting the aforementioned target front depth image by using formula (4) is to filter out the depth information of noise points, that is, to remove the depth information of the background in the target front depth image.
  • the device for reconstructing the three-dimensional model of the object determines the original area where the target object is located in the target front color image.
  • the device for reconstructing the three-dimensional model of the object to determine the original area of the target object in the target front color image may be: adding the pixel points of the adjusted target front depth image whose pixel value is not zero in the target front color image
  • the area composed of the corresponding pixels serves as the original area (corresponding to the second area).
  • the pixels in the adjusted target front depth image correspond to the pixels in the target front color image one to one.
  • the device for reconstructing the three-dimensional model of the object may determine a rectangular area in the target front color image that includes the original area as an ROI in the target front color image.
  • the device for reconstructing the three-dimensional model of the object determines whether the preset condition is satisfied.
  • the preset condition may be that the identification bit is the first reference value and the original area is not included in the reference area.
  • the original area is not included in the reference area means that at least a part of the original area is not in the reference area. If the original area is not included in the reference area, it is considered that the target object is out of the frame. Before the 3D object model reconstruction device enters the online scanning stage from the preview stage, it is necessary to ensure that the pose of the target object remains unchanged.
  • the purpose of the object 3D model reconstruction device executing 406 is to determine whether the pose of the target object when scanned to obtain the foregoing target front color image is the same as the pose of the target object when scanned to obtain the foregoing front color image of the previous frame. If the pose is different, it indicates that the pose of the target object has changed, so the user needs to be instructed to keep the pose of the target object unchanged.
  • the device for reconstructing the three-dimensional model of the object sets the identification position to the aforementioned second reference value and instructs the user to keep the pose of the target object unchanged.
  • the identification bit is the first reference value and the original area is not included in the reference area, it indicates that the target object is in a proper position when the front color image of the previous frame of the target front color image is obtained by scanning, and The pose when the target object is scanned to obtain the front color image of the previous frame is different from the pose when the target front color image is scanned to obtain the target object. Therefore, the user needs to be instructed to keep the pose of the target object unchanged.
  • the device for reconstructing the three-dimensional model of the object can output a voice or a corresponding picture to instruct the user to keep the pose of the target object unchanged.
  • the device for reconstructing the three-dimensional model of the object judges whether the position and size of the original area meet the requirements.
  • the object three-dimensional model reconstruction device determines whether the position and size of the original area meet the requirements. It may be determined that the ratio of the area of the original area to the area of the target front color image is within the target interval, and the target front color image The image of the target object is in the target area of the front color image of the target.
  • the target area may be a rectangular area or a regular polygonal area, and the distance between the center point of the target area and the center point of the target front color image is less than the above-mentioned distance threshold.
  • step 406 is executed before step 408 is executed to ensure the pose of the target object when the front color image of the frame is scanned No change.
  • the device for reconstructing the three-dimensional model of the object sets the identification position to the aforementioned first reference value and determines the bounding box of the target object.
  • the device for reconstructing the three-dimensional model of the object can determine a bounding box of the target object by using the adjusted front depth image of the target. Each time the device for reconstructing the three-dimensional model of the object executes step 409, a bounding box of the target object can be obtained. The device for reconstructing the three-dimensional model of the object can store each bounding box in the determined sequence to obtain a bounding box sequence. The method of determining the bounding box of the target object will be described in detail below, and will not be described in detail here.
  • the device for reconstructing the three-dimensional object model instructs the user to adjust the position of the target object.
  • the object 3D model reconstruction device can output corresponding prompt information according to the current position of the target object to guide the user to place the target object in a suitable area. For example, when the distance between the target object and the object 3D model reconstruction device is not suitable, the distance information is not appropriate; when the image of the target object deviates from the center area of the entire image, it is prompted that the image of the target object is not in the entire image. The central area of the image.
  • the device for reconstructing the three-dimensional model of the object prompts the user to place the target object in a suitable area through a voice or a corresponding interface.
  • the device for reconstructing the three-dimensional model of the object judges whether the bounding box is stable.
  • Judging whether the bounding box is stable can be judging whether the distance between the center points of any two adjacent bounding boxes in the target bounding box sequence is less than the aforementioned first threshold, and the bounding boxes included in the target bounding box sequence are those in the aforementioned bounding box sequence.
  • the last F bounding boxes of, F is an integer greater than 1. That is to say, judging whether the bounding box is stable can be understood as judging that the distance between the center points of any two adjacent bounding boxes in the most recently determined F bounding box is less than the aforementioned first threshold.
  • the number of bounding boxes obtained by performing step 409 is less than F, it is directly determined that the bounding box is unstable. It can be understood that after the device for reconstructing the three-dimensional model of the object determines that the bounding box is stable (that is, the position of the target object remains unchanged), it enters the online scanning stage.
  • the pre-color image and the pre-depth image are preprocessed in the preview stage of the object 3D model reconstruction device, so that every frame of the image acquired by the object 3D model reconstruction device in the scanning stage meets the requirements, thereby improving scanning s efficiency.
  • Step 409 in FIG. 4 involves the process of determining the bounding box of the target object.
  • the operation of the device for reconstructing the three-dimensional model of the object to determine the bounding box of the target object may be as follows:
  • the object three-dimensional model reconstruction device determines the depth binary image based on the adjusted target front depth image.
  • the object 3D model reconstruction device may use the following formula to determine the depth binary image corresponding to the adjusted target front depth image:
  • depth_mask(i,j) represents the value of the pixel in the i-th row and j-th column of the depth binary image.
  • the pixel in the adjusted target front depth image is the same as the pixel in the depth binary image.
  • the object three-dimensional model reconstruction device performs an expansion operation and/or an erosion operation on the above-mentioned depth binary image.
  • Performing the expansion operation on the depth binary map first, and then the corrosion operation can make the area corresponding to the target object in the depth binary map as closed as possible.
  • Corrosion is a process of eliminating boundary points and shrinking the boundary to the inside. Corrosion operations can be used to eliminate small and meaningless objects. Expansion is the process of merging all the background points in contact with the object into the object to expand the boundary outward. The expansion operation can be used to fill holes in objects. Expansion operation and corrosion operation are commonly used technical means in this field, and will not be detailed here.
  • the depth binary image after the expansion operation and the corrosion operation is likely to connect the object area (that is, the area where the image of the target object is located) with other areas, so it is necessary to calculate that each pixel in the depth binary image is in front of the above target Set the difference between the corresponding pixel value in the depth image and the pixel point adjacent to the pixel point (the pixel point in the eight neighborhood of the pixel point) in the target pre-depth image, if the difference is absolute If the value is less than the pixel value threshold, the two pixels are considered to be continuous; otherwise, the two pixels are considered discontinuous, and black is drawn between the pixel and the adjacent pixel on the depth binary image Line, so that the object area is disconnected from other areas (such as face, sleeves, etc.).
  • the pixel value threshold can be 4cm, 5cm, 6cm, 7cm, 8cm, 10cm, etc., which is not limited in this application.
  • the pixel value corresponding to a certain pixel in the depth binary image in the target pre-depth image is the pixel value of the pixel corresponding to the pixel in the target pre-depth image.
  • the device for reconstructing the three-dimensional model of the object determines the visible region binary image included in the above-mentioned depth binary image.
  • the visible area binary image is an area in the P areas included in the depth binary image that corresponds to the smallest average pixel value in the adjusted target front depth image; the pixels in any of the P areas The value of the point is 255, any two of the P areas do not overlap, and P is an integer greater than 0.
  • the average pixel value corresponding to any one of the P areas in the adjusted target front depth image is the average pixel value of each pixel in the corresponding area of the any area in the adjusted target front depth image Value, the pixel points in the any area correspond to the pixel points in the corresponding area of the any area in the adjusted target pre-depth image.
  • the visible area binary image includes at least M pixels, and M can be 100, 200, 300, and so on.
  • the visible area binary image in the depth binary image includes at least 200 pixels.
  • the difference between the pixel values of any two adjacent pixels in the corresponding area in the adjusted target pre-depth image of any area is smaller than the aforementioned pixel value threshold.
  • the object three-dimensional model reconstruction device expands the above-mentioned visible region binary image.
  • the pixels in the expanded visible area binary image correspond to the pixels in the depth binary image one-to-one, and the values of the pixels in the expanded area are all Is 0; the value of the extended pixel in the extended area is set to 255, each pixel in the extended pixel has a corresponding value of 255 in the fifth binary image or the sixth binary image, and each The pixel value corresponding to the pixel point in the target pre-depth image is greater than the sixth threshold and less than the seventh threshold.
  • the sixth threshold is not greater than the smallest pixel value of the visible area binary image in the corresponding area of the adjusted target front depth image
  • the seventh threshold is not less than the visible area binary image in the adjusted target The largest pixel value in the corresponding area in the front depth image.
  • the device for reconstructing the three-dimensional model of the object calculates the three-dimensional space coordinates of each pixel whose pixel value is not zero in the target image area in the target front depth image.
  • the pixels in the target image area correspond to the pixels with the pixel value of 255 in the binary image of the expanded visible area, and the pixels in the target pre-depth image correspond to the pixels in the binary image of the expanded visible area.
  • a pixel in the depth image whose pixel value is not zero its pixel coordinate in the depth image is (u d , v d ), and the pixel value is d.
  • x, y, z are the three-dimensional space coordinates of the pixel, and the pixel coordinates of the pixel are (u d , v d ).
  • the object three-dimensional model reconstruction device determines the three-dimensional space area of the target object according to the three-dimensional space coordinates of each pixel with a non-zero pixel value in the target image area.
  • the three-dimensional space area is expanded, and the finally expanded three-dimensional empty area is used as the object reconstruction area.
  • the object three-dimensional model reconstruction device can reconstruct the three-dimensional model and texture of the object in the extended three-dimensional space area.
  • the three-dimensional space area is expanded according to a certain proportion in the x direction, the y direction, and the z direction.
  • the z direction needs to be expanded more than the x direction and the y direction due to the possibility of visual blind zones.
  • the object three-dimensional model reconstruction device determines the bounding box of the target object according to the three-dimensional space area.
  • the above-mentioned bounding box of the target object may be an AABB bounding box (Axis-aligned bounding box), or a direction bounding box, which is not limited in this application.
  • the device for reconstructing the three-dimensional model of the object can accurately determine the three-dimensional space of the target object, and then obtain the bounding box of the target object.
  • Step 222 in FIG. 2B is to determine the pixels corresponding to the interfering objects in each frame of the color image.
  • the device for reconstructing the three-dimensional model of the object can determine the ROI (that is, the region of the image including the target object) in each frame of the color image according to the bounding box of the target object determined in the preview stage. In this way, the object three-dimensional model reconstruction device only needs to determine the pixel points corresponding to the interfering object in the ROI in each frame of color image, and does not need to determine the pixel points corresponding to the interfering object in the entire color image, thereby improving calculation efficiency.
  • the device for reconstructing the three-dimensional model of the object can obtain a bounding box of the target object in the preview stage, for example, an AABB bounding box, and the bounding box corresponds to 8 vertices.
  • the device for reconstructing the three-dimensional model of the object can determine the ROI in any color image in the following manner: according to the camera pose corresponding to the previous frame of the color image of the any color image, the 8 vertices of the bounding box of the target object are projected into 2D Eight two-dimensional coordinate points are obtained on the image; a smallest rectangular frame including these eight two-dimensional coordinate points is used as the ROI in any color image.
  • the device for reconstructing the three-dimensional model of the object may use the following formula to calculate the two-dimensional coordinate point of any vertex of the bounding box of the target object projected to the 2D image:
  • K d is the camera internal parameters
  • T] is the camera pose corresponding to the previous frame of the color image of any color image
  • [x y z] is any vertex of the bounding box of the target object.
  • the 3D object model reconstruction device can determine a rectangular frame as the next frame of the color image according to the bounding box of the target object obtained in the preview stage and the camera pose corresponding to the currently processed frame of color image. ROI.
  • T] -1 is the identity matrix.
  • the depth camera can include a 3D/depth sensor or a 3D/depth sensor module to obtain the depth information of a static object. It should be understood that the scanned object should theoretically be a stationary object, and in actual operation, slight dynamics are acceptable to a certain extent. Structured light technology and TOF can be used to obtain depth information.
  • FIG. 5 A schematic diagram of structured light is shown in Figure 5, where 501 is an invisible infrared light source, 502 is a grating that generates a certain light pattern, 503 is an object to be scanned (target object), and 504 is an infrared camera.
  • the returned light pattern is compared with the expected light pattern, and the depth information of the scanned part of the target object is obtained through calculation.
  • a TOF depth camera is shown in Figure 6, where 611 is the target object, 612 is the infrared emitting end of the TOF camera, and 613 is the infrared receiving end.
  • 612 emits infrared light (for example but not limited to: 850nm-1200nm)
  • the target object reflects infrared rays, and the reflected infrared rays are received by 613.
  • the sensor of 613 generates a series of voltage difference signals due to the reflected infrared light; the depth calculation unit 614 performs calculations based on the series of voltage difference signals.
  • the depth information 615 of the scanned part of the target object is obtained.
  • the depth camera and the color camera will be called synchronously, and a certain correction algorithm is adopted to make the images of the target object scanned by the two are consistent.
  • the way that a color camera acquires images during scanning is basically the same as that of a normal camera. This section will not repeat them.
  • the object when scanning the target object, it is necessary to scan the object within a certain angle range (usually limited by the smaller of the depth camera or the color camera) and within a certain distance, which is limited by the depth information (For example: the quality of the depth map), usually the distance between the object and the depth camera (or mobile terminal) is between 20 cm and 80 cm.
  • a certain angle range usually limited by the smaller of the depth camera or the color camera
  • a certain distance which is limited by the depth information (For example: the quality of the depth map)
  • the distance between the object and the depth camera (or mobile terminal) is between 20 cm and 80 cm.
  • a specific scanning method can be that the mobile terminal does not move.
  • the user directly holds the target object and places it in the range of 30cm to 80cm in front of the depth camera and keeps it still.
  • the object is slowly rotated in all directions to make all scanned images The union can construct a complete object.
  • the scene information includes the full picture of the object without leaving blind spots. Therefore, in the process of panoramic scanning, there will be multiple depth maps (depth map sequences) corresponding to each depth map. All correspond to the scene within the scan range during one scan; there will also be corresponding multi-frame color images (color image sequence), and each color image corresponds to the scene within the scan range during one scan.
  • Scanning the target object may include interfering objects that are in direct contact with the target object, such as the user's hand.
  • the pixel value of the pixel corresponding to the interfering object in the depth image can be adjusted to remove the depth information of the interfering object in the depth image .
  • the shooting frame rate of the depth camera during the scanning process may be greater than or equal to 25 fps, such as 30 fps, 60 fps, and 120 fps.
  • the mobile terminal can present the scanning progress of the target object, so that the user can observe whether the panorama of the target object has been covered, and the user can independently choose to continue scanning or terminate the scanning.
  • the position of the depth camera and the color camera can be front or rear, there are correspondingly two ways of pre-scanning and post-scanning.
  • the depth camera when the depth camera is located above the front of the mobile phone, it can be used in conjunction with the front color camera, and the front scanning can realize self-portrait scanning; when the depth camera is located above the back of the mobile phone, it can be used in conjunction with the rear color camera, and the rear scanning can be used.
  • the above-mentioned color camera scan and depth camera scan can be turned on when the user triggers the scan function, and the trigger operations include timing, triggering the shutter, gesture operation, space sensing, device manipulation, etc.
  • the system can prompt which objects are suitable for scanning or 3D modeling in the preview image; for example, a box can be used to identify objects in the preview image to prompt the user.
  • the specific device parameters involved in the above-mentioned depth camera and color camera are related to the manufacturing process, user requirements, and terminal design constraints, and there is no specific limitation in the present invention.
  • Step 223 in FIG. 2B involves the reconstruction process of the three-dimensional model of the object, which will be described in detail below.
  • a depth image sequence 701 and a color image sequence 702 will be obtained, where each frame image obtained by the depth camera is a depth image of the scanned scene ( For example: Depth map).
  • each frame of the image obtained by the color camera is a color image of the scanned scene (for example: RGB map); the depth image sequence 701 is meshed to obtain the mesh model of the target object.
  • Texture mapping is performed on the mesh model according to the color image sequence 702 to obtain a mesh model 703 after texture mapping, that is, a 3D model of the object.
  • texture mapping can also be performed on all frames or a few frames in the color sequence image.
  • Step 331 Acquire a color image (including but not limited to RGB) and a depth image (Depth) of the target object in each scanned scene.
  • Depth Map depth image
  • Depth Map contains information about the distance between multiple points on the surface of the target object and the depth camera.
  • Depth Map is similar to a grayscale image, except that one pixel value represents the actual distance between the depth camera and a point on the surface of the target object.
  • the color image and the Depth image are registered.
  • Step 332 includes, but is not limited to, performing step 222 to adjust at least one frame of the depth image in the depth image sequence, convert the depth image into a 3D point cloud, estimate the normal vector of each vertex, and crop out the scanned object point.
  • Step 333 Determine the pose transformation relationship of the target object between different scanning positions.
  • the depth image sequence and color image sequence of the target object at different scanning positions are collected.
  • Pose estimation is to estimate the 3D pose of an object based on an image sequence.
  • registration based on feature points There are two ideas for pose estimation: registration based on feature points and registration based on point clouds.
  • the precise registration based on the point cloud is used, for example, the iterative nearest neighbor point algorithm (ICP) is used to estimate the object pose.
  • ICP iterative nearest neighbor point algorithm
  • Step 334 Convert the 2D depth image into a 3D point cloud and merge it into a unified 3D voxel model.
  • the truncated signed distance function (Truncated Signed Distance Function, TSDF) algorithm is used to fuse the above 3D point clouds.
  • the voxel values are SDF (Signed Distance Function) value and Weight (weight) Value, and optional color value.
  • the TSDF algorithm is currently the mainstream processing algorithm for 3D point cloud fusion.
  • the weight calculation uses the method of finding the average value. Each time the fusion is performed, the old weight value is increased by one. The new value has a weight of 1.
  • the new and old SDF values are multiplied by their weights, added, and then divided by the number of fusions (new weight values) to obtain the new normalized SDF value.
  • Step 335 Determine whether there is a preset number of key frames saved at intervals of a certain angle (for example, but not limited to preset angles such as 30, 45, 60, 90, etc.) in the three directions of Roll/Yaw/Pitch, such as saved key frames
  • a preset number of key frames saved at intervals of a certain angle for example, but not limited to preset angles such as 30, 45, 60, 90, etc.
  • the number is less than the preset number (according to whether the panorama of the target object is covered or not)
  • the mobile terminal will instruct the user to perform more scans.
  • the number of key frames is sufficient to cover the panoramic view of the target object, the user is prompted that the scan is complete, and the scan can be ended and proceed to the subsequent steps.
  • Step 336 In the process of real-time fusion, select and cache input key frame information required for texture mapping, including information such as color images, poses (position and pose differences between different images).
  • the preset number (F) key frames are selected for each of the Roll/Yaw/Pitch directions, and the 360-degree texture of the object can be completely restored.
  • ICP results determine the angle of each frame in the input image stream (YAW/Pitch/Roll), then calculate the definition of each frame, construct a selection strategy based on the angle and definition, and select key frames.
  • the angle strategy is to divide 360 degrees into F 360/F areas in different directions, and each area must have a clear color image.
  • the principle of sharpness detection Generally, there are gradient method and Sobel operator to evaluate the sharpness of the image. You can choose the gradient method to calculate the sharpness.
  • Step 337 Use the Marching Cubes algorithm to achieve 3D point cloud meshing, and generate triangular patches.
  • the main idea of the Marching Cubes algorithm is to find the boundary between the content part and the background part of the 3D point cloud in the unit of voxel. Triangular slices are extracted from the voxel to fit this boundary. In simple terms, it can be said that the volume contains the content of the volume data.
  • the voxel points are called real points, and the background voxel points outside are called virtual points.
  • Such a three-dimensional point cloud is a lattice composed of various real points and virtual points.
  • each of the 8 voxels of a voxel may be a real point or an imaginary point, then a voxel has a total of 2 to the 8th power, that is, 256 possible situations.
  • the core idea of Marching Cubes algorithm is to use these 256 enumerable situations to extract iso-triangular patches in voxels.
  • a voxel is a cube grid composed of eight adjacent voxel points in a three-dimensional image.
  • the semantics of the Cube of the MarchingCubes algorithm can also refer to this voxel. Note the distinction between voxels and voxels.
  • a voxel is a grid of 8 voxels, and each voxel (except for the boundary) is shared by the 8 voxels.
  • Step 341 According to the mesh model (triangular patch information) and the pose information of the key frame, it is determined whether all the triangle faces are visible under the pose of each key frame.
  • the input to the mesh model is the information of all triangles and the space coordinates of the key frames, and the output is the information about whether all the triangles are visible under the pose of each key frame.
  • Step 342 According to the result of step 341 and the grid model, the method of region division and graph cut (Graph Cuts) is adopted to mark each face on the grid model, and determine which key frame image (view ) To generate textures.
  • the result of patch labeling can be used to generate a preliminary texture map.
  • Step 343 Map the texture of the corresponding area in the key frame image to the texture map, and perform boundary smoothing on patches (seam patches) of different key frames.
  • the result of the patch labeling is saved as a patch with the same label in the adjacent area, the vertices of all patches are smoothed, the pixel value of each vertex is adjusted, and the triangle line of the final vertex surrounded by Based on the position and pixel affine transformation, the final texture map is formed.
  • an embodiment of the present invention provides an object three-dimensional model reconstruction device 1100.
  • the device 1100 includes: a determination module 1101, a depth map processing module 1102, and a model reconstruction module 1103.
  • the device may be a mobile terminal.
  • the mobile terminal includes a color camera and a depth camera. The color camera and the depth camera are located on the same side of the mobile terminal. For related characteristics, refer to the description in the foregoing method embodiment.
  • the determining module 1100 is configured to determine the first color image and the first depth image of the target object, transmit the first color image to the model reconstruction module 1103, and transmit the first depth image to the depth map processing module 1102.
  • the target object includes a target object and an interfering object; the pixels in the first color image correspond to the pixels in the first depth image in a one-to-one correspondence;
  • the determining module 1101 is further configured to determine the color interference pixel points corresponding to the interference object in the first color image, and send first description information to the depth map processing module; the first description information is used to describe the first color image The coordinates of the aforementioned color interference pixels in
  • the depth map processing module 1102 is configured to adjust the pixel value of the depth interference pixel in the first depth image according to the first description information to obtain a processed first depth image, and the depth interference pixel is in the first depth image One-to-one correspondence with the aforementioned color interference pixels;
  • the model reconstruction module 1103 is configured to construct a three-dimensional model of the target object according to the first color image and the processed first depth image.
  • the foregoing device further includes:
  • the acquisition module 1104 is specifically used to acquire a color image sequence and a depth image sequence, transmit the color image sequence to the determination module 1101 and the model reconstruction module 1103, and transmit the depth image sequence to the depth map processing module 1102;
  • the color image sequence It includes multi-frame color images of the target object in multiple poses, and the depth image sequence includes multiple-frame depth images of the target object in the multiple poses;
  • the first depth image is the multi-frame depth image Any frame of image in the above, the first color image is an image corresponding to the first depth image among the multiple color images;
  • the determining module 1101 is specifically configured to determine the color interference pixel corresponding to the interference object in each frame of the color image, and send the second description information to the depth map processing module; the second description information is used to describe each frame of the color image The coordinates of the pixels corresponding to the above-mentioned interfering objects;
  • the depth map processing module 1102 is used to adjust the pixel value of the depth interference pixel in the depth image of each frame according to the second description information to obtain the processed depth image sequence; the pixel point corresponding to the interference object in the color image is The corresponding pixel in the depth image corresponding to the color image is the depth interference pixel in the depth image;
  • the model reconstruction module 1103 is specifically configured to construct a three-dimensional model of the target object according to multiple frames of the first color image and multiple frames of the processed first depth image.
  • the foregoing device further includes:
  • the scanning module 1105 is configured to scan the target object to obtain a front color image sequence and a front depth image sequence, and transmit the front depth color image sequence and the front depth image sequence to the determination module; the front depth image
  • the front depth image in the sequence corresponds to the front color image in the preceding color image sequence
  • the pixels in the front depth image correspond to the pixels in the front color image corresponding to the front depth image.
  • the scanning module 1105 is also used for determining in the determining module 1101 that the ratio of the area of the image of the target object in each frame of the front color image to the area of the front color image is within the target interval, and the front color image
  • the target object is scanned to obtain the color image sequence and the Depth image sequence; the target area is an area including the center point of the front color image; the target area is an area including the center point of the front color image.
  • the above-mentioned device further includes: an indication module 1106 for determining, in the above-mentioned determining module, the area and the area where the image of the above-mentioned target object in each frame of the above-mentioned front color image is located The ratio of the area of the front color image is in the target interval, and the image of the target object in the front color image is in the target area of the front color image, and the displacement of the target object is determined to be less than In the case of the first threshold, instruct the user to adjust the posture of the target object multiple times;
  • the scanning module 1105 is specifically configured to scan the target object to obtain the color image sequence and the depth image sequence when the user adjusts the posture of the target object multiple times.
  • the apparatus 1100 provided in the embodiment of the present invention can implement operations from image preprocessing (that is, preview stage) to online scanning, and then to 3D model reconstruction.
  • image preprocessing that is, preview stage
  • the user can hold the target object to change the position of the target object, so that the mobile terminal can collect color images and depth images that meet the requirements
  • the online scanning stage the user holds the target object to adjust the posture of the target object. So that the mobile terminal scans the images of different parts of the target object to obtain the color image sequence and the depth image sequence
  • the post-processing stage the mobile terminal adjusts the depth image sequence and determines according to the color image sequence and the adjusted depth image sequence
  • the three-dimensional model of the target object In the solution of the present application, the user can hold an object for scanning, which has high operation efficiency and can improve user experience.
  • each of the above modules can be separately set up processing elements, or they can be integrated in a certain chip of the terminal for implementation.
  • they can also be stored in the storage element of the controller in the form of program codes and processed by a certain processor.
  • the component calls and executes the functions of the above modules.
  • various modules can be integrated together or implemented independently.
  • the above-mentioned processing element here may be an integrated circuit chip with signal processing capability.
  • each step of the above method or each of the above modules can be completed by hardware integrated logic circuits in the processor element or instructions in the form of software.
  • the processing element can be a general-purpose processor, such as a central processing unit (English: central processing unit, CPU for short), or one or more integrated circuits configured to implement the above methods, such as one or more specific integrated circuits. Circuit (English: application-specific integrated circuit, abbreviation: ASIC), or, one or more microprocessors (English: digital signal processor, abbreviation: DSP), or, one or more field programmable gate arrays (English: field-programmable gate array, referred to as FPGA), etc.
  • ASIC application-specific integrated circuit
  • DSP digital signal processor
  • FPGA field-programmable gate array
  • the device 12 is another object 3D model reconstruction device provided by an embodiment of the application.
  • the device includes: a processor 1201, a depth sensor module 1202, a color camera 1203, and a bus 1204; a depth sensor module 1202, a color camera 1203, and processing
  • the device 1201 is connected by a bus 1204; the depth sensor module 1202 and the color camera 1203223 are used to perform a panoramic scan of the target object under the control of the processor 1202; the above-mentioned processor is used to control the object 3D model reconstruction device to perform the following operations: obtain a color image sequence And a depth image sequence; the color image sequence includes color images of the target object in multiple poses, the depth image sequence includes depth images of the target object in the multiple poses, and the target object includes a target object and an interfering object; The depth images in the depth image sequence correspond to the color images in the color image sequence, and the pixels in the depth image correspond to the pixels in the color image corresponding to the depth image; determine each frame of the color image The pixel points corresponding to
  • Fig. 13 is a mobile terminal provided by an embodiment of the application.
  • the mobile terminal includes a processor 1301, a memory 1302, a depth sensor module 1303, a color camera 1304, and a bus 1305; the color camera 1304 and the depth sensor module 1303 are located in the mobile terminal.
  • the memory 1302 is used to store computer programs and instructions; the processor 1301 is used to call the computer programs and instructions stored in the memory 1302 to enable the mobile terminal to perform the following operations: obtain a color image sequence and a depth image sequence; the color image sequence includes the target object Color images in multiple poses, the depth image sequence includes depth images of the target object in the multiple poses, the target object includes target objects and interfering objects; the depth image in the depth image sequence and the color image sequence One-to-one correspondence between the color images in the above-mentioned depth image and the pixels in the above-mentioned color image corresponding to the above-mentioned depth image; determine the pixel points corresponding to the interference object in each frame of the above-mentioned color image; adjust each frame The pixel values of the
  • the processor 1301 can realize the functions of the acquiring module 1101, the determining module 1102, the depth map processing module 1103, the model reconstruction module 1104, and the indicating module 1106; the depth sensor module 1303 and the color camera 1304 can realize the functions of the scanning module 1105.
  • the processor 1301 may control a display device or an audio device to implement the function of the indicator module 1106.
  • An embodiment of the present application also provides a computer-readable storage medium.
  • the computer-readable storage medium stores a computer program.
  • the computer program includes software program instructions. When the program instructions are executed by a processor, the program instructions are implemented to obtain color image sequence and depth.
  • Image sequence the color image sequence includes color images of the target object in multiple poses
  • the depth image sequence includes the depth image of the target object in the multiple poses
  • the target object includes the target object and the interfering object
  • the depth image The depth images in the sequence correspond to the color images in the color image sequence
  • the pixels in the depth image correspond to the pixels in the color image corresponding to the depth image
  • the pixel corresponding to the interference object adjust the pixel value of the interference pixel in the above-mentioned depth image of each frame to obtain the processed depth image sequence
  • the pixel corresponding to the interference object in the above-mentioned color image is at the above-mentioned depth corresponding to the above-mentioned color image
  • the embodiments of the present invention may be provided as methods, systems, or computer program products. Therefore, the present invention may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present invention may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.
  • a computer-usable storage media including but not limited to disk storage, CD-ROM, optical storage, etc.
  • the present invention is described according to the flowcharts and/or block diagrams of the method, device (system), and computer program product of the embodiments of the present invention. It should be understood that each process and/or block in the flowchart and/or block diagram, and the combination of processes and/or blocks in the flowchart and/or block diagram can be implemented by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing equipment to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing equipment are generated It is a device that realizes the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • These computer program instructions can also be stored in a computer-readable memory that can guide a computer or other programmable data processing equipment to work in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture including the instruction device.
  • the device implements the functions specified in one process or multiple processes in the flowchart and/or one block or multiple blocks in the block diagram.
  • These computer program instructions can also be loaded on a computer or other programmable data processing equipment, so that a series of operation steps are executed on the computer or other programmable equipment to produce computer-implemented processing, so as to execute on the computer or other programmable equipment.
  • the instructions provide steps for implementing functions specified in a flow or multiple flows in the flowchart and/or a block or multiple blocks in the block diagram.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Computer Graphics (AREA)
  • Geometry (AREA)
  • Software Systems (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Processing Or Creating Images (AREA)

Abstract

L'invention concerne un procédé de reconstruction de modèle tridimensionnel d'objet dans le domaine de la vision par ordinateur. Le procédé consiste à : déterminer une première image couleur et une première image de profondeur d'une cible, la cible comprenant un objet cible et un objet distracteur, des points de pixel dans la première image de couleur ayant une correspondance biunivoque avec des points de pixel dans la première image de profondeur ; déterminer des points de pixel de distraction de couleur dans la première image de couleur correspondant à l'objet de distracteur ; ajuster des valeurs de pixel de points de pixel de distraction de profondeur dans la première image de profondeur pour obtenir une première image de profondeur traitée, les points de pixel de distraction de profondeur étant des points de pixel dans la première image de profondeur ayant une correspondance biunivoque avec les points de pixel de distraction de couleur ; et construire un modèle tridimensionnel de l'objet cible en fonction de la première image de couleur et de la première image de profondeur traitée. Dans la présente invention, un utilisateur peut directement tenir l'objet cible à la main pour mettre en œuvre un balayage, ce qui permet de construire un modèle tridimensionnel de l'objet cible. Le procédé présente des opérations simples et une efficacité de balayage élevée, et améliore l'expérience de l'utilisateur.
PCT/CN2020/081222 2019-03-25 2020-03-25 Système et procédé de reconstruction de modèle tridimensionnel d'objet WO2020192706A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910235504.4A CN111739146A (zh) 2019-03-25 2019-03-25 物体三维模型重建方法及装置
CN201910235504.4 2019-03-25

Publications (1)

Publication Number Publication Date
WO2020192706A1 true WO2020192706A1 (fr) 2020-10-01

Family

ID=72609601

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/081222 WO2020192706A1 (fr) 2019-03-25 2020-03-25 Système et procédé de reconstruction de modèle tridimensionnel d'objet

Country Status (2)

Country Link
CN (1) CN111739146A (fr)
WO (1) WO2020192706A1 (fr)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112967381A (zh) * 2021-03-05 2021-06-15 北京百度网讯科技有限公司 三维重建方法、设备和介质
CN113199479A (zh) * 2021-05-11 2021-08-03 梅卡曼德（北京）机器人科技有限公司 轨迹生成方法、装置、电子设备、存储介质和3d相机
US20220051372A1 (en) * 2020-08-12 2022-02-17 Niantic, Inc. Feature matching using features extracted from perspective corrected image
CN115965758A (zh) * 2022-12-28 2023-04-14 无锡东如科技有限公司 一种图协同单目实例三维重建方法

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112926543A (zh) * 2021-04-09 2021-06-08 商汤集团有限公司 图像生成、三维模型生成方法、装置、电子设备及介质
CN113290863B (zh) * 2021-04-23 2022-10-14 湖南华曙高科技股份有限公司 用于增材制造零件模型的处理方法、装置和计算机设备
CN113643421B (zh) * 2021-07-06 2023-08-25 北京航空航天大学 图像的三维重建方法和三维重建装置
CN113507627B (zh) * 2021-07-08 2022-03-25 北京的卢深视科技有限公司 视频生成方法、装置、电子设备及存储介质
CN113570701B (zh) * 2021-07-13 2023-10-24 聚好看科技股份有限公司 一种头发重建方法及设备
TWI778756B (zh) * 2021-08-20 2022-09-21 財團法人資訊工業策進會 三維邊界框重建方法、三維邊界框重建系統及非暫態電腦可讀取媒體
WO2023070555A1 (fr) * 2021-10-29 2023-05-04 华为技术有限公司 Procédé d'obtention de valeur de vérité de profondeur et appareil associé
CN113902875A (zh) * 2021-11-23 2022-01-07 北京智创华科半导体研究院有限公司 目标物体的多视影像全局密集匹配三维重建方法及装置
CN114322768B (zh) * 2021-12-29 2022-08-26 成都翰瑞威自动化测控设备有限责任公司 一种移动式接触网几何参数测量系统、方法及装置
CN115205485B (zh) * 2022-06-30 2023-03-24 广州极点三维信息科技有限公司 基于包围盒的三维空间探测扩充方法、系统、装置及介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103597516A (zh) * 2011-06-06 2014-02-19 微软公司 控制虚拟环境中的对象
CN104599314A (zh) * 2014-06-12 2015-05-06 深圳奥比中光科技有限公司 三维模型重建方法与系统
CN106997605A (zh) * 2017-03-22 2017-08-01 浙江大学 一种通过智能手机采集脚型视频和传感器数据获取三维脚型的方法
CN108053435A (zh) * 2017-11-29 2018-05-18 深圳奥比中光科技有限公司 基于手持移动设备的动态实时三维重建方法和系统
CN108805979A (zh) * 2018-02-05 2018-11-13 清华-伯克利深圳学院筹备办公室 一种动态模型三维重建方法、装置、设备和存储介质

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US10311648B2 (en) * 2016-06-22 2019-06-04 Aquifi, Inc. Systems and methods for scanning three-dimensional objects
US10055882B2 (en) * 2016-08-15 2018-08-21 Aquifi, Inc. System and method for three-dimensional scanning and for capturing a bidirectional reflectance distribution function
CN112132881A (zh) * 2016-12-12 2020-12-25 华为技术有限公司 一种动态三维图像获取的方法和设备
CN108269280A (zh) * 2018-01-05 2018-07-10 厦门美图之家科技有限公司 一种深度图像的处理方法及移动终端

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103597516A (zh) * 2011-06-06 2014-02-19 微软公司 控制虚拟环境中的对象
CN104599314A (zh) * 2014-06-12 2015-05-06 深圳奥比中光科技有限公司 三维模型重建方法与系统
CN106997605A (zh) * 2017-03-22 2017-08-01 浙江大学 一种通过智能手机采集脚型视频和传感器数据获取三维脚型的方法
CN108053435A (zh) * 2017-11-29 2018-05-18 深圳奥比中光科技有限公司 基于手持移动设备的动态实时三维重建方法和系统
CN108805979A (zh) * 2018-02-05 2018-11-13 清华-伯克利深圳学院筹备办公室 一种动态模型三维重建方法、装置、设备和存储介质

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
OLEG MURATOV, YURY SLYNKO, VITALY CHERNOV, MARIA LYUBIMTSEVA, ARTEM SHAMSUAROV, VICTOR BUCHA: "3DCapture: 3D Reconstruction for a Smartphone", 2016 IEEE CONFERENCE ON COMPUTER VISION AND PATTERN RECOGNITION WORKSHOPS (CVPRW), 19 December 2016 (2016-12-19), pages 75 - 82, XP055737176, ISSN: 2160-7516, DOI: 10.1109/CVPRW.2016.116 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20220051372A1 (en) * 2020-08-12 2022-02-17 Niantic, Inc. Feature matching using features extracted from perspective corrected image
US12008740B2 (en) * 2020-08-12 2024-06-11 Niantic, Inc. Feature matching using features extracted from perspective corrected image
CN112967381A (zh) * 2021-03-05 2021-06-15 北京百度网讯科技有限公司 三维重建方法、设备和介质
CN112967381B (zh) * 2021-03-05 2024-01-16 北京百度网讯科技有限公司 三维重建方法、设备和介质
CN113199479A (zh) * 2021-05-11 2021-08-03 梅卡曼德（北京）机器人科技有限公司 轨迹生成方法、装置、电子设备、存储介质和3d相机
CN115965758A (zh) * 2022-12-28 2023-04-14 无锡东如科技有限公司 一种图协同单目实例三维重建方法

Also Published As

Publication number Publication date
CN111739146A (zh) 2020-10-02

Similar Documents

Publication Publication Date Title
WO2020192706A1 (fr) Système et procédé de reconstruction de modèle tridimensionnel d'objet
JP7176012B2 (ja) オブジェクト・モデリング動作方法及び装置並びにデバイス
CN109978936B (zh) 视差图获取方法、装置、存储介质及设备
US9269003B2 (en) Diminished and mediated reality effects from reconstruction
CN112639664A (zh) 用于确定和/或评价图像显示设备的定位地图的方法和装置
JP7387202B2 (ja) ３次元顔モデル生成方法、装置、コンピュータデバイス及びコンピュータプログラム
US20110148868A1 (en) Apparatus and method for reconstructing three-dimensional face avatar through stereo vision and face detection
KR102455468B1 (ko) 객체의 3차원 모델을 복원하는 방법 및 장치
CN112784621B (zh) 图像显示方法及设备
JP2010109783A (ja) 電子カメラ
US10354399B2 (en) Multi-view back-projection to a light-field
US11908096B2 (en) Stereoscopic image acquisition method, electronic device and storage medium
CN115997379A (zh) 用于立体渲染的图像fov的复原
CN114241127A (zh) 全景图像生成方法、装置、电子设备和介质
EP4258221A2 (fr) Appareil de traitement d'image, procédé de traitement d'image et programme
US20230291865A1 (en) Image processing apparatus, image processing method, and storage medium
US20230326136A1 (en) Apparatus and methods for image reconstruction using machine learning processes
US20230072247A1 (en) Method and device for processing image content
WO2021120052A1 (fr) Reconstruction 3d à partir d'un nombre insuffisant d'images
CN116778075A (zh) 一种三维模型重建的方法以及装置
WO2023129855A1 (fr) Systèmes et procédés de reprojection d'image
CN118096604A (zh) 人脸图像的校正方法、人脸图像的校正装置、终端及存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20778214

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20778214

Country of ref document: EP

Kind code of ref document: A1