WO2022019128A1 - Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement lisible par ordinateur - Google Patents

Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement lisible par ordinateur Download PDF

Info

Publication number
WO2022019128A1
WO2022019128A1 PCT/JP2021/025736 JP2021025736W WO2022019128A1 WO 2022019128 A1 WO2022019128 A1 WO 2022019128A1 JP 2021025736 W JP2021025736 W JP 2021025736W WO 2022019128 A1 WO2022019128 A1 WO 2022019128A1
Authority
WO
WIPO (PCT)
Prior art keywords
value
information processing
depth
depth value
target portion
Prior art date
Application number
PCT/JP2021/025736
Other languages
English (en)
Japanese (ja)
Inventor
宏基 水野
Original Assignee
ソニーグループ株式会社
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by ソニーグループ株式会社 filed Critical ソニーグループ株式会社
Publication of WO2022019128A1 publication Critical patent/WO2022019128A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
    • G01B11/00—Measuring arrangements characterised by the use of optical techniques
    • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures

Definitions

  • Non-Patent Document 1 describes a method of generating a target three-dimensional model based on depth information.
  • the distance value to the surface of the target is calculated for each position (voxel) in the space based on the distance image mapping the depth to the target.
  • a three-dimensional model is generated by connecting the positions where the distance value becomes 0. This makes it possible to accurately restore the shape of the target even if the depth detection accuracy is uneven (pages 3-4 of Non-Patent Document 1, FIGS. 2, 5, etc.). ..
  • the technology for detecting 3D shapes in this way has become more familiar due to the miniaturization of sensors, etc., and is expected to be applied in various scenes. Therefore, there is a demand for a technique capable of detecting a three-dimensional shape with high accuracy.
  • the information processing device includes an acquisition unit and a setting unit.
  • the acquisition unit acquires the depth value of the target portion of the real object irradiated with the irradiation light by the ToF method.
  • a computer-readable recording medium records a program that causes a computer system to perform the following steps.
  • the outward-facing camera 12 is a monocular RGB camera capable of capturing color moving images and still images.
  • the image taken by the outward-facing camera 12 is output to the display 10 in real time, for example. This enables the user to shoot an object while checking the state of the image to be shot.
  • a digital camera including an image sensor such as a CMOS (Complementary Metal-Oxide Semiconductor) sensor or a CCD (Charge Coupled Device) sensor can be used. In addition, any configuration may be adopted.
  • CMOS Complementary Metal-Oxide Semiconductor
  • CCD Charge Coupled Device
  • the 3D model 2 of the object 1 is generated based on the depth map acquired while the user moves around the object 1. More specifically, volume data representing the position of the surface of the object 1 is generated for each depth map. Then, the 3D model 2 is generated based on the data (integrated volume data) in which each volume data is integrated. This point will be described in detail later.
  • the microphone 16 is a sound collecting element that detects voice around the mobile terminal 100.
  • the speaker 17 is a reproduction element that reproduces voice or the like output from the mobile terminal 100. As shown in FIG. 1A, the microphone 16 is arranged below the front surface 13 of the mobile terminal 100. The speaker 17 is arranged on the upper side of the front surface 13 of the mobile terminal 100.
  • the communication unit 18 is a module for executing network communication, short-range wireless communication, and the like with other devices. For example, a wireless LAN module such as WiFi and a communication module such as Bluetooth (registered trademark) are provided.
  • the specific configuration of the microphone 16, the speaker 17, and the communication unit 18 is not limited.
  • the data acquisition unit 31 generates various data from the output of each sensor (ToF camera 11 or the like) provided in the mobile terminal 100. As shown in FIG. 3, the data acquisition unit 31 includes a shooting parameter acquisition unit 38, a depth map acquisition unit 39, and an infrared image acquisition unit 40.
  • the depth map acquisition unit 39 acquires the depth value of the target portion of the object 1 irradiated with infrared light by the ToF method. Specifically, the depth value of the target portion is calculated based on the output of the ToF camera 11.
  • the target portion is a portion to be measured for depth (depth value), and is, for example, a portion on the surface of the object 1 irradiated with infrared light.
  • the ToF camera 11 measures the round-trip flight time of the infrared light with respect to the target portion by detecting the infrared light reflected at the target portion.
  • the depth map acquisition unit 39 calculates the depth value of the target portion by using the data indicating the flight time and the speed of light.
  • the infrared image acquisition unit 40 generates an infrared image (IR image) of the object 1 based on the output of the ToF camera 11.
  • the ToF camera 11 measures the brightness (intensity) of the infrared light in, for example, a fixed exposure time when detecting the infrared light reflected by the target portion.
  • the brightness of the infrared light represents the brightness of the infrared light reflected at the target portion, and is, for example, a value corresponding to the reflection characteristic at the target portion.
  • the depth map acquisition unit 39 generates an infrared image of the object 1 based on the data indicating the brightness. That is, the infrared image is data in which the reflection characteristics of the target portion, which are different for each pixel, are recorded.
  • the process of generating an infrared image is always executed at a predetermined frame rate at the same time as the depth map, for example.
  • the reflection intensity is low in a portion where the incident angle of infrared light is shallow (that is, a portion where infrared light is incident along the surface) such as the side surface of the nose or the contour of the face.
  • the reflection intensity is high in the portion where the incident angle of the infrared light is deep (that is, the portion where the infrared light is incident at an angle close to perpendicular to the surface).
  • the infrared image 4 records the reflection intensity according to the shape of the object 1.
  • such an infrared image 4 is generated by the infrared image acquisition unit 40 described above.
  • the standard deviation ⁇ of the depth value becomes small. That is, the brighter the reflected infrared light, the lower the fluctuation (noise level) of the depth value tends to be.
  • the standard deviation ⁇ of the depth value tends to increase sharply, and the fluctuation of the depth value (noise level) tends to increase.
  • the TSDF calculation unit 44 sets the position coordinates of the voxel 7 as v, sets the distance between the voxel 7 and the target portion P in the detection direction of the depth value as D (v), and sets the voxel 7.
  • the weight value W (v) is set according to the equation (3), where W (v) is the weight value, the noise level of the depth value of the target portion P is ⁇ , and the preset adjustment coefficient is ⁇ .
  • the distribution of the weight value W (v) is not limited to the normal distribution, and any distribution having the maximum value at the target site P can be used.
  • FIG. 16 is a diagram showing an example of generating a 3D model.
  • FIG. 16A shows a 3D model 2 of a person's face generated by adjusting the width of the distribution of the weight value W (v) by applying the present technique.
  • FIG. 16B shows a 3D model 2 of a person's face generated without adjusting the width of the distribution of the weight value W (v).
  • the object 1 is arranged in the photographing area 201 surrounded by the ToF camera 11. That is, the photographing system 200 is a system that surrounds the object 1 with a plurality of ToF cameras 211 and photographs the object 1 from various directions.
  • the depth value is calculated based on the outputs of a plurality of ToF cameras 211 provided so as to surround the object 1.
  • the object 1 is simultaneously photographed by a plurality of ToF cameras 211, and the depth value is calculated for each pixel based on the output of each camera. This makes it possible to generate a plurality of depth maps 3 at once. Therefore, in the photographing system 200, even when the object 1 is moving, it is possible to accurately measure the shape at each moment.
  • Volume data 6 (position parameter F (v) and weight value W (v)) are generated for each of these depth maps 3. At this time, shooting parameters such as the position and posture of each ToF camera 211 are appropriately referred to. The shooting parameters are acquired, for example, by pre-calibration.
  • the weight value W (v) for example, the width of the distribution of the weight value W (v) is set according to the noise level calculated from the infrared image 4. in this way.
  • a ToF camera capable of acquiring an infrared image together with a depth value has been described as an example.
  • an infrared image cannot be acquired, it is possible to estimate the noise level of the depth value by using an image of another camera or the like.
  • a color camera outward facing camera, etc.
  • it can be used as a substitute for the brightness value of infrared light based on the image taken by the color camera. be.
  • the information processing method according to the present technology is executed by a computer such as a mobile terminal operated by the user has been described.
  • the information processing method and the program according to the present technology may be executed by a computer operated by the user and another computer capable of communicating via a network or the like.
  • the information processing method and program according to the present technology can be executed not only in a computer system composed of a single computer but also in a computer system in which a plurality of computers operate in conjunction with each other.
  • the system means a set of a plurality of components (devices, modules (parts), etc.), and it does not matter whether or not all the components are in the same housing. Therefore, a plurality of devices housed in separate housings and connected via a network, and one device in which a plurality of modules are housed in one housing are both systems.
  • the information processing method and program execution related to this technology by a computer system are, for example, acquisition of a depth value and setting of a weight value based on the target part regarding the distance parameter to the target part based on the depth value. Includes both when performed by one computer and when each process is performed by a different computer. Further, the execution of each process by a predetermined computer includes having another computer execute a part or all of the process and acquiring the result.
  • the setting unit is an information processing device that sets the weight values for each of the plurality of voxels so that the distribution of the weight values becomes a normal distribution that peaks at the target site.
  • the information processing apparatus according to any one of (3) to (5).
  • the irradiation light is infrared light, and is In the setting unit, the noise level of the depth value of the target portion is ⁇ , the brightness value of the infrared light reflected by the target portion is I, the first coefficient set in advance is A, and the preset portion is set in advance. With the set second coefficient as B, the noise level is calculated according to the following equation (1).
  • Information processing equipment (7) The information processing apparatus according to (6).
  • the information processing apparatus includes any one of (3) to (8).
  • the distance parameter is a distance obtained by normalizing the distance between the voxel and the target site in the detection direction of the depth value with a threshold value.
  • the model generation unit is an information processing device that calculates the distance parameter for the voxel in which the distance between the voxel and the target portion in the detection direction of the depth value is equal to or less than the threshold value.
  • the position coordinates of the voxel are v
  • the distance between the voxel and the target portion in the detection direction of the depth value is D (v)
  • the weight value set in the voxel is W (v).
  • the noise level of the depth value of the target portion is ⁇
  • the preset adjustment coefficient is ⁇
  • the weight value is set according to the following equation (2).
  • Information processing equipment (11) The information processing apparatus according to any one of (1) to (10).
  • the acquisition unit calculates the depth value based on the output of the ToF sensor that detects the irradiation light reflected by the target portion.

Abstract

L'invention concerne un dispositif de traitement d'informations qui, selon un aspect de la présente technologie, comprend une unité d'acquisition et une unité de définition. L'unité d'acquisition acquiert, par un procédé de ToF, une valeur de profondeur pour un site d'intérêt d'un objet réel irradié avec une lumière de rayonnement. L'unité de définition définit la largeur d'une distribution de valeurs de pondération ayant le site d'intérêt comme référence de façon à ce qu'elle soit inférieure, par rapport à un paramètre pour la distance jusqu'au site d'intérêt en fonction de la valeur de profondeur, à mesure qu'une valeur de luminosité de la lumière de rayonnement réfléchie par le site d'intérêt augmente.
PCT/JP2021/025736 2020-07-21 2021-07-08 Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement lisible par ordinateur WO2022019128A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2020-124369 2020-07-21
JP2020124369A JP2022021027A (ja) 2020-07-21 2020-07-21 情報処理装置、情報処理方法、及びコンピュータが読み取り可能な記録媒体

Publications (1)

Publication Number Publication Date
WO2022019128A1 true WO2022019128A1 (fr) 2022-01-27

Family

ID=79728743

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2021/025736 WO2022019128A1 (fr) 2020-07-21 2021-07-08 Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement lisible par ordinateur

Country Status (2)

Country Link
JP (1) JP2022021027A (fr)
WO (1) WO2022019128A1 (fr)

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011123071A (ja) * 2009-12-10 2011-06-23 Honda Motor Co Ltd 撮像装置、オクルージョン領域の探索方法、及びプログラム
JP2012078942A (ja) * 2010-09-30 2012-04-19 Toshiba Corp デプス補正装置及び方法
JP5906258B2 (ja) * 2011-01-31 2016-04-20 マイクロソフト テクノロジー ライセンシング，エルエルシー 三次元環境の再現
WO2018216341A1 (fr) * 2017-05-22 2018-11-29 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2011123071A (ja) * 2009-12-10 2011-06-23 Honda Motor Co Ltd 撮像装置、オクルージョン領域の探索方法、及びプログラム
JP2012078942A (ja) * 2010-09-30 2012-04-19 Toshiba Corp デプス補正装置及び方法
JP5906258B2 (ja) * 2011-01-31 2016-04-20 マイクロソフト テクノロジー ライセンシング，エルエルシー 三次元環境の再現
WO2018216341A1 (fr) * 2017-05-22 2018-11-29 ソニー株式会社 Dispositif de traitement d'informations, procédé de traitement d'informations et programme

Also Published As

Publication number Publication date
JP2022021027A (ja) 2022-02-02

Similar Documents

Publication Publication Date Title
US11210804B2 (en) Methods, devices and computer program products for global bundle adjustment of 3D images
CN107836012B (zh) 投影图像生成方法及其装置、图像像素与深度值之间的映射方法
CN110462686B (zh) 用于从场景获得深度信息的设备和方法
US8699005B2 (en) Indoor surveying apparatus
US9460517B2 (en) Photogrammetric methods and devices related thereto
Pandey et al. Extrinsic calibration of a 3d laser scanner and an omnidirectional camera
CN111344644B (zh) 用于基于运动的自动图像捕获的技术
US20170134713A1 (en) Image calibrating, stitching and depth rebuilding method of a panoramic fish-eye camera and a system thereof
CN106643699A (zh) 一种虚拟现实系统中的空间定位装置和定位方法
TW201709718A (zh) 在使用者裝置上顯示光場為基影像之方法及裝置，及對應之電腦程式產品
WO2018227576A1 (fr) Procédé et système de détection de forme de sol, procédé d'atterrissage de drone et drone
JP2013539147A5 (fr)
CN110419208B (zh) 成像系统、成像控制方法、图像处理设备和计算机可读介质
WO2015068470A1 (fr) Dispositif de mesure de forme 3d, procédé de mesure de forme 3d, et programme de mesure de forme 3d
US11012677B1 (en) Systems and methods for correcting rolling shutter artifacts
CN110377148A (zh) 计算机可读介质、训练对象检测算法的方法及训练设备
US10388069B2 (en) Methods and systems for light field augmented reality/virtual reality on mobile devices
CN112312113A (zh) 用于生成三维模型的方法、装置和系统
Wan et al. A study in 3D-reconstruction using kinect sensor
US20210118160A1 (en) Methods, devices and computer program products for 3d mapping and pose estimation of 3d images
KR20210145734A (ko) 정보 처리 장치, 정보 처리 방법, 및 프로그램
Wang et al. Measurement and analysis of depth resolution using active stereo cameras
WO2020019175A1 (fr) Procédé et dispositif de traitement d'image et dispositif photographique et véhicule aérien sans pilote
WO2022019128A1 (fr) Dispositif de traitement d'informations, procédé de traitement d'informations et support d'enregistrement lisible par ordinateur
US11069121B2 (en) Methods, devices and computer program products for creating textured 3D images

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 21845780

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 21845780

Country of ref document: EP

Kind code of ref document: A1