WO2022239543A1 - 情報処理装置、情報処理方法および記憶媒体 - Google Patents
情報処理装置、情報処理方法および記憶媒体 Download PDFInfo
- Publication number
- WO2022239543A1 WO2022239543A1 PCT/JP2022/015015 JP2022015015W WO2022239543A1 WO 2022239543 A1 WO2022239543 A1 WO 2022239543A1 JP 2022015015 W JP2022015015 W JP 2022015015W WO 2022239543 A1 WO2022239543 A1 WO 2022239543A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- input image
- map
- real space
- current input
- information
- Prior art date
Links
- 230000010365 information processing Effects 0.000 title claims description 73
- 238000003672 processing method Methods 0.000 title claims description 7
- 238000000034 method Methods 0.000 claims abstract description 121
- 238000012545 processing Methods 0.000 claims abstract description 97
- 230000008569 process Effects 0.000 claims abstract description 95
- 238000003780 insertion Methods 0.000 claims description 156
- 230000037431 insertion Effects 0.000 claims description 156
- 238000001914 filtration Methods 0.000 claims description 6
- 238000001514 detection method Methods 0.000 description 16
- 238000010586 diagram Methods 0.000 description 13
- 238000004364 calculation method Methods 0.000 description 11
- 238000004891 communication Methods 0.000 description 11
- 238000000605 extraction Methods 0.000 description 8
- 230000006870 function Effects 0.000 description 8
- 238000005259 measurement Methods 0.000 description 6
- 239000000284 extract Substances 0.000 description 5
- 238000013507 mapping Methods 0.000 description 5
- 230000003139 buffering effect Effects 0.000 description 4
- 238000005266 casting Methods 0.000 description 4
- 230000008859 change Effects 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 239000004065 semiconductor Substances 0.000 description 4
- 230000009471 action Effects 0.000 description 2
- 230000005540 biological transmission Effects 0.000 description 2
- 238000004590 computer program Methods 0.000 description 2
- 230000000694 effects Effects 0.000 description 2
- 238000005401 electroluminescence Methods 0.000 description 2
- 238000005516 engineering process Methods 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 230000003068 static effect Effects 0.000 description 2
- 230000003190 augmentative effect Effects 0.000 description 1
- 239000000872 buffer Substances 0.000 description 1
- 230000000295 complement effect Effects 0.000 description 1
- 239000000470 constituent Substances 0.000 description 1
- 230000008602 contraction Effects 0.000 description 1
- 238000007796 conventional method Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 230000003111 delayed effect Effects 0.000 description 1
- 238000012217 deletion Methods 0.000 description 1
- 230000037430 deletion Effects 0.000 description 1
- 230000010339 dilation Effects 0.000 description 1
- 230000005484 gravity Effects 0.000 description 1
- 230000006872 improvement Effects 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 230000007774 longterm Effects 0.000 description 1
- 239000011159 matrix material Substances 0.000 description 1
- 229910044991 metal oxide Inorganic materials 0.000 description 1
- 150000004706 metal oxides Chemical class 0.000 description 1
- AYOOGWWGECJQPI-NSHDSACASA-N n-[(1s)-1-(5-fluoropyrimidin-2-yl)ethyl]-3-(3-propan-2-yloxy-1h-pyrazol-5-yl)imidazo[4,5-b]pyridin-5-amine Chemical compound N1C(OC(C)C)=CC(N2C3=NC(N[C@@H](C)C=4N=CC(F)=CN=4)=CC=C3N=C2)=N1 AYOOGWWGECJQPI-NSHDSACASA-N 0.000 description 1
- 230000001151 other effect Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 238000009877 rendering Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 230000005236 sound signal Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Images
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T15/00—3D [Three Dimensional] image rendering
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
- G06V10/00—Arrangements for image or video recognition or understanding
- G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
- G06V10/762—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using clustering, e.g. of similar faces in social networks
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
- G01B11/00—Measuring arrangements characterised by the use of optical techniques
- G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T19/00—Manipulating 3D models or images for computer graphics
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/70—Denoising; Smoothing
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/70—Determining position or orientation of objects or cameras
Definitions
- the present disclosure relates to an information processing device, an information processing method, and a storage medium.
- AR Augmented Reality
- VR Virtual Reality
- robotics the environment around the user or robot is updated in three dimensions in real time.
- the present disclosure proposes an information processing device, an information processing method, and a storage medium capable of updating a map with low delay and high accuracy.
- an information processing device includes an information acquisition unit, a determination unit, and an update processing unit.
- the information acquisition unit stores a real space map corresponding to a real space stored in a storage medium, a current input image representing the real space acquired by a sensor, a past input image, and the sensor corresponding to the current input image. and past position and orientation information of the sensor corresponding to the past input image.
- the determination unit determines whether or not the current input image includes an insertion object that is not included in the real space map, based on the real space map and the current input image.
- the update processing unit updates the real space map according to the current position and orientation information and the past position and orientation information based on a determination result that the current input image does not include the insertion object. different from the first map update process, performing an update process, and updating the real space map according to the current position and orientation information based on a determination result that the current input image includes the insertion object; 2 map update processing is executed.
- FIG. 10 is a diagram for explaining the purpose of processing for calculating the distance between a candidate region, which is a cluster, and an insertion object included in an insertion point group list;
- FIG. 3 is a diagram for explaining a pixel of interest and a voxel of interest;
- FIG. 10 is a diagram for explaining a pixel of interest and a voxel of interest;
- 10 is a diagram showing an overview of update processing by an update processing unit; 7 is a flowchart showing a processing procedure of real space map update processing executed by the information processing apparatus according to the embodiment; 4 is a flow chart showing a processing procedure of map update processing executed by the information processing apparatus according to the embodiment; 7 is a flowchart showing a processing procedure of insertion object area detection processing executed by the information processing apparatus according to the embodiment; It is a block diagram showing an example of hardware constitutions of an information processor concerning this embodiment.
- a plurality of components having substantially the same functional configuration may be distinguished by attaching different numbers after the same reference numerals.
- the same reference numerals are used.
- a 3D map a three-dimensional map obtained by performing the
- a typical method for reconstructing a scene in real time is a method of integrating multi-viewpoint depth images into a signed distance field (see, for example, Non-Patent Documents 1 and 2). These methods are currently used in various situations because they can be processed in real time and can extract polygon meshes that are important for shielding and physical simulation (see, for example, Non-Patent Document 3). .
- FIGS. 1 and 2 are diagrams for explaining the signed distance field.
- FIG. 1 shows voxels V obtained by dividing a three-dimensional space, which is a real space, into a grid-like array. The entire voxel V is composed of a plurality of voxels V that are unit elements.
- the signed distance field consists of the signed distance to the object surface (the outside of the object is positive, the inside is negative, and the object surface is zero), and the weight parameter is stored in each voxel V to represent the distance field.
- Non-Patent Documents 1 and 2 a 3D map is reconstructed by integrating multi-viewpoint depth images into a signed distance field using a moving average.
- the above-described conventional technique has a problem in that there is a delay in updating the 3D map and extracting its polygon mesh in areas where there has been a change from the scene at the time of previous reconstruction to the current scene. .
- this method requires a strong precondition that the 3D shape of the object that can be inserted into the scene is known. is delayed or cannot be updated with high accuracy.
- Non-Patent Document 4 As a document that focuses on 3D scanning in the same space at different times.
- Non-Patent Document 4 two or more 3D maps obtained by scanning the same room at different times are input, and a method of separating the static 3D model of the room and the dynamic 3D model of the room is used. disclosed.
- Non-Patent Document 4 aims to post-separate static and dynamic regions from multiple scanned maps in an offline environment. It doesn't update the map.
- Patent Document 1 discloses a method of performing self-position estimation and mapping in an environment where the position of an object changes.
- mapping is performed by matching processing with a known object database, in addition to being unable to handle unknown objects, a sparse feature point map is assumed as a map representation method, and dense 3D reconstruction and It is not intended for mesh extraction.
- the information processing apparatus 1 determines whether or not the current input image includes an insertion object that is not included in the real space map, and inserts the real space map according to the determination result. Perform update processing.
- the information processing apparatus 1 updates the real space map according to the current position and orientation information and the past position and orientation information based on the determination result that the current input image does not include the insertion object.
- the first map update process is, for example, an update process of updating the real space map using a moving average based on current position and orientation information and past position and orientation information.
- the information processing apparatus 1 executes second map update processing for updating the real space map according to the current position and orientation information based on the determination result that the current input image includes the insertion object.
- the second map update processing is update processing different from the first map update processing, and is update processing for updating the real space map based on current position and orientation information without using past position and orientation information. .
- the information processing apparatus 1 executes the first map update process for areas in the current input image in which the new insertion object does not exist, and performs the first map update process for areas in which the new insertion object exists.
- the second map update process is executed.
- the information processing apparatus 1 performs update processing using a moving average based on the current position and orientation information and the past position and orientation information for regions in the real space map in which the insertion object does not exist, so that the input image It is possible to perform high-precision updating while reducing the noise included in . Further, the information processing apparatus 1 performs update processing based on the current position and orientation information for a region in which an insertion object exists in the real space map, so that the newly appearing insertion object can be immediately displayed on the real space map. can be updated to reflect Thus, according to the information processing apparatus 1 according to the embodiment, it is possible to update the real space map with low delay and high accuracy.
- FIG. 3 is a block diagram showing a functional configuration example of the information processing device 1 according to the embodiment.
- the information processing device 1 according to the embodiment includes a control section 3 , a storage section 4 , a sensor 100 , an orientation detection section 200 and a display section 300 .
- FIG. 3 shows the configuration in which the sensor 100, the orientation detection unit 200, and the display unit 300 are incorporated inside the information processing apparatus 1, but at least one of the sensor 100, the orientation detection unit 200, and the display unit 300 One may be arranged outside the information processing device 1 and connected to the information processing device 1 .
- the sensor 100 acquires a depth image as an input image.
- the sensor 100 includes, for example, a TOF (Time of Flight) range-finding sensor, a stereo camera, a LiDAR (Light Detection and Ranging) range-finding sensor, or the like.
- the sensor 100 generates a depth image indicating the distance to an object (insertion object) existing in real space and outputs it to the control unit 3 .
- the sensor 100 may also acquire the captured image as the input image.
- the sensor 100 includes, for example, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, a CCD (Charge Coupled Device) image sensor, and the like.
- the sensor 100 outputs the acquired captured image to the control unit 3 .
- the posture detection unit 200 detects the posture of the sensor unit 21 used to acquire the input image using arbitrary odometry to acquire position and posture information.
- the posture detection unit 22 acquires position and posture information (for example, 6Dof (Degrees of freedom)) using an IMU sensor or the like and outputs it to the control unit 3 .
- the display unit 300 is a display such as an LCD (Liquid Crystal Display), and displays information output from the control unit 3. Also, the display unit 300 may have a function of displaying a three-dimensional virtual object such as AR or VR.
- LCD Liquid Crystal Display
- the storage unit 4 is realized, for example, by semiconductor memory elements such as RAM (Random Access Memory), ROM (Read Only Memory), flash memory, or storage devices such as hard disks and optical disks.
- the storage unit 4 stores a real space map 41, an insertion point cloud list 42, and various programs.
- the real space map 41 is real space map information based on the input image.
- the real space map 41 may be three-dimensional map information or two-dimensional map information.
- the real space map 41 is represented by voxels V obtained by dividing a three-dimensional space, which is a real space, into a grid-like array.
- the entire voxel V is composed of a plurality of voxels V that are unit elements.
- Each voxel V stores a signed distance to the object surface (the outside of the object is positive, the inside is negative, and the object surface is zero) and a weight parameter representing the reliability of the signed distance.
- the insertion point group list 42 is list information regarding the point group (pixel group) of the insertion object. Specifically, the insertion point cloud list 42 is information about point clouds of newly detected insertion objects that are included in the current input image and not included in the real space map 41 .
- the information of the insertion point cloud list 42 may be a configuration included in the real space map 41.
- each voxel V of the real space map 41 may be assigned a label indicating whether or not it is the region of the insertion object.
- the control unit 3 is a controller. For example, various programs stored in the storage unit 4 are executed by a CPU (Central Processing Unit), MPU (Micro Processing Unit), etc., using RAM as a work area. It is realized by Also, the control unit 3 can be implemented by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).
- ASIC Application Specific Integrated Circuit
- FPGA Field Programmable Gate Array
- the control unit 3 includes an information acquisition unit 31, a determination unit 32, an update processing unit 33, a PM extraction unit 34, a physics calculation unit 35, and a display control unit 36, and has information processing functions described below. achieve or perform any action.
- the information acquisition unit 31 acquires various types of information. For example, the information acquisition unit 31 reads (acquires) the real space map 41 from the storage unit 4 . The information acquisition unit 31 also acquires the current input image acquired by the sensor 100 . The information acquisition unit 31 also acquires the current position and orientation information of the sensor 100 corresponding to the current input image from the orientation detection unit 200 . Specifically, the information acquisition unit 31 acquires the position and orientation information of the sensor 100 detected by the orientation detection unit 200 when the current input image was acquired as the current position and orientation information.
- the information acquisition unit 31 also acquires past input images acquired by the sensor 100 .
- the information acquisition unit 31 buffers the input images acquired from the sensor 100 at predetermined intervals in the storage unit 4, and converts the input image, which is one or more frames before the current input image, into the past input image. to get as
- the information acquisition unit 31 also acquires past position and orientation information of the sensor 100 corresponding to past input images from the orientation detection unit 200 .
- the information acquisition unit 31 acquires the position and orientation information of the sensor 100 detected by the orientation detection unit 200 when the past input image was acquired as the past position and orientation information.
- the determination unit 32 determines whether the current input image includes an insertion object that is not included in the real space map 41 .
- FIG. 4 is a diagram showing an overview of determination processing by the determination unit 32.
- the image labeled "Live” corresponds to the current input image.
- An image described as “Virtual” corresponds to a virtual input image, which will be described later.
- “Inserted” is an image showing the residual obtained by subtracting "Virtual” from “Live”.
- the determination unit 32 subtracts information of a virtual input image, which will be described later, from the current input image, and determines whether or not there is an insertion object area (the chair area in FIG. 4) from the residual that is the subtraction result.
- the determination unit 32 generates a virtual input image based on the current input image and the real space map 41. Specifically, the determination unit 32 generates from the real space map 41 a virtual input image having substantially the same position and orientation information as the current position and orientation information of the sensor 100 corresponding to the current input image.
- the generation of the virtual input image includes, for example, a method using a ray marching method, a method of rendering a polygon mesh extracted from a 3D map, which is the real space map 41, using a graphics pipeline, and the like.
- the generated virtual input image may be regarded as a two-dimensional image (2D image) virtually generated from a 3D map, which is the real space map 41 .
- the determination unit 32 calculates a residual map using the current input image and the generated virtual input image.
- the residual map can be calculated by the following equation (1), where R(u) is the residual map, D(u) is the current input image, and D(u) is the virtual input image.
- the residual map may be N(u) for the virtual normal image, V(u) for information obtained by converting the current input image into a point cloud using the internal parameters of the sensor 100, and V(u) for the virtual input image inside the sensor 100. It may be calculated by the following formula (2), with information converted into a point group using parameters as V ⁇ (u). Note that the virtual normal image can be generated at the same time as the virtual input image is generated.
- the determination unit 32 performs spatial filtering on the calculated residual map.
- the filtering process for example, an opening process that combines contraction and dilation can be used. Thereby, noise contained in the residual map due to noise contained in the current input image and the virtual input image can be removed.
- the determination unit 32 generates a binarized image by binarizing each pixel of the residual map after the filtering process using a preset threshold value. Subsequently, the determination unit 32 clusters the binarized image based on the connected components. A cluster extracted by this clustering is a set of connected pixels having the same value in binarization, and becomes a candidate area for an insertion object.
- the determination unit 32 determines whether or not the candidate area, which is the extracted cluster, is truly the area of the insertion object.
- the determination unit 32 calculates the distance between the extracted candidate region, which is the cluster, and the insertion object included in the insertion point group list 42 .
- the purpose of calculating the distance will be described later with reference to FIG.
- the determination unit 32 uses the internal parameters of the sensor 100 to convert the depth of each pixel forming a cluster into a point group.
- Euclidean distance such as the following formula (3) can be used.
- the distance d may be a Mahalanobis distance that considers the distribution of cluster point groups, as in the following equation (4).
- ⁇ j in the following equation (4) is the variance-covariance matrix of the point group P j of the cluster
- P ⁇ j is the center of gravity of the point group P j of the cluster.
- the determining unit 32 determines that a cluster whose distance d is less than a preset threshold value is the same as an insertion object detected in the past.
- the inserted object is determined to be point clouds detected from different angles (or the same angle). That is, when the distance d is less than a preset threshold, the determination unit 32 determines that the cluster is a point cloud of an insertion object already registered in the insertion point cloud list 42 and is newly detected in the current input image. is not an inserted object. Then, the determination unit 32 updates the information of the insertion objects already registered in the insertion point group list 42 based on the clusters.
- the determination unit 32 determines whether clusters whose distance d is equal to or greater than a preset threshold value are insertion objects newly detected in the current input image, or are generated by noise in the residual map. It is judged to be one of the outliers.
- the determination unit 32 determines that the cluster is an outlier, and inserts Exclude from object area.
- the determination unit 32 determines that an insertion newly detected in the current input image. It is determined to be an object area, and is registered in the insertion point group list 42 as a new insertion object area.
- the determination unit 32 determines that the current input image includes an insertion object that is not included in the real space map 41. judge.
- the determination unit 32 selects the insertion object that is not included in the real space map 41 as the current is not included in the input image.
- FIG. 5 is a diagram for explaining the purpose of the process of calculating the distance between candidate regions, which are clusters, and insertion objects included in the insertion point group list 42.
- this process is a process of buffering the previously detected insertion object area in the insertion point group list 42 and comparing it with the currently detected candidate area.
- the sensor 100 detects a predetermined insertion object OB over two frames at time t-1 and time t.
- the area OBt-1 of the insertion object OB is detected, and the real space map 41 is updated.
- the area OBt is detected.
- a zero crossing plane occurs in the area OBt ⁇ 1 of the area OBt. Only the region Rt has a difference map value equal to or greater than the threshold.
- the region Rt is the noise of the residual map. It is difficult to distinguish between outliers caused by , and regions of the inserted object that have already been partially measured.
- the threshold for the number of pixels in the cluster is increased, the number of pixels in the region Rt is erroneously determined to be an outlier less than the threshold. This is because there is a high possibility that other clusters generated by noise will be erroneously determined as the region of the insertion object.
- the area Rt shown in FIG. is part of the area of the insertion object OB that has already been measured.
- a small cluster (region Rt shown in FIG. 5) generated by measuring the previously measured insertion object OB from a slightly different angle can be reliably determined to be the region of the insertion object OB, and at the same time it is an outlier.
- Other small clusters can be excluded by thresholding the number of pixels.
- the area of the insertion object OB registered in the insertion point cloud list 42 is updated to an area obtained by combining the area OBt-1 with the area Rt.
- the update processing unit 33 performs different map update processing according to the determination result of the insertion object by the determination unit 32 . Specifically, the update processing unit 33 executes the first map update processing based on the determination result by the determination unit 32 that the current input image does not include the new insertion object, and changes the current input image to a new insertion object. A second map update process is performed based on the determination that the inserted object is included.
- the first map update process executes an update process of updating the real space map 41 according to the current position and orientation information and the past position and orientation information.
- the second map update process is an update process for updating the real space map 41 according to current position and orientation information without using past position and orientation information.
- the update processing unit 33 performs ray casting from the center of the sensor 100 for each pixel (pixel of interest) of the current input image, and acquires a voxel (voxel of interest) where the ray intersects.
- FIG. 6 is a diagram for explaining a pixel of interest and a voxel of interest.
- a point (depth) corresponding to the object surface is obtained from the input image based on the current position and orientation information of the sensor 100, this point is determined as the target pixel IP.
- a line passing through the target pixel IP and the sensor 100 is determined as a ray, and a voxel at which the ray intersects is determined as a target voxel IV.
- all of the colored voxels are voxels of interest IV, and the lighter the color, the closer it is to the pixel of interest IP.
- FIG. 7 is a diagram showing an overview of update processing by the update processing unit 33. As shown in FIG. The example shown in FIG. 7 shows an example in which a new insertion object is inserted.
- the object surface of the insertion object is detected by the sensor 100 as a depth, and the detected depth becomes the pixel of interest IP.
- a voxel of interest IV is extracted.
- the update processing unit 33 performs the second map update processing on the voxels of interest corresponding to the insertion object, and performs the first map update processing on the voxels of interest IV not corresponding to the insertion object. process.
- the update processing unit 33 performs two determination processes using the pixel of interest and the voxel of interest, and performs the first map update process or the second map update process according to the determination results of the two determination processes. conduct.
- the update processing unit 33 determines whether or not the pixel of interest is the area of the insertion object. Specifically, the update processing unit 33 determines whether or not the pixel of interest is a pixel included in the region of the insertion object newly registered in the insertion point group list 42 . In addition, as the second determination process, the update processing unit 33 determines whether or not the distance between the voxel of interest and the measurement point (pixel of interest IP) of the input image of interest is less than a preset threshold value.
- the update processing unit 33 If either of these two determination processes does not satisfy the condition, the update processing unit 33 considers that the space occupied by the target voxel has not changed significantly since the time when the real space map 41 was previously generated (updated). Therefore, the first map update process is executed. That is, when the pixel of interest is not in the region of the insertion object, or when the distance between the voxel of interest and the measurement point of the input image of interest is equal to or greater than a threshold, the update processing unit 33 uses the following equation (5), which is a moving average, and The signed distance and weight parameters of voxels in the real space map 41 are updated by executing the first map update process using equation (6).
- D t ⁇ 1 (v) and W t ⁇ 1 (v) are the signed distance and weight parameters before updating
- d t (v, u) and w t (v, u) are signed distance and weight parameters calculated based on the current input image and current pose information.
- the update processing unit 33 changes the voxel of interest due to the insertion of a new object compared to the time when the real space map 41 was previously generated (updated).
- a second map update process is executed because it is considered that the space occupied has changed. That is, when the pixel of interest is in the region of the insertion object and the distance between the voxel of interest and the measurement point of the input image of interest is less than the threshold, the update processing unit 33 uses the following formulas (7) and (8) to update the signed distance and weight parameters of voxels in the real space map 41 by executing the second map update process using .
- the equations (7) and (8) mean that the input image of the current scene acquired from the sensor 100 is immediately reflected on the real space map 41. In this way, by explicitly determining whether the voxel of interest is the space occupied by the insertion object and adaptively switching the updating method, the noise reduction effect of the first map updating process and the second map It is possible to achieve both the immediacy of update processing. That is, low-delay and high-precision map update can be realized.
- the PM extraction unit 34 extracts polygon meshes for each insertion object from the real space map 41 updated by the update processing unit 33 . Specifically, the PM extraction unit 34 extracts voxels having a signed distance of zero in the real space map 41 for each insertion object, and extracts a polygon mesh for each insertion object based on the extracted voxels.
- the physics calculation unit 35 Based on the polygon meshes extracted by the PM extraction unit 34, the physics calculation unit 35 performs various calculations regarding the actions of AR, VR, robots, etc., and reflects the calculation results in AR, VR, and robots.
- the display control unit 36 performs display calculations related to AR and VR based on the polygon meshes extracted by the PM extraction unit 34, and reflects the calculation results on the display unit 300.
- FIG. 8 is a flow chart showing the processing procedure of the real space map updating process executed by the information processing apparatus 1 according to the embodiment.
- control unit 3 reads the real space map 41 stored in the storage unit 4 (step S101).
- control unit 3 generates an empty insertion point cloud list 42 regarding the point cloud of the insertion object corresponding to the real space map 41 in the storage unit 4 (step S102).
- control unit 3 acquires the current input image and the current position and orientation information of the sensor 100 corresponding to the current input image (step S103).
- control unit 3 detects a new insertion object area included in the current input image based on the current input image, the current position and orientation information, and the real space map 41 (step S104).
- control unit 3 registers the point cloud corresponding to the area of the detected insertion object in the insertion point cloud list 42 (step S105).
- control unit 3 updates the real space map 41 based on whether each pixel of the current input image is included in the area of the insertion object (step S106).
- control unit 3 extracts polygon meshes from the updated real space map 41 (step S107).
- step S108 determines whether or not the mapping has been completed. If the mapping has been completed (step S108: Yes), the real space map 41 is stored in the storage unit 4 (step S109). , terminate the process. On the other hand, if the mapping has not ended (step S108: No), the controller 3 returns to step S103.
- FIG. 9 is a flowchart showing the procedure of map update processing executed by the information processing apparatus 1 according to the embodiment.
- control unit 3 performs ray casting from the center of the sensor 100 for each pixel of the input image (step S201).
- control unit 3 acquires voxels intersected by rays (step S202).
- control unit 3 determines whether or not the pixel of interest is the region of the insertion object (step S203).
- control unit 3 determines whether or not the distance between the voxel of interest and the measurement point is within a threshold value (step S204).
- step S204 When the distance between the voxel of interest and the measurement point is within the threshold (step S204: Yes), the control unit 3 updates the voxel by the second map update process (step S205).
- step S203 determines whether the pixel of interest is the region of the insertion object (step S203: No), or if the distance between the voxel of interest and the measurement point is not within the threshold (step S204: No).
- the control unit 3 updates the first map. Voxels are updated by processing (step S206).
- control unit 3 determines whether or not to continue ray casting (step S207). ) and returns to step S202.
- step S207 No
- step S207 the control unit 3 repeats steps S201 to S207 until it is completed.
- FIG. 10 is a flow chart showing the processing procedure of the insertion object area detection processing executed by the information processing apparatus 1 according to the embodiment.
- control unit 3 synthesizes a virtual depth image (past input image) from the real space map 41 (step S301).
- control unit 3 calculates a residual map between the depth image (current input image) acquired from the sensor and the past input image (step S302).
- control unit 3 filters the residual map (step S303).
- control unit 3 binarizes and clusters the filtered residual map (step S304).
- control unit 3 determines whether the cluster exists within a threshold distance from the point group registered in the insertion point group list 42 (step S305).
- control unit 3 designates the pixels included in the cluster as the region of the insertion object (step S306).
- control unit 3 adds the point cloud in the cluster to the insertion point cloud list 42 (step S307), and ends the process when the above-described processing is completed for each cluster. If the above-described processing for each cluster has not been completed, the control unit 3 repeats steps S305 to S308 until the processing is completed.
- step S305 if the cluster does not exist within the threshold distance from the point group registered in the insertion point group list 42 (step S305: No), the control unit 3 determines that the number of pixels in the cluster is equal to or greater than the threshold. It is determined whether or not there is (step S308).
- step S308 Yes
- the control unit 3 proceeds to step S306. That is, it is designated as the region of the new insertion object.
- step S308: No the control unit 3 detects the cluster as noise and proceeds to process the next cluster.
- FIG. 11 is a block diagram showing an example of the hardware configuration of the information processing device 1 according to this embodiment.
- the information processing apparatus 1 includes a CPU (Central Processing Unit) 901, a ROM (Read Only Memory) 902, a RAM (Random Access Memory) 903, a host bus 905, a bridge 907, an external bus 906, and an interface 908. , an input device 911 , an output device 912 , a storage device 913 , a drive 914 , a connection port 915 and a communication device 916 .
- the information processing apparatus 1 may be provided with a processing circuit such as an electric circuit, DSP, or ASIC in place of or together with the CPU 901 .
- the CPU 901 functions as an arithmetic processing device and a control device, and controls general operations within the information processing device 1 according to various programs.
- the CPU 901 may be a microprocessor.
- the ROM 902 stores programs and calculation parameters used by the CPU 901 .
- the RAM 903 temporarily stores programs used in the execution of the CPU 901, parameters that change as appropriate during the execution, and the like.
- the CPU 901 may execute the functions of the information acquisition unit 31, the determination unit 32, the update processing unit 33, the PM extraction unit 34, the physics calculation unit 35, and the display control unit 36, for example.
- the CPU 901, ROM 902 and RAM 903 are interconnected by a host bus 905 including a CPU bus.
- the host bus 905 is connected via a bridge 907 to an external bus 906 such as a PCI (Peripheral Component Interconnect/Interface) bus.
- PCI Peripheral Component Interconnect/Interface
- the host bus 905, bridge 907, and external bus 906 do not necessarily have to be configured separately, and these functions may be implemented in one bus.
- the input device 911 is, for example, a device such as a mouse, keyboard, touch panel, button, microphone, switch, or lever, through which information is input by the user.
- the input device 911 may be a remote control device using infrared rays or other radio waves, or may be an external connection device such as a mobile phone or a PDA corresponding to the operation of the information processing device 1 .
- the input device 911 may include, for example, an input control circuit that generates an input signal based on information input by the user using the input means described above.
- the output device 912 is a device capable of visually or audibly notifying the user of information.
- the output device 912 is, for example, a CRT (Cathode Ray Tube) display device, a liquid crystal display device, a plasma display device, an EL (Electro Luminescence) display device, a laser projector, an LED (Light Emitting Diode) projector or a display device such as a lamp.
- a CTR Cathode Ray Tube
- a liquid crystal display device a plasma display device
- an EL (Electro Luminescence) display device a laser projector
- LED Light Emitting Diode
- a display device such as a lamp.
- it may be an audio output device such as a speaker or headphones.
- the output device 912 may, for example, output results obtained by various processes performed by the information processing device 1 .
- the output device 912 may visually display the results obtained by various processes performed by the information processing device 1 in various formats such as text, images, tables, graphs, and the like.
- the output device 912 may convert an audio signal such as voice data or acoustic data into an analog signal and audibly output it.
- Input device 911 and output device 912 may, for example, perform the functions of an interface.
- the storage device 913 is a data storage device formed as an example of the storage unit 4 of the information processing device 1 .
- the storage device 913 may be implemented by, for example, a magnetic storage device such as a HDD (Hard Disk Drive), a semiconductor storage device, an optical storage device, a magneto-optical storage device, or the like.
- the storage device 913 may include a storage medium, a recording device that records data on the storage medium, a reading device that reads data from the storage medium, and a deletion device that deletes data recorded on the storage medium.
- the storage device 913 may store programs executed by the CPU 901, various data, various data obtained from the outside, and the like.
- the storage device 913 may perform the function of storing the real space map 41 and the insertion point cloud list 42, for example.
- the drive 914 is a storage medium reader/writer, and is built in or externally attached to the information processing apparatus 1 .
- the drive 914 reads information recorded on a removable storage medium such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and outputs the information to the RAM 903 .
- Drives 914 can also write information to removable storage media.
- connection port 915 is an interface that is connected to an external device.
- the connection port 915 is a connection port capable of data transmission with an external device, and may be, for example, a USB (Universal Serial Bus).
- the communication device 916 is, for example, an interface formed by a communication device or the like for connecting to the network N.
- the communication device 916 may be, for example, a communication card for wired or wireless LAN (Local Area Network), LTE (Long Term Evolution), Bluetooth (registered trademark), or WUSB (Wireless USB).
- the communication device 916 may be a router for optical communication, a router for ADSL (Asymmetric Digital Subscriber Line), a modem for various types of communication, or the like.
- the communication device 916 can, for example, transmit and receive signals to and from the Internet or other communication devices according to a predetermined protocol such as TCP/IP.
- the network N is a wired or wireless transmission path for information.
- the network N may include the Internet, a public line network such as a telephone line network or a satellite communication network, various LANs (Local Area Networks) including Ethernet (registered trademark), or WANs (Wide Area Networks).
- the network N may also include a dedicated line network such as IP-VPN (Internet Protocol-Virtual Private Network).
- a computer program is also created for causing hardware such as the CPU, ROM, and RAM built into the information processing apparatus 1 to exhibit functions equivalent to the respective components of the information processing apparatus 1 according to the present embodiment described above. It is possible. It is also possible to provide a storage medium storing the computer program.
- each component of each device illustrated is functionally conceptual and does not necessarily need to be physically configured as illustrated.
- the specific forms of distribution and integration of each device are not limited to those illustrated, and all or part of them can be functionally or physically distributed and integrated in arbitrary units according to various loads and usage conditions. Can be integrated and configured.
- the information processing device 1 includes the information acquisition unit 31, the determination unit 32, and the update processing unit 33.
- the information acquisition unit 31 acquires a real space map 41 corresponding to the real space stored in the storage medium (storage unit 4), a current input image representing the real space acquired by the sensor 100, a past input image, and a current input image. and the past position and orientation information of the sensor 100 corresponding to past input images.
- the determination unit 32 determines whether the current input image includes an insertion object that is not included in the real space map 41 .
- the update processing unit 33 executes first map update processing for updating the real space map 41 according to the current position and orientation information and the past position and orientation information. Then, based on the determination result that the current input image includes the insertion object, the real space map 41 is updated according to the current position and orientation information, and a second map update process different from the first map update process is executed. .
- the real space map 41 can be updated with low delay and high accuracy.
- the update processing unit 33 updates the target pixel IP corresponding to the target pixel IP.
- the second map update process is performed on the target voxel IV, and if the target pixel IP is not included in the area of the insertion object, the target voxel IV corresponding to the target pixel IP is subjected to the first map update process. to run.
- low-latency map updating is realized by executing the second map update process for the area of the insertion object in the current input image, and the first map update process is performed for the area other than the insertion object.
- High-precision map update that eliminates noise can be achieved by executing map update processing.
- the update processing unit 33 updates the voxel of interest IV.
- the distance between the target pixel IP and the target voxel IV is equal to or greater than a predetermined threshold value
- the first map update process is performed for the target voxel IV.
- map update accuracy can be increased.
- the determination unit 32 generates a virtual input image from the real space map 41 according to the current position and orientation information, and uses a residual map calculated based on the current input image and the virtual input image to replace the insertion object with the current input image. is included.
- the determination unit 32 determines whether or not the current input image includes an insertion object using the residual map that has undergone filtering processing to remove noise contained in the residual map.
- the determination unit 32 generates a binarized image by binarizing each pixel in the residual map, and determines that the current input image includes the insertion object based on the cluster regions obtained by clustering the connected components in the binarized image. Determine whether or not
- the area of the insertion object included in the current input image can be extracted with high accuracy.
- the determination unit 32 determines that the cluster area is the insertion object area.
- a small cluster generated by measuring the already measured insertion object from a slightly different angle with the sensor 100 can be reliably determined to be the region of the insertion object OB.
- the determination unit 32 determines that the cluster area is an insertion object area when the number of pixels in the cluster area is equal to or greater than a predetermined threshold.
- the generated virtual input image is a two-dimensional image having substantially the same position and orientation information as the current position and orientation information.
- voxels V containing signed distances and weights indicating the reliability of the signed distances are arranged.
- the first map update processing updates moving averages of signed distances and weights calculated based on the current input image corresponding to the current position and orientation information and the past input image corresponding to the past position and orientation information.
- the second map update process is an update process in which the signed distance and weight calculated based on the current input image corresponding to the current position and orientation information are updated values. be.
- a real space map corresponding to a real space stored in a storage medium a current input image showing the real space acquired by a sensor, the past input image, and the current position and orientation of the sensor corresponding to the current input image.
- an information acquisition unit that acquires information and past position and orientation information of the sensor corresponding to the past input image; a determination unit that determines whether or not the current input image includes an insertion object that is not included in the real space map, based on the real space map and the current input image; executing a first map update process for updating the real space map according to the current position and orientation information and the past position and orientation information based on a determination result that the current input image does not include the insertion object; performing a second map update process different from the first map update process for updating the real space map according to the current position and orientation information based on a determination result that the current input image includes the insertion object;
- An information processing device comprising: an update processing unit that executes.
- the update processing unit When the pixel of interest in the current input image is included in the region of the insertion object and the distance between the pixel of interest and the voxel of interest is less than a predetermined threshold, the second Execute the map update process of The information processing apparatus according to (2), wherein when the distance between the pixel of interest and the voxel of interest is equal to or greater than a predetermined threshold, the first map update process is performed on the voxel of interest.
- the determination unit is generating a virtual input image from the real space map according to the current position and orientation information;
- the information processing apparatus according to any one of (1) to (3) above, which determines whether or not includes (5)
- the determination unit is The information according to (4) above, wherein, using the residual map subjected to filtering processing for removing noise contained in the residual map, it is determined whether or not the current input image includes the insertion object. processing equipment.
- the determination unit is generating a binarized image by binarizing each pixel in the residual map, and determining whether the current input image includes the insertion object based on cluster regions obtained by clustering the connected components in the binarized image;
- the information processing apparatus according to (4) or (5), which determines whether or not.
- the determination unit is If the distance between the cluster area extracted this time and the cluster area extracted last time is less than a predetermined threshold, the cluster area is determined to be the insertion object area, ).
- the determination unit is The information processing apparatus according to (6) or (7) above, wherein the cluster area is determined to be the insertion object area when the number of pixels in the cluster area is equal to or greater than a predetermined threshold.
- the information processing apparatus according to any one of (4) to (8), wherein the generated virtual input image is a two-dimensional image having substantially the same position and orientation information as the current position and orientation information. .
- the real space map is Voxels containing signed distances and weights indicating the reliability of said signed distances are arranged
- the first map update process includes: After updating the moving average of the signed distance and the weight calculated based on the current input image corresponding to the current position and orientation information and the past input image corresponding to the past position and orientation information is an update process with a value of
- the second map update process includes: Any one of (1) to (9) above, wherein the updating process uses the signed distance and the weight calculated based on the current input image corresponding to the current position and orientation information as updated values.
- a computer-executed information processing method comprising: A real space map corresponding to a real space stored in a storage medium, a current input image showing the real space acquired by a sensor, the past input image, and the current position and orientation of the sensor corresponding to the current input image.
- An information processing method comprising: an update processing step to be performed.
- an information acquisition unit that acquires information and past position and orientation information of the sensor corresponding to the past input image;
- a determination unit that determines whether or not the current input image includes an insertion object that is not included in the real space map, based on the real space map and the current input image; executing a first map update process for updating the real space map according to the current position and orientation information and the past position and orientation information based on a determination result that the current input image does not include the insertion object; performing a second map update process different from the first map update process for updating the real space map according to the current position and orientation information based on a determination result that the current input image includes the insertion object;
- a storage medium storing a program for functioning as an update processing unit to be executed.
Landscapes
- Engineering & Computer Science (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Software Systems (AREA)
- Computer Graphics (AREA)
- Health & Medical Sciences (AREA)
- Artificial Intelligence (AREA)
- Computing Systems (AREA)
- Databases & Information Systems (AREA)
- Evolutionary Computation (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Multimedia (AREA)
- Computer Hardware Design (AREA)
- General Engineering & Computer Science (AREA)
- Image Analysis (AREA)
Abstract
Description
1.はじめに
2.本開示の概要
3.情報処理装置の機能構成
4.処理フロー
5.ハードウェア構成例
6.まとめ
ARやVR、ロボティクスにおいては、デプスセンサやステレオカメラ、測距センサ等のデバイスを用いて、ユーザやロボットの周囲の環境を三次元に再構成することが行われており、このような再構成をリアルタイムに行うことは重要である。
そこで、本開示では、既知物体の形状データベースを用いることなく、上述した従来技術で生じる問題を解決する手法を提案する。なお、<<2.本開示の概要>>では、実施形態に係る情報処理装置1が実行する処理の概要を説明し、より詳細な処理については、<<3.情報処理装置の機能構成>>以降で後述する。
まず、図3を用いて、上述した情報処理装置1の機能構成例について説明する。図3は、実施形態に係る情報処理装置1の機能構成例を示すブロック図である。図3に示すように、実施形態に係る情報処理装置1は、制御部3と、記憶部4と、センサ100と、姿勢検出部200と、表示部300とを備える。なお、図3では、センサ100、姿勢検出部200および表示部300は、情報処理装置1の内部に組み込まれた構成を示しているが、センサ100、姿勢検出部200および表示部300の少なくとも1つが情報処理装置1の外部に配置され、情報処理装置1に接続される構成であってもよい。
次に、図8を用いて、実施形態に係る情報処理装置1によって実行される実空間マップの更新処理の処理手順について説明する。図8は、実施形態に係る情報処理装置1によって実行される実空間マップの更新処理の処理手順を示すフローチャートである。
続いて、図11を参照して、本実施形態に係る情報処理装置1等のハードウェア構成の一例について説明する。図11は、本実施形態に係る情報処理装置1のハードウェア構成の一例を示すブロック図である。
以上説明したように、本開示の一実施形態によれば、情報処理装置1は、情報取得部31と、判定部32と、更新処理部33とを備える。情報取得部31は、記憶媒体(記憶部4)に格納された実空間に対応する実空間マップ41、センサ100が取得した実空間を示す現在の入力画像、過去の入力画像、現在の入力画像に対応するセンサ100の現在の位置姿勢情報、および過去の入力画像に対応するセンサ100の過去の位置姿勢情報を取得する。判定部32は、実空間マップ41および現在の入力画像に基づいて、実空間マップ41に含まれない挿入オブジェクトを現在の入力画像が含むか否かを判定する。更新処理部33は、現在の入力画像が挿入オブジェクトを含まないという判定結果に基づいて、現在の位置姿勢情報および過去の位置姿勢情報に従って実空間マップ41を更新する第1のマップ更新処理を実行し、現在の入力画像が挿入オブジェクトを含むという判定結果に基づいて、現在の位置姿勢情報に従って実空間マップ41を更新する、第1のマップ更新処理とは異なる第2のマップ更新処理を実行する。
(1)
記憶媒体に格納された実空間に対応する実空間マップ、センサが取得した前記実空間を示す現在の入力画像、過去の前記入力画像、前記現在の入力画像に対応する前記センサの現在の位置姿勢情報、および前記過去の入力画像に対応する前記センサの過去の位置姿勢情報を取得する情報取得部と、
前記実空間マップおよび前記現在の入力画像に基づいて、前記実空間マップに含まれない挿入オブジェクトを前記現在の入力画像が含むか否かを判定する判定部と、
前記現在の入力画像が前記挿入オブジェクトを含まないという判定結果に基づいて、前記現在の位置姿勢情報および前記過去の位置姿勢情報に従って前記実空間マップを更新する第1のマップ更新処理を実行し、
前記現在の入力画像が前記挿入オブジェクトを含むという判定結果に基づいて、前記現在の位置姿勢情報に従って前記実空間マップを更新する、前記第1のマップ更新処理とは異なる第2のマップ更新処理を実行する更新処理部とを備える、情報処理装置。
(2)
前記更新処理部は、
前記現在の入力画像が前記挿入オブジェクトを含むという判定結果に基づいて、前記現在の入力画像における注目画素が前記挿入オブジェクトの領域に含まれる場合には、該注目画素に対応した注目ボクセルに対して前記第2のマップ更新処理を実行し、
前記注目画素が前記挿入オブジェクトの領域に含まれない場合には、該注目画素に対応した注目ボクセルに対して前記第1のマップ更新処理を実行する、前記(1)に記載の情報処理装置。
(3)
前記更新処理部は、
前記現在の入力画像における前記注目画素が前記挿入オブジェクトの領域に含まれる場合において、前記注目画素と、前記注目ボクセルとの距離が所定の閾値未満であるときには、該注目ボクセルに対して前記第2のマップ更新処理を実行し、
前記注目画素と、前記注目ボクセルとの距離が所定の閾値以上であるときには、該注目ボクセルに対して前記第1のマップ更新処理を実行する、前記(2)に記載の情報処理装置。
(4)
前記判定部は、
前記現在の位置姿勢情報に従って前記実空間マップから仮想入力画像を生成し、前記現在の入力画像および前記仮想入力画像に基づいて算出した残差マップを用いて、前記挿入オブジェクトを前記現在の入力画像が含むか否かを判定する、前記(1)~(3)のいずれか1つに記載の情報処理装置。
(5)
前記判定部は、
前記残差マップに含まれるノイズを除去するフィルタリング処理を行った前記残差マップを用いて、前記挿入オブジェクトを前記現在の入力画像が含むか否かを判定する、前記(4)に記載の情報処理装置。
(6)
前記判定部は、
前記残差マップにおける各画素を二値化した二値化画像を生成し、前記二値化画像における連結成分をクラスタリングしたクラスタの領域に基づいて、前記挿入オブジェクトを前記現在の入力画像が含むか否かを判定する、前記(4)または(5)に記載の情報処理装置。
(7)
前記判定部は、
今回抽出された前記クラスタの領域と、前回抽出された前記クラスタの領域との距離が所定の閾値未満である場合に、前記クラスタの領域が前記挿入オブジェクトの領域であると判定する、前記(6)に記載の情報処理装置。
(8)
前記判定部は、
前記クラスタの領域の画素数が所定の閾値以上である場合に、前記クラスタの領域が前記挿入オブジェクトの領域であると判定する、前記(6)または(7)に記載の情報処理装置。
(9)
前記生成される仮想入力画像は、前記現在の位置姿勢情報と実質的に同じ位置姿勢情報を有する二次元画像である、前記(4)~(8)のいずれか1つに記載の情報処理装置。
(10)
前記実空間マップは、
符号付距離および前記符号付距離の信頼度を示す重みを含むボクセルが配列され、
前記第1のマップ更新処理は、
前記現在の位置姿勢情報に対応した前記現在の入力画像と、前記過去の位置姿勢情報に対応した前記過去の入力画像とに基づいて算出した前記符号付距離および前記重みの移動平均を更新後の値とする更新処理であり、
前記第2のマップ更新処理は、
前記現在の位置姿勢情報に対応した前記現在の入力画像に基づいて算出した前記符号付距離および前記重みを更新後の値とする更新処理である、前記(1)~(9)のいずれか1つに記載の情報処理装置。
(11)
コンピュータが実行する情報処理方法であって、
記憶媒体に格納された実空間に対応する実空間マップ、センサが取得した前記実空間を示す現在の入力画像、過去の前記入力画像、前記現在の入力画像に対応する前記センサの現在の位置姿勢情報、および前記過去の入力画像に対応する前記センサの過去の位置姿勢情報を取得する情報取得工程と、
前記実空間マップおよび前記現在の入力画像に基づいて、前記実空間マップに含まれない挿入オブジェクトを前記現在の入力画像が含むか否かを判定する判定工程と、
前記現在の入力画像が前記挿入オブジェクトを含まないという判定結果に基づいて、前記現在の位置姿勢情報および前記過去の位置姿勢情報に従って前記実空間マップを更新する第1のマップ更新処理を実行し、
前記現在の入力画像が前記挿入オブジェクトを含むという判定結果に基づいて、前記現在の位置姿勢情報に従って前記実空間マップを更新する、前記第1のマップ更新処理とは異なる第2のマップ更新処理を実行する更新処理工程とを含む、情報処理方法。
(12)
コンピュータを、
記憶媒体に格納された実空間に対応する実空間マップ、センサが取得した前記実空間を示す現在の入力画像、過去の前記入力画像、前記現在の入力画像に対応する前記センサの現在の位置姿勢情報、および前記過去の入力画像に対応する前記センサの過去の位置姿勢情報を取得する情報取得部と、
前記実空間マップおよび前記現在の入力画像に基づいて、前記実空間マップに含まれない挿入オブジェクトを前記現在の入力画像が含むか否かを判定する判定部と、
前記現在の入力画像が前記挿入オブジェクトを含まないという判定結果に基づいて、前記現在の位置姿勢情報および前記過去の位置姿勢情報に従って前記実空間マップを更新する第1のマップ更新処理を実行し、
前記現在の入力画像が前記挿入オブジェクトを含むという判定結果に基づいて、前記現在の位置姿勢情報に従って前記実空間マップを更新する、前記第1のマップ更新処理とは異なる第2のマップ更新処理を実行する更新処理部と
として機能させるためのプログラムが記憶された記憶媒体。
3 制御部
4 記憶部
21 センサ部
22 姿勢検出部
31 情報取得部
32 判定部
33 更新処理部
34 PM抽出部
35 物理演算部
36 表示制御部
41 実空間マップ
42 挿入点群リスト
100 センサ
200 姿勢検出部
300 表示部
IP 注目画素
IV 注目ボクセル
OB 挿入オブジェクト
V ボクセル
Claims (12)
- 記憶媒体に格納された実空間に対応する実空間マップ、センサが取得した前記実空間を示す現在の入力画像、過去の前記入力画像、前記現在の入力画像に対応する前記センサの現在の位置姿勢情報、および前記過去の入力画像に対応する前記センサの過去の位置姿勢情報を取得する情報取得部と、
前記実空間マップおよび前記現在の入力画像に基づいて、前記実空間マップに含まれない挿入オブジェクトを前記現在の入力画像が含むか否かを判定する判定部と、
前記現在の入力画像が前記挿入オブジェクトを含まないという判定結果に基づいて、前記現在の位置姿勢情報および前記過去の位置姿勢情報に従って前記実空間マップを更新する第1のマップ更新処理を実行し、
前記現在の入力画像が前記挿入オブジェクトを含むという判定結果に基づいて、前記現在の位置姿勢情報に従って前記実空間マップを更新する、前記第1のマップ更新処理とは異なる第2のマップ更新処理を実行する更新処理部とを備える、情報処理装置。 - 前記更新処理部は、
前記現在の入力画像が前記挿入オブジェクトを含むという判定結果に基づいて、前記現在の入力画像における注目画素が前記挿入オブジェクトの領域に含まれる場合には、該注目画素に対応した注目ボクセルに対して前記第2のマップ更新処理を実行し、
前記注目画素が前記挿入オブジェクトの領域に含まれない場合には、該注目画素に対応した注目ボクセルに対して前記第1のマップ更新処理を実行する、請求項1に記載の情報処理装置。 - 前記更新処理部は、
前記現在の入力画像における前記注目画素が前記挿入オブジェクトの領域に含まれる場合において、前記注目画素と、前記注目ボクセルとの距離が所定の閾値未満であるときには、該注目ボクセルに対して前記第2のマップ更新処理を実行し、
前記注目画素と、前記注目ボクセルとの距離が所定の閾値以上であるときには、該注目ボクセルに対して前記第1のマップ更新処理を実行する、請求項2に記載の情報処理装置。 - 前記判定部は、
前記現在の位置姿勢情報に従って前記実空間マップから仮想入力画像を生成し、前記現在の入力画像および前記仮想入力画像に基づいて算出した残差マップを用いて、前記挿入オブジェクトを前記現在の入力画像が含むか否かを判定する、請求項1~3のいずれか1つに記載の情報処理装置。 - 前記判定部は、
前記残差マップに含まれるノイズを除去するフィルタリング処理を行った前記残差マップを用いて、前記挿入オブジェクトを前記現在の入力画像が含むか否かを判定する、請求項4に記載の情報処理装置。 - 前記判定部は、
前記残差マップにおける各画素を二値化した二値化画像を生成し、前記二値化画像における連結成分をクラスタリングしたクラスタの領域に基づいて、前記挿入オブジェクトを前記現在の入力画像が含むか否かを判定する、請求項4または5に記載の情報処理装置。 - 前記判定部は、
今回抽出された前記クラスタの領域と、前回抽出された前記クラスタの領域との距離が所定の閾値未満である場合に、前記クラスタの領域が前記挿入オブジェクトの領域であると判定する、請求項6に記載の情報処理装置。 - 前記判定部は、
前記クラスタの領域の画素数が所定の閾値以上である場合に、前記クラスタの領域が前記挿入オブジェクトの領域であると判定する、請求項6または7に記載の情報処理装置。 - 前記生成される仮想入力画像は、前記現在の位置姿勢情報と実質的に同じ位置姿勢情報を有する二次元画像である、請求項4~8のいずれか1つに記載の情報処理装置。
- 前記実空間マップは、
符号付距離および前記符号付距離の信頼度を示す重みを含むボクセルが配列され、
前記第1のマップ更新処理は、
前記現在の位置姿勢情報に対応した前記現在の入力画像と、前記過去の位置姿勢情報に対応した前記過去の入力画像とに基づいて算出した前記符号付距離および前記重みの移動平均を更新後の値とする更新処理であり、
前記第2のマップ更新処理は、
前記現在の位置姿勢情報に対応した前記現在の入力画像に基づいて算出した前記符号付距離および前記重みを更新後の値とする更新処理である、請求項1~9のいずれか1つに記載の情報処理装置。 - コンピュータが実行する情報処理方法であって、
記憶媒体に格納された実空間に対応する実空間マップ、センサが取得した前記実空間を示す現在の入力画像、過去の前記入力画像、前記現在の入力画像に対応する前記センサの現在の位置姿勢情報、および前記過去の入力画像に対応する前記センサの過去の位置姿勢情報を取得する情報取得工程と、
前記実空間マップおよび前記現在の入力画像に基づいて、前記実空間マップに含まれない挿入オブジェクトを前記現在の入力画像が含むか否かを判定する判定工程と、
前記現在の入力画像が前記挿入オブジェクトを含まないという判定結果に基づいて、前記現在の位置姿勢情報および前記過去の位置姿勢情報に従って前記実空間マップを更新する第1のマップ更新処理を実行し、
前記現在の入力画像が前記挿入オブジェクトを含むという判定結果に基づいて、前記現在の位置姿勢情報に従って前記実空間マップを更新する、前記第1のマップ更新処理とは異なる第2のマップ更新処理を実行する更新処理工程とを含む、情報処理方法。 - コンピュータを、
記憶媒体に格納された実空間に対応する実空間マップ、センサが取得した前記実空間を示す現在の入力画像、過去の前記入力画像、前記現在の入力画像に対応する前記センサの現在の位置姿勢情報、および前記過去の入力画像に対応する前記センサの過去の位置姿勢情報を取得する情報取得部と、
前記実空間マップおよび前記現在の入力画像に基づいて、前記実空間マップに含まれない挿入オブジェクトを前記現在の入力画像が含むか否かを判定する判定部と、
前記現在の入力画像が前記挿入オブジェクトを含まないという判定結果に基づいて、前記現在の位置姿勢情報および前記過去の位置姿勢情報に従って前記実空間マップを更新する第1のマップ更新処理を実行し、
前記現在の入力画像が前記挿入オブジェクトを含むという判定結果に基づいて、前記現在の位置姿勢情報に従って前記実空間マップを更新する、前記第1のマップ更新処理とは異なる第2のマップ更新処理を実行する更新処理部と
として機能させるためのプログラムが記憶された記憶媒体。
Priority Applications (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2023520910A JPWO2022239543A1 (ja) | 2021-05-11 | 2022-03-28 | |
US18/556,290 US20240221275A1 (en) | 2021-05-11 | 2022-03-28 | Information processing apparatus, information processing method, and storage medium |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2021-080305 | 2021-05-11 | ||
JP2021080305 | 2021-05-11 |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2022239543A1 true WO2022239543A1 (ja) | 2022-11-17 |
Family
ID=84028211
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2022/015015 WO2022239543A1 (ja) | 2021-05-11 | 2022-03-28 | 情報処理装置、情報処理方法および記憶媒体 |
Country Status (3)
Country | Link |
---|---|
US (1) | US20240221275A1 (ja) |
JP (1) | JPWO2022239543A1 (ja) |
WO (1) | WO2022239543A1 (ja) |
Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2008304268A (ja) * | 2007-06-06 | 2008-12-18 | Sony Corp | 情報処理装置、および情報処理方法、並びにコンピュータ・プログラム |
JP2012103144A (ja) * | 2010-11-11 | 2012-05-31 | Univ Of Tokyo | 3次元環境復元装置、3次元環境復元方法、及びロボット |
JP2017509035A (ja) * | 2014-01-03 | 2017-03-30 | インテル・コーポレーション | 深度カメラを用いたリアルタイム3d再構成 |
US20200097006A1 (en) * | 2017-07-28 | 2020-03-26 | Qualcomm Incorporated | Systems and Methods for Utilizing Semantic Information for Navigation of a Robotic Device |
-
2022
- 2022-03-28 WO PCT/JP2022/015015 patent/WO2022239543A1/ja active Application Filing
- 2022-03-28 US US18/556,290 patent/US20240221275A1/en active Pending
- 2022-03-28 JP JP2023520910A patent/JPWO2022239543A1/ja active Pending
Patent Citations (4)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP2008304268A (ja) * | 2007-06-06 | 2008-12-18 | Sony Corp | 情報処理装置、および情報処理方法、並びにコンピュータ・プログラム |
JP2012103144A (ja) * | 2010-11-11 | 2012-05-31 | Univ Of Tokyo | 3次元環境復元装置、3次元環境復元方法、及びロボット |
JP2017509035A (ja) * | 2014-01-03 | 2017-03-30 | インテル・コーポレーション | 深度カメラを用いたリアルタイム3d再構成 |
US20200097006A1 (en) * | 2017-07-28 | 2020-03-26 | Qualcomm Incorporated | Systems and Methods for Utilizing Semantic Information for Navigation of a Robotic Device |
Also Published As
Publication number | Publication date |
---|---|
US20240221275A1 (en) | 2024-07-04 |
JPWO2022239543A1 (ja) | 2022-11-17 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
WO2020119684A1 (zh) | 一种3d导航语义地图更新方法、装置及设备 | |
Salas-Moreno et al. | Dense planar SLAM | |
JP2021534495A (ja) | ビデオデータを使用するオブジェクトインスタンスのマッピング | |
CN106940704B (zh) | 一种基于栅格地图的定位方法及装置 | |
CN108898676B (zh) | 一种虚实物体之间碰撞及遮挡检测方法及系统 | |
Carozza et al. | Markerless vision‐based augmented reality for urban planning | |
JP4392507B2 (ja) | 3次元サーフェス生成方法 | |
CN110568447A (zh) | 视觉定位的方法、装置及计算机可读介质 | |
CN109521879B (zh) | 交互式投影控制方法、装置、存储介质及电子设备 | |
JP2021522607A (ja) | ポイントクラウドの着色において使用される方法及びシステム | |
JP2011095797A (ja) | 画像処理装置、画像処理方法及びプログラム | |
AU2022345532B2 (en) | Browser optimized interactive electronic model based determination of attributes of a structure | |
CN109906600B (zh) | 模拟景深 | |
JP2001067463A (ja) | 異なる視点からの複数のフェイシャル画像に基づき新たな視点からのフェイシャル画像を生成するフェイシャル画像生成装置及び方法並びにその応用装置及び記録媒体 | |
TW202309834A (zh) | 模型重建方法及電子設備和電腦可讀儲存介質 | |
Shalaby et al. | Algorithms and applications of structure from motion (SFM): A survey | |
US20220180545A1 (en) | Image processing apparatus, image processing method, and program | |
CN113256696A (zh) | 基于自然场景的激光雷达和相机的外参标定方法 | |
WO2022239543A1 (ja) | 情報処理装置、情報処理方法および記憶媒体 | |
JP2023168262A (ja) | データ分割装置及び方法 | |
CN117036653A (zh) | 基于超体素聚类的点云分割方法及系统 | |
WO2023088127A1 (zh) | 室内导航方法、服务器、装置和终端 | |
JP2016527574A (ja) | プリミティブの組を用いてデータをレジストレーションする方法 | |
CN111198563A (zh) | 一种用于足式机器人动态运动的地形识别方法及系统 | |
Afzal et al. | Kinect Deform: enhanced 3d reconstruction of non-rigidly deforming objects |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 22807248 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2023520910 Country of ref document: JP |
|
WWE | Wipo information: entry into national phase |
Ref document number: 18556290 Country of ref document: US |
|
NENP | Non-entry into the national phase |
Ref country code: DE |
|
122 | Ep: pct application non-entry in european phase |
Ref document number: 22807248 Country of ref document: EP Kind code of ref document: A1 |