Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
  • G01B11/00—Measuring arrangements characterised by the use of optical techniques
  • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
  • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object
  • G01B11/254—Projection of a pattern, viewing through a pattern, e.g. moiré
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01B—MEASURING LENGTH, THICKNESS OR SIMILAR LINEAR DIMENSIONS; MEASURING ANGLES; MEASURING AREAS; MEASURING IRREGULARITIES OF SURFACES OR CONTOURS
  • G01B11/00—Measuring arrangements characterised by the use of optical techniques
  • G01B11/24—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures
  • G01B11/25—Measuring arrangements characterised by the use of optical techniques for measuring contours or curvatures by projecting a pattern, e.g. one or more lines, moiré fringes on the object

Definitions

• the present disclosure relates to a photodetection device, a system, and an information processing device.
• Three-dimensional shape measurement processing is sometimes used to obtain such surrounding conditions.
• Three-dimensional shape measurement is achieved by, for example, calculating depth information based on the principle of triangulation from the phase information obtained by reading a predetermined projection pattern onto the subject with an image sensor. can.
• the present disclosure provides a photodetection device that sets an appropriate point cloud density and achieves highly accurate and high-speed three-dimensional shape measurement.
• the photodetection device includes a light receiving sensor, a processing circuit, and a register.
• the light receiving sensor acquires pattern information projected onto the subject.
• the processing circuit acquires depth information based on the pattern information, sets a point cloud density for each area in the depth information, sets a point cloud based on the point cloud density, and performs processing related to the point cloud. Output information.
• the register stores parameters for processing in the processing circuit and control signals for the processing circuit.
• the processing circuit may set the point cloud density based on the pattern information.
• the processing circuit may generate a mask based on the pattern information, and set the point cloud density based on the mask.
• the processing circuit may extract an edge region from the pattern information and generate the mask based on the edge information.
• the processing circuit may extract a flat area from the pattern information and generate the mask based on the information on the flat area.
• the processing circuit may obtain information about the flat area based on the reliability map and the edge area, and generate the mask.
• the processing circuit may generate the mask based on the reliability map and the edge region.
• the processing circuit may generate the reliability map based on a region in which a pattern is projected onto the subject.
• the processing circuit may generate the mask by calculating the product of the reliability map indicating a region where the pattern is projected onto the subject and information obtained by inverting the edge region.
• the processing circuit may set the point cloud density lower for the flat area than for the edge area.
• It may further include a light emitting element that projects a phase shift pattern onto the subject, and the light receiving sensor may acquire reflected light from the subject onto which the phase shift pattern is projected as the pattern information.
• the system includes one or more solid-state imaging devices including one or more of the photodetection devices described above, and position and position information based on depth information in a point cloud acquired from the solid-state imaging device.
• the present invention includes an estimation unit that acquires a posture, and a register control unit that transmits parameters and control related to the point cloud acquisition process to a register of the solid-state imaging device.
• an information processing device includes a processing circuit.
• the processing circuit acquires depth information based on the acquired pattern information on the subject, sets a point cloud density for each region in the depth information, sets a point cloud based on the point cloud density, and sets the point cloud density based on the point cloud density. Output information about point clouds.
• FIG. 1 is a block diagram schematically showing a system according to an embodiment.
• 1 is a flowchart illustrating an example of processing of a photodetection device according to an embodiment.
• FIG. 3 is a diagram illustrating an example of a phase pattern to be projected according to an embodiment.
• FIG. 3 is a diagram illustrating an example of photographed pattern information according to an embodiment.
• FIG. 3 is a diagram illustrating an example of a reconstructed phase image according to an embodiment.
• FIG. 3 is a diagram illustrating an example of a depth image according to an embodiment.
• FIG. 3 is a diagram illustrating an example of an edge region according to an embodiment.
• FIG. 3 is a diagram illustrating an example of flat area information according to an embodiment.
• FIG. 3 is a diagram illustrating an example of thinning according to an embodiment.
• FIG. 3 is a diagram illustrating an example of thinning according to an embodiment.
• FIG. 3 is a diagram illustrating an example of thinning according to an embodiment.
• FIG. 3 is a diagram illustrating an example of an output obtained by controlling the density of a point group according to an embodiment.
• FIG. 1 is a block diagram schematically showing a system 1 according to an embodiment.
• the system 1 includes a solid-state imaging device 2 and a post-processing unit 3.
• the system 1 acquires information using, for example, a solid-state imaging device 2 .
• the system 1 uses the post-processing unit 3 to estimate the three-dimensional shape of the object, or the position and orientation of a vehicle, robot, etc. equipped with the solid-state imaging device 2.
• one solid-state imaging device 2 is shown in the system 1 in FIG. 1, the system is not limited to this, and a plurality of solid-state imaging devices 2 may be provided.
• the solid-state imaging device 2 includes a photodetector 20 and an interface (hereinafter referred to as I/F 210).
• the photodetector 20 performs signal processing based on the intensity of the received light, and outputs the signal processing results to the outside of the solid-state imaging device 2 via the I/F 210 .
• the solid-state imaging device 2 is equipped with a storage circuit such as a memory or a storage at least either inside or outside the photodetection device 20 as necessary.
• a storage circuit such as a memory or a storage at least either inside or outside the photodetection device 20 as necessary.
• information processing by software is concretely realized using hardware resources including general-purpose processing circuits, programs, etc. may be stored in these storage circuits.
• the photodetector 20 includes a light receiving section 200, a control circuit 202, a register 204, and a processing circuit 206.
• the photodetector 20 may be configured to include a light receiving element included in a general camera module, a processing circuit capable of performing the processing described below, and the like.
• the light receiving section 200, the control circuit 202, the register 204, and the processing circuit 206 may be mounted on stacked semiconductors.
• the light receiving unit 200 includes, for example, a light receiving element (photoelectric conversion element) such as a PD (Photo Diode), and a pixel circuit that appropriately outputs an analog signal output from the light receiving element.
• the output from the pixel circuit may be an analog signal or a digital signal after analog-to-digital conversion.
• the light receiving unit 200 includes, for example, a light receiving sensor whose light receiving area is defined by a pixel array in which light receiving elements are arranged in a two-dimensional array.
• the control circuit 202 is a circuit that executes control of the solid-state imaging device 20.
• the register 204 is, for example, a register that stores predefined parameters or parameters set by external control.
• the control circuit 202 controls the light receiving section 200 or the processing circuit 206 based on control signals or parameters stored in a register.
• the processing circuit 206 is a circuit that executes various signal processing in the photodetector 20 and the solid-state imaging device 2.
• the processing circuit 206 may be a general-purpose processor capable of executing information processing using software, or may be a circuit limited to use, such as an ASIC (Application Specified Integrated Circuit). Alternatively, it may be a programmable circuit such as FPGA (Field-Programmable Gate Array).
• the photodetector 20 outputs a signal processed by the processing circuit 206.
• the solid-state imaging device 2 outputs necessary data to the outside via the I/F 210. This necessary data may include data processed by the processing circuit 206 .
• the post-processing unit 3 is a unit that executes processing based on the data output from the solid-state imaging device 2.
• the post-processing unit 3 includes, for example, a simple configuration: an estimation section 300 , a register control section 302 , and a mechanism control section 304 .
• the post-processing unit 3 estimates the position and orientation information of the housing of the vehicle, robot, etc. in which the solid-state imaging device 2 is mounted, generates appropriate control signals for this housing, and controls the housing appropriately. do.
• the estimation unit 300 includes various circuits, and acquires the three-dimensional shape of the object based on the signal acquired from the solid-state imaging device 2, and acquires information on the position and orientation of the above-mentioned housing. .
• the register control unit 302 sets appropriate parameters in the register inside the photodetector 20 based on the estimation result of the estimation unit 300 or the data output from the processing circuit 206. As another example, the register control unit 302 may write a signal for controlling the photodetector 20 into the register 204 .
• the mechanism control unit 304 controls the housing so that it can move safely, for example, based on the position and orientation information estimated by the estimation unit 300.
• the mechanism control unit 304 may control the imaging direction of the solid-state imaging device 2 based on the position and orientation information estimated by the estimation unit 300 .
• the solid-state imaging device 2 may further include a light emitting unit (light emitting element) inside or outside the photodetection device 20 that projects a predetermined pattern onto the subject.
• this light emitting unit may be provided at any location inside or outside the system.
• the photodetector 20 acquires an image of the phase pattern projected through the light emitting element on the subject.
• System 1 estimates the three-dimensional shape of the object or the position and orientation information of the housing and achieves appropriate control using the configuration described above. Next, the processing of the photodetector 2 will be explained.
• FIG. 2 is a flowchart showing the processing of the photodetector 20 according to one embodiment.
• the solid-state imaging device 2 or the system 1 projects a projection having a phase pattern shown in FIG. 3 onto the subject.
• the photodetector 20 acquires information about the subject onto which such a phase pattern is projected.
• the pattern to be projected may include a pattern having uniform intensity for use in removing the influence in the normal direction.
• the light receiving unit 200 photographs the phase pattern reflected by the subject and acquires it as pattern information for each projected phase information (S100).
• the processing circuit or the pixel circuit may output the result of adding an offset to distinguish it from the non-projection range if the phase of the projection range starts from 0 as necessary.
• FIG. 4 is a diagram showing an example of pattern information obtained by photographing a subject onto which a phase pattern is projected.
• the processing circuit 206 acquires a phase image based on the pattern information acquired by the light receiving unit 200 (S102).
• This phase image is an image obtained by a general method based on the plurality of photographed pattern information shown in FIG. 4.
• the processing circuitry obtains this phase image using a phase shift method.
• FIG. 5 is a diagram showing an example of a reconstructed phase image when the pattern of FIG. 4 is acquired.
• the processing circuit 206 may apply filter processing such as a noise removal filter to the acquired phase image as necessary.
• the processing circuit 206 may perform noise removal by using, by way of non-limiting example, a moving average filter, a median filter, etc. on the phase image.
• the processing circuit 206 generates a depth image from the acquired phase image or the noise-removed phase image (S104).
• FIG. 6 is a diagram showing an example of a depth image generated by the processing circuit 206.
• the processing circuit 206 may acquire the depth image using a phase shift method, as a non-limiting example. This process may also be executed by the processing circuit 206 using a general method.
• the processing circuit 206 In parallel with the processing in S104, or before or after the processing in S104, the processing circuit 206 generates a mask based on the phase image (i.e., pattern information) (S106). As an example, the processing circuit 206 obtains edge information as shown in FIG. 7 from the phase image, and generates a mask based on this edge information.
• the processing circuit 206 may obtain the edge image by using a Sobel filter, a Laplacian filter, etc., as a non-limiting example. In the figure, white areas indicate edge areas.
• the processing circuit 206 acquires flat area information by integrating an image obtained by inverting the acquired edge information and a reliability map.
• the processing circuit 206 generates a mask from the obtained flat area information. Further, the processing circuit 206 may generate a mask from information on the edge region. That is, the processing circuit 206 may generate a mask from either edge region information or flat region information, or may generate a mask corresponding to each region from both.
• the processing circuit 206 may generate a reliability map in which a region of light-receiving pixels where information can be appropriately acquired for the region on which the phase pattern is projected is a region where highly reliable information can be acquired.
• the processing circuit 206 may generate a predetermined image in an image obtained by applying a low-pass filter to the image obtained by the light receiving element when projecting the uniform pattern in FIG. 3 (another phase pattern may also be used).
• a reliability map may be generated in which regions having pixel values greater than or equal to the pixel value are defined as regions with high reliability.
• the processing circuit 206 can generate a mask indicating a flat area by obtaining the product of the generated reliability map and the inverted edge information.
• FIG. 8 is a diagram showing an example of a flat area obtained by the above calculation. The flat area shown in this figure may be used as a mask area. In the figure, white areas indicate flat areas.
• the processing circuit 206 After generating the mask, the processing circuit 206 generates a thinning pattern that sets the density of the point cloud that determines the density of the data to be output to the post-processing unit 3 (S108).
• This thinning pattern is a pattern for controlling the density of points that output depth information or information related to depth information.
• the estimation unit 300 in the system 1 receives appropriate information about points in the acquired image as input depending on the estimation method.
• the processing circuit 206 determines the point from which the point cloud information required by the estimation unit 300 is output based on the mask generated in S106.
• Information in this regard may be determined based on, for example, PLY (Polygon File Format) or a format similar to PLY.
• the processing circuit performs control so that more information about the edge region is output than information about the flat region.
• the processing circuit 206 may output depth information, etc. for all pixels for edge regions, and may output depth information, etc. for thinned out pixels for flat regions.
• FIG. 9 is a diagram illustrating an example of thinning according to an embodiment.
• the processing circuit 206 may perform control to thin out the information in the shaded area in the figure in a flat area and acquire information on other pixels.
• the rate of outputting information in a flat area is approximately 1/2.
• FIG. 10 is a diagram showing another example of thinning according to one embodiment.
• the processing circuit 206 may control to thin out the information in the diagonally shaded area in the flat area and acquire the other pixel information.
• the rate of outputting information in a flat area is approximately 1/3.
• FIG. 11 is a diagram showing another example of thinning according to one embodiment.
• the processing circuit 206 may control to thin out the information in the diagonally shaded area in the flat area and acquire the other pixel information.
• the rate of outputting information in a flat area is approximately 1/4.
• the processing circuit 206 determines the density of the point cloud for outputting information based on the mask and based on a preset thinning rate, as in some of the examples above.
• the thinning rate is determined uniformly in a flat area within the image, but the thinning rate is not limited to this.
• the processing circuit 206 may calculate the area of the flat region and change the thinning rate based on this area.
• the processing circuit 206 may be set so as not to thin out too much in a flat area where the area is narrow, and to increase the thinning rate in an area that is wider than the narrow area.
• the processing circuit 206 acquires and outputs data in which the point cloud densities of the edge region and the flat region differ, as point cloud information, based on the thinning pattern generated in S108 (S110).
• the estimation unit 300 can restore the three-dimensional shape using this point cloud data.
• FIG. 12 is a diagram showing an example of a point cloud output from the photodetector 20 according to an embodiment. Points that output point cloud data are represented in black, and points that do not output point cloud data are represented in white. As an example, the output rate of the point cloud in a flat area is set to 1/3. As shown in FIG. 12, it is shown that a point group with high density in edge regions and low density in flat regions can be appropriately output.
• the present embodiment it is possible to appropriately output the point cloud data of the edge region and to reduce and output the point cloud data of the flat region.
• the amount of data can be reduced by appropriately setting the density of the point cloud for each region.
• the time and calculation costs for acquiring point cloud data are reduced on the photodetector side, and the memory access time, which can be a bottleneck, is significantly reduced on the post-processing unit side. can be reduced to Furthermore, it is possible to reduce the latency in the photodetector, and it is possible to further improve the accuracy of shape restoration by, for example, increasing the frame rate.
• the photodetection device has been described as having a configuration including a light receiving element, as can be understood from the description, the present disclosure naturally includes an information processing apparatus that executes processing without a light receiving element.
• a light receiving sensor, a processing circuit; register and Equipped with The light receiving sensor acquires pattern information projected on the subject,
• the processing circuit includes: Obtaining depth information based on the pattern information, Setting a point cloud density for each region in the depth information, setting a point cloud based on the point cloud density; outputting information regarding the point cloud;
• the register is storing parameters for processing in the processing circuit and control signals for the processing circuit; Photodetection device.
• the processing circuit includes: setting the point cloud density based on the pattern information; The photodetector according to (1).
• the processing circuit includes: generating a mask based on the pattern information; setting the point cloud density based on the mask; The photodetector according to (2).
• the processing circuit includes: extracting an edge region from the pattern information; generating the mask based on the edge information; The photodetector according to (3).
• the processing circuit includes: Extracting a flat area from the pattern information, generating the mask based on information of the flat area; The photodetector according to (4).
• the processing circuit includes: obtaining information on the flat area based on the reliability map and the edge area and generating the mask; The photodetection device according to (5).
• the processing circuit includes: generating the mask based on the confidence map and the edge region; The photodetection device described in (5) or (6).
• the processing circuit includes: generating the reliability map based on a region in which a pattern is projected onto the subject; The photodetector according to (7).
• the processing circuit includes: generating the mask by calculating the product of the reliability map indicating a region where the pattern is projected onto the subject and information obtained by inverting the edge region;
• the optical information device according to (8).
• the processing circuit includes: setting the point cloud density lower for the flat area than for the edge area; The photodetector according to any one of (5) to (9).
• the photodetector according to any one of (1) to (10).
• the processing circuit includes: Acquire depth information based on the acquired pattern information of the subject, Setting a point cloud density for each region in the depth information, setting a point cloud based on the point cloud density; outputting information regarding the point cloud; Information processing device.

Landscapes

• Engineering & Computer Science (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Length Measuring Devices By Optical Means (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Publications (1)

Family

ID=89118267

Family Applications (1)

Country Status (3)

Citations (3)

• 2023
  • 2023-05-30 JP JP2024526392A patent/JPWO2023238741A1/ja active Pending
  • 2023-05-30 WO PCT/JP2023/020141 patent/WO2023238741A1/ja not_active Ceased
  • 2023-05-30 US US18/870,806 patent/US20250369753A1/en active Pending

Patent Citations (3)

Also Published As

Similar Documents

Legal Events