JP2022176974A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an imaging apparatus that does not require much preparation to acquire distance information.

SOLUTION: The imaging apparatus according to an embodiment includes: imaging means for imaging a measurement target; projection means for projecting light to the measurement target; light receiving means for receiving the light reflected by the measurement target; and acquisition means for acquiring distance information to the measurement target on the basis of the light received by the light receiving means. The imaging means, the light projection means, the light receiving means, and the acquisition means are integrally provided in a common housing.

SELECTED DRAWING: Figure 2

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to an imaging device and an imaging processing method.

Real objects, buildings, and the entire space including them are measured by imaging with a camera or by irradiating laser light, and the 3D structure in the real space is reconstructed as electronic data from the measurement results. Three-dimensional model reconstruction techniques are known. Various targets, from those targeting small objects such as chairs, desks, and models, to those targeting multiple large objects and open spaces such as construction sites and tourist spots. possible to use.

There is also a technique for acquiring depth information by irradiating patterned light on areas with no pattern, such as monochromatic floors and walls. There is also a document disclosing a LiDAR (Light Detection and Ranging) device that receives projected light (see Patent Document 1).

However, conventionally, an imaging device and an irradiation device that emits laser light or the like are separate entities, and complex preparations are required for a system that restores a 3D model (for example, alignment of the irradiation direction and the imaging direction of the imaging device, etc.). ).

SUMMARY OF THE INVENTION It is an object of the present invention to provide an imaging apparatus that can be easily prepared for acquiring distance information.

In order to solve the above-described problems and achieve the object, an imaging apparatus according to an embodiment of the present invention includes imaging means for imaging an object to be measured, projection means for projecting light onto the object to be measured, and light receiving means for receiving the light reflected from the light receiving means; and acquisition means for acquiring information on the distance to the measurement object based on the light received by the light receiving means, wherein the imaging means and the projection means and the light-receiving means and the acquiring means are integrally provided in a common housing.

ADVANTAGE OF THE INVENTION According to this invention, it is effective in the ability to implement|achieve the imaging device with which the preparation for acquiring distance information is easy.

FIG. 1 is a diagram showing an example of the appearance of an imaging device according to an embodiment. FIG. 2 is a diagram for explaining the configuration of the imaging device. FIG. 3 is a diagram illustrating an example of a configuration of processing blocks of a processing circuit; FIG. 4 is a diagram showing an example of a Ctof cost curve function. FIG. 5 is a diagram schematically showing reprojection processing. FIG. 6 is an explanatory diagram of adding segmentation information indicating an object to an input image of a measurement range. FIG. 7 is a flow chart showing an example of the operation of the processing circuit of the imaging device. FIG. 8 is a diagram for explaining the configuration of an imaging device that performs measurement using a phase difference detection method according to Modification 1. As shown in FIG. FIG. 9 is a diagram schematically showing an example of the configuration of the projection section. FIG. 10 is a diagram illustrating an example of the configuration of a MEMS mirror. FIG. 11 is a diagram for explaining an example of the configuration of an imaging device according to Modification 2. As shown in FIG. FIG. 12 is an explanatory diagram of a projection system such as EquiRectangular.

Exemplary embodiments of an imaging device and an imaging processing method will be described in detail below with reference to the accompanying drawings.

[First embodiment]
1 and 2 are diagrams showing an example of the configuration of an imaging device according to the first embodiment. FIG. 1 is a diagram showing an example of the appearance of an imaging device. FIG. 2 is a diagram for explaining the configuration of the imaging device. FIG. 2 shows the internal configuration of the imaging apparatus of FIG. Furthermore, in FIG. 2, a diagram for explaining the path of light is superimposed. First, the configuration of the imaging apparatus will be described with reference to FIGS. 1 and 2. FIG.

The imaging device 1 includes an imaging unit (camera) 11, a projection unit (a portion corresponding to a light emitting unit of a distance sensor) 12 that projects light other than visible light, and distance information based on the light projected by the projection unit 12. A distance information acquisition unit (portion corresponding to the light receiving unit of the distance sensor) 13 for acquiring distance information is provided integrally with the housing 10 . Each part is electrically connected to the processing circuit 14 inside the housing 10 by a synchronization signal line L, and operates in synchronization with each other. Here, the imaging unit 11 corresponds to "imaging means", the projection unit 12 corresponds to "projecting means", and the distance information acquisition unit 13 corresponds to "light receiving means". The processing circuit 14 corresponds to "acquisition means".

A photographing switch 15 is used by the user to input a photographing instruction signal to the processing circuit 14 . Based on the photographing instruction, the processing circuit 14 controls each part to acquire data of the RGB image and distance information, and further functions as "information processing means" to convert the acquired data of the distance information into a high-level image based on the data of the RGB image and the distance information. A process of reconstructing the density three-dimensional point cloud data is performed. Although it is possible to construct 3D point cloud data by using the distance information data as it is, in that case, the accuracy of the 3D point cloud data is limited to the number of pixels (resolution) of the distance information acquisition unit 13. . This example also shows the processing for reconstructing it into high-density three-dimensional point cloud data. The reconstructed data is output to an external PC or the like via a portable recording medium or communication, and used to display the three-dimensional reconstruction model.

Power is supplied to each unit and the processing circuit 14 from a battery housed inside the housing 10 . In addition to this, a configuration in which power is supplied from the outside of the housing 10 through a connection cord may be employed.

The imaging unit 11 has an imaging element 11a, a fisheye lens (wide-angle lens) 11b, and the like. The projection unit 12 has a light source unit 12a, a wide-angle lens 12b, and the like. The distance information acquisition unit 13 has a TOF (Time Of Fright) sensor 13a, a wide-angle lens 13b, and the like. Although not shown, each part may constitute an optical system such as a prism and a lens group. For example, the imaging unit 11 may include an optical system for forming an image of the light collected by the fisheye lens 11b on the imaging device 11a. Further, the projection unit 12 may include an optical system that guides the light from the light source unit 12a to the wide-angle lens 12b. Further, an optical system may be configured in the distance information acquisition section 13 for forming an image of the light collected by the wide-angle lens 13b on the TOF sensor 13a. Each optical system may be appropriately determined according to the configuration and arrangement of the imaging device 11a, light source unit 12a, TOF sensor 13a, etc. Here, optical systems such as prisms and lens groups will be omitted from the description. .

The imaging element 11a, the light source section 12a, and the TOF sensor 13a are integrally housed inside the housing 10. As shown in FIG. The fisheye lens 11b, the wide-angle lens 12b, and the wide-angle lens 13b are provided on the first surface of the housing 10, respectively. On the first surface, the inner ranges of each of the fisheye lens 11b, the wide-angle lens 12b, and the wide-angle lens 13 are open.

The imaging device 11a is a two-dimensional resolution image sensor (area sensor). The imaging element 11a has an imaging area in which a large number of light receiving elements (photodiodes) for each pixel are arranged in a two-dimensional direction. In the imaging area, R (red), G (green), and B (blue) color filters such as a Bayer array are provided to receive visible light, and light passing through the color filters is stored in photodiodes. be. Here, an image sensor with a large number of pixels is used so that a wide-angle (for example, the range of a 180-degree hemispherical sphere with the imaging direction shown in FIG. 2 as the front) can be acquired with high resolution. . The imaging element 11a converts the light imaged on the imaging area into an electric signal by the pixel circuit of each pixel, and outputs a high-resolution RGB image. The fisheye lens 11b collects light from a wide angle (for example, a 180-degree hemispherical range with the imaging direction shown in FIG. 2 as the front) and forms an image of the light on the imaging area of the imaging device 11a.

The light source unit 12a is a semiconductor laser, and emits laser light in a wavelength band other than the visible light range used for distance measurement (infrared as an example). One semiconductor laser may be used for the light source unit 12a, or a plurality of semiconductor lasers may be used in combination. As the semiconductor laser, a surface emitting semiconductor laser such as a VCSEL (Vertical Cavity Surface Emitting LASER) may be used. In addition, the light of a semiconductor laser is shaped vertically by an optical lens, and the vertically elongated light is scanned in the one-dimensional direction of the measurement range with an optical deflection element such as a MEMS (Micro Electro Mechanical Systems) mirror. may be configured. In the present embodiment, as the light source unit 12a, a form is shown in which light from the semiconductor laser LA is spread over a wide angle range via a wide-angle lens 12b without using an optical deflection element such as a MEMS mirror.

The wide-angle lens 12b of the light source unit 12a has a function of widening the light emitted from the light source unit 12a into a wide-angle range (for example, a hemispherical range of 180 degrees around the imaging direction shown in FIG. 2).

The wide-angle lens 13b of the distance information acquisition unit 13 converts the reflected light of the light from the light source unit 12a projected by the projection unit 12 into a wide-angle measurement range (for example, a hemisphere with a circumference of 180 degrees with the imaging direction shown in FIG. 2 as the front). ), and forms an image on the light receiving area of the TOF sensor 13a. The measurement range includes one or a plurality of objects to be projected (for example, buildings), and light reflected by the objects to be projected (reflected light) enters the wide-angle lens 13b. Reflected light may be captured by, for example, providing a filter that cuts off light having wavelengths in the infrared region or higher over the entire surface of the wide-angle lens 13b. In addition, as long as light in the infrared region is incident on the light receiving area, means such as a filter for passing light in the infrared region may be provided in the optical path from the wide-angle lens 13b to the light receiving area.

The TOF sensor 13a is a two-dimensional resolution optical sensor. The TOF sensor 13a is light-receiving means having a light-receiving area in which a large number of light-receiving elements (photodiodes) are arranged two-dimensionally. The TOF sensor 13a receives the reflected light of each area (each area is also called a position) in the measurement range with the light receiving element corresponding to each area, and measures the distance to each area based on the light detected by each light receiving element. (calculate.

For example, in the configuration for measuring the distance by the pulse method shown in FIG. In synchronism, the TOF sensor 13a measures the time (t) required to receive the reflected pulse P2, which is the reflected light of the irradiation pulse P1 emitted from the light source unit 12a. When this method is adopted, for example, a TOF sensor 13a having a circuit for measuring time mounted on the output side of the light receiving element is used. Each circuit converts the time taken by the light source unit 12a from emitting the irradiation pulse P1 to receiving the reflected pulse P2 for each light-receiving element into a distance, and obtains the distance to each area. In this way, the TOF sensor 13a is driven in synchronization with the irradiation of light from the projection unit 12, and the distance corresponding to each pixel is calculated from the time required for each light receiving element (corresponding to a pixel) to receive the reflected light. and outputs distance information (also referred to as “distance image” or “TOF image”) in which pixel information is associated with information indicating the distance to each area within the measurement range. The number of areas into which the measurement range can be divided is determined by the resolution of the light receiving areas. Therefore, if a low-resolution image is used for miniaturization, the number of pixel information in the range image is reduced, and the number of three-dimensional point groups is also reduced. This method is suitable for widening the angle of view of the imaging device 1 because it can output strong light using peak light. In addition, when a MEMS mirror or the like is used to wave (scan) light, it is possible to irradiate strong light to a long distance while suppressing spread, which leads to an increase in measurement distance. In this case, the arrangement relationship is such that the laser light emitted from the light source unit 12a is scanned (deflected) toward the wide-angle lens 12b by the MEMS mirror.

Although it is desirable that the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 match, for example, 180 degrees or more, they do not necessarily have to match. The effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 may be reduced as necessary. In this embodiment, the imaging unit 11 and the distance information acquisition unit 13 reduce the effective pixels within a range of, for example, 100 degrees to 180 degrees so that nothing interferes with the angle of view. Further, the resolution of the TOF sensor 13a may be set lower than the resolution of the imaging device 11a in order to prioritize miniaturization of the imaging device 1. FIG. In the case of the pulse method as in this example, it is difficult to provide the TOF sensor 13a with high resolution. By making the TOF sensor 13a have a resolution lower than that of the imaging device 11a, it is possible to suppress an increase in the size of the light receiving area. For this reason, the TOF sensor 13a has a low resolution, and the three-dimensional point cloud obtained by the TOF sensor 13a has a low density. can be converted. The process of converting into a high-density three-dimensional point group in the processing circuit 14 will be described later.

In the present embodiment, as an example, the imaging element 11a, the light source section 12a, and the TOF sensor 13a are provided so as to line up in a straight line in the longitudinal direction of the housing 10. FIG. The imaging area (imaging surface) of the imaging element 11a and the light receiving area (light receiving surface) of the TOF sensor 13a may be arranged in a direction perpendicular to the longitudinal direction as shown in FIG. It may be arranged in the longitudinal direction by providing a prism or the like that converts the optical path by 90 degrees and enters the light. In addition, they may be arranged in any direction depending on the configuration. That is, the imaging device 11a, the light source unit 12a, and the TOF sensor 13a are arranged so as to cover the same measurement range. An imaging unit 11, a projection unit 12, and a distance information acquisition unit 13 are arranged from one side of the housing 10 toward the measurement range. In this case, the imaging device 11a and the TOF sensor 13a should be arranged on the same base line so as to achieve parallel stereo. By arranging for parallel stereo, even if there is only one imaging device 11a, it is possible to obtain parallax data using the output of the TOF sensor 13a. The light source unit 12a is configured to irradiate the measurement range of the TOF sensor 13a with light.

(processing circuit)
Next, processing of the processing circuit 14 will be described. A TOF image obtained only by the TOF sensor 13a has a low resolution as it is. For this reason, this example shows an example in which the resolution is increased by the processing circuit 14 to reconstruct high-density three-dimensional point cloud data. Part or all of the processing described below as the "information processing means" in the processing circuit 14 may be performed by an external device.

FIG. 3 is a diagram showing an example of a configuration of processing blocks of the processing circuit 14. As shown in FIG. Processing circuit 14 shown in FIG. It has a projection processing unit 147 , a semantic segmentation unit 148 , a parallax calculation unit 149 and a three-dimensional reconstruction processing unit 150 . In FIG. 3, solid line arrows indicate the flow of signals, and broken line arrows indicate the flow of data.

Upon receiving an ON signal (shooting start signal) from the shooting switch 15, the control section 141 outputs a synchronization signal to the image sensor 11a, the light source section 12a, and the TOF sensor 13a, and controls the processing circuit 14 as a whole. The control unit 141 first outputs a signal instructing the light source unit 12a to emit an ultrashort pulse, and at the same timing outputs a signal instructing the TOF sensor 13a to generate TOF image data. Furthermore, the control unit 141 outputs a signal instructing the imaging element 11a to perform imaging. The imaging by the imaging element 11a may be performed during the period during which the ultrashort pulse is emitted from the light source section 12a, or during the most recent period before or after that period.

The RGB image data acquisition unit 142 acquires RGB image data captured by the image sensor 11a based on an image capturing instruction from the control unit 141 . The monochrome processing unit 143 performs processing for aligning data types for matching processing with the TOF image data obtained from the TOF sensor 13a. In this example, the monochrome processing unit 143 performs processing for converting RGB image data into a monochrome image.

The TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensor 13a based on the TOF image data generation instruction from the control unit 141 .

The resolution enhancement unit 145 treats the TOF image data as a monochrome image and enhances the resolution thereof. Specifically, the resolution enhancement unit 145 replaces the distance value associated with each pixel of the TOF image data with a monochrome image value (grayscale value) and uses it. Furthermore, the resolution enhancement unit 145 enhances the resolution of the monochrome image to the resolution of the RGB image data obtained from the imaging element 11a. Conversion to high resolution is performed, for example, by performing normal up-conversion processing. As another conversion method, for example, a plurality of frames of continuously generated TOF image data may be acquired, and the distances between adjacent points may be added using them to perform super-resolution processing.

The matching processing unit 146 extracts feature amounts of textured portions of the monochrome image obtained by increasing the resolution of the TOF image data and the monochrome image of the RGB image data, and performs matching processing using the extracted feature amounts. For example, the matching processing unit 146 extracts edges from each monochrome image and performs matching processing between the extracted edge information. As another method, for example, matching processing may be performed by a method such as SIFT, which converts changes in texture into features. Here, matching processing means searching for corresponding pixels.

As a specific method of matching processing, there is block matching, for example. Block matching consists of pixel values cut out as M×M (M is a positive integer) pixel size blocks around the reference pixel, and In this method, the similarity of pixel values cut out as a block of M×M pixels is calculated, and the central pixel with the highest similarity is taken as the corresponding pixel.

There are various methods of calculating similarity. For example, (Formula 1) shown below is a formula representing a normalized autocorrelation coefficient C _NCC (NCC: Normalized Correlation Coefficient). A higher value of the normalized autocorrelation coefficient C _NCC indicates a higher degree of similarity, and becomes 1 when the pixel values of the blocks are completely matched.

The center pixel of the block showing the highest coefficient value in the search range is taken as the corresponding pixel.

Further, since the distance data of the textureless area can be obtained from the TOF image data, the matching process may be weighted according to the area. For example, in the calculation of Equation 1, weighting may be performed on locations other than edges (textureless regions) as shown in Equation 2 below.

Also, instead of Equation 1, a selective normalized correlation (SCC: Selective Correlation Coefficient) such as Equation 3 below may be used.

Equation 3 is basically the same as Equation 1, except that it passes through mask coefficients called selection functions _Cn . In general, _Cn is defined as _bn and _b'n as incremental codes for each image, respectively. In other words, edges are more emphasized in the calculation. By using the inverse, the TOF data can be made available in areas that are not edges. Here, C _tof is a weighted cost curve function with a width around the distance position detected by the TOF sensor 13a. FIG. 4 is an example of a C _tof cost curve function.

The reprojection processing unit 147 performs processing for reprojecting the TOF image data indicating the distance of each position in the measurement range onto the two-dimensional coordinates (screen coordinate system) of the imaging unit 11 . To re-project is to determine at which coordinates the three-dimensional point calculated by the TOF sensor 13a appears in the image of the imaging element 11a. The TOF image data indicates the positions of three-dimensional points in a coordinate system centered on the distance information acquisition section 13 (mainly the wide-angle lens 13b). Therefore, the three-dimensional point indicated by the TOF image data is reprojected onto the coordinate system centered on the imaging unit 11 (mainly the fisheye lens 11b). For example, the reprojection processing unit 147 translates the coordinates of the three-dimensional point of the TOF image data to the coordinates of the three-dimensional point centered on the imaging unit 11, and after the translation, the two-dimensional coordinate system indicated by the RGB image data. (screen coordinate system). FIG. 5 is a diagram schematically showing reprojection processing. The process of converting to the screen coordinate system will be described using Equation 4.

In Expression 4, (X, Y, Z) represent three-dimensional coordinates in the coordinate system of the imaging unit 11 . The 3×3 matrix is called a projection matrix, and the focal length (f _x , f _y ) in the (x, y) direction of the screen coordinate system and the shift (c _x , c _y ) of the screen coordinate system with respect to the optical center and These are coordinates (reprojection coordinates) when λ(u, v) obtained by Equation 2 is transformed (reprojection) into the screen coordinate system.

The parallax calculator 149 calculates the parallax of each position from the difference in distance from the corresponding pixel obtained by the matching process.

In the parallax matching process, the reprojection coordinates converted by the reprojection processing unit 147 are used to search for pixels around the position of the reprojection coordinates. It becomes possible to acquire distance information.

Further, segmentation data obtained by the semantic segmentation processing of the semantic segmentation unit 148 may be used for the parallax matching processing. In that case, more detailed and high-resolution distance information can be obtained.

In addition, parallax matching processing is performed only for edges and only portions with strong feature amounts, and for other portions, TOF image data is also used, for example, RGB image features and probabilistic methods are used to perform propagation processing. may

The semantic segmentation unit 148 uses deep learning to assign a segmentation label indicating an object to the input image of the measurement range, as shown in FIG. 6, for example. As a result, each pixel of the TOF image data can be constrained to one of a plurality of distance areas divided by distance, thereby further increasing the reliability of calculation.

The three-dimensional reconstruction processing unit 145 acquires RGB image data from the RGB image data acquisition unit 142, reconstructs three-dimensional data based on the distance information output by the parallax calculation unit 149, and assigns color information to each three-dimensional point. is added to output a high-density 3D point cloud.

(Operation of processing circuit)
FIG. 7 is a flow chart showing an example of the operation of the processing circuit 14 of the imaging device 1. As shown in FIG. When the photographing switch 15 is turned on by the user and a photographing instruction signal is input, the control unit 141 of the processing circuit 14 performs an operation of generating a high-density three-dimensional point cloud by the following method.

First, the control unit 141 drives the light source unit 12a, the TOF sensor 13a, and the imaging device 11a to photograph the measurement range (step S1). Driven by the control unit 141, the light source unit 12a emits infrared light (irradiation pulse), and the TOF sensor 13a receives the reflected pulse, which is the reflected light. In addition, the imaging device 11a captures an image of the measurement range at the timing when the driving of the light source unit 12a is started or in the period immediately preceding it.

Next, the TOF image data acquisition unit 144 acquires TOF image data indicating the distance of each position in the two-dimensional area from the TOF sensor 13a (step S2). Further, the RGB image data acquisition unit 142 acquires RGB image data of the measurement range from the imaging element 11a (step S3). Note that steps S2 and S3 may be performed in the reverse order.

Next, the monochrome processing unit 143 converts the RGB image data into a monochrome image (step S4). The TOF image data and the RGB image data are different in data type between the distance data and the RGB data, respectively, and matching cannot be performed as they are. Therefore, each data is first converted into a monochrome image. As for the TOF image data, before the resolution is increased, the resolution increasing unit 145 converts the value indicating the distance of each pixel by replacing it with the value of the monochrome image.

Next, the resolution enhancement unit 145 enhances the resolution of the TOF image data (step S5).

Next, the matching processing unit 146 extracts feature amounts of textured portions of each monochrome image, and performs matching processing using the extracted feature amounts (step S6).

Next, the parallax calculator 149 calculates the parallax of each position from the distance deviation of the corresponding pixels (step S7).

Then, the three-dimensional reconstruction processing unit 145 acquires the RGB image data from the RGB image data acquisition unit 142, reconstructs the three-dimensional data based on the distance information output by the parallax calculation unit 149, and converts each three-dimensional point into A high-density three-dimensional point group to which color information is added is output (step S8).

As described above, the imaging apparatus according to the present embodiment includes an imaging unit that receives visible light to obtain an image, a projection unit that projects light other than visible light, and an image based on the light projected by the projection unit. A distance information acquisition unit for acquiring distance information is provided integrally with the housing. As a result, it is possible to realize an imaging device that is easy to prepare for acquiring distance information. In the case where the imaging device and the projection device are separate units as in the conventional art, calibration of the imaging device and the projection device is required each time shooting is performed. Calibration is alignment of the projection direction and the imaging direction. However, in the present embodiment, since they are integrally provided in a common housing, such calibration is not necessary, so preparation for acquiring distance information is simplified. Also, in the processing circuit, high-density three-dimensional point cloud data is reconstructed based on the data output from each section. As a result, it becomes possible to realize "high-resolution" and "high-precision" "three-dimensional reconstruction of long distances and wide angles" in a single small device.

(Modification 1)
Although the configuration for measuring the distance by the pulse method has been shown in the embodiment, the present invention is not limited to this method, and may be modified to other methods as appropriate. Modification 1 shows a configuration for measuring a distance by a phase difference detection method as an example other than the pulse method.

Generally, in the phase difference detection method, a measurement range is irradiated with a laser beam amplitude-modulated at the fundamental frequency, the reflected light is received, and the time is obtained by measuring the phase difference between the irradiated light and the reflected light. is multiplied by the speed of light to calculate the distance. The advantage of this method is that a certain degree of resolution can be expected, but it is necessary to keep emitting light in order to obtain the phase difference. For this reason, strong light cannot be output, and it is difficult to increase the measurement range to a long distance. In addition, it is difficult to widen the angle because spot light is the basis. Modification 1 also improves this point.

FIG. 8 is a diagram for explaining the configuration of an imaging device that performs measurement using a phase difference detection method according to Modification 1. As shown in FIG. FIG. 8 shows a configuration in which the configuration of the imaging device 1 shown in FIG. 2 is modified to the phase difference detection method. Here, portions different from the configuration of the imaging apparatus shown in FIG. 2 will be described, and descriptions of common portions will be omitted as appropriate.

In the imaging device 2 according to Modification 1, the light source unit irradiates laser light amplitude-modulated at the fundamental frequency. Since the phase difference detection method is adopted here, spot light is used. In the example shown in FIG. 8, a MEMS mirror 231 and a semiconductor laser 232, which are an example of an optical deflection element, are provided as a “scanning means”.

The MEMS mirror 231 is driven in one-dimensional direction, and scans the laser light of the semiconductor laser 232 in one direction (for example, the X-axis direction) of the measurement range. When viewed from the projection unit, the entire measurement range can be projected onto a secondary plane of the X-axis and the Y-axis orthogonal thereto. Therefore, the laser beam is scanned by the MEMS mirror 231 in the X-axis direction when considering the entire measurement range as a secondary plane. The range in the Y-axis direction in the secondary plane is covered by using vertically elongated laser light. By configuring in this way, it is possible to widen the angle in the phase difference detection method.

In the example shown in FIG. 8, a semiconductor laser 232 is provided and a scanning system magnifying lens 233 shapes the laser beam to have a vertically elongated diameter, thereby covering the range of the secondary plane in the Y-axis direction. Here, as an example, multiple semiconductor lasers 232 are used. Note that the number of semiconductor lasers 232 may be determined as appropriate. A plurality of semiconductor lasers 232 are shown to obtain strong light, but the number of semiconductor lasers 232 is not limited to this and may be one. It should be noted that as the number of semiconductor lasers 232 is increased, it becomes possible to continue emitting strong light.

A VCSEL may be used as the semiconductor laser. When constructing a plurality of semiconductor lasers, it is possible to use a VCSEL in which light-emitting portions that emit laser light are arranged in a one-dimensional direction or in a two-dimensional direction.

FIG. 9 is a diagram schematically showing an example of a configuration of a projection section according to Modification 1. As shown in FIG. As shown in FIG. 9, the control unit 141 of the processing circuit 14 controls both the light emitting circuit 232a of the semiconductor laser 232 and the MEMS mirror 231 that scans the spot light in the X-axis direction. Specifically, after the semiconductor laser 232 emits light, the control unit 141 reciprocates the MEMS mirror 231 in one axial direction (around the rotation axis in FIG. 9). With this control, the laser light P emitted from the semiconductor laser 232 is scanned in one-dimensional direction (X-axis direction) of the measurement range via the MEMS mirror 231 .

In FIG. 9, the dashed arrows indicate how the laser light P is emitted in a different direction through the MEMS mirror 231 . Further, in this example, since the scanning system magnifying lens 233 is provided, the laser light P is shaped so that the diameter thereof becomes vertically long and is emitted. In FIG. 9, in the laser light P scanned in the one-dimensional direction by the MEMS mirror 231, the scanning system magnifying lens 233 shapes the diameter of the arbitrary laser light P so that it is emitted vertically. indicated by an arrow. Thus, with the configuration shown in Modification 1, the entire measurement range is irradiated with the laser light P only by driving the MEMS mirror 231 in one axial direction. Further, in the phase difference detection method, it is possible to continuously emit strong light and widen the angle of view.

(Optical deflection element)
FIG. 10 is a diagram illustrating an example of the configuration of the MEMS mirror 231. As shown in FIG. A MEMS mirror 231 shown in FIG. 10 has a movable portion 132 and two sets of meandering beam portions 133 on a support substrate 131 .

The movable part 132 has a reflecting mirror 1320 . One end of each of the two sets of meandering beam portions 133 is connected to the movable portion 132 and the other end is supported by the support substrate 131 . The two sets of meandering beams 133 each consist of a plurality of meandering beams, and both consist of a first piezoelectric member 1331 deformed by application of a first voltage and a second piezoelectric member 1331 deformed by application of a second voltage. of piezoelectric members 1332 are alternately provided on each beam. The first piezoelectric member 1331 and the second piezoelectric member 1332 are provided independently for each adjacent beam portion. The two sets of meandering beam portions 133 are deformed by voltage application to the first piezoelectric member 1331 and the second piezoelectric member 1332, respectively, and rotate the reflecting mirror 1320 of the movable portion 132 around the rotation axis.

Specifically, voltages having opposite phases are applied to the first piezoelectric member 1331 and the second piezoelectric member 1332 to cause each beam to warp. As a result, the adjacent beams flex in different directions, which are accumulated, and the reflecting mirror 1320 reciprocates around the rotation axis together with the movable part 132 connected to the two sets of meandering beams 133 . Furthermore, by applying a sine wave having a driving frequency matching the mirror resonance mode with the rotation axis as the rotation center in opposite phases to the first piezoelectric member 1331 and the second piezoelectric member 1332, a very low voltage can be achieved. A large rotation angle can be obtained.

Note that the drive waveform is not limited to a sine wave. For example, it may be a sawtooth wave. In addition, it may be driven not only in resonance mode but also in non-resonance mode.

(Operation of processing circuit)
When receiving an ON signal (shooting start signal) from the shooting switch 15, the control unit 141 controls the imaging device 11a, the light source unit (the semiconductor laser 232 and the MEMS mirror 231), and the TOF sensor 13a (in this example, a phase difference detection type TOF sensor). ) is driven by outputting a synchronizing signal to control the entire processing circuit 14 . In Modification 1, the control unit 141 drives the plurality of semiconductor lasers 232 of the light source unit, and continues to irradiate the MEMS mirror 231 with strong light during one scan by the MEMS mirror 231 . The MEMS mirror 231 starts scanning when the semiconductor laser 232 is irradiated. During the scanning period, the TOF sensor 13 a receives the reflected light from the measurement range with the corresponding light receiving element, and outputs data of each pixel to the processing circuit 14 .

The data output to the processing circuit 14 is acquired by the distance information acquiring unit 13, the time is obtained from the phase difference between the irradiated light and the reflected light, and the time is multiplied by the speed of light to calculate the distance. From the TOF image data generated in this manner, high-density three-dimensional point cloud data is reconstructed in the same procedure as in the embodiment. The process of reconstructing the 3D point cloud data is a repetition of the description of the embodiment, so the description is omitted here.

As described above, the configuration of Modification 1 shows the case where the optical deflection element and the plurality of semiconductor lasers are provided. In this configuration, by adopting a MEMS mirror, a VCSEL, or the like, the imaging element 11a, the light source (the semiconductor laser 232 and the MEMS mirror 231), and the TOF sensor 13a are arranged in a straight line in the longitudinal direction of the housing 10. Since it can be provided as follows, it becomes possible to make the imaging device 1 more compact.

(Modification 2)
The first embodiment and modification 1 include an imaging unit that acquires an RGB image, a projection unit that projects light outside the visible light region, and a distance information acquisition unit that acquires distance information based on the light projected by the projection unit. and are provided on the housing 10 for a measurement range of a hemispherical sphere (approximately 180 degrees or less than 180 degrees). The number of each part is not limited to one set, and may be increased as appropriate. For example, two sets of imaging units may be provided on the same base line for a measurement range of a half celestial sphere (approximately 180 degrees or less than 180 degrees), and the imaging units may be stereo cameras. Also, the same number of imaging units, projection units, and distance information acquiring units may be provided not only on the front side of the housing 10 but also on the rear side or the like. As described above, when a plurality of sets of the imaging unit, the projection unit, and the distance information acquisition unit are provided, one set (for example, the front side set) and the other set (for example, the back side set) of the plurality of sets have a measurement range. They are provided in directions that provide mutually different measurement ranges.

FIG. 11 is a diagram for explaining an example of the configuration of an imaging device according to Modification 2. As shown in FIG. FIG. 11 shows a configuration in which the configuration of the imaging device 1 shown in FIG. 2 is modified. Here, portions different from the configuration of the imaging apparatus shown in FIG. 2 will be described, and descriptions of common portions will be omitted as appropriate.

The image pickup device 3 shown in FIG. 11 has two sets of image pickup units 11 provided on the same base line on one surface (referred to as the front surface) of the housing 10 . In addition, on the rear surface of the housing 10, the imaging units 11, the projection units 12, and the distance information acquisition units 13 are integrally provided in the same number and arrangement as the front surface.

As shown in FIG. 11, by providing not only the front but also the back side, the front half sphere (approximately 180 degrees) is added to the back side half sphere (approximately 180 degrees), resulting in a full sphere (around 360 degrees). ) can be covered. In other words, the processing circuit 14 can acquire 360-degree omnidirectional RGB images and distance information data by one shot, and data for omnidirectional 3D reconstruction models (high-density 3D point cloud data). ) can be generated at once. Note that the process of generating high-density three-dimensional point cloud data is substantially the same as in the embodiment except for using an omnidirectional RGB image and distance information data.

In addition, in the configuration shown in FIG. 11, two sets of RGB imaging units 11 are provided on one surface of the housing 10 on the same base line. In this case, multi-view processing becomes possible in the processing circuit 14 . That is, RGB images of two viewpoints can be obtained by simultaneously driving the two imaging units 11 provided on one surface with a predetermined distance therebetween. This allows the use of parallax calculated based on the two RGB images, further improving distance accuracy over the entire measurement range.

Specifically, when a plurality of RGB imaging units 11 are provided, multi-baseline stereo (MSB) using SSSD and EPI processing can be used like conventional parallax calculation. Therefore, by using this, the reliability of parallax is increased, and high spatial resolution and accuracy can be achieved.

(Operation of processing circuit)
For example, as shown in Equation 5 below, based on Equation 2, the ratio of using the RGB image at the edge is increased. Here, _CRGB is a parallax calculation process using an RGB multiview image, and is a cost value calculated by MBS or EPI. _w1 is a weight value that determines which of _Cwncc and _CRGB is prioritized, but w shown in Equation 2 may be used as it is.

Calculation from the matched parallax to the actual distance is performed by Equation 6 below. Z is the depth, f is the focus, and baseline is called the baseline length, which is the distance between the imaging units 11 on the baseline. d is the parallax, which is the value calculated by the above matching process, that is, the value obtained by multiplying the difference between the coordinate values of the reference pixel and the corresponding pixel in the baseline direction by the pixel pitch.

Note that this is a case where the projection form of the image is perspective projection, and the following equation 7 is used in a projection system such as EquiRectangular used for an omnidirectional image or the like. FIG. 12 shows an explanatory diagram for deriving this formula.

(Application example)
The 3D restoration technology for the measurement range mainly includes a method called structure from motor and a method called stereo method using equipment equipped with a plurality of image capturing means. As a basic process, these methods search for points that are supposed to have been taken at the same location in images taken from different positions by cameras, based on the degree of similarity of the images. This processing is called a corresponding point search or the like. Following this corresponding point searching process, depending on the case, complicated calculations are performed to attempt to restore a large-scale space as smoothly as possible without gaps.

In images taken from different points, the characteristics of the images are fundamentally important when searching for a point where the same point is assumed to have been taken. For example, areas (textureless areas) that are the same no matter where they are photographed, such as a single-color wall or a single-color floor, are captured based on the similarity of the image. It becomes very difficult to find corresponding points. If this corresponding point is not found, the parallax of the human eye cannot be found, and the depth cannot be found. In other words, it is difficult to perform correct 3D reconstruction of points for which corresponding points cannot be found.

As a solution to this problem, there is a known method of irradiating patterned light or random light to make it easier to find corresponding pixels. This is generically called an active stereo method or the like. As an example of the active stereo method, a device that obtains depth information by irradiating a known pattern of light (infrared light) and taking an image with a camera, as represented by Kinect, which is a device that allows game players to input gestures. However, these are aimed only at obtaining depth information, and cannot obtain natural color information of the subject at the same time. Therefore, it is difficult to reconstruct objects and spaces in three dimensions and reuse them as digital 3D data. Separately, color information can be acquired by normal camera photography and superimposed as a 3D mesh texture or the like, but this is impossible when the subject or imaging device is moving.

In addition, when estimating distance using camera matching, processing is basically performed by matching corresponding points in the image, so even for feature area points, distance accuracy deteriorates due to calibration performance. There is a problem that the distance measurement accuracy deteriorates according to the distance to. This is the same problem in active stereo. In addition to this, in the case of active stereo, feature points can be obtained in the area where the pattern is projected, but in order to project the pattern over a wide angle, there is a problem that a large projection device is required.

On the other hand, in the method using TOF, high distance measurement accuracy can be achieved regardless of the distance in the area where the projected light can be measured by the light receiving device. In order to measure, a system that forms a light receiving system for a certain projected light is required. Therefore, in order to obtain a large number of distance measuring points at once, it is necessary to provide a large number of light projecting and light receiving systems, which increases the size of the apparatus and increases the cost. For this reason, it is common to implement a device by providing a rotation mechanism in a limited light projecting/receiving system. Even in such a case, resolution as high as that of a camera cannot be achieved, and a time difference occurs due to the provision of a rotation system, which inevitably increases the size of the apparatus. As for the rotating mechanism, in recent years, MEMS technology that eliminates the mechanical rotating mechanism and shakes the light emitting laser and light receiving sensor, a TOF sensor that enables light reception at multiple points, and a VCSEL equipped with multiple lasers have been released. is coming. With this, the fundamental problems of lack of resolution and wide angle are improved, but still it is not possible to realize high resolution, wide angle and miniaturization as much as a camera. Also, like active stereo, these are aimed only at obtaining depth information and cannot simultaneously obtain natural color information of the subject.

If a conventional device is simply connected, it becomes very large, but these problems are solved in each imaging device shown in this embodiment and its modifications. That is, in spite of its small size, wide-angle three-dimensional information targeting approximately 180 degrees to 360 degrees (whole circumference) can be restored at a high density at once, including textureless areas. In particular, when constructing a light projecting system for projecting light other than visible light and a sensor for receiving the light so that the resolution may be rough, a very small integrated structure can be realized.

The above-described embodiments and modifications are preferred examples of the present invention, but the present invention is not limited thereto, and various modifications can be made without departing from the gist of the present invention. .

1 imaging device 10 housing 11 imaging unit 11a imaging element 11b fisheye lens 12 projection unit 12a light source unit 12b wide-angle lens 13 distance information acquisition unit 13a TOF sensor 13b wide-angle lens 14 processing circuit 15 photographing switch L synchronization signal line

JP 2018-028555 A

The present invention relates to an imaging device .

In order to solve the above-described problems and achieve the object, an imaging apparatus according to an embodiment of the present invention includes an imaging element and a plurality of imaging optical elements each having an imaging range of a different range. a light source , and a plurality of projection optical elements arranged in different directions to project the light emitted from the light source onto the 360- degree range light receiving means for receiving the light reflected from the range of 360 degrees , including projection means, a light receiving element, and a plurality of light receiving optical elements each having a different light receiving range; It is characterized in that the projecting means and the light receiving means are integrally provided in a longitudinally shaped housing whose longitudinal direction is the first axial direction .

The imaging unit 11 has an imaging element 11a, a fisheye lens (wide-angle lens) 11b, and the like. The projection unit 12 has a light source unit 12a, a wide-angle lens 12b, and the like. The distance information acquisition unit 13 has a TOF (Time Of Fright) sensor 13a, a wide-angle lens 13b, and the like. Although not shown, each part may constitute an optical system such as a prism and a lens group. For example, the imaging unit 11 may be configured with an optical system for forming an image on the imaging element 11a with the light collected from the imaging range by the fisheye lens 11b. Further, the projection unit 12 may include an optical system that guides the light from the light source unit 12a to the wide-angle lens 12b. Further, an optical system may be configured in the distance information acquisition section 13 for forming an image of the light collected by the wide-angle lens 13b on the TOF sensor 13a. Here, an optical element arranged to receive light in the surrounding imaging range is called an "imaging optical element". An optical element arranged to project light emitted from a light source to the surroundings is called a "projection optical element". Further, an optical element arranged to receive reflected light (reflected light of projected light) from the surroundings is called a "light receiving optical element". Each optical system may be appropriately determined according to the configuration and arrangement of the imaging device 11a, light source unit 12a, TOF sensor 13a, etc. Here, optical systems such as prisms and lens groups will be omitted from the description. .

Claims (12)

imaging means for imaging a measurement target;
Projection means for projecting light onto the object to be measured;
a light receiving means for receiving the light reflected from the object to be measured;
Acquisition means for acquiring distance information from the measurement target based on the light received by the light receiving means;
has
The imaging means, the projection means, the light receiving means, and the acquisition means are integrally provided in a common housing,
An imaging device characterized by: The projection means includes scanning means for scanning the light,
2. The imaging device according to claim 1, wherein: The projection means projects the light so that the diameter of the light becomes longer in a one-dimensional direction of the measurement range.
3. The imaging apparatus according to claim 2, characterized by: The imaging means, the projection means, and the light receiving means are provided on a straight line in the housing,
4. The imaging apparatus according to any one of claims 1 to 3, characterized by: The imaging means and the light receiving means are set so that there is no interference within an angle of view of 100 degrees to 180 degrees.
5. The imaging apparatus according to any one of claims 1 to 4, characterized in that: A plurality of the imaging means are provided on the same baseline,
6. The imaging apparatus according to any one of claims 1 to 5, characterized by: the resolution of the light receiving means is lower than the resolution of the imaging means;
7. The imaging apparatus according to any one of claims 1 to 6, characterized by: The imaging means, the projection means, and the light receiving means are all provided on one side of the housing toward the same measurement range,
8. The imaging apparatus according to any one of claims 1 to 7, characterized by: A plurality of sets of the imaging means, the projection means, and the light receiving means are provided in the housing,
One set and the other set of the plurality of sets are provided in the direction that the measurement ranges are different from each other,
8. The imaging apparatus according to any one of claims 1 to 7, characterized by: Information processing means for forming three-dimensional point cloud data based on the distance information obtained by the obtaining means and the image obtained by imaging by the imaging means,
10. The imaging device according to any one of claims 1 to 9, characterized in that: The information processing means is
By increasing the resolution of the distance image based on the distance information acquired by the acquisition means to the resolution of the image obtained by imaging by the imaging means, using the parallax data obtained between the image and the image, Constructing 3D point cloud data with a higher density than the 3D point cloud data obtained from the range image,
11. The imaging device according to claim 10, characterized by: An imaging processing method using an imaging device,
a step of projecting light onto a measurement range by means of projection means of the imaging device;
a step of receiving the light reflected from the measurement range by a light receiving means of the imaging device;
a step of acquiring distance information of the measurement range based on the light received by the light receiving means;
a step of constructing three-dimensional point cloud data based on an image of the measurement range captured by an imaging unit of the imaging device and the acquired distance information;
An imaging processing method comprising:

Images