JP7120365B1 - IMAGING DEVICE, IMAGING METHOD AND INFORMATION PROCESSING DEVICE - Google Patents

Abstract

An object of the present invention is to easily identify a specific object included in a displayed image. An imaging device 1 includes an imaging unit 11 for imaging an object, a projection unit 12 for projecting light onto the object, and a distance information acquisition unit 13 (an example of a light receiving unit) for receiving light reflected from the object. , a determination unit 160 that determines whether there is a highly reflective object based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11; and a display control unit 170 for displaying. [Selection drawing] Fig. 15

Description

The present invention relates to an imaging device, an imaging method, and an information processing device.

Patent Literature 1 describes a distance measuring device capable of stably and accurately measuring the distance to an object.

Japanese Patent Application Laid-Open No. 2002-200002 describes an image pickup apparatus that performs image processing to reduce the influence of the reflection of a finger or the like when the reflection occurs.

In Patent Document 3, a block whose measurement data density is lower than a predetermined threshold is extracted, and the coordinates within the range of the extracted block are presented as the presented measurement position where the three-dimensional measurement device should be installed. A three-dimensional synthesis processing system is described that has a measurement position presentation unit that performs measurement.

JP 2018-077071 A Japanese Patent No. 5423287 Japanese Patent No. 6192938

An object of the present invention is to provide an imaging device, an imaging method, and an information processing device that can easily identify a specific object included in a display image.

An imaging apparatus according to the present invention includes an imaging unit for imaging an object, a projection unit for projecting amplitude-modulated light onto the object, and a light receiving unit for receiving light reflected from the object in synchronization with the projection of the light by the projection unit. and acquiring distance information to an object based on the output of the light-receiving unit, wherein the presence of a highly reflective object is determined when the amount of charge in a pixel due to light received by the light-receiving unit is equal to or greater than a predetermined value. a determination unit that determines based on an area corresponding to a pixel in the image information captured by the imaging unit; and if it is determined that there is a highly reflective object, the image information and an identification that identifies the highly reflective object. and a display control unit that displays a display image including information on the display unit so as to display a different image than when it is determined that there is no highly reflective object on the display unit.

According to the present invention, it is possible to provide an imaging device, an imaging method, and an information processing device that can easily identify a specific object included in a display image.

It is a figure showing an example of the appearance of the imaging device concerning the embodiment of the present invention. It is a figure for demonstrating the structure of the imaging device in the same embodiment. It is a figure for demonstrating the usage condition of the imaging device in the same embodiment. It is a figure which shows an example of a structure of the processing block of the processing circuit in the same embodiment. FIG. 4 is a flow chart showing an example of the operation of the processing circuit of the imaging device in the same embodiment; FIG. 4 is a flow chart for generating omnidirectional image data in the same embodiment; FIG. 10 is a flow chart for judging an approaching object in the same embodiment; It is a figure for demonstrating the display content of the display part in the same embodiment. It is a figure which shows the external appearance of the imaging device which concerns on the modification of the same embodiment. It is a figure which shows the structure of the processing block of the processing circuit in a modification. It is a figure which shows the external appearance of the imaging device based on the 2nd modification of embodiment of this invention. FIG. 10 is a diagram showing the configuration of processing blocks of a processing circuit in a second modified example; FIG. 11 is a flow chart for judging an approaching object in the second modified example; It is a figure for demonstrating the structure of the imaging device based on the 3rd modification of embodiment of this invention. FIG. 4 is a flowchart for determining a highly reflective object according to the embodiment of the present invention; FIG. 10 is a flowchart for determining a distant object and a low-reflection object in the same embodiment; FIG. 10 is a flowchart for determining blurring of an image in the same embodiment; It is a judgment flow figure in the 4th modification of the embodiment of the present invention. It is a figure which shows an example of a structure of the processing block of the processing circuit in the 5th modification of embodiment of this invention. It is a figure which shows an example of a structure of the information processing system in the 6th modification of embodiment of this invention. It is a figure which shows an example of a structure of the information processing system in the 7th modification of embodiment of this invention. FIG. 14 is a diagram for explaining display contents of a display unit in fifth to seventh modifications; FIG. 4 is a diagram for explaining a three-dimensional image displayed by a display unit according to the embodiment of the present invention; FIG. FIG. 11 is a flowchart for judgment in fifth to seventh modifications; FIG. 14 is another diagram for explaining display contents of the display unit in the fifth to seventh modifications; FIG. 11 is a flow diagram illustrating processing in fifth to seventh modifications; FIG. 12 is another flow diagram for explaining the processing in the fifth to seventh modifications;

Exemplary embodiments of an imaging device and an imaging processing method will be described in detail below with reference to the accompanying drawings.

FIG. 1 is a diagram showing an example of the appearance of an imaging device according to an embodiment of the present invention. FIG. 2 is a diagram for explaining the configuration of the imaging device. FIG. 2 shows the internal configuration of the imaging apparatus of FIG.

The imaging device 1 is an example of an information processing device that outputs three-dimensional information determined based on received light. A portion corresponding to a light emitting portion) 12 and a distance information acquisition portion (a portion corresponding to a light receiving portion of a distance sensor) 13 for acquiring distance information based on the light projected by the projection portion 12 are integrated with the housing 10. It is set in Each part is electrically connected to the processing circuit 14 inside the housing 10 by a synchronization signal line L, and operates in synchronization with each other.

A photographing switch 15 is used by the user to input a photographing instruction signal to the processing circuit 14 . The display unit 20 displays the content according to the output signal of the processing circuit 14, and is configured by a liquid crystal screen or the like. The display unit 20 is configured by a touch panel or the like, and may receive user's operation input. Based on the photographing instruction, the processing circuit 14 controls each part to acquire the RGB image and distance information data, and reproduces the acquired distance information data into high-density three-dimensional point cloud data based on the RGB image and the distance information data. Do the process to build.

Although it is possible to construct 3D point cloud data by using the distance information data as it is, in that case, the accuracy of the 3D point cloud data is limited to the number of pixels (resolution) of the distance information acquisition unit 13. . This example also shows the processing for reconstructing it into high-density three-dimensional point cloud data. The reconstructed data is output to an external PC or the like via a portable recording medium or communication, and used to display the three-dimensional reconstruction model.

Power is supplied to each unit and the processing circuit 14 from a battery housed inside the housing 10 . In addition to this, a configuration in which power is supplied from the outside of the housing 10 through a connection cord may be employed.

The imaging unit 11 captures two-dimensional image information, and includes imaging elements 11a and 11A, fisheye lenses (wide-angle lenses) 11b and 11B, and the like. The projection unit 12 includes light source units 12a and 12A, wide-angle lenses 12b and 12B, and the like. The distance information acquisition unit 13 includes TOF (Time Of Flight) sensors 13a, 13A, wide-angle lenses 13b, 13B, and the like. Although not shown, each part may constitute an optical system such as a prism and a lens group.

For example, the imaging unit 11 may be configured with an optical system for forming images of the light collected by the fisheye lenses 11b and 11B on the imaging elements 11a and 11A. Further, the projection section 12 may be configured with an optical system that guides the light from the light source sections 12a and 12A to the wide-angle lenses 12b and 12B. Further, an optical system may be configured in the distance information acquisition section 13 to image the light collected by the wide-angle lenses 13b and 13B on the TOF sensors 13a and 13A. Each optical system may be appropriately determined according to the configuration and arrangement of the imaging elements 11a, 11A, light source units 12a, 12A, TOF sensors 13a, 13A, etc. Here, optical systems such as prisms and lens groups are described. are omitted in the description.

The imaging elements 11a and 11A, the light sources 12a and 12A, and the TOF sensors 13a and 13A are housed inside the housing 10 integrally. The fisheye lens 11b, the wide-angle lens 12b, the wide-angle lens 13b, and the display unit 20 are provided on the first surface on the front side of the housing 10, respectively. On the first surface, the inner ranges of the fisheye lens 11b, the wide-angle lens 12b, and the wide-angle lens 13b are open.

The fisheye lens 11B, the wide-angle lens 12B, the wide-angle lens 13B, and the shooting switch 15 are provided on the second surface on the back side of the housing 10, respectively. On the second surface, the inner ranges of the fisheye lens 11B, the wide-angle lens 12B, and the wide-angle lens 13B are open.

The imaging elements 11a and 11A are two-dimensional resolution image sensors (area sensors). The imaging elements 11a and 11A have an imaging area in which a large number of light receiving elements (photodiodes) for each pixel are arranged in two-dimensional directions. In the imaging area, R (red), G (green), and B (blue) color filters such as a Bayer array are provided to receive visible light, and light passing through the color filters is stored in photodiodes. be. Here, an image sensor with a large number of pixels is used so that a wide-angle (for example, the range of a 180-degree hemispherical sphere with the imaging direction shown in FIG. 2 as the front) can be acquired with high resolution. .

The imaging elements 11a and 11A convert the light imaged on the imaging area into an electric signal by the pixel circuit of each pixel and output a high-resolution RGB image. The fisheye lenses 11b and 11B collect light from a wide angle (for example, a 180-degree hemispherical range with the imaging direction shown in FIG. 2 as the front) and form an image of the light on the imaging areas of the imaging elements 11a and 11A.

The light source units 12a and 12A are semiconductor lasers, and emit laser light in a wavelength band other than the visible light range used for distance measurement (infrared as an example here). One semiconductor laser may be used for the light source units 12a and 12A, or a plurality of semiconductor lasers may be used in combination. As the semiconductor laser, a surface emitting semiconductor laser such as a VCSEL (Vertical Cavity Surface Emitting LASER) may be used.

In addition, the light of a semiconductor laser is formed by an optical lens so as to be vertically elongated, and the vertically elongated light is scanned in a one-dimensional direction of the measurement range by an optical deflection element such as a MEMS (Micro Electro Mechanical Systems) mirror. may be configured. In this embodiment, the light sources 12a and 12A are configured to spread the light from the semiconductor laser LA over a wide angle range through wide-angle lenses 12b and 12B without using an optical deflection element such as a MEMS mirror.

The wide-angle lenses 12b and 12B of the light source units 12a and 12A have a function of widening the light emitted from the light source units 12a and 12A into a wide-angle range (for example, a 180-degree hemispherical range with the imaging direction shown in FIG. 2 as the front). have

The wide-angle lenses 13b and 13B of the distance information acquisition unit 13 convert the light reflected from the light sources 12a and 12A projected by the projection unit 12 into a wide-angle measurement range (for example, a circumference with the imaging direction shown in FIG. 2 as the front). 180-degree hemispheric range, etc.), and the light is imaged on the light receiving areas of the TOF sensors 13a and 13A. The measurement range includes one or more projection objects (for example, buildings), and light reflected by the projection objects (reflected light) enters wide-angle lenses 13b and 13B. The reflected light may be captured by, for example, providing a filter that cuts off light having wavelengths in the infrared region or higher on the entire surfaces of the wide-angle lenses 13b and 13B. In addition to this, since light in the infrared region may be incident on the light-receiving area, a means such as a filter for passing light in the infrared region may be provided in the optical path from the wide-angle lenses 13b and 13B to the light-receiving area. good.

The TOF sensors 13a and 13A are optical sensors with two-dimensional resolution. The TOF sensors 13a and 13A have light receiving areas in which a large number of light receiving elements (photodiodes) are arranged two-dimensionally. In this sense, it can be said that it is a "second imaging light receiving means". The TOF sensors 13a and 13A receive reflected light from each area (each area is also referred to as a position) in the measurement range with a light receiving element corresponding to each area, and the distance to each area is determined based on the light detected by each light receiving element. is measured (calculated).

In this embodiment, the distance is measured by the phase difference detection method. In the phase difference detection method, a measurement range is irradiated with a laser beam amplitude-modulated at the fundamental frequency, the reflected light is received, and the time is obtained by measuring the phase difference between the irradiated light and the reflected light. to calculate the distance. The strength of this method is that a certain degree of resolution can be expected.

The TOF sensors 13a and 13A are driven in synchronization with the irradiation of light by the projection unit 12, and the distance corresponding to each pixel is calculated from the phase difference between the light receiving element (corresponding to the pixel) and the reflected light, and the pixel information is obtained. Distance information image data (hereinafter also referred to as "distance image" or "TOF image") associated with information indicating the distance to each area within the measurement range is output. The TOF sensors 13a and 13A may output phase information image data in which phase information is associated with pixel information, and acquire distance information image data based on the phase information image data in post-processing.

The number of areas into which the measurement range can be divided is determined by the resolution of the light receiving areas. Therefore, if a low-resolution image is used for miniaturization, the number of pixel information in the distance image data is reduced, so the number of three-dimensional point groups is also reduced.

As another form, the distance may be measured by a pulse method instead of the phase difference detection method. In that case, for example, the light source units 12a and 12A emit an ultrashort irradiation pulse P1 having a rise time of several nanoseconds (ns) and a high optical peak power. , the time (t) required to receive the reflected pulse P2, which is the reflected light of the irradiation pulse P1 emitted by the light source units 12a and 12A.

When this method is adopted, for example, the TOF sensors 13a and 13A are used, in which a circuit for measuring time is mounted on the output side of the light receiving element. Each circuit converts the time required for the light source units 12a and 12A from emitting the irradiation pulse P1 to receiving the reflected pulse P2 for each light-receiving element into a distance, and obtains the distance to each area.

This method is suitable for widening the angle of view of the imaging device 1 because it can output strong light using peak light. In addition, when a MEMS mirror or the like is used to wave (scan) light, it is possible to irradiate strong light to a long distance while suppressing spread, which leads to an increase in measurement distance. In this case, the arrangement relationship is such that the laser beams emitted from the light source units 12a and 12A are scanned (deflected) toward the wide-angle lenses 12b and 12B by the MEMS mirrors.

Although it is desirable that the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 match, for example, 180 degrees or more, they do not necessarily have to match. The effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 may be reduced as necessary. In this embodiment, the imaging unit 11 and the distance information acquisition unit 13 are effective within a range of, for example, 100 degrees to 180 degrees so that the angle of view does not include the main body of the imaging device 1, the distance information acquisition unit 13, etc. reducing pixels.

Further, the resolution of the TOF sensors 13a and 13A may be set lower than the resolution of the imaging elements 11a and 11A in order to prioritize miniaturization of the imaging apparatus 1. FIG. By setting the TOF sensors 13a and 13A to have a resolution lower than that of the imaging elements 11a and 11A, it is possible to suppress an increase in the size of the light-receiving area. Therefore, the TOF sensors 13a and 13A have a low resolution, and the three-dimensional point cloud obtained by the TOF sensors 13a and 13A has a low density. It can be transformed into a dimensional point cloud. The process of converting into a high-density three-dimensional point group in the processing circuit 14 will be described later.

In the present embodiment, as an example, the imaging element 11a, the light source section 12a, and the TOF sensor 13a are provided so as to line up in a straight line in the longitudinal direction of the housing 10. FIG. Similarly, the imaging element 11A, the light source section 12A, and the TOF sensor 13A are arranged in a straight line in the longitudinal direction of the housing 10. As shown in FIG. An example of the imaging device 11a, the light source unit 12a, and the TOF sensor 13a will be described below.

The imaging area (imaging surface) of the imaging element 11a and the light receiving area (light receiving surface) of the TOF sensor 13a may be arranged in a direction perpendicular to the longitudinal direction as shown in FIG. It may be arranged in the longitudinal direction by providing a prism or the like that converts the optical path by 90 degrees and enters the light. In addition, they may be arranged in any direction depending on the configuration. That is, the imaging element 11a, the light source unit 12a, and the TOF sensor 13a are arranged so as to cover the same measurement range. An imaging unit 11, a projection unit 12, and a distance information acquisition unit 13 are arranged from one side of the housing 10 toward the measurement range.

In this case, the imaging device 11a and the TOF sensor 13a should be arranged on the same base line so as to achieve parallel stereo. By arranging for parallel stereo, even if there is only one imaging device 11a, it is possible to obtain parallax data using the output of the TOF sensor 13a. The light source unit 12a is configured to irradiate the measurement range of the TOF sensor 13a with light.

(processing circuit)
Next, processing of the processing circuit 14 will be described. The TOF images obtained only by the TOF sensors 13a and 13A have low resolution as they are. Therefore, in this example, an example of reconstructing high-density three-dimensional point group data by increasing the resolution by the processing circuit 14 is shown. Part or all of the processing described below as the "information processing means" in the processing circuit 14 may be performed by an external device.

As described above, the 3D point cloud data reconstructed by the imaging device 1 is output to an external device such as a PC via a portable recording medium or communication, and used to display a 3D restored model. .

As a result, compared to the case where the imaging device 1 itself displays a three-dimensional restored model, it is possible to provide the imaging device 1 that is superior in portability due to speeding up, downsizing, and weight reduction.

However, after the 3D information is restored by an external device away from the site where the 3D information is acquired, the 3D information is acquired in a desired layout, such as the reflection of the photographer himself or a tripod in the captured image. It may be found that there is not, in which case revisiting the site from which the 3D information is to be obtained is troublesome.

In order to solve this problem, it is conceivable to bring an external device such as a PC to the site, but then the merits of speeding up, downsizing, and weight reduction will be lost.

It is also conceivable to transmit the acquired three-dimensional information to an external device via a communication line and receive the restored three-dimensional information. Therefore, it is difficult to visually check whether the photographer himself or the tripod is reflected in the captured image.

In particular, in the case of omnidirectional three-dimensional information, it is extremely difficult to visually check whether the photographer himself or a tripod is reflected in the captured image.

In view of the above problems, the present embodiment provides an imaging apparatus 1 that can easily check in real time whether the photographer himself or a tripod is reflected in the captured image, or whether the three-dimensional information of the desired layout has not been acquired. intended to provide

FIG. 3 is a diagram for explaining the usage status of the imaging device according to the embodiment.

In the state shown in FIG. 3(a), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the omnidirectional imaging range R, and the photographer M and the selfie stick 1A are not included in the omnidirectional imaging range R. It does not appear in captured images of the celestial sphere.

In the state shown in FIG. 3B, the photographer M is included in the omnidirectional imaging range R, and the photographer M appears in the omnidirectional captured image.

In the state shown in FIG. 3C, the tripod 1B supporting the imaging device 1 is included in the omnidirectional imaging range R, and the tripod 1B appears in the omnidirectional captured image.

In the state shown in FIG. 3(d), the photographer M and the selfie stick 1A supporting the imaging device 1 are not included in the omnidirectional imaging range R, and the photographer M and the selfie stick 1A are not included in the omnidirectional imaging range R. Although it will not be reflected in the captured image of the celestial sphere, there is a possibility that it will be erroneously determined to be reflected due to the strong external light (for example, the sun or lighting).

In the states shown in FIGS. 3(b) and 3(c), it is difficult to uniformly judge whether or not there is an object in the image, because the color, type, and appearance of the object in the image vary. rice field.

In the above situation, when determining the presence or absence of a specific object (proximity object) such as the photographer himself or a tripod based on the distance information image data output from the TOF sensors 13a and 13A, the actual specific object It was difficult to distinguish between the existence of a large amount of light and whether the outside light was too strong.

That is, when the amount of electricity stored in a specific pixel of the TOF sensor 13a, 13A is saturated, whether the cause is the presence of a specific object or the intensity of outside light is too strong, the TOF sensor 13a, 13A It was difficult to distinguish just from the output of

In view of the above problems, the present embodiment is an imaging apparatus 1 capable of accurately confirming whether or not a specific object such as a photographer or a tripod is reflected in a captured image by distinguishing it from the influence of external light. for the other purpose of providing Another object of the present embodiment is to make it possible to confirm that objects such as highly reflective objects, distant objects, low reflective objects, and image blur are included in the image, in addition to the near objects.

FIG. 4 is a diagram showing an example of a configuration of processing blocks of the processing circuit 14. As shown in FIG. Processing circuit 14 shown in FIG. A projection processing unit 147, a semantic segmentation unit 148, a parallax calculation unit 149, a three-dimensional reconstruction processing unit 150, a determination unit 160, a display control unit 170 as an example of an output unit, and an example of an output unit. and a transmitting/receiving unit 180 . In FIG. 4, solid line arrows indicate the flow of signals, and broken line arrows indicate the flow of data.

When the control unit 141 receives an ON signal (shooting start signal) from the shooting switch 15, it outputs a synchronization signal to the image sensors 11a and 11A, the light source units 12a and 12A, and the TOF sensors 13a and 13A, thereby controlling the processing circuit 14 as a whole. do. The control unit 141 first outputs signals instructing the light source units 12a and 12A to emit ultrashort pulses, and at the same timing outputs signals instructing the TOF sensors 13a and 13A to generate TOF image data. Further, the control unit 141 outputs signals instructing imaging to the imaging elements 11a and 11A. The imaging by the imaging devices 11a and 11A may be performed during the period during which light is emitted from the light source units 12a and 12A, or during the most recent period before or after that period.

The RGB image data acquisition unit 142 acquires RGB image data captured by the image sensors 11a and 11A based on an image capturing instruction from the control unit 141, and outputs omnidirectional RGB image data. The monochrome processing unit 143 performs processing for aligning data types for matching processing with the TOF image data obtained from the TOF sensors 13a and 13A. In this example, the monochrome processing unit 143 performs processing for converting omnidirectional RGB image data into an omnidirectional monochrome image.

The TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13a and 13A based on the TOF image data generation instruction from the control unit 141, and outputs omnidirectional TOF image data.

The resolution enhancement unit 145 treats the omnidirectional TOF image data as a monochrome image and enhances its resolution. Specifically, the resolution enhancement unit 145 replaces the value of the distance associated with each pixel of the omnidirectional TOF image data with the value (grayscale value) of the omnidirectional monochrome image. . Furthermore, the resolution enhancement unit 145 enhances the resolution of the omnidirectional monochrome image to the resolution of the omnidirectional RGB image data obtained from the imaging elements 11a and 11A. Conversion to high resolution is performed, for example, by performing normal up-conversion processing. As another conversion method, for example, multiple frames of omnidirectional TOF image data generated continuously are acquired, and the distances between adjacent points are added using them to perform super-resolution processing. good.

The matching processing unit 146 calculates the feature quantity of the textured portion of the omnidirectional monochrome image obtained by increasing the resolution of the omnidirectional TOF image data and the omnidirectional monochrome image of the omnidirectional RGB image data. is extracted, and matching processing is performed using the extracted feature amount. For example, the matching processing unit 146 extracts edges from each monochrome image and performs matching processing between the extracted edge information. As another method, for example, matching processing may be performed by a method such as SIFT, which converts changes in texture into features. Here, matching processing means searching for corresponding pixels.

As a specific method of matching processing, there is block matching, for example. Block matching consists of pixel values cut out as M×M (M is a positive integer) pixel size blocks around the reference pixel, and In this method, the similarity of pixel values cut out as a block of M×M pixels is calculated, and the central pixel with the highest similarity is taken as the corresponding pixel.

There are various methods of calculating similarity. For example, an equation representing a normalized autocorrelation coefficient NCC (Normalized Correlation Coefficient) may be used. The higher the normalized autocorrelation coefficient CNCC, the higher the degree of similarity.

Further, since the distance data of the textureless area can be obtained from the omnidirectional TOF image data, the matching process may be weighted according to the area. For example, in the calculation of the formula representing CNCC, calculation may be performed to weight portions other than edges (textureless regions).

Alternatively, selective normalization correlation (SCC) or the like may be used instead of the formula representing the NCC.

The reprojection processing unit 147 performs a process of reprojecting the omnidirectional TOF image data indicating the distance of each position in the measurement range onto the two-dimensional coordinates (screen coordinate system) of the imaging unit 11 . To re-project is to find out at which coordinates the three-dimensional points calculated by the TOF sensors 13a and 13A appear in the images of the imaging devices 11a and 11A. The omnidirectional TOF image data indicates the positions of three-dimensional points in a coordinate system centered on the distance information acquisition section 13 (mainly the wide-angle lenses 13b and 13B). Therefore, the three-dimensional point indicated by the omnidirectional TOF image data is reprojected onto the coordinate system centered on the imaging unit 11 (mainly the fisheye lenses 11b and 11B).

For example, the reprojection processing unit 147 translates the coordinates of the 3D point of the omnidirectional TOF image data to the coordinates of the 3D point centered on the imaging unit 11, and after the translation, performs the omnidirectional TOF A process of converting the coordinates of the three-dimensional points of the image data into a two-dimensional coordinate system (screen coordinate system) indicated by the omnidirectional RGB image data is performed. As a result, the coordinates of the three-dimensional point of the omnidirectional TOF image data and the coordinates of the omnidirectional two-dimensional image information captured by the imaging unit 11 are associated with each other. The reprojection processing unit 147 associates the coordinates of the three-dimensional point of the omnidirectional TOF image data with the coordinates of the omnidirectional two-dimensional image information captured by the imaging unit 11 .

The parallax calculator 149 calculates the parallax of each position from the difference in distance from the corresponding pixel obtained by the matching process. In the parallax matching process, the reprojection coordinates converted by the reprojection processing unit 147 are used to search for pixels around the position of the reprojection coordinates. It becomes possible to acquire distance information.

Further, segmentation data obtained by the semantic segmentation processing of the semantic segmentation unit 148 may be used for the parallax matching processing. In that case, more detailed and high-resolution distance information can be obtained.

In addition, parallax matching processing is performed only for edges and only parts with strong feature amounts, and for other parts, omnidirectional TOF image data is also used. may be used for propagation processing.

The semantic segmentation unit 148 uses deep learning to assign a segmentation label indicating an object to the input image of the measurement range. As a result, each pixel of the omnidirectional TOF image data can be constrained to one of a plurality of distance areas divided by distance, thereby further increasing the reliability of calculation.

The three-dimensional reconstruction processing unit 150 acquires omnidirectional RGB image data from the RGB image data acquisition unit 142 and reconstructs omnidirectional three-dimensional data based on the distance information output by the parallax calculation unit 149. , outputs a high-density 3D point cloud of the sphere with color information added to each 3D point. The 3D reconstruction processing unit 150 is an example of a 3D information determination unit that determines 3D information.

The determination unit 160 acquires the omnidirectional RGB image data from the RGB image data acquisition unit 142, and from the reprojection processing unit 147, converts the omnidirectional RGB image data into a two-dimensional coordinate system indicated by the omnidirectional RGB image data. The TOF image data of the sphere is acquired, and based on these data, it is determined whether or not the specific object is reflected in the captured image, and the determination result is output to the display control unit 170 .

The display control unit 170 acquires omnidirectional RGB image data from the RGB image data acquiring unit 142 and causes the display unit 20 to display two-dimensional image information based on the acquired omnidirectional RGB image data. Further, the display control unit 170 causes the display unit 20 to display a display image including information indicating the determination result acquired from the determination unit 160 and two-dimensional image information.

The display control unit 170 is an example of an output unit that outputs two-dimensional image information captured by the imaging unit 11 separately from the three-dimensional information, and the display unit 20 is an example of an output destination for outputting the two-dimensional image information. be.

The display control unit 170 may acquire omnidirectional three-dimensional data from the three-dimensional reconstruction processing unit 150 and cause the display unit 20 to display the three-dimensional information. Specifically, the display control unit 170 may select between displaying two-dimensional image information on the display unit 20 and displaying three-dimensional information on the display unit 20 according to a predetermined condition. Thereby, the display control section 170 can output the two-dimensional image information separately from the three-dimensional information.

The transmitting/receiving unit 180 communicates with an external device by a wired or wireless technology, and is used to receive omnidirectional 3D data output from the 3D reconstruction processing unit 150 and omnidirectional data output from the RGB image data acquisition unit 142 . The two-dimensional image information of the celestial sphere is transmitted (output) via the network 400 to the external device 300 that performs the three-dimensional restoration processing.

In the present embodiment, the two-dimensional image information captured by the imaging unit 11 is "original two-dimensional image information" for creating "two-dimensional image data for display", or "for display". two-dimensional image data of". For example, when "two-dimensional image data for display" is created from "original two-dimensional image information" inside the imaging device 1, or when "original two-dimensional image information" is transmitted from the imaging device 1 to an external device Then, an external device may create "two-dimensional image data for display" from the "original two-dimensional image information".

The transmitting/receiving unit 180 is an example of an output unit that outputs 3D information, and the external device 300 is an example of an output destination that outputs 3D information.

The transmitting/receiving unit 180 may transmit only omnidirectional 3D data without transmitting omnidirectional 2D image information. Further, the transmission/reception unit 180 may be configured by an interface circuit with a portable storage medium such as an SD card or a personal computer.

(Operation of processing circuit)
FIG. 5 is a flow chart showing an example of the operation of the processing circuit 14 of the imaging device 1. As shown in FIG. When the user turns on the photographing switch 15 and receives a photographing instruction signal, the control unit 141 of the processing circuit 14 performs an operation of generating a high-density three-dimensional point cloud by the following method (imaging processing method and an example of an information processing method).

First, the control section 141 drives the light source sections 12a and 12A, the TOF sensors 13a and 13A, and the imaging elements 11a and 11A to photograph the measurement range (step S1). By being driven by the controller 141, the light sources 12a and 12A emit infrared light (an example of a projecting step), and the TOF sensors 13a and 13A receive the reflected light (an example of a light receiving step). In addition, the imaging elements 11a and 11A capture images of the measurement range at the timing when the light source units 12a and 12A start to be driven or in the period immediately preceding it (an example of the imaging step).

Next, the RGB image data acquisition unit 142 acquires RGB image data of the measurement range from the imaging devices 11a and 11A (step S2). Then, the display control unit 170 acquires the omnidirectional RGB image data from the RGB image data acquiring unit 142, and causes the display unit 20 to display two-dimensional image information based on the acquired omnidirectional RGB image data (display An example of steps) (step S3).

The display control unit 170 causes the display unit 20 to display two-dimensional image information of a partial region of the acquired omnidirectional RGB image data, and according to various user inputs, the two-dimensional image information displayed on the display unit 20 is displayed. Change the image information area. Various inputs by the user can be realized by providing an operation switch other than the photographing switch 15 or configuring the display section 20 as an input section such as a touch panel.

At this stage, from the two-dimensional image information displayed on the display unit 20, the photographer can confirm that the two-dimensional image information of the desired layout has not been obtained, and that the photographer himself, the tripod, etc. have not been reflected in the captured image. can be confirmed.

Next, the TOF image data acquisition unit 144 acquires TOF image data indicating the distance of each position in the two-dimensional area from the TOF sensors 13a and 13A (step S4).

Next, the monochrome processing unit 143 converts the RGB image data into a monochrome image (step S5). The TOF image data and the RGB image data are different in data type between the distance data and the RGB data, respectively, and matching cannot be performed as they are. Therefore, each data is first converted into a monochrome image. As for the TOF image data, before the resolution is increased, the resolution increasing unit 145 converts the value indicating the distance of each pixel by replacing it with the value of the monochrome image.

Next, the resolution enhancement unit 145 enhances the resolution of the TOF image data (step S6). Next, the matching processing unit 146 extracts feature amounts of textured portions of each monochrome image, and performs matching processing using the extracted feature amounts (step S7).

Next, the parallax calculator 149 calculates the distance by calculating the parallax of each position from the distance deviation of the corresponding pixels (step S8).

Next, the determination unit 160 acquires omnidirectional RGB image data from the RGB image data acquisition unit 142, and omnidirectional coordinates converted into a two-dimensional coordinate system indicated by the RGB image data from the reprojection processing unit 147. TOF image data is acquired, based on these data, it is determined whether or not a nearby object as a specific target is reflected in the captured image, and the determination result is output to the display control unit 170 (an example of the determination step) .

The display control unit 170 causes the display unit 20 to display the information indicating the determination result acquired from the determination unit 160 by superimposing or including the information on the two-dimensional image information (an example of a display step) (step S9). In step S<b>9 , the determination unit 160 determines whether or not there are not only near objects but also highly reflective objects, distant objects, low reflective objects, image blur, and the like as specific objects, and outputs the determination result to the display control unit 170 . .

Then, the three-dimensional reconstruction processing unit 150 acquires the RGB image data from the RGB image data acquisition unit 142, reconstructs the three-dimensional data based on the distance information output by the parallax calculation unit 149, and converts each three-dimensional point into A high-density three-dimensional point group to which color information is added is output (step S10).

Next, the transmission/reception unit 180 performs 3D reconstruction processing on the 3D data output from the 3D reconstruction processing unit 150 and the 2D image information output from the RGB image data acquisition unit 142 via the network 400. 300 (an example of a three-dimensional information output step) (step S11).

The transmission/reception unit 180 may transmit the three-dimensional data output from the three-dimensional reconstruction processing unit 150 without transmitting the two-dimensional image information output from the RGB image data acquisition unit 142 .

As described above, the image capturing apparatus 1 includes the image capturing unit 11 and the display control unit 170 that outputs two-dimensional image information captured by the image capturing unit 11 separately from three-dimensional information.

As a result, it is possible to easily check from the two-dimensional image information whether the photographer himself or the tripod is reflected in the captured image, or whether the three-dimensional information of the desired layout has not been acquired, without checking the three-dimensional information. it becomes possible to

Therefore, it is possible to reacquire the 3D information while staying at the site where the 3D information is to be acquired. This reduces the trouble of visiting the site to acquire the three-dimensional information again, compared with the case where the user finds that the three-dimensional information of the desired layout has not been acquired.

The three-dimensional information includes omnidirectional three-dimensional information. In this case, even with omnidirectional 3D information, where it is difficult to confirm that the photographer himself or his tripod is reflected in the captured image, or that 3D information with a desired layout is not acquired, the photographer can It is possible to easily confirm from the two-dimensional image information imaged by the imaging unit 11 that the user himself or a tripod is reflected in the imaged image and that the three-dimensional information of the desired layout is not acquired.

The display control unit 170 outputs the two-dimensional image information G in step S3 before the transmitting/receiving unit 180 transmits (outputs) the three-dimensional information in step S11. The display control unit 170 outputs the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

As a result, it is possible to check from the two-dimensional image information before checking the three-dimensional information whether the photographer himself or the tripod is reflected in the captured image, or whether the three-dimensional information of the desired layout has not been acquired. becomes possible.

The display control unit 170 causes the display unit 20 to display two-dimensional image information. The imaging device 1 includes a display section 20 .

As a result, it is possible to easily confirm from the two-dimensional image information displayed on the display unit 20 that the photographer himself or the tripod is reflected in the captured image and that the three-dimensional information of the desired layout is not acquired. becomes possible.

The display control unit 170 outputs the two-dimensional image information to the display unit 20 different from the external device 300 to which the transmitting/receiving unit 180 outputs the three-dimensional information.

As a result, it is possible to check the three-dimensional information output to the external device 300 without checking the three-dimensional information output to the external device 300, such as whether the photographer himself or the tripod is reflected in the captured image, or whether the three-dimensional information of the desired layout has not been acquired. It is possible to confirm from the two-dimensional image information output to the display unit 20 different from the device 300 .

The imaging device 1 includes a three-dimensional reconstruction processing section 150 that determines three-dimensional information based on the output of the distance information acquisition section 13 . The 3D reconstruction processing unit 150 determines 3D information based on the output of the distance information acquisition unit 13 and the 2D image information.

As a result, the three-dimensional information determined by the three-dimensional reconstruction processing unit 150 is checked to confirm that the photographer himself or the tripod is reflected in the captured image, or that the three-dimensional information of the desired layout has not been acquired. It is possible to confirm from the two-dimensional image information imaged by the imaging unit 11 without having to remove the image.

FIG. 6 is a flow chart of omnidirectional image data generation in the same embodiment.

FIG. 6A is a flow chart showing the process of generating omnidirectional RGB image data corresponding to step S2 described in FIG.

The RGB image data acquisition unit 142 inputs two pieces of RGB image data in the fisheye image format (step S201).

The RGB image data acquisition unit 142 converts each RGB image data into an equirectangular image format (step S202). The RGB image data acquisition unit 142 converts the two RGB image data into an equirectangular image format based on the same coordinate system, thereby facilitating image combination in the next step. Note that the RGB image data can be converted to image data using one or more image formats other than the equirectangular image format, if desired. For example, it is possible to convert the coordinates of an image perspectively projected onto an arbitrary surface or an image perspectively projected onto each surface of an arbitrary polyhedron.

The equirectangular image format will now be described. The equirectangular image format is a format capable of expressing an omnidirectional image, and is a format of an image (equirectangular image) created using the equirectangular projection method. The equirectangular projection is a projection that expresses a three-dimensional direction with two variables, such as the latitude and longitude of a globe, and displays it on a plane so that the latitude and longitude are orthogonal. Therefore, the equirectangular image is an image generated using the equirectangular projection method, and is represented by coordinates having two angular variables of the spherical coordinate system as two axes.

The RGB image data acquisition unit 142 combines the two RGB image data generated in step S202 to generate one omnidirectional RGB image data (step S203). The two pieces of input RGB image data cover an area with an angle of view exceeding 180 degrees. Therefore, the omnidirectional RGB image data generated by appropriately connecting the two RGB image data can cover the omnidirectional area.

Note that the combining process in step S203 can use an existing technique for connecting a plurality of images, and the method is not particularly limited.

FIG. 6(b) is a flow chart showing the process of generating omnidirectional TOF image data corresponding to step S4 described in FIG.

The TOF image data acquisition unit 144 acquires two distance image data in the fisheye image format (step S401).

The TOF image data acquisition unit 144 converts the two pieces of TOF image data in the fisheye image format into the equirectangular image format (step S402). The equirectangular image format is, as described above, a format capable of representing an omnidirectional image. In this step S402, the two TOF image data are converted into the equirectangular image format based on the same coordinate system, thereby facilitating the image combination in the next step S403.

The TOF image data acquisition unit 144 combines the two TOF image data generated in step S402 to generate one omnidirectional TOF image data (step S403). The two pieces of input TOF image data cover an area with an angle of view exceeding 180 degrees. Therefore, the omnidirectional TOF image data generated by appropriately connecting the two pieces of TOF image data can cover the omnidirectional area.

Note that the combining process in step S403 can use an existing technique for connecting multiple images, and the method is not particularly limited.

FIG. 7 is a flow chart for judging an approaching object in the same embodiment.

FIG. 7 is a flowchart showing the process of determining whether or not a nearby object is included in the captured image corresponding to step S9 described in FIG.

Based on the omnidirectional TOF image data acquired from the re-projection processing unit 147, the determination unit 160 determines whether the omnidirectional TOF image data has a saturated amount of electricity as an example of a pixel whose amount of electricity storage is equal to or greater than a predetermined value. It is determined whether or not there is a pixel in which the number of pixels is 0 (step S801).

If there is a pixel with a saturated charge amount in step S801, the determining unit 160 determines, based on the omnidirectional RGB image data acquired from the RGB image data acquiring unit 142, among the omnidirectional RGB image data, For pixels at the same coordinates as the pixel with the saturated charged amount in step S801, it is determined whether the charged amount is saturated as an example of the charged amount being equal to or greater than a predetermined value (step S802).

If the amount of stored electricity is saturated in step S802, the determining unit 160 determines that the pixels in which the amount of stored electricity is saturated in step S801 are due to external light (for example, the sun or illumination), and outputs error information. Output to the display control unit 170 . The display control unit 170 causes the display unit 20 to display a display image including the error information and the two-dimensional image information based on the error information acquired from the determination unit 160 (step S803).

If the amount of stored electricity is not saturated in step S802, the determination unit 160 determines that the pixels in which the amount of stored electricity is saturated in step S801 are due to the presence of a nearby object, and the amount of stored electricity is saturated in step S801. The coordinate position information of the pixel that has been displayed is output to display control section 170 . The display control unit 170 causes the display unit 20 to display a display image including identification information for identifying a nearby object and two-dimensional image information based on the pixel coordinate position information acquired from the determination unit 160 (step S804). .

If there is no pixel in which the amount of stored electricity is saturated in step S801, the determination unit 160 determines 0 It is determined whether there is a pixel indicating distance information of 5 m or less (step S805).

If there is no pixel indicating distance information of 0.5 m or less in step S805, the determination unit 160 ends the process.

If there is a pixel indicating distance information of 0.5 m or less in step S805, the determination unit 160 proceeds to the above-described step S804. and output the coordinate position information of the pixel indicating the distance information of 0.5 m or less to the display control unit 170 in step S805. The display control unit 170 causes the display unit 20 to display a display image including identification information for identifying a nearby object and two-dimensional image information based on the pixel coordinate position information acquired from the determination unit 160 .

As described above, the display control unit 170 superimposes or includes the identification information on the two-dimensional image information when it determines that there is a nearby object, and when it determines that there is no nearby object, the display control unit 170 superimposes the identification information on the two-dimensional image information. Do not superimpose or include image information.

That is, the display control unit 170 causes the display unit 20 to display differently depending on whether or not there is a nearby object.

Further, the display control unit 170 causes the display unit 20 to display a display image including identification information for identifying a nearby object and two-dimensional image information based on the pixel coordinate position information acquired from the determination unit 160 .

That is, the display control unit 170 causes the display unit 20 to perform different displays depending on the position of the nearby object.

FIG. 8 is a diagram for explaining display contents of the display unit in the embodiment.

FIG. 8 is an explanatory diagram corresponding to step S2 shown in FIG. 5 and steps S803 and S804 shown in FIG.

Two-dimensional image information G is displayed on the display unit 20 by the display control unit 170 . Further, the display control unit 170 causes the display unit 20 to display identification information G1 and G2 (for example, a finger and a tripod) for identifying objects such as nearby objects, error information G3, and two-dimensional image information G. An image is displayed. The error information G3 can be indicated using a mark such as "sun, lighting", as shown in FIG.

As described above, the imaging device 1 includes the imaging unit 11 that images an object, the projection unit 12 that projects light onto the object, the distance information acquisition unit 13 that receives light reflected from the object, and the distance information acquisition unit. A display control unit 170 that causes the display unit 20 to display differently according to the presence or absence of an object such as a nearby object determined based on the output of the unit 13 and the output of the imaging unit 11 .

As a result, the photographer can accurately check whether or not the photographer himself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the influence of external light.

The imaging device 1 includes a display section 20 . As a result, the photographer can reliably confirm whether or not a nearby object is included in the captured image.

The display control unit 170 causes the display unit 20 to perform different displays depending on the position of the nearby object. As a result, the photographer can confirm the position at which the nearby object is reflected in the captured image.

The display control unit 170 causes the display unit 20 to display the image information G captured by the imaging unit 11, and displays a display image including the image information and the identification information G1 and G2 for identifying nearby objects on the display unit 20. Let As a result, the photographer can reliably confirm the position at which the nearby object is reflected in the captured image.

In the imaging device 1, the amount of stored electricity is saturated as an example of a pixel in which the amount of stored electricity due to the light received by the distance information acquisition unit 13 is equal to or greater than a predetermined value, and the amount of stored electricity of the pixel of the imaging unit 11 is equal to or less than the predetermined value. As an example, it includes a determination unit 160 that determines that there is an approaching object when the amount of stored electricity is not saturated.

As a result, the photographer can accurately check whether or not a nearby object is reflected in the captured image, distinguishing it from the influence of external light.

FIG. 9 is a diagram showing the appearance of an imaging device according to a modification of the embodiment. FIG. 10 is a diagram showing a configuration of processing blocks of a processing circuit in a modified example.

In this modification, the display control unit 170 acquires omnidirectional RGB image data from the RGB image data acquisition unit 142, and displays two-dimensional image information based on the acquired omnidirectional RGB image data on the display device 500. 520 to display. The display unit 520 is an example of an output destination for outputting two-dimensional image information.

As a result, it is possible to easily confirm from the two-dimensional image information displayed on the display unit 520 that the photographer himself or the tripod is reflected in the captured image and that the three-dimensional information of the desired layout is not acquired. becomes possible.

The display control unit 170 outputs the two-dimensional image information to the display unit 520 different from the external device 300 to which the transmitting/receiving unit 180 outputs the three-dimensional information.

As a result, it is possible to check the three-dimensional information output to the external device 300 without checking the three-dimensional information output to the external device 300, such as whether the photographer himself or the tripod is reflected in the captured image, or whether the three-dimensional information of the desired layout has not been acquired. It is possible to confirm from the two-dimensional image information output to the display unit 520 different from the device 300 .

The display control unit 170 may acquire omnidirectional three-dimensional data from the three-dimensional reconstruction processing unit 150 and cause the display unit 520 to display the three-dimensional information. Specifically, the display control unit 170 may select, according to a predetermined condition, to display two-dimensional image information on the display unit 520 or to display three-dimensional information on the display unit 520 . Thereby, the display control section 170 can output the two-dimensional image information separately from the three-dimensional information.

Based on the error information acquired from the determination unit 160, the display control unit 170 causes the display unit 520 to display a display image including the error information and the two-dimensional image information.

The display control unit 170 causes the display unit 520 to display a display image including identification information for identifying a nearby object and two-dimensional image information based on the pixel coordinate position information acquired from the determination unit 160 .

That is, the display control unit 170 causes the display unit 520 to display differently depending on whether or not there is a nearby object determined based on the output of the distance information acquisition unit 13 and the output of the imaging unit 11 .

As a result, the photographer can accurately check whether or not the photographer himself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the influence of external light.

The display control section 170 causes the position of the display section 520 to display differently according to the position of the nearby object. As a result, the photographer can confirm the position at which the nearby object is reflected in the captured image.

The display control unit 170 causes the display unit 520 to display image information captured by the imaging unit 11, and also causes the display unit 520 to display a display image including image information and identification information for identifying a nearby object. As a result, the photographer can reliably confirm the position at which the nearby object is reflected in the captured image.

FIG. 11 is a diagram showing the appearance of an imaging device according to a second modification of the embodiment of the present invention. FIG. 12 is a diagram showing the configuration of processing blocks of a processing circuit in the second modification.

In a second modification shown in FIG. 11, an imaging device 1 includes a plurality of display units 20A and 20a instead of the display unit 20 shown in FIG. The display units 20A and 20a are composed of LEDs or the like, and flash or light up according to the output signal of the processing circuit 14. FIG.

The display unit 20 a is provided on the first surface on the front side of the housing 10 , and the display unit 20 A is provided on the second surface on the rear side of the housing 10 .

In the second modification shown in FIG. 12, the display control unit 170 causes the display units 20A and 20a to display information indicating the determination result acquired from the determination unit 160. FIG. For example, the displays 20a and 20b may flash red when there is an object close to either side of the imaging device 1 .

Further, the transmission/reception unit 180 transmits (outputs) the omnidirectional two-dimensional image information output from the RGB image data acquisition unit 142 to the display device 500 via the network 400 . The display device 500 is an example of an output destination for outputting two-dimensional image information.

That is, in the second modification, in step S3 shown in FIG. 5, the transmission/reception unit 180 acquires omnidirectional RGB image data from the RGB image data acquisition unit 142, to the display device 500 (output).

The transmitting/receiving unit 510 of the display device 500 receives the two-dimensional image information transmitted from the transmitting/receiving unit 180 of the imaging device 1 .

The display control unit 530 of the display device 500 causes the display unit 520 to display the two-dimensional image information received by the transmission/reception unit 510 . A display device 500 including a display control unit 530 is an example of an information processing device.

As described above, the image capturing apparatus 1 includes the image capturing unit 11 and the transmitting/receiving unit 180 that outputs two-dimensional image information captured by the image capturing unit 11 separately from three-dimensional information.

As a result, it is possible to easily check from the two-dimensional image information whether the photographer himself or the tripod is reflected in the captured image, or whether the three-dimensional information of the desired layout has not been acquired, without checking the three-dimensional information. it becomes possible to

Therefore, it is possible to reacquire the 3D information while staying at the site where the 3D information is to be acquired. This reduces the trouble of visiting the site to acquire the three-dimensional information again, compared with the case where the user finds that the three-dimensional information of the desired layout has not been acquired.

The transmitting/receiving unit 180 transmits (outputs) the two-dimensional image information G in step S3 before transmitting (outputting) the three-dimensional information in step S11. The transmitting/receiving unit 180 transmits (outputs) the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

As a result, it is possible to check from the two-dimensional image information before checking the three-dimensional information whether the photographer himself or the tripod is reflected in the captured image, or whether the three-dimensional information of the desired layout has not been acquired. becomes possible.

The transmitting/receiving unit 180 transmits the two-dimensional image information to the display device 500, and the display device 500 causes the display unit 520 to display the two-dimensional image information.

As a result, it is possible to easily confirm from the two-dimensional image information displayed on the display unit 520 that the photographer himself or the tripod is reflected in the captured image and that the three-dimensional information of the desired layout is not acquired. becomes possible.

The transmitting/receiving unit 180 transmits the two-dimensional image information to the display device 500 different from the external device 300 that outputs the three-dimensional information.

As a result, it is possible to check the three-dimensional information output to the external device 300 without checking the three-dimensional information output to the external device 300, such as whether the photographer himself or the tripod is reflected in the captured image, or whether the three-dimensional information of the desired layout has not been acquired. It is possible to confirm from the two-dimensional image information output to the display unit 520 of the display device 500 different from the device 300 .

The transmitting/receiving unit 180 may transmit the 3D information to the display device 500 . Specifically, the transmitting/receiving unit 180 may select a case of transmitting two-dimensional image information to the display device 500 and a case of transmitting three-dimensional information to the display device 500 according to a predetermined condition. Thereby, the transmitting/receiving section 180 can transmit the two-dimensional image information to the display device 500 separately from the three-dimensional information.

FIG. 13 is a flow chart for judging an approaching object in the second modified example.

FIG. 13 is a flowchart showing the process of determining whether or not a nearby object is reflected in the captured image corresponding to step S9 described in FIG. 5 in the second modification.

Based on the omnidirectional TOF image data acquired from the re-projection processing unit 147, the determination unit 160 determines whether the omnidirectional TOF image data has a saturated amount of electricity as an example of a pixel whose amount of electricity storage is equal to or greater than a predetermined value. It is determined whether or not there is a pixel in which the number of pixels is 0 (step S811).

If there is a pixel with a saturated charge amount in step S811, the determining unit 160 determines, based on the omnidirectional RGB image data acquired from the RGB image data acquiring unit 142, among the omnidirectional RGB image data, As an example of a pixel having the same coordinates as the pixel whose charged amount is saturated in step S811, it is determined whether the charged amount is saturated (step S812).

If the amount of stored electricity is saturated in step S812, the determination unit 160 determines that the pixels in which the amount of stored electricity is saturated in step S811 are caused by external light, and outputs error information to the display control unit 170. . The display control unit 170 causes the display units 20A and 20a to display the error information based on the error information acquired from the determination unit 160 (step S813).

If the amount of stored electricity is not saturated in step S812, the determination unit 160 determines that the pixels in which the amount of stored electricity is saturated in step S811 are due to the presence of a nearby object, and the amount of stored electricity is saturated in step S811. The coordinate position information of the pixel that has been displayed is output to display control section 170 . Based on the pixel coordinate position information acquired from the determination unit 160, the display control unit 170 determines whether the coordinate position information indicates the front side of the housing 10 (step S814).

If there is no pixel in which the amount of stored electricity is saturated in step S811, the determination unit 160 determines 0 It is determined whether there is a pixel indicating distance information of 5 m or less (step S815).

If there is no pixel indicating distance information of 0.5 m or less in step S815, the determination unit 160 ends the process.

If there is a pixel indicating distance information of 0.5 m or less in step S815, the determining unit 160 proceeds to the above-described step S814. and output the coordinate position information of the pixel indicating the distance information of 0.5 m or less to the display control unit 170 in step S815. Based on the pixel coordinate position information acquired from the determination unit 160 , the display control unit 170 determines whether the coordinate position information indicates the front side of the housing 10 .

If the display control unit 170 determines in step S814 that it is on the front side, it blinks the display unit 20a arranged on the front side of the housing 10 (step S816).

If the display control unit 170 does not determine in step S814 that it is on the front side, it blinks the display unit 20A arranged on the back side of the housing 10 (step S817).

As described above, the display control unit 170 blinks the display unit 20a or the display unit 20A when it determines that there is a nearby object, and blinks the display unit 20a or 20A when it determines that there is no nearby object. do not flash.

That is, the display control unit 170 causes the display unit 20a and the display unit 20A to display differently depending on the presence or absence of the nearby object.

As a result, the photographer can accurately check whether or not the photographer himself or a nearby object such as a tripod is reflected in the captured image, distinguishing it from the influence of external light.

Further, the display control unit 170 blinks the display unit 20a or the display unit 20A based on the pixel coordinate position information acquired from the determination unit 160 .

That is, the display control unit 170 causes the position of the display unit according to the position of the nearby object, that is, the display unit 20a or the display unit 20A to display differently. As a result, the photographer can confirm the position at which the nearby object is reflected in the captured image.

Then, the display control unit 170 causes the display unit closer to the nearby object, of the display units 20A and 20a, to display differently according to the presence or absence of the nearby object. As a result, the photographer can reliably confirm the position where the specific object is reflected in the captured image.

FIG. 14 is a diagram for explaining the configuration of an imaging device according to a third modification of the embodiment of the invention.

In a third modified example shown in FIG. 14, the imaging apparatus 1 includes other imaging elements 111a and 111A, other fisheye lenses (wide-angle lenses) 111b and 111B, etc. in addition to the configuration shown in FIG. An imaging unit 111 is provided.

In the third modification, the RGB imaging unit 11 and another imaging unit 111 are provided on the same base line. In this case, multi-view processing becomes possible in the processing circuit 14 . In other words, RGB images of two viewpoints can be obtained by simultaneously driving the image pickup section 11 and the other image pickup section 111 which are provided on one side with a predetermined distance therebetween. This allows the use of parallax calculated based on the two RGB images, further improving distance accuracy over the entire measurement range.

Specifically, when the RGB image capturing unit 11 and another image capturing unit 111 are provided, multi-baseline stereo (MSB) using an SSD, EPI processing, and the like can be used like conventional parallax calculation. Therefore, by using this, the reliability of parallax is increased, and high spatial resolution and accuracy can be achieved.

As described above, the imaging apparatus 1 includes another imaging unit 111, and the three-dimensional reconstruction processing unit 150 uses the output of the distance information acquisition unit 13, the two-dimensional image information, and the image captured by the other imaging unit 111. 3D information is determined based on the other 2D image information.

The imaging device 1 obtains three-dimensional information based on the other imaging unit 111, the two-dimensional image information, and the other two-dimensional image information imaged by the other imaging unit 111, without depending on the output of the distance information acquisition unit 13. and a three-dimensional information determination unit that determines the .

As a result, the three-dimensional reconstruction processing unit 150 determines based on the two-dimensional image information that the photographer himself or the tripod is reflected in the captured image, or that the three-dimensional information of the desired layout has not been acquired. It is possible to confirm from the two-dimensional image information captured by the imaging unit 11 without confirming the three-dimensional information.

FIG. 15 is a flow chart for determining a highly reflective object according to the embodiment of the present invention, and is a flowchart showing the process of determining whether or not a highly reflective object is reflected in the captured image corresponding to step S9 described in FIG. .

Based on the omnidirectional TOF image data acquired from the re-projection processing unit 147, the determination unit 160 determines whether the omnidirectional TOF image data has a saturated amount of electricity as an example of a pixel whose amount of electricity storage is equal to or greater than a predetermined value. It is determined whether or not there is a pixel in which the number of pixels is 0 (step S21).

If there is a pixel with a saturated charge amount in step S21, the determining unit 160 determines, based on the omnidirectional RGB image data acquired from the RGB image data acquiring unit 142, of the omnidirectional RGB image data, It is determined whether or not the RGB image data including the pixels at the same coordinates as the pixels whose charged amount is saturated in step S21 matches the reference information indicating the highly reflective object (step S22). A model image may be used as reference information indicating a highly reflective object, and the degree of matching between the RGB image data and the model image may be determined by image recognition. Further, parameters such as spectrum and color may be used as the reference information indicating the highly reflective object and the RGB image data, and the degree of matching may be determined based on a predetermined threshold value. Furthermore, the reference information may be stored in a table or may use a learning model.

The processing circuit 14 stores images of highly reflective objects such as metals and mirrors as model image information. It is determined whether the image matches the image of the highly reflective object.

If the determination unit 160 determines that the image acquired in step S22 matches the stored image of the highly reflective object, the determination unit 160 outputs the coordinate position information of the pixel determined in step S22 to the display control unit 170 (step S23). The display control unit 170 causes the display units 20 and 520 to display a display image including identification information for identifying a highly reflective object and two-dimensional image information based on the pixel coordinate position information acquired from the determination unit 160 ( Step S24), the process ends.

Steps S22 and S23 are examples of determination steps, and step S24 is an example of display steps.

If the judgment unit 160 judges that the image acquired in step S22 does not match the stored image of the highly reflective object, the judging unit 160 shifts to judgment of a nearby object (step S23), and performs the close object judgment flow shown in FIG. Execute (step S25).

As described above, the imaging apparatus 1 includes the determination unit 160 that determines whether or not there is a highly reflective object based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and and a display control unit 170 that causes the display units 20 and 520 to display differently accordingly.

As a result, the photographer can accurately confirm that a highly reflective object such as a mirror is included in the captured image by distinguishing it from the effects of nearby objects and external light.

The imaging device 1 includes a display section 20 . This allows the photographer to reliably confirm that the highly reflective object is included in the captured image.

The display control unit 170 causes the display units 20 and 520 to display differently according to the position of the highly reflective object. This allows the photographer to confirm the position of the highly reflective object.

Similar to the near object shown in FIG. 13, the display unit 20 includes a plurality of display units 20A and 20a. display differently depending on the presence or absence of an object. This allows the photographer to reliably confirm the position of the highly reflective object.

3, the display control unit 170 causes the display units 20 and 520 to display the image information G captured by the imaging unit 11, and the identification information for identifying the highly reflective object and the image information G and a display image including and are displayed on the display unit 20 or 520 . This allows the photographer to reliably confirm the position of the highly reflective object.

The determining unit 160 determines that the amount of stored electricity is saturated as an example of a pixel in which the amount of stored electricity due to the light received by the distance information acquisition unit 13 is equal to or greater than a predetermined value, and that the image information captured by the imaging unit is a highly reflective object. If it matches the model image information, which is an example of the reference information indicating , it is determined that there is a highly reflective object.

Accordingly, the photographer can accurately confirm that the highly reflective object is included in the captured image by distinguishing it from the effects of nearby objects and external light.

The imaging device 1 acquires distance information to an object based on the light received by the distance information acquiring section 13 . In this case, the photographer can confirm that the reason why the desired distance information is not acquired is not the nearby object or external light but the highly reflective object.

The imaging device 1 includes a transmission/reception section 180 that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition section 13 . In this case, the photographer can confirm that the reason why the desired three-dimensional information is not acquired is not a nearby object or external light but a highly reflective object.

FIG. 16 is a flowchart for determining distant objects and low-reflectance objects in this embodiment. It is a flow chart showing.

Based on the omnidirectional TOF image data acquired from the reprojection processing unit 147, the determination unit 160 determines whether or not there is a pixel in the omnidirectional TOF image data whose charge amount is equal to or less than a threshold for obtaining distance information. (Step S41).

If it is determined in step S41 that there is no pixel with the amount of stored electricity equal to or less than the threshold value, the determining unit 160 determines whether the omnidirectional TOF image data is If there is a pixel indicating distance information of 10 m or more, it is determined that the object is a distant object, and the coordinate position information of the pixel is output to the display control unit 170 (step S42).

Based on the pixel coordinate position information acquired from the determination unit 160, the display control unit 170 causes the display units 20 and 520 to display a display image including identification information for identifying distant objects and two-dimensional image information ( Step S43), the process ends.

If there is no pixel indicating distance information of 10 m or more in step S42, the determination unit 160 ends the process.

If in step S41 there is a pixel in which the amount of stored electricity is equal to or less than the threshold, the determining unit 160 determines, based on the omnidirectional RGB image data acquired from the RGB image data acquiring unit 142, the omnidirectional RGB image data, step It is determined whether or not the stored electricity amount is equal to or less than the threshold at which the object can be recognized, for pixels at the same coordinates as the pixels whose accumulated electricity amount is equal to or less than the threshold value in S41 (step S44).

If determination unit 160 determines in step S<b>44 that the amount of stored electricity is equal to or less than the object recognizable threshold, determination unit 160 determines that the object is a low-reflection object, and outputs the coordinate position information of the pixel to display control unit 170 .

The display control unit 170 displays, on the display units 20 and 520, a display image including identification information for identifying a low-reflection object and two-dimensional image information based on the pixel coordinate position information acquired from the determination unit 160 ( Step S45), the process ends.

If the judgment unit 160 judges in step S44 that the amount of stored electricity is not equal to or less than the object recognizable threshold, it is an example of reference information in which the image and the distance are associated with each other regarding the RGB image data including the pixels judged in step S44. A distance is determined based on the model information (step S46). When a model image is used as the reference information, the matching degree between the RGB image data and the model image may be determined by image recognition. Further, the degree of matching may be determined based on a predetermined threshold using parameters as the reference information and the RGB image data. Furthermore, the reference information may be stored in a table or may use a learning model.

The processing circuit 14 stores, as model information, an image associated with a distance for each of a plurality of distances, and the determination unit 160 uses a determination device such as AI in step S46 to acquire It is determined whether the obtained image matches the stored image for each distance.

The determination unit 160 determines whether the distance associated with the image that matches the image acquired in step S46 is 10 m or more. It outputs to the control unit 170 (step S47), and proceeds to step S43.

If the distance associated with the image that matches the image acquired in step S46 is not 10 m or more, the determination unit 160 determines that the object is a low-reflection object, and outputs the coordinate position information of the pixel to the display control unit 170. (Step S47), the process proceeds to step S45.

Steps S41, S42, S44, and S47 are examples of determination steps, and steps S43 and S45 are examples of display steps.

As described above, the imaging apparatus 1 includes the determination unit 160 that determines whether there is a distant object or a low-reflection object based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11; and a display control unit 170 that causes the display units 20 and 520 to display differently depending on the presence or absence of the low-reflection object.

This allows the photographer to accurately confirm that a distant object or a low-reflection object such as black is included in the captured image.

The imaging device 1 includes a display section 20 . This allows the photographer to reliably confirm that a distant object or a low-reflection object is included in the captured image.

The display control unit 170 causes the display units 20 and 520 to display differently according to the position of the distant object or the low-reflection object. This allows the photographer to confirm the position of a distant object or a low-reflection object.

Similar to the near object shown in FIG. 13, the display unit 20 includes a plurality of display units 20A and 20a. The display section of is made to display differently according to the presence or absence of an object. This allows the photographer to reliably confirm the position of the distant object or the low-reflectance object.

3, the display control unit 170 causes the display units 20 and 520 to display the image information G captured by the imaging unit 11, and identification information for identifying a distant object or a low-reflection object; A display image including the image information G is displayed on the display unit 20 or 520 . This allows the photographer to reliably confirm the position of the distant object or the low-reflectance object.

The determining unit 160 determines whether the object is a low-reflective object or a distant object based on the output of the imaging unit 11 when the amount of charge in the pixel due to the light received by the distance information acquiring unit 13 is equal to or less than a threshold. As a result, the photographer can accurately confirm which of the low-reflection object and the distant object is included in the captured image.

The determination unit 160 determines that there is a low-reflection object when the amount of charge in the pixel due to the light received by the distance information acquisition unit 13 is equal to or less than the threshold and the amount of charge in the pixel of the imaging unit 11 is equal to or less than the threshold.
This allows the photographer to accurately confirm that the low-reflection object is included in the captured image.

The determining unit 160 determines whether the amount of electricity stored in the pixels due to the light received by the distance information acquisition unit 13 is equal to or less than the threshold, the amount of electricity stored in the pixels of the imaging unit 11 is equal to or greater than the threshold, and the distance determined based on the pixels is equal to or greater than the threshold. If so, it is determined that there is a distant object.

This allows the photographer to accurately confirm that the distant object is included in the captured image.

The imaging device 1 acquires distance information to an object based on the light received by the distance information acquiring section 13 . In this case, the photographer can confirm that the reason why the desired distance information is not acquired is the distant object or the low-reflection object.

The imaging device 1 includes a transmission/reception section 180 that is an example of an output section that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition section 13 . In this case, the photographer can confirm that the reason why the desired three-dimensional information is not acquired is the distant object or the low-reflection object.

FIG. 17 is a flow chart for judging image blur in this embodiment, and is a flow chart showing the process for judging the presence or absence of image blur in a captured image corresponding to step S9 described with reference to FIG.

Based on the omnidirectional RGB image data acquired from the RGB image data acquiring unit 142, the determination unit 160 determines whether or not there is a pixel of an image including an edge peripheral region in the omnidirectional RGB image (step S51). .

Here, the determination unit 160 detects edges included in the image by comparing changes in luminance values of pixels, primary and secondary differential values thereof, and the like with threshold values, and detects pixels of the image including edge peripheral regions. Although specified, other methods may be used to detect edges.

Next, if there is a pixel of the image including the edge peripheral region in step S51, based on the omnidirectional TOF image data acquired from the reprojection processing unit 147, out of the omnidirectional TOF image data, in step S51 It is determined whether the edge of the TOF phase image is displaced with respect to the TOF image data including the pixel of the same coordinate as the pixel of the image determined to include the edge peripheral region. If it is determined that the edge is deviated, step S51. The coordinate position information of the pixel determined in 1 is output to the display control unit 170 (step S52).

The display control unit 170 causes the display units 20 and 520 to display a display image including identification information for identifying image blur and two-dimensional image information based on the pixel coordinate position information acquired from the determination unit 160 ( Step S53), the process ends.

Steps S51 and S52 are examples of determination steps, and step S53 is an example of a display step.

If there is no pixel of the image including the edge peripheral region in step S51, and if the edge of the TOF phase image is not shifted in step S52, the determination unit 160 ends the process.

In the present embodiment, the distance is measured by a phase difference detection method, and the imaging device 1 acquires N TOF phase images of the same phase for each phase of 0°, 90°, 180°, and 270°. to add.

By adding N phase images of the same phase in this manner, the dynamic range of the phase images of the phase is expanded. In addition, the time required to pick up N phase images to be added for each phase is shortened, so that phase images with excellent positional accuracy and less influence of blurring can be obtained. Therefore, by using the phase image with the expanded dynamic range, it is possible to accurately perform the image shift amount detection processing described below.

The determination unit 160 calculates the amount of pixel shift for each phase using a process for obtaining a general optical flow or the machine learning method disclosed in the following reference paper, and calculates the amount of pixel shift for each phase. The value added in all phases is compared with a threshold value to finally determine whether or not there is blurring in the image, but other methods may be used to determine whether or not there is blurring in the image.

Paper title Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset
Author Qi Guo (SEAS, Harvard University),
Iuri Frosio
Orazio Gallo
Todd Zickler (SEAS, Harvard University)
Jan Kautz
release date
Monday, September 10, 2018
Publisher
ECCV (European Conference on Computer Vision) 2018
URLs (Uniform Resource Locators)
https://research. nvidia. com/publication/2018-09_Tackling-3D-ToF

As described above, the imaging apparatus 1 includes the determination unit 160 that determines whether or not there is image blur based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and the and a display control unit 170 that causes the display units 20 and 520 to display differently accordingly.

This allows the photographer to accurately confirm that the image blur is included in the captured image.

The imaging device 1 includes a display section 20 . This allows the photographer to reliably confirm that the image blur is included in the captured image.

The display control unit 170 causes the positions of the display units 20 and 520 to display differently according to the position of the image blur. This allows the photographer to confirm the position of the image blur.

13, the display unit 20 includes a plurality of display units 20A and 20a. A display unit is made to display differently according to the presence or absence of an object. As a result, the photographer can reliably confirm the location of the image blurring.

Similar to the close object shown in FIG. and a display image including and are displayed on the display unit 20 or 520 . As a result, the photographer can reliably confirm the location of the image blurring.

The determination unit 160 detects an edge of the image based on the information of the image captured by the imaging unit 11, and determines that the image blurs when the light received by the distance information acquisition unit 13 causes pixel displacement.

This allows the photographer to accurately confirm that the image blur is included in the captured image.

The imaging device 1 acquires distance information to an object based on the light received by the distance information acquiring section 13 . In this case, the photographer can confirm that the reason why the desired distance information is not acquired is the blurring of the image.

The imaging device 1 includes a transmission/reception section 180 that is an example of an output section that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition section 13 . In this case, the photographer can confirm that the reason why the desired three-dimensional information is not acquired is the blurring of the image.

FIG. 18 is a decision flow diagram in the fourth modification of the embodiment of the invention.

In step S9 described with reference to FIG. 5, the determination unit 160 determines whether or not there is a specific object such as a nearby object, and the display control unit 170 controls the display units 20 and 520 to display different images depending on the presence or absence of the specific object. However, in the fourth modification, the determination unit 160 does not determine the presence or absence of the specific target, and the display control unit 170 controls the display units 20 and 520 according to the presence or absence of the specific target. does not allow the display to be different, but allows the user to recognize a particular object.

In the flow shown in FIG. 18( a ), the determination unit 160 determines whether the amount of charge in the omnidirectional TOF image data is equal to or greater than a predetermined value based on the omnidirectional TOF image data acquired from the reprojection processing unit 147 . As an example of a certain pixel, it is determined whether or not there is a pixel in which the amount of stored electricity is saturated and is equal to or greater than a threshold for obtaining distance information. Output to the control unit 170 (step S31).

Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 performs display including position identification information for identifying the position and two-dimensional image information, similar to the proximity object shown in FIG. The image is displayed on the display unit 20, 520 (step S32), and the process ends.

Judgment part 160 ends processing, when the amount of accumulation of electricity is not more than a threshold in Step S31.

In the flow shown in FIG. 18B, the determination unit 160 acquires the amount of stored electricity and the distance information from the omnidirectional TOF image data based on the omnidirectional TOF image data acquired from the reprojection processing unit 147. It is determined whether there is a pixel whose charge amount is below the threshold, and if there is a pixel whose charge amount is below the threshold, coordinate position information of the pixel is output to the display control unit 170 (step S33).

Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 performs display including position identification information for identifying the position and two-dimensional image information, similar to the proximity object shown in FIG. The image is displayed on the display unit 20, 520 (step S34), and the process ends.

Judgment part 160 ends processing, when the amount of accumulation of electricity is not below a threshold in Step S33.

In the flow shown in FIG. 18(c), the determination unit 160 determines whether the TOF phase image is shifted in the omnidirectional TOF image data based on the omnidirectional TOF image data acquired from the reprojection processing unit 147. It is determined whether there is a pixel for which distance information cannot be obtained, and if there is a pixel for which the TOF phase image is shifted, the coordinate position information of the pixel is output to the display control unit 170 (step S35).

Here, the judgment unit 160 judges the deviation of the TOF phase image by a method similar to the method described in step S52 of FIG.

Based on the coordinate position information of the pixel acquired from the determination unit 160, the display control unit 170 performs display including position identification information for identifying the position and two-dimensional image information, similar to the proximity object shown in FIG. The image is displayed on the display unit 20, 520 (step S36), and the process ends.

If there is no pixel for which the TOF phase image is shifted, the determination unit 160 ends the process.

As described above, the imaging device 1 determines whether the output of the distance information acquisition unit 13 is equal to or greater than the threshold value or equal to or less than the threshold value based on the position information indicating the position determined by the determination unit 160. , and two-dimensional image information G captured by the imaging unit 11 that captures an image of the target, and a display control unit 170 that causes the display units 20 and 520 to display a display image including the above.

As a result, a position where the output of the distance information acquisition unit 13 is equal to or greater than the threshold value or less than or equal to the threshold value, that is, a position where the output of the distance information acquisition unit 13 is too strong or too weak and a desired output cannot be obtained is displayed in the two-dimensional image G By confirming with , it is possible to confirm the cause of the failure to obtain the desired output.

In addition, the imaging device 1 obtains position identification information for identifying the position based on the position information indicating the position determined by the determination unit 160 that the distance information to the object cannot be acquired based on the output of the distance information acquisition unit 13, and the object and the two-dimensional image information G captured by the imaging unit 11 that captures .

Accordingly, by confirming the position where the distance information to the object cannot be obtained in the two-dimensional image G, it is possible to confirm the cause of the inability to obtain the distance information to the object.

The determination units 160, 560, and 660 determine that the distance information to the object cannot be acquired when the output of the distance information acquisition unit 13 is not equal to or greater than the threshold value or the threshold value or less. It also includes cases where blurring is detected.

FIG. 19 is a diagram showing an example of a configuration of processing blocks of a processing circuit in the fifth modification of the embodiment of the invention.

The processing blocks of the processing circuit in the fifth modification shown in FIG. determination unit 160 acquires spherical three-dimensional data from three-dimensional reconstruction processing unit 150; outputs determination results to transmission/reception unit 180; display control unit 170 performs three-dimensional reconstruction; A point of acquiring omnidirectional three-dimensional data from the configuration processing unit 150 is added.

In addition to the omnidirectional three-dimensional data output from the three-dimensional reconstruction processing unit 150 and the omnidirectional two-dimensional image information output from the RGB image data acquisition unit 142, the transmitting/receiving unit 180 receives the data from the determination unit 160. The judgment result is transmitted (output) via the network 400 to the external device 300 that performs the three-dimensional restoration processing.

The display control unit 170 causes the display unit 20 to display a three-dimensional image based on the omnidirectional three-dimensional data acquired from the three-dimensional reconstruction processing unit 150. A display image including identification information for identifying a specific target and a three-dimensional image is displayed on the display unit 20 based on the determination result of the determination unit 160 that determines whether there is a specific target based on both the output of Let Specific objects include near objects, highly reflective objects, distant objects, low reflective objects and also image blur regions.

Accordingly, it is possible to confirm, by viewing the 3D image 3G, which of the nearby object, the high reflective object, the distant object, the low reflective object, and the blurring of the image is the reason why the desired 3D image 3G is not displayed. can.

FIG. 20 is a diagram showing an example of the configuration of an information processing system according to the sixth modification of the embodiment of the invention.

An information processing system according to the sixth modification shown in FIG. 20 includes an imaging device 1 and a display device 500 .

The image pickup apparatus 1 shown in FIG. 20 includes image pickup elements 11a and 11A, TOF sensors 13a and 13A, light source sections 12a and 12A, and a photographing switch 15, which are configured similarly to those shown in FIG.

The processing circuit 4 in the imaging device 1 shown in FIG. The control unit 141 is configured similarly to that shown in FIG.

Similar to FIG. 4, the RGB image data acquisition unit 142 acquires the RGB image data captured by the image sensors 11a and 11A based on the image capturing instruction from the control unit 141, and outputs the omnidirectional RGB image data. , in that the output destination is the transmission/reception unit 180. FIG.

As in FIG. 4, the TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13a and 13A based on the TOF image data generation instruction from the control unit 141, and obtains omnidirectional TOF image data. , but differs from FIG. 4 in that the output destination is the transmission/reception unit 180 .

4, the transmitting/receiving unit 180 transmits the omnidirectional RGB image data output from the RGB image data obtaining unit 142 and the omnidirectional TOF image data output from the TOF image data obtaining unit 144 to the display device 500. Send (output).

A display device 500 shown in FIG. 20 includes a transmitting/receiving unit 510, a display unit 520, and a display control unit 530, similarly to the second modification shown in FIG.
RGB image data acquisition unit 542, monochrome processing unit 543, TOF image data acquisition unit 544, resolution enhancement unit 545, matching processing unit 546, reprojection processing unit 547, semantic segmentation unit 548, and parallax calculation It further includes a unit 549 , a three-dimensional reconstruction processing unit 550 and a determination unit 560 .

The transmitting/receiving unit 180 receives the omnidirectional RGB image data and the omnidirectional TOF image data transmitted from the imaging device 1 .

The RGB image data acquisition unit 542 acquires omnidirectional RGB image data from the transmission/reception unit 180, and the TOF image data acquisition unit 544 acquires omnidirectional RGB image data from the transmission/reception unit 180. , are configured similarly to the RGB image data acquisition unit 142 and the TOF image data acquisition unit 144 shown in FIG.

A monochrome processing unit 543, a TOF image data acquisition unit 544, a resolution enhancement unit 545, a matching processing unit 546, a reprojection processing unit 547, a semantic segmentation unit 548, a parallax calculation unit 549, and three-dimensional reconstruction. The processing unit 550 and the determination unit 560 include the monochrome processing unit 143, the TOF image data acquisition unit 144, the resolution enhancement unit 145, the matching processing unit 146, the reprojection processing unit 147, and It is configured in the same manner as the semantic segmentation unit 148 , parallax calculation unit 149 , 3D reconstruction processing unit 150 and determination unit 160 .

The display control unit 530 may acquire the omnidirectional RGB image data from the RGB image data acquiring unit 542 and cause the display unit 520 to display a two-dimensional image based on the acquired omnidirectional RGB image data. The three-dimensional data of the sphere may be acquired from the dimensional reconstruction processing unit 545 and the three-dimensional image information may be displayed on the display unit 520 .

Display control unit 530 causes display unit 520 to display a display image including information indicating the determination result acquired from determination unit 160 and a two-dimensional image or a three-dimensional image.

As described above, the display device 500 includes the output of the imaging unit 11 that captures an image of the target, and the receiving unit that receives the output of the distance information acquisition unit 13 that projects light onto the target and receives the light reflected from the target. A transmitting/receiving unit 510 which is an example, a determining unit 560 which determines whether or not there is a specific object based on both the output of the distance information acquiring unit 13 received by the transmitting/receiving unit 510 and the output of the imaging unit 11; and a display control unit 530 that causes the display unit to display differently according to the presence or absence of a specific target based on the determination result of the unit 560 .

Specific objects include near objects, highly reflective objects, distant objects, low reflective objects and blurred areas of the image.

Further, the display device 500 detects a specific target based on both the output of the imaging unit 11 that captures an image of the target and the output of the distance information acquisition unit 13 that projects light onto the target and receives light reflected from the target. Based on the result of determination by determination unit 560 that determines whether there is A display control unit 530 for displaying is provided.

FIG. 21 is a diagram showing an example of the configuration of an information processing system according to the seventh modification of the embodiment of the invention.

The information processing system according to the seventh modification shown in FIG. 21 includes an imaging device 1 , a display device 500 and a server 600 .

The imaging device 1 shown in FIG. 21 is configured similarly to the imaging device 1 shown in FIG. 20, and the display device 500 shown in FIG. 21 is configured similarly to the display device 500 shown in FIG.

The server 600 shown in FIG. 21 includes a reception unit 610, an RGB image data acquisition unit 642, a monochrome processing unit 643, a TOF image data acquisition unit 644, a resolution enhancement unit 645, a matching processing unit 646, a reprojection A processing unit 647 , a semantic segmentation unit 648 , a parallax calculation unit 649 , a three-dimensional reconstruction processing unit 650 , a determination unit 660 and a transmission unit 680 are provided.

The receiving unit 610 receives omnidirectional RGB image data and omnidirectional TOF image data transmitted from the imaging apparatus 1 via the network 400 .

The RGB image data obtaining unit 642 obtains omnidirectional RGB image data from the receiving unit 610, and the TOF image data obtaining unit 644 obtains omnidirectional RGB image data from the receiving unit 610. , are configured similarly to the RGB image data acquisition unit 142 and the TOF image data acquisition unit 144 shown in FIG.

A monochrome processing unit 643, a TOF image data acquisition unit 644, a resolution enhancement unit 645, a matching processing unit 646, a reprojection processing unit 647, a semantic segmentation unit 648, a parallax calculation unit 649, and three-dimensional reconstruction. The processing unit 650 and the determination unit 660 include the monochrome processing unit 143, the TOF image data acquisition unit 144, the resolution enhancement unit 145, the matching processing unit 146, the reprojection processing unit 147, and It is configured in the same manner as the semantic segmentation unit 148 , parallax calculation unit 149 , 3D reconstruction processing unit 150 and determination unit 160 .

The transmission unit 680 receives the spherical three-dimensional data output from the three-dimensional reconstruction processing unit 150, the spherical two-dimensional image information output from the RGB image data acquisition unit 142, and the judgment of the judgment unit 160. , are transmitted (output) to the display device 500 via the network 400 .

The transmitting/receiving unit 510 of the display device 510 receives the spherical three-dimensional data, the two-dimensional image information, and the determination result of the determination unit 160 transmitted from the server 600 .

The display control unit 530 of the display device 510 may acquire the omnidirectional RGB image data from the transmitting/receiving unit 510 and cause the display unit 520 to display a two-dimensional image based on the acquired omnidirectional RGB image data. It is also possible to obtain omnidirectional three-dimensional data from the transmitting/receiving unit 510 and display the three-dimensional image information on the display unit 20 .

Display control unit 530 causes display unit 520 to display a display image including information indicating the determination result acquired from transmission/reception unit 510 and a two-dimensional image or a three-dimensional image.

As described above, the display device 500 is based on both the output of the imaging unit 11 that captures an image of the target and the output of the distance information acquisition unit 13 that projects light onto the target and receives light reflected from the target. Transmitting/receiving unit 510 that receives the result of determination by determination unit 660 of server 600 as to whether there is a specific target, and based on the determination result received by transmitting/receiving unit 510, different displays are displayed on display unit 520 according to the presence or absence of the specific target. and a display control unit 530 that causes the Specific objects include near objects, highly reflective objects, distant objects, low reflective objects and blurred areas of the image.

Further, the display device 500 detects a specific target based on both the output of the imaging unit 11 that captures an image of the target and the output of the distance information acquisition unit 13 that projects light onto the target and receives light reflected from the target. Based on the determination result of determination unit 660 that determines whether there is A display control unit 530 for displaying is provided.

FIG. 22 is a diagram for explaining display contents of the display section in the fifth to seventh modifications.

As shown in FIG. 22, the display control unit 530 displays a three-dimensional image 3G including identification information 3Ga, 3Gb, and 3Gc for identifying a specific target on the display unit 520 . 3Ga, 3Gb and 3Gc may be location identification information that identifies the location of a particular object.

Although FIG. 22 shows the display unit 520, the display control unit 170 similarly displays a three-dimensional image 3G including identification information 3Ga, 3Gb, and 3Gc for identifying a specific target on the display unit 20 as well.

Here, the identification information 3Ga indicates a blind spot and is identified and displayed in pink or the like, the identification information 3Gb indicates a low reflective object and is identified and displayed in orange or the like, and the identification information 3Gc indicates a distant object and is identified and displayed by mosaic or the like. there is

All of these identification information 3Ga, 3Gb and 3Gc may be displayed at the same time, or any one or two of them may be displayed.

FIG. 23 is a diagram for explaining a three-dimensional image displayed by the display unit according to the embodiment of the present invention;

FIG. 23(a) is a diagram showing the positions of the virtual camera and the predetermined area when the omnidirectional image is a three-dimensional solid sphere. The virtual camera IC corresponds to the position of the viewpoint of the user viewing the omnidirectional image CE displayed as a three-dimensional sphere.

FIG. 23(b) is a three-dimensional perspective view of FIG. 23(a), and FIG. 23(c) is a diagram showing a predetermined area image displayed on the display.

FIG. 23(b) represents the omnidirectional image CE shown in FIG. 23(a) with a three-dimensional solid sphere CS. Assuming that the omnidirectional image CE generated in this way is the stereosphere CS, the virtual camera IC is positioned inside the omnidirectional image CE as shown in FIG. ing.

A predetermined region T in the omnidirectional image CE is a photographing region of the virtual camera IC, and is specified by predetermined region information indicating the photographing direction and angle of view of the virtual camera IC in the three-dimensional virtual space including the omnidirectional image CE. be.

Further, the zooming of the predetermined area T can also be expressed by moving the virtual camera IC closer to or farther from the omnidirectional image CE. The predetermined area image Q is an image of the predetermined area T in the omnidirectional image CE. Therefore, the predetermined area T can be specified by the angle of view α and the distance f from the virtual camera IC to the omnidirectional image CE.

That is, the display control units 170 and 530 change the position and orientation of the virtual camera IC at the position of the viewpoint for viewing the three-dimensional image 3G, thereby changing the display area of the three-dimensional image 3G to be displayed on the display units 20 and 520. change.

In the above description, the three-dimensional image displayed by the display unit is described as an example using a omnidirectional image, but the same applies when using three-dimensional point cloud data. A three-dimensional point group is arranged in a virtual space, and a virtual camera is arranged in the virtual space. A three-dimensional image is obtained by projecting a three-dimensional point group onto a predetermined projection plane in the virtual space based on predetermined area information indicating the virtual camera's viewpoint position, photographing direction, and angle of view. Also, by changing the viewpoint position and orientation of the virtual camera, the display area of the three-dimensional image is changed.

FIG. 24 is a judgment flow diagram in the fifth to seventh modifications. Based on the spherical three-dimensional data acquired from the three-dimensional reconstruction processing units 150, 550, and 650, the determination units 160, 560, and 660 determine the density of point cloud data in the spherical three-dimensional data as a threshold value. It is determined whether there is an area (coordinates) of less than (step S61).

If the determination units 160, 560, and 660 determine in step S61 that there is an area (coordinates) in which the density of the point cloud data is less than the threshold, a plurality of pixels having the same coordinates as the area in which the density of the point cloud data is less than the threshold are determined. , based on the output of the imaging unit 11 in the flow shown in FIG. Output to the units 170 and 530 (step S62).

Based on the pixel coordinate position information acquired from the determination units 160, 560, and 660, the display control units 170 and 530, as shown in FIG. A display image including G and is displayed on the display unit 20, 520 (step S63), and the process ends.

In step S62, the determining units 160, 560, and 660 determine that a plurality of pixels having the same coordinates as the area where the density of the point cloud data is less than the threshold do not include pixels determined to be distant objects, as shown in FIG. In the flow, based on the output of the imaging unit 11, it is determined whether a pixel determined as a low-reflectance object is included. (step S64).

Based on the pixel coordinate position information acquired from the determination units 160, 560, and 660, the display control units 170 and 530, as shown in FIG. A display image including the image G is displayed on the display unit 20, 520 (step S65), and the process ends.

In step S64, the determining units 160, 560, and 660 determine that a plurality of pixels having the same coordinates as the area in which the density of the point cloud data is less than the threshold value are blind spots when pixels determined as low-reflecting objects are not included. Then, the pixel coordinate position information is output to the display control units 170 and 530 .

Based on the pixel coordinate position information acquired from the determination units 160, 560, and 660, the display control units 170 and 530, as shown in FIG. and is displayed on the display unit 20, 520 (step S66), and the process ends. Steps S61, S62 and S64 are examples of determination steps, and steps S63, S65 and S66 are examples of display steps.

As described above, the imaging device 1 and the display device 500 combine the output of the imaging unit 11 that images an object with the output of the distance information acquisition unit 13 that projects light onto the object and receives light reflected from the object. Based on the determination results of the determination units 160, 560, and 660 that determine whether there is a specific target based on both, based on the identification information 3Ga, 3Gb, and 3Gc for identifying the specific target, and based on the output of the distance information acquisition unit 13 A display control unit 170 that causes the display units 20 and 520 to display differently a display image including a three-dimensional image 3G determined by the three-dimensional reconstruction processing units 150, 550, and 650, which are examples of the three-dimensional information determination unit; 530.

Specific objects include not only distant objects, low reflectivity objects and blind spots, but also close objects, high reflectivity objects and blurred areas of the image.

As a result, by viewing the 3D image 3G, it is confirmed which of the distant object, the low reflective object, the blind spot, the close object, the high reflective object, and the blurring of the image is the reason why the desired 3D image 3G is not displayed. can do.

In addition, the imaging device 1 and the display device 500 cause the display units 20 and 520 to display a three-dimensional image 3G determined based on the output of the distance information acquisition unit 13 that receives light projected onto and reflected from the object. Display control units 170 and 530 are provided, and the display control units 170 and 530 control a distant object, which is far from the distance information acquisition unit 13 when light reflected from the object is received in the three-dimensional image 3G, and a projected object. Based on the position information indicating the position determined to be at least one of the low reflective object with low reflectance for the light reflected from the object and the blind spot with respect to the distance information acquisition unit 13 when the light reflected from the object is received. , position identification information 3Ga, 3Gb or 3Gc for identifying at least one position of a low-reflective object and a blind spot, and a three-dimensional image 3G.

As a result, it is possible to confirm by viewing the three-dimensional image 3G which of the distant object, the low-reflecting object, and the blind spot is the reason why the desired three-dimensional image 3G is not displayed. It becomes possible to take measures such as imaging.

The three-dimensional image 3G is determined by three-dimensional reconstruction processing units 150, 550, and 650, which are examples of three-dimensional information determination units.

The display control units 170 and 530 display one of the position identification information 3Ga, 3Gb, and 3Gc and the three-dimensional image 3G based on the position information of any one of the distant object, the low-reflection object, and the blind spot. The image may be displayed on the display unit 20, 520, and any two or more of the position identification information 3Ga, 3Gb and 3Gc may be displayed based on the position information of any two or all of the distant object, the low reflective object, and the blind spot. A display image including all and the three-dimensional image 3G may be displayed on the display unit 20, 520. FIG.

When the information processing device is the imaging device 1, the imaging device 1 includes the distance information acquisition unit 13 and the three-dimensional reconstruction processing unit 150 as shown in FIG.

When the information processing device is the display device 500, as shown in FIGS. Then, the output of the distance information acquisition unit 13 is transmitted to the display device 500 or the server 600 .

The display device 500 may or may not include a three-dimensional reconstruction processing unit 550 as shown in FIG.

When the display device 500 does not include the three-dimensional reconstruction processing unit 550, the imaging device 1 may include the three-dimensional reconstruction processing unit 150 and transmit a three-dimensional image to the display device 500, as shown in FIG. Alternatively, the server 600 may include a 3D reconstruction processing unit 650 and transmit the 3D image to the display device 500 .

The display control unit 170, 530 indicates a position where the density of the point cloud data included in the 3D image 3G is lower than the threshold and is determined to be at least one of a distant object, a low reflective object, and a blind spot. Based on the positional information, the display unit 20, 520 is caused to display a display image including the positional identification information 3Ga, 3Gb, 3Gc and the three-dimensional image 3G.

With this, it is possible to confirm, by viewing the three-dimensional image 3G, which of the distant object, the low-reflection object, and the blind spot is the cause of the density of the point cloud data being lower than the threshold.

The display control units 170 and 530 display a position that is determined to be at least one of a distant object, a low-reflection object, and a blind spot in the three-dimensional image 3G based on the output of the imaging unit 11 that images the object. Based on the information, the display unit 20, 520 is caused to display a display image including the position identification information 3Ga, 3Gb, 3Gc and the three-dimensional image 3G.

Accordingly, it is possible to accurately determine whether the cause of not displaying the desired three-dimensional image 3G is a distant object, a low-reflection object, or a blind spot, based on the output of the imaging unit 11 .

When the information processing device is the imaging device 1, the imaging device 1 includes an imaging unit 11 as shown in FIG. When the information processing device is the display device 500, the display device 500 does not include the imaging unit 11 as shown in FIGS. 11 to display device 500 or server 600 .

The imaging device 1 and the display device 500 include determination units 160, 560, and 660 that determine positions of at least one of a distant object, a low-reflection object, and a blind spot in the three-dimensional image 3G, a display control unit 170, Based on the determination results of the determination units 160, 560, and 660, the 530 causes the display units 20 and 520 to display display images including the position identification information 3Ga, 3Gb, and 3Gc and the three-dimensional image 3G.

When the information processing device is the imaging device 1, the imaging device 1 is provided with a determination unit 160 as shown in FIG.

When the information processing device is the display device 500, the display device 500 may or may not include the determination section 560 as shown in FIG.

If the display device 500 does not include the determination unit 560, the imaging device 1 may include the determination unit 160 and transmit the determination result to the display device 500, and the server 600 may include the determination unit 660 as shown in FIG. You may transmit a judgment result to the display apparatus 500 by pressing.

FIG. 25 is another diagram for explaining the display contents of the display section in the fifth to seventh modifications.

As shown in FIG. 25, the display unit 520 has a three-dimensional display including position identification information 3G1 and 3G2 for identifying the position of the distance information acquisition unit 13 when the light reflected from the object is received by the display control unit 530. Image 3G is displayed.

The three-dimensional image 3G is determined based on the output of the distance information acquisition unit 13 positioned at the first position and the output of the distance information acquisition unit 13 positioned at a second position different from the first position. The identification information 3G1 is an example of first position identification information that identifies a first position, and the position identification information 3G2 is an example of first position identification information that identifies a second position.

Although FIG. 25 shows the display unit 520, the display control unit 170 also displays the position identification information for identifying the position of the distance information acquisition unit 13 when the light reflected from the object is received. A three-dimensional image 3G including 3G1 and 3G2 is displayed.

The display control units 170 and 530 cause the display units 20 and 520 to display display images including a three-dimensional image 3G and identification information 3Ga, 3Gb and 3Gc, which are examples of low-density identification information, as shown in FIG. Along with the display, as shown in FIG. 25, the display image may include position identification information 3G1 and 3G2 for identifying the position of the distance information acquisition unit 13 when the light reflected from the object is received.

FIG. 26 is a flowchart for explaining processing in the fifth to seventh modifications.

The three-dimensional reconstruction processing units 150, 550, and 650 read the omnidirectional high-density three-dimensional point cloud data (step S71), and set the origin of the three-dimensional point cloud data to the light reflected from the object. It is acquired as position information indicating the imaging position of the distance information acquisition unit 13 (step S72).

The three-dimensional reconstruction processing units 150, 550, and 650 confirm whether or not there is pre-loaded three-dimensional point cloud data. The dimensional point cloud data and the position information acquired in step S72 are output to the display control units 170 and 530 (step S73).

The display control units 170 and 530 receive the light reflected from the object as shown in FIG. A display image including the position identification information 3G1 for identifying the position of the distance information acquisition unit 13 at that time and the three-dimensional image 3G is displayed on the display units 20 and 520 (step S74), and the process ends.

If there is 3D point cloud data read in advance in step S73, the 3D reconstruction processing units 150, 550, and 650 combine the 3D point cloud data read in step S71 with the 3D data read in advance. The point cloud data are integrated (step S75).

The three-dimensional reconstruction processing units 150, 550, and 650 convert the origin of the three-dimensional point cloud data read in step S71 and the origin of the three-dimensional point cloud data read in advance into the integrated three-dimensional data in step S75. The coordinates in the point cloud data are calculated as the position information of the imaging position, and the three-dimensional point cloud data integrated in step S75 and the calculated plurality of position information are output to the display control units 170 and 530 (step S76).

In step S74, the display control units 170 and 530, based on the three-dimensional point cloud data acquired from the three-dimensional reconstruction processing units 150, 550, and 650 and a plurality of pieces of position information, as shown in FIG. A display image including a plurality of position identification information 3G1 and 3G2 for identifying the position of the distance information acquisition section 13 when receiving the light coming from the display section 20 and the three-dimensional image 3G is displayed on the display sections 20 and 520 .

FIG. 27 is another flowchart for explaining the processing in the fifth to seventh modifications.

The three-dimensional reconstruction processing units 150, 550, and 650 read the high-density three-dimensional point cloud data of the omnisphere (step S81). Based on the spherical three-dimensional data acquired from 650, steps S61, S62, and S64 of the flow shown in FIG. ).

When the virtual camera IC shown in FIG. 23 is at the position of the position identification information 3G1 or 3G2 shown in FIG. Then, the orientation of the virtual camera IC is changed so that at least one of the identification information 3Ga, 3Gb, and 3Gc, which are examples of the low-density identification information shown in FIG. 22, is included in the displayed image (step S83).

As described above, the imaging device 1 and the display device 500 include the display control units 170 and 530 that cause the display units 20 and 520 to display the three-dimensional image 3G determined based on the output of the distance information acquisition unit 13. The control units 170 and 530 determine the position of the distance information acquisition unit 13 when the light reflected from the object is received based on the position information indicating the position of the distance information acquisition unit 13 when the light reflected from the object is received. and a three-dimensional image 3G are displayed on the display units 20 and 520.

Thereby, the positional relationship between the imaging position indicating the position of the distance information acquisition unit 13 when the light reflected from the target is received and the specific target can be grasped in the three-dimensional image 3G.

The three-dimensional image 3G and position information are determined by the three-dimensional reconstruction processing units 150, 550, and 650. FIG.

When the information processing device is the imaging device 1, the imaging device 1 includes the distance information acquisition unit 13 and the three-dimensional reconstruction processing unit 150 as shown in FIG.

When the information processing device is the display device 500, as shown in FIGS. Then, the output of the distance information acquisition unit 13 is transmitted to the display device 500 or the server 600 .

The display device 500 may or may not include a three-dimensional reconstruction processing unit 550 as shown in FIG.

If the display device 500 does not include the three-dimensional reconstruction processing unit 550, the imaging device 1 may include the three-dimensional reconstruction processing unit 150 and transmit the three-dimensional image and the position information to the display device 500. FIG. As indicated, the server 600 may include a 3D reconstruction processor 650 to transmit 3D images and location information to the display device 500 .

The display control units 170 and 530 generate identification information 3Ga, 3Gb, and 3Gc, which are examples of low-density identification information for identifying regions, based on region information indicating regions in the three-dimensional image 3G where the density of the point cloud data is lower than the threshold value. , and the three-dimensional image 3G are displayed on the display units 20 and 520 .

In this case, since the positional relationship between the imaging position and the area where the density of the point cloud data is lower than the threshold can be grasped, it is possible to specify the factor that the density of the point cloud data is lower than the threshold. For example, if the area is far from the imaging position, a distant object is the factor, if the area is in the blind spot of the imaging position, the blind spot is the factor, and if neither the distant object nor the blind spot is the factor, the low reflective object is the factor. One thing can be identified.

The display control units 170 and 530 change the display area of the three-dimensional image 3G to be displayed on the display units 20 and 520 by changing the position and orientation of the virtual camera IC at the position of the viewpoint for viewing the three-dimensional image 3G. .

The display control units 170 and 530 change the orientation of the virtual camera IC to a predetermined orientation when the position of the virtual camera IC is at the positions 3G1 and 3G2 identified by the position identification information. Predetermined orientations include locations that cause re-imaging such as low-density point cloud regions, locations that meet preset conditions such as locations to be confirmed in field surveys, and locations that the photographer or other confirmation person focuses on. An arbitrary portion or the like is an area that is included in the display area. Examples of sites to be checked in a site survey include, for example, a construction site, where changes occur continuously on the site (material storage area), the position of each object in the permanent structure (the building itself), the gap distance between objects, Space for new installations, temporary facilities (material storage, scaffolding, etc., that are attached and removed during the construction process), storage for heavy machinery (forks, cranes), workspace (rotating range, delivery route), flow line for residents (Detour during construction) etc.

This makes it possible to direct the line of sight of the user positioned at the imaging position to a specific object that the user wishes to see on site.

The display control units 170 and 530 change the orientation of the virtual camera IC so that the display area includes a low-density portion in which the density of predetermined coordinates and point cloud data in the three-dimensional image 3G is lower than a threshold value. The predetermined coordinates do not specify the image, and are maintained even if the image at the predetermined coordinates changes before and after the three-dimensional point cloud data are integrated in step S75 of FIG. 26, for example.

As a result, it is possible to direct the line of sight of the user positioned at the imaging position to the specific object that is the low-density portion in the three-dimensional image 3G.

The display control units 170 and 530 are determined based on the output of the distance information acquisition unit 13 positioned at the first position and the output of the distance information acquisition unit 13 positioned at a second position different from the first position. The dimensional image 3G is displayed on the display unit 20, 520, the first position identification information 3G1 for identifying the first position, the second position identification information 3G2 for identifying the second position, the three-dimensional image 3G, is displayed on the display unit 20, 520.

Thereby, the positional relationship between the first imaging position, the second imaging position, and the specific target can be grasped in the three-dimensional image 3G.

●Summary●
As described above, the image capturing apparatus 1 according to the embodiment of the present invention includes the image capturing unit 11 that captures an image of an object, the projection unit 12 that projects light onto the object, and the distance information that receives the light reflected from the object. an acquisition unit 13 (an example of a light receiving unit), a determination unit 160 that determines whether there is a highly reflective object based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11; and a display control unit 170 that causes the display units 20 and 520 to display differently accordingly.

As a result, the photographer can accurately confirm that a highly reflective object such as a mirror is included in the captured image by distinguishing it from the effects of nearby objects and external light.

The imaging device 1 includes a display section 20 . This allows the photographer to reliably confirm that the highly reflective object is included in the captured image.

The display control unit 170 causes the display units 20 and 520 to display differently according to the position of the highly reflective object. This allows the photographer to confirm the position of the highly reflective object.

The display unit 20 includes a plurality of display units 20A and 20a, and the display control unit 170 causes the display unit 20A and 20a closer to the highly reflective object to display different images depending on the presence or absence of the object. Let This allows the photographer to reliably confirm the position of the highly reflective object.

The display control unit 170 causes the display units 20 and 520 to display the image information G captured by the imaging unit 11, and displays a display image including the image information G and identification information for identifying a highly reflective object on the display units 20 and 520. 520 to display. This allows the photographer to reliably confirm the position of the highly reflective object.

The determining unit 160 determines that the amount of stored electricity is saturated as an example of a pixel in which the amount of stored electricity due to the light received by the distance information acquisition unit 13 is equal to or greater than a predetermined value, and that the image information captured by the imaging unit is a highly reflective object. If it matches the model image information as an example of the reference information indicating , it is determined that there is a highly reflective object.

Accordingly, the photographer can accurately confirm that the highly reflective object is included in the captured image by distinguishing it from the effects of nearby objects and external light.

The imaging device 1 acquires distance information to an object based on the light received by the distance information acquiring section 13 . In this case, the photographer can confirm that the reason why the desired distance information is not acquired is not the nearby object or external light but the highly reflective object.

The imaging device 1 includes a transmission/reception section 180 that is an example of an output section that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition section 13 . In this case, the photographer can confirm that the reason why the desired three-dimensional information is not acquired is not a nearby object or external light but a highly reflective object.

An imaging processing method according to an embodiment of the present invention includes an imaging step of imaging an object with an imaging unit 11, a projection step of projecting light onto the object with a projection unit 12, and a distance information acquiring unit that receives light reflected from the object. 13, a determination step of determining whether or not there is a highly reflective object based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a high reflective and a display step of causing the display units 20 and 520 to perform different displays by the display control units 170 and 530 according to the presence or absence of the object.

An imaging device 1 and a display device 500, which are examples of an information processing apparatus according to an embodiment of the present invention, receive an output of an imaging unit 11 that images an object, project light onto the object, and receive light reflected from the object. Based on the judgment results of the judgment units 160, 560, and 660 that judge whether there is a highly reflective object based on both the output of the distance information acquisition unit 13 and the output of the distance information acquisition unit 13, the display units 20 and 520 display It has display control units 170 and 530 that cause different displays.

A display device 500, which is an example of an information processing device according to an embodiment of the present invention, acquires output of an imaging unit 11 that images an object, and distance information acquisition that projects light onto the object and receives light reflected from the object. A transmitting/receiving unit 510, which is an example of a receiving unit that receives a determination result by the determining unit 160 of the imaging device 1 or the determining unit 660 of the server 600 as to whether there is a specific target based on both the output of the unit 13 and the transmitting/receiving unit 510 and a display control unit 530 that causes the display unit 520 to display differently according to the presence or absence of a specific target based on the determination result received by the. Specific objects include near objects, highly reflective objects, distant objects, low reflective objects, blind spots and image blur areas.

A display device 500, which is an example of an information processing device according to an embodiment of the present invention, acquires output of an imaging unit 11 that images an object, and distance information acquisition that projects light onto the object and receives light reflected from the object. Transmitting/receiving unit 510, which is an example of a receiving unit that receives the output of unit 13, and whether there is a specific target based on both the output of distance information acquiring unit 13 received by transmitting/receiving unit 510 and the output of imaging unit 11 and a display control unit 530 that causes the display unit to display differently according to the presence or absence of a specific target based on the determination result of the determination unit 560 . Specific objects include near objects, highly reflective objects, distant objects, low reflective objects, blind spots and image blur areas.

An imaging device 1 and a display device 500, which are examples of an information processing apparatus according to an embodiment of the present invention, receive an output of an imaging unit 11 that images an object, project light onto the object, and receive light reflected from the object. Identification information 3Ga, 3Gb, and 3Gc for identifying a specific target based on the determination results of determination units 160 and 560 that determine whether there is a specific target based on both the output of the distance information acquisition unit 13 and the three-dimensional and a display control unit 170, 530 for causing the display unit 20, 520 to display a display image including the image 3G. Specific objects include not only distant objects, low reflectivity objects and blind spots, but also close objects, high reflectivity objects and blurred areas of the image.

The three-dimensional image 3G is determined by the three-dimensional reconstruction processing units 150, 550, and 650, which are examples of the three-dimensional information determination unit, based on the output of the distance information acquisition unit 13.

As a result, by viewing the 3D image 3G, it is confirmed which of the distant object, the low reflective object, the blind spot, the close object, the high reflective object, and the blurring of the image is the reason why the desired 3D image 3G is not displayed. can do.

In the imaging device 1 and the display device 500, which are examples of an information processing device according to an embodiment of the present invention, the output of the distance information acquisition unit 13 that projects light onto an object and receives light reflected from the object is equal to or greater than a threshold. Alternatively, based on the position information indicating the position determined by the determination unit 160 or 560 as to whether it is equal to or less than the threshold, the position identification information for identifying the position, the two-dimensional image information G captured by the imaging unit 11 that captures the image of the target, and display control units 170 and 530 that cause the display units 20 and 520 to display a display image containing

As a result, a position where the output of the distance information acquisition unit 13 is equal to or greater than the threshold value or less than or equal to the threshold value, that is, a position where the output of the distance information acquisition unit 13 is too strong or too weak and a desired output cannot be obtained is displayed in the two-dimensional image G By confirming with , it is possible to confirm the cause of the failure to obtain the desired output.

An imaging device 1 and a display device 500, which are examples of an information processing device according to an embodiment of the present invention, project light onto a target and receive light reflected from the target based on the output of the distance information acquisition unit 13. Position identification information for identifying the position based on the position information indicating the position determined by the determination unit 160, 560 that the distance information to the target cannot be acquired, and two-dimensional image information G captured by the image capturing unit 11 that captures an image of the target. and display control units 170 and 530 that cause the display units 20 and 520 to display display images including .

Accordingly, by confirming the position where the distance information to the object cannot be obtained in the two-dimensional image G, it is possible to confirm the cause of the inability to obtain the distance information to the object.

The determination units 160, 560, and 660 determine that the distance information to the object cannot be acquired when the output of the distance information acquisition unit 13 is not equal to or greater than the threshold value or the threshold value or less. It also includes cases where blurring is detected.

In the above, when the information processing device is the imaging device 1, the imaging device 1 includes the imaging unit 11, the distance information acquisition unit 13, the three-dimensional reconstruction processing unit 150, and the determination unit 160, as shown in FIG. .

When the information processing device is the display device 500, as shown in FIGS. , the distance information acquisition unit 13, and transmits these outputs to the display device 500 or the server 600. FIG.

The display device 500 may or may not include the determination section 560 as shown in FIG.

If the display device 500 does not include the determination unit 560, the imaging device 1 may include the determination unit 160 and transmit the determination result to the display device 500, and the server 600 may include the determination unit 660 as shown in FIG. You may transmit a judgment result to the display apparatus 500 by pressing.

Similarly, the display device 500 may or may not include a three-dimensional reconstruction processing unit 550 as shown in FIG.

When the display device 500 does not include the three-dimensional reconstruction processing unit 550, the imaging device 1 may include the three-dimensional reconstruction processing unit 150 and transmit a three-dimensional image to the display device 500, as shown in FIG. Alternatively, the server 600 may include the 3D reconstruction processing unit 650 and transmit the 3D image to the display device 500 .

As described above, the image capturing apparatus 1 according to the embodiment of the present invention includes the image capturing unit 11 that captures an image of an object, the projection unit 12 that projects light onto the object, and the distance information that receives the light reflected from the object. an acquisition unit 13 (an example of a light receiving unit); a determination unit 160 that determines whether there is a distant object or a low-reflection object based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11; and a display control unit 170 that causes the display units 20 and 520 to display differently depending on the presence or absence of the low-reflection object.

This allows the photographer to accurately confirm that a distant object or a low-reflection object such as black is included in the captured image.

The imaging device 1 includes a display section 20 . This allows the photographer to reliably confirm that a distant object or a low-reflection object is included in the captured image.

The display control unit 170 causes the display units 20 and 520 to display differently according to the position of the distant object or the low-reflection object. This allows the photographer to confirm the position of a distant object or a low-reflection object.

The display unit 20 includes a plurality of display units 20A and 20a, and the display control unit 170 controls the display unit closer to a distant object or a low-reflection object among the plurality of display units 20A and 20a according to the presence or absence of an object. display differently. This allows the photographer to reliably confirm the position of the distant object or the low-reflectance object.

The display control unit 170 causes the display units 20 and 520 to display the image information G captured by the imaging unit 11, and displays a display image including identification information for identifying a distant object or a low-reflection object and the image information G. 20, 520 is caused to display. This allows the photographer to reliably confirm the position of the distant object or the low-reflectance object.

The determining unit 160 determines whether the object is a low-reflective object or a distant object based on the output of the imaging unit 11 when the amount of charge in the pixel due to the light received by the distance information acquiring unit 13 is equal to or less than a threshold. As a result, the photographer can accurately confirm which of the low-reflection object and the distant object is included in the captured image.

The determination unit 160 determines that there is a low-reflection object when the amount of charge in the pixel due to the light received by the distance information acquisition unit 13 is equal to or less than the threshold and the amount of charge in the pixel of the imaging unit 11 is equal to or less than the threshold. This allows the photographer to accurately confirm that the low-reflection object is included in the captured image.

The determination unit 160 determines whether the amount of charge in the pixel due to the light received by the distance information acquisition unit 13 is equal to or less than the threshold, the amount of charge in the pixel of the imaging unit 11 is equal to or greater than the threshold, and the distance determined based on the pixel is equal to or greater than the threshold. If there is, it is determined that there is a distant object.

This allows the photographer to accurately confirm that the distant object is included in the captured image.

The imaging device 1 acquires distance information to an object based on the light received by the distance information acquiring section 13 . In this case, the photographer can confirm that the reason why the desired distance information is not acquired is the distant object or the low-reflection object.

The imaging device 1 includes a transmission/reception section 180 that is an example of an output section that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition section 13 . In this case, the photographer can confirm that the reason why the desired three-dimensional information is not acquired is the distant object or the low-reflection object.

An imaging processing method according to an embodiment of the present invention includes an imaging step of imaging an object with an imaging unit 11, a projection step of projecting light onto the object with a projection unit 12, and a distance information acquiring unit that receives light reflected from the object. a light receiving step of receiving light by 13, and a determining step of determining whether there is a distant object or a low-reflecting object by determining units 160, 560, and 660 based on both the output of the distance information acquiring unit 13 and the output of the imaging unit 11; and a display step of causing the display units 20 and 520 to perform different displays by the display control units 170 and 530 according to the presence or absence of a distant object or a low-reflection object.

An imaging device 1 and a display device 500, which are examples of an information processing apparatus according to an embodiment of the present invention, receive an output of an imaging unit 11 that images an object, project light onto the object, and receive light reflected from the object. Based on the judgment results of the judgment units 160, 560, and 660 that judge whether there is a distant object or a low-reflection object based on both the output of the distance information acquisition unit 13 and the presence or absence of a distant object or a low-reflection object, Display control units 170 and 530 are provided to cause the display units 20 and 520 to display different images.

As described above, the image capturing apparatus 1 according to the embodiment of the present invention includes the image capturing unit 11 that captures an image of an object, the projection unit 12 that projects light onto the object, and the distance information that receives the light reflected from the object. an acquisition unit 13 (an example of a light receiving unit), a determination unit 160 that determines whether there is image blur based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and the presence or absence of image blur. and a display control unit 170 that causes the display units 20 and 520 to display differently accordingly.

This allows the photographer to accurately confirm that the image blur is included in the captured image.

The imaging device 1 includes a display section 20 . This allows the photographer to reliably confirm that the image blur is included in the captured image.

The display control unit 170 causes the positions of the display units 20 and 520 to display differently according to the position of the image blur. This allows the photographer to confirm the position of the image blur.

The display unit 20 includes a plurality of display units 20A and 20a, and the display control unit 170 causes the display unit closer to the position of the blurring of the image among the plurality of display units 20A and 20a to display an image depending on the presence or absence of the target object. display differently. As a result, the photographer can reliably confirm the location of the image blurring.

The display control unit 170 causes the display units 20 and 520 to display the image information G captured by the imaging unit 11, and displays a display image including the image information G and identification information for identifying blurring of the image on the display units 20 and 520. 520 to display. As a result, the photographer can reliably confirm the location of the image blurring.

The determination unit 160 detects an edge of the image based on the information of the image captured by the imaging unit 11, and determines that the image blurs when the light received by the distance information acquisition unit 13 causes pixel displacement.

This allows the photographer to accurately confirm that the image blur is included in the captured image.

The imaging device 1 acquires distance information to an object based on the light received by the distance information acquiring section 13 . In this case, the photographer can confirm that the reason why the desired distance information is not acquired is the blurring of the image.

The imaging device 1 includes a transmission/reception section 180 that is an example of an output section that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition section 13 . In this case, the photographer can confirm that the reason why the desired three-dimensional information is not acquired is the blurring of the image.

An imaging processing method according to an embodiment of the present invention includes an imaging step of imaging an object with an imaging unit 11, a projection step of projecting light onto the object with a projection unit 12, and a distance information acquiring unit that receives light reflected from the object. 13, a determination step of determining whether or not there is an image blur based on both the output of the distance information acquisition section 13 and the output of the imaging section 11, and and a display step of causing the display units 20 and 520 to display differently depending on the presence or absence of blurring.

An imaging device 1 and a display device 500, which are examples of an information processing apparatus according to an embodiment of the present invention, receive an output of an imaging unit 11 that images an object, project light onto the object, and receive light reflected from the object. Based on the judgment results of the judgment units 160, 560, and 660 that judge whether there is image blur based on both the output of the distance information acquisition unit 13 and the presence or absence of image blur, the display units 20 and 520 display It has display control units 170 and 530 that cause different displays.

An imaging device 1 and a display device 500, which are examples of an information processing device according to an embodiment of the present invention, have a distance information acquisition unit 13, which is an example of a light receiving unit that receives light projected onto and reflected from an object. The display control units 170 and 530 are provided to display the three-dimensional image 3G determined based on the output on the display units 20 and 520. The display control units 170 and 530 control the light reflected from the object in the three-dimensional image 3G. Distant objects that are far away from the distance information acquisition unit 13 when light is received, low-reflection objects that have a low reflectance with respect to projected light, and blind spots for the distance information acquisition unit 13 when light reflected from an object is received. position identification information 3Ga, 3Gb, and 3Gc for identifying at least one position of a distant object, a low-reflection object, and a blind spot based on position information indicating at least one position determined to be at least one, and a three-dimensional image 3G; The display image containing the image is displayed on the display unit 20 or 520 .

As a result, it is possible to confirm by viewing the three-dimensional image 3G which of the distant object, the low-reflecting object, and the blind spot is the reason why the desired three-dimensional image 3G is not displayed. It becomes possible to take measures such as imaging.

The three-dimensional image 3G is determined by three-dimensional reconstruction processing units 150, 550, and 650, which are examples of three-dimensional information determination units.

The display control units 170 and 530 display one of the position identification information 3Ga, 3Gb, and 3Gc and the three-dimensional image 3G based on the position information of any one of the distant object, the low-reflection object, and the blind spot. The image may be displayed on the display unit 20, 520, and any two or more of the position identification information 3Ga, 3Gb and 3Gc may be displayed based on the position information of any two or all of the distant object, the low reflective object, and the blind spot. A display image including all and the three-dimensional image 3G may be displayed on the display unit 20, 520. FIG.

When the information processing device is the imaging device 1, the imaging device 1 includes the distance information acquisition unit 13 and the three-dimensional reconstruction processing unit 150 as shown in FIG.

When the information processing device is the display device 500, as shown in FIGS. , the output of the distance information acquisition unit 13 is transmitted to the display device 500 or the server 600 .

The display device 500 may or may not include the three-dimensional reconstruction processing unit 550. When the display device 500 does not include the three-dimensional reconstruction processing unit 550, the imaging device 1 includes the three-dimensional reconstruction processing unit 150. The server 600 may include a three-dimensional reconstruction processing unit 650 and transmit the three-dimensional image to the display device 500 as shown in FIG. .

The display control unit 170, 530 indicates a position where the density of the point cloud data included in the 3D image 3G is lower than the threshold and is determined to be at least one of a distant object, a low reflective object, and a blind spot. A display image including position identification information 3Ga, 3Gb, and 3Gc and a three-dimensional image 3G is displayed on the display unit 20 or 520 based on the position information.

With this, it is possible to confirm, by viewing the three-dimensional image 3G, which of the distant object, the low-reflection object, and the blind spot is the cause of the density of the point cloud data being lower than the threshold.

The display control units 170 and 530 display a position that is determined to be at least one of a distant object, a low-reflection object, and a blind spot in the three-dimensional image 3G based on the output of the imaging unit 11 that images the object. Based on the information, the display unit 20, 520 is caused to display a display image including the position identification information 3Ga, 3Gb, 3Gc and the three-dimensional image 3G.

Accordingly, it is possible to accurately determine whether the cause of not displaying the desired three-dimensional image 3G is a distant object, a low-reflection object, or a blind spot, based on the output of the imaging unit 11 .

When the information processing device is the imaging device 1, the imaging device 1 includes an imaging unit 11 as shown in FIG. When the information processing device is the display device 500, the display device 500 does not include the imaging unit 11 as shown in FIGS. 11 to display device 500 or server 600 .

The imaging device 1 and the display device 500 are provided with determination units 160 and 560 that determine the position of at least one of a distant object, a low-reflection object, and a blind spot in the three-dimensional image 3G, and the display control units 170 and 530 , based on the determination results of the determination units 160 and 560, the display units 20 and 520 are caused to display a display image including the position identification information 3Ga, 3Gb, and 3Gc and the three-dimensional image 3G.

When the information processing device is the imaging device 1, the imaging device 1 is provided with a determination unit 160 as shown in FIG.

When the information processing device is the display device 500, the display device 500 may or may not include the determination section 560 as shown in FIG.

If the display device 500 does not include the determination unit 560, the imaging device 1 may include the determination unit 160 and transmit the determination result to the display device 500, and the server 600 may include the determination unit 660 as shown in FIG. You may transmit a judgment result to the display apparatus 500 by pressing.

The display control units 170 and 530 change the display area of the three-dimensional image 3G to be displayed on the display units 20 and 520 by changing the position and orientation of the virtual camera IC at the position of the viewpoint for viewing the three-dimensional image 3G. .

An imaging device 1 and a display device 500, which are examples of an information processing device according to an embodiment of the present invention, have a distance information acquisition unit 13, which is an example of a light receiving unit that receives light projected onto and reflected from an object. Display control units 170 and 530 for displaying a three-dimensional image 3G determined based on the output on the display units 20 and 520 are provided, and the display control units 170 and 530 obtain distance information when light reflected from an object is received. A display image including position identification information 3G1 and 3G2 for identifying the position of the distance information acquisition unit 13 when light reflected from the object is received based on position information indicating the position of the unit 13, and a three-dimensional image 3G. is displayed on the display unit 20 or 520 .

Thereby, the positional relationship between the imaging position indicating the position of the distance information acquisition unit 13 when the light reflected from the target is received and the specific target can be grasped in the three-dimensional image 3G. That is, it is possible to easily compare the positional relationship between the imaging position and the specific target at the site where the three-dimensional image was acquired and the positional relationship between the imaging position and the specific target in the three-dimensional image.

The three-dimensional image 3G and position information are determined by three-dimensional reconstruction processing units 150, 550, and 650, which are examples of three-dimensional information determination units.

When the information processing device is the imaging device 1 , the imaging device 1 includes the distance information acquisition unit 13 and the three-dimensional reconstruction processing unit 150 .

When the information processing device is the display device 500, the display device 500 does not include the distance information acquisition unit 13, and the imaging device 1 includes the distance information acquisition unit 13 and outputs the output of the distance information acquisition unit 13 to the display device. 500 or server 600.

The display device 500 may or may not include the three-dimensional reconstruction processing unit 550. When the display device 500 does not include the three-dimensional reconstruction processing unit 550, the imaging device 1 includes the three-dimensional reconstruction processing unit 150. The server 600 may include the three-dimensional reconstruction processing section 650 and transmit the three-dimensional image and the position information to the display device 500 .

The display control units 170 and 530 generate identification information 3Ga, 3Gb, and 3Gc, which are examples of low-density identification information for identifying regions, based on region information indicating regions in the three-dimensional image 3G where the density of the point cloud data is lower than the threshold value. , and the three-dimensional image 3G are displayed on the display units 20 and 520 .

In this case, since the positional relationship between the imaging position and the area where the density of the point cloud data is lower than the threshold can be grasped, it is possible to specify the factor that the density of the point cloud data is lower than the threshold. For example, if the area is far from the imaging position, a distant object is the factor, if the area is in the blind spot of the imaging position, the blind spot is the factor, and if neither the distant object nor the blind spot is the factor, the low reflective object is the factor. One thing can be identified.

The display control units 170 and 530 change the display area of the three-dimensional image 3G to be displayed on the display units 20 and 520 by changing the position and orientation of the virtual camera IC at the position of the viewpoint for viewing the three-dimensional image 3G. .

The display control units 170 and 530 change the orientation of the virtual camera IC to a predetermined orientation when the position of the virtual camera IC is at the positions 3G1 and 3G2 identified by the position identification information.

This makes it possible to direct the line of sight of the user positioned at the imaging position to a specific object that the user wishes to see on site.

The display control units 170 and 530 change the orientation of the virtual camera IC so that the display area includes predetermined coordinates and a low-density portion in which the density of point cloud data in the three-dimensional image 3G is lower than a threshold value. .

This makes it possible to direct the line of sight of the user positioned at the imaging position to predetermined coordinates or to a specific object that is a low-density portion in the three-dimensional image 3G.

The display control units 170 and 530 are determined based on the output of the distance information acquisition unit 13 positioned at the first position and the output of the distance information acquisition unit 13 positioned at a second position different from the first position. The dimensional image 3G is displayed on the display unit 20, 520, the first position identification information 3G1 for identifying the first position, the second position identification information 3G2 for identifying the second position, the three-dimensional image 3G, is displayed on the display unit 20, 520.

Thereby, the positional relationship between the first imaging position, the second imaging position, and the specific target can be grasped in the three-dimensional image 3G.

1 imaging device (an example of an information processing device)
3G three-dimensional image 3Ga, 3Gb, 3Gc identification information 3G1, 3G2 position identification information 10 housing 11 imaging unit 11a, 11A imaging device 11b, 11B fisheye lens 12 projection unit 12a, 12A light source unit 12b, 12B wide-angle lens 13 distance information acquisition unit (Example of light receiving part)
13a, 13A TOF sensor 13b, 13B wide-angle lens 14 processing circuit 15 shooting switch 20 display unit 20A, 20a display unit 111 other imaging unit 150, 550, 650 three-dimensional reconstruction processing unit (an example of a three-dimensional information determination unit)
160, 560, 660 judgment unit 170 display control unit (an example of an output unit)
180 Transmitter/receiver (an example of an output unit)
300 external device (example of output destination)
500 display device (output destination, example of information processing device)
520 Display (an example of output destination)
530 Display control unit (an example of an output unit)
600 Server L Synchronization signal line

Claims (6)

an imaging unit that images an object;
a projection unit that projects amplitude-modulated light onto the target;
a light receiving unit that receives the light reflected from the object in synchronization with the projection of the light by the projecting unit ;
An imaging device that acquires distance information to the object based on the output of the light receiving unit,
Whether or not there is a highly reflective object is determined based on the area corresponding to the pixel in the image information captured by the imaging unit when the amount of charge in the pixel due to the light received by the light receiving unit is equal to or greater than a predetermined value . a judgment unit;
When it is determined that the highly reflective object is present , the display image including the image information and the identification information for identifying the highly reflective object is displayed on the display unit. A display control unit that displays a different display than when it is determined that there is no
imaging device. 2. The imaging apparatus according to claim 1, comprising said display unit. 3. The imaging apparatus according to claim 1, wherein the determination unit determines that the highly reflective object exists when the area matches reference information indicating the highly reflective object. 4. The imaging apparatus according to any one of claims 1 to 3, further comprising an output section that outputs three-dimensional information determined based on the distance information acquired from the light receiving section. an imaging step of imaging an object with an imaging unit;
a projecting step of projecting amplitude-modulated light onto the object;
a light receiving step of receiving the light reflected from the object by a light receiving unit in synchronization with the projection of the light by the projecting step;
a distance information acquisition step of acquiring distance information to the object based on the output of the light receiving unit;
Whether or not there is a highly reflective object is determined based on the area corresponding to the pixel in the image information captured by the imaging unit when the amount of charge in the pixel due to the light received by the light receiving unit is equal to or greater than a predetermined value . a decision step;
When it is determined that there is no highly reflective object by displaying a display image including the image information and identification information for identifying the highly reflective object when it is determined that the highly reflective object is present. and a display step of displaying a different image from the image pickup processing method. Light reception for projecting amplitude-modulated light onto an object to determine if there is a highly reflective object, and receiving the light reflected from the object in synchronization with the projection of the light in order to acquire distance information to the object when the amount of charge in a pixel due to the light received by the unit is equal to or greater than a predetermined value , based on the determination result based on the region corresponding to the pixel in the image information captured by the imaging unit that captures the image of the object ; When it is determined that the highly reflective object is present , the display image including the image information and the identification information for identifying the highly reflective object is displayed on the display unit so that the highly reflective object is not detected on the display unit. An information processing apparatus having a display control unit that displays a different display from when it is determined that there is no information.

Images