JP6966011B1 - Imaging device, imaging method and information processing device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To easily identify blurring of an image included in a display image. An image pickup device (1) includes an image pickup unit (11) for imaging an object, a projection unit (12) for projecting light on the object, and a distance information acquisition unit (13) (an example of a light receiving unit) for receiving light reflected from the object. The determination unit 160 for determining whether or not there is image blur based on both the output of the distance information acquisition unit 13 and the output of the image pickup unit 11 and the display units 20 and 520 differ depending on the presence or absence of image blur. A display control unit 170 for displaying is provided. [Selection diagram] FIG. 17

Description

The present invention relates to an imaging device, an imaging method, and an information processing device.

Patent Document 1 describes a distance measuring device capable of measuring the distance to an object in a stable and accurate manner.

Patent Document 2 describes an image pickup apparatus that performs image processing to reduce the influence of the reflection when the reflection of a finger or the like occurs.

In Patent Document 3, a block whose measurement data density is lower than a predetermined threshold value is extracted, and the coordinates within the range of the extracted block are presented as a presentation measurement position which is a position where a three-dimensional measurement device should be installed. A three-dimensional synthesis processing system having a measurement position presenting unit and a measurement position presenting unit is described.

JP-A-2018-077071 Japanese Patent No. 5423287 Japanese Patent No. 6192938

An object of the present invention is to provide an image pickup apparatus, an imaging method, and an information processing apparatus capable of easily identifying blurring of an image included in a display image.

The image pickup apparatus according to the present invention includes an image pickup unit that images an object, a projection unit that projects light onto the target, a light receiving unit that receives light reflected from the target, an output of the light receiving unit, and an output of the image pickup unit. It is provided with a determination unit for determining whether or not there is image blurring based on both of the above, and a display control unit for causing the display unit to display differently depending on the presence or absence of image blurring.

According to the present invention, it is possible to provide an imaging device, an imaging method, and an information processing device that can easily identify blurring of an image included in a display image.

It is a figure which shows an example of the appearance of the image pickup apparatus which concerns on embodiment of this invention. It is a figure for demonstrating the structure of the image pickup apparatus in the same embodiment. It is a figure for demonstrating the usage situation of the image pickup apparatus in the same embodiment. It is a figure which shows an example of the structure of the processing block of the processing circuit in the same embodiment. It is a flow chart which shows an example of the operation of the processing circuit of the image pickup apparatus in the same embodiment. It is a omnidirectional image data generation flow diagram in the same embodiment. It is a proximity object judgment flow chart in the same embodiment. It is a figure for demonstrating the display content of the display part in the same embodiment. It is a figure which shows the appearance of the image pickup apparatus which concerns on the modification of this embodiment. It is a figure which shows the structure of the processing block of the processing circuit in the modification. It is a figure which shows the appearance of the image pickup apparatus which concerns on the 2nd modification of embodiment of this invention. It is a figure which shows the structure of the processing block of the processing circuit in the 2nd modification. It is a close object judgment flow figure in the 2nd modification. It is a figure for demonstrating the structure of the image pickup apparatus which concerns on the 3rd modification of embodiment of this invention. It is a judgment flow figure of a highly reflective object in embodiment of this invention. It is a judgment flow chart of a distant object and a low reflection object in the same embodiment. It is a judgment flow figure of image blur in the same embodiment. It is a judgment flow diagram in the 4th modification of embodiment of this invention. It is a figure which shows an example of the structure of the processing block of the processing circuit in the 5th modification of the embodiment of this invention. It is a figure which shows an example of the structure of the information processing system in the 6th modification of the embodiment of this invention. It is a figure which shows an example of the structure of the information processing system in the 7th modification of the embodiment of this invention. It is a figure for demonstrating the display content of the display part in 5th to 7th modification. It is a figure for demonstrating the 3D image displayed by the display part in embodiment of this invention. It is a judgment flow chart in the 5th to 7th modification. It is another figure for demonstrating the display content of the display part in 5th to 7th modification. It is a flow figure explaining the process in 5th to 7th modification. It is another flow diagram explaining the process in 5th to 7th modification.

Hereinafter, embodiments of the imaging apparatus and the imaging processing method will be described in detail with reference to the accompanying drawings.

FIG. 1 is a diagram showing an example of the appearance of the image pickup apparatus according to the embodiment of the present invention. FIG. 2 is a diagram for explaining the configuration of the image pickup apparatus. FIG. 2 shows the internal configuration of the image pickup apparatus of FIG.

The image pickup device 1 is an example of an information processing device that outputs three-dimensional information determined based on the received light, and is an image pickup unit (camera) 11 and a projection unit (distance sensor) that projects light other than visible light. A portion (12) corresponding to a light emitting unit and a distance information acquisition unit (a portion corresponding to a light receiving portion of a distance sensor) 13 that acquires distance information based on the light projected by the projection unit 12 are integrated with the housing 10. It is provided in. Each part is electrically connected to the processing circuit 14 inside the housing 10 by a synchronization signal line L, and operates in synchronization with each other.

The shooting switch 15 is for the user to input a shooting instruction signal to the processing circuit 14. The display unit 20 displays the contents corresponding to the output signal of the processing circuit 14, and is composed of a liquid crystal screen or the like. The display unit 20 may be configured by a touch panel or the like and may accept a user's operation input. The processing circuit 14 controls each part based on the shooting instruction to acquire RGB image and distance information data, and reconverts the acquired distance information data into high-density three-dimensional point cloud data based on the RGB image and distance information data. Perform the process of building.

It is possible to construct 3D point cloud data by using the distance information data as it is, but in that case, the accuracy of the 3D point cloud data is limited to the number of pixels (resolution) of the distance information acquisition unit 13. .. In this example, the processing when reconstructing it into high-density three-dimensional point cloud data is also shown. The reconstructed data is output to an external PC or the like via a portable recording medium or communication, and is used for displaying a three-dimensional restoration model.

Electric power is supplied to each part and the processing circuit 14 from a battery housed inside the housing 10. In addition to this, a configuration in which power is supplied from the outside of the housing 10 by a connection cord may be used.

The image pickup unit 11 captures two-dimensional image information and includes image pickup elements 11a and 11A, fisheye lenses (wide-angle lenses) 11b and 11B, and the like. The projection unit 12 includes light source units 12a and 12A, wide-angle lenses 12b and 12B, and the like. The distance information acquisition unit 13 includes TOF (Time Of Fright) sensors 13a and 13A, wide-angle lenses 13b and 13B, and the like. Although not shown, each part may constitute an optical system such as a prism or a lens group.

For example, the image pickup unit 11 may be configured with an optical system for forming an image of the light collected by the fisheye lenses 11b and 11B on the image pickup devices 11a and 11A. Further, the projection unit 12 may be configured with an optical system that guides the light of the light source units 12a and 12A to the wide-angle lenses 12b and 12B. Further, the distance information acquisition unit 13 may be configured with an optical system for forming an image of the light collected by the wide-angle lenses 13b and 13B on the TOF sensors 13a and 13A. Each optical system may be appropriately determined according to the configuration and arrangement of the image pickup elements 11a and 11A, the light source units 12a and 12A, the TOF sensors 13a and 13A, and the like. Will be omitted for explanation.

The image pickup elements 11a and 11A, the light source units 12a and 12A, and the TOF sensors 13a and 13A are integrally housed inside the housing 10. The fisheye lens 11b, the wide-angle lens 12b, the wide-angle lens 13b, and the display unit 20 are each provided on the first surface on the front side of the housing 10. On the first surface, the inner range of each of the fisheye lens 11b, the wide-angle lens 12b, and the wide-angle lens 13b is open.

The fisheye lens 11B, the wide-angle lens 12B, the wide-angle lens 13B, and the photographing switch 15 are each provided on the second surface on the back side of the housing 10. On the second surface, the inner range of each of the fisheye lens 11B, the wide-angle lens 12B, and the wide-angle lens 13B is open.

The image sensors 11a and 11A are two-dimensional resolution image sensors (area sensors). The image pickup elements 11a and 11A have an image pickup area in which a large number of light receiving elements (photodiodes) of each pixel are arranged in a two-dimensional direction. R (Red), G (Green), and B (Blue) color filters such as a bayer array are provided in the imaging area to receive visible light, and the light that has passed through the color filters is stored in the photodiode. NS. Here, an image sensor having a large number of pixels is used so that a wide-angle (for example, a range of a hemispherical sphere with a circumference of 180 degrees with the imaging direction as the front as shown in FIG. 2) can be acquired with high resolution. ..

The image pickup elements 11a and 11A convert the light imaged in the image pickup area into an electric signal by the pixel circuit of each pixel and output a high-resolution RGB image. The fisheye lenses 11b and 11B collect light from a wide angle (for example, a hemispherical range of 180 degrees around the image pickup direction shown in FIG. 2), and image the light in the image pickup areas of the image pickup elements 11a and 11A.

The light source units 12a and 12A are semiconductor lasers, and emit laser light in a wavelength band other than the visible light region (here, infrared is used as an example) used for distance measurement. One semiconductor laser may be used for the light source units 12a and 12A, or a plurality of semiconductor lasers may be used in combination. Further, as the semiconductor laser, for example, a surface emitting type semiconductor laser such as VCSEL (Vertical Cavity Surface Emitting LASER) may be used.

Further, the light of the semiconductor laser is molded by an optical lens so as to be vertically long, and the vertically long light is scanned in a one-dimensional direction of the measurement range by an optical deflection element such as a MEMS (Micro Electro Mechanical Systems) mirror. It may be configured. In the present embodiment, the light sources 12a and 12A show a form in which the light of the semiconductor laser LA is spread over a wide-angle range through the wide-angle lenses 12b and 12B without using a light deflection element such as a MEMS mirror.

The wide-angle lenses 12b and 12B of the light source units 12a and 12A have a function of expanding the light emitted by the light source units 12a and 12A to a wide-angle range (for example, a hemispherical range of 180 degrees around the imaging direction shown in FIG. 2). Has.

The wide-angle lenses 13b and 13B of the distance information acquisition unit 13 measure the reflected light of the light sources 12a and 12A projected by the projection unit 12 with respect to the wide-angle lens 13b and 13B whose front is the imaging direction shown in FIG. It is taken in from each direction (such as a 180-degree hemispherical range), and the light is imaged in the light receiving area of the TOF sensors 13a and 13A. The measurement range includes one or more projectiles (for example, a building), and the light reflected by the projectile (reflected light) is incident on the wide-angle lenses 13b and 13B. The reflected light may be captured, for example, by providing a filter that cuts light having a wavelength in the infrared region or higher on the entire surface of the wide-angle lenses 13b and 13B. Not limited to this, since it is sufficient that the light in the infrared region is incident on the light receiving area, a means for passing light having a wavelength in the infrared region such as a filter may be provided in the optical path from the wide-angle lenses 13b and 13B to the light receiving area. good.

The TOF sensors 13a and 13A are two-dimensional resolution optical sensors. The TOF sensors 13a and 13A have a light receiving area in which a large number of light receiving elements (photodiodes) are arranged in a two-dimensional direction. In this sense, it can be said to be a "second image receiving / receiving means". The TOF sensors 13a and 13A receive the reflected light of each area of the measurement range (each area is also called a position) by the light receiving element corresponding to each area, and the distance to each area based on the light detected by each light receiving element. Is measured (calculated).

In this embodiment, the distance is measured by the phase difference detection method. In the phase difference detection method, the measurement range is irradiated with laser light amplitude-modulated at the basic frequency, the reflected light is received and the phase difference between the irradiation light and the reflected light is measured to obtain the time, and the speed of light is obtained at that time. To calculate the distance. The strength of this method is that a certain level of resolution can be expected.

The TOF sensors 13a and 13A are driven in synchronization with the irradiation of light by the projection unit 12, and each light receiving element (corresponding to a pixel) calculates the distance corresponding to each pixel from the phase difference with the reflected light and converts it into pixel information. It outputs distance information image data (later also referred to as "distance image" or "TOF image") associated with information indicating the distance to each area within the measurement range. The TOF sensors 13a and 13A may output phase information image data in which phase information is associated with pixel information, and may acquire distance information image data based on the phase information image data in post-processing.

The number of areas in which the measurement range can be divided is determined by the resolution of the light receiving area. Therefore, when a low-resolution image is used for miniaturization, the number of pixel information of the distance image data is reduced, so that the number of three-dimensional point clouds is also reduced.

As another form, the distance may be measured by the pulse method instead of the phase difference detection method. In that case, for example, the light source units 12a and 12A emit an ultrashort pulse irradiation pulse P1 having a rise time of several nanoseconds (ns) and a strong optical peak power, and the TOF sensors 13a and 13A synchronize with this. The time (t) required to receive the reflected pulse P2, which is the reflected light of the irradiation pulse P1 emitted by the light source units 12a and 12A, is measured.

When this method is adopted, for example, TOF sensors 13a and 13A on which a circuit for measuring time or the like is mounted on the output side of the light receiving element is used. In each circuit, the time required for the light source units 12a and 12A to receive the reflection pulse P2 after emitting the irradiation pulse P1 is converted into a distance for each light receiving element, and the distance to each area is obtained.

This method is suitable for widening the angle of the image pickup apparatus 1 because it can output strong light by using peak light. Further, when the light is shaken (scanned) by using a MEMS mirror or the like, strong light can be irradiated to a long distance while suppressing the spread, which leads to an increase in the measurement distance. In this case, the arrangement is such that the laser light emitted from the light source units 12a and 12A is scanned (deflected) toward the wide-angle lenses 12b and 12B by the MEMS mirror.

It is desirable that the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 match, for example, at 180 degrees or more, but they do not necessarily have to match. If necessary, the effective angle of view of the imaging unit 11 and the effective angle of view of the distance information acquisition unit 13 may be reduced. In the present embodiment, the image pickup unit 11 and the distance information acquisition unit 13 are effective, for example, within the range of 100 degrees to 180 degrees so that the image angle does not include the image pickup device 1 main body, the distance information acquisition unit 13, and the like. The number of pixels is reduced.

Further, the resolutions of the TOF sensors 13a and 13A may be set lower than the resolutions of the image pickup devices 11a and 11A in order to prioritize the miniaturization of the image pickup device 1. By making the TOF sensors 13a and 13A have a lower resolution than the image pickup elements 11a and 11A, it is possible to suppress an increase in the size of the light receiving area, which can lead to a miniaturization of the image pickup device 1. Therefore, the TOF sensors 13a and 13A have a low resolution, and the three-dimensional point cloud obtained by the TOF sensors 13a and 13A has a low density. It can be converted into a dimensional point cloud. The process of converting to a high-density three-dimensional point cloud in the processing circuit 14 will be described later.

In the present embodiment, as an example, the image sensor 11a, the light source unit 12a, and the TOF sensor 13a are provided so as to be aligned in a straight line in the longitudinal direction of the housing 10. Similarly, the image sensor 11A, the light source unit 12A, and the TOF sensor 13A are provided so as to be aligned in a straight line in the longitudinal direction of the housing 10. Hereinafter, an example of the image sensor 11a, the light source unit 12a, and the TOF sensor 13a will be described.

The image pickup area (imaging surface) of the image pickup device 11a and the light receiving area (light receiving surface) of the TOF sensor 13a may be arranged in a direction orthogonal to the longitudinal direction as shown in FIG. The optical path) may be arranged in the longitudinal direction by providing a prism or the like that converts the light path by 90 degrees to cause an incident. In addition to this, it may be arranged in any direction depending on the configuration. That is, the image sensor 11a, the light source unit 12a, and the TOF sensor 13a are arranged so as to cover the same measurement range. The imaging unit 11, the projection unit 12, and the distance information acquisition unit 13 are arranged from one side of the housing 10 toward the measurement range.

At this time, the image sensor 11a and the TOF sensor 13a may be arranged on the same baseline so as to be parallel stereo. By arranging them in parallel stereo, parallax data can be obtained by using the output of the TOF sensor 13a even if there is only one image sensor 11a. The light source unit 12a is configured so that the measurement range of the TOF sensor 13a can be irradiated with light.

(Processing circuit)
Subsequently, the processing of the processing circuit 14 will be described. The TOF image obtained only by the TOF sensors 13a and 13A has a low resolution as it is. Therefore, in this example, an example in which the resolution is increased by the processing circuit 14 and high-density three-dimensional point cloud data is reconstructed is shown. In addition, a part or all of the following processing as "information processing means" in the processing circuit 14 may be performed by an external device.

As described above, the 3D point cloud data reconstructed by the image pickup device 1 is output to an external device such as a PC via a portable recording medium or communication, and is used for displaying the 3D restoration model. ..

As a result, it is possible to provide the image pickup device 1 which is excellent in portability by speeding up, downsizing, and weight reduction as compared with the case where the image pickup device 1 itself displays the three-dimensional restoration model.

However, after moving away from the site where the 3D information is acquired and restoring the 3D information by an external device, the image is reflected in the captured image of the photographer himself or the tripod, and the 3D information of the desired layout is acquired. You may notice that there is no such thing, and in that case, it takes time and effort to revisit the site where the 3D information is acquired.

To solve this, it is conceivable to bring an external device such as a PC to the site, but doing so eliminates the merits of speeding up, downsizing, and weight reduction.

It is also conceivable to send the acquired 3D information to an external device via a communication line and receive the restored 3D information, but the merit of speeding up is lost and the 3D information has a large amount of information in the first place. Therefore, it is difficult to visually confirm the reflection on the captured image of the photographer himself or the tripod.

In particular, in the case of spherical three-dimensional information, it is extremely difficult to visually confirm the reflection on the captured image of the photographer himself or a tripod.

In view of the above problems, the present embodiment provides an image pickup device 1 that can easily confirm in real time that the image is reflected in a captured image of the photographer himself or a tripod, and that the three-dimensional information of a desired layout is not acquired. The purpose is to provide.

FIG. 3 is a diagram for explaining a usage situation of the image pickup apparatus in the same embodiment.

In the state shown in FIG. 3A, the photographer M and the selfie stick 1A supporting the image pickup device 1 are not included in the global celestial imaging range R, and the photographer M and the selfie stick 1A are all included. It does not appear in the captured image of the celestial sphere.

In the state shown in FIG. 3B, the photographer M is included in the spherical imaging range R, and the photographer M is reflected in the captured image of the spherical image.

In the state shown in FIG. 3C, the tripod 1B supporting the imaging device 1 is included in the spherical imaging range R, and the tripod 1B is reflected in the captured image of the spherical image.

In the state shown in FIG. 3D, the photographer M and the selfie stick 1A supporting the image pickup device 1 are not included in the global imaging range R, and the photographer M and the selfie stick 1A are all included. Although it does not appear in the captured image of the celestial sphere, it may be misjudged because the outside light (for example, the sun or lighting) is strong.

Further, in the states shown in FIGS. 3B and 3C, it is difficult to uniformly determine the presence or absence of reflection because the colors and types of the reflected objects and their appearance are various. rice field.

In response to the above situation, when determining the presence or absence of a specific object (proximity object) such as the photographer himself or a tripod based on the distance information image data output from the TOF sensors 13a and 13A, a truly specific object It was difficult to distinguish whether there was an image or the outside light was too strong.

That is, when the amount of electricity stored in the specific pixels of the TOF sensors 13a and 13A is saturated, it may be due to the presence of a specific object, or the intensity of the external light is too strong, or the TOF sensors 13a, 13A. It was difficult to distinguish from the output of.

In view of the above problems, the present embodiment can accurately confirm whether or not a specific object such as the photographer or a tripod is reflected in the captured image by distinguishing it from the influence of external light. The other purpose is to provide. Another object of the present embodiment is to be able to confirm that the image includes an object such as a high-reflecting object, a distant object, a low-reflecting object, and an image blur, not limited to a near object.

FIG. 4 is a diagram showing an example of the configuration of the processing block of the processing circuit 14. The processing circuit 14 shown in FIG. 4 includes a control unit 141, an RGB image data acquisition unit 142, a monochrome processing unit 143, a TOF image data acquisition unit 144, a high resolution unit 145, a matching processing unit 146, and the like. It is an example of a projection processing unit 147, a semantic segmentation unit 148, a parallax calculation unit 149, a three-dimensional reconstruction processing unit 150, a judgment unit 160, a display control unit 170 which is an example of an output unit, and an output unit. It has a transmission / reception unit 180. In FIG. 4, the solid line arrow indicates the signal flow, and the broken line arrow indicates the data flow.

When the control unit 141 receives an ON signal (shooting start signal) from the shooting switch 15, it outputs a synchronization signal to the image sensors 11a and 11A, the light source units 12a and 12A, and the TOF sensors 13a and 13A, and controls the entire processing circuit 14. do. The control unit 141 first outputs a signal instructing the light source units 12a and 12A to emit an ultrashort pulse, and outputs a signal instructing the TOF sensors 13a and 13A to generate TOF image data at the same timing. Further, the control unit 141 outputs a signal instructing the image pickup elements 11a and 11A to take an image. The image pickup by the image pickup devices 11a and 11A may be a period during which the light source units 12a and 12A are emitted, or may be the latest period before and after that period.

The RGB image data acquisition unit 142 acquires the RGB image data captured by the imaging elements 11a and 11A based on the imaging instruction by the control unit 141, and outputs the RGB image data of the entire celestial sphere. The monochrome processing unit 143 performs processing for aligning the data types for matching processing with the TOF image data obtained from the TOF sensors 13a and 13A. In this example, the monochrome processing unit 143 performs processing for converting RGB image data of the spherical image into a monochrome image of the spherical image.

The TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13a and 13A based on the TOF image data generation instruction by the control unit 141, and outputs the TOF image data of the entire celestial sphere.

The high-resolution unit 145 regards the TOF image data of the whole celestial sphere as a monochrome image and raises the resolution. Specifically, the high-resolution unit 145 replaces the distance value associated with each pixel of the TOF image data of the spherical image with the monochrome image value (grayscale value) of the spherical image. .. Further, the high resolution unit 145 raises the resolution of the monochrome image of the whole celestial sphere to the resolution of the RGB image data of the whole celestial sphere obtained from the image pickup devices 11a and 11A. The conversion to high resolution is performed, for example, by performing a normal up-conversion process. As another conversion method, for example, it is possible to acquire a plurality of frames of continuously generated TOF image data of all celestial spheres, add the distances of adjacent points using them, and perform super-resolution processing. good.

The matching processing unit 146 describes the feature amount of the textured portion of the monochrome image of the spherical image in which the TOF image data of the spherical image is increased in resolution and the monochrome image of the spherical image of the RGB image data of the spherical image. Is extracted, and matching processing is performed according to the extracted feature amount. For example, the matching processing unit 146 extracts edges from each monochrome image and performs matching processing between the extracted edge information. As another method, for example, the matching process may be performed by a method such as SIFT that quantifies the change in texture. Here, the matching process means searching for the corresponding pixel.

As a specific method of matching processing, for example, there is block matching. Block matching is performed around the pixel value that is cut out as a block of M × M (M is a positive integer) pixel size around the pixel to be referenced, and around the pixel that is the center of the search in the other image. This is a method in which the similarity of pixel values cut out as a block of M × M pixels is calculated, and the central pixel having the highest similarity is set as the corresponding pixel.

There are various methods for calculating the degree of similarity. For example, an equation showing a normalized autocorrelation coefficient NCC (Normalized Correlation Cooperative) may be used. The normalized autocorrelation coefficient CNCC indicates that the higher the numerical value, the higher the similarity, and it becomes 1 when the pixel values of the blocks are exactly the same.

Further, since the data of the distance of the textureless region can be obtained from the TOF image data of the spherical image, the matching process may be weighted according to the region. For example, in the calculation of the equation showing CNCC, the calculation may be performed by weighting the portion other than the edge (textureless region).

Further, instead of the formula indicating NCC, a selective normalization correlation (SCC) or the like may be used.

The reprojection processing unit 147 performs a process of reprojecting the TOF image data of the entire celestial sphere indicating the distance of each position in the measurement range onto the two-dimensional coordinates (screen coordinate system) of the imaging unit 11. Reprojection means finding the coordinates at which the three-dimensional points calculated by the TOF sensors 13a and 13A appear in the images of the image sensors 11a and 11A. The TOF image data of the whole celestial sphere indicates the position of a three-dimensional point in the coordinate system centered on the distance information acquisition unit 13 (mainly the wide-angle lenses 13b and 13B). Therefore, the three-dimensional points indicated by the TOF image data of the whole celestial sphere are reprojected on the coordinate system centered on the imaging unit 11 (mainly fisheye lenses 11b and 11B).

For example, the reprojection processing unit 147 moves the coordinates of the three-dimensional points of the TOF image data of the whole celestial sphere in parallel with the coordinates of the three-dimensional points centered on the imaging unit 11, and after the parallel movement, the TOF of the whole celestial sphere A process is performed to convert the coordinates of the three-dimensional points of the image data into the two-dimensional coordinate system (screen coordinate system) indicated by the RGB image data of the whole sky. As a result, the coordinates of the three-dimensional points of the TOF image data of the whole celestial sphere and the coordinates of the two-dimensional image information of the whole celestial sphere captured by the imaging unit 11 are associated with each other. The reprojection processing unit 147 associates the coordinates of the three-dimensional points of the TOF image data of the whole celestial sphere with the coordinates of the two-dimensional image information of the whole celestial sphere captured by the imaging unit 11.

The parallax calculation unit 149 calculates the parallax at each position from the deviation of the distance from the corresponding pixel obtained by the matching process. The parallax matching process uses the reprojection coordinates converted by the reprojection processing unit 147 to search for peripheral pixels at the position of the reprojection coordinates, thereby shortening the processing time and achieving more detailed and high resolution. It becomes possible to acquire distance information.

Further, the segmentation data obtained by the semantic segmentation process of the semantic segmentation unit 148 may be used for the parallax matching process. In that case, it becomes possible to acquire more detailed and high-resolution distance information.

In addition, parallax matching processing is performed only on the edges and only the parts with strong features, and the TOF image data of the whole celestial sphere is also used for the other parts. May be used to perform propagation processing.

The semantic segmentation unit 148 uses deep learning to assign a segmentation label indicating an object to the input image of the measurement range. As a result, each pixel of the TOF image data of the spherical image can be constrained to any of a plurality of distance regions divided for each distance, so that the reliability of the calculation is further improved.

The three-dimensional reconstruction processing unit 150 acquires the RGB image data of the whole celestial sphere from the RGB image data acquisition unit 142, and reconstructs the three-dimensional data of the whole celestial sphere based on the distance information output by the parallax calculation unit 149. , Outputs a high-density three-dimensional point cloud of the entire celestial sphere with color information added to each three-dimensional point. The three-dimensional reconstruction processing unit 150 is an example of a three-dimensional information determination unit that determines three-dimensional information.

The determination unit 160 acquires the RGB image data of the whole sky from the RGB image data acquisition unit 142, and the reprojection processing unit 147 converts the whole sky into the two-dimensional coordinate system indicated by the RGB image data of the whole sky. The TOF image data of the sphere is acquired, and based on these data, it is determined whether or not a specific object is reflected in the captured image, and the determination result is output to the display control unit 170.

The display control unit 170 acquires RGB image data of all celestial spheres from the RGB image data acquisition unit 142, and causes the display unit 20 to display two-dimensional image information based on the acquired RGB image data of all celestial spheres. Further, the display control unit 170 causes the display unit 20 to display a display image including the information indicating the determination result acquired from the determination unit 160 and the two-dimensional image information.

The display control unit 170 is an example of an output unit that outputs two-dimensional image information captured by the image pickup unit 11 separately from the three-dimensional information, and the display unit 20 is an example of an output destination that outputs two-dimensional image information. be.

The display control unit 170 may acquire three-dimensional data of the whole celestial sphere from the three-dimensional reconstruction processing unit 150 and display the three-dimensional information on the display unit 20. Specifically, the display control unit 170 may select a case where the two-dimensional image information is displayed on the display unit 20 and a case where the three-dimensional information is displayed on the display unit 20 according to a predetermined condition. As a result, the display control unit 170 can output two-dimensional image information in addition to the three-dimensional information.

The transmission / reception unit 180 communicates with an external device by wired or wireless technology, and the three-dimensional data of the entire celestial sphere output from the three-dimensional reconstruction processing unit 150 and all output from the RGB image data acquisition unit 142. The two-dimensional image information of the celestial sphere is transmitted (output) to the external device 300 that performs the three-dimensional restoration processing via the network 400.

In the present embodiment, the two-dimensional image information captured by the imaging unit 11 is "original two-dimensional image information" for creating "two-dimensional image data for display", or "for display". 2D image data ". For example, when creating "two-dimensional image data for display" from "original two-dimensional image information" inside the image pickup device 1, or transmitting "original two-dimensional image information" from the image pickup device 1 to an external device. Then, an external device may create "two-dimensional image data for display" from the "original two-dimensional image information".

The transmission / reception unit 180 is an example of an output unit that outputs three-dimensional information, and the external device 300 is an example of an output destination that outputs three-dimensional information.

The transmission / reception unit 180 may not transmit the two-dimensional image information of the whole celestial sphere, but may transmit only the three-dimensional data of the whole celestial sphere. Further, the transmission / reception unit 180 may be configured by an interface circuit with a portable storage medium such as an SD card or a personal computer.

(Operation of processing circuit)
FIG. 5 is a flow chart showing an example of the operation of the processing circuit 14 of the image pickup apparatus 1. When the shooting switch 15 is turned on by the user and a shooting instruction signal is input, the control unit 141 of the processing circuit 14 performs an operation of generating a high-density three-dimensional point cloud by the following method (imaging processing method and An example of information processing method).

First, the control unit 141 drives the light source units 12a and 12A, the TOF sensors 13a and 13A, and the image pickup elements 11a and 11A to photograph the measurement range (step S1). Driven by the control unit 141, the light source units 12a and 12A irradiate infrared light (an example of a projection step), and the TOF sensors 13a and 13A receive the reflected light (an example of a light receiving step). Further, the image pickup devices 11a and 11A take an image of the measurement range at the timing of starting the driving of the light source units 12a and 12A or the period immediately before the same (an example of the image pickup step).

Next, the RGB image data acquisition unit 142 acquires RGB image data in the measurement range from the image pickup devices 11a and 11A (step S2). Then, the display control unit 170 acquires RGB image data of all celestial spheres from the RGB image data acquisition unit 142, and displays (displays) two-dimensional image information based on the acquired RGB image data of all celestial spheres on the display unit 20. Example of step) (step S3).

The display control unit 170 causes the display unit 20 to display the two-dimensional image information of a part of the acquired RGB image data of the entire celestial sphere, and the two-dimensional display unit 20 is displayed by various inputs of the user. Change the area of image information. Various inputs of the user can be realized by providing an operation switch other than the shooting switch 15 and configuring the display unit 20 as an input unit such as a touch panel.

At this stage, the photographer has not acquired the image in the captured image of the photographer himself or the tripod, or the two-dimensional image information of the desired layout by the two-dimensional image information displayed on the display unit 20. Can be confirmed.

Next, the TOF image data acquisition unit 144 acquires TOF image data indicating the distance of each position in the two-dimensional region from the TOF sensors 13a and 13A (step S4).

Next, the monochrome processing unit 143 converts the RGB image data into a monochrome image (step S5). The TOF image data and the RGB image data have different data types for the distance data and the RGB data, respectively, and matching cannot be performed as they are. Therefore, first, each data is once converted into a monochrome image. The TOF image data is converted by the high resolution unit 145 by replacing the value indicating the distance of each pixel with the value of the monochrome image as it is before the high resolution.

Next, the high-resolution unit 145 increases the resolution of the TOF image data (step S6). Next, the matching processing unit 146 extracts the feature amount of the textured portion of each monochrome image, and performs the matching process with the extracted feature amount (step S7).

Next, the parallax calculation unit 149 calculates the parallax at each position from the deviation of the distance of the corresponding pixel to calculate the distance (step S8).

Next, the determination unit 160 acquires the RGB image data of the entire celestial sphere from the RGB image data acquisition unit 142, and the reprojection processing unit 147 converts the entire celestial sphere into the two-dimensional coordinate system indicated by the RGB image data. TOF image data is acquired, and based on these data, it is determined whether or not a nearby object as a specific target is reflected in the captured image, and the determination result is output to the display control unit 170 (an example of the determination step). ..

The display control unit 170 superimposes or includes information indicating the determination result acquired from the determination unit 160 on the two-dimensional image information and displays it on the display unit 20 (an example of a display step) (step S9). In step S9, the determination unit 160 determines whether or not there is a high-reflecting object, a distant object, a low-reflecting object, an image blur, etc. as a specific target, and outputs the determination result to the display control unit 170. ..

Then, the three-dimensional reconstruction processing unit 150 acquires the RGB image data from the RGB image data acquisition unit 142, reconstructs the three-dimensional data based on the distance information output by the parallax calculation unit 149, and converts the three-dimensional data into each three-dimensional point. A high-density three-dimensional point cloud to which color information is added is output (step S10).

Next, the transmission / reception unit 180 performs a three-dimensional restoration process of the three-dimensional data output from the three-dimensional reconstruction processing unit 150 and the two-dimensional image information output from the RGB image data acquisition unit 142 via the network 400. It is transmitted to 300 (an example of a three-dimensional information output step) (step S11).

The transmission / reception unit 180 may transmit the three-dimensional data output from the three-dimensional reconstruction processing unit 150 without transmitting the two-dimensional image information output from the RGB image data acquisition unit 142.

As described above, the image pickup apparatus 1 includes an image pickup unit 11 and a display control unit 170 that outputs two-dimensional image information captured by the image pickup unit 11 in addition to the three-dimensional information.

As a result, it is easy to confirm from the 2D image information that the photographer himself or the tripod is not reflected in the captured image or the 3D information of the desired layout is not acquired without confirming the 3D information. Will be possible.

Therefore, it is possible to reacquire the 3D information while being at the site where the 3D information is acquired, and after leaving the site where the 3D information is acquired, the image is taken on the photographer himself or the captured image of a tripod or the like. Compared to the case of crowding or noticing that the 3D information of the desired layout has not been acquired, the time and effort to visit the site for acquiring the 3D information again is reduced.

The three-dimensional information includes spherical three-dimensional information. In this case, even in the case of all-sky 3D information, where it is difficult to confirm that the image is reflected in the captured image of the photographer himself or a tripod, or that the 3D information of the desired layout has not been acquired. It is possible to easily confirm from the two-dimensional image information captured by the imaging unit 11 that the image is reflected in the captured image of itself, a tripod, or the like, and that the three-dimensional information of the desired layout is not acquired.

The display control unit 170 outputs the two-dimensional image information G in step S3 before the transmission / reception unit 180 transmits (outputs) the three-dimensional information in step S11. The display control unit 170 outputs the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

As a result, it is necessary to confirm from the 2D image information before confirming the 3D information that the photographer himself or the tripod is reflected in the captured image and the 3D information of the desired layout is not acquired. Becomes possible.

The display control unit 170 causes the display unit 20 to display two-dimensional image information. The image pickup apparatus 1 includes a display unit 20.

As a result, it is possible to easily confirm from the two-dimensional image information displayed on the display unit 20 that the photographer himself or the tripod is reflected in the captured image and that the three-dimensional information of the desired layout is not acquired. Becomes possible.

The display control unit 170 outputs the two-dimensional image information to the display unit 20, which is different from the external device 300 to which the transmission / reception unit 180 outputs the three-dimensional information.

As a result, it is possible to confirm that the image is reflected in the captured image of the photographer himself or the tripod, and that the 3D information of the desired layout has not been acquired, without checking the 3D information output to the external device 300. It is possible to confirm from the two-dimensional image information output to the display unit 20 different from the device 300.

The image pickup apparatus 1 includes a three-dimensional reconstruction processing unit 150 that determines three-dimensional information based on the output of the distance information acquisition unit 13. The three-dimensional reconstruction processing unit 150 determines the three-dimensional information based on the output of the distance information acquisition unit 13 and the two-dimensional image information.

As a result, the 3D information determined by the 3D reconstruction processing unit 150 is confirmed that the image is reflected in the captured image of the photographer himself or the tripod, and that the 3D information of the desired layout is not acquired. It is possible to confirm from the two-dimensional image information captured by the imaging unit 11 without any problem.

FIG. 6 is a spherical image data generation flow diagram in the same embodiment.

FIG. 6A is a flowchart showing a process of generating RGB image data of the spherical image corresponding to step S2 described with reference to FIG.

The RGB image data acquisition unit 142 inputs two RGB image data in the fisheye image format (step S201).

The RGB image data acquisition unit 142 converts each RGB image data into an equirectangular image format (step S202). The RGB image data acquisition unit 142 converts the two RGB image data into a regular distance cylindrical image format based on the same coordinate system, thereby facilitating image combination in the next step. The RGB image data can be converted into image data by using one or a plurality of image format formats other than the regular distance cylindrical image format, if necessary. For example, it can be converted into the coordinates of an image perspectively projected on an arbitrary surface or an image perspectively projected on each surface of an arbitrary polyhedron.

Here, the equirectangular image format will be described. The equirectangular image format is a method capable of expressing a spherical image, and is a format of an image (equirectangular image) created by using the equirectangular projection. The equirectangular projection is a projection that represents a three-dimensional direction with two variables, such as the latitude and longitude of a globe, and is displayed in a plane so that the latitude and longitude are orthogonal to each other. Therefore, the equirectangular image is an image generated by using the equirectangular projection, and is represented by coordinates with two angular variables in the spherical coordinate system as two axes.

The RGB image data acquisition unit 142 combines the two RGB image data generated in step S202 to generate one all-sky RGB image data (step S203). The two input RGB image data cover an area having a total angle of view of more than 180 degrees. Therefore, the spherical RGB image data generated by appropriately connecting the two RGB image data can cover the spherical region.

The combination process in step S203 can use an existing technique for connecting a plurality of images, and the method is not particularly limited.

FIG. 6B is a flowchart showing a processing of generating TOF image data of the spherical image corresponding to step S4 described with reference to FIG.

The TOF image data acquisition unit 144 acquires two distance image data in the fisheye image format (step S401).

The TOF image data acquisition unit 144 converts each of the two TOF image data in the fisheye image format into an equirectangular image format (step S402). As described above, the equirectangular image format is a method capable of expressing a spherical image. In this step S402, by converting the two TOF image data into an equirectangular image format based on the same coordinate system, the image combination in the next step S403 is facilitated.

The TOF image data acquisition unit 144 combines the two TOF image data generated in step S402 to generate one spherical TOF image data (step S403). The two input TOF image data cover an area having a total angle of view of more than 180 degrees. Therefore, the spherical TOF image data generated by appropriately connecting the two TOF image data can cover the spherical region.

The combination process in step S403 can use an existing technique for connecting a plurality of images, and the method is not particularly limited.

FIG. 7 is a proximity object determination flow diagram in the same embodiment.

FIG. 7 is a flowchart showing a process of determining whether or not a nearby object is reflected in the captured image corresponding to step S9 described with reference to FIG.

Based on the TOF image data of the whole celestial sphere acquired from the reprojection processing unit 147, the determination unit 160 saturates the stored amount of the TOF image data of the whole celestial sphere as an example of a pixel in which the stored amount is equal to or more than a predetermined value. It is determined whether or not there is a pixel (step S801).

When there is a pixel whose storage amount is saturated in step S801, the determination unit 160 is based on the RGB image data of the whole celestial sphere acquired from the RGB image data acquisition unit 142, and among the RGB image data of the whole celestial sphere, With respect to the pixel having the same coordinates as the pixel whose stored amount is saturated in step S801, it is determined whether the stored amount is saturated as an example in which the stored amount is equal to or higher than a predetermined value (step S802).

When the storage amount is saturated in step S802, the determination unit 160 determines that the pixel whose storage amount is saturated in step S801 is due to external light (for example, the sun or lighting), and outputs error information. Output to the display control unit 170. The display control unit 170 causes the display unit 20 to display a display image including the error information and the two-dimensional image information based on the error information acquired from the determination unit 160 (step S803).

When the storage amount is not saturated in step S802, the determination unit 160 determines that the pixel whose storage amount is saturated in step S801 is due to the presence of a nearby object, and the storage amount is saturated in step S801. The coordinate position information of the pixel is output to the display control unit 170. The display control unit 170 causes the display unit 20 to display a display image including identification information for identifying a nearby object and two-dimensional image information based on the coordinate position information of the pixels acquired from the determination unit 160 (step S804). ..

When there is no pixel whose storage amount is saturated in step S801, the determination unit 160 determines 0 of the TOF image data of the whole celestial sphere based on the TOF image data of the whole celestial sphere acquired from the reprojection processing unit 147. It is determined whether there is a pixel showing distance information of .5 m or less (step S805).

If there is no pixel indicating the distance information of 0.5 m or less in step S805, the determination unit 160 ends the process.

If there is a pixel showing distance information of 0.5 m or less in step S805, the determination unit 160 proceeds to step S804 described above, and the pixel showing distance information of 0.5 m or less in step S805 depends on the presence of a nearby object. It is determined that this is the case, and the coordinate position information of the pixel indicating the distance information of 0.5 m or less in step S805 is output to the display control unit 170. The display control unit 170 causes the display unit 20 to display a display image including identification information for identifying a nearby object and two-dimensional image information based on the coordinate position information of the pixels acquired from the determination unit 160.

As described above, the display control unit 170 superimposes or includes the identification information on the two-dimensional image information when it determines that a nearby object exists, and two-dimensionally displays the identification information when it does not determine that a nearby object exists. Do not superimpose or include on image information.

That is, the display control unit 170 causes the display unit 20 to display differently depending on the presence or absence of a nearby object.

Further, the display control unit 170 causes the display unit 20 to display a display image including identification information for identifying a nearby object and two-dimensional image information based on the coordinate position information of the pixels acquired from the determination unit 160.

That is, the display control unit 170 causes the position of the display unit 20 to display differently according to the position of the adjacent object.

FIG. 8 is a diagram for explaining the display contents of the display unit in the same embodiment.

FIG. 8 is an explanatory diagram corresponding to step S2 shown in FIG. 5, step S803 and step S804 shown in FIG. 7.

Two-dimensional image information G is displayed on the display unit 20 by the display control unit 170. Further, the display unit 20 includes display including identification information G1 and G2 (for example, a finger and a tripod), error information G3, and two-dimensional image information G for identifying an object such as a nearby object by the display control unit 170. The image is displayed. The error information G3 can be indicated by using a mark such as "sun, lighting" as shown in FIG.

As described above, the imaging device 1 includes an imaging unit 11 that images an object, a projection unit 12 that projects light onto the object, a distance information acquisition unit 13 that receives light reflected from the object, and distance information acquisition. A display control unit 170 that causes the display unit 20 to display differently depending on the output of the unit 13 and the output of the imaging unit 11 and the presence or absence of an object such as a nearby object determined based on the output is provided.

As a result, the photographer can accurately confirm whether or not the photographer himself or a nearby object such as a tripod is reflected in the captured image, distinguishing from the influence of external light.

The image pickup apparatus 1 includes a display unit 20. As a result, the photographer can surely confirm whether or not a close object is reflected in the captured image.

The display control unit 170 causes the position of the display unit 20 to display differently according to the position of a nearby object. This allows the photographer to confirm the position of the reflection of a nearby object in the captured image.

The display control unit 170 displays the image information G captured by the imaging unit 11 on the display unit 20, and displays a display image including the identification information G1 and G2 for identifying nearby objects and the image information on the display unit 20. Let me. As a result, the photographer can surely confirm the position of the image of the close object in the captured image.

The image pickup device 1 has a saturated amount of electricity stored as an example of pixels in which the amount of electricity stored by the light received by the distance information acquisition unit 13 is equal to or greater than a predetermined value, and the amount of electricity stored in the pixels of the image pickup unit 11 is equal to or less than a predetermined value. As an example, when the amount of stored electricity is not saturated, a determination unit 160 for determining that there is a nearby object is provided.

As a result, the photographer can accurately confirm whether or not a nearby object is reflected in the captured image, distinguishing it from the influence of external light.

FIG. 9 is a diagram showing the appearance of the image pickup apparatus according to the modified example of the same embodiment. FIG. 10 is a diagram showing a configuration of a processing block of a processing circuit in a modified example.

In this modification, the display control unit 170 acquires the RGB image data of the whole celestial sphere from the RGB image data acquisition unit 142, and displays the two-dimensional image information based on the acquired RGB image data of the celestial sphere on the display device 500. Displayed in unit 520. The display unit 520 is an example of an output destination that outputs two-dimensional image information.

As a result, it is possible to easily confirm from the two-dimensional image information displayed on the display unit 520 that the photographer himself or the tripod is reflected in the captured image and that the three-dimensional information of the desired layout is not acquired. Becomes possible.

The display control unit 170 outputs the two-dimensional image information to the display unit 520, which is different from the external device 300 from which the transmission / reception unit 180 outputs the three-dimensional information.

As a result, it is possible to confirm that the image is reflected in the captured image of the photographer himself or the tripod, and that the 3D information of the desired layout has not been acquired, without checking the 3D information output to the external device 300. It is possible to confirm from the two-dimensional image information output to the display unit 520 different from the device 300.

The display control unit 170 may acquire three-dimensional data of the whole celestial sphere from the three-dimensional reconstruction processing unit 150 and display the three-dimensional information on the display unit 520. Specifically, the display control unit 170 may select a case where the two-dimensional image information is displayed on the display unit 520 and a case where the three-dimensional information is displayed on the display unit 520 according to a predetermined condition. As a result, the display control unit 170 can output two-dimensional image information in addition to the three-dimensional information.

The display control unit 170 causes the display unit 520 to display a display image including the error information and the two-dimensional image information based on the error information acquired from the determination unit 160.

The display control unit 170 causes the display unit 520 to display a display image including identification information for identifying a nearby object and two-dimensional image information based on the coordinate position information of the pixels acquired from the determination unit 160.

That is, the display control unit 170 causes the display unit 520 to display differently depending on the presence or absence of a nearby object determined based on the output of the distance information acquisition unit 13 and the output of the image pickup unit 11.

As a result, the photographer can accurately confirm whether or not the photographer himself or a nearby object such as a tripod is reflected in the captured image, distinguishing from the influence of external light.

The display control unit 170 causes the position of the display unit 520 to display differently according to the position of a nearby object. This allows the photographer to confirm the position of the reflection of a nearby object in the captured image.

The display control unit 170 causes the display unit 520 to display the image information captured by the image pickup unit 11, and causes the display unit 520 to display a display image including identification information for identifying a nearby object and image information. As a result, the photographer can surely confirm the position of the image of the close object in the captured image.

FIG. 11 is a diagram showing the appearance of the image pickup apparatus according to the second modification of the embodiment of the present invention. FIG. 12 is a diagram showing a configuration of a processing block of the processing circuit in the second modification.

In the second modification shown in FIG. 11, the image pickup apparatus 1 includes a plurality of display units 20A and 20a in place of the display unit 20 shown in FIG. The display units 20A and 20a are composed of LEDs and the like, and blink or light according to the output signal of the processing circuit 14.

The display unit 20a is provided on the first surface on the front side of the housing 10, and the display unit 20A is provided on the second surface on the back side of the housing 10.

In the second modification shown in FIG. 12, the display control unit 170 causes the display units 20A and 20a to display information indicating the determination result acquired from the determination unit 160. For example, the display units 20a and 20b may blink red when there is an object close to each side of the image pickup apparatus 1.

Further, the transmission / reception unit 180 transmits (outputs) the two-dimensional image information of the entire celestial sphere output from the RGB image data acquisition unit 142 to the display device 500 via the network 400. The display device 500 is an example of an output destination that outputs two-dimensional image information.

That is, in the second modification, in step S3 shown in FIG. 5, the transmission / reception unit 180 acquires the RGB image data of the whole celestial sphere from the RGB image data acquisition unit 142, and the acquired RGB image data of the whole celestial sphere. The two-dimensional image information based on the above is transmitted (output) to the display device 500.

The transmission / reception unit 510 of the display device 500 receives the two-dimensional image information transmitted from the transmission / reception unit 180 of the image pickup device 1.

The display control unit 530 of the display device 500 causes the display unit 520 to display the two-dimensional image information received by the transmission / reception unit 510. The display device 500 including the display control unit 530 is an example of an information processing device.

As described above, the image pickup apparatus 1 includes an image pickup unit 11 and a transmission / reception unit 180 that outputs two-dimensional image information captured by the image pickup unit 11 in addition to the three-dimensional information.

As a result, it is easy to confirm from the 2D image information that the photographer himself or the tripod is not reflected in the captured image or the 3D information of the desired layout is not acquired without confirming the 3D information. Will be possible.

Therefore, it is possible to reacquire the 3D information while being at the site where the 3D information is acquired, and after leaving the site where the 3D information is acquired, the image is taken on the photographer himself or the captured image of a tripod or the like. Compared to the case of crowding or noticing that the 3D information of the desired layout has not been acquired, the time and effort to visit the site for acquiring the 3D information again is reduced.

The transmission / reception unit 180 transmits (outputs) the two-dimensional image information G in step S3 before transmitting (outputting) the three-dimensional information in step S11. The transmission / reception unit 180 transmits (outputs) the two-dimensional image information G in step S3 before the three-dimensional reconstruction processing unit 150 determines the three-dimensional information in step S10.

As a result, it is necessary to confirm from the 2D image information before confirming the 3D information that the photographer himself or the tripod is reflected in the captured image and the 3D information of the desired layout is not acquired. Becomes possible.

The transmission / reception unit 180 transmits the two-dimensional image information to the display device 500, and the display device 500 causes the display unit 520 to display the two-dimensional image information.

As a result, it is possible to easily confirm from the two-dimensional image information displayed on the display unit 520 that the photographer himself or the tripod is reflected in the captured image and that the three-dimensional information of the desired layout is not acquired. Becomes possible.

The transmission / reception unit 180 transmits the two-dimensional image information to the display device 500 different from the external device 300 that outputs the three-dimensional information.

As a result, it is not necessary to confirm the 3D information output to the external device 300 that the photographer himself or the tripod is reflected in the captured image or the 3D information of the desired layout is not acquired. It is possible to confirm from the two-dimensional image information output to the display unit 520 of the display device 500, which is different from the device 300.

The transmission / reception unit 180 may transmit the three-dimensional information to the display device 500. Specifically, the transmission / reception unit 180 may select a case where the two-dimensional image information is transmitted to the display device 500 and a case where the three-dimensional information is transmitted to the display device 500 according to a predetermined condition. As a result, the transmission / reception unit 180 can transmit the two-dimensional image information to the display device 500 in addition to the three-dimensional information.

FIG. 13 is a proximity object determination flow diagram in the second modification.

FIG. 13 is a flowchart showing a process of determining whether or not a nearby object is reflected in the captured image corresponding to step S9 described in FIG. 5 in the second modification.

Based on the TOF image data of the whole celestial sphere acquired from the reprojection processing unit 147, the determination unit 160 saturates the stored amount of the TOF image data of the whole celestial sphere as an example of a pixel in which the stored amount is equal to or more than a predetermined value. It is determined whether or not there is a pixel (step S811).

When there is a pixel whose storage amount is saturated in step S811, the determination unit 160 is based on the RGB image data of the whole celestial sphere acquired from the RGB image data acquisition unit 142, and among the RGB image data of the whole celestial sphere, With respect to the pixels having the same coordinates as the pixels in which the stored amount is saturated in step S811, it is determined whether the stored amount is saturated as an example in which the stored amount is equal to or higher than a predetermined value (step S812).

When the storage amount is saturated in step S812, the determination unit 160 determines that the pixel whose storage amount is saturated in step S811 is due to external light, and outputs error information to the display control unit 170. .. The display control unit 170 causes the display units 20A and 20a to display the error information based on the error information acquired from the determination unit 160 (step S813).

When the storage amount is not saturated in step S812, the determination unit 160 determines that the pixel whose storage amount is saturated in step S811 is due to the presence of a nearby object, and the storage amount is saturated in step S811. The coordinate position information of the pixel is output to the display control unit 170. The display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 based on the coordinate position information of the pixels acquired from the determination unit 160 (step S814).

When there is no pixel whose storage amount is saturated in step S811, the determination unit 160 has 0 of the TOF image data of all celestial spheres based on the TOF image data of all celestial spheres acquired from the reprojection processing unit 147. It is determined whether there is a pixel showing distance information of .5 m or less (step S815).

If there is no pixel indicating the distance information of 0.5 m or less in step S815, the determination unit 160 ends the process.

If there is a pixel showing distance information of 0.5 m or less in step S815, the determination unit 160 proceeds to step S814 described above, and the pixel showing distance information of 0.5 m or less in step S815 depends on the presence of a nearby object. It is determined that this is the case, and the coordinate position information of the pixel indicating the distance information of 0.5 m or less in step S815 is output to the display control unit 170. The display control unit 170 determines whether the coordinate position information is on the front side of the housing 10 based on the coordinate position information of the pixels acquired from the determination unit 160.

When the display control unit 170 determines in step S814 that it is on the front side, the display control unit 170 blinks the display unit 20a arranged on the front side of the housing 10 (step S816).

If the display control unit 170 does not determine in step S814 that it is on the front side, the display control unit 170 blinks the display unit 20A arranged on the back side of the housing 10 (step S817).

As described above, the display control unit 170 blinks the display unit 20a or the display unit 20A when it is determined that a nearby object exists, and the display unit 20a and the display unit 20A when it does not determine that a nearby object exists. Do not blink.

That is, the display control unit 170 causes the display unit 20a and the display unit 20A to display differently depending on the presence or absence of a nearby object.

As a result, the photographer can accurately confirm whether or not the photographer himself or a nearby object such as a tripod is reflected in the captured image, distinguishing from the influence of external light.

Further, the display control unit 170 blinks the display unit 20a or the display unit 20A based on the coordinate position information of the pixels acquired from the determination unit 160.

That is, the display control unit 170 causes the position of the display unit, that is, either the display unit 20a or the display unit 20A, to display differently according to the position of the adjacent object. This allows the photographer to confirm the position of the reflection of a nearby object in the captured image.

Then, the display control unit 170 causes the display unit of the display units 20A and 20a on the side closer to the nearby object to display differently depending on the presence or absence of the nearby object. As a result, the photographer can surely confirm the position of the reflection of a specific object in the captured image.

FIG. 14 is a diagram for explaining a configuration of an image pickup apparatus according to a third modification of the embodiment of the present invention.

In the third modification shown in FIG. 14, in addition to the configuration shown in FIG. 2, the image pickup device 1 has other image pickup elements 111a, 111A, other fisheye lenses (wide-angle lenses) 111b, 111B, and the like. The image pickup unit 111 is provided.

In the third modification, the RGB imaging unit 11 and the other imaging unit 111 are provided on the same baseline. In this case, the processing circuit 14 enables multi-eye processing. That is, RGB images of two viewpoints can be obtained by simultaneously driving the imaging unit 11 and the other imaging unit 111 provided at a predetermined distance on one surface. Therefore, the parallax calculated based on the two RGB images can be used, and the distance accuracy of the entire measurement range can be further improved.

Specifically, when the RGB imaging unit 11 and another imaging unit 111 are provided, multi-baseline stereo (MSB) using SSD, EPI processing, and the like can be used as in the conventional parallax calculation. Therefore, by using this, the reliability of parallax is increased, and it becomes possible to realize high spatial resolution and accuracy.

As described above, the imaging device 1 includes another imaging unit 111, and the three-dimensional reconstruction processing unit 150 is imaged by the output of the distance information acquisition unit 13, the two-dimensional image information, and the other imaging unit 111. The 3D information is determined based on the other 2D image information.

The image pickup device 1 does not depend on the output of the distance information acquisition unit 13, but has three-dimensional information based on the other image pickup unit 111, the two-dimensional image information, and the other two-dimensional image information captured by the other image pickup unit 111. It may be provided with a three-dimensional information determination unit for determining the above.

As a result, the 3D reconstruction processing unit 150 determines that the image is reflected in the captured image of the photographer himself or the tripod, and that the 3D information of the desired layout has not been acquired based on the 2D image information. It is possible to confirm from the two-dimensional image information captured by the imaging unit 11 without confirming the three-dimensional information.

FIG. 15 is a flow chart for determining a highly reflective object according to the embodiment of the present invention, and is a flowchart showing a process for determining whether or not a highly reflective object is reflected in a captured image corresponding to step S9 described with reference to FIG. ..

Based on the TOF image data of the whole celestial sphere acquired from the reprojection processing unit 147, the determination unit 160 saturates the stored amount of the TOF image data of the whole celestial sphere as an example of a pixel in which the stored amount is equal to or more than a predetermined value. It is determined whether or not there is a pixel (step S21).

When there is a pixel whose storage amount is saturated in step S21, the determination unit 160 is based on the RGB image data of the whole celestial sphere acquired from the RGB image data acquisition unit 142, and among the RGB image data of the whole celestial sphere, It is determined in step S21 whether the RGB image data including the pixels having the same coordinates as the pixels whose stored amount is saturated matches the reference information indicating the highly reflective object (step S22). A model image may be used as reference information indicating a highly reflective object, and the degree of coincidence between the RGB image data and the model image may be determined by image recognition. Further, the degree of coincidence may be determined by a predetermined threshold value by using parameters such as spectrum and tint as reference information indicating a highly reflective object and RGB image data. Further, the reference information may be stored in a table, or a learning model may be used.

The processing circuit 14 stores an image of a highly reflective object such as metal or a mirror as model image information, and the determination unit 160 stores the acquired image using a determination device such as AI in step S22. It is determined whether the image matches the image of the highly reflective object.

When the determination unit 160 determines that the image acquired in step S22 matches the stored image of the highly reflective object, the determination unit 160 outputs the coordinate position information of the pixel determined in step S22 to the display control unit 170 (step S23). The display control unit 170 causes the display units 20 and 520 to display a display image including identification information for identifying a highly reflective object and two-dimensional image information based on the coordinate position information of the pixels acquired from the determination unit 160 ( Step S24), the process is terminated.

Step S22 and step S23 are examples of determination steps, and step S24 is an example of display steps.

When the determination unit 160 determines that the image acquired in step S22 does not match the stored image of the highly reflective object, the determination unit 160 proceeds to the determination of the proximity object (step S23), and performs the proximity object determination flow shown in FIG. Execute (step S25).

As described above, the image pickup apparatus 1 has a determination unit 160 for determining whether or not there is a highly reflective object based on both the output of the distance information acquisition unit 13 and the output of the image pickup unit 11, and the presence or absence of the highly reflective object. A display control unit 170 that causes the display units 20 and 520 to display different displays is provided accordingly.

This allows the photographer to accurately confirm that a highly reflective object such as a mirror is included in the captured image, distinguishing it from the influence of a nearby object or external light.

The image pickup apparatus 1 includes a display unit 20. As a result, the photographer can surely confirm that the highly reflective object is included in the captured image.

The display control unit 170 causes different displays at the positions of the display units 20 and 520 according to the position of the highly reflective object. This allows the photographer to confirm the position of the highly reflective object.

Similar to the proximity object shown in FIG. 13, the display unit 20 includes a plurality of display units 20A and 20a, and the display control unit 170 is a display unit on the side of the plurality of display units 20A and 20a that is closer to the highly reflective object. Is displayed differently depending on the presence or absence of the object. This allows the photographer to reliably confirm the position of the highly reflective object.

Similar to the proximity object shown in FIG. 3, the display control unit 170 displays the image information G imaged by the image pickup unit 11 on the display units 20 and 520, and also identifies the identification information for identifying the highly reflective object and the image information G. The display image including the above is displayed on the display units 20 and 520. This allows the photographer to reliably confirm the position of the highly reflective object.

The determination unit 160 has a saturated amount of electricity stored as an example of a pixel in which the amount of electricity stored in the pixel due to the light received by the distance information acquisition unit 13 is equal to or greater than a predetermined value, and the image information captured by the imaging unit is a highly reflective object. If it matches the model image information which is an example of the reference information indicating, it is judged that there is a highly reflective object.

This allows the photographer to accurately confirm that a highly reflective object is included in the captured image, distinguishing it from the influence of nearby objects and external light.

The image pickup apparatus 1 acquires the distance information to the target based on the light received by the distance information acquisition unit 13. In this case, the photographer can confirm that the factor that the desired distance information is not acquired is not a close object or external light but a highly reflective object.

The image pickup apparatus 1 includes a transmission / reception unit 180 that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition unit 13. In this case, the photographer can confirm that the factor that the desired three-dimensional information is not acquired is not a close object or external light but a highly reflective object.

FIG. 16 is a flow chart for determining a distant object and a low-reflecting object in the present embodiment, and a process for determining whether or not a distant object and a low-reflecting object are reflected in a captured image corresponding to step S9 described with reference to FIG. 5 is performed. It is a flowchart which shows.

Based on the TOF image data of all celestial spheres acquired from the reprojection processing unit 147, the determination unit 160 determines whether or not there are pixels in the TOF image data of all celestial spheres whose stored amount is equal to or less than the threshold at which distance information can be acquired. (Step S41).

When there are no pixels whose storage amount is equal to or less than the threshold value in step S41, the determination unit 160 has 10 m or more of the TOF image data of all celestial spheres based on the TOF image data of all celestial spheres acquired from the reprojection processing unit 147. If there is a pixel indicating the distance information of 10 m or more, it is determined that the object is a distant object, and the coordinate position information of the pixel is output to the display control unit 170 (step S42).

The display control unit 170 causes the display units 20 and 520 to display a display image including identification information for identifying a distant object and two-dimensional image information based on the coordinate position information of the pixels acquired from the determination unit 160 ( Step S43), the process is terminated.

If there is no pixel indicating distance information of 10 m or more in step S42, the determination unit 160 ends the process.

When the determination unit 160 has a pixel whose storage amount is equal to or less than the threshold value in step S41, the determination unit 160 is based on the RGB image data of the whole celestial sphere acquired from the RGB image data acquisition unit 142, and among the RGB image data of the whole celestial sphere, the step With respect to the pixel having the same coordinates as the pixel whose electricity storage amount is equal to or less than the threshold value in S41, it is determined whether the electricity storage amount is equal to or less than the object recognizable threshold value (step S44).

When the determination unit 160 determines in step S44 that the amount of stored electricity is equal to or less than the threshold value at which the object can be recognized, the determination unit 160 determines that the object is a low-reflecting object and outputs the coordinate position information of the pixels to the display control unit 170.

The display control unit 170 displays a display image including identification information for identifying a low-reflecting object and two-dimensional image information on the display units 20 and 520 based on the coordinate position information of the pixels acquired from the determination unit 160 ( Step S45), the process is terminated.

When the determination unit 160 determines in step S44 that the amount of electricity stored is not equal to or less than the threshold value at which the object can be recognized, the determination unit 160 is an example of reference information in which the image and the distance are associated with each other for the RGB image data including the pixels determined in step S44. The distance is determined based on the model information (step S46). When a model image is used as the reference information, the degree of matching between the RGB image data and the model image may be determined by image recognition. Further, the degree of agreement may be determined by a predetermined threshold value by using parameters as reference information and RGB image data. Further, the reference information may be stored in a table, or a learning model may be used.

The processing circuit 14 stores an image associated with the distance as model information for each of the plurality of distances, and the determination unit 160 acquires the image in step S46 using a determination device such as AI. It is determined whether or not the image is matched with the stored image for each distance.

The determination unit 160 determines whether the distance associated with the image matching the image acquired in step S46 is 10 m or more, and if it is 10 m or more, determines that it is a distant object and displays the coordinate position information of the pixels. Output to the control unit 170 (step S47), and the process proceeds to step S43.

If the distance associated with the image matching the image acquired in step S46 is not 10 m or more, the determination unit 160 determines that the image is a low-reflecting object, and outputs the coordinate position information of the pixels to the display control unit 170. (Step S47), the process proceeds to step S45.

Steps S41, S42, S44, and S47 are examples of determination steps, and steps S43 and S45 are examples of display steps.

As described above, the image pickup apparatus 1 includes a determination unit 160 for determining whether or not there is a distant object or a low reflection object based on both the output of the distance information acquisition unit 13 and the output of the image pickup unit 11, and the distant object or the image pickup device 1. A display control unit 170 that causes the display units 20 and 520 to display differently depending on the presence or absence of a low-reflecting object is provided.

This allows the photographer to accurately confirm that a distant object or a low-reflecting object such as black is included in the captured image.

The image pickup apparatus 1 includes a display unit 20. This allows the photographer to reliably confirm that a distant object or a low-reflection object is included in the captured image.

The display control unit 170 causes different displays at the positions of the display units 20 and 520 according to the positions of the distant object or the low reflection object. This allows the photographer to confirm the position of a distant object or a low reflection object.

Similar to the proximity object shown in FIG. 13, the display unit 20 includes a plurality of display units 20A and 20a, and the display control unit 170 is located on the side of the plurality of display units 20A and 20a that is closer to the distant object or the low reflection object. The display unit of is displayed differently depending on the presence or absence of the object. This allows the photographer to reliably confirm the position of a distant object or a low reflection object.

Similar to the proximity object shown in FIG. 3, the display control unit 170 displays the image information G imaged by the image pickup unit 11 on the display units 20 and 520, and also provides identification information for identifying a distant object or a low reflection object. A display image including the image information G is displayed on the display units 20 and 520. This allows the photographer to reliably confirm the position of a distant object or a low reflection object.

When the storage amount of the pixel by the light received by the distance information acquisition unit 13 is equal to or less than the threshold value, the determination unit 160 determines whether the object is a low reflection object or a distant object based on the output of the image pickup unit 11. As a result, the photographer can accurately confirm whether the captured image includes a low-reflecting object or a distant object.

The determination unit 160 determines that there is a low-reflecting object when the amount of electricity stored in the pixels due to the light received by the distance information acquisition unit 13 is less than or equal to the threshold value and the amount of electricity stored in the pixels of the image pickup unit 11 is less than or equal to the threshold value.
This allows the photographer to accurately confirm that the low-reflection object is included in the captured image.

In the determination unit 160, the amount of electricity stored in the pixels due to the light received by the distance information acquisition unit 13 is equal to or less than the threshold value, the amount of electricity stored in the pixels of the imaging unit 11 is equal to or greater than the threshold value, and the distance determined based on the pixels is equal to or greater than the threshold value. In that case, it is judged that there is a distant object.

This allows the photographer to accurately confirm that a distant object is included in the captured image.

The image pickup apparatus 1 acquires the distance information to the target based on the light received by the distance information acquisition unit 13. In this case, the photographer can confirm that the factor that the desired distance information is not acquired is a distant object or a low reflection object.

The image pickup apparatus 1 includes a transmission / reception unit 180, which is an example of an output unit that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition unit 13. In this case, the photographer can confirm that the factor for not acquiring the desired three-dimensional information is a distant object or a low-reflection object.

FIG. 17 is a flow chart for determining image blurring in the present embodiment, and is a flowchart showing a determination process for determining the presence or absence of image blurring in the captured image corresponding to step S9 described with reference to FIG.

The determination unit 160 determines whether or not there are pixels of an image including an edge peripheral region in the RGB image of the entire celestial sphere based on the RGB image data of the entire celestial sphere acquired from the RGB image data acquisition unit 142 (step S51). ..

Here, the determination unit 160 detects the edge included in the image by comparing the change in the luminance value in the pixel and its primary and secondary differential values with the threshold value, and determines the pixel of the image including the edge peripheral region. Although specified, edges may be detected by other methods.

Next, when there are pixels of the image including the edge peripheral region in step S51, in step S51 of the TOF image data of all celestial spheres based on the TOF image data of all celestial spheres acquired from the reprojection processing unit 147. With respect to the TOF image data including the pixels of the image having the same coordinates as the pixels of the image determined to include the edge peripheral region, it is determined whether the edges of the TOF phase image are deviated, and if it is determined that the edges are deviated, step S51. The coordinate position information of the pixel determined in step 2 is output to the display control unit 170 (step S52).

The display control unit 170 causes the display units 20 and 520 to display a display image including identification information for identifying image blur and two-dimensional image information based on the coordinate position information of the pixels acquired from the determination unit 160 ( Step S53), the process is terminated.

Steps S51 and S52 are examples of determination steps, and step S53 is an example of display steps.

The determination unit 160 ends the process when there are no pixels of the image including the edge peripheral region in step S51 and when the edges of the TOF phase image are not displaced in step S52.

In the present embodiment, the distance is measured by the phase difference detection method, and the image pickup apparatus 1 acquires N TOF phase images of the same phase for each phase of 0 °, 90 °, 180 °, and 270 °. to add.

By adding N phase images of the same phase in this way, the dynamic range of the phase images of that phase is expanded. In addition, the time required to capture the N phase images to which each phase is added has been shortened, and a phase image having excellent position accuracy with less influence of blurring and the like can be obtained. Therefore, with the phase image having the expanded dynamic range, the following image shift amount detection process can be executed with high accuracy.

The determination unit 160 calculates the amount of pixel shift for each phase by a process for obtaining a general optical flow or a machine learning method disclosed in the following reference paper, and calculates the amount of pixel shift for each phase. The value added in all phases is compared with the threshold value to finally determine the presence or absence of image blur, but the presence or absence of image blur may be determined by another method.

Paper title Tackling 3D ToF Artifacts Through Learning and the FLAT Dataset
Author Qi Guo (SEAS, Harvard University),
Iuri Frosio
Orazio Gallo
Todd Zickler (SEAS, Harvard University)
Jan Kautz
release date
Monday, September 10, 2018
Publisher
ECCV (European Computer Vision) 2018
URL (Uniform Resource Identifier)
https: // research. nvidia. com / publication / 2018-09_Tackling-3D-ToF

As described above, the image pickup apparatus 1 has a determination unit 160 for determining whether or not there is image blurring based on both the output of the distance information acquisition unit 13 and the output of the image pickup unit 11, and the presence or absence of image blurring. A display control unit 170 that causes the display units 20 and 520 to display different displays is provided accordingly.

This allows the photographer to accurately confirm that the image blur is included in the captured image.

The image pickup apparatus 1 includes a display unit 20. As a result, the photographer can surely confirm that the blurred image is included in the captured image.

The display control unit 170 causes different displays at the positions of the display units 20 and 520 according to the position of the blur of the image. This allows the photographer to confirm the position of the blur in the image.

Similar to the proximity object shown in FIG. 13, the display unit 20 includes a plurality of display units 20A and 20a, and the display control unit 170 is located on the side of the plurality of display units 20A and 20a that is closer to the blur position of the image. The display unit is made to display differently depending on the presence or absence of the object. As a result, the photographer can surely confirm the position of the blur of the image.

Similar to the proximity object shown in FIG. 3, the display control unit 170 displays the image information G imaged by the image pickup unit 11 on the display units 20 and 520, and also identifies the identification information for identifying the blur of the image and the image information G. The display image including the above is displayed on the display units 20 and 520. As a result, the photographer can surely confirm the position of the blur of the image.

The determination unit 160 detects the edge of the image based on the image information captured by the image pickup unit 11, and determines that the image is blurred when the pixels due to the light received by the distance information acquisition unit 13 are displaced.

This allows the photographer to accurately confirm that the image blur is included in the captured image.

The image pickup apparatus 1 acquires the distance information to the target based on the light received by the distance information acquisition unit 13. In this case, the photographer can confirm that the factor that the desired distance information is not acquired is the blurring of the image.

The image pickup apparatus 1 includes a transmission / reception unit 180, which is an example of an output unit that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition unit 13. In this case, the photographer can confirm that the factor that the desired three-dimensional information is not acquired is the blurring of the image.

FIG. 18 is a determination flow chart in a fourth modification of the embodiment of the present invention.

In step S9 described with reference to FIG. 5, the determination unit 160 determines the presence / absence of a specific object such as a nearby object, and the display control unit 170 differs from the display units 20 and 520 depending on the presence / absence of the specific object. However, in the fourth modification, the determination unit 160 does not determine the presence or absence of a specific object, and the display control unit 170 determines the presence or absence of the specific object, depending on the presence or absence of the specific object. Does not display differently, but allows the user to recognize the object in a specific way.

In the flow shown in FIG. 18A, the determination unit 160 has a storage amount of a predetermined value or more in the TOF image data of all celestial spheres based on the TOF image data of all celestial spheres acquired from the reprojection processing unit 147. As an example of a certain pixel, it is determined whether there is a pixel whose storage amount is saturated and the distance information can be acquired or more than the threshold value, and when there is a pixel whose storage amount is equal to or more than the threshold value, the coordinate position information of the pixel is displayed. Output to the control unit 170 (step S31).

The display control unit 170 is based on the coordinate position information of the pixels acquired from the determination unit 160, and similarly to the proximity object shown in FIG. 3, the display control unit 170 includes the position identification information for identifying the position and the two-dimensional image information. The image is displayed on the display units 20 and 520 (step S32), and the process ends.

If the amount of electricity stored is not equal to or greater than the threshold value in step S31, the determination unit 160 ends the process.

In the flow shown in FIG. 18B, the determination unit 160 acquires distance information in the TOF image data of all celestial spheres based on the TOF image data of all celestial spheres acquired from the reprojection processing unit 147. It is determined whether or not there are pixels that are below the possible threshold, and if there are pixels whose storage amount is below the threshold, the coordinate position information of the pixels is output to the display control unit 170 (step S33).

The display control unit 170 is based on the coordinate position information of the pixels acquired from the determination unit 160, and similarly to the proximity object shown in FIG. 3, the display control unit 170 includes the position identification information for identifying the position and the two-dimensional image information. The image is displayed on the display units 20 and 520 (step S34), and the process ends.

If the storage amount is not equal to or less than the threshold value in step S33, the determination unit 160 ends the process.

In the flow shown in FIG. 18C, the determination unit 160 shifts the TOF phase image into the TOF image data of all celestial spheres based on the TOF image data of all celestial spheres acquired from the reprojection processing unit 147. It is determined whether there is a pixel whose distance information cannot be acquired, and if there is a pixel whose TOF phase image is out of alignment, the coordinate position information of the pixel is output to the display control unit 170 (step S35).

Here, the determination unit 160 determines the deviation of the TOF phase image by the same method as that described in step S52 of FIG.

The display control unit 170 is based on the coordinate position information of the pixels acquired from the determination unit 160, and similarly to the proximity object shown in FIG. 3, the display control unit 170 includes the position identification information for identifying the position and the two-dimensional image information. The image is displayed on the display units 20 and 520 (step S36), and the process ends.

The determination unit 160 ends the process when there are no pixels whose TOF phase image is out of alignment.

As described above, the image pickup apparatus 1 identifies the position based on the position information indicating the position determined by the determination unit 160 whether the output of the distance information acquisition unit 13 is equal to or greater than the threshold value or equal to or less than the threshold value. A display control unit 170 that displays a display image including the two-dimensional image information G captured by the image pickup unit 11 that captures the target on the display units 20 and 520 is provided.

As a result, the position where the output of the distance information acquisition unit 13 is equal to or greater than or equal to the threshold value, that is, the position where the output of the distance information acquisition unit 13 is too strong or too weak to obtain a desired output is the two-dimensional image G. By confirming with, it is possible to confirm the reason why the desired output cannot be obtained.

Further, the image pickup apparatus 1 has position identification information for identifying the position and the target based on the position information indicating the position determined by the determination unit 160 that the distance information to the target cannot be acquired based on the output of the distance information acquisition unit 13. The display control unit 170 is provided with the two-dimensional image information G captured by the image pickup unit 11 that captures the image, and the display control unit 170 that displays the display image including the image information G on the display units 20 and 520.

Thereby, by confirming the position where the distance information to the target cannot be acquired with the two-dimensional image G, it is possible to confirm the factor that the distance information to the target cannot be acquired.

The determination units 160, 560, and 660 determine that the distance information to the target cannot be acquired by the output of the distance information acquisition unit 13 except when the output of the distance information acquisition unit 13 is equal to or less than the threshold value. Including the case where the blur is detected.

FIG. 19 is a diagram showing an example of a configuration of a processing block of a processing circuit in a fifth modification of the embodiment of the present invention.

In the processing block of the processing circuit in the fifth modification shown in FIG. 19, the determination unit 160 outputs the determination result to the transmission / reception unit 180 with respect to the processing block of the processing circuit 14 of the present embodiment shown in FIG. The point, the point where the judgment unit 160 acquires the three-dimensional data of the entire celestial sphere from the three-dimensional reconstruction processing unit 150, the point where the judgment result is output to the transmission / reception unit 180, and the display control unit 170 re-three-dimensionally The point of acquiring the three-dimensional data of the whole celestial sphere from the configuration processing unit 150 is added.

In addition to the three-dimensional data of all celestial spheres output from the three-dimensional reconstruction processing unit 150 and the two-dimensional image information of all celestial spheres output from the RGB image data acquisition unit 142, the transmission / reception unit 180 of the determination unit 160 The determination result is transmitted (output) to the external device 300 that performs the three-dimensional restoration process via the network 400.

The display control unit 170 displays a three-dimensional image on the display unit 20 based on the three-dimensional data of the entire celestial sphere acquired from the three-dimensional reconstruction processing unit 150, and also outputs the image pickup unit 11 and the distance information acquisition unit 13. A display image including identification information for identifying a specific target and a three-dimensional image is displayed on the display unit 20 based on the judgment result by the judgment unit 160 that determines whether or not there is a specific target based on both the output of the above. To let. Specific objects also include near objects, high reflectors, distant objects, low reflectors and blurred areas of the image.

As a result, it is possible to confirm by observing the 3D image 3G whether the cause of not displaying the desired 3D image 3G is a near object, a high-reflecting object, a distant object, a low-reflecting object, or an image blur. can.

FIG. 20 is a diagram showing an example of the configuration of the information processing system in the sixth modification of the embodiment of the present invention.

The information processing system according to the sixth modification shown in FIG. 20 includes an image pickup device 1 and a display device 500.

The image pickup device 1 shown in FIG. 20 includes image pickup elements 11a and 11A, TOF sensors 13a and 13A, light source units 12a and 12A, and a photographing switch 15, which are configured in the same manner as those shown in FIG.

The processing circuit 4 in the image pickup apparatus 1 shown in FIG. 20 includes a control unit 141, an RGB image data acquisition unit 142, a TOF image data acquisition unit 144, and a transmission / reception unit 180. The control unit 141 is configured in the same manner as that shown in FIG.

Similar to FIG. 4, the RGB image data acquisition unit 142 acquires the RGB image data captured by the image pickup devices 11a and 11A based on the image pickup instruction by the control unit 141, and outputs the RGB image data of the entire celestial sphere. The difference from FIG. 4 is that the output destination is the transmission / reception unit 180.

Similar to FIG. 4, the TOF image data acquisition unit 144 acquires the TOF image data generated by the TOF sensors 13a and 13A based on the TOF image data generation instruction by the control unit 141, and the TOF image data of the entire celestial sphere. Is output, but it differs from FIG. 4 in that the output destination is the transmission / reception unit 180.

Unlike FIG. 4, the transmission / reception unit 180 displays the RGB image data of the whole celestial sphere output from the RGB image data acquisition unit 142 and the TOF image data of the whole celestial sphere output from the TOF image data acquisition unit 144 to the display device 500. Send (output).

The display device 500 shown in FIG. 20 includes a transmission / reception unit 510, a display unit 520, and a display control unit 530, as in the second modification shown in FIG.
RGB image data acquisition unit 542, monochrome processing unit 543, TOF image data acquisition unit 544, high resolution unit 545, matching processing unit 546, reprojection processing unit 547, semantic segmentation unit 548, and parallax calculation. A unit 549, a three-dimensional reconstruction processing unit 550, and a determination unit 560 are further provided.

The transmission / reception unit 180 receives the RGB image data of the whole celestial sphere and the TOF image data of the whole celestial sphere transmitted from the image pickup apparatus 1.

The RGB image data acquisition unit 542 acquires the RGB image data of the whole celestial sphere from the transmission / reception unit 180, and the TOF image data acquisition unit 544 acquires the RGB image data of the whole celestial sphere from the transmission / reception unit 180. , Each of the same as the RGB image data acquisition unit 142 and the TOF image data acquisition unit 144 shown in FIG.

Monochrome processing unit 543, TOF image data acquisition unit 544, high resolution unit 545, matching processing unit 546, reprojection processing unit 547, semantic segmentation unit 548, parallax calculation unit 549, and three-dimensional reconstruction. The processing unit 550 and the determination unit 560 include the monochrome processing unit 143, the TOF image data acquisition unit 144, the high resolution unit 145, the matching processing unit 146, the reprojection processing unit 147, and the reprojection processing unit 147 shown in FIG. It is configured in the same manner as the semantic segmentation unit 148, the parallax calculation unit 149, the three-dimensional reconstruction processing unit 150, and the determination unit 160.

The display control unit 530 may acquire RGB image data of all celestial spheres from the RGB image data acquisition unit 542 and display a two-dimensional image based on the acquired RGB image data of all celestial spheres on the display unit 520. The three-dimensional data of the whole celestial sphere may be acquired from the dimensional reconstruction processing unit 545, and the three-dimensional image report may be displayed on the display unit 520.

The display control unit 530 causes the display unit 520 to display a display image including the information indicating the determination result acquired from the determination unit 160 and the two-dimensional image or the three-dimensional image.

As described above, the display device 500 receives the output of the imaging unit 11 that images the target and the output of the distance information acquisition unit 13 that projects the light on the target and receives the light reflected from the target. A determination unit 560 that determines whether or not there is a specific object based on both the transmission / reception unit 510, which is an example, the output of the distance information acquisition unit 13 received by the transmission / reception unit 510, and the output of the imaging unit 11. A display control unit 530 that causes the display unit to display different displays depending on the presence or absence of a specific target based on the determination result of the unit 560 is provided.

Specific objects include near objects, high reflectors, distant objects, low reflectors and blurred areas of the image.

Further, the display device 500 determines a specific target based on both the output of the imaging unit 11 that images the target and the output of the distance information acquisition unit 13 that projects light onto the target and receives the light reflected from the target. A display image including identification information for identifying a specific target and a three-dimensional image 3G determined by the three-dimensional reconstruction processing unit 550 is displayed on the display unit 520 based on the determination result by the determination unit 560 that determines the presence or absence. A display control unit 530 for displaying is provided.

FIG. 21 is a diagram showing an example of the configuration of the information processing system in the seventh modification of the embodiment of the present invention.

The information processing system according to the seventh modification shown in FIG. 21 includes an image pickup device 1, a display device 500, and a server 600.

The image pickup device 1 shown in FIG. 21 is configured in the same manner as the image pickup device 1 shown in FIG. 20, and the display device 500 shown in FIG. 21 is configured in the same manner as the display device 500 shown in FIG.

The server 600 shown in FIG. 21 is a reprojection of the receiving unit 610, the RGB image data acquisition unit 642, the monochrome processing unit 643, the TOF image data acquisition unit 644, the high resolution unit 645, the matching processing unit 646, and the matching processing unit 646. It includes a processing unit 647, a semantic segmentation unit 648, a parallax calculation unit 649, a three-dimensional reconstruction processing unit 650, a determination unit 660, and a transmission unit 680.

The receiving unit 610 receives the RGB image data of the whole celestial sphere and the TOF image data of the whole celestial sphere transmitted from the image pickup apparatus 1 via the network 400.

The RGB image data acquisition unit 642 acquires the RGB image data of the whole sky from the reception unit 610, and the TOF image data acquisition unit 644 acquires the RGB image data of the whole sky from the reception unit 610, but other than that. , Each of the same as the RGB image data acquisition unit 142 and the TOF image data acquisition unit 144 shown in FIG.

Monochrome processing unit 643, TOF image data acquisition unit 644, high resolution unit 645, matching processing unit 646, reprojection processing unit 647, semantic segmentation unit 648, parallax calculation unit 649, and three-dimensional reconstruction. The processing unit 650 and the determination unit 660 include the monochrome processing unit 143, the TOF image data acquisition unit 144, the high resolution unit 145, the matching processing unit 146, the reprojection processing unit 147, and the reprojection processing unit 147 shown in FIG. It is configured in the same manner as the semantic segmentation unit 148, the parallax calculation unit 149, the three-dimensional reconstruction processing unit 150, and the determination unit 160.

The transmission unit 680 determines the three-dimensional data of all celestial spheres output from the three-dimensional reconstruction processing unit 150, the two-dimensional image information of all celestial spheres output from the RGB image data acquisition unit 142, and the determination unit 160. The result and the result are transmitted (output) to the display device 500 via the network 400.

The transmission / reception unit 510 of the display device 510 receives the three-dimensional data of the whole celestial sphere, the two-dimensional image information, and the determination result of the determination unit 160 transmitted from the server 600.

The display control unit 530 of the display device 510 may acquire the RGB image data of the whole celestial sphere from the transmission / reception unit 510 and display the two-dimensional image based on the acquired RGB image data of the celestial sphere on the display unit 520. The three-dimensional data of the whole celestial sphere may be acquired from the transmission / reception unit 510, and the three-dimensional image report may be displayed on the display unit 20.

The display control unit 530 causes the display unit 520 to display a display image including information indicating a determination result acquired from the transmission / reception unit 510 and a two-dimensional image or a three-dimensional image.

As described above, the display device 500 is based on both the output of the imaging unit 11 that images the target and the output of the distance information acquisition unit 13 that projects the light on the target and receives the light reflected from the target. Based on the determination result received by the transmission / reception unit 510 and the transmission / reception unit 510 that receive the determination result by the determination unit 660 of the server 600 whether or not there is a specific target, the display unit 520 has a different display depending on the presence or absence of the specific object. The display control unit 530 and the like are provided. Specific objects include near objects, high reflectors, distant objects, low reflectors and blurred areas of the image.

Further, the display device 500 determines a specific target based on both the output of the imaging unit 11 that images the target and the output of the distance information acquisition unit 13 that projects light onto the target and receives the light reflected from the target. A display image including identification information for identifying a specific target and a three-dimensional image 3G determined by the three-dimensional reconstruction processing unit 650 is displayed on the display unit 520 based on the determination result by the determination unit 660 that determines the presence or absence. A display control unit 530 for displaying is provided.

FIG. 22 is a diagram for explaining the display contents of the display unit in the fifth to seventh modified examples.

As shown in FIG. 22, the display control unit 530 displays a three-dimensional image 3G including identification information 3Ga, 3Gb, and 3Gc for identifying a specific target on the display unit 520. 3Ga, 3Gb and 3Gc may be position identification information that identifies the position of a specific target.

Although the display unit 520 is shown in FIG. 22, the display unit 170 also displays the three-dimensional image 3G including the identification information 3Ga, 3Gb, and 3Gc for identifying a specific target by the display control unit 170.

Here, the identification information 3Ga indicates a blind spot and is identified and displayed in pink or the like, the identification information 3Gb indicates a low-reflecting object and is identified and displayed in orange or the like, and the identification information 3Gc indicates a distant object and is identified and displayed by a mosaic or the like. There is.

All of these identification information 3Ga, 3Gb and 3Gc may be displayed at the same time, or any one or two may be displayed.

FIG. 23 is a diagram for explaining a three-dimensional image displayed by the display unit according to the embodiment of the present invention.

FIG. 23A is a diagram showing the positions of a virtual camera and a predetermined area when the spherical image is a three-dimensional three-dimensional sphere. The virtual camera IC corresponds to the position of the viewpoint of the user who views the image with respect to the spherical image CE displayed as a three-dimensional three-dimensional sphere.

23 (b) is a three-dimensional perspective view of FIG. 23 (a), and FIG. 23 (c) is a diagram showing a predetermined area image when displayed on a display.

FIG. 23 (b) represents the spherical image CE shown in FIG. 23 (a) as a three-dimensional three-dimensional sphere CS. Assuming that the spherical image CE generated in this way is a three-dimensional sphere CS, the virtual camera IC is located inside the spherical image CE as shown in FIG. 23 (a). ing.

The predetermined area T in the spherical image CE is a shooting area of the virtual camera IC, and is specified by predetermined area information indicating the shooting direction and angle of view of the virtual camera IC in the three-dimensional virtual space including the spherical image CE. NS.

Further, the zoom of the predetermined area T can be expressed by moving the virtual camera IC closer to or further away from the spherical image CE. The predetermined region image Q is an image of the predetermined region T in the spherical image CE. Therefore, the predetermined region T can be specified by the angle of view α and the distance f from the virtual camera IC to the spherical image CE.

That is, the display control units 170 and 530 change the position and orientation of the virtual camera IC at the position of the viewpoint for viewing the 3D image 3G to display the display area of the 3D image 3G to be displayed on the display units 20 and 520. change.

In the above, the three-dimensional image displayed by the display unit has been described with the example of using the spherical image, but the same applies to the case of using the three-dimensional point cloud data. A three-dimensional point cloud is placed in the virtual space, and a virtual camera is placed in the virtual space. A three-dimensional image is obtained by projecting a three-dimensional point cloud onto a predetermined projection surface in the virtual space based on predetermined area information indicating the viewpoint position, shooting direction, and angle of view of the virtual camera. Further, the display area of the three-dimensional image is changed by changing the viewpoint position and orientation of the virtual camera.

FIG. 24 is a judgment flow diagram in the fifth to seventh modified examples. Judgment units 160, 560, and 660 have a threshold value of the density of point cloud data in the three-dimensional data of all celestial spheres based on the three-dimensional data of all celestial spheres acquired from the three-dimensional reconstruction processing units 150, 550, and 650. It is determined whether there is an area (coordinates) less than (step S61).

When the determination units 160, 560, and 660 determine in step S61 that there is a region (coordinate) in which the density of the point cloud data is less than the threshold value, the determination units 160, 560, and 660 refer to a plurality of pixels having the same coordinates as the region in which the density of the point cloud data is less than the threshold value. , Based on the output of the imaging unit 11 in the flow shown in FIG. 16, it is determined whether or not the pixel determined to be a distant object is included, and if the pixel determined to be a distant object is included, the coordinate position information of the pixel is displayed and controlled. Output to units 170 and 530 (step S62).

As shown in FIG. 22, the display control units 170 and 530 have position identification information 3Gc for identifying the position of a distant object and a three-dimensional image based on the coordinate position information of the pixels acquired from the judgment units 160, 560 and 660. A display image including G and is displayed on the display units 20 and 520 (step S63), and the process ends.

The determination units 160, 560, and 660 are shown in FIG. 16 when the pixels determined to be distant objects are not included in the plurality of pixels having the same coordinates as the region where the density of the point cloud data is less than the threshold in step S62. Based on the output of the imaging unit 11 in the flow, it is determined whether or not the pixel determined to be a low-reflecting object is included, and if the pixel determined to be a low-reflecting object is included, the coordinate position information of the pixel is displayed in the display control unit 170, 530. Output to (step S64).

As shown in FIG. 22, the display control units 170 and 530 have three-dimensional position identification information 3Gb for identifying the position of the low-reflecting object based on the coordinate position information of the pixels acquired from the determination units 160, 560 and 660. The display image including the image G is displayed on the display units 20 and 520 (step S65), and the process ends.

In step S64, the determination units 160, 560, and 660 determine that a plurality of pixels having the same coordinates as the region where the density of the point cloud data is less than the threshold value are blind spots when the pixels determined to be low-reflecting objects are not included. Judgment is made, and the coordinate position information of the pixel is output to the display control units 170 and 530.

As shown in FIG. 22, the display control units 170 and 530 have the position identification information 3Ga for identifying the position of the blind spot and the three-dimensional image G based on the coordinate position information of the pixels acquired from the judgment units 160, 560 and 660. The display image including the above is displayed on the display units 20 and 520 (step S66), and the process ends. Steps S61, S62 and S64 are examples of determination steps, and steps S63, S65 and S66 are examples of display steps.

As described above, the image pickup device 1 and the display device 500 include the output of the image pickup unit 11 that images the target and the output of the distance information acquisition unit 13 that projects light on the target and receives the light reflected from the target. Based on the judgment results of the judgment units 160, 560, and 660 that determine whether or not there is a specific target based on both of the above, the identification information 3Ga, 3Gb, and 3Gc that identify the specific target, and the output of the distance information acquisition unit 13. The display control unit 170, which causes the display units 20 and 520 to display different display images including the three-dimensional image 3G determined by the three-dimensional reconstruction processing units 150, 550, and 650, which is an example of the three-dimensional information determination unit. 530 is provided.

Specific objects include not only distant objects, low reflectors and blind spots, but also near objects, high reflectors, and blurred areas of the image.

As a result, it is confirmed by observing the 3D image 3G whether the cause of not displaying the desired 3D image 3G is a distant object, a low-reflecting object, a blind spot, a near object, a high-reflecting object, or an image blur. can do.

Further, the image pickup device 1 and the display device 500 cause the display units 20 and 520 to display the three-dimensional image 3G determined based on the output of the distance information acquisition unit 13 that receives the light projected from the target and reflected from the target. The display control units 170 and 530 are provided, and the display control units 170 and 530 project a distant object that is far from the distance information acquisition unit 13 when the light reflected from the object is received in the three-dimensional image 3G. A distant object based on a low-reflecting object having a low reflectance to the light, and a position information indicating a position determined to be at least one blind spot with respect to the distance information acquisition unit 13 when the light reflected from the object is received. A display image including the position identification information 3Ga, 3Gb or 3Gc for identifying at least one position of the low reflector and the blind spot, and the three-dimensional image 3G is displayed on the display units 20 and 520.

As a result, it is possible to confirm by observing the 3D image 3G whether the factor that the desired 3D image 3G is not displayed is a distant object, a low reflection object, or a blind spot. It becomes possible to take measures such as imaging.

The three-dimensional image 3G is determined by the three-dimensional reconstruction processing units 150, 550, and 650, which are examples of the three-dimensional information determination unit.

The display control unit 170, 530 displays a display including any one of the position identification information 3Ga, 3Gb, and 3Gc and the three-dimensional image 3G based on the position information of any one of the distant object, the low-reflecting object, and the blind spot. The image may be displayed on the display units 20, 520, and based on the position information of any two or all of the distant object, the low-reflecting object, and the blind spot, any two of the position identification information 3Ga, 3Gb, and 3Gc or A display image including all and the three-dimensional image 3G may be displayed on the display units 20 and 520.

When the information processing device is the image pickup device 1, the image pickup device 1 includes a distance information acquisition unit 13 and a three-dimensional reconstruction processing unit 150, as shown in FIG.

When the information processing device is the display device 500, as shown in FIGS. 20 and 21, the display device 500 does not include the distance information acquisition unit 13, and the image pickup device 1 includes the distance information acquisition unit 13. Then, the output of the distance information acquisition unit 13 is transmitted to the display device 500 or the server 600.

As shown in FIG. 20, the display device 500 may or may not include the three-dimensional reconstruction processing unit 550.

When the display device 500 does not include the three-dimensional reconstruction processing unit 550, the imaging device 1 may include the three-dimensional reconstruction processing unit 150 and transmit a three-dimensional image to the display device 500, as shown in FIG. In addition, the server 600 may include a three-dimensional reconstruction processing unit 650 and transmit a three-dimensional image to the display device 500.

The display control units 170 and 530 indicate positions where the density of the point cloud data included in the three-dimensional image 3G is lower than the threshold value and is determined to be at least one of a distant object, a low reflection object, or a blind spot. Based on the position information, a display image including the position identification information 3Ga, 3Gb, 3Gc, and the three-dimensional image 3G is displayed on the display units 20 and 520.

Thereby, it is possible to confirm by observing the three-dimensional image 3G whether the factor that the density of the point cloud data is lower than the threshold value is a distant object, a low reflection object, or a blind spot.

The display control units 170 and 530 are positions indicating a position determined to be at least one of a distant object, a low reflection object, or a blind spot in the three-dimensional image 3G based on the output of the image pickup unit 11 that images the object. Based on the information, a display image including the position identification information 3Ga, 3Gb, 3Gc, and the three-dimensional image 3G is displayed on the display units 20 and 520.

Thereby, it is possible to accurately determine whether the factor that the desired three-dimensional image 3G is not displayed is a distant object, a low-reflecting object, or a blind spot based on the output of the imaging unit 11.

When the information processing device is the image pickup device 1, the image pickup device 1 includes an image pickup unit 11 as shown in FIG. When the information processing device is the display device 500, the display device 500 does not include the image pickup unit 11 as shown in FIGS. 20 and 21, and the image pickup device 1 includes the image pickup unit 11 and includes the image pickup unit 11. The output of 11 is transmitted to the display device 500 or the server 600.

The image pickup device 1 and the display device 500 include determination units 160, 560, and 660 for determining a position that is at least one of a distant object, a low reflection object, or a blind spot in the three-dimensional image 3G, and the display control unit 170, Based on the determination results of the determination units 160, 560, and 660, the 530 causes the display units 20 and 520 to display a display image including the position identification information 3Ga, 3Gb, 3Gc, and the three-dimensional image 3G.

When the information processing device is the image pickup device 1, the image pickup device 1 includes a determination unit 160 as shown in FIG.

When the information processing device is the display device 500, the display device 500 may or may not include the determination unit 560 as shown in FIG. 20.

When the display device 500 does not include the determination unit 560, the image pickup device 1 may include the determination unit 160 and transmit the determination result to the display device 500, and the server 600 includes the determination unit 660 as shown in FIG. The determination result may be transmitted to the display device 500.

FIG. 25 is another diagram for explaining the display contents of the display unit in the fifth to seventh modified examples.

As shown in FIG. 25, the display unit 520 is three-dimensionally including position identification information 3G1 and 3G2 that identify the position of the distance information acquisition unit 13 when the display control unit 530 receives the light reflected from the target. Image 3G is displayed.

The three-dimensional image 3G is determined based on the output of the distance information acquisition unit 13 located at the first position and the output of the distance information acquisition unit 13 located at a second position different from the first position. The identification information 3G1 is an example of the first position identification information for identifying the first position, and the position identification information 3G2 is an example of the first position identification information for identifying the second position.

Although the display unit 520 is shown in FIG. 25, the display unit 20 also has the position identification information for identifying the position of the distance information acquisition unit 13 when the light reflected from the target is received by the display control unit 170. A three-dimensional image 3G including 3G1 and 3G2 is displayed.

The display control units 170 and 530 display a display image including the three-dimensional image 3G and the identification information 3Ga, 3Gb and 3Gc, which are examples of the low-density identification information, on the display units 20 and 520 as shown in FIG. In addition to displaying the image, as shown in FIG. 25, the display image may include position identification information 3G1 and 3G2 that identify the position of the distance information acquisition unit 13 when the light reflected from the object is received.

FIG. 26 is a flow chart illustrating processing in the fifth to seventh modified examples.

The three-dimensional reconstruction processing units 150, 550, and 650 read the high-density three-dimensional point cloud data of the entire celestial sphere (step S71), and receive the light reflected from the target at the origin of the three-dimensional point cloud data. It is acquired as position information indicating the imaging position of the distance information acquisition unit 13 (step S72).

The three-dimensional reconstruction processing units 150, 550, and 650 confirm whether there is the three-dimensional point cloud data read in advance, and if there is no three-dimensional point cloud data read in advance, the three-dimensional point cloud data read in step S71 3 The 3D point cloud data and the position information acquired in step S72 are output to the display control units 170 and 530 (step S73).

The display control units 170 and 530 received the light reflected from the target as shown in FIG. 25 based on the three-dimensional point cloud data and the position information acquired from the three-dimensional reconstruction processing units 150, 550 and 650. A display image including the position identification information 3G1 for identifying the position of the distance information acquisition unit 13 and the three-dimensional image 3G is displayed on the display units 20 and 520 (step S74), and the process ends.

If there is 3D point cloud data read in advance in step S73, the 3D reconstruction processing units 150, 550, and 650 have the 3D point cloud data read in step S71 and the 3D point cloud read in advance. The point cloud data is integrated (step S75).

The three-dimensional reconstruction processing units 150, 550, and 650 integrate the origin of the three-dimensional point cloud data read in step S71 and the origin of the three-dimensional point cloud data read in advance in step S75 in three dimensions. The coordinates in the point cloud data are calculated as the position information of the imaging position, and the three-dimensional point cloud data integrated in step S75 and the calculated plurality of position information are output to the display control units 170 and 530 (step S76).

In step S74, the display control units 170 and 530 reflect from the target as shown in FIG. 25 based on the three-dimensional point cloud data acquired from the three-dimensional reconstruction processing units 150, 550 and 650 and the plurality of position information. A display image including a plurality of position identification information 3G1, 3G2 and a three-dimensional image 3G for identifying the position of the distance information acquisition unit 13 when the received light is received is displayed on the display units 20 and 520.

FIG. 27 is another flow chart illustrating the processing in the fifth to seventh modified examples.

The three-dimensional reconstruction processing units 150, 550, and 650 read the high-density three-dimensional point cloud data of the entire celestial sphere (step S81), and the determination units 160, 560, and 660 read the three-dimensional reconstruction processing units 150, 550, Based on the three-dimensional data of the whole celestial sphere acquired from 650, steps S61, S62, and S64 of the flow shown in FIG. 24 are executed to extract a low density part where the density of the point cloud data is lower than the threshold value (step S82). ).

When the virtual camera IC shown in FIG. 23 is at the position of the position identification information 3G1 or 3G2 shown in FIG. 25, the display control units 170 and 530 execute steps S63, S65 and S66 of the flow shown in FIG. 24. The orientation of the virtual camera IC is changed so that at least one of the identification information 3Ga, 3Gb, and 3Gc, which is an example of the low-density identification information shown in FIG. 22, is included in the display image (step S83).

As described above, the image pickup device 1 and the display device 500 include display control units 170 and 530 for displaying the three-dimensional image 3G determined based on the output of the distance information acquisition unit 13 on the display units 20 and 520. The control units 170 and 530 are based on the position information indicating the position of the distance information acquisition unit 13 when the light reflected from the target is received, and the position of the distance information acquisition unit 13 when the light reflected from the target is received. A display image including the position identification information 3G1, 3G2 and the three-dimensional image 3G for identifying the above is displayed on the display units 20 and 520.

As a result, the positional relationship between the imaging position indicating the position of the distance information acquisition unit 13 when the light reflected from the object is received and the specific object can be grasped in the three-dimensional image 3G.

The three-dimensional image 3G and the position information are determined by the three-dimensional reconstruction processing units 150, 550, and 650.

When the information processing device is the image pickup device 1, the image pickup device 1 includes a distance information acquisition unit 13 and a three-dimensional reconstruction processing unit 150, as shown in FIG.

When the information processing device is the display device 500, as shown in FIGS. 20 and 21, the display device 500 does not include the distance information acquisition unit 13, and the image pickup device 1 includes the distance information acquisition unit 13. Then, the output of the distance information acquisition unit 13 is transmitted to the display device 500 or the server 600.

As shown in FIG. 20, the display device 500 may or may not include the three-dimensional reconstruction processing unit 550.

When the display device 500 does not include the three-dimensional reconstruction processing unit 550, the imaging device 1 may include the three-dimensional reconstruction processing unit 150 and transmit the three-dimensional image and the position information to the display device 500. As shown, the server 600 may include the 3D reconstruction processing unit 650 and transmit the 3D image and the position information to the display device 500.

The display control units 170 and 530 are identification information 3Ga, 3Gb, 3Gc, which is an example of low-density identification information for identifying an area based on the area information indicating an area in which the density of the point cloud data in the three-dimensional image 3G is lower than the threshold value. A display image including the three-dimensional image 3G and the three-dimensional image 3G is displayed on the display units 20 and 520.

In this case, since the positional relationship between the imaging position and the region where the density of the point cloud data is lower than the threshold value can be grasped, it is possible to identify the factor in which the density of the point cloud data is lower than the threshold value. For example, if the area is far from the imaging position, a distant object is a factor, if the area is in the blind spot of the imaging position, a blind spot is a factor, and if it is neither a distant object nor a blind spot, a low-reflecting object is a factor. It can be identified that there is.

The display control units 170 and 530 change the display area of the 3D image 3G to be displayed on the display units 20 and 520 by changing the position and orientation of the virtual camera IC at the position of the viewpoint for viewing the 3D image 3G. ..

When the position of the virtual camera IC is at the positions 3G1 and 3G2 identified by the position identification information, the display control units 170 and 530 change the orientation of the virtual camera IC to a predetermined orientation. As the predetermined orientation, the location that causes re-imaging such as the low density point cloud region, the location that meets the preset conditions such as the confirmation location of the field survey, and the photographer or other confirmers pay attention. This is an area such that any part of the display area is included in the display area. Examples of sites to be confirmed in the site survey include the location where changes occur continuously at the site (material storage area), the position of each object in the main building (building itself), and the gap distance between the objects. Space for new installations, temporary construction (material storage, scaffolding, etc. that may be removed during the construction process), heavy equipment (forks, cranes) storage, workspace (turnable range, carry-in route), resident flow lines (Detour during construction) etc. are applicable.

This makes it possible to direct the line of sight of the user located at the imaging position to a specific object to be seen in the field.

The display control units 170 and 530 change the orientation of the virtual camera IC so that the display area includes a low-density portion in which the density of predetermined coordinates and point cloud data in the three-dimensional image 3G is lower than the threshold value. The predetermined coordinates do not specify the image, and are maintained even when the image at the predetermined coordinates changes before and after the integration of the three-dimensional point cloud data in step S75 of FIG. 26, for example.

This makes it possible to direct the line of sight of the user located at the imaging position to a specific object which is a low-density portion in the three-dimensional image 3G.

The display control units 170 and 530 are determined based on the output of the distance information acquisition unit 13 located at the first position and the output of the distance information acquisition unit 13 located at a second position different from the first position. The first position identification information 3G1 for displaying the three-dimensional image 3G on the display units 20 and 520 and identifying the first position, the second position identification information 3G2 for identifying the second position, and the three-dimensional image 3G, The display image including the above is displayed on the display units 20 and 520.

Thereby, the positional relationship between the first imaging position and the second imaging position and the specific target can be grasped in the three-dimensional image 3G.

● Summary ●
As described above, the image pickup apparatus 1 according to the embodiment of the present invention includes an image pickup unit 11 that images an object, a projection unit 12 that projects light onto the object, and distance information that receives light reflected from the object. Based on both the acquisition unit 13 (an example of the light receiving unit), the output of the distance information acquisition unit 13, and the output of the imaging unit 11, the determination unit 160 that determines whether or not there is a highly reflective object, and the presence or absence of the highly reflective object A display control unit 170 that causes the display units 20 and 520 to display different displays is provided accordingly.

This allows the photographer to accurately confirm that a highly reflective object such as a mirror is included in the captured image, distinguishing it from the influence of a nearby object or external light.

The image pickup apparatus 1 includes a display unit 20. As a result, the photographer can surely confirm that the highly reflective object is included in the captured image.

The display control unit 170 causes different displays at the positions of the display units 20 and 520 according to the position of the highly reflective object. This allows the photographer to confirm the position of the highly reflective object.

The display unit 20 includes a plurality of display units 20A and 20a, and the display control unit 170 displays different displays on the display unit closer to the highly reflective object among the plurality of display units 20A and 20a depending on the presence or absence of an object. To let. This allows the photographer to reliably confirm the position of the highly reflective object.

The display control unit 170 displays the image information G imaged by the image pickup unit 11 on the display units 20 and 520, and displays a display image including the identification information for identifying the highly reflective object and the image information G on the display unit 20, Display on 520. This allows the photographer to reliably confirm the position of the highly reflective object.

The determination unit 160 has a saturated amount of electricity stored as an example of a pixel in which the amount of electricity stored in the pixel due to the light received by the distance information acquisition unit 13 is equal to or greater than a predetermined value, and the image information captured by the imaging unit is a highly reflective object. If it matches the model image information as an example of the reference information indicating, it is judged that there is a highly reflective object.

This allows the photographer to accurately confirm that a highly reflective object is included in the captured image, distinguishing it from the influence of nearby objects and external light.

The image pickup apparatus 1 acquires the distance information to the target based on the light received by the distance information acquisition unit 13. In this case, the photographer can confirm that the factor that the desired distance information is not acquired is not a close object or external light but a highly reflective object.

The image pickup apparatus 1 includes a transmission / reception unit 180, which is an example of an output unit that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition unit 13. In this case, the photographer can confirm that the factor that the desired three-dimensional information is not acquired is not a close object or external light but a highly reflective object.

The imaging processing method according to the embodiment of the present invention includes an imaging step of capturing an image of an object by an imaging unit 11, a projection step of projecting light onto an object by a projection unit 12, and a distance information acquisition unit that acquires light reflected from the object. A determination step of determining whether or not there is a highly reflective object based on both the light receiving step of receiving light by 13 and the output of the distance information acquisition unit 13 and the output of the imaging unit 11, and a determination step of determining whether or not there is a highly reflective object, and high reflection. A display step is provided in which display control units 170 and 530 cause display units 20 and 520 to display differently depending on the presence or absence of an object.

The image pickup device 1 and the display device 500, which are examples of the information processing device according to the embodiment of the present invention, receive the output of the image pickup unit 11 that images the target and the light reflected from the target by projecting the light onto the target. Based on the judgment results by the judgment units 160, 560, and 660 that determine whether or not there is a high-reflecting object based on both the output of the distance information acquisition unit 13 and the output of the high-reflecting object, the display units 20 and 520 are displayed according to the presence or absence of the high-reflecting object. A display control unit 170, 530 for displaying different displays is provided.

The display device 500, which is an example of the information processing device according to the embodiment of the present invention, acquires the output of the imaging unit 11 that images the target and the distance information that receives the light reflected from the target by projecting the light on the target. The transmission / reception unit 510 and the transmission / reception unit 510 are examples of the reception unit that receives the determination result by the determination unit 160 of the image pickup apparatus 1 or the determination unit 660 of the server 600 as to whether or not there is a specific target based on both the output of the unit 13. A display control unit 530 that causes the display unit 520 to display differently depending on the presence or absence of a specific target based on the determination result received by the user. Specific objects include near objects, high reflectors, distant objects, low reflectors, blind spots and blurred areas of the image.

The display device 500, which is an example of the information processing device according to the embodiment of the present invention, acquires the output of the imaging unit 11 that images the target and the distance information that receives the light reflected from the target by projecting the light on the target. Is there a specific object based on both the transmission / reception unit 510, which is an example of the reception unit that receives the output of the unit 13, the output of the distance information acquisition unit 13 received by the transmission / reception unit 510, and the output of the imaging unit 11. A determination unit 560 for determining the above, and a display control unit 530 for causing the display unit to display different displays depending on the presence or absence of a specific target based on the determination result of the determination unit 560. Specific objects include near objects, high reflectors, distant objects, low reflectors, blind spots and blurred areas of the image.

The image pickup device 1 and the display device 500, which are examples of the information processing device according to the embodiment of the present invention, receive the output of the image pickup unit 11 that images the target and the light reflected from the target by projecting the light on the target. Identification information 3Ga, 3Gb, 3Gc and three-dimensional identification information 3Ga, 3Gb, 3Gc that identify a specific target based on the judgment result by the judgment units 160 and 560 that determine whether or not there is a specific target based on both the output of the distance information acquisition unit 13 to be processed. The display control units 170 and 530 are provided with an image 3G and a display control unit 170 and 530 for displaying a display image including the image 3G on the display units 20 and 520. Specific objects include not only distant objects, low reflectors and blind spots, but also near objects, high reflectors, and blurred areas of the image.

The 3D image 3G is determined by the 3D reconstruction processing units 150, 550, and 650, which are examples of the 3D information determination unit, based on the output of the distance information acquisition unit 13.

As a result, it is confirmed by observing the 3D image 3G whether the cause of not displaying the desired 3D image 3G is a distant object, a low-reflecting object, a blind spot, a near object, a high-reflecting object, or an image blur. can do.

In the image pickup device 1 and the display device 500, which are examples of the information processing device according to the embodiment of the present invention, the output of the distance information acquisition unit 13 that projects light on the target and receives the light reflected from the target is equal to or higher than the threshold value. Alternatively, the position identification information for identifying the position based on the position information indicating the position determined by the determination units 160 and 560 to determine whether the value is below the threshold value, and the two-dimensional image information G imaged by the imaging unit 11 for imaging the target. The display control units 170 and 530 are provided so that the display image including the above can be displayed on the display units 20 and 520.

As a result, the position where the output of the distance information acquisition unit 13 is equal to or greater than or equal to the threshold value, that is, the position where the output of the distance information acquisition unit 13 is too strong or too weak to obtain a desired output is the two-dimensional image G. By confirming with, it is possible to confirm the reason why the desired output cannot be obtained.

The image pickup device 1 and the display device 500, which are examples of the information processing device according to the embodiment of the present invention, are targets based on the output of the distance information acquisition unit 13 that projects light onto the target and receives the light reflected from the target. Based on the position information indicating the position determined by the determination units 160 and 560 that it is not possible to acquire the distance information up to, the position identification information for identifying the position and the two-dimensional image information G imaged by the imaging unit 11 for imaging the target. A display control unit 170, 530 for displaying a display image including, on the display units 20 and 520 is provided.

Thereby, by confirming the position where the distance information to the target cannot be acquired with the two-dimensional image G, it is possible to confirm the factor that the distance information to the target cannot be acquired.

The determination units 160, 560, and 660 determine that the distance information to the target cannot be acquired by the output of the distance information acquisition unit 13 except when the output of the distance information acquisition unit 13 is equal to or less than the threshold value. Including the case where the blur is detected.

In the above, when the information processing device is the image pickup device 1, the image pickup device 1 includes an image pickup unit 11, a distance information acquisition unit 13, a three-dimensional reconstruction processing unit 150, and a determination unit 160, as shown in FIG. ..

When the information processing device is the display device 500, as shown in FIGS. 20 and 21, the display device 500 does not include the image pickup unit 11 and the distance information acquisition unit 13, and the image pickup device 1 is the image pickup unit 11. The distance information acquisition unit 13 is provided, and these outputs are transmitted to the display device 500 or the server 600.

As shown in FIG. 20, the display device 500 may or may not include the determination unit 560.

When the display device 500 does not include the determination unit 560, the image pickup device 1 may include the determination unit 160 and transmit the determination result to the display device 500, and the server 600 includes the determination unit 660 as shown in FIG. The determination result may be transmitted to the display device 500.

Similarly, the display device 500 may or may not include the three-dimensional reconstruction processing unit 550 as shown in FIG.

When the display device 500 does not include the three-dimensional reconstruction processing unit 550, the imaging device 1 may include the three-dimensional reconstruction processing unit 150 and transmit a three-dimensional image to the display device 500, as shown in FIG. In addition, the server 600 may include a three-dimensional reconstruction processing unit 650 and transmit a three-dimensional image to the display device 500.

As described above, the image pickup apparatus 1 according to the embodiment of the present invention includes an image pickup unit 11 that images an object, a projection unit 12 that projects light onto the object, and distance information that receives light reflected from the object. A determination unit 160 that determines whether there is a distant object or a low-reflecting object based on both the acquisition unit 13 (an example of a light receiving unit), the output of the distance information acquisition unit 13, and the output of the imaging unit 11, and the distant object or A display control unit 170 that causes the display units 20 and 520 to display differently depending on the presence or absence of a low-reflecting object is provided.

This allows the photographer to accurately confirm that a distant object or a low-reflecting object such as black is included in the captured image.

The image pickup apparatus 1 includes a display unit 20. This allows the photographer to reliably confirm that a distant object or a low-reflection object is included in the captured image.

The display control unit 170 causes different displays at the positions of the display units 20 and 520 according to the positions of the distant object or the low reflection object. This allows the photographer to confirm the position of a distant object or a low reflection object.

The display unit 20 includes a plurality of display units 20A and 20a, and the display control unit 170 determines the presence or absence of an object on the display unit on the side of the plurality of display units 20A and 20a that is closer to the distant object or the low reflection object. To display differently. This allows the photographer to reliably confirm the position of a distant object or a low reflection object.

The display control unit 170 displays the image information G captured by the image pickup unit 11 on the display units 20 and 520, and displays a display image including identification information for identifying a distant object or a low reflection object and image information G. Displayed in units 20 and 520. This allows the photographer to reliably confirm the position of a distant object or a low reflection object.

When the storage amount of the pixel by the light received by the distance information acquisition unit 13 is equal to or less than the threshold value, the determination unit 160 determines whether the object is a low reflection object or a distant object based on the output of the image pickup unit 11. As a result, the photographer can accurately confirm whether the captured image includes a low-reflecting object or a distant object.

The determination unit 160 determines that there is a low-reflecting object when the amount of electricity stored in the pixels due to the light received by the distance information acquisition unit 13 is less than or equal to the threshold value and the amount of electricity stored in the pixels of the image pickup unit 11 is less than or equal to the threshold value. This allows the photographer to accurately confirm that the low-reflection object is included in the captured image.

In the determination unit 160, the amount of electricity stored in the pixels due to the light received by the distance information acquisition unit 13 is equal to or less than the threshold value, the amount of electricity stored in the pixels of the imaging unit 11 is equal to or greater than the threshold value, and the distance determined based on the pixels is equal to or greater than the threshold value. If there is, it is judged that there is a distant object.

This allows the photographer to accurately confirm that a distant object is included in the captured image.

The image pickup apparatus 1 acquires the distance information to the target based on the light received by the distance information acquisition unit 13. In this case, the photographer can confirm that the factor that the desired distance information is not acquired is a distant object or a low reflection object.

The image pickup apparatus 1 includes a transmission / reception unit 180, which is an example of an output unit that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition unit 13. In this case, the photographer can confirm that the factor for not acquiring the desired three-dimensional information is a distant object or a low-reflection object.

The imaging processing method according to the embodiment of the present invention includes an imaging step of capturing an image of an object by an imaging unit 11, a projection step of projecting light onto an object by a projection unit 12, and a distance information acquisition unit that acquires light reflected from the object. A light receiving step of receiving light from the 13 and a judgment step of determining whether there is a distant object or a low-reflecting object based on both the output of the distance information acquisition unit 13 and the output of the imaging unit 11 by the determination units 160, 560, and 660. A display step is provided in which display control units 170 and 530 cause display units 20 and 520 to display differently depending on the presence or absence of a distant object or a low reflection object.

The image pickup device 1 and the display device 500, which are examples of the information processing device according to the embodiment of the present invention, receive the output of the image pickup unit 11 that images the target and the light reflected from the target by projecting the light onto the target. Based on the judgment results by the judgment units 160, 560, and 660 that determine whether there is a distant object or a low-reflecting object based on both the output of the distance information acquisition unit 13 and the presence or absence of the distant object or the low-reflecting object. The display control units 170 and 530 are provided so that the display units 20 and 520 display different displays.

As described above, the image pickup apparatus 1 according to the embodiment of the present invention includes an image pickup unit 11 that images an object, a projection unit 12 that projects light onto the object, and distance information that receives light reflected from the object. The acquisition unit 13 (an example of the light receiving unit), the determination unit 160 that determines whether or not there is image blur based on both the output of the distance information acquisition unit 13 and the output of the image pickup unit 11, and the presence or absence of image blur. A display control unit 170 that causes the display units 20 and 520 to display different displays is provided accordingly.

This allows the photographer to accurately confirm that the image blur is included in the captured image.

The image pickup apparatus 1 includes a display unit 20. As a result, the photographer can surely confirm that the blurred image is included in the captured image.

The display control unit 170 causes different displays at the positions of the display units 20 and 520 according to the position of the blur of the image. This allows the photographer to confirm the position of the blur in the image.

The display unit 20 includes a plurality of display units 20A and 20a, and the display control unit 170 displays the display unit 20A and 20a on the side of the plurality of display units 20A and 20a that is closer to the blurring position, depending on the presence or absence of an object. Make a different display. As a result, the photographer can surely confirm the position of the blur of the image.

The display control unit 170 displays the image information G captured by the image pickup unit 11 on the display units 20 and 520, and displays a display image including the identification information for identifying the blur of the image and the image information G. Display on 520. As a result, the photographer can surely confirm the position of the blur of the image.

The determination unit 160 detects the edge of the image based on the image information captured by the image pickup unit 11, and determines that the image is blurred when the pixels due to the light received by the distance information acquisition unit 13 are displaced.

This allows the photographer to accurately confirm that the image blur is included in the captured image.

The image pickup apparatus 1 acquires the distance information to the target based on the light received by the distance information acquisition unit 13. In this case, the photographer can confirm that the factor that the desired distance information is not acquired is the blurring of the image.

The image pickup apparatus 1 includes a transmission / reception unit 180, which is an example of an output unit that outputs three-dimensional information determined based on the distance information acquired from the distance information acquisition unit 13. In this case, the photographer can confirm that the factor that the desired three-dimensional information is not acquired is the blurring of the image.

The imaging processing method according to the embodiment of the present invention includes an imaging step of capturing an image of an object by an imaging unit 11, a projection step of projecting light onto an object by a projection unit 12, and a distance information acquisition unit that acquires light reflected from the object. A determination step of determining whether or not there is image blur based on both the light receiving step of receiving light from the image 13 and the output of the distance information acquisition unit 13 and the output of the imaging unit 11 and the determination step of determining whether the image is blurred by the determination units 160, 560, and 660. A display step is provided in which display control units 170 and 530 cause display units 20 and 520 to display different displays depending on the presence or absence of blurring.

The image pickup device 1 and the display device 500, which are examples of the information processing device according to the embodiment of the present invention, receive the output of the image pickup unit 11 that images the target and the light reflected from the target by projecting the light on the target. Based on the judgment results of the judgment units 160, 560, and 660 that determine whether or not there is image blur based on both the output of the distance information acquisition unit 13 and the output of the distance information acquisition unit 13, the display units 20 and 520 are displayed according to the presence or absence of image blur. A display control unit 170, 530 for displaying different displays is provided.

The image pickup device 1 and the display device 500, which are examples of the information processing device according to the embodiment of the present invention, are distance information acquisition units 13 which are examples of light receiving units that receive light reflected from the target and projected onto the target. The display control units 170 and 530 are provided to display the three-dimensional image 3G determined based on the output on the display units 20 and 520, and the display control units 170 and 530 emit the light reflected from the object in the three-dimensional image 3G. A distant object that is far from the distance information acquisition unit 13 when it receives light, a low-reflecting object that has a low reflectance to the projected light, and a blind spot with respect to the distance information acquisition unit 13 when the light reflected from the object is received. Position identification information 3Ga, 3Gb, 3Gc, and three-dimensional image 3G that identify at least one position of a distant object, a low-reflecting object, and a blind spot based on the position information indicating the position determined to be at least one. The display image including the display image is displayed on the display units 20 and 520.

As a result, it is possible to confirm by observing the 3D image 3G whether the factor that the desired 3D image 3G is not displayed is a distant object, a low reflection object, or a blind spot. It becomes possible to take measures such as imaging.

The three-dimensional image 3G is determined by the three-dimensional reconstruction processing units 150, 550, and 650, which are examples of the three-dimensional information determination unit.

The display control unit 170, 530 displays a display including any one of the position identification information 3Ga, 3Gb, and 3Gc and the three-dimensional image 3G based on the position information of any one of the distant object, the low-reflecting object, and the blind spot. The image may be displayed on the display units 20, 520, and based on the position information of any two or all of the distant object, the low-reflecting object, and the blind spot, any two of the position identification information 3Ga, 3Gb, and 3Gc or A display image including all and the three-dimensional image 3G may be displayed on the display units 20 and 520.

When the information processing device is the image pickup device 1, the image pickup device 1 includes a distance information acquisition unit 13 and a three-dimensional reconstruction processing unit 150, as shown in FIG.

When the information processing device is the display device 500, as shown in FIGS. 20 and 21, the display device 500 does not include the distance information acquisition unit 13, and the image pickup device 1 includes the distance information acquisition unit 13. The output of the distance information acquisition unit 13 is transmitted to the display device 500 or the server 600.

The display device 500 may or may not include the three-dimensional reconstruction processing unit 550, and when the display device 500 does not include the three-dimensional reconstruction processing unit 550, the imaging device 1 includes the three-dimensional reconstruction processing unit 150. The 3D image may be transmitted to the display device 500, or as shown in FIG. 21, the server 600 may include the 3D reconstruction processing unit 650 and transmit the 3D image to the display device 500. ..

The display control units 170 and 530 indicate positions where the density of the point cloud data included in the three-dimensional image 3G is lower than the threshold value and is determined to be at least one of a distant object, a low reflection object, or a blind spot. A display image including the position identification information 3Ga, 3Gb, 3Gc, and the three-dimensional image 3G is displayed on the display units 20 and 520 based on the position information.

Thereby, it is possible to confirm by observing the three-dimensional image 3G whether the factor that the density of the point cloud data is lower than the threshold value is a distant object, a low reflection object, or a blind spot.

The display control units 170 and 530 are positions indicating a position determined to be at least one of a distant object, a low reflection object, or a blind spot in the three-dimensional image 3G based on the output of the image pickup unit 11 that images the object. Based on the information, a display image including the position identification information 3Ga, 3Gb, 3Gc, and the three-dimensional image 3G is displayed on the display units 20 and 520.

Thereby, it is possible to accurately determine whether the factor that the desired three-dimensional image 3G is not displayed is a distant object, a low-reflecting object, or a blind spot based on the output of the imaging unit 11.

When the information processing device is the image pickup device 1, the image pickup device 1 includes an image pickup unit 11 as shown in FIG. When the information processing device is the display device 500, the display device 500 does not include the image pickup unit 11 as shown in FIGS. 20 and 21, and the image pickup device 1 includes the image pickup unit 11 and includes the image pickup unit 11. The output of 11 is transmitted to the display device 500 or the server 600.

The image pickup device 1 and the display device 500 include determination units 160 and 560 for determining a position which is at least one of a distant object, a low reflection object, or a blind spot in a three-dimensional image 3G, and display control units 170 and 530. Based on the determination results of the determination units 160 and 560, the display units 20 and 520 display the display image including the position identification information 3Ga, 3Gb, 3Gc and the three-dimensional image 3G.

When the information processing device is the image pickup device 1, the image pickup device 1 includes a determination unit 160 as shown in FIG.

When the information processing device is the display device 500, the display device 500 may or may not include the determination unit 560 as shown in FIG. 20.

When the display device 500 does not include the determination unit 560, the image pickup device 1 may include the determination unit 160 and transmit the determination result to the display device 500, and the server 600 includes the determination unit 660 as shown in FIG. The determination result may be transmitted to the display device 500.

The display control units 170 and 530 change the display area of the 3D image 3G to be displayed on the display units 20 and 520 by changing the position and orientation of the virtual camera IC at the position of the viewpoint for viewing the 3D image 3G. ..

The image pickup device 1 and the display device 500, which are examples of the information processing device according to the embodiment of the present invention, are the distance information acquisition section 13 which is an example of the light receiving section that receives the light projected from the target and reflected from the target. The display control units 170 and 530 are provided to display the three-dimensional image 3G determined based on the output on the display units 20 and 520, and the display control units 170 and 530 acquire distance information when the light reflected from the target is received. A display image including position identification information 3G1, 3G2, and a three-dimensional image 3G that identify the position of the distance information acquisition unit 13 when the light reflected from the target is received based on the position information indicating the position of the unit 13. Is displayed on the display units 20 and 520.

As a result, the positional relationship between the imaging position indicating the position of the distance information acquisition unit 13 when the light reflected from the object is received and the specific object can be grasped in the three-dimensional image 3G. That is, it is possible to easily compare the positional relationship between the imaging position and the specific target in the field where the three-dimensional image is acquired and the positional relationship between the imaging position and the specific object in the three-dimensional image.

The 3D image 3G and the position information are determined by the 3D reconstruction processing units 150, 550, and 650, which are examples of the 3D information determination unit.

When the information processing device is the image pickup device 1, the image pickup device 1 includes a distance information acquisition unit 13 and a three-dimensional reconstruction processing unit 150.

When the information processing device is the display device 500, the display device 500 does not include the distance information acquisition unit 13, and the image pickup device 1 includes the distance information acquisition unit 13 to display the output of the distance information acquisition unit 13. Send to 500 or server 600.

The display device 500 may or may not include the three-dimensional reconstruction processing unit 550, and when the display device 500 does not include the three-dimensional reconstruction processing unit 550, the imaging device 1 includes the three-dimensional reconstruction processing unit 150. The 3D image and the position information may be transmitted to the display device 500, or the server 600 may include the 3D reconstruction processing unit 650 and transmit the 3D image and the position information to the display device 500.

The display control units 170 and 530 are identification information 3Ga, 3Gb, 3Gc, which is an example of low-density identification information for identifying an area based on area information indicating an area in which the density of point cloud data in the three-dimensional image 3G is lower than the threshold value. A display image including the three-dimensional image 3G and the three-dimensional image 3G is displayed on the display units 20 and 520.

In this case, since the positional relationship between the imaging position and the region where the density of the point cloud data is lower than the threshold value can be grasped, it is possible to identify the factor in which the density of the point cloud data is lower than the threshold value. For example, if the area is far from the imaging position, a distant object is a factor, if the area is in the blind spot of the imaging position, a blind spot is a factor, and if it is neither a distant object nor a blind spot, a low-reflecting object is a factor. It can be identified that there is.

The display control units 170 and 530 change the display area of the 3D image 3G to be displayed on the display units 20 and 520 by changing the position and orientation of the virtual camera IC at the position of the viewpoint for viewing the 3D image 3G. ..

When the position of the virtual camera IC is at the positions 3G1 and 3G2 identified by the position identification information, the display control units 170 and 530 change the orientation of the virtual camera IC to a predetermined orientation.

This makes it possible to direct the line of sight of the user located at the imaging position to a specific object to be seen in the field.

The display control units 170 and 530 change the orientation of the virtual camera IC so that the display area includes the predetermined coordinates and the low density part where the density of the point cloud data in the three-dimensional image 3G is lower than the threshold value. ..

This makes it possible to direct the line of sight of the user located at the imaging position to a specific object having predetermined coordinates or a low density portion in the three-dimensional image 3G.

The display control units 170 and 530 are determined based on the output of the distance information acquisition unit 13 located at the first position and the output of the distance information acquisition unit 13 located at a second position different from the first position. The first position identification information 3G1 for displaying the three-dimensional image 3G on the display units 20 and 520 and identifying the first position, the second position identification information 3G2 for identifying the second position, and the three-dimensional image 3G, The display image including the above is displayed on the display units 20 and 520.

Thereby, the positional relationship between the first imaging position and the second imaging position and the specific target can be grasped in the three-dimensional image 3G.

1 Imaging device (an example of information processing device)
3G 3D image 3Ga, 3Gb, 3Gc Identification information 3G1, 3G2 Position identification information 10 Housing 11 Image sensor 11a, 11A Image sensor 11b, 11B Fisheye lens 12 Projection unit 12a, 12A Light source unit 12b, 12B Wide-angle lens 13 Distance information acquisition unit (Example of light receiving part)
13a, 13A TOF sensor 13b, 13B Wide-angle lens 14 Processing circuit 15 Shooting switch 20 Display unit 20A, 20a Display unit 111 Other imaging unit 150, 550, 650 3D reconstruction processing unit (example of 3D information determination unit)
160, 560, 660 Judgment unit 170 Display control unit (example of output unit)
180 Transmitter / receiver (example of output)
300 External device (example of output destination)
500 Display device (output destination, example of information processing device)
520 display unit (example of output destination)
530 Display control unit (example of output unit)
600 server L sync signal line

Claims (10)

An imaging unit that captures an object and
A projection unit that projects light onto the target,
A light receiving unit that receives the light reflected from the target, and a light receiving unit.
A determination unit that determines whether or not there is image blur based on both the output of the light receiving unit and the output of the image pickup unit.
A display control unit that causes the display unit to display different images depending on the presence or absence of blurring of the image.
Imaging device equipped with. The imaging device according to claim 1, further comprising the display unit. The imaging device according to claim 1 or 2, wherein the display control unit causes the different display to be displayed at the position of the display unit according to the position of the blur of the image. The display unit includes a plurality of display units.
The display control unit is any one of claims 1 to 3, wherein the display unit on the side closer to the position of the blur of the image among the plurality of display units displays differently depending on the presence or absence of the blur of the image. The imaging apparatus described. The imaging device according to any one of claims 1 to 4, wherein the display control unit displays a display image including image information captured by the imaging unit and identification information for identifying blurring of the image on the display unit. .. The determination unit detects an edge of an image based on the image information captured by the image pickup unit, and determines that the image is blurred when the pixels due to the light received by the light receiving unit are displaced. The imaging device according to any one of claims 1 to 5. The imaging device according to any one of claims 1 to 6, wherein the distance information to the target is acquired based on the light received by the light receiving unit. The imaging device according to claim 7, further comprising an output unit that outputs three-dimensional information determined based on the distance information acquired from the light receiving unit. An imaging step in which an object is imaged by an imaging unit,
A projection step that projects light onto the target,
A light receiving step in which the light reflected from the target is received by the light receiving unit, and
A determination step for determining whether or not there is image blur based on both the output of the light receiving unit and the output of the imaging unit, and
An imaging processing method comprising a display step of displaying different images depending on the presence or absence of blurring of the image. Based on both the output of the image pickup unit that images the target and the output of the light receiving unit that projects the light onto the target and receives the light reflected from the target, the judgment result is determined as to whether or not there is image blur. Based on this, an information processing device including a display control unit that causes the display unit to display differently depending on the presence or absence of blurring of the image.

Images