JP7414090B2 - Imaging device and method of controlling the imaging device - Google Patents

Description

The present invention relates to an imaging device and a method of controlling the imaging device.

3D reconstruction model technology that measures real objects, buildings, and the entire space that contains them using camera imaging and laser light irradiation, and restores 3D information as electronic data based on the measurement results. It has been known. In recent years, the use of this three-dimensional reconstruction model has been considered in a variety of situations, such as bridge inspections and construction sites. For example, in the construction of a large-scale facility such as a building, a comparison may be made between CAD (Computer-Aided Design) drawings at the time of design and a three-dimensional reconstruction model created by measuring at the construction site. Another possible application is to create a three-dimensional reconstruction model and check the situation during post-construction inspections.

Depth information is obtained by acquiring multiple captured images from different viewpoints using the imaging system disclosed in Patent Document 1 that is capable of capturing the entire surrounding area as a spherical image, and by comparing the acquired multiple captured images. It is possible to create a three-dimensional reconstruction model for a wider range of areas. However, with the technology described in Patent Document 1, it is difficult to compare captured images for areas with poor image characteristics and textures, such as monochrome walls, and it is difficult to obtain depth information with high accuracy. It was difficult.

On the other hand, Patent Document 2 discloses a technique of irradiating a space with a known pattern of light and acquiring depth information through stereo matching based on the irradiated pattern and a pattern imaged by a stereo camera. .

However, in Patent Document 2, the purpose is to acquire only depth information, and it is not considered to overlap and simultaneously acquire color information of an object with depth information. Furthermore, in Patent Document 2, since matching is performed based on a known pattern and an imaged pattern, a pattern light irradiation device, an imaging device, and a calculation device for performing matching processing need to work together. However, there were also problems in that the arrangement and miniaturization of the device were restricted.

The present invention has been made in view of the above, and an object of the present invention is to enable creation of a more accurate three-dimensional reconstruction model with a simpler configuration.

In order to solve the above-mentioned problems and achieve the objects, the present invention provides a plurality of first optical systems facing in mutually different directions, and a plurality of visible light irradiated through each of the plurality of first optical systems. A plurality of first image sensors that capture images, a plurality of second optical systems facing different directions from each other, and a plurality of second optical systems that capture images of invisible light irradiated through each of the plurality of second optical systems. 2 image sensors, a plurality of third optical systems facing in different directions from each other, and a plurality of third image sensors that respectively image the invisible light irradiated through each of the plurality of third optical systems; The plurality of first optical systems, the plurality of first image sensors, the plurality of second optical systems, the plurality of second image sensors, the plurality of third optical systems, and the plurality of third image sensors a casing provided therein, a first image including color information is calculated by the first image sensor, and a second image and a third image are used to calculate depth information by the second image sensor and the third image sensor. The casing has an elongated shape in a first direction and has a bottom surface on one end side of the elongated shape, and the plurality of first optical systems, the plurality of second optical systems, and the plurality of Among the third optical systems, the second optical system is closest to the bottom surface, and there is an area between the second optical system and the bottom surface where no optical system is disposed, and the second optical system is located on the floor surface of the space to be imaged. The first direction is oriented in a direction perpendicular to the first optical system, and the plurality of first optical systems, the plurality of second optical systems, and the plurality of third optical systems are arranged above an area where the optical systems are not arranged. The image is taken with the bottom surface facing the floor surface , and the plurality of first optical systems, the plurality of second optical systems, and the plurality of third optical systems are arranged at different positions in the first direction, It is possible to image a 360° circumference of the housing in the first direction.

According to the present invention, it is possible to create a more accurate three-dimensional reconstructed model with a simpler configuration.

FIG. 1 is a diagram for schematically explaining a method for creating a three-dimensional reconstructed model according to each embodiment. FIG. 2 is a diagram illustrating an example of a case where an imaging device capable of capturing omnidirectional images is used, which is applicable to each embodiment. FIG. 3 is a diagram showing the appearance of an example of the imaging device according to the first embodiment. FIG. 4 is a diagram schematically showing the internal structure of the imaging device according to the first embodiment. FIG. 5 is a diagram showing the structure of an example of an imaging body applicable to the first embodiment. FIG. 6 is a three-sided view schematically showing the appearance of the imaging device according to the first embodiment. FIG. 7 is a diagram illustrating an example of an imaging range that can be imaged by each imaging body according to the first embodiment. FIG. 8 is a diagram showing a connection between an image processing device and an imaging device, which is applicable to the first embodiment. FIG. 9 is a block diagram showing the configuration of an example of an imaging device according to the first embodiment. FIG. 10 is a block diagram illustrating an example configuration of a control unit and a memory in the imaging device according to the first embodiment. FIG. 11 is a block diagram showing the configuration of an example of an image processing device applicable to the first embodiment. FIG. 12 is an example functional block diagram for explaining the functions of the image processing apparatus according to the first embodiment. FIG. 13 is a flowchart of an example of a three-dimensional reconstruction model creation process applicable to the first embodiment. FIG. 14 is a diagram illustrating the captured image search method according to the first embodiment in comparison with a general search method. FIG. 15 is a diagram for explaining trigonometry applicable to the first embodiment. FIG. 16 is a diagram for explaining reprojection processing according to the first embodiment. FIG. 17 is a diagram schematically showing the internal structure of an imaging device according to the second embodiment. FIG. 18 is a diagram illustrating an example of pixel search processing using a plurality of stereo pairs, which is applicable to the second embodiment.

Embodiments of an imaging device and a method of controlling the imaging device will be described in detail below with reference to the accompanying drawings.

FIG. 1 is a diagram for schematically explaining a method for creating a three-dimensional reconstructed model according to each embodiment. The imaging device 50a includes one or more first imaging bodies capable of capturing visible light in the visible light region and non-visible light region, and at least non-visible light in the visible light region and non-visible light region. Two or more second imaging bodies capable of imaging light in the optical region are included. Here, for the sake of explanation, it is assumed that the imaging device 50a includes one first imaging body and two second imaging bodies.

A predetermined pattern 81 is projected onto objects 60, 60, . . . by a light source 51 that emits light 80 in an invisible light region. The pattern 81 is imaged by two second imaging bodies included in the imaging device 50a. Matching processing is performed based on the pattern 81 of the light 80 in the invisible light region captured by each of the second imaging bodies of 2, depth information is obtained based on the matching processing result, and three-dimensional point cloud information ( Hereinafter, three-dimensional point cloud information) is generated.

Further, in synchronization with the imaging by the above-described second imaging body, the objects 60, 60, . . . are imaged by one first imaging body included in the imaging device 50a. The captured image captured by the first imaging body is an image including color information in the visible light region. Further, the first imaging body is configured to prevent light in the non-visible light region from being imaged using a filter or the like, and the captured image captured by the first imaging body is a pattern of light in the non-visible light region. The image does not include 81.

Color information is added to the 3D point cloud information by reprojecting the 3D point cloud information generated based on the captured image captured by the second imaging body onto the captured image captured by the first imaging body. A three-dimensional reconstructed model is generated.

In this way, in the embodiment, depth information is obtained based on a captured image of the pattern 81 formed by light in the invisible light region, three-dimensional point cloud information is generated, and the generated three-dimensional point cloud information is combined with the captured image. The image is reprojected using light in the visible light range that was captured synchronously. Therefore, a more accurate three-dimensional reconstruction model can be created with a simpler configuration.

In FIG. 1, the imaging device 50a is shown to capture an image in one direction (the front side) at an angle of view 70 of less than 180°, but this is not limited to this example; An imaging device capable of capturing a spherical image can also be used. FIG. 2 is a diagram illustrating an example of a case where an imaging device 50b capable of capturing omnidirectional images is used, which is applicable to each embodiment. Like the imaging device 50a described above, the imaging device 50b includes one or more first imaging bodies that capture light in the visible light region of the visible light region and the non-visible light region; and two or more imaging bodies that capture images of at least light in the non-visible light region. Here, each image pickup body has two fisheye lenses each having an angle of view of 180° or more, and is arranged so that the directions of the optical axes are the same and images are taken in opposite directions, so that a spherical image can be taken. configured as possible.

Note that, also here, for the sake of explanation, it is assumed that the imaging device 50b includes one first imaging body and two second imaging bodies.

In FIG. 2, a plane 40 is a plane represented by an X-axis and a Y-axis perpendicular to the Z-axis, when the vertical direction is represented by the Z-axis. Examples of the plane 40 include a horizontal piece of land or a floor surface. For example, the imaging device 50b is placed at the Z-axis position on the plane 40, and a plurality of light sources 51, 51, . The patterns 81, 81, . . . are arranged on the plane 40 so that they can be irradiated. Further, when a subject exists in the positive direction of the Z-axis (above the imaging device 50b), such as indoors, the light source 51 is arranged so that the pattern 81 is also irradiated in the positive direction of the Z-axis.

In the imaging device 50b, the patterns 81, 81, . . . generated by the respective light sources 51, 51, . The imaging by the second imaging body and the imaging by the second imaging body are executed in synchronization. As a result, one spherical image captured using light in the visible light range and two spherical images captured including light in the non-visible light range are obtained. Matching processing is performed based on patterns 81, 81, ... included in the spherical image by light 80 in the invisible light region captured by each of the second imaging bodies of 2, and depth information is obtained based on the matching processing result. 3D point cloud information is generated.

Furthermore, the spherical image captured by the first imaging body is an image that includes color information in the visible light region. By reprojecting the 3D point cloud information generated based on the spherical image captured by the second imaging body onto the spherical image captured by the first imaging body, the 3D point cloud information is converted into 3D point cloud information. Color information is added and a three-dimensional reconstructed model is generated. In this case, since a three-dimensional reconstructed model can be generated by one-time imaging for 360 degrees around the imaging device 50b, the three-dimensional reconstructed model can be generated more efficiently.

Note that it is preferable that the pattern 81 formed by the light 80 in the non-visible light region emitted by the light source 51 be a pattern with few repeating elements (for example, a random pattern) because this facilitates the matching process.

In the following, a case will be described in which the imaging device 50b shown in FIG. 2, which is capable of capturing a spherical image, is used as the imaging device.

[First embodiment]
A first embodiment will be described. FIG. 3 is a diagram showing the appearance of an example of the imaging device according to the first embodiment. In FIG. 3, the imaging device 1 corresponds to the imaging device 50b shown in FIG. 2, and has a plurality of (three in this example) imaging lenses _20IRa1 , 20VL _a and 20IR _a2 and a shutter button 30 are provided. In the housing 10, image sensors are provided corresponding to each of 20IR _a1 , 20VL _a and 20IR _a2 .

The light incident on each of the imaging lenses 20IR _a1 , 20VL _a and 20IR _a2 is transmitted to the corresponding imaging element via an imaging optical system including the imaging lenses 20IR _a1 , 20VL _a and 20IR _a2 provided in the housing 10, respectively. is irradiated. The image sensor is, for example, a CCD (Charge Coupled Device), and is a light receiving element that converts irradiated light into electric charge. The image sensor is not limited to this, and may be a CMOS (Complementary Metal Oxide Semiconductor) image sensor.

Further, a plurality of imaging lenses 20IR b1 , 20VL are provided on a second surface of the housing 10 _on the back side of the first surface, corresponding to the respective imaging lenses 20IR _a1 , 20VL _a and 20IR _a2 on the first surface. _b and 20IR _b2 are provided. These imaging lenses 20IR _b1 , 20VL _b and 20IR _b2 are also provided with corresponding imaging elements in the housing 10, similarly to the above-mentioned imaging lenses 20IR _a1 , 20VL _a and 20IR _a2 . The light incident on each imaging lens 20IR _b1 , 20VL _b and 20IR _b2 is irradiated onto the corresponding imaging element via each imaging optical system including each imaging lens 20IR _b1 , 20VL _b and 20IR _b2 . Each image sensor converts the irradiated light into electric charge.

Although the details will be described later, each drive unit that drives each image sensor performs shutter control on each image sensor according to a trigger signal, and reads out the charge from which light is converted from each image sensor. Each drive section converts the charge read from each image sensor into an electrical signal, converts the electrical signal into a captured image as digital data, and outputs the captured image. Each captured image output from each drive unit is stored in a memory, for example. Hereinafter, for convenience, the operation of outputting a captured image based on light incident on the imaging lens 20IR _a1 in response to a trigger signal will be described as "imaging by the imaging lens 20IR _a1 ," for example.

Note that when the housing 10 is regarded as a rectangular parallelepiped, the surfaces in contact with the sides (long sides) along the direction in which the imaging lenses 20IR _a1 , 20VL _a and 20IR _a2 are aligned are referred to as side surfaces, first surfaces, second surfaces, and The surface touching the upper end of each of the two side surfaces is the upper surface, and the surface touching the lower end is the bottom surface. In the example of FIG. 3, the surface in contact with the upper end on the side where the imaging lenses 20IR _a1 and 20IR _b1 are arranged is the top surface, and the surface in contact with the lower end on the side where the shutter button 30 is arranged is the bottom surface.

Further, in the example of FIG. 3, the imaging lenses 20IR _a1 , _{20VL a} and 20IR _a2 are arranged with the distance between them being the distance d _. On the other hand, the imaging lens VL _a is arranged with distances dv ₁ and dv ₂ from the imaging lenses 20IR _a1 and 20IR _a2 , respectively. The distance dv ₁ and the distance dv ₂ are preferably equal values, but are not limited to this. Furthermore, the imaging lenses 20IR _a1 and 20IR _b1 , the imaging lenses 20VL _a and 20VL _b , and the imaging lenses 20IR _a2 and 20IR _b2 are arranged in the housing 10 such that their heights from, for example, the bottom of the housing 10 are the same. be done.

In each imaging lens 20IR _a1 , 20VL _a and 20IR _a2 and each imaging lens 20IR _b1 , 20VL _b and 20IR _b2 , there is a set of imaging lenses whose heights from the bottom of the housing 10 match, and a set of the imaging lens. Each imaging optical system and each image sensor corresponding to the image forming apparatus are collectively referred to as an imaging body. In the example of FIG. 3, the set of imaging lenses _20IRa1 and _20IRb1 , including each corresponding imaging optical system and each imaging element, is set as an imaging body _21IR1 . Similarly, the imaging bodies 21VL and 21IR ₂ including the corresponding imaging optical systems and imaging elements are respectively referred to as imaging bodies 21VL and 21IR ₂ .

The shutter button 30 is a button for instructing imaging by each of the imaging lenses 20IR _a1 , 20VL _a and 20IR _a2 and each of the imaging lenses 20IR _b1 , 20VL _b and 20IR _b2 in response to an operation. When the shutter button 30 is operated, imaging by each imaging lens 20IR _b1 , 20VL _b and 20IR _b2 and each imaging lens 20IR _b1 , 20VL _b and 20IR _b2 is performed synchronously.

Note that, hereinafter, for example, "imaging is performed by the imaging lenses 20IR _a1 and 20IR _b1 in the imaging body 21IR ₁ " will be appropriately described as "imaging is performed by the imaging body 21R ₁ ".

FIG. 4 is a diagram schematically showing the internal structure of the imaging device 1 according to the first embodiment. In FIG. 4, the casing 10 of the imaging device 1 is shown as a cross-sectional view from the side, with the left side being the first surface (the front direction) and the right side being the second surface (the back direction). The imaging body 21IR ₁ includes an imaging lens 20IR _a1 on the first surface side including the imaging element 200a ₁ and an imaging lens 20IR _b1 on the second surface side including the imaging element 200b ₁ .

In FIG. 4, the image sensors 200 _a1 and 200 _b1 are arranged with their light-receiving surfaces facing the front direction and the back direction, respectively, but in reality, the image sensors 200 _a1 and 200 _b1 are are arranged in the vertical direction of the casing 10 with their light-receiving surfaces facing inward. The light incident from the front direction and the back direction is respectively converted in direction by a prism or the like, and then enters the image pickup elements 200 _a1 and 200 _b1 . Details regarding this configuration will be described later.

The imaging bodies 21VL and _21IR2 are also configured similarly to the imaging body _21IR1 . That is, the imaging body 21VL includes an imaging lens 20VL _a on the first surface side that includes the imaging device 200a ₂ and an imaging lens 20VL _b on the second surface side that includes the imaging device 200b ₂ . Further, the imaging body 21IR ₂ includes an imaging lens 20IR _a2 on the first surface side including the imaging element 200a ₃ and an imaging lens 20IR _b2 on the second surface side including the imaging element 200b ₃ .

Furthermore, in the imaging device 1 according to the first embodiment, the imaging bodies 21IR ₁ and 21IR ₂ are configured to be able to image at least light in the non-visible light region of light in the visible light region and light in the non-visible light region. has been done. On the other hand, the imaging body 21VL is configured to be able to selectively image light in the visible light region out of light in the visible light region and light in the non-visible light region, and not to image light in the non-visible light region. There is.

For example, in the imaging bodies 21IR ₁ and 21IR ₂ , filters 22IR that selectively transmit light in the non-visible light region are provided in the imaging lenses 20IR _a1 and 20IR _b1 and the imaging lenses 20IR _a2 and 20IR _b2 , respectively. For example, in the imaging body 21IR ₁ , the light incident on the imaging lenses 20IR _a1 and 20IR _b1 is selectively transmitted by each filter 22IR in the non-visible light region, and is irradiated onto the imaging elements 200a ₁ and 200b ₁ . Ru. This also applies to the imaging body _21IR2 .

Further, in the imaging body 21VL, filters 22VL that selectively transmit light in the visible light range are provided on the imaging lenses 20VL _a and 20VL _b , respectively. The light incident on the imaging lenses 20VL _a and VL _b is selectively transmitted in the visible light range by each filter 22VL, and is irradiated onto the imaging elements 200a ₂ and 200b ₂ . Therefore, the images captured by the image sensors 200a ₂ and 200b ₂ have color information.

In the first embodiment, it is assumed that the non-visible light region is an infrared light region. Here, the infrared light region is defined as a region with a wavelength of 780 nm or more, and the filter 22IR has a characteristic of cutting off light with a wavelength of 780 nm or less, for example. Further, the filter 22VL has a characteristic of cutting light having a wavelength of 780 nm or more.

However, the present invention is not limited to this, and the imaging lenses 20IR _a1 and 20IR _b1 and the imaging lenses 20IR _a2 and 20IR _b2 may be configured to be able to transmit both light in the non-visible light region and visible light region without providing the filter 22IR. That is, the imaging bodies 21IR ₁ and 21IR ₂ can capture at least the light in the non-visible light region and the light in the visible light region, and the imaging body 21VL can capture the light in the non-visible light region and the visible light region. Any configuration may be used as long as it is possible to selectively image light in the visible light region and selectively disable image capturing of light in the non-visible light region.

Furthermore, in the case of the imaging lens 20IR _a1 , for example, the filter 22IR may be provided for a lens included in the imaging lens 20IR _a1 , or may be provided for the imaging element 200a ₁ . This also applies to the filter 22VL. Furthermore, without providing the filters 22IR and 22VL, the imaging bodies 21IR ₁ , 21VL and 21IR ₂ can image light in the non-visible light region and the visible light region, respectively, and in subsequent image processing, the images captured by the imaging body 21VL. It is also conceivable to cut off the non-visible light region components of the captured image data.

In FIG. 4, a processing circuit board 31 is equipped with a drive control circuit for driving each image sensor 200a ₁ to a ₃ and 200b ₁ to 200b ₃ . A drive control circuit mounted on the processing circuit board 31 generates a trigger signal for synchronously executing imaging by each of the image sensors 200a ₁ to a ₃ and 200b ₁ to 200b ₃ in response to the user's operation of the shutter button 30. A synchronous imaging instruction signal 32 is output.

FIG. 5 is a diagram showing an example of the structure of the imaging bodies 21IR ₁ , 21VL, and 21IR ₂ applicable to the first embodiment. Note that, in the following, when there is no need to distinguish between the imaging bodies 21IR ₁ , 21VL, and 21IR ₂ , these imaging bodies 21IR ₁ , 21VL, and 21IR ₂ will be collectively described as the imaging body 21.

In FIG. 5, the imaging body 21 includes imaging optical systems 201a and 201b including imaging lenses 20a and 20b, respectively, and imaging elements 200a and 200b using CCD or CMOS sensors. Each of the imaging optical systems 201a and 201b is configured, for example, as a fisheye lens with seven lenses in six groups. This fisheye lens has a total angle of view larger than 180° (=360 degrees/n, where n is the number of optical systems=2), preferably has an angle of view of 185° or more, and more preferably 190°. The angle of view is as follows.

Each imaging optical system 201a and 201b includes prisms 202a and 202b that change the optical path by 90°, respectively. The seven fisheye lenses in six groups included in each of the imaging optical systems 201a and 201b can be divided into a group on the entrance side of the prisms 202a and 202b, and a group on the exit side (on the imaging elements 200a and 200b side). For example, in the imaging optical system 201a, light incident on the imaging lens 20a is incident on the prism 202a via each lens belonging to a group on the incident side of the prism 202a. The light incident on the prism 202a has its optical path converted by 90 degrees and is irradiated onto the image sensor 200a through each lens, aperture stop, and filter belonging to the group on the exit side of the prism 202a.

The optical elements (lenses, prisms 202a and 202b, filters, and aperture stops) of the two imaging optical systems 201a and 201b have a positional relationship with respect to the imaging elements 200a and 200b. More specifically, the optical axes of the optical elements of the imaging optical systems 201a and 201b are positioned perpendicular to the centers of the light receiving areas of the corresponding image sensors 200a and 200b, and the light receiving areas are Positioning is performed so that the image forming plane of the fisheye lens becomes the image plane of the fisheye lens. Furthermore, in the imaging body 21, the imaging optical systems 201a and 201b have the same specifications and are combined in opposite directions so that their respective optical axes match.

FIG. 6 is a three-sided view schematically showing the appearance of the imaging device 1 according to the first embodiment. FIGS. 6A, 6B, and 6C are examples of a top view, a front view, and a side view of the imaging device 1, respectively. In FIG. 6B, the imaging lens 20IR _a2 is arranged such that the center of the lens is located at a height h from the bottom surface of the housing 10. An imaging lens 20IR _a1 is arranged above the imaging lens 20IR _a2 by a distance d, and an imaging lens 20VL _a is arranged at a distance dv ₂ from the imaging lens 20IR _a2 towards the upper surface. Each of the imaging lenses 20IR _a1 , 20VL, and 20IR _a2 is arranged with the lens center aligned with the center line C in the long side direction of the housing 10 .

As shown in FIG. 6(c), on the back side of each imaging lens _20IRa1 , _20VLa , and _20IRa2 , each imaging lens _20IRb1 , _20VLb , and _20IRb2 is provided _, respectively _. and 20IR _a2 . That is, these imaging lenses 20IR _b1 , 20VL _b and 20IR _b2 are also arranged with their lens centers aligned with the above-mentioned center line C. Furthermore, as described above, among the imaging lenses 20IR _a1 , 20VL _a and 20IR _a2 and the imaging lenses 20IR _b1 , 20VL _b and 20IR _b2 , two imaging lenses having the same height (for example, the imaging lens 20IR _a1 and 20IR _b1 ) are included in one imaging body (for example, imaging body 21IR ₁ ).

Note that in FIGS. 6(a) and 6(c), the angle α indicates an example of the angle of view (imaging range) of the imaging lenses 20IR _a1 and 20IR _b1 .

FIG. 7 is a diagram showing an example of an imaging range that can be imaged by each of the imaging bodies 21IR ₁ , 21VL, and 21IR ₂ according to the first embodiment. Each imaging body 21IR ₁ , 21VL and 21IR ₂ has a similar imaging range. In FIG. 7, the imaging ranges of the imaging bodies 21IR ₁ , 21VL, and 21IR ₂ are represented by the imaging range of the imaging body 21IR ₁ .

In the following, as shown in FIG. 7, for example, the direction in which the imaging lenses 20IR _a1 , 20VL _a and 20IR _a2 are aligned corresponds to the Z axis, and the direction of the optical axis of the imaging lenses 20IR _a1 and 20IR _b1 corresponds to the X axis. shall correspond.

The imaging body 21IR ₁ has an imaging range that covers the entire celestial sphere centered on the center of the imaging body 21IR ₁ by a combination of the imaging lenses 20IR _a1 and 20IR _b1 . That is, as described above, the imaging lenses 20IR _a1 and 20IR _b1 each have an angle of view of more than 180°, more preferably 185° or more. Therefore, by combining these imaging lenses 20IR _a1 and 20IR _b1 , the imaging range A on the XY plane and the imaging range B on the is realized.

In other words, the imaging body 21IR ₁ includes a range including a hemisphere centered on the optical axis of the imaging lens 20IR _a1 and oriented in the first direction of the optical axis, and an imaging lens that is common to the optical axis of the imaging lens 20IR _a1 . It is possible to image a range including a hemisphere centered on the optical axis of 20IR _b1 and directed in a second direction opposite to the first direction of the optical axis.

Furthermore, the imaging bodies 21IR ₁ and 21IR ₂ are arranged with a distance d apart in the Z-axis direction. Therefore, each captured image captured by the imaging bodies 21IR ₁ and 21IR ₂ with the entire celestial sphere as the imaging range becomes an image with a different viewpoint by the distance d in the Z-axis direction.

At this time, in the first embodiment, imaging in each imaging lens 20IR _a1 , 20VL _a and 20IR _a2 and each imaging lens 20IR _b1 , 20VL _b and 20IR _b2 is synchronized according to the operation of the shutter button 30. It will be done. Therefore, by using the imaging device 1 according to the first embodiment, the viewpoints differ by the distance d in the Z-axis direction for each of the first and second surfaces of the housing 10, and the light in the non-visible light region is It is possible to obtain two captured images in which light in the visible light region is captured and one captured image in which light in the visible light region is captured with the imaging position set between the imaging positions of the two captured images.

[3D restoration model creation process applicable to the first embodiment]
Next, a three-dimensional reconstruction model creation process applicable to the first embodiment will be schematically described. In the first embodiment, for example, as shown in FIG. 8, the image processing device 100 connected to the imaging device 1 executes a three-dimensional reconstruction model creation process. The image processing apparatus 100 can be configured by, for example, running an image processing program on a personal computer (PC). The image processing device 100 is not limited to this, and may be dedicated hardware for executing the three-dimensional restoration model creation process.

For example _, the image processing device 100 selects from the memory of the imaging device ₁ a plurality of images (in this example, a first surface and the second surface (6 images in total) are read out. The image processing device 100 converts two captured images of the six images read from the memory of the image capturing device 1 into the image capturing bodies 21IR ₁ , 21VL and 21IR ₂ into the image capturing bodies 21IR ₁ , 21VL and 21IR 2 . Each of 21IR ₂ is synthesized. As a result, two omnidirectional images captured including light in the non-visible light region, whose viewpoints differ by a distance d in the Z-axis direction, and approximately each of the two omnidirectional images, On the other hand, one spherical image that is taken selectively including light in the visible light region and whose viewpoints differ by distances dv ₁ and dv ₂ is created.

Next, the configuration regarding signal processing of the imaging device 1 and the image processing device 100 according to the first embodiment will be described. FIG. 9 is a block diagram showing the configuration of an example of the imaging device 1 according to the first embodiment. Note that in FIG. 9, parts corresponding to those in FIGS. 4 and 5 described above are designated by the same reference numerals, and detailed description thereof will be omitted.

In FIG. 9, the imaging device 1 includes image sensors 200a ₁ , 200a ₂ and 200a ₃ , image sensors 200b ₁ , 200b ₂ and 200b ₃ , drive units 210a ₁ , 210a ₂ and 210a ₃ , and drive unit 210b. ₁ , 210b ₂ and 210b ₃ , buffer memories 211a ₁ , 211a ₂ and 211a ₃ , and buffer memories 211b ₁ , 211b ₂ and 211b ₃ .

Among these, the imaging elements 200a ₁ , 200a ₂ and 200a ₃ , the drive units 210a ₁ , 210a ₂ and 210a ₃ , and the buffer memories 211a ₁ , 211a ₂ and 211a ₃ are the imaging lenses 20IR _a1 and 20VL _{a ,} respectively. and 20IR _a2 , and are included in the imaging bodies 21IR ₁ , 21VL and 21IR ₂ , respectively. In addition, in FIG. 9, in order to avoid complexity, only the imaging body 21IR ₁ is shown among the imaging bodies 21IR ₁ , 21VL, and 21IR ₂ .

Similarly, the imaging elements 200b ₁ , 200b ₂ and 200b ₃ , the driving units 210b ₁ , 210b ₂ and 210b ₃ , and the buffer memories 211b ₁ , 211b ₂ and 211b ₃ respectively have the imaging lenses 20IR _b1 , 20VL _b and _20IRb2 , and are included in the imaging bodies _21IR1 , 21VL, and _21IR2 , respectively.

The imaging device 1 further includes a control section 220, a memory 221, and a switch (SW) 222. Of these, at least the control unit 220 and the memory 221 are mounted on the processing circuit board 31. The switch 222 corresponds to the shutter button 30 shown in FIGS. 3 and 4. For example, this corresponds to a state in which the switch 222 is closed and the shutter button 30 is operated.

The imaging body 21IR ₁ will be explained. The image sensor 21IR ₁ includes an image sensor 200a ₁ , a drive section 210a ₁ and a buffer memory 211a ₁ , and an image sensor 200b ₁ , a drive section 210b ₁ and a buffer memory 211b ₁ .

The drive unit 210a ₁ drives the image sensor 200a ₁ and reads charges from the image sensor 200a ₁ in response to the synchronous imaging instruction signal 32 supplied as a trigger signal from the control unit 220. The drive unit 210a ₁ converts the charge read from the image sensor 200a ₁ into an electrical signal, and further converts this electrical signal into a captured image that is digital data and outputs the captured image. Note that, in response to one synchronous imaging instruction signal 32, the driving unit 210a ₁ outputs one frame of a captured image based on the charge read from the image sensor 200a ₁ .

The buffer memory 211a ₁ is a memory that can store at least one frame worth of captured images. The captured image output from the drive unit 210a ₁ is stored in this buffer memory 211a ₁ .

In the imaging body 21IR ₁ , the imaging element 200b ₁ , drive unit 210b ₁ and buffer memory 211b ₁ have the same functions as the above-mentioned imaging element 200a ₁ , drive unit 210a ₁ and buffer memory 211a ₁ , so they will be described here. The explanation will be omitted. Furthermore, since the functions of the other imaging bodies 21VL and _21IR2 are the same as those of the imaging body _21IR1 , the description thereof will be omitted here.

The control unit 220 controls the overall operation of the imaging device 1. When the control unit 220 detects the transition of the switch 222 from the open state to the closed state, it outputs the synchronous imaging instruction signal 32. The synchronous imaging instruction signal 32 is simultaneously supplied to each drive section 210a ₁ , 210a ₂ and 210a ₃ and each drive section 210b ₁ , 210b ₂ and 210b ₃ .

The memory 221 stores each captured image from each buffer memory 211a ₁ , 211a ₂ and 211a ₃ and each buffer memory 211b ₁ , 211b 2 and 211b ₃ under the control of the control unit 220 in accordance with the output of the synchronous imaging instruction signal ₃₂ . and stores each read-out captured image. Each captured image stored in the memory 221 can be read out by the image processing device 100 connected to the imaging device 1, as described using FIG. 8, for example.

In this way, the control unit 220 functions as an imaging control unit that controls imaging by each of the imaging bodies 21IR ₁ , 21VL, and 21IR ₂ and acquires the captured images.

FIG. 10 is a block diagram showing an example of the configuration of the control unit 220 and memory 221 in the imaging device 1 according to the first embodiment. In FIG. 10, the control unit 220 includes a CPU (Central Processing Unit) 2000, a ROM (Read Only Memory) 2001, a trigger I/F 2004, a switch (SW) circuit 2005, a data I/F 2006, and a communication I/F 2004. F2007, and each of these units is communicably connected to the bus 2010. Further, the memory 221 includes a RAM (Random Access Memory) 2003 and a memory controller 2002, and the memory controller 2002 is further connected to the bus 2010. The battery 2020 is a secondary battery such as a lithium ion secondary battery, for example, and is a power supply unit that supplies power to various parts inside the imaging device 1 that require power supply.

The CPU 2000 operates according to a program stored in advance in the ROM 2001, for example, using a part of the storage area of the RAM 2003 as a work memory, and controls the overall operation of the imaging apparatus 1. The memory controller 2002 controls data storage and reading from the RAM 2003 according to instructions from the CPU 2000. The memory controller 2002 also controls reading of captured images from each of the buffer memories 211a ₁ , 211a ₂ and 211a ₃ and each of the buffer memories 211b ₁ , 211b ₂ and 211b ₃ according to instructions from the CPU 2000.

The switch circuit 2005 detects the transition between the closed state and the open state of the switch 222 and passes the detection result to the CPU 2000. When the CPU 2000 receives a detection result indicating that the switch 222 has transitioned from the open state to the closed state from the switch circuit 2005, the CPU 2000 outputs the synchronous imaging instruction signal 32 as a trigger signal. The synchronous imaging instruction signal 32 is outputted via the trigger I/F 2004, branched off, and supplied to each drive unit 210a ₁ , 210a ₂ and 210a ₃ and each drive unit 210b ₁ , 210b ₂ and 210b ₃ . Ru.

Data I/F 2006 is an interface for performing data communication with external devices. As the data I/F 2006, for example, a USB (Universal Serial Bus) can be applied. Communication I/F 2007 is connected to a network and controls communication with the network. The network connected to the communication I/F 2007 may be either wired or wireless, or may be connectable to both wired and wireless. The image processing device 100 described above is connected to the imaging device 1 via at least one of the data I/F 2006 and the communication I/F 2007.

Note that, in the above description, the CPU 2000 outputs the synchronous imaging instruction signal 32 in accordance with the detection result by the switch circuit 2005, but this is not limited to this example. For example, the CPU 2000 may output the synchronous imaging instruction signal 32 in response to a signal supplied via the data I/F 2006 or the communication I/F 2007. Further, the trigger I/F 2004 generates a synchronized imaging instruction signal 32 according to the detection result of the switch circuit 2005, and each drive unit 210a ₁ , 210a ₂ and 210a ₃ and each drive unit 210b ₁ , 210b ₂ and 210b ₃ may be supplied.

In such a configuration, when the control unit 220 detects the transition of the switch 222 from the open state to the closed state, it generates and outputs the synchronous imaging instruction signal 32. The synchronous imaging instruction signal 32 is supplied to each drive unit 210a ₁ , 210a ₂ and 210a ₃ and each drive unit 210b ₁ , 210b ₂ and 210b ₃ at the same timing. Each drive unit 210a ₁ , 210a ₂ and 210a ₃ and each drive unit 210b ₁ , 210b ₂ and 210b ₃ drive each image sensor 200a ₁ , 200a ₂ and 200a in synchronization with the supplied synchronous imaging instruction signal 32. ₃ and the charges are read out from each image sensor 200b ₁ , 200b ₂ and 200b ₃ .

Each drive unit 210a ₁ , 210a ₂ and 210a ₃ and each drive unit 210b ₁ , 210b ₂ and 210b ₃ drive each image sensor 200a ₁ , 200a ₂ and 200a ₃ and each image sensor 200b ₁ , 200b ₂ and The electric charges read from 200b ₃ are each converted into a captured image, and each converted captured image is stored in each buffer memory 211a ₁ , 211a ₂ and 211a ₃ and each buffer memory 211b ₁ , 211b ₂ and 211b ₃ .

At a predetermined timing after the output of the synchronous imaging instruction signal 32, the control unit 220 causes the memory 221 to read data from each of the buffer memories 211a ₁ , 211a ₂ and 211a ₃ and from each of the buffer memories 211b ₁ , 211b ₂ and 211b ₃ . instructs to read out the captured image. In the memory 221, the memory controller 2002 reads each captured image from each buffer memory 211a ₁ , 211a ₂ and 211a ₃ and each buffer memory 211b ₁ , 211b ₂ and 211b ₃ according to this instruction, The captured image is stored in a predetermined area of the RAM 2003.

For example, when the image processing device 100 is connected to the imaging device 1 via the data I/F 2006, the image processing device 100 requests reading of each captured image stored in the RAM 2003 via the data I/F 2006. In response to this request, CPU 2000 instructs memory controller 2002 to read each captured image from RAM 2003. The memory controller 2002 reads each captured image from the RAM 2003 in response to this instruction, and transmits each read captured image to the image processing apparatus 100 via the data I/F 2006. The image processing device 100 executes a three-dimensional restoration model creation process according to the first embodiment, which will be described later, based on each captured image transmitted from the imaging device 1.

FIG. 11 is a block diagram showing the configuration of an example of an image processing apparatus 100 applicable to the first embodiment. The image processing apparatus 100 includes a CPU 1000 , a ROM 1001 , a RAM 1002 , a graphic I/F 1003 , a storage 1004 , an input device 1005 , a data I/F 1006 , and a communication I/F 1007 , and these units are connected to a bus 1010 . communicably connected to each other. Note that in FIG. 11, the input device 1005 can be detachably attached to the image processing apparatus 100.

In this way, the image processing apparatus 100 includes a CPU 1000 and various storage media such as a ROM 1001, a RAM 1002, and a storage 1004, and can be configured using a general computer.

The storage 1004 is a storage medium that can store data in a nonvolatile manner, and is, for example, a nonvolatile semiconductor memory such as a flash memory. The present invention is not limited to this, and a hard disk drive may be used as the storage 1004. The storage 1004 stores programs and various data for the CPU 1000 to execute. Note that the storage 1004 and the ROM 1001 may share, for example, one rewritable nonvolatile semiconductor memory.

CPU 1000 controls the entire image processing apparatus 100 according to programs stored in ROM 1001 and storage 1004, using RAM 1002 as a work memory. The graphic I/F 1003 converts the display control signal generated by the CPU 1000 into a signal that can be displayed by the display 1020 and outputs the signal. The display 1020 includes, for example, an LCD (Liquid Crystal Display) as a display device, and is driven by a signal output from the graphic I/F 1003 to display a screen according to a display control signal. The display 1020 may be built into the image processing apparatus 100 or may be detachably attached to the image processing apparatus 100.

Data I/F 1006 inputs and outputs data to and from external devices. As the data I/F 1006, a USB can be applied, for example, in association with the data I/F 2006 of the imaging device 1 described above. The image processing device 100 is connected to the imaging device 1 through a data I/F 1006, and can transmit a request for a captured image to the imaging device 1, acquire a captured image from the imaging device 1, and the like.

Input device 1005 receives, for example, user input. As the input device 1005, a pointing device such as a mouse or a keyboard, which is used by being connected to the image processing apparatus 100, can be used. The input device 1005 is not limited to this, and may be an operator built into the image processing apparatus 100. The user can input instructions to the image processing apparatus 100 by operating the input device 1005 according to the information displayed on the display 1020, for example.

The communication I/F 1007 performs communication via the network under the control of the CPU 1000.

FIG. 12 is an example functional block diagram for explaining the functions of the image processing apparatus 100 according to the first embodiment. In FIG. 12, the image processing device 100 includes an acquisition section 110, a matching processing section 111, a 3D information generation section 112, a reprojection processing section 113, an output processing section 114, a UI (User Interface) section 120, A control unit 121 is included.

These acquisition section 110, matching processing section 111, 3D information generation section 112, reprojection processing section 113, output processing section 114, UI section 120, and control section 121 run the image processing program according to the first embodiment on the CPU 1000. It is realized through execution. However, the present invention is not limited to this, and some or all of the acquisition section 110, matching processing section 111, 3D information generation section 112, reprojection processing section 113, output processing section 114, UI section 120, and control section 121 may cooperate with each other. It may also be realized by a hardware circuit that operates by

The acquisition unit 110 is connected to the imaging device 1 to capture images captured by the imaging devices 200a ₁ , 200a 2 and 200a ₃ and 200b ₁ , 200b _{2 and} 200b ₃ stored in the memory 221 of the imaging device 1. Acquire each captured image. The acquisition unit 110 acquires the captured images captured by the imaging bodies 21IR ₁ and 21IR ₂ , that is, the captured images captured by the imaging elements 200a ₁ , 200a ₃ , 200b ₁ and 200b ₃ , among the acquired captured images. , is passed to the matching processing section 111. Furthermore, among the acquired images, the acquisition unit 110 passes the captured images captured by the imaging body 21VL, that is, the captured images captured by the imaging elements 200a ₂ and 200b ₂ to the reprojection processing unit 113.

The matching processing unit 111 performs matching processing using the captured images captured by the imaging bodies 21IR ₁ and 21IR ₂ passed from the acquisition unit 110. The 3D information generation section 112 obtains depth information using the results of the matching process performed by the matching processing section 111, and generates three-dimensional point group information based on the obtained depth information.

The reprojection processing unit 113 reprojects each captured image captured by the imaging body 21VL passed from the acquisition unit 110 onto the three-dimensional point cloud information generated by the matching processing unit 111, and generates three-dimensional point cloud information. Add color information to.

The output processing unit 114 outputs the three-dimensional point group information to which color information has been added by the reprojection processing unit 113 as a three-dimensional restored model. The output processing unit 114 may output the three-dimensional restored model to the outside of the image processing apparatus 100, or may store it in the RAM 1002 or the storage 1004. The output processing unit 114 can also display the three-dimensional reconstructed model on the display 1020.

The UI unit 120 controls screen display on the display 1020, accepts input according to user operations on the input device 1005, and provides a user interface. The control unit 121 controls the overall operation of the image processing apparatus 100.

The image processing program for realizing each function according to the first embodiment in the image processing device 100 is a file in an installable format or an executable format, and is stored on a CD (Compact Disk), a flexible disk (FD), or a DVD ( It is recorded and provided on a computer-readable recording medium such as a digital versatile disk (Digital Versatile Disk). The present invention is not limited to this, and the image processing program may be provided by being stored on a computer connected to a network such as the Internet and downloaded via the network. Further, the image processing program may be provided or distributed via a network such as the Internet.

The image processing program has a module configuration including an acquisition section 110, a matching processing section 111, a 3D information generation section 112, a reprojection processing section 113, an output processing section 114, a UI section 120, and a control section 121. In actual hardware, when the CPU 1000 reads out and executes the image processing program from a storage medium such as the storage 1004, the above-mentioned units are loaded onto a main storage device such as the RAM 1002, and the acquisition unit 110, matching processing unit 111, a 3D information generation section 112, a reprojection processing section 113, an output processing section 114, a UI section 120, and a control section 121 are generated on the main storage device.

[Details of 3D reconstruction model creation process applicable to the first embodiment]
Next, the three-dimensional reconstruction model creation process according to the first embodiment will be described in more detail. FIG. 13 is a flowchart of an example of a three-dimensional reconstruction model creation process applicable to the first embodiment.

It is assumed that, prior to execution of the processing in the flowchart of FIG. 13, the imaging apparatus 1 performs imaging in which the imaging bodies 21IR ₁ , 21VL, and 21IR ₂ are synchronized. Therefore, the memory 221 of the imaging device 1 contains four captured images captured by the imaging bodies 21IR ₁ and 21IR ₂ that include light in the non-visible light region, and four captured images that do not include light in the non-visible light region by the imaging body 21VL. , two captured images selectively capturing light in the visible light region, and six captured images are stored.

In step S100, the acquisition unit 110 acquires from the memory 221 of the imaging device 1 a captured image that includes light in the non-visible light region. More specifically, the acquisition unit 110 retrieves two captured images captured by the imaging body 21IR 1 and two captured images captured by the imaging body 21IR ₂ from the memory ₂₂₁ of the imaging device 1. get. The acquisition unit 110 passes the acquired four captured images to the matching processing unit 111.

In the next step S101, the matching processing unit 111 performs matching processing based on the four captured images passed from the acquisition unit 110. Furthermore, in the next step S102, the 3D information generation unit 112 calculates depth information based on the matching processing result in step S101, and generates three-dimensional point group information.

The processing in step S101 and step S102 will be explained in more detail. In the first embodiment, depth information is calculated by a stereo method using two spherical images based on images captured by imaging bodies 21IR ₁ and 21IR ₂ that capture light in the non-visible light region. . The stereo method referred to here uses two images taken from different viewpoints by two cameras, and a certain pixel (reference pixel) in one image is compared to a corresponding pixel (reference pixel) in the other image. This method calculates the depth (depth value) by trigonometry based on the reference pixel and the corresponding pixel.

In step S101, the matching processing unit 111 combines two captured images captured by the imaging body 21IR ₁ among the four captured images passed from the acquisition unit 110, and creates one omnidirectional image. (referred to as the first spherical image). Similarly, the matching processing unit 111 combines two captured images captured by the imaging body 21IR ₂ to create one omnidirectional image (referred to as a second omnidirectional image).

Here, in the general stereo method, as illustrated in FIG. 14(a), captured images 300a and 300b captured by shifting the viewpoint in the horizontal direction are used, and a reference pixel in the captured image 300a serving as a standard is used. The search is performed by moving an area 302 in the captured image 300b to be searched, which corresponds to an area 301 of a predetermined size centered at 303, in the horizontal direction as shown by arrow D in the figure. At this time, the captured images 300a and 300b are captured by matching the horizontal heights of the imaging positions and matching the imaging directions, so that the captured images 300a and 300b are parallel to the horizontal direction. As a result, the pixels (corresponding pixels) in the captured image 300b, which are captured at locations corresponding to the pixels (reference pixels) in the captured image 300a, exist on the same line in the vertical direction, and the one-dimensional search It becomes possible to search for corresponding pixels, and it becomes possible to suppress the amount of calculation.

In the imaging device 1 according to the first embodiment, imaging bodies 21IR ₁ and 21IR ₂ for acquiring captured images used in the stereo method are arranged in alignment in the vertical direction. In this case, as illustrated in FIG. 14(b), using captured images 300a' and 300b' taken with the viewpoint shifted in the vertical direction, a captured image 300b corresponding to a region 301' in the captured image 300a' is used. The search is performed by moving the inner region 302 in the vertical direction as shown by arrow E in the figure. In this case as well, the captured images 300a' and 300b' are captured with the same horizontal position and the same imaging direction, so that the captured images 300a' and 300b' are parallel to the vertical direction. As a result, the reference pixel in the captured image 300a' and the corresponding pixel in the captured image 300b have their horizontal coordinates on the same line, making it possible to search for the corresponding pixels by one-dimensional search.

Note that when using the imaging device 1 according to the first embodiment, for two omnidirectional images based on captured images captured by the imaging bodies 21IR ₁ and 21RI ₂ arranged in the vertical direction. The stereo method will be applied. In the example of the first embodiment, in the imaging device 1, each piece of spherical information based on the captured image captured by each of the imaging bodies 21IR ₁ and 21IR ₂ is expressed in plane coordinates using the equi-rectangular projection method. By expanding the image into two spherical images, two spherical images are generated. This makes it possible to search for corresponding pixels using the stereo method using a one-dimensional search in the vertical direction as shown in FIG. 14(b), which facilitates processing.

Note that the equirectangular projection is a projection method that transforms spherical information into plane coordinates by using a coordinate system in which the vertical direction corresponds to latitude and the horizontal direction corresponds to longitude. The present invention is not limited to this, and other projection methods may be applied to the first embodiment as a projection method for converting spherical surface information into plane coordinates.

The matching processing unit 111 searches for corresponding pixels between a reference image (referred to as a first spherical image) and an image to be searched (referred to as a second spherical image), and performs matching. Perform processing. Various methods are known for searching for corresponding pixels; for example, a block matching method can be applied.

The block matching method acquires pixel values of an area cut out as a block of M pixels by N pixels with the reference pixel as the center in the first spherical image. Furthermore, in the second spherical image, the pixel values of a region cut out as a block of M pixels×N pixels with the target pixel as the center are acquired. Based on the pixel values, the degree of similarity between the region containing the reference pixel and the region containing the target pixel is calculated. The degree of similarity is compared while moving the block of M pixels by N pixels within the image to be searched, and the target pixel in the block at the position where the degree of similarity is highest is determined as the corresponding pixel corresponding to the reference pixel.

Similarity can be calculated using various calculation methods. For example, the normalized cross-correlation coefficient (NCC) shown in Equation (1) is one of the cost functions, and the larger the value of the numerical value C _NCC indicating the cost, the higher the degree of similarity. In equation (1), the values M and N represent the size of the pixel block for performing the search. Further, the value I(i,j) represents the pixel value of a pixel in the pixel block in the first omnidirectional image that is the reference, and the value T(i,j) represents the pixel value of the pixel in the pixel block in the first omnidirectional image that is the reference Represents pixel values within a pixel block in a celestial sphere image.

As explained using FIG. 14(b), the matching processing unit 111 generates a pixel block of M pixels by N pixels (corresponding to area 301') in the first spherical image (corresponding to captured image 300a'). ) in the second spherical image (corresponding to the captured image 300b') in the vertical direction, for example, as shown by the arrow E in FIG. 14(b). The calculation of equation (1) is executed while moving pixel by pixel, and the numerical value C _NCC is calculated. In the second spherical image, the center pixel of the pixel block with the maximum numerical value C _NCC is set as the target pixel corresponding to the reference pixel.

Returning to the explanation of FIG. 13, in step S102, the 3D information generation unit 112 uses trigonometry to calculate a depth value (depth information) based on the reference pixel and the corresponding pixel obtained by the matching process in step S101. Then, three-dimensional point group information related to the first spherical image and the second spherical image is generated.

FIG. 15 is a diagram for explaining trigonometry applicable to the first embodiment. The purpose of the processing is to calculate the distance D to the target object 403 in the figure from the imaging position information within the image captured by the image sensor 402. That is, this distance D corresponds to the depth information of the target pixel. The distance D is calculated using the following equation (2).

Note that in equation (2), the value baseline represents the length of the baseline between the cameras 400a and 400b (baseline length). In the example of the imaging device 1, as shown in FIGS. 3 and 6, the distance d between the imaging bodies 21IR ₁ and 21IR ₂ corresponds to the value baseline. Note that a line connecting the optical centers of the imaging body 21IR ₁ and the imaging body 21IR ₂ constitutes a base line. The value f represents the focal length of the lens 401. The value q represents the parallax. The parallax q is a value obtained by multiplying the difference between the coordinate values of the reference pixel and the corresponding pixel by the pixel pitch of the image sensor. The coordinate values of the corresponding pixels are obtained based on the result of the matching process in step S101.

This equation (2) is the method for calculating the distance D when the two cameras 400a and 400b are used. This is to calculate the distance D from the captured images respectively captured by the two cameras 400a and 400b. In the first embodiment, the calculation method according to equation (2) is applied to the first spherical image and the second spherical image based on the captured images captured by the imaging bodies 21IR ₁ and 21IR ₂ . to calculate the distance D.

The 3D information generation unit 112 passes the three-dimensional point cloud information generated in step S102 to the reprojection processing unit 113.

In FIG. 13, in the next step S103, the acquisition unit 110 acquires a captured image obtained by selectively capturing light in the visible light region from the memory 221 of the imaging device 1. More specifically, the acquisition unit 110 acquires two captured images captured by the imaging body 21VL from the memory 221 of the imaging device 1. The acquisition unit 110 passes the acquired two captured images to the reprojection processing unit 113. The reprojection processing unit 113 combines the two captured images passed from the acquisition unit 110 to create one omnidirectional image (referred to as a visible light omnidirectional image).

In step S104, the reprojection processing unit 113 converts the three-dimensional point group information passed from the 3D information generation unit 112 in step S102 to the two images captured by the imaging body 21VL passed from the acquisition unit 110 in step S103. Performs reprojection processing to project onto a visible light spherical image based on the captured image.

The reprojection process in step S104 will be explained using FIG. 16. In FIG. 16, the camera center is a camera position for acquiring a captured image having color information, and in the imaging device 1, corresponds to the optical center of the imaging body 21VL. The image 320 shows a captured image having color information captured from the center of the camera, and in the imaging device 1 corresponds to a visible light spherical image based on the captured image captured by the imaging body 21VL. Restoration points #1 and #2 respectively indicate points in the three-dimensional point group information from which three-dimensional information has been restored corresponding to the image 320.

Further, in FIG. 16, reprojection coordinate #1 indicates a position corresponding to restoration point #1 in image 320. Similarly, reprojection coordinate #2 indicates the position corresponding to restoration point #2 in image 320. The reprojection process is a process of associating each point of the three-dimensional point cloud information with each point having a corresponding position in the image 320. In the example of FIG. 16, the process is to convert the coordinates of restoration points #1 and #2 in the three-dimensional point group information to reprojection coordinates #1 and #2 in the image 320.

The reprojection process can be realized by translating the three-dimensional point group information according to the camera position that captured the image 320, and then converting it into the image coordinate system (two-dimensional screen coordinate system) of the image 320. Equation (3) shows an example of a conversion equation for performing such reprojection processing.

In equation (3), (X, Y, Z) on the right side represents the three-dimensional coordinates of the three-dimensional point group information in the camera coordinate system at the camera position shown as the camera center in FIG. Note that in equation (3), the value Z corresponds to the distance D in FIG. 15 and indicates the depth. Further, the values (X, Y) indicate coordinates in the horizontal and vertical directions, respectively. The 3×3 matrix in Equation (3) is a matrix called a projection matrix, and the values f _x and f _y each indicate the focal length in the (x, y) direction (horizontal, vertical direction). Moreover, the values c _x and c _y represent the amount of shift in the horizontal and vertical directions of the two-dimensional screen coordinate system with respect to the optical center, respectively. The left side indicates the coordinates of the image 320, and among the values (u, v, 1), the value (u, v) corresponds to the coordinates (x, y) in the image 320. Further, the value λ on the left side is a coefficient indicating a ratio.

Returning to the explanation of FIG. 13, in the next step S105, the reprojection processing unit 113 adds the color information of the visible light spherical image to the three-dimensional point cloud information based on the result of the reprojection processing in step S104. . Referring to FIG. 16, this is a process of adding color information of pixels at reprojection coordinates #1 and #2 in image 320 to restoration points #1 and #2 of three-dimensional point group information.

That is, the reprojection process in step S104 has determined the coordinates to be included in the visible light spherical image for each point in the three-dimensional point group information, so based on these coordinates, each point in the visible light spherical image The color information of the pixel is added to the corresponding coordinates of the three-dimensional point cloud information. That is, the color information of a pixel in the visible light spherical image is added to the coordinates of the three-dimensional point group information corresponding to the coordinates of the pixel in the visible light spherical image.

Here, the coordinates calculated by the reprojection process may include digits below the decimal point. Therefore, by interpolation processing, the pixel values of pixels at integer coordinates around the coordinate in the visible light spherical image are determined in order to obtain the desired brightness value at the coordinate.

The three-dimensional point cloud information to which color information has been added in step S105 is acquired as a three-dimensional restored model. The acquired three-dimensional restored model may be output from the image processing apparatus 100 to the outside, or may be stored in the storage 1004 or the like.

As described above, in the first embodiment, three-dimensional point group information is obtained based on the captured images captured by the imaging bodies 21IR ₁ and 21IR ₂ that capture light in the non-visible light region, and light in the visible light region is obtained. The color information of each pixel in the captured image captured by the imaging body 21VL that selectively captures the image can be added to the obtained three-dimensional point group information.

[Second embodiment]
Next, a second embodiment will be described. The imaging device 1 according to the first embodiment described above includes two imaging bodies 21IR ₁ and 21IR _{2 that each image light including a non-visible light region, and a first imaging body 21IR 2} that selectively images light in the visible light region. Although the configuration includes the imaging body 21VL, this is not limited to this example. The imaging device according to the second embodiment has a configuration that includes more imaging bodies than the configuration of the first embodiment.

FIG. 17 is a diagram schematically showing the internal structure of an imaging device 1' according to the second embodiment. In FIG. 17, the left side is the front direction and the right side is the back direction, similar to FIG. 4 described above.

In the example of FIG. 17, the imaging device 1' has additional imaging bodies 21VL ₁ and 21VL ₂ for selectively imaging light in the visible light region, in addition to the imaging device 1 shown in FIG. 4. That is, the imaging device 1' includes three imaging bodies 21VL, 21VL ₁ and 21VL ₂ for selectively imaging light including the visible light region, and two imaging bodies 21VL 1 and 21VL 2 for selectively imaging light in the non-visible light region. Image pickup bodies 21IR ₁ and 21IR ₂ are included. In the example of FIG. 17, the imaging body 21VL ₁ is provided above the imaging body 21IR ₁ , and the imaging body 21VL ₂ is provided below the imaging body 21IR ₂ .

In the imaging device 1', each imaging lens 20VL _a1 , 20IR _a1 , 20VL _a , 20IR _a2 and 20VL _a2 in the front direction has its lens center aligned with the center line in the long side direction of the housing 10, as in FIG. 6(b). are arranged in alignment. Similarly, the imaging lenses 20VL _b1 , 20IR _b1 , 20VL _b , 20IR _b2 and 20VL _b2 in the rear direction are also arranged with their lens centers aligned with the center line in the long side direction of the housing 10 .

Further, like the imaging body 21VL, the imaging body 21VL ₁ includes an imaging lens 20VL _a1 and an imaging element 200a ₄ in the front direction, and an imaging lens 20VL _b1 and an imaging element 200b ₄ in the rear direction. Similarly, the imaging body 21VL ₂ includes an imaging lens 20VL _a2 and an imaging element 200 _a5 in the front direction, and an imaging lens 20VL _b2 and an imaging element 200b ₅ in the rear direction.

Further, in each of the imaging bodies 21VL ₁ and 21VL ₂ , similarly to the imaging body 21VL, each of the imaging lenses 20VLa ₁ , 20VL _a2 , 20VL _b1 and 20VL _b2 is provided with a filter 22VL that selectively transmits light in the visible light region. provided. The filter 22VL is not limited to this, and the filter 22VL may be provided in the imaging elements 200a ₁ and 200b ₁ and the imaging devices 200a ₅ and 200b ₅ , or the filter 22VL may be provided in the non-visible light region by image processing without providing the filter 22VL. You may cut the ingredients.

The processing circuit board 31' is equipped with a circuit for synchronizing and driving the imaging elements 200a ₁ to 200a ₅ and the imaging elements 200b ₁ to 200b ₅ included in the imaging apparatus 1'. This circuit has a configuration in which a set of a driving section and a buffer memory is added in accordance with the four added image pickup elements 200a ₄ , 200a ₅ , 200b ₄ and 200b ₅ to the configuration shown in FIG. 9 . The processing circuit board 31' generates a synchronous imaging instruction which is a trigger signal for synchronously executing imaging by the imaging elements 200a ₁ to 200a ₅ and the imaging elements 200b ₁ to 200b ₅ in response to an operation on the shutter button 30. A signal 32' is output.

The imaging body 21VL, the imaging bodies 21IR ₁ and 21IR ₂ obtain three-dimensional point group information based on the captured images of light in the non-visible light region by the imaging bodies 21IR ₁ and 21IR ₂ in the same manner as in the first embodiment. generate. Color information is given to this three-dimensional point group information based on a captured image obtained by selectively capturing light in the visible light region by the imaging body 21VL, and a three-dimensional restoration model is created.

In the imaging device 1' according to the second embodiment, _the multi _- base A three-dimensional point cloud is generated using a line stereo method.

That is, in the imaging device 1' according to the second embodiment, the three imaging bodies 21VL, 21VL ₁ and 21VL ₂ for selectively imaging light in the visible light region are aligned on the center line of the housing 10. It is arranged as follows. Therefore, it is possible to configure a plurality of base lines existing on one straight line, and it is possible to form a plurality of pairs of imaging objects (referred to as stereo pairs) to which trigonometry can be applied. In the example of FIG. 17, a first stereo pair includes the image pickup body 21VL ₁ and the image pickup body 21VL, a second stereo pair includes the image pickup body 21VL and the image pickup body VL ₂ , and a third stereo pair includes the image pickup body 21VL ₁ and the image pickup body 21VL ₂ . It is possible to form three stereo pairs of pairs.

Here, a common imaging range is imaged using multiple stereo pairs that include portions that overlap with the baseline, depth information is obtained for each image based on the imaged image, and three-dimensional point cloud information is generated by combining each depth information. Can be done. FIG. 18 is a diagram illustrating an example of pixel search processing using a plurality of stereo pairs, which is applicable to the second embodiment. For example, consider images 300d, 300e, and 300f of omnidirectional images captured by the imaging bodies 21VL ₁ , 21VL, and 21VL ₂ , respectively. In this case, a stereo pair of images (referred to as a stereo pair image) is formed by the captured images 300d and 300e, the captured images 300e and 300f, and the captured images 300d and 300f, respectively.

In the captured images 300d, 300e, and 300f, among the imaging bodies 21VL ₁ , 21VL, and 21VL ₂ , for example, the captured image 300e captured by the centrally located imaging body 21VL is set as a reference image including the reference pixel 333. A region 332 in the captured image 300d captured by the imaging body 21VL ₁ to be searched, which corresponds to a region 331 of a predetermined size centered on the reference pixel 333, is vertically moved as shown by arrow F in the figure. Move and explore. Similarly, the captured image 300f captured by the imaging body 21VL ₂ is searched, and a region 332' corresponding to the region 331 in the captured image 300f is searched in the vertical direction as shown by arrow F' in the figure. Move to and explore. The correspondence between the region 331 and the regions 332 and 332' is indicated by a straight line 340 in the figure.

Here, the process of finding a pixel corresponding to the reference pixel 333 is a process of searching for a pixel corresponding to one point in space in each of the captured image 300d by the imaging body 21VL ₁ and the captured image 300f by the imaging body VL ₂ . becomes. This is a common depth value (distance This is a search process for the corresponding pixel for D).

The following method can be applied by utilizing the fact that the distance D is common between the two stereo pair images. That is, using the parallax q as a parameter, a numerical value indicating the similarity, that is, the cost, for each of the first stereo pair image and the second stereo pair image, based on the corresponding pixels corresponding to the parallax q in the captured images 300d and 300f, which are the target images. Calculate C _NCC (hereinafter referred to as cost C _NCC ). Then, the first stereo pair image and the second stereo pair image have the maximum addition result ΣC _NCC of the cost C _NCC1 calculated for the first stereo pair image and the cost C _NCC2 calculated for the second stereo pair image. Find the parallax q ₁ and q ₂ for each. According to this method, it is possible to obtain each parallax q ₁ and q ₂ more reliably and with fewer errors than when using a single stereo pair image, and the distance D can be calculated with higher accuracy.

In this way, the three-dimensional point group information generated based on the distance D obtained using each image captured by the imaging bodies 21VL ₁ , 21VL and 21VL ₂ and the imaging bodies 21IR ₁ and 21VL described in the first embodiment and three-dimensional point group information generated based on a captured image containing light in the non-visible light region using 21IR ₂ can be combined and used. As a result, more point group information can be used, and a three-dimensional reconstructed model can be generated with higher accuracy.

The above method can also be applied to other stereo pair images by using the parallax q with the point of overlap as the reference point if the baselines overlap. In the example of the imaging device 1' in FIG. 17, in addition to the first stereo pair image formed by each of the captured images of the image capturing bodies 21VL ₁ and 21VL mentioned above, and the second stereo pair image formed by each captured image of the image capturing bodies 21VL and 21VL ₂ , , a third stereo pair of images formed by the respective captured images of the imaging bodies 21VL ₁ and 21VL ₂ can be used. It is also possible to further use a stereo pair of images of the images of the imaging bodies 21IR ₁ and 21IR ₂ in which images are captured including light in the non-visible light region. As a result, more point group information can be used, and a more accurate three-dimensional reconstructed model can be generated.

[Modification of second embodiment]
The imaging device 1' shown in FIG. 17 includes three imaging bodies 21VL, 21VL ₁ and 21VL ₂ that selectively capture light in the visible light region, and a second imaging body that captures images including light in the non-visible light region. It has bodies 21IR ₁ and 21IR ₂ . In the imaging device 1', the imaging bodies 21VL, 21VL ₁ and 21VL ₂ and the imaging bodies 21IR ₁ and 21IR ₂ are arranged alternately.

This is not limited to this example, and the imaging device may include, for example, a third imaging body that captures images including light in the non-visible light range, and a second imaging body that selectively captures light in the visible light range. , may be arranged alternately. Furthermore, the number of imaging bodies is not limited to five, but may be six or more. When 6 imaging bodies are arranged in the imaging device, for example, 3 imaging bodies that capture images including light in the non-visible light region, 3 imaging bodies that selectively capture images of light in the visible light region, It is conceivable to arrange them alternately.

The above-described embodiments are preferred examples of the present invention, but the present invention is not limited thereto, and various modifications can be made without departing from the gist of the present invention.

1, 1', 50a Imaging device 10 Housing 20IR _a1 , 20IR _a2 , 20IR _b1 , 20IR _b2 , 20VL _a , 20VL _a1 , 20VL _a2 , 20VL _b , 20VL _b1 , 20VL _b2 Imaging lens 21IR ₁ , 21IR ₂ , 21VL, 21VL ₁ , 21VL ₂ Image pickup bodies 22IR, 22VL Filter 30 Shutter button 31, 31' Processing circuit board 32, 32' Synchronous imaging instruction signal 51 Light source 80 Light in invisible light region 81 Pattern 100 Image processing device 110 Acquisition unit 111 Matching process Section 112 3D information generation section 113 Reprojection processing section 200a, 200b, 200a ₁ , 200a ₂ , 200a ₃ , 200a ₄ , 200a ₅ , 200b ₁ , 200b ₂ , 200b ₃ , 200b ₄ , 200b ₅ Image sensor 220 Control section 221 Memory 222 Switches 300a, 300b, 300a', 300b', 300d, 300e, 300f Captured images 301, 302, 331, 332, 332' Regions 303, 333 Reference pixels

JP2013-218278A US Patent Application Publication No. 2014/0132501

Claims (7)

a plurality of first optical systems facing in different directions;
a plurality of first image sensors that respectively image visible light irradiated through each of the plurality of first optical systems;
a plurality of second optical systems facing in different directions;
a plurality of second image pickup elements each capturing an image of the invisible light irradiated through each of the plurality of second optical systems;
a plurality of third optical systems facing in different directions;
a plurality of third image pickup elements each capturing an image of the invisible light irradiated through each of the plurality of third optical systems;
The plurality of first optical systems, the plurality of first image sensors, the plurality of second optical systems, the plurality of second image sensors, the plurality of third optical systems, and the plurality of third image sensors are provided. A casing and
Equipped with
outputting a first image including color information by the first image sensor, and outputting a second image and a third image for calculating depth information by the second image sensor and the third image sensor, respectively;
The casing has an elongated shape in a first direction, and has a bottom surface on one end side of the elongated shape,
Among the plurality of first optical systems, the plurality of second optical systems, and the plurality of third optical systems, the second optical system is closest to the bottom surface, and between the second optical system and the bottom surface. There are areas where the optical system is not placed,
The first direction is oriented in a direction perpendicular to the floor surface of the space to be imaged, and the plurality of first optical systems, the plurality of second optical systems, and the plurality of third optical systems are arranged in the optical systems. The image is taken with the bottom surface facing the floor surface so that the bottom surface is above the area that is not
The plurality of first optical systems, the plurality of second optical systems, and the plurality of third optical systems are arranged at different positions in the first direction, and image a 360° circumference of the casing in the first direction. Imaging device that is possible. The imaging device according to claim 1, wherein the plurality of first optical systems, the plurality of second optical systems, and the plurality of third optical systems are arranged on the other end side of the housing in the first direction. The plurality of first optical systems or the plurality of third optical systems are arranged on the other end side of the housing,
The imaging device according to claim 1, wherein a button to be operated is arranged in an area of the housing where the optical system is not arranged . 4. A second optical system of each of the plurality of second optical systems and a third optical system of each of the plurality of third optical systems each have an angle of view of 180° or more. The imaging device according to item 1. a plurality of fourth optical systems facing in mutually different directions;
a plurality of fourth image sensors that respectively image visible light incident on each of the plurality of fourth optical systems;
Furthermore,
An area where the optical system is not placed is located between the fourth optical system and the bottom surface,
Any one of claims 1 to 4, wherein the plurality of second optical systems or the plurality of third optical systems are arranged between the plurality of first optical systems and the plurality of fourth optical systems. The imaging device described in . The imaging device according to any one of claims 1 to 5, wherein the invisible light that enters the plurality of second optical systems and the plurality of third optical systems is pattern light emitted from a light source. A method for controlling an imaging device, the method comprising:
The imaging device includes:
a plurality of first optical systems facing in different directions;
a plurality of first image sensors that respectively image visible light irradiated through each of the plurality of first optical systems;
a plurality of second optical systems facing in different directions;
a plurality of second image pickup elements each capturing an image of the invisible light irradiated through each of the plurality of second optical systems;
a plurality of third optical systems facing in different directions;
a plurality of third image pickup elements each capturing an image of the invisible light irradiated through each of the plurality of third optical systems;
The plurality of first optical systems, the plurality of first image sensors, the plurality of second optical systems, the plurality of second image sensors, the plurality of third optical systems, and the plurality of third image sensors are provided. A casing and
including;
The casing has an elongated shape in a first direction, and has a bottom surface on one end side of the elongated shape,
Among the plurality of first optical systems, the plurality of second optical systems, and the plurality of third optical systems, the second optical system is closest to the bottom surface, and between the second optical system and the bottom surface. There are areas where the optical system is not placed,
outputting a first image including color information by the first image sensor, and outputting a second image and a third image for calculating depth information by the second image sensor and the third image sensor, respectively;
The first direction is oriented in a direction perpendicular to the floor surface of the space to be imaged, and the plurality of first optical systems, the plurality of second optical systems, and the plurality of third optical systems are arranged in the optical systems. A step of taking an image with the bottom surface facing the floor surface so that the bottom surface is above the area where the bottom surface is not
The plurality of first optical systems, the plurality of second optical systems, and the plurality of third optical systems are arranged at different positions in the first direction, and image a 360° circumference of the casing in the first direction. the step of
A method for controlling an imaging device having the following.

Images