JP2018152632A - Imaging apparatus and imaging method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To record distance information to a subject in pixel information of an optical image taken by the imaging apparatus on a pixel-by-pixel basis.SOLUTION: An imaging apparatus 1 including an imaging optical system 5 for forming an image of a subject on a sensor and a light source 21 for irradiating the subject. The sensor includes an optical image sensor 10 for imaging at least visible light and a TOF image sensor 20 for measuring a distance to the subject. The optical image sensor 10 or the TOF image sensor 20 is used to image the image formed by the imaging optical system 5, and then the other sensor is displaced to the imaging position by the imaging optical system 5 and performs imaging.SELECTED DRAWING: Figure 1

Description

  The present invention relates to an imaging apparatus and an imaging method.

  In recent years, the imaging device has been replaced by a digital camera that is sensitive to an optical sensor (optical image sensor) such as a CCD (Charged-coupled devices) or CMOS (Complementary metal-oxide-semiconductor) instead of a silver salt camera that captures a subject on film. The signal is recorded as an optical image (optical image) on a pixel on the optical image sensor.

  On the other hand, development of ranging technology using a three-dimensional sensor for measuring the distance to an object to be measured has been actively conducted. One of the techniques is three-dimensional laser measurement. The distance from the laser device to the object is measured based on the time until the laser beam emitted from the laser device is reflected back to the object and the reflected intensity. By emitting the laser beam radially at regular intervals, 3 Dimensional point cloud data can be obtained. In the following description, an object that is a measurement target of the three-dimensional sensor is also referred to as a subject.

  According to the three-dimensional laser measurement, the three-dimensional shape of the object can be examined, but since it is not acquired as an image as in the photograph, for example, whether the metal fitting is removed or the member is not cracked or corroded. It is not suitable for grasping the situation of an object. In addition, 3D laser measurement devices are generally much more expensive than imaging devices, and the measurement time is longer than that of photography. Therefore, the overall cost is high. In addition, since the three-dimensional laser measuring instrument is large and heavy and is not easy to handle, it is unsuitable for measurement in high places and uneasy places.

  As a distance measurement technique using a three-dimensional sensor, a TOF (Time of Flight) calculation method for obtaining the distance to an object based on the time from when the pulsed light is projected until the pulsed light is reflected by the object and returned. Ranging technology using a sensor is known (for example, Patent Documents 1 and 2).

  In the distance measurement technique using TOF, an area image sensor (TOF image sensor) in which pixels are two-dimensionally arranged on the sensor is used, and distance information is recorded in each pixel. A pixel in which pixel information is recorded in each pixel of the TOF image sensor is also referred to as a TOF image (TOF image).

  Since the number of pixels of the area image sensor used for such distance measurement is smaller than the number of pixels of the optical image sensor generally used for imaging devices, even if the area image sensor used for distance measurement is used, the optical It is not possible to record a high-definition optical image that can be recorded by an image sensor.

  As a technique for acquiring both an optical image and distance information, a technique is known in which an optical image sensor and its imaging optical system, and an area image sensor and its imaging optical system are arranged in one apparatus. Document 3 discloses a first filter that passes light in multiple wavelength bands included in light reflected from a subject, and a second filter that passes light in specific wavelength bands included in light in multiple wavelength bands by a control signal. A detection unit that detects a first image signal obtained by photoelectrically converting light in the multiple wavelength band and a second image signal obtained by photoelectrically converting light in a specific wavelength band; and a second image signal from the first image signal. An image processing apparatus that generates a color image and a depth image using the removed difference image signal and the second image signal is disclosed.

  However, when the optical image sensor and its imaging optical system, and the area image sensor and its imaging optical system are provided side by side in one apparatus, the shooting position in each imaging optical system will shift, so at the pixel level of the optical image The matched distance information cannot be acquired. In the technique of Patent Document 3, since the same pixel by time-division switching is shared for acquisition of the optical image and distance information, one distance information is obtained for a plurality of pixels of the optical image. It is not possible to acquire distance information that matches at the pixel level.

  As described above, conventionally, the distance information to the subject cannot be recorded in the pixel information of the high-definition optical image photographed by the imaging apparatus.

  SUMMARY OF THE INVENTION An object of the present invention is to provide an imaging apparatus capable of recording distance information to a subject in pixel units in pixel information of an optical image photographed by the imaging apparatus.

  In order to achieve such an object, an imaging apparatus according to the present invention is an imaging apparatus including an imaging optical system that forms an image of a subject on a sensor, and a light emitting unit that irradiates the subject. An optical sensor that captures at least visible light, and a three-dimensional sensor that measures a distance to the subject, and is imaged by the imaging optical system using either the optical sensor or the three-dimensional sensor. After the captured image is picked up, an image pickup control means for picking up an image by displacing the other sensor at the image forming position by the image pickup optical system is provided.

  According to the present invention, distance information to a subject can be recorded in pixel units in pixel information of an optical image photographed by an imaging device.

It is a section lineblock diagram showing an example of an imaging device. It is a flowchart which shows an example of the imaging | photography process by an imaging device. (A) The light-receiving part of an optical image sensor, (b) It is a schematic diagram which shows the light-receiving part of a TOF image sensor. It is explanatory drawing of the array pattern of a Bayer array. It is a figure which shows an example of the data file structure of the optical image to which distance information was provided. It is a block diagram which shows an example of a function structure of a TOF image sensor. It is a block diagram which shows an example of a function structure of a light projection system. It is an example of the timing chart of a pulse control signal. It is an example of the timing chart of a light source drive signal. It is a block diagram which shows an example of a function structure of a light reception type | system | group. It is explanatory drawing of transmission / reception of the signal between a three-dimensional information acquisition part and a light-receiving part. It is the schematic of 1 frame in a TOF image sensor. It is a principal part block diagram of an omnidirectional imaging device. It is a top view of an omnidirectional imaging device. It is a flowchart which shows an example of the imaging | photography process by an omnidirectional imaging device. It is a schematic side view of an omnidirectional imaging device. It is the schematic which shows an example of the light source of an omnidirectional imaging device. It is explanatory drawing of the irradiation angle of a light source. It is explanatory drawing of the imaging | photography process by an omnidirectional imaging device.

  Hereinafter, the configuration according to the present invention will be described in detail based on the embodiment shown in FIGS.

[First Embodiment]
An imaging apparatus 1 that is an embodiment of an imaging apparatus according to the present invention will be described. The imaging apparatus according to the present embodiment includes an imaging optical system (imaging optical system 5) that forms an image of a subject on a sensor, and a light emitting unit (light source 21) that irradiates the subject (imaging apparatus 1). The sensor includes at least an optical sensor (optical image sensor 10) that captures visible light and a three-dimensional sensor (TOF image sensor 20) that measures the distance to the subject. An image pickup control means for picking up an image formed by the image pickup optical system using any one of the dimension sensors and then displacing the other sensor at the image formation position by the image pickup optical system is provided. . In addition, the code | symbol in embodiment and the example of application are shown in a parenthesis.

(Imaging device configuration)
FIG. 1 is a cross-sectional configuration diagram illustrating an example of an imaging apparatus 1 according to the present embodiment. The imaging device 1 includes a housing 3, a substrate 4, an optical image sensor 10, a TOF image sensor 20, an imaging optical system 5 (shown simply as a lens 5), and a light source 21.

  An optical image sensor 10 as an image pickup device for taking an image and a TOF image sensor 20 as a light receiving element for distance acquisition are fixedly provided on the substrate 4 disposed in the housing 3. Is supported by a means for supporting the substrate 4 so as to be displaceable, and its displacement is controlled by the control means (imaging control means) of the imaging apparatus 1, and the position shown in FIG. It is possible to reciprocate between the positions shown in (2). The optical image sensor 10 is, for example, a CMOS image sensor. In this embodiment, the case where the TOF image sensor 20 is used as the three-dimensional sensor will be described. However, other three-dimensional sensors can be applied. Details of each image sensor will be described later.

  When shooting with the imaging device 1, first, an image with visible light is shot with the optical image sensor 10 in the state shown in FIG. Next, as shown in FIG. 1B, the substrate 4 is displaced (referred to as a second state) so that the TOF image sensor 20 comes to substantially the same position taken by the optical image sensor 10. Then, the pulsed light is emitted from the light source 21 and imaging is performed again. After photographing using the TOF image sensor 20, the substrate 4 is displaced and returned to the first state again.

(Shooting process)
FIG. 2 is a flowchart illustrating an example of a shooting process performed by the imaging apparatus 1. In this imaging process, the imaging apparatus 1 is fixed by a fixing means such as a tripod in order to image the two sensors at substantially the same position by displacing the substrate 4 during the imaging process. Further, the substrate 4 in this photographing process has the first state (FIG. 1A) as the initial position.

  When receiving an imaging instruction such as pressing the release button from the operator (S101), the imaging apparatus 1 performs imaging using the optical image sensor 10 in the first state (S102), and the optical image sensor 10 performs optical imaging. (Optical image) is acquired (S103).

  Next, the substrate 4 is displaced into the second state (FIG. 1B) (S104), and photographing using the TOF image sensor 20 in the second state is performed (S105). At this time, pulsed light is emitted from the light source 21 (S106), and the TOF image sensor 20 acquires a TOF image (TOF image) (S107).

  Next, control for associating the acquired optical image with the position of the pixel of the TOF image is executed (S108), and distance information is given to each pixel of the optical image (S109). In the present embodiment, an example is described in which shooting using the optical image sensor 10 is performed first, and then shooting using the TOF image sensor 20 is performed. However, shooting using the TOF image sensor 20 is performed first. Thereafter, photographing using the optical image sensor 10 may be performed.

(Corresponding pixel position between sensors)
In the photographing by the imaging apparatus 1 according to the present embodiment, the optical image sensor 10 and the TOF image sensor 20 are photographed at substantially the same position, so the same place (field of view) is photographed. Therefore, the pixels of the two sensors can correspond to the same place. As a result, the distance information measured by the TOF image sensor 20 can be recorded in correspondence with each pixel of the optical image captured by the optical image sensor 10.

  Here, if the sensor size and the number of pixels of the optical image sensor 10 and the TOF image sensor 20 can be made the same, the pixels can be made to correspond one-to-one. The number of pixels of the image sensor 20 is smaller than the number of pixels of the optical image sensor 10. For this reason, when the sensor size and the number of pixels of the optical image sensor 10 and the TOF image sensor 20 are different, it is necessary to control the pixels of the optical image sensor 10 and the TOF image sensor 20 to correspond to each other.

  Control (S108) for associating the positions of the pixels of the two sensors will be described by taking as an example a case where the number of pixels of the optical image sensor 10 is four times that of the TOF image sensor 20.

  FIG. 3A is a schematic diagram illustrating a part of the light receiving unit 19 of the optical image sensor 10, and FIG. 3B is a schematic diagram illustrating a part of the light receiving unit 29 of the TOF image sensor 20. As shown in FIG. 3, the light receiving unit 19 of the optical image sensor 10 and the light receiving unit 29 of the TOF image sensor 20 are configured by two-dimensionally arraying light receiving elements (for example, photodiodes and phototransistors), An image is created based on the light information entered from the light receiving element. One grating portion shown in FIG. 3 represents one light receiving element (that is, a pixel). Here, the number of pixels of the optical image sensor 10 is 2048 (horizontal) × 1024 (vertical), for example, a TOF image sensor 20. It is assumed that the number of pixels is 1024 (horizontal) × 512 (vertical).

  Here, the four pixels in the shaded portion of the optical image sensor 10 shown in FIG. 3A correspond to one pixel in the shaded portion of the TOF image sensor 20 shown in FIG. By assigning the distance information measured by one pixel of the TOF image sensor 20 to these four pixels, the distance information can be assigned to each pixel of the optical image sensor 10.

  FIG. 4 is an explanatory diagram of an array pattern of the Bayer array. In the Bayer arrangement, red (R), green (G), and blue (B), which are the three primary colors, are arranged in units of R, G, G, and B. When a Bayer color filter is used for the optical image sensor 10, four pixels are the smallest unit on the color image. Therefore, as shown in FIG. 3, the TOF image is applied to the four pixels of the optical image sensor 10. By assigning the distance information measured by one pixel of the sensor 20, the distance information can be assigned to each pixel constituting the color image.

(data structure)
After correspondence control of the pixels of the two sensors (S108), distance information is given to the data of each pixel of the optical image.

  FIG. 5 is a diagram illustrating an example of a data file structure of an optical image to which distance information is added. The data file 30 includes header data 31, image data 32, and distance data 33. The image data 32 is, for example, data indicating a RAW image in a Bayer array, and the distance data 33 is, for example, a distance at each pixel. It is binary data.

(TOF image sensor)
Next, the TOF image sensor 20 which is a preferable example of a three-dimensional sensor will be described. The TOF image sensor emits modulated reference light (pulse light from the light source 21), and detects the time until the reference light is reflected by an object (subject) to be measured and returned. , To measure the distance to the subject.

  Note that TOF is being developed for various uses because of its responsive principle, among various 3D sensing methods. For example, gesture recognition and the position of mobile objects such as robots and automobiles are being developed. Application to control is expected. In addition, as the TOF, a direct TOF method that directly detects the time difference between the light emission timing and the light reception timing of the reference light and an indirect TOF method that detects the time difference from the calculation using the light reception signal are known. The indirect TOF method is considered to be more advantageous.

  FIG. 6 is a block diagram illustrating an example of a functional configuration of the TOF image sensor 20. The TOF image sensor 20 includes a light projecting system 201, a light receiving system 202, and a three-dimensional information acquisition unit 203. The three-dimensional information acquisition unit 203 is connected to the control unit of the imaging apparatus 1 via a bus 204. ing.

  Each unit of the light projecting system 201, the light receiving system 202, and the three-dimensional information acquisition unit 203 is housed in, for example, the housing of the TOF image sensor 20, and light is projected from the light projecting system 201 to the housing. And a window through which light reflected by an object and light directed toward the light receiving system 202 pass, and glass is attached to the window.

  FIG. 7 is a block diagram illustrating an example of a functional configuration of the light projecting system 201. The light projecting system 201 is disposed on the −Z side of the light receiving system 202 and includes the light source 21 and the light source driving unit 25.

  The light source 21 is arranged so as to emit light in the + X direction, and lighting on and off are controlled by the light source driving unit 25. As the light source 21, for example, an LED can be used. However, the light source 21 is not limited to this, and other light sources such as an edge emitting laser and a semiconductor laser such as a surface emitting laser can also be used.

  FIG. 8 is an example of a timing chart of the pulse control signal. FIG. 9 is an example of a timing chart of the light source drive signal. A pulse control signal (FIG. 8) sent from the three-dimensional information acquisition unit 203 is input to the light source driving unit 25, and the light source driving unit 25 drives the light source for driving the light source 21 based on the pulse control signal. A signal (FIG. 9) is generated. The light source drive signal (drive pulse) is sent to the light source 21 and the three-dimensional information acquisition unit 203.

  Thereby, the light source 21 emits pulsed light having a pulse width instructed from the three-dimensional information acquisition unit 203. When there are a plurality of light sources, each light source is controlled as pulsed light having the same pulse width. The pulsed light emitted from the light source 21 is set in the three-dimensional information acquisition unit 203 so that the duty is 50% or less.

  A part of the light emitted from the light source 21 of the light projecting system 201 and reflected by the object returns to the TOF image sensor 20. The light receiving system 202 detects light (reflected light from the object) that is reflected by the object and returns to the TOF image sensor 20.

  FIG. 10 is a block diagram illustrating an example of a functional configuration of the light receiving system 202. The light receiving system 202 is disposed on the + Z side of the light projecting system 201 and includes the imaging optical system 28 and the light receiving unit 29.

  The imaging optical system 28 is disposed on the optical path of the reflected light from the object, and collects the reflected light from the object. In the example of FIG. 10, the example in which the imaging optical system 28 is configured by a single lens is shown, but it may be configured by a plurality of lenses or a mirror optical system.

  The light receiving unit 29 receives the reflected light from the object via the imaging optical system 28, converts it into an electrical signal, divides the electrical signal in time, and distributes it to a plurality of signals for each time.

  The three-dimensional information acquisition unit 203 calculates a time difference between the light emission timing of the light source 21 and the light reception timing of the light receiving unit 29 using a signal input from the light receiving unit 29, and calculates a distance to the object based on the time difference. Thus, the three-dimensional information of the object is acquired.

  FIG. 11 is an explanatory diagram of signal transmission / reception between the three-dimensional information acquisition unit 203 and the light receiving unit 29. The three-dimensional information acquisition unit 203 includes a modulation frequency control unit 203a that controls a modulation frequency that is a frequency of a drive pulse applied to the light source 21, and an object based on a time difference between the emission timing of the light source 21 and the light reception timing of the light receiving unit 29. A distance image generation unit 203b that generates a distance image that is three-dimensional information of the object by calculating the distance to the object, and the three-dimensional information acquisition unit 203 includes a TX1 signal, a TX2 signal, a TXD signal, and a reset signal. Is output to the light receiving unit 29.

  The light receiving unit 29 is an area image sensor (FIG. 3B) in which pixels are two-dimensionally arranged, and receives light reflected from an object via the imaging optical system 28. The output signal (analog signal) of the light receiving unit 29 is converted into a digital signal by an analog-digital converter (ADC) and sent to the three-dimensional information acquisition unit 203.

  The light receiving unit 29 has two charge storage units for one light receiving element. When the TX1 signal is at a high level, the charge photoelectrically converted by the light receiving system 202 is stored in one charge storage unit. When the TX2 signal is at the high level, the charge photoelectrically converted by the light receiving system 202 is accumulated in the other charge accumulation unit.

  The light receiving unit 29 does not accumulate charges when the TXD signal is at a high level, and sets the amount of charges accumulated in the two charge accumulation units to 0 when the reset signal is at a high level. As described above, the image sensor for obtaining distance information has a structure different from that of a normal optical image sensor, and it is not easy to incorporate the image sensor on the same sensor.

  FIG. 12 is a schematic diagram of one frame in the TOF image sensor 20. The TOF image sensor 20 basically performs two or more TOF detections (detection using a TOF calculation method) having different modulation frequencies, and acquires distance information with high distance resolution by high-frequency TOF detection. In the low frequency side TOF detection, the aliasing component included in the high frequency side TOF detection information is determined to perform de-aliasing (reference document: Patent Document 1).

  Here, an example in which one frame is configured by two subframes using different modulation frequencies will be described. The modulation frequency (frequency of one drive pulse) in one subframe is a first frequency f1, and the modulation frequency (frequency of another drive pulse) in the other subframe is a second frequency f2.

  The three-dimensional information acquisition unit 203 performs TOF detection at the first frequency f1, and stores the phase information in the memory in the distance image generation unit 203b. Next, the modulation frequency control unit 203a switches from the first frequency f1 to the second frequency f2, and performs TOF detection using the second frequency f2. Here, a relationship of the first frequency f1> the second frequency f2 is assumed. Needless to say, TOF detection at the first frequency f1 may be performed after the TOF detection at the second frequency f2.

  The three-dimensional information acquisition unit 203 supplies the light source 21 with the distance to the object calculated from the time difference obtained in the first time zone in which the first drive pulse is supplied to the light source 21 and the second drive pulse. The three-dimensional information is calculated using the distance to the object calculated from the time difference obtained in the second time zone.

  At this time, three or more drive pulses having different pulse widths may be supplied to the light source 21 in different time zones. In this case, three or more drive pulses having different pulse widths are divided into a side having a larger pulse width and a side having a smaller pulse width, the drive pulse having a smaller pulse width is formed into a sine wave, and the drive pulse having a larger pulse width is formed into a rectangular waveform. It is preferable to do.

  The imaging apparatus 1 according to the present embodiment described above is a distance information simultaneous acquisition type imaging apparatus 1 capable of recording accurate distance information in association with each pixel of an optical image captured by the optical image sensor 10. can do.

  Thereby, for example, by making each pixel of the image image and the TOF image in the same shooting range correspond to each other, the color image of the image can be displayed as a color image corresponding to the distance. At this time, since a color image corresponding to the distance can be added to the image image in units of pixels, the distance image can be of high quality like the picture quality.

  In addition, since accurate distance information from the shooting position can be converted into space coordinates, the space of the optical photograph can be recognized in 3D coordinates. For this reason, for example, spatial data can be easily input to software such as three-dimensional CAD (computer-aided design).

  In the present embodiment, the light source 21 has a single light emitting point (LD or LED), pulsed light is emitted from this, and the reflected light from the object is received by the TOF image sensor 20. For example, the light source 21 has a plurality of light emitting points arranged two-dimensionally, sequentially turns on the plurality of light emitting points, and receives light reflected from an object of each light emitting pulse sequentially by a single light receiving system 202. A distance image may be generated.

  In the present embodiment, the non-scanning light projecting system 201 has been described as an example. However, the light projecting system 201 includes a scanning light projecting device (for example, a polygon mirror, a galvano mirror, a MEMS mirror, etc.). The optical system 201 may be used. In this case, for example, a plurality of lights respectively emitted from the line light sources arranged in one direction are scanned in a direction (for example, a vertical direction) that is non-parallel to the arrangement direction of the line light sources to correspond to the line light sources. Thus, a plurality of light receiving systems 202 arranged in parallel to the arrangement direction can receive light to generate a distance image. Alternatively, the distance image may be generated by two-dimensionally scanning the light from a single light emitting point with the light deflecting means and receiving the reflected light from the object with the TOF image sensor 20.

  In the present embodiment, the imaging apparatus 1 provided with one optical image sensor 10 and one TOF image sensor 20 as the imaging sensor has been described as an example. However, the imaging apparatus 1 includes the optical image sensor 10 and the optical image sensor 10. Any one having at least one TOF image sensor 20 may be used, and three or more imaging sensors may be provided and these may be displaced to the imaging position of one imaging optical system 5.

[Second Embodiment]
Hereinafter, other embodiments of the imaging device according to the present invention will be described. In addition, the description about the same point as the said embodiment is abbreviate | omitted suitably.

  The imaging apparatus according to the second embodiment includes two or more imaging optical systems (imaging optical systems 5A and 5B) that form an image of a subject on a sensor, a light emitting unit (light source 21) that irradiates the subject, and a sensor. An omnidirectional image capturing device (omnidirectional image capturing device 2) including an image capturing control unit that obtains an image within a solid angle of 4π radians from the captured image, and the sensor captures at least visible light. An optical sensor (optical image sensor 10), and a three-dimensional sensor (TOF image sensor 20) that measures the distance to the subject. In addition, the code | symbol in embodiment and an application example are shown in a parenthesis.

  In recent years, as a camera technology for performing special optical image recording, there are two imaging optical systems including a wide-angle lens having an angle of view wider than 180 degrees and an optical image sensor that captures an image by this wide-angle lens. There is known an omnidirectional image pickup device that is combined so as to be reversed and synthesizes images taken by the respective image pickup optical systems to obtain an “image within a solid angle of 4π radians” (reference: Japanese Patent Laid-Open 2010-271675). In such an omnidirectional imaging apparatus, image information in all directions can be acquired at the same time, and can be effectively used for, for example, a surveillance camera for security or a vehicle-mounted camera.

  In this omnidirectional imaging apparatus, if distance information can be acquired as described in the first embodiment, distance information can be recorded in pixels of an omnidirectional image image while obtaining a high-definition photograph. It becomes possible. Hereinafter, a case where the imaging device is an omnidirectional imaging device will be described.

(Spherical imaging device configuration)
FIG. 13 is a configuration diagram of a main part of an omnidirectional imaging device 2 which is another embodiment of the imaging device according to the present invention. The omnidirectional imaging device 2 includes two imaging optical systems 5A and 5B. The imaging optical systems 5A and 5B have the same structure except for the imaging sensor. Hereinafter, the components constituting the imaging optical system 5A are distinguished by adding A at the end of the reference numerals, and those constituting the imaging optical system 5B by adding B at the end of the reference numerals.

  The omnidirectional imaging device 2 has the TOF image sensor 20 installed as one of the imaging optical systems 5A and 5B, and the optical image sensor 10 installed as the other imaging sensor. Here, a case where the TOF image sensor 20 is installed in the imaging optical system 5A and the optical image sensor 10 is installed in the imaging optical system 5B will be described.

  The imaging optical system 5A will be described. The imaging optical system 5A includes a wide-angle lens 6A having a field angle wider than 180 degrees, and a TOF image sensor 20 that captures an image by the wide-angle lens.

  The TOF image sensor 20 is provided on the substrate 4A. The substrate 4A is fixed and has circuit elements for driving the TOF image sensor 20 and circuit elements necessary for electrical connection with the outside, and is larger in size than the TOF image sensor 20.

Wide-angle lens 6A is front group 6A ₁ of negative power, the prism 6A ₂ forming the substance of the reflection surface, a positive arranged group 6A ₃ after power, the rear group 6A of the optical axis LA ₁ of the front group by the reflecting surface It is bent 90 degrees toward ₃ . Note that the front group 6A ₁ and the rear group 6A ₃ are composed of a plurality of lenses, but in FIG. 13, they are simplified and shown as one lens.

Light from the object side is incident on the front unit 6A _1, it is reflected by the reflecting surface of the prism 6A _2, folded optical axis 90 degrees towards the rear unit 6A3. The light beam reflected by the reflecting surface of the prism 6A ₂ is incident on the rear unit 6A ₃ along the rear group optical axis LA _2, when transmitted through the rear group 6A _3, is formed on the light receiving surface of the TOF image sensor 20 .

  The imaging optical system 5B will be described. The imaging optical system 5B includes a wide-angle lens having a field angle wider than 180 degrees and an optical image sensor 10 that captures an image by the wide-angle lens 6B.

  The optical image sensor 10 is provided on the substrate 4B. The substrate 4B is fixed and has circuit elements for driving the optical image sensor 10 and circuit elements necessary for electrical connection with the outside, and is larger in size than the optical image sensor 10.

The wide-angle lens 6B includes a negative power front group 6B ₁ , a prism 6B ₂ that forms a reflective surface, and a positive power rear group 6B _3, and the front group optical axis LB ₁ is reflected by the reflective surface to the rear group 6B. It is bent 90 degrees toward ₃ . The front group 6B ₁ and the rear group 6B ₃ are composed of a plurality of lenses, but in FIG. 13, they are each shown as a single lens in a simplified manner.

Light from the object side is incident on the front unit 6B _1, is reflected by the reflecting surface of the prism 6B _2, folded optical axis 90 degrees towards the rear unit 6B3. The light beam reflected by the reflecting surface of the prism 6B ₂ is incident on the rear group 6B ₃ along the rear group optical axis LB ₂ and passes through the rear group 6B ₃ , so that an object image is formed on the light receiving surface of the optical image sensor 10. Form an image.

The two imaging optical systems 5A and 5B described so far are combined as shown in FIG. That is, the front group optical axes LA ₁ and LB ₁ of the front groups 6A ₁ and 6B ₁ are made to coincide with each other so that the direction of the front group is reversed (in FIG. 13, the front group 6A ₁ of the imaging optical system 5A is facing right, and the imaging optics front group 6B ₁ is left.) as the system 5B, and the rear group _6A 3, 6B ₃ of the optical axis _LA 2, LB ₂ are parallel to each other, the orientation between the rear group are opposite to each other (in FIG. 13, group 6A ₃ after the imaging optical system 5A is upward, the group 6B ₃ after the imaging optical system 5B are combined to downward) so.

  The omnidirectional imaging device 2 having the two imaging optical systems 5A and 5B can simultaneously photograph a reverse TOF image and an optical image having different 180 ° directions (referred to as first imaging). Then, after this first shooting, the omnidirectional imaging device 2 is rotated 180 ° to shoot, thereby simultaneously shooting a TOF image and an optical image that are 180 ° different from the first shooting. (Second shooting).

  By combining the images acquired in the first and second imaging, an image within a solid angle of 4π radians of the optical image (that is, an omnidirectional image) can be obtained, and the TOF image is also 4π. An image within a solid angle of radians can be obtained. And since each is image | photographing in the substantially same place, it becomes possible to make each pixel which image | photographed the same place respond | correspond.

  FIG. 14 is a top view of the omnidirectional imaging device 2. Since it is necessary to project pulsed light from the light source 21 on the photographing side of the TOF image sensor, the omnidirectional imaging device 2 has a light source 21 that emits pulsed light on the photographing side of the TOF image sensor 20. doing. As shown in FIG. 14, when the light source 21 is provided on both sides of the omnidirectional image pickup device 2 and the photographing is performed by rotating 180 degrees, the range where the light source 21 is provided (shaded portion) becomes a blind spot and becomes a omnidirectional sky. The ball image is kicked (the light source 21 is reflected). This point will be described later.

(Shooting process)
FIG. 15 is a flowchart illustrating an example of an imaging process performed by the omnidirectional imaging apparatus 2. In this photographing process, since the omnidirectional imaging device 2 is rotated by 180 ° during the photographing processing, the omnidirectional imaging device 2 is fixed to a fixing means such as a tripod, and further. It is fixed to a rotation control means such as a rotary head for accurately rotating a desired angle.

  When the omnidirectional imaging device 2 receives a shooting instruction such as pressing the release button from the operator (S201), the omnidirectional imaging device 2 performs shooting (first shooting) (S202). In photographing, pulse light is emitted from the light source 21 (S203), the optical image sensor 10 acquires an optical image, and the TOF image sensor 20 acquires a TOF image at a position different from the optical image by 180 ° (S204). .

  Next, after the rotary head is rotated 180 ° by the rotation control means (S205), photographing (second photographing) is performed (S206).

  The rotary head is preferably rotated by an automatic rotary head that can be controlled to rotate at an arbitrary angle as the rotary head. However, the user does not use the automatic rotary head and the user manually rotates the head. Also good. In the case of manual operation, the omnidirectional imaging device 2 notifies that the rotary head needs to be rotated by a predetermined angle when the first imaging is completed. As the rotating pan head at this time, it is preferable to use a rotating pan head having a scale or a fixing function for being able to be rotated and fixed accurately at a desired angle.

  In the second imaging, pulse light is emitted from the light source 21 (S207), the optical image sensor 10 acquires an optical image, and the TOF image sensor 20 acquires a TOF image at a position 180 ° different from the optical image. (S208).

  The optical image and the TOF image acquired in the first shooting and the second shooting are combined to create an omnidirectional image (S209).

  Next, the acquired optical image and the position of the pixel of the TOF image are associated with each other (S210), and distance information is given to each pixel of the optical image (S211).

  In this embodiment, the example has been described in which the omnidirectional imaging device 2 supported by a rotatable support means such as a tripod head is rotated by 180 °, but the housing of the omnidirectional imaging device 2 has been described. The imaging optical systems 5A and 5B may be rotated 180 ° inside the body 3. In this case, if the light source 21 of the omnidirectional imaging device 2 can irradiate only the photographing side (one side) of the TOF image sensor 20, it is necessary to rotate the light source 21 together with the rotation of the TOF image sensor 20. The light source 21 can be fixed by using the light source 21 that can irradiate the photographing side (both sides) of both sensors.

  According to the omnidirectional imaging device 2 according to the second embodiment described above, accurate distance information is recorded in association with each pixel of the optical image within the solid angle of 4π radians photographed by the optical image sensor. The distance information simultaneous acquisition type omnidirectional imaging device can be obtained.

  Further, as described above, since accurate distance information from the photographing position can be converted into spatial coordinates, the spatial coordinate data of the celestial sphere and a high-definition optical image can be easily obtained. For example, when it is necessary to acquire information such as the arrangement and dimensions of piping, wiring, ancillary facilities, structural members, etc. when renovating or reinforcing a building, In addition to grasping the situation based on the panoramic image of the azimuth, it is possible to perform dimension measurement and drawing creation.

  In the second embodiment, the imaging optical system is two imaging optical systems having the same structure each having a wide-angle lens having a field angle wider than 180 degrees, and each of the optical image sensor 10 and the TOF image sensor 20 is provided. However, when the angle of view of each imaging system is narrower than 180 degrees, etc., it has three or more imaging optical systems and at least one of the optical image sensor 10 and the TOF image sensor 20. It may be a thing.

[Third Embodiment]
As described above, in order to take a TOF image, the light source 21 is required. However, in the omnidirectional imaging apparatus 2, if the omnidirectional sphere is omnidirectionally radiated, the size of the light source 21 is large. Thus, many of the omnidirectional shooting angles of view are hidden by the light source 21.

  In contrast, as described in the second embodiment, two imaging optical systems having the same structure are combined, one is an optical image sensor, and the other is a TOF image sensor. It is only necessary to irradiate the hemispherical surface on the imaging side (FIG. 14). Thereby, since the scale of the light source 21 can be reduced, the portion hidden by the light source 21 at the time of omnidirectional photographing becomes very small, and the omnidirectional imaging device 2 can be reduced in weight and slim. it can.

  However, as shown in FIG. 14, even when a hemispherical surface is irradiated, there is a range that is partially hidden by the light source 21 during omnidirectional shooting. Therefore, in the present embodiment, the omnidirectional imaging apparatus 2 is rotated twice or more to shoot, and the captured images are combined to obtain an omnidirectional captured image without a portion kicked by the light source 21. is there.

  FIG. 16 is a schematic horizontal view of the omnidirectional imaging device 2 according to the present embodiment. FIG. 17 is a schematic diagram illustrating an example of the light source 21 of the omnidirectional imaging device 2, which is (a) a side view, (b) a top view, and (c) a front view. FIG. 18 is an explanatory diagram of the irradiation angle of the light source 21 shown in FIG.

  The light sources 21 are provided on both sides in the photographing direction on the TOF image sensor 20 side, and the light emitting units 22 are capable of emitting pulsed light evenly from the upper side to the lower side of the omnidirectional imaging device 2. A plurality of light emitting points 23 are arranged in a semicircular shape in the vertical direction. Here, an example is shown in which the installation angles of the light emitting points 23 are shifted by 36 ° in the vertical direction with respect to the horizontal direction. In addition, a diffusion plate 24 that diffuses the pulsed light is provided in front of the light emitting point 23, and the pulsed light from the light emitting point 23 is made uniform by the diffusion plate 24, so that the omnidirectional imaging device 2 can be viewed from above. It becomes possible to irradiate the pulsed light evenly to the lower side.

  Further, in this embodiment, two rows of light emitting units 22 each having light emitting points 23 arranged in the vertical direction are arranged, and the two rows of light emitting units 22 are arranged at an angle of 60 degrees when viewed from the top, and one row of light emission. The portion 22 irradiates left and right at an angle of 30 °. As a result, the light source 21 projects pulsed light at an imaging angle of 120 ° from the upper surface.

  Note that it is desirable that the light emitting points 23 of the light emitting unit 22 have a sufficient number to illuminate the entire celestial sphere as evenly as possible, and it is preferable that there are at least eight light emitting points on the left and right sides.

  In addition, the light sources 21 are preferably provided on both sides of the shooting direction on the TOF image sensor 20 side, but the present invention is not limited to this. It is good also as a structure which installs and irradiates.

  FIG. 19 is an explanatory diagram of the photographing process by the omnidirectional imaging device 2 according to the present embodiment, where (a) is the first photographing, (b) is the second photographing, and (c) is It is a top view of the omnidirectional imaging device 2 at the time of 3rd imaging | photography. It is assumed that the angle of view kicked by the two light sources 21 provided on both sides in the photographing direction on the TOF image sensor side of the omnidirectional imaging device 2 is less than 60 °.

  First, the omnidirectional imaging device 2 executes the first imaging in the direction shown in FIG. Next, the rotating pan head is rotated by 120 °, and the second imaging is executed at the position shown in FIG. Further, the rotating pan head is rotated by 120 ° from the position, and the third photographing is executed as the position shown in FIG. Then, three images with an angle of view of 120 ° are combined to create an omnidirectional image.

  If the image is shot twice with a rotation of 180 °, there will be an angle of view kicked by the two light sources 21, but the image will be rotated three times or more and executed three times or more as in this embodiment. By doing so, it is possible to acquire an image image and a TOF image for all the angles of view. Then, by synthesizing these, an omnidirectional image for all angles of view can be created. By setting the rotation angle and the number of times of shooting according to the angle of view kicked by the light source 21, it becomes possible to create an omnidirectional image for all angles of view regardless of the angle of view kicked by the light source 21. .

  In addition, when combining a plurality of images into an omnidirectional image, in principle, it is necessary to have overlapping portions. That is, in the above example, it is necessary to synthesize an image with a shooting angle of 120 ° or more in one shooting, but in this specification, in order to simplify the description, the angle of view of 120 ° is obtained. As described.

  The above-described embodiment is a preferred embodiment of the present invention, but is not limited thereto, and various modifications can be made without departing from the gist of the present invention.

DESCRIPTION OF SYMBOLS 1 Imaging device 2 Omnispherical imaging device 3 Case 4, 4A, 4B Substrate 5, 5A, 5B Imaging optical system (lens)
6A, 6B Wide-angle lens 6A ₁ , 6B ₁ Front group 6A ₂ , 6B ₂ Prism 6A ₃ , 6B ₃ Rear group LA ₁ , LB ₁ Front group optical axis LA ₂ , LB ₂ Rear group optical axis 10 Optical image sensor ( Optical sensor)
19 Light-receiving part 20 TOF image sensor (three-dimensional sensor)
21 light source 22 light emitting unit 23 light emitting point 24 diffuser plate 25 light source driving unit 28 imaging optical system 29 light receiving unit 30 data file 31 header data 32 image data 33 distance data 201 light projecting system 202 light receiving system 203 three-dimensional information obtaining unit 203a modulation Frequency control unit 203b Distance image generation unit 204 Bus

JP-T-2015-501927 Special table 2013-538342 gazette JP2015-70619A

Claims (10)

An imaging optical system for forming an image of a subject on a sensor;
An imaging device comprising: a light emitting unit that irradiates the subject;
The sensor includes at least an optical sensor that captures visible light, and a three-dimensional sensor that measures a distance to the subject.
Imaging in which either the optical sensor or the three-dimensional sensor is used to capture an image formed by the imaging optical system, and then the other sensor is displaced to an imaging position by the imaging optical system. An imaging apparatus comprising a control means. Two or more imaging optical systems for forming an image of a subject on a sensor;
A light emitting unit for illuminating the subject;
An omnidirectional imaging device comprising imaging control means for obtaining an image within a solid angle of 4π radians from an image captured by the sensor,
The image pickup apparatus includes: an optical sensor that picks up at least visible light; and a three-dimensional sensor that measures a distance to the subject. The two or more imaging optical systems are two imaging optical systems having the same structure each having a wide-angle lens having a field angle wider than 180 degrees,
The optical sensor is provided at an imaging position by one of the two imaging optical systems,
The imaging apparatus according to claim 2, wherein the three-dimensional sensor is provided at an imaging position formed by the other of the two imaging optical systems. A rotation control means for rotating the imaging device by a predetermined angle;
The image pickup control unit causes the image pickup apparatus to pick up an image at the same time using the optical sensor and the three-dimensional sensor while rotating the image pickup device at a predetermined angle by the rotation control unit. Imaging device.   The imaging device according to claim 2, wherein the light emitting unit is provided so as to be able to irradiate an imaging range side by the three-dimensional sensor. The optical sensor and the three-dimensional sensor are provided on one member,
The imaging apparatus according to claim 1, wherein the imaging control unit moves the optical sensor and the three-dimensional sensor to an imaging position by the imaging optical system by displacing the member.   The imaging control means stores distance information obtained by imaging by the three-dimensional sensor in association with pixel information of each pixel of the optical sensor that has imaged substantially the same range. The imaging device according to any one of the above.   The imaging apparatus according to claim 1, wherein the three-dimensional sensor is a TOF (Time of Flight) image sensor. An imaging optical system for forming an image of a subject;
An imaging method using an imaging device including a light emitting unit that irradiates the subject,
Imaging an optical sensor for imaging visible light so as to be positioned at an imaging position by the imaging optical system;
Illuminating the subject with the light emitting unit, and imaging a three-dimensional sensor for measuring the distance to the subject so as to be positioned at an imaging position by the imaging optical system. Method. Two or more imaging optical systems for forming an image of a subject on a sensor;
An imaging method using an omnidirectional imaging device including a light emitting unit that irradiates the subject,
The sensor includes at least an optical sensor that captures visible light and a three-dimensional sensor that measures a distance to the subject, and the light-emitting unit uses the three-dimensional sensor on the imaging range side of the three-dimensional sensor. Is provided at the time of imaging
Simultaneously imaging using the optical sensor and the three-dimensional sensor;
And a step of repeating the simultaneous imaging using the optical sensor and the three-dimensional sensor while rotating the imaging device at a predetermined angle after the step.

Images