JP7025811B1 - Imaging device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve visibility of an imaging device in a dark place.
SOLUTION: This imaging device separates a light condensing unit that condenses light from the outside and light incident through the condensing unit into first light including visible light and far infrared light. When the first beam splitter and the first light are received, it corresponds to the near-infrared light signal corresponding to the near-infrared light of the first light and the visible light of the first light. A far-infrared light sensor that generates a far-infrared light signal corresponding to the far-infrared light when it receives a far-infrared light separated by a generator that generates a visible light signal and the first beam splitter. A visible light image processing unit that generates a visible light image corresponding to the visible light signal, a near infrared thermal image processing unit that generates a near infrared thermal image corresponding to the near infrared light signal, and a far infrared image processing unit. It is characterized by including a far-infrared thermal image processing unit that generates a far-infrared thermal image corresponding to an infrared light signal.
[Selection diagram] Fig. 1

Description

The present invention relates to an image pickup device and a beam splitter.

Conventionally, in a surveillance system, lighting is installed at night or in a dark place to brighten the area to be monitored, or a far-infrared optical camera that detects the heat of a subject is adopted as a surveillance camera.

Japanese Unexamined Patent Publication No. 2011-232606

When installing surveillance cameras and lighting at night, problems such as deterioration of the night environment and increase in the cost of installing lighting may occur. In addition, when no lighting is installed, the image recognition technology using a visible light camera that shoots with visible light has extremely low detection accuracy at night or in dark places, and false detection is caused by the influence of shadows even in the daytime. was there. When a far-infrared light camera is used for surveillance, the far-infrared light camera cannot obtain a high-definition image as high as that of a visible light camera, which may cause a decrease in visibility in surveillance, and a surveillance system including a surveillance camera. There is a risk of complicating the system configuration of.

One aspect of the disclosed technique is to generate a highly visible image even in a dark place.

One aspect of the disclosed technique is exemplified by the following imaging devices. This image pickup apparatus has a condensing unit that condenses light from the outside, and a first beam that separates light incident through the condensing unit into first light including visible light and far-infrared light. When the splitter and the first light are received, a near-infrared light signal corresponding to the near-infrared light of the first light and a visible light signal corresponding to the visible light of the first light are obtained. A far-infrared light sensor that generates a far-infrared light signal corresponding to the far-infrared light when it receives the far-infrared light separated by the first beam splitter and the above-mentioned visible. A visible light image processing unit that generates a visible light image corresponding to an optical signal, a near-infrared thermal image processing unit that generates a near-infrared thermal image corresponding to the near-infrared light signal, and the far-infrared optical signal. It is characterized by including a far-infrared thermal image processing unit that generates a far-infrared thermal image according to the above.

According to the above image pickup apparatus, light from the outside is separated into near-infrared light, visible light, and far-infrared light, and a near-infrared thermal image corresponding to near-infrared light, a visible light image corresponding to visible light, and a visible light image are used. It is possible to generate a far-infrared thermal image corresponding to the far-infrared light. That is, the image pickup apparatus can generate a near-infrared thermal image, a far-infrared thermal image, and a visible light image on the same optical axis. Since it can be generated on the same optical axis, the subject in each of the near-infrared thermal image, the far-infrared thermal image, and the visible light image is different from the case where the visible light camera and the far-infrared light camera are separately provided for shooting. Can be in the same position. Therefore, it becomes easy to compare the near-infrared thermal image, the far-infrared thermal image, and the visible light image with each other.

Further, since the near-infrared thermal image and the far-infrared thermal image are images generated by sensing the heat of the outside, it is easy to capture the outside as an image even in a dark place. The visible light image also includes the above external colors. On the other hand, since the near-infrared thermal image and the far-infrared thermal image are images generated by sensing heat, the outside is not clearly reflected. Such a tendency is more remarkable in the far-infrared thermal image than in the near-infrared thermal image. Since the image pickup apparatus can generate a near-infrared thermal image, a visible light image, and a far-infrared thermal image, these images can be used properly according to the purpose and the external environment. For example, a visible light image can be used in a bright place, and a near-infrared thermal image or a far-infrared thermal image can be used in a dark place. Therefore, according to the above-mentioned image pickup apparatus, it is possible to generate an image with high visibility even in a shooting in a dark place.

In the image pickup apparatus, the generation unit uses a prism-type second beam splitter that separates the first light into visible light and near-infrared light, and the visible light separated by the second beam splitter. When it receives light, it receives the near-infrared light separated by the visible light sensor that generates the visible light signal corresponding to the received visible light and the second beam splitter, and responds to the received near-infrared light. It may also include a near-infrared light sensor that produces a near-infrared light signal. Further, when the generation unit receives the first light, the first sensor generates a first signal indicating the first light, and when the first signal is input, the generation unit displays the first signal as visible light. It may include a separation part for separating into a visible light signal indicating and a near-infrared light signal indicating near-infrared light.

The image pickup apparatus may have the following features. The image pickup apparatus may include a synthesis processing unit that generates a composite image obtained by synthesizing the visible light image and the near-infrared thermal image. Here, the composite image may be a composite of the far-infrared thermal image. Then, the composite image is generated by the composite processing unit, for example, the first brightness signal showing the brightness component of the visible light image, the second brightness signal showing the brightness component of the near-infrared thermal image, and the far-infrared thermal image. The process of adding the third brightness signal indicating the brightness component to generate the composite image brightness signal, the brightness component is indicated by the composite image brightness signal, and the color component is indicated by the color signal indicating the color component of the visible light image. It may include a process of generating a composite image.

Further, the process of generating the composite image luminance signal may include a process of emphasizing the high frequency components of the first luminance signal and the second luminance signal. The process of generating the composite image includes a process of correcting the contrast of the composite image luminance signal and a process of adjusting the intensity of the color signal according to the brightness indicated by the composite image luminance signal corrected with the contrast. It may be included. Then, in the composite image, the color component may be shown by the color signal whose intensity of the color signal is adjusted. By performing such processing, the image pickup apparatus can generate a composite image having a higher contrast feeling and a higher resolution feeling.

The image pickup device uses the first image of the detection target as visible light, the second image of the detection target as near-infrared light, and the third image of the detection target as far-infrared light as teacher data. A determination unit for inputting the composite image and the far-infrared thermal image to the constructed learning model and determining whether or not the detection target exists outside the outside may be further provided. Visible light images, near-infrared thermal images, and far-infrared thermal images each have the above-mentioned characteristics. By synthesizing the visible light image and the near-infrared thermal image by the synthesis processing unit, it is possible to generate a composite image that is clearer than the near-infrared thermal image and contains color information. The image pickup apparatus can suitably detect a detection target even in a dark place by inputting such a composite image and a far-infrared thermal image reflecting the external heat distribution into the determination unit. ..

In the image pickup apparatus, the determination unit includes a plurality of learning models constructed for each detection target, and the determination unit selects a learning model according to the detection target specified by the user from among the plurality of learning models. You may. By having such a feature, the image pickup apparatus can suitably detect a designated detection target.

Further, in the image pickup apparatus, when the condensing unit employs a transmission optical system including a plurality of lenses as the condensing unit, the image pickup apparatus can be easily miniaturized. Further, when a catadioptric system including a plurality of concave mirrors is used as the light collecting unit, it is possible to manufacture an image pickup device at a lower cost than when a lens is used. The catadioptric system may include a convex mirror.

Further, in the image pickup apparatus, the first beam splitter contains at least one of silicon, germanium, zinc selenium, and chalcogenide glass, and a substrate that transmits far-infrared light having a wavelength of 8 μm to 12 μm and the above. A high refractive index film provided on the substrate and containing at least one of zinc sulfide and zinc selenium and a low refractive index film containing a fluoride dielectric are alternately laminated, and the wavelength is 0.4 μm to 0. It may include a dielectric multilayer film that reflects visible light up to 7 μm and near-infrared light with a wavelength of 0.7 μm to 3 μm. Here, examples of the fluoride dielectric include magnesium fluoride, calcium fluoride, lithium fluoride, barium fluoride, lanthanum fluoride, and potassium fluoride. By adopting such a configuration, the beam splitter of the above image pickup apparatus reflects visible light and near-infrared light with high reflectance without transmitting them, and also reflects far-infrared light with high transmittance. It can be made transparent with.

In the image pickup apparatus, the substrate of the first beam splitter preferably has a refractive index of 3 or more with respect to the far infrared light. Further, it is preferable that the dielectric multilayer film of the first beam splitter has a refractive index of less than 3 with respect to the far infrared light.

Further, the disclosed technology can also be grasped as the beam splitter.

According to the disclosed technique, it is possible to provide an image pickup apparatus capable of enhancing visibility in a dark place.

FIG. 1 is a diagram showing a configuration of an image pickup apparatus according to an embodiment. FIG. 2 is a diagram schematically showing a near-infrared thermal image, a visible light image, and a far-infrared thermal image in the embodiment. FIG. 3 is a flowchart showing the first variation of the synthesis processing by the synthesis processing unit. FIG. 4 is a flowchart showing a second variation of the composition processing by the composition processing unit. FIG. 5 is a diagram showing an example of a computer included in the image pickup apparatus according to the embodiment. FIG. 6 is a diagram illustrating the characteristics of the first beam splitter in the embodiment. FIG. 7 is an example of a processing flow for detecting a detection target by the image pickup apparatus according to the embodiment. FIG. 8 is an example of a processing flow for detecting a detection target by the image pickup apparatus according to the first modification. FIG. 9 is a diagram showing a configuration of an image pickup apparatus according to a second modification. FIG. 10 is a diagram schematically illustrating a first application example. FIG. 11 is a diagram schematically showing a second application example.

<Embodiment>
Hereinafter, embodiments will be further described with reference to the drawings. FIG. 1 is a diagram showing a configuration of an image pickup apparatus 1 according to an embodiment. The image pickup device 1 is, for example, a surveillance camera installed outdoors or indoors. The image pickup apparatus 1 may capture a still image or a moving image. The image pickup apparatus 1 includes a computer 100 and an optical unit 200.

The optical unit 200 causes the subject light from the subject M to enter the image pickup apparatus 1, and converts the incident subject light into an electric signal. The optical unit 200 includes a lens 11, a first beam splitter 121, a second beam splitter 122, a near-infrared light sensor 131, a visible light sensor 132, and a far-infrared light sensor 133. The lens 11, the first beam splitter 121, the second beam splitter 122, the near infrared light sensor 131, the visible light sensor 132, and the far infrared light sensor 133 are arranged on the same optical axis.

The lens 11 is an optical member that transmits the subject light from the subject M including visible light, near-infrared light, and far-infrared light, and condenses the transmitted subject light. The lens 11 is an example of a “lens” and a “condensing unit”. Although two lenses 11 are illustrated in FIG. 1, the number of lenses 11 is not limited to two, and may be three or more. The plurality of lenses 11 can also be said to be transmission optical systems.

The first beam splitter 121 is a plate-shaped beam splitter (wavelength separating element). The first beam splitter 121 is arranged behind the lens 11 (between the lens 11 and the far-infrared light sensor 133 in the image pickup apparatus 1). The first beam splitter 121 separates the subject light incident through the lens 11 into first light including visible light and near-infrared light and far-infrared light. When the subject light is incident on the first beam splitter 121, the first light is reflected and the far infrared light is transmitted. The first light reflected by the first beam splitter 121 is incident on the second beam splitter 122. The far-infrared light transmitted through the first beam splitter 121 is incident on the far-infrared light sensor 133. The first beam splitter 121 is an example of a "first beam splitter".

The first beam splitter 121 is produced by providing a dielectric multilayer film 1211 that reflects visible light and near-infrared light on a transparent substrate 1212. The dielectric multilayer film 1211 is, for example, a dielectric multilayer film in which a dielectric film having a high refractive index and a dielectric film having a low refractive index are alternately laminated. The lamination of the dielectric in the dielectric multilayer film is performed by, for example, thin film deposition or sputtering. The relationship between the refractive index n, the wavelength λ, and the film thickness d of each of the dielectric films can be, for example, nd = λ / 4. The refractive index n of the high-refractive index dielectric film and the low-refractive index dielectric film with respect to far-infrared light (wavelength 8 μm to 12 μm) is, for example, “n <3”. A high-refractive index dielectric film is an example of a “high-refractive index film”. A low refractive index dielectric film is an example of a “low refractive index film”.

Examples of the dielectric having a high refractive index include zinc sulfide (ZnS) and zinc selenide (ZnSe). Further, as the dielectric having a low refractive index, for example, a fluoride dielectric can be mentioned. Examples of the fluoride dielectric include magnesium fluoride (MgF ₂ ), calcium fluoride (CaF ₂ ), lithium fluoride (LiF), barium fluoride (BaF ₂ ), lanthanum fluoride (LaF ₃ ), and fluoride. Potassium (KF) and the like can be mentioned. The transparent substrate 1212 is a substrate capable of transmitting at least far infrared light. The transparent substrate 1212 is formed of, for example, silicon (Si), germanium (Ge), zinc selenide (ZnSe), chalcogenide glass, or the like. The refractive index of the transparent substrate 1212 is transparent to far infrared light. The refractive index n of the transparent substrate 1212 with respect to far-infrared light is, for example, “n> = 3”. The transparent substrate 1212 is formed by, for example, double-sided optical polishing of a material that is transparent to far-infrared light and has a refractive index n to far-infrared light of “n> = 3”. Since the refractive index of the transparent substrate 1212 is larger than that of the dielectric multilayer film 1211, the reflection of far-infrared light by the transparent substrate 1212 is suppressed. The first beam splitter 121 is arranged with the dielectric multilayer film 1211 facing the lens 11.

The second beam splitter 122 is a prism-shaped beam splitter. The second beam splitter 122 separates the first light reflected by the first beam splitter 121 into visible light and near-infrared light. When the first light is incident on the second beam splitter 122, the visible light is reflected and the near infrared light is transmitted. The visible light reflected by the second beam splitter 122 is incident on the visible light image processing unit 142. The near-infrared light transmitted through the second beam splitter 122 is incident on the near-infrared thermal image processing unit 141.

The second beam splitter 122 includes triangular prisms 24a and 24b having a quadrangular side surface and a right triangle shape at the bottom surface. The second beam splitter 122 is made, for example, as follows. A visible light reflector 1221 is provided on one of the side surfaces of the triangular prism 122b. The second beam splitter 122 is produced by abutting one of the side surfaces of the triangular prism 122a with the side surface of the triangular prism 122b provided with the visible light reflector 1221. The second beam splitter 122 thus produced is arranged so that the visible light reflector 1221 is slanted with respect to the light reflected by the first beam splitter 121. The second beam splitter 122 is an example of a "second beam splitter".

The near-infrared light sensor 131 is, for example, an image pickup device that generates an electric signal according to the incident near-infrared light. That is, the near-infrared light sensor 131 can also be said to be an image pickup device that detects the heat of the subject M and converts it into an electric signal. The near-infrared light sensor 131 has a peak of sensitivity to light having a wavelength of 0.7 μm to 3 μm, for example. The near-infrared light sensor 131 is, for example, a CCD image sensor or a CMOS image sensor having sensitivity to near-infrared light. The near-infrared light sensor 131 is an example of a "near-infrared light sensor". An electric signal corresponding to near-infrared light is an example of a "near-infrared light signal".

The visible light sensor 132 is, for example, an image pickup device that generates an electric signal according to the incident visible light. The visible light sensor 132 can also be said to be an image pickup device that detects the color of the subject M and converts it into an electric signal. The visible light sensor 132 has, for example, a peak of sensitivity to light having a wavelength of 0.4 μm to 0.7 μm or less. The visible light sensor 132 is, for example, a CCD image sensor or a CMOS image sensor having sensitivity to visible light. The visible light sensor 132 is an example of a “visible light sensor”. An electric signal corresponding to visible light is an example of a "visible light signal".

The far-infrared light sensor 133 is, for example, an image pickup device that generates an electric signal according to the incident far-infrared light. The far-infrared light sensor 133 can also be said to be an image pickup device that detects the heat of the subject M and converts it into an electric signal. The far-infrared light sensor 133 has, for example, a peak of sensitivity to light having a wavelength of 8 μm to 12 μm. The far-infrared light sensor 133 is, for example, a CCD image sensor, a CMOS image sensor, a thermopile, a volometer, and a pyroelectric sensor having sensitivity to far-infrared light. The far-infrared light sensor 133 is an example of a "far-infrared light sensor". An electric signal corresponding to far-infrared light is an example of a "far-infrared light signal".

The computer 100 generates a near-infrared thermal image, a visible light image, and a far-infrared thermal image based on electric signals input from the near-infrared light sensor 131, the visible light sensor 132, and the far-infrared light sensor 133. , Performs monitoring processing of the subject M. The computer 100 includes a near-infrared thermal image processing unit 141, a visible light image processing unit 142, a far-infrared thermal image processing unit 143, a synthesis processing unit 15, a learning function indicator 16, and a machine learning device 17.

The near-infrared thermal image processing unit 141 is a processing unit that generates a near-infrared thermal image (referred to as "IR image" in the figure) based on an electric signal input from the near-infrared light sensor 131. That is, the near-infrared thermal image generated by the near-infrared thermal image processing unit 141 is an image taken with near-infrared light. The near-infrared thermal image processing unit 141 is an example of the "near-infrared thermal image processing unit".

The visible light image processing unit 142 is a processing unit that generates a visible light image based on an electric signal input from the visible light sensor 132. That is, the visible light image generated by the visible light image processing unit 142 is an image taken with visible light. The visible light image processing unit 142 is an example of the “visible light image processing unit”. The near-infrared light sensor 131, the visible light sensor 132, the near-infrared thermal image processing unit 141, and the visible light image processing unit 142 are examples of the “generation unit”.

The far-infrared thermal image processing unit 143 is a processing unit that generates a far-infrared thermal image (referred to as "FIR image" in the figure) based on an electric signal input from the far-infrared light sensor 133. That is, the far-infrared thermal image generated by the far-infrared thermal image processing unit 143 is an image obtained by photographing the subject M with far-infrared light. Since far-infrared light is mainly emitted by the heat of an object, the far-infrared thermal image can be said to be an image showing the heat distribution in the photographing range of the image pickup apparatus 1. The far-infrared thermal image processing unit 143 is an example of the “far-infrared thermal image processing unit”.

FIG. 2 is a diagram schematically showing a near-infrared thermal image, a visible light image, and a far-infrared thermal image in the embodiment. FIG. 2A exemplifies a near-infrared thermal image generated by the near-infrared thermal image processing unit 141. FIG. 2B illustrates a visible light image generated by the visible light image processing unit 142. FIG. 2C illustrates a far-infrared thermal image generated by the far-infrared thermal image processing unit 143.

For near-infrared thermal images generated by the near-infrared thermal image processing unit 141, visible light images generated by the visible light image processing unit 142, and far-infrared thermal images generated by the far-infrared thermal image processing unit 143. Each has its own characteristics. Since the near-infrared thermal image and the far-infrared thermal image generated based on the near-infrared light and the far-infrared light are images generated by sensing the heat of the subject M, even in a dark place. It is easy to capture the subject M as an image. Further, the visible light image generated based on the visible light also includes the color of the subject M. On the other hand, since the near-infrared thermal image and the far-infrared thermal image are images generated by sensing heat, the subject M is not clearly reflected. Such a tendency is more remarkable in the far-infrared thermal image than in the near-infrared thermal image. Therefore, the image pickup apparatus 1 compensates for the shortcomings of these images by generating a composite image obtained by synthesizing the near-infrared thermal image, the visible light image, and the far-infrared thermal image by the composite processing unit 15.

Returning to FIG. 1, the synthesis processing unit 15 is, for example, a near-infrared thermal image generated by the near-infrared thermal image processing unit 141, a visible light image generated by the visible light image processing unit 142, and a far-infrared thermal image. It is a processing unit that generates a composite image based on the far-infrared thermal image generated by the processing unit 143. The compositing processing unit 15 synthesizes a near-infrared thermal image, a visible light image, and a far-infrared thermal image to generate a composite image that complements the defects of each image. The compositing processing unit 15 corrects the subject M of the near-infrared thermal image more clearly by correcting the near-infrared thermal image with a visible light image, and adds color information to the near-infrared thermal image. The first corrected image is generated. Then, the composition processing unit 15 corrects the far-infrared thermal image by the first correction image, thereby correcting the far-infrared thermal image more clearly and generating a composite image to which color information is added. The synthesis processing unit 15 is an example of a “synthesis processing unit”.

Here, a signal indicating each image synthesized by the synthesis processing unit 15 will be described. Visible light images contain color information. Therefore, the signal indicating the visible light image includes a color signal indicating the color of each pixel and a luminance signal indicating the brightness of each pixel. Near-infrared thermal images and far-infrared thermal images do not contain color information. Therefore, the signal showing the near-infrared thermal image and the signal showing the far-infrared thermal image include the luminance signal indicating the luminance of each pixel, but do not include the color signal. A variation of the composition process for synthesizing each image represented by such a signal will be described with reference to the drawings.

FIG. 3 is a flowchart showing the first variation of the synthesis processing by the synthesis processing unit 15. Hereinafter, the first variation of the synthesis processing by the synthesis processing unit 15 will be described with reference to FIG.

In G11, the synthesis processing unit 15 cuts out a part of the far-infrared thermal image and resizes the cut-out image to reduce the number of pixels of the far-infrared thermal image and the size of the image to the near-infrared thermal image and the visible light image. Perform the resizing process to align with. If the number of pixels of the far-infrared light sensor 133 is equal to the number of pixels of the near-infrared light sensor 131 and the number of pixels of the visible light sensor 132, the processing of G11 may be omitted.

In the G12, the synthesis processing unit 15 adds the luminance signal of the visible light image, the luminance signal of the near-infrared thermal image, and the luminance signal of the far-infrared thermal image to generate the luminance signal of the composite image. Here, when the processing of G11 is executed, the luminance signal of the far-infrared thermal image after the processing of G11 is used to generate the composite image luminance signal. The composite image luminance signal is generated, for example, by adding the luminance values indicated by the respective luminance signals in each pixel.

In G13, the synthesis processing unit 15 generates a composite image by synthesizing the color signal of the visible light image with the composite image luminance signal generated in G12. That is, the color component of the composite image is indicated by the color signal of the visible light image, and the luminance component of the composite image is indicated by the luminance signal of the composite image.

Subsequently, other variations of the synthesis processing by the synthesis processing unit 15 will be described. FIG. 4 is a flowchart showing a second variation of the synthesis processing by the synthesis processing unit 15. The same processing as in FIG. 3 is designated by the same reference numerals, and the description thereof will be omitted. Hereinafter, the second variation of the synthesis processing by the synthesis processing unit 15 will be described with reference to FIG.

In the G21, the synthesis processing unit 15 executes a process of emphasizing high frequency components in each of the luminance signal of the near-infrared thermal image and the luminance signal of the visible light image. In the process of emphasizing the high frequency component, a predetermined value may be added to each high frequency component of the luminance signal of the near-infrared thermal image and the luminance signal of the visible light image, or the predetermined value may be integrated. The frequency range and predetermined value of the high frequency component may be appropriately determined. By the addition process of G21, it is possible to increase the resolution of the far-infrared thermal image, which is often generated by the sensor having a lower number of pixels as compared with the visible light image or the near-infrared thermal image.

In G22, the synthesis processing unit 15 adds the luminance signal of the far-infrared thermal image, the luminance signal of the near-infrared thermal image in which the high-frequency component is emphasized in G21, and the luminance signal of the visible light image to obtain the added luminance signal. Generate.

In G23, the synthesis processing unit 15 generates a composite luminance signal by performing contrast correction processing for correcting the contrast with respect to the added luminance signal generated in G22. The method used for the contrast correction process is not limited, and various known methods can be adopted.

In the G24, the synthesis processing unit 15 generates a composite image luminance signal by executing a process of emphasizing the high frequency component of the composite luminance signal generated by the G23. In the process of emphasizing the high frequency component, the same method as G21 can be adopted.

In G25, the synthesis processing unit 15 corrects the intensity of the color signal of the visible light image according to the ratio of the luminance of the luminance signal of the visible light image and the luminance signal of the synthetic luminance signal generated by G23. For example, the compositing processing unit 15 may increase the intensity of the color signal of the visible light image so that the brightness of each pixel of the composite luminance signal is larger than the brightness of the same pixel of the luminance signal of the visible light image.

In G26, the synthesis processing unit 15 synthesizes the composite image luminance signal generated by G24 and the color signal of the visible light image corrected by G25 to generate a composite image. That is, the color component of the composite image is indicated by the color signal corrected by G25, and the luminance component of the composite image is indicated by the luminance signal of the composite image generated by G24.

The synthesis process by the synthesis processing unit 15 is not limited to the processing described with reference to FIGS. 3 and 4. The synthesis processing unit 15 may perform the synthesis processing by another method.

The learning function indicator 16 instructs the machine learning device 17 to detect a subject from the image captured by the image pickup device 1. Examples of the detection target include people, animals, vehicles, and the like. The learning function indicator 16 is designated as a detection target, for example, by an instruction from the user via the user interface of the image pickup device 1.

The machine learning device 17 captures a detection target in advance to generate a near-infrared thermal image generated by the near-infrared thermal image processing unit 141, a visible light image generated by the visible light image processing unit 142, and far-infrared heat. A learning model used for detecting the detection target is constructed using the far-infrared thermal image generated by the image processing unit 143 and the composite image generated by the composite processing unit 15 as teacher data. The machine learning device 17 may construct a plurality of learning models corresponding to each of the plurality of detection targets. Further, the machine learner 17 uses the constructed learning model to obtain a near-infrared thermal image input from the near-infrared thermal image processing unit 141, a visible light image input from the visible light image processing unit 142, and a far-infrared image. The designated detection target is detected based on the far-infrared thermal image input from the external thermal image processing unit 143 and the composite image input from the composite processing unit 15. When the machine learning device 17 detects a detection target, it may notify a higher-level device such as a server device. The machine learning device 17 is an example of a “determination unit”.

The near-infrared thermal image processing unit 141, the visible light image processing unit 142, the far-infrared thermal image processing unit 143, the synthesis processing unit 15, the learning function indicator 16 and the machine learning device 17 are, for example, a Central Processing Unit (CPU). And realized by a computer with memory. FIG. 5 is a diagram showing an example of a computer 100 included in the image pickup apparatus 1 according to the embodiment. The computer 100 is an information processing device included in the image pickup device 1.

The CPU 101 is also referred to as a microprocessor unit (MPU) or a processor. The CPU 101 may be a combination of a processor and an integrated circuit. The combination is called, for example, a microcontroller unit (MCU), a system-on-a-chip (SoC), a system LSI, a chipset, or the like. In the computer 100, the CPU 101 expands the program stored in the auxiliary storage unit 103 into the work area of the main storage unit 102, and executes the program to execute the near-infrared thermal image processing unit 141 and the visible light image processing unit 142. , A far-infrared thermal image processing unit 143, a synthesis processing unit 15, a learning function indicator 16, and a machine learning device 17 are realized.

The main storage unit 102 is exemplified as a storage unit that is directly accessed from the CPU 101. The main storage unit 102 includes a Random Access Memory (RAM) and a Read Only Memory (ROM).

The auxiliary storage unit 103 stores various programs and various data in a readable / writable recording medium. The auxiliary storage unit 103 is also called an external storage device. The auxiliary storage unit 103 stores an operating system (Operating System, OS), various programs, various tables, a learning model built by the machine learning device 17, and the like.

The auxiliary storage unit 103 is, for example, an Erasable Program ROM (EPROM), a solid state drive (Sold State Drive, SSD), a hard disk drive (Hard Disk Drive, HDD), or the like.

The communication unit 104 is an interface that receives inputs of electric signals from the near-infrared light sensor 131, the visible light sensor 132, and the far-infrared light sensor 133. The CPU 101 acquires electric signals from the near-infrared light sensor 131, the visible light sensor 132, and the far-infrared light sensor 133 via the communication unit 104.

The computer 100 may further include, for example, an input unit that receives an operation instruction or the like from a user or the like. As such an input unit, an input device such as a keyboard, a pointing device, a touch panel, an acceleration sensor, or a voice input device can be exemplified.

(Characteristics of beam splitter)
FIG. 6 is a diagram illustrating the characteristics of the first beam splitter 121 in the embodiment. FIG. 6 also illustrates the characteristics of the transparent substrate 1212 without the dielectric multilayer film 1211 for comparison. The vertical axis of FIG. 6 exemplifies the transmittance or the reflectance, and the horizontal axis exemplifies the wavelength. In FIG. 6, the curve showing the reflectance is indicated by "reflectance", and the curve showing the transmittance is indicated by "transmittance". Further, in FIG. 6, the solid line curve exemplifies the characteristics of the first beam splitter 121, and the dotted line curve exemplifies the characteristics of the transparent substrate 1212.

The transmittance of the transparent substrate 1212 for visible light having a wavelength of 0.6 μm or less and near-infrared light having a wavelength of 1.2 μm or less is approximately 0%, and the reflectance is 70% or less. Further, the transmittance of the transparent substrate 1212 for far-infrared light having a wavelength of 10 μm or more is 30% or less, and the reflectance is 40% or less. On the other hand, the transmittance of the first beam splitter 121 for visible light having a wavelength of 0.6 μm or less and near-infrared light having a wavelength of 1.2 μm or less is approximately 0%, and the reflectance is 70% or more. Further, the transmittance of the first beam splitter 121 for far-infrared light having a wavelength of 10 μm or more is approximately 100%, and the reflectance is approximately 0%. That is, by providing the dielectric multilayer film 1211 on the transparent substrate 1212, the reflectance of visible light and near-infrared light can be increased, and the transmittance of far-infrared light can be increased. That is, the first beam splitter 121 having the dielectric multilayer film 1211 provided on the transparent substrate 1212 reflects visible light and near-infrared light with high reflectance without transmitting them, and does not reflect far-infrared light. Can be transmitted with high transmittance.

(Detection flow of detection target)
FIG. 7 is an example of a processing flow for detecting a detection target by the image pickup apparatus 1 according to the embodiment. Hereinafter, a processing flow for detecting a detection target by the image pickup apparatus 1 according to the embodiment will be described with reference to FIG. 7.

In T1, the synthesis processing unit 15 is subjected to the near-infrared thermal image generated by the near-infrared thermal image processing unit 141, the visible light image generated by the visible light image processing unit 142, and the far-infrared thermal image processing unit 143. A composite image is generated based on the generated far-infrared thermal image.

In T2, the machine learning device 17 detects the detection target by inputting the composite image generated in T1 into the learning model generated by the machine learning device 17 in advance. When the detection target is detected (YES in T2), the process proceeds to T3. If the detection target is not detected (NO at T2), the process ends.

In T3, the machine learning device 17 outputs an alert when the detection target is detected. The output of the alert may be, for example, a notification to a higher-level device. Further, the machine learning device 17 may output a warning sound from a speaker (not shown) included in the image pickup device 1, and outputs a warning message to a display (not shown) to which the image pickup device 1 is connected. May be good.

<Action and effect of the embodiment>
In the image pickup apparatus 1 according to the embodiment, the subject light incident through the lens 11 is split into first light including visible light and near-infrared light and far-infrared light by the first beam splitter 121. The first light separated from the subject light by the first beam splitter 121 is incident on the second beam splitter 122, and is split into visible light and near infrared light by the second beam splitter 122. The near-infrared light separated by the second beam splitter 122 is incident on the near-infrared light sensor 131, and the visible light separated by the second beam splitter 122 is incident on the visible light sensor 132. Further, the far-infrared light separated from the subject light by the first beam splitter 121 is incident on the far-infrared light sensor 133. The lens 11, the near-infrared light sensor 131, the visible light sensor 132, and the far-infrared light sensor 133 are arranged on the same optical axis, and the near-infrared light sensor 131, the visible light sensor 132, and the far-infrared light sensor 133 are arranged. The light incident on the lens 11 is separated from the same subject light incident on the lens 11. Therefore, the image pickup apparatus 1 according to the embodiment can simultaneously acquire a near-infrared thermal image, a visible light image, and a far-infrared thermal image of the subject M.

In the image pickup apparatus 1 according to the embodiment, the near-infrared thermal image generated by the near-infrared thermal image processing unit 141, the visible light image generated by the visible light image processing unit 142, and the far-infrared thermal image processing unit 143 The generated far-infrared thermal image is synthesized by the synthesis processing unit 15. Therefore, the image pickup apparatus 1 according to the embodiment can generate a clear composite image that compensates for the shortcomings of the near-infrared thermal image, the visible light image, and the far-infrared thermal image.

In the present embodiment, for example, the addition process of G21 in FIG. 4 increases the resolution of a far-infrared thermal image, which is often generated by a sensor having a lower number of pixels than a visible light image or a near-infrared thermal image. Can be planned. Further, the processing of G23 and G24 in FIG. 4 improves the contrast feeling and the resolution feeling of the composite image. According to the present embodiment, by generating a composite image by such processing, it is possible to generate a composite image with further improved contrast feeling and resolution feeling.

In the image pickup apparatus 1 according to the embodiment, the near-infrared thermal image generated by the near-infrared thermal image processing unit 141, the visible light image generated by the visible light image processing unit 142, and the far-infrared thermal image processing unit 143 The machine learner 17 performs machine learning using the generated far-infrared thermal image and the composite image generated by the synthesis processing unit 15 as teacher data, and a learning model used for detecting the detection target is constructed. Here, the subject to be detected is instructed by the user to the learning function indicator 16, and the learning function indicator 16 instructs the machine learning device 17 to detect the instructed detection target. The machine learning device 17 selects and detects a learning model corresponding to the designated detection target. Therefore, according to the image pickup apparatus 1 according to the embodiment, it is possible to detect a subject desired by the user.

As described with reference to FIG. 6, the first beam splitter 121 included in the image pickup apparatus 1 according to the embodiment reflects visible light and near-infrared light with high reflectance without transmitting them, and far-infrared light. Light is transmitted with high transmittance and is not reflected. Therefore, the first beam splitter 121 can separate the first light including visible light and near-infrared light from the far-infrared light with high accuracy. Further, since the image pickup apparatus 1 can separate the first light and the far infrared light with high accuracy by the first beam splitter 121, the first light is incident on the far infrared light sensor 133. Is suppressed. As a result, the deterioration of the image quality of the far-infrared light sensor 133 is suppressed.

Further, the image pickup apparatus 1 according to the embodiment generates a near-infrared thermal image, a visible light image, and a far-infrared thermal image on the same optical axis. Therefore, when a near-infrared light camera that shoots with near-infrared light, a visible light camera that shoots with visible light, and a far-infrared light camera that shoots with far-infrared light are prepared to shoot the subject M (different). Unlike the case where the subject M is photographed from a position), the position of the subject M does not shift between the near-infrared thermal image, the visible light image, and the far-infrared thermal image. Therefore, the image pickup apparatus 1 can show the position of the subject M with high accuracy even in the composite image.

<First modification>
In the embodiment, the near-infrared thermal image generated by the near-infrared thermal image processing unit 141, the visible light image generated by the visible light image processing unit 142, and the far-infrared generated by the far-infrared thermal image processing unit 143. The external heat image was synthesized by the synthesis processing unit 15. In the first modification, the composite processing unit 15 generates a composite image of the near-infrared thermal image generated by the near-infrared thermal image processing unit 141 and the visible light image generated by the visible light image processing unit 142. The processing of is described. In this case, the far-infrared thermal image generated by the far-infrared thermal image processing unit 143 does not have to be input to the synthesis processing unit 15. Hereinafter, the first modification will be described with reference to the drawings.

(Detection flow of detection target)
FIG. 8 is an example of a processing flow for detecting a detection target by the image pickup apparatus 1 according to the first modification. In FIG. 8, the same processing as in FIG. 7 is designated by the same reference numerals, and the description thereof will be omitted. Hereinafter, with reference to FIG. 8, a processing flow for detecting a detection target by the image pickup apparatus 1 according to the first modification will be described.

In T1a, the synthesis processing unit 15 and in T1, the synthesis processing unit 15 displays the near-infrared thermal image generated by the near-infrared thermal image processing unit 141 and the visible light image generated by the visible light image processing unit 142. Based on this, a composite image is generated.

In T2a, the machine learning device 17 detects the detection target by using the learning model generated by the machine learning device 17 in advance. The composite image generated by T1a and the far-infrared thermal image generated by the far-infrared thermal image processing unit 143 are input to the machine learning device 17. The machine learning device 17 detects a detection target by inputting a composite image and a far-infrared thermal image into the learning model. When the detection target is detected (YES in T2a), the process proceeds to T3. If the detection target is not detected (NO in T2a), the process ends.

<Second modification>
In the image pickup apparatus 1 according to the embodiment, the lens 11 is adopted as an optical member that collects the subject light. In the second modification, an image pickup apparatus provided with a concave mirror instead of the lens 11 will be described. The components common to the embodiments are designated by the same reference numerals, and the description thereof will be omitted. Hereinafter, the second modification will be described with reference to the drawings.

FIG. 9 is a diagram showing the configuration of the image pickup apparatus 1a according to the second modification. The image pickup device 1a is different from the image pickup device 1 in that the image pickup device 1a includes the optical section 200a instead of the optical section 200. The optical unit 200a includes a concave mirror 11a instead of the lens 11, and a near-infrared light / visible light sensor 131a instead of the near-infrared light sensor 131 and the visible light sensor 132. Further, the optical unit 200a further includes an RGB / IR separation element 132b, and the second beam splitter 122 is omitted.

The concave mirror 11a reflects the incident subject light in a direction different from the incident direction. The subject light incident on the concave mirror 11a is bent in the optical path by the concave mirror 11a and is incident on the first beam splitter 121. Although three concave mirrors 11a are illustrated in FIG. 9, the number of concave mirrors 11a is not limited to three. The three concave mirrors 11a can also be said to be a catadioptric system.

The near-infrared light / visible light sensor 131a is an image pickup device that generates an electric signal corresponding to incident visible light and near-infrared light. The near-infrared light sensor 131a has a peak of sensitivity to light having a wavelength of 2.5 μm or less. The near-infrared light / visible light sensor 131a is, for example, a CCD image sensor or a CMOS image sensor having sensitivity to light having a wavelength of 2.5 μm or less. The electric signal generated by the near-infrared light / visible light sensor 131a is input to the RGB / IR separation element 132b. The near-infrared light / visible light sensor 131a is an example of the "first sensor". The electric signal corresponding to the visible light and the near infrared light is an example of the "first signal".

The RGB / IR separation element 132b separates the electric signal input from the near-infrared light / visible light sensor 131a into an electric signal indicating visible light and an electric signal indicating near-infrared light. The electric signal indicating the near-infrared light separated by the RGB / IR separation element 132b is input to the near-infrared thermal image processing unit 141. An electric signal indicating visible light separated by the RGB / IR separation element 132b is input to the visible light image processing unit 142. The RGB / IR separation element 132b is an example of a “separation unit”. The near-infrared light / visible light sensor 131a, the RGB / IR separation element 132b, the near-infrared thermal image processing unit 141, and the visible light image processing unit 142 are examples of the “generation unit”.

By adopting the concave mirror 11a instead of the lens 11, the image pickup apparatus 1a according to the second modification can be manufactured at a lower cost than the image pickup apparatus 1. On the other hand, the image pickup apparatus 1 can be made smaller than the image pickup apparatus 1a by adopting the lens 11.

<Other variants>
In the embodiment, the first beam splitter 121 including the dielectric multilayer film 1211 has been described. However, the dielectric multilayer film 1211 does not have to be provided with the first beam splitter 121. For example, the first beam splitter 121 can also reflect visible light and near-infrared light by forming periodic grooves having a pitch smaller than the wavelength of visible light instead of the dielectric multilayer film 1211. .. This periodic groove has a structure much smaller than the wavelength of far-infrared light. Therefore, even with such a structure, the first beam splitter 121 can transmit far infrared light.

<Application example>
The image pickup apparatus according to the embodiment and each modification described above can be applied to various systems. Hereinafter, application examples of the image pickup apparatus according to the embodiment and each modification will be described with reference to the drawings.

(First application example)
FIG. 10 is a diagram schematically illustrating a first application example. In the first application example, the image pickup apparatus 1 according to the embodiment is adopted as a surveillance camera of the vehicle 800. FIG. 10 is a top view of the vehicle 800 traveling on the road. In the example of FIG. 10, one image pickup apparatus 1 is provided so as to photograph the traveling direction of the vehicle 800.

In the first application example, the image pickup apparatus 1 photographs the traveling direction of the vehicle (for example, the monitoring area W1). At night or in a dark place such as in a tunnel, the visibility of the passerby M1 crossing the pedestrian crossing is lowered due to the influence of the headlight light 811 of the oncoming vehicle 810. As described above, it is difficult to detect the passerby M1 with a visible light image in a dark place, and it is not possible to clearly capture the passerby M1 to be detected in a near-infrared thermal image or a far-infrared thermal image. Have difficulty. The image pickup apparatus 1 acquires a near-infrared thermal image taken with near-infrared light, a visible light image taken with visible light, and a far-infrared thermal image taken with far-infrared light on the same optical axis. Then, the image pickup apparatus 1 synthesizes a near-infrared thermal image, a visible light image, and a far-infrared thermal image to generate a composite image that complements the shortcomings of these images. In other words, the image pickup apparatus 1 detects the passerby M1 from the viewpoints of both visible light and heat distribution by using the composite image. By using such a composite image, the image pickup apparatus 1 can improve the detection accuracy of the passerby M1 in a dark place. Further, the machine learning device 17 of the image pickup apparatus 1 may instruct the vehicle 800 to control the brake when the passerby M1 is detected.

(Second application example)
The image pickup device 1 can also be applied to an entry / exit management system. FIG. 11 is a diagram schematically showing a second application example. In the second application example, the image pickup apparatus 1 is adopted for entry / exit management of the server room 850. FIG. 11 is a top view of the server room 850. In the example of FIG. 11, one image pickup device 1 is provided beside the entrance / exit door 851 of the server room 850. The shooting direction of the image pickup apparatus 1 is directed to the outside of the server room 850.

The image pickup apparatus 1 photographs an operator who intends to perform work in the server room 850, and transmits the captured image to a higher-level device. The host device can perform face recognition using the visible light image received from the image pickup device 1. Further, the host device measures the body temperature of the worker based on the far-infrared thermal image received from the image pickup device 1, and if the measured body temperature is equal to or higher than the predetermined body temperature, the server room 850 is considered to be in poor physical condition. You may refuse to enter the room (do not unlock the entrance / exit door 851). In the second application example, the image pickup device 1 was used for entry / exit control to the server room 850, but as a measure against the new corona (COVID-19) and the like, the customer's face recognition and body temperature at the time of entering the store are recognized. The image pickup apparatus 1 may be used for the measurement.

(Other application examples)
Since the image pickup devices 1 and 1a described above can capture a suitable image regardless of the brightness of the region to be captured, they can be applied to various other uses. Since the image pickup devices 1 and 1a can take pictures not only by visible light but also by heat, they can also be applied to a forest fire surveillance camera. Further, since the image pickup devices 1 and 1a can take a suitable image regardless of the brightness of the area to be photographed, the disaster prevention monitoring camera and the natural environment monitoring camera corresponding to various environments such as day and night, rain, fog, and snow. It can also be used as a day / night security camera.

The embodiments and modifications disclosed above can be combined.

1 ・ ・ Imaging device 1a ・ ・ Imaging device 11 ・ ・ Lens 11a ・ ・ Concave mirror 15 ・ ・ Synthesis processing unit 16 ・ ・ Learning function indicator 17 ・ ・ Machine learning device 100 ・ ・ Computer 200 ・ ・ Optical unit 200a ・ ・ Optical Part 101 ... CPU
102 ... Main storage unit 103 ... Auxiliary storage unit 104 ... Communication unit 121 ... First beam splitter 122 ... Second beam splitter 131 ... Near infrared light sensor 131a ... Near infrared light / visible light sensor 132 ... Visible light sensor 132b ... RGB / IR separation element 133 ... Far infrared light sensor 141 ... Near infrared thermal image processing unit 142 ... Visible light image processing unit 143 ... Far infrared thermal image processing unit 800 ・ ・ Vehicle 810 ・ ・ Oncoming vehicle 811 ・ ・ Headlight light 850 ・ ・ Server room 851 ・ ・ Entrance / exit door 1221 ・ ・ Visible light reflector 122a ・ ・ Triangular prism 122b ・ ・ Triangular prism 1211 ・ ・ Dielectric multilayer film 1212 ... Transparent substrate M ... Subject W1 ... Monitoring area

Claims (10)

A condensing unit that collects light from the outside and
A first beam splitter that splits the light incident through the condensing unit into first light including visible light and far-infrared light.
When the first light is received, a near-infrared light signal corresponding to the near-infrared light of the first light and a visible light signal corresponding to the visible light of the first light are generated. The generator and
A far-infrared light sensor that generates a far-infrared light signal corresponding to the far-infrared light when receiving the far-infrared light separated by the first beam splitter.
A visible light image processing unit that generates a visible light image corresponding to the visible light signal, and a visible light image processing unit.
A near-infrared thermal image processing unit that generates a near-infrared thermal image corresponding to the near-infrared optical signal, and a near-infrared thermal image processing unit.
A far-infrared thermal image processing unit that generates a far-infrared thermal image corresponding to the far-infrared optical signal, and a far-infrared thermal image processing unit.
A synthesis processing unit for generating a composite image obtained by synthesizing the visible light image, the near-infrared thermal image, and the far-infrared thermal image is provided .
The synthesis processing unit is
A composite image by adding a first luminance signal indicating the luminance component of the visible light image, a second luminance signal indicating the luminance component of the near-infrared thermal image, and a third luminance signal indicating the luminance component of the far-infrared thermal image. The process of generating a luminance signal and
A process of generating a composite image in which the luminance component is indicated by the composite image luminance signal and the color component is indicated by the color signal indicating the color component of the visible light image is executed.
The process of generating the composite image luminance signal includes a process of emphasizing the high frequency components of the first luminance signal and the second luminance signal.
The process of generating the composite image is
The process of correcting the contrast of the composite image luminance signal and
A process of adjusting the intensity of the color signal according to the brightness indicated by the contrast-corrected composite image luminance signal, and the like.
The composite image is characterized in that a color component is indicated by a color signal whose intensity of the color signal is adjusted .
Imaging device. The generator is
A prism-type second beam splitter that separates the first light into visible light and near-infrared light,
A visible light sensor that generates a visible light signal corresponding to the received visible light when the visible light separated by the second beam splitter is received.
A near-infrared light sensor that generates a near-infrared light signal corresponding to the received near-infrared light when the near-infrared light separated by the second beam splitter is received is included.
The imaging device according to claim 1. The generator is
When the first light is received, the first sensor that generates the first signal indicating the first light and the first sensor
When the first signal is input, the first signal includes a separation unit that separates the first signal into a visible light signal indicating visible light and a near-infrared light signal indicating near-infrared light.
The imaging device according to claim 1. A learning model constructed using the first image of the detection target as visible light, the second image of the detection target as near-infrared light, and the third image of the detection target as far-infrared light as teacher data. Further, a determination unit for inputting the composite image and the far-infrared thermal image to determine whether or not the detection target exists outside the outside is further provided.
The imaging device according to any one of claims 1 to 3 . The determination unit includes a plurality of learning models constructed for each detection target.
The determination unit selects a learning model according to the detection target specified by the user from the plurality of learning models.
The imaging device according to claim 4 . The condensing unit is a transmission optical system including a plurality of lenses.
The imaging device according to any one of claims 1 to 5 . The condensing unit is a catadioptric system including a plurality of concave mirrors.
The imaging device according to any one of claims 1 to 5 . The first beam splitter
A substrate containing at least one of silicon, germanium, zinc selenide and chalcogenide glass and transmitting far infrared light having a wavelength of 8 μm to 12 μm.
A high refractive index film provided on the substrate and containing at least one of zinc sulfide and zinc selenium and a low refractive index film containing a fluoride dielectric are alternately laminated and have a wavelength of 0.4 μm to 0. Includes a dielectric multilayer film that reflects visible light up to 7 μm and near-infrared light with wavelengths from 0.7 μm to 3 μm.
The imaging device according to any one of claims 1 to 7 . The substrate has a refractive index of 3 or more with respect to the far infrared light.
The imaging device according to claim 8 . The dielectric multilayer film has a refractive index of less than 3 with respect to the far infrared light.
The imaging device according to claim 8 or 9 .

Images