WO2023181558A1

WO2023181558A1 - Imaging device

Info

Publication number: WO2023181558A1
Application number: PCT/JP2022/047594
Authority: WO
Inventors: 拓洋澁谷
Original assignee: 株式会社日立国際電気
Priority date: 2022-03-24
Filing date: 2022-12-23
Publication date: 2023-09-28
Also published as: JPWO2023181558A1

Abstract

The purpose of the present patent is to provide an imaging device capable of obtaining an image with a deep depth of field even when an aperture is open.　This imaging device comprises: an imaging element unit that separately receives visible light and near-infrared light, and outputs a visible light signal and a near-infrared light signal; and a video signal processing unit that extracts a contour component from a video by means of the near-infrared light signal, and superimposes the extracted contour component on a video of the visible light signal. The imaging device further comprises: a lens; an aperture that adjusts the amount of light entering the lens, and a control unit that controls the aperture opening, wherein the control unit opens the aperture in accordance with the luminance level of the video by means of the visible light signal.

Description

Imaging device

The present invention relates to an imaging device, and particularly to an imaging device that expands the apparent depth of field.

In recent years, indoor and outdoor fixed-point surveillance cameras have become widely used as a means of monitoring to strengthen security, and vehicle-mounted surveillance cameras have become a means of recording evidence in the event of an accident. When these surveillance cameras take pictures in environments with low light levels, such as in the evening or at night, the images become dark and lose their value as surveillance images. To prevent this, an operation or process is performed to increase the brightness level of the video captured by the surveillance camera.

Means to increase the brightness level are: [1] Open the aperture to increase the amount of light received, [2] Increase the exposure time by lengthening the time for one frame, [3] Irradiate light onto the subject. [4] There is a method of applying gain correction to the video signal.

Furthermore, Patent Document 1 discloses that it is determined whether it is daytime, nighttime, early morning, or evening based on the brightness of the photographed image acquired by the camera and the brightness of the reference image of the road surface, and the aperture of the camera is adjusted according to the determination result. A technique for controlling this is disclosed.

Japanese Patent Application Publication No. 2001-21958

Regarding the means [1] above, when the aperture is opened, the depth of field becomes shallow, and images of objects other than objects at a certain distance become blurred. For this reason, it is not suitable for cameras that monitor not only one specific point but many points over a wide range. The same problem applies to the technique disclosed in Patent Document 1, and for example, when the aperture is opened to increase the amount of light received at night, a similar problem occurs.

Regarding the means for increasing the exposure time in [2] above, it becomes necessary to lower the frame rate, and the number of afterimages that occur on a moving subject increases. For this reason, the images are unsuitable for surveillance cameras that are required to record instantaneous events.

Regarding the means for irradiating light in [3] above, it is necessary to install a device for irradiating light, which increases cost and power consumption. Furthermore, there are cases where the irradiation of visible light may affect the subject. Furthermore, when near-infrared light is used, a problem arises in that color images cannot be obtained.

Regarding the means for performing gain correction on the video signal in [4] above, video noise also increases during gain correction, resulting in a noisy image. Furthermore, a problem arises in that the gradation of the image becomes rough. Therefore, there is a problem in that the quality of the acquired image is significantly degraded.

In view of the above problems, it is an object of the present invention to provide an imaging device that can obtain an image with a deep apparent depth of field even when the aperture is opened.

In order to achieve the above object, one of the typical imaging devices of the present invention is capable of separately receiving visible light and near-infrared light with respect to incident light, and is capable of receiving a visible light signal based on the visible light and a near-infrared light signal based on the visible light. an image sensor unit that outputs a near-infrared light signal based on infrared light, and a process for extracting a contour component from an image generated by the near-infrared light signal and superimposing the extracted contour component on the image generated by the visible light signal. and a video signal processing section.

According to the present invention, even when the lens aperture is opened in the imaging device, an image with a deep apparent depth of field can be obtained.
Problems, configurations, and effects other than those described above will be clarified by the following embodiments.

FIG. 1 is a block diagram of a computer system for implementing aspects according to embodiments of the present disclosure. FIG. 2 is a block diagram showing an example of the configuration of an imaging device according to the present invention. FIG. 3 is a diagram showing a first example of the image sensor section in the image pickup apparatus of the present invention. FIG. 4 is a diagram showing a second example of the image sensor section in the image pickup apparatus of the present invention. FIG. 5 is a diagram showing a third example of the image sensor section in the image pickup apparatus of the present invention. FIG. 6 is a diagram showing a fourth example of the image sensor section in the image pickup apparatus of the present invention. FIG. 7 is a diagram showing a fifth example of the image sensor section in the image pickup apparatus of the present invention. FIG. 8 is a diagram showing an example of a flowchart of the imaging apparatus of the present invention. FIG. 9 is a diagram illustrating a specific example of superimposing contour components from a visible light image and a near-infrared light image in the imaging apparatus of the present invention.

A mode for carrying out the present invention will be described.

<Computer system for implementing aspects according to embodiments>
FIG. 1 is a block diagram of a computer system 300 for implementing aspects according to embodiments of the present disclosure. The mechanisms and apparatus of the various embodiments disclosed herein may be applied to any suitable computing system. The main components of computer system 300 include one or more processors 302 , memory 304 , terminal interface 312 , storage interface 314 , I/O (input/output) device interface 316 , and network interface 318 . These components may be interconnected via memory bus 306, I/O bus 308, bus interface unit 309, and I/O bus interface unit 310.

Computer system 300 may include one or

more processing devices

302A and 302B, collectively referred to as processor 302. Each processor 302 executes instructions stored in memory 304 and may include onboard cache. In some embodiments, computer system 300 may include multiple processors, and in other embodiments, computer system 300 may be a single processing unit system. Processing devices include CPU (Central Processing Unit), FPGA (Field-Programmable Gate Array), GPU (Graphics Processing Unit), and DSP (Digital Processing Unit). al Signal Processor) etc. can be applied.

In some embodiments, memory 304 may include random access semiconductor memory, storage devices, or storage media (either volatile or nonvolatile) for storing data and programs. In some embodiments, memory 304 represents the entire virtual memory of computer system 300 and may include the virtual memory of other computer systems connected to computer system 300 via a network. Although memory 304 may be conceptually considered a single entity, in other embodiments this memory 304 may be a more complex arrangement, such as a hierarchy of caches and other memory devices. . For example, memory may exist as multiple levels of caches, and these caches may be divided by function. As a result, one cache may hold instructions while the other cache holds non-instruction data used by the processor. Memory may be distributed and associated with various different processing units, such as in a so-called NUMA (Non-Uniform Memory Access) computer architecture.

Memory 304 may store all or a portion of the programs, modules, and data structures that perform the functions described herein. For example, memory 304 may store latent factor identification application 350. In some embodiments, latent factor identification application 350 may include instructions or writings that perform functions described below on processor 302, or may include instructions or writings that are interpreted by other instructions or writings. In some embodiments, latent factor identification application 350 may be applied to semiconductor devices, chips, logic gates, circuits, circuit cards, and/or other physical hardware instead of or in addition to processor-based systems. It may also be implemented in hardware via a device. In some embodiments, latent factor identification application 350 may include data other than instructions or descriptions. In some embodiments, cameras, sensors, or other data input devices (not shown) may be provided to communicate directly with bus interface unit 309, processor 302, or other hardware of computer system 300. . Such a configuration may reduce the need for processor 302 to access memory 304 and latent factor identification applications.

Computer system 300 may include a bus interface unit 309 that provides communication between processor 302 , memory 304 , display system 324 , and I/O bus interface unit 310 . I/O bus interface unit 310 may be coupled to I/O bus 308 for transferring data to and from various I/O units. I/O bus interface unit 310 connects, via I/O bus 308, a plurality of I/

O interface units

312, 314, 316, also known as I/O processors (IOPs) or I/O adapters (IOAs). and 318. Display system 324 may include a display controller, display memory, or both. A display controller may provide video, audio, or both data to display device 326. Computer system 300 may also include devices, such as one or more sensors, configured to collect data and provide the data to processor 302. For example, computer system 300 may include environmental sensors that collect humidity data, temperature data, pressure data, etc., motion sensors that collect acceleration data, motion data, etc., and the like. Other types of sensors can also be used. The display memory may be a dedicated memory for buffering video data. Display system 324 may be connected to a display device 326, such as a standalone display screen, a television, a tablet, or a handheld device. In some embodiments, display device 326 may include speakers to render audio. Alternatively, a speaker for rendering audio may be connected to the I/O interface unit. In other embodiments, the functionality provided by display system 324 may be implemented by an integrated circuit that includes processor 302. Similarly, the functionality provided by bus interface unit 309 may be implemented by an integrated circuit that includes processor 302.

The I/O interface unit has the ability to communicate with various storage or I/O devices. For example, the terminal interface unit 312 may include a user output device such as a video display device, a speaker television, or a user input device such as a keyboard, mouse, keypad, touch pad, trackball, button, light pen, or other pointing device. It is possible to attach user I/O devices 320 such as: Using the user interface, a user operates a user input device to input input data and instructions to user I/O device 320 and computer system 300, and to receive output data from computer system 300. Good too. The user interface may be displayed on a display device, played through a speaker, or printed through a printer, for example, via the user I/O device 320.

Storage interface 314 may include one or more disk drives or direct access storage devices 322 (typically magnetic disk drive storage devices, but an array of disk drives or other storage devices configured to appear as a single disk drive). ) can be installed. In some embodiments, storage device 322 may be implemented as any secondary storage device. The contents of memory 304 may be stored in storage device 322 and read from storage device 322 as needed. Network interface 318 may provide a communication pathway so that computer system 300 and other devices can communicate with each other. This communication path may be, for example, network 330.

The computer system 300 shown in FIG. 1 includes a bus structure that provides a direct communication path between a processor 302, a memory 304, a bus interface 309, a display system 324, and an I/O bus interface unit 310; In embodiments, computer system 300 may include point-to-point links in a hierarchical, star, or web configuration, multiple hierarchical buses, and parallel or redundant communication paths. Additionally, although I/O bus interface unit 310 and I/O bus 308 are shown as a single unit, computer system 300 may actually include multiple I/O bus interface units 310 or multiple I/O bus A bus 308 may also be provided. Additionally, although multiple I/O interface units are shown to separate I/O bus 308 from various communication paths leading to various I/O devices, in other embodiments, one of the I/O devices may Some or all may be directly connected to one system I/O bus.

In some embodiments, computer system 300 is a device that receives requests from other computer systems (clients) that do not have a direct user interface, such as a multi-user mainframe computer system, a single-user system, or a server computer. There may be. In other embodiments, computer system 300 may be a desktop computer, portable computer, laptop, tablet computer, pocket computer, telephone, smart phone, or any other suitable electronic device.

<Block diagram of imaging device>
FIG. 2 is a block diagram showing an example of the configuration of an imaging device according to the present invention. In FIG. 2, the imaging device 1 includes a lens section 2, an image sensor section 3, a video signal processing section 4, and a control section 5. The imaging device 1 captures video at, for example, 3 frames per second (3 fps) or more, and performs processing to be described later. Incident light from a subject is imaged by the lens section 2, and photoelectrically converted into an electrical signal by the image sensor section 3.

The lens section 2 includes concave and convex lenses 6 that adjust zoom and focus, an aperture 7 that adjusts the amount of light, and a near-infrared light cut filter 8. These are provided in this order from the incident light side: each concave-convex lens 6, an aperture 7, and a near-infrared light cut filter 8. Note that various configurations can be applied to the configuration of each concave-convex lens 6 depending on the purpose.

Here, the aperture 7 will be explained. The aperture 7 adjusts the amount of light entering from the concave-convex lens 6 by changing the size of the central hole using a structure such as a plurality of blades. Opening the aperture increases the amount of light, allowing you to capture good images even in dark surroundings. However, in this case, the depth of field becomes shallow, and objects at a certain distance are in focus, while objects at other distances tend to be blurred. This makes it difficult to identify objects at a distance other than a specific distance. On the other hand, if the surroundings are bright, there will be enough light even when the aperture is closed, and the depth of field will be deep. In this case, it is easy to focus over a wide range of different distances, and even at distances that are out of focus, the degree of blur is small.

The near-infrared light cut filter 8 is a filter that cuts only near-infrared light and allows visible light to pass through. Here, near-infrared light is light with a wavelength of approximately 800 to 2,500 nanometers. Furthermore, visible light is light with a wavelength of about 380 to 770 nanometers. The near-infrared light cut filter 8 is located adjacent to the diaphragm 7, and is open only in the center, so that it is transparent without a filter. The size of the aperture in the central part where there is no filter can be set to be smaller. For example, the aperture may be made smaller than the aperture when the diaphragm 7 is at its most closed position. Further, the aperture may be smaller than the aperture of the diaphragm 7 at the time of the brightness level threshold value in step S101 in FIG. 8, which will be described later. As a result, it is possible to set a deep depth of field for near-infrared light.

The image sensor section 3 converts the incident light that has passed through the lens section 2, the aperture 7, and the near-infrared cut filter 8 into an electrical signal, and transmits the signal to the video signal processing section 4. Examples of the image sensor here include a CCD (Charge-Coupled Device) image sensor, a CMOS (Complementary Metal Oxide Semiconductor) image sensor, and the like. The image sensor section 3 is configured to be able to separately receive visible light and near-infrared light, and the detailed configuration will be described later with reference to FIGS. 3 to 7.

The video signal processing unit 4 performs various video signal processing such as gain correction, gamma correction, knee correction, contour correction, and color correction on the signal input from the image sensor unit 3. The video signal processing section 4 includes a contour extraction section 9. The contour extraction section 9 performs contour correction processing on the near-infrared light signal input from the image sensor section 3 and outputs it as a contour signal. Then, the above contour signal is added to the visible light signal inputted from the image sensor section 3 to create a video signal. An HD-SDI (High Definition Serial Digital Interface) signal is generated from this video signal and output to the outside. Note that the video signal output from the video signal processing unit 4 is not limited to the HD-SDI described above, and other types of video signals such as compressed and encrypted video signals may be applied.

The control unit 5 controls the lens unit 2, the image sensor unit 3, and the video signal processing unit 4 of the imaging device 1. The control unit 5 can be configured by, for example, a CPU.

The computer system 300 shown in FIG. 1 can be applied to the video signal processing section 4 and the control section 5.

<Specific example of image sensor section>
A specific example of the image sensor section 3 will be explained using FIGS. 3 to 7.

FIG. 3 is a diagram showing a first example of the image sensor section in the image pickup device of the present invention. The image sensor section 3 shown in FIG. 3 includes an image sensor 31 having a filter 31a. The filter 31a is a filter installed on the incident light side, and is a filter in which a plurality of visible light filters "Vis" and a plurality of near-infrared light filters "IR" are arranged alternately in a mosaic pattern (checkerboard pattern). be. The visible light filter "Vis" is a filter that transmits only visible light. The near-infrared light filter "IR" is a filter that transmits only near-infrared light. Therefore, the image sensor section 3 in FIG. 3 is a monochrome single-plate type, and serves as an image sensor for monochrome images. The incident light passes through the filter 31a and reaches the image sensor 31, but at this time, the visible light filter "Vis" and the near-infrared light filter "IR" separate visible light and near-infrared light for each light receiving element. Receive light. Then, it is output to the video signal processing unit 4 as a visible light signal and a near-infrared light signal. Although FIG. 3 shows an example of 16 filters, the arrangement is not limited to this and can be arranged according to the number of light receiving elements.

FIG. 4 is a diagram showing a second example of the image sensor section in the imaging device of the present invention. The image sensor section 3 shown in FIG. 4 includes a prism 12, an image sensor 32, and an image sensor 33. The prism 12 is a prism that separates incident light into visible light and near-infrared light. The image sensor 32 is an image sensor that receives visible light separated by the prism 12. The image sensor 33 is an image sensor that receives near-infrared light separated by the prism 12. For this reason, the image sensor section 3 in FIG. 4 is a monochrome two-plate type using two monochrome image sensors that each receive visible light and near-infrared light. The incident light passes through the prism 12 and is separated into visible light and near-infrared light. The image sensor 32 receives only visible light and outputs it to the video signal processing section 4 as a visible light signal. The image sensor 33 receives only near-infrared light and outputs it to the video signal processing section 4 as a near-infrared light signal. The video signal processing unit 4 generates a monochrome image based on the received visible light signal. Therefore, both image sensors do not have a mosaic filter as shown in FIG. 3.

FIG. 5 is a diagram showing a third example of the image sensor section in the imaging device of the present invention. The image sensor section 3 shown in FIG. 5 includes an image sensor 34 having a filter 34a. The filter 34a is a filter installed on the incident light side, and includes a plurality of red light filters "R", a plurality of green light filters "G", a plurality of blue light filters "B", and a plurality of near-infrared light filters. Optical filters "IR" are arranged alternately in a mosaic pattern. In Figure 5, multiple patterns are lined up with four patterns: a green light filter "G" in the upper left, a red light filter "R" in the upper right, a green light filter "G" in the lower left, and a near-infrared light filter "IR" in the lower right. Out. Green light filter "G" is a filter that passes only green light. The red light filter "R" is a filter that passes only red light. Blue light filter "B" is a filter that passes only blue light. The near-infrared light filter "IR" is a filter that transmits only near-infrared light. As a result, the image sensor section 3 in FIG. 5 is of a color single-plate type, and serves as an image sensor for color images.

In FIG. 5, the incident light passes through the filter 34a and reaches the image sensor 34, but at this time, the visible light passes through the red light filter "R" as red light and passes through the green light filter "G". The light that has passed is received as green light, and the light that has passed through the blue light filter "B" is received as blue light by the image sensor 34. Signals based on these received lights are output to the video signal processing section 4 as visible light signals. Further, the light that has passed through the near-infrared light filter "IR" is received by the image sensor 34 as near-infrared light. A signal based on this received light is output to the video signal processing section 4 as a near-infrared light signal. This makes it possible to receive visible light and near-infrared light separately for each light receiving element. Although FIG. 5 shows an example of 16 filters, the arrangement is not limited to this and can be arranged according to the number of light receiving elements.

FIG. 6 is a diagram showing a fourth example of the image sensor section in the imaging device of the present invention. The image sensor section 3 shown in FIG. 6 includes a prism 12, an image sensor 35, and an image sensor 36. The image sensor 35 has a filter 35a. The prism 12 is a prism that separates incident light into visible light and near-infrared light, and is similar to the prism 12 in FIG. 4 . The filter 35a is a filter in which a plurality of red light filters "R", a plurality of green light filters "G", and a plurality of blue light filters "B" are alternately arranged in a mosaic pattern. FIG. 6 shows a Bayer array filter 35a that receives visible light. The image sensor 35 is an image sensor that receives visible light separated by the prism 12. The image sensor 36 is an image sensor that receives near-infrared light separated by the prism 12. For this reason, the image sensor section 3 in FIG. 6 is a color two-plate type using an image sensor that receives visible light and a monochrome image sensor that receives near-infrared light.

In FIG. 6, incident light passes through a prism 12 and is separated into visible light and near-infrared light. The image sensor 35 receives only visible light and outputs it to the video signal processing section 4 as a visible light signal. The image sensor 36 receives only near-infrared light and outputs it to the video signal processing section 4 as a near-infrared light signal. At this time, the image sensor 35 can receive light as an RGB color image through the filter 35a. Although FIG. 6 shows an example of 16 filters, the arrangement is not limited to this and can be arranged according to the number of light receiving elements.

FIG. 7 is a diagram showing a fifth example of the image sensor section in the imaging device of the present invention. The image sensor section 3 shown in FIG. 7 includes a prism 15, an image sensor 37, an image sensor 38, an image sensor 39, and an image sensor 40. The prism 15 is a prism that separates incident light into red light, green light, blue light, and near-infrared light, respectively. The image sensor 37 is an image sensor that receives red light separated by the prism 15. The image sensor 38 is an image sensor that receives green light separated by the prism 15. The image sensor 39 is an image sensor that receives blue light separated by the prism 15. The image sensor 40 is an image sensor that receives near-infrared light separated by the prism 15. For this reason, the image sensor section 3 in FIG. 7 is a color four-plate type using four monochrome image sensors that receive red light, green light, blue light, and near-infrared light, respectively.

In FIG. 7, incident light passes through a prism 15 and is separated into red light, green light, blue light, and near-infrared light. The image sensor 37 receives only red light, the image sensor 38 receives only green light, and the image sensor 39 receives only blue light. Signals based on these received lights are output to the video signal processing section 4 as visible light signals. Further, the image sensor 40 receives only near-infrared light. A signal based on this received light is output to the video signal processing section 4 as a near-infrared light signal. The video signal processing unit 4 generates a color video based on the received red light, green light, and blue light. Therefore, both image sensors do not have a mosaic filter as shown in FIG. 6.

<Example of flowchart>
FIG. 8 is a diagram showing an example of a flowchart of the imaging apparatus of the present invention. The operation of expanding the apparent depth of field when the aperture 7 of the lens section 2 is opened will be described using FIG. 8. Here, unless otherwise explained, processing in the video signal processing section 4 is shown.

First, in step S101, it is determined whether the brightness level of the video input to the video signal processing unit 4 is smaller than a threshold value. If it is determined that the brightness level of the input video is smaller than the threshold, the process advances to step S102. If the brightness level of the input video is equal to or higher than the threshold, the process ends. The brightness level threshold can be set in advance. As the input video here, a video based on a visible light signal can be used.

In step S102, the aperture 7 of the lens section 2 is opened so that a predetermined brightness level is obtained. The predetermined brightness level here may be, for example, the brightness level threshold value in step S101. If the brightness level is low, the image will be dark, so by opening the aperture 7, the brightness of the image can be maintained at a predetermined level. The degree to which the aperture 7 opens may be determined according to the brightness level detected in step S101. Here, the aperture 7 can be controlled and opened under the control of the control section 5 according to the brightness level detected by the video signal processing section 4. However, although opening the aperture 7 increases the amount of light and raises the brightness level, the depth of field becomes shallower and the degree of blur in the image at a distance other than the specific distance increases.

Next, in step S103, the contour extraction unit 9 extracts contour components from the near-infrared light signal. The near-infrared light signal is a signal of near-infrared light obtained by passing through a near-infrared light cut filter 8 adjacent to the aperture 7. The near-infrared light cut filter 8 is open only at the center and is transparent, so that most of the near-infrared light does not pass through, but only a part of the center passes through. That is, the optical path of the near-infrared light is not affected by the diaphragm 7, and is always in the same state as if the diaphragm were closed. As a result, an image with a deep depth of field can be obtained although the brightness level of the near-infrared light signal is low.

The video signal processing unit 4 creates a video (near-infrared video) based on the near-infrared light signal, but since the near-infrared video has a low brightness level, the video is dark as a whole. For this reason, the near-infrared light image may be subjected to gain correction as necessary, or may be passed through an LPF (Low-Pass Filter) to remove noise. Here, although the LPF may slightly degrade the resolution, compared to a blurred image with a shallow depth of field, it is possible to obtain an image sufficient for extracting contour components. Extraction of the contour component can be performed by performing image processing on each image of the near-infrared light video and extracting the contour of each subject.

Next, in step S104, the video signal processing unit 4 superimposes the contour component extracted from the near-infrared light image onto the visible light image. The visible light image is an image created by a visible light signal that has passed through the aperture 7. The superimposition involves aligning and superimposing outline components from visible light images and near-infrared light images taken at the same time. As a result, the outline component of the near-infrared light image is added to the blurred visible light image with a shallow depth of field, so the outline of each subject can be made clear. For visible light images with a shallow depth of field, blurred images (brightness blur in monochrome images, brightness and color blur in color images) cannot be completely eliminated, but the addition of contour components This makes it possible to create an image with a seemingly deep depth of field. At this time, the visible light image is not subjected to large gain correction to increase brightness, nor is processing to reduce the frame rate. Therefore, it is possible to obtain an image without deterioration such as noise, gradation, afterimage, etc., and with an appropriate brightness level.

<Specific example of superimposition>
FIG. 9 is a diagram illustrating a specific example of superimposing contour components from a visible light image and a near-infrared light image in the imaging apparatus of the present invention.

The visible light image 201 is an image obtained when the aperture 7 is opened in step S102 of FIG. Although the brightness level can be increased by opening the aperture 7, the depth of field becomes shallower. In the example of FIG. 9, the car in the foreground is in focus, and other subjects are out of focus, resulting in a blurred image (an image with blurred colors). The brightness level of the image is maintained at a predetermined level so as not to become dark.

The contour image 202 is an image representing the contour components extracted in step S103 of FIG. 8. The near-infrared light passes through the near-infrared light cut filter 8 that is open only in the center, resulting in an image with a deep depth of field, but initially the entire image is dark. For this reason, gain correction is performed to increase the gain and brighten the image. At this time, since noise also increases, noise correction such as DNR (Digital Noise Reduction) is also applied. Then, contour components are extracted based on the image. The contour image 202 in FIG. 9 has a deep depth of field, so that not only the car in the foreground but also buildings, roads, billboards, and other cars in the background are in focus or slightly out of focus. Therefore, it is possible to capture the contour components of each subject (each object) through image processing. The contour component of the contour image 202 can represent the contour of each subject with a line or the like.

The superimposed video 203 is a video that has undergone the superimposition process in step S104 of FIG. 8. In the example of FIG. 9, a visible light image 201 and an outline image 202 are superimposed. Since the visible light image 201 maintains the brightness level, the image does not become dark as a whole. Furthermore, since the contour image 202 adds contour components over a wide range, the contour can be made clear and the apparent depth of field can be increased. This allows you to clearly capture not only the car in the foreground, but also other subjects.

<Effect>
According to the embodiments described above, even when the aperture of the lens is opened, it is possible to obtain an image with a deep apparent depth of field. As a result, for example, even when a surveillance camera takes pictures in an environment with a low amount of light, such as in the evening or at night, it is possible to obtain images that do not become dark. In this case, the apparent depth of field becomes deeper, making it possible to monitor not only one specific point but a wide range of multiple points. At this time, there is no need to lower the frame rate, so there is no increase in afterimages that occur on the subject.

Note that the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the embodiments described above are described in detail to explain the present invention in an easy-to-understand manner, and the present invention is not necessarily limited to having all the configurations described. Furthermore, it is possible to replace a part of the configuration of one embodiment with the configuration of another embodiment, and it is also possible to add the configuration of another embodiment to the configuration of one embodiment. Further, it is possible to add, delete, or replace a part of the configuration of each embodiment with other configurations.

For example, the present invention can be applied not only when the amount of light is insufficient, but also when the amount of light is sufficient. In this case, a predetermined effect can be obtained by combining it with other operations that affect the brightness level. For example, it is possible to increase the frame rate by shortening the exposure time by the amount of light increased by opening the aperture 7 of the lens, or to improve the image quality by performing gain correction in the compression direction.

Further, although FIG. 2 shows an example in which the aperture 7 and the near-infrared light cut filter 8 are close to each other, the present invention is also applicable even if they are separated. For example, if the aperture 7 is narrowed down to the same size as the aperture of the near-infrared light cut filter 8, near-infrared light will not reach anything other than parallel light from the central aperture, but in this case, the visible light image will be Because the depth is deep, contour correction is not necessary. Furthermore, even if there is a difference in focal position, no problem will occur because near-infrared light has a deep depth of field. Furthermore, if there is a difference in zoom magnification, it can be corrected by correcting it using aberration correction (digital zoom).

DESCRIPTION OF SYMBOLS 1...Imaging device, 2...Lens part, 3...Image sensor part, 4...Video signal processing part, 5...Control part, 6...Concave-convex lens, 7...Aperture, 8...Near-infrared light cut filter, 9...Contour extraction Part, 10... Imaging device, 12, 15... Prism, 31-40... Imaging element, 31a, 34a, 35a... Filter, 201... Visible light image, 202... Contour image, 203... Superimposed image, 300... Computer system, 302 ...Processor, 302A, 302B...Processing device, 304...Memory, 306...Memory bus, 308...I/O bus, 309...Bus interface unit, 310...I/O bus interface unit, 312...Terminal interface unit, 314...Storage Interface, 316...I/O device interface, 318...Network interface, 320...User I/O device, 324...Display system, 326...Display device, 330...Network, 350...Latent factor identification application

Claims

an image sensor section capable of separately receiving visible light and near-infrared light with respect to incident light and outputting a visible light signal based on the visible light and a near-infrared light signal based on the near-infrared light; An imaging device comprising: a video signal processing unit that extracts a contour component from an image based on an infrared light signal and superimposes the extracted contour component on a video based on the visible light signal.
The imaging device according to claim 1,
It has a lens, an aperture that adjusts the amount of light entering from the lens, and a control unit that controls the opening degree of the aperture, and the control unit adjusts the aperture according to the brightness level of the image based on the visible light signal. An imaging device characterized by performing control to open an image.
The imaging device according to claim 1,
An imaging device comprising a near-infrared cut filter having an opening with no filter only in the center, and wherein the image sensor section receives light that has passed through the near-infrared cut filter.