JP2022189835A - Imaging apparatus - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a technique for imaging both a visual image and an infrared image, and imaging a high-quality image by improving color reproducibility in visible light imaging.

SOLUTION: An imaging system is configured to enable stereo imaging for both a visible image and an infrared image, and improve color reproducibility in visible light imaging. The imaging system includes: two imaging sensors 12; and two DBPFs which are optical filters having a visible light band and a second wavelength band with transparency and arranged for the two imaging sensors. The imaging system has at least four types of filter units which are different in spectral transmission characteristics in accordance with a wavelength in the visible light band and similar in transmittance in the second wavelength band, and two color filters arranged for the two imaging sensors, respectively. The imaging system measures a distance to an object based on two visible or infrared image signals.

SELECTED DRAWING: Figure 8

COPYRIGHT: (C)2023,JPO&INPIT

Description

The present invention relates to an imaging device and an imaging system.

In recent years, by combining image recognition technology and biometric authentication, the development of surveillance cameras that detect criminals, shoplifters, terrorists, etc. by facial recognition, and in-vehicle cameras that are used for automatic driving of automobiles is underway.

For example, as a surveillance camera, there is known a system that detects an intrusion of a suspicious person by detecting a distance using a stereo camera (see Patent Document 1). Patent Literature 1 describes an "image recognition device for distance measurement using stereo images" as a method for measuring the shape and distance of an object, and a technique called a stereo method for obtaining distances from stereo images is known. ing. In this stereo method, two left and right images, called stereo images, are first input, and the corresponding points in the left and right images are calculated by calculating the feature values in the images. Is it in the picture?) For details of how to find the corresponding points, see, for example. It is described in Patent Document 2 as an "image matching method". Then, when the corresponding points of the two left and right images are obtained, the distance to the object surface can be calculated by the principle of triangulation, so the distance to the object and the shape of the object surface can be known.

In addition, Patent Document 1 proposes a moving object recognition apparatus that can detect a moving object with high precision and high speed and measure its shape and distance by using known correspondence relationships of stereo images. .

Surveillance cameras and self-driving cameras need to shoot regardless of location or time, such as outdoors and indoors, day and night, but depending on the situation, there may be cases where sufficient lighting cannot be obtained. In this case, it is conceivable to perform infrared photographing using infrared illumination that is invisible to humans. Also, in the camera for automatic driving, taking into consideration the effect of headlights on oncoming vehicles at night, it is conceivable to perform infrared photography using infrared light as lighting for illuminating a distant place. In any case, during the day when there is a high possibility that the amount of visible light is sufficient, visible light photography is performed without lighting, and when nighttime lighting is required, infrared photography is performed using infrared lighting that is difficult for the human eye to see. can be considered.

Considering such a situation, it is preferable that the surveillance camera and the automatic driving camera are capable of photographing with both visible light and infrared light.

2. Description of the Related Art An imaging device such as a monitoring camera that takes images continuously day and night detects infrared light to take images at night. A photodiode (light-receiving element), which is a light-receiving part of an imaging sensor such as a CCD sensor or a CMOS sensor, can receive light up to a near-infrared wavelength band of about 1300 nm. In principle, it is possible to capture images up to the infrared band.

Since the wavelength band of light for which human visibility is high is 400 nm to 700 nm, when the image sensor detects near-infrared light, the image appears redder to the human eye. For this reason, when shooting in the daytime or in a bright place indoors, in order to match the sensitivity of the image sensor to human visibility, an infrared cut filter that cuts off light in the infrared band is provided in front of the image sensor. It is desirable to remove light with a wavelength of 700 nm or more. On the other hand, when photographing at night or in a dark place, it is necessary to photograph without providing an infrared cut filter.

As such an imaging device, an imaging device that manually attaches and detaches an infrared cut filter and an imaging device that automatically inserts and removes an infrared cut filter are conventionally known. Furthermore, an imaging device is disclosed that eliminates the need to insert and remove the infrared cut filter described above. For example, a second wavelength band that has transmission characteristics in the visible light band, has cutoff characteristics in a first wavelength band adjacent to the long wavelength side of the visible light band, and is a part of the first wavelength band An optical filter has been proposed that has a transmission characteristic of . According to this filter, light can be transmitted both in the visible light band and in a second wavelength band on the longer wavelength side of the visible light band, that is, on the infrared side and away from the visible light band.

Hereinafter, an optical filter that transmits light in the visible light band and the second infrared wavelength band and blocks light in other wavelength bands as described above will be referred to as a DBPF (double band pass filter). .

In recent years, as biometric authentication, various authentication technologies such as fingerprint, face, iris, vein, signature, voiceprint, and walking have been developed. Biometric authentication used together mainly includes face authentication and iris authentication.

JP-A-3-81878 JP-A-62-107386 Japanese Patent No. 5009395

In the DBPF of Patent Document 3, the light in the second wavelength band (relatively narrow wavelength band included in the infrared wavelength band) included in the infrared (near-infrared) wavelength band is transmitted without being blocked at all times. will do. That is, unlike the case of using an infrared cut filter that cuts wavelengths longer than the visible light band, the infrared light transmitted through the second wavelength band is not a little affected when photographing in the visible light band. Become.

Color filters are used in imaging sensors that perform color imaging in the visible light band. The color filter consists of red, green, and blue regions (filter sections) arranged in a predetermined pattern corresponding to each pixel of the image sensor. and blocks the transmission of light in wavelength bands of other colors.

However, on the longer wavelength side than the visible light band, light is basically transmitted although the light transmittance differs depending on the region and wavelength of each color. Therefore, when infrared light is transmitted in the second wavelength band on the infrared side as in the DBPF described above, the infrared light passes through the color filter and reaches the photodiode (light receiving element) of the image sensor. It increases the amount of electrons generated by the effect.

Further, when both color photography with visible light and photography with infrared light illumination are performed, for example, a color filter in which regions of red, green, and blue are arranged in a predetermined pattern is provided with the above-described second A region for infrared light (infrared region) having a peak of light transmittance in the wavelength band of . That is, the array (pattern) of the color filters consists of four areas of red R, green G, blue B, and infrared IR. In this case, the region for infrared light blocks light in the visible light band and mainly transmits light in the second wavelength band. It is conceivable to remove infrared light components from image signals of red, green, and blue colors by using infrared light image signals output from an imaging sensor that receives the image. However, even with such signal processing, it has been difficult to achieve color reproduction substantially equivalent to that in color photography when an infrared cut filter is used. Further, when the distance is calculated in stereo, if there is a difference between the left and right signal levels, it causes an error in the parallax calculation.

Furthermore, when using face authentication using images captured by a camera, the user is asked to face a predetermined camera in an entry/exit management system for offices, buildings, etc., boarding procedures at airports, immigration control, etc. Authentication for type and an unspecified number of users who unconsciously take pictures with multiple cameras at public facilities, airports, transportation facilities, etc. There is a type to In the former type, since the shooting conditions are limited, it is possible to recognize with high accuracy even with recent technology. , the recognition rate is greatly affected.

In order to detect suspicious persons using surveillance cameras, it is possible to shoot continuously for 24 hours using both visible light and infrared light without depending on the shooting location or shooting time. Furthermore, if the captured images can be made clear and high-resolution with as little noise as possible, it is believed that the ability to detect suspicious persons early and analyze the situation when a crime occurs will be greatly improved.

In addition, by adopting a configuration that enables distance measurement using the stereo method using two cameras, even in the case of the camera for autonomous driving, when using both infrared and visible imaging as described above, clear images with little noise can be obtained. can be obtained, the accuracy of image recognition can be improved.

From the above, it is possible to capture both infrared and visible images at the same time, and the image quality such as noise, resolution, and color reproducibility is at the same level or higher than that of normal visible images that do not include infrared images. Furthermore, it is desirable to be able to shoot in stereo with a two-camera configuration.

INDUSTRIAL APPLICABILITY The present invention provides a technology capable of capturing both a visible image and an infrared image, improving color reproducibility during visible light capturing, and capturing a high-quality image.

An imaging device or an imaging system according to the present invention includes an imaging device, and a filter unit that has a characteristic of transmitting at least a visible light wavelength region and an infrared wavelength region, and filters a signal from the imaging device based on the characteristic. a signal processing unit that processes the signal filtered by the filter unit and outputs a visible light signal and an infrared signal; first data including at least one of a visible light signal and an infrared signal output from the signal processing; and the moving object region extraction unit generated by the and a signal output control unit that transmits second data based on information about the moving body to the outside.

Further, the imaging device or imaging system according to the present invention has two imaging elements and a characteristic of transmitting at least a visible light wavelength region and an infrared wavelength region, and filters signals from the imaging elements based on the characteristics. two filter units, two signal processing units that process the signals filtered by the filter units and output visible light signals and infrared signals, two visible image signals output from the signal processing units, and / Or a distance calculation unit for calculating a distance to a subject captured in a visible image by a visible image signal and/or an infrared image by an infrared image signal using two infrared image signals, and output from the signal processing unit a moving object region extracting unit that generates information about a moving object in an image captured by the imaging element from the infrared signal obtained; and at least one of the visible light signal and the infrared signal output from the signal processing 1 data, second data based on the information about the moving object generated by the moving object region extracting unit, and third data based on the distance information calculated by the distance calculating unit, are transmitted to the outside. and a control unit.

According to the present invention, it is possible to obtain high-quality images. More specifically, for example, according to one aspect of the present invention, both a high-quality infrared image and a visible image can be simultaneously captured by a camera configured with one imaging sensor and one optical film. Visibility can be improved even in changing environments such as lack of lighting or at night. According to another aspect of the invention, both the infrared image and the visible image can more accurately measure the distance of an object, and the information can be provided to an external system along with the visible image or the infrared image. According to another aspect of the present invention, by using an infrared image instead of a visible image, it is possible to extract a moving object region in the image at a higher speed, and the information is transmitted to an external system together with the visible image or the infrared image. can be provided to

1 is a schematic diagram showing an imaging system according to Embodiment 1 of the present invention; FIG. 1 is a schematic diagram showing the configuration of an imaging sensor unit of an imaging system according to Embodiment 1 of the present invention; FIG. 4 is a graph showing transmittance spectra of the DBPF and color filters of the imaging sensor of the imaging system according to Embodiment 1 of the present invention; 1 is a schematic diagram showing one configuration example of a color filter of an imaging system according to Embodiment 1 of the present invention; FIG. 2 is a block diagram for explaining a signal processing section of the imaging system according to Embodiment 1 of the present invention; FIG. FIG. 4 is a flow chart diagram explaining a communication flow between the camera and the controller of the imaging system according to Embodiment 1 of the present invention; 4 is a diagram showing one configuration of various image information handled by the imaging system according to Embodiment 1 of the present invention; FIG. FIG. 4 is a schematic diagram showing an imaging system according to Embodiment 2 of the present invention; FIG. 10 is a diagram showing one configuration of various types of analysis metadata information handled by the imaging system according to Embodiment 2 of the present invention; FIG. 10 is a flow chart diagram illustrating a communication flow between a camera and a controller of the imaging system according to Embodiment 2 of the present invention; FIG. 10 is a diagram showing an image screen example of an image handled by the imaging system according to Embodiment 2 of the present invention; FIG. 10 is a diagram showing a configuration example of analysis metadata information handled by the imaging system according to Embodiment 2 of the present invention; FIG. 5 is a schematic diagram showing an imaging system according to Embodiment 3 of the present invention; FIG. 11 is a schematic diagram showing another configuration example of the imaging system according to Embodiment 3 of the present invention; FIG. 11 is a schematic diagram showing another configuration example of the imaging system according to Embodiment 3 of the present invention; FIG. 11 is a schematic diagram showing another configuration example of the imaging system according to Embodiment 3 of the present invention; FIG. 10 is a diagram showing one configuration of various image information handled by an imaging system according to Embodiment 3 of the present invention; FIG. 10 is a diagram showing one configuration of various image information handled by an imaging system according to Embodiment 3 of the present invention; FIG. 10 is a diagram showing one configuration of analysis metadata information handled by the imaging system according to Embodiment 3 of the present invention; FIG. 10 is a diagram showing a configuration example of analysis metadata information handled by the imaging system according to Embodiment 3 of the present invention; 4 is a graph showing transmittance spectra of the DBPF and color filters of the imaging sensor of the imaging system according to Embodiment 1 of the present invention; FIG. 10 is a diagram showing a processing flow of an imaging device of an imaging system according to Embodiment 3 of the present invention; It is an example of operating the imaging system according to Embodiment 3 of the present invention. It is an example of operating the imaging system according to Embodiment 3 of the present invention.

(Embodiment 1)
BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 shows a configuration example of an imaging system according to Embodiment 1 of the present invention. The imaging system is roughly composed of one or more imaging devices 100 ((a) to (n)) and one or more controller devices 200 . The imaging device 100 and controller device 200 are connected via a network 303 . Although the network 303 is assumed to be a wired LAN (Local Area Network) in this embodiment, it may be a general-purpose network such as a wireless LAN (WiFi), USB (Universal Serial Bus), IEEE1394, or the like.

In the network 303, standard IP (Internet Protocol) is used as a network protocol, and TCP (Transmission Control Protocol) and UDP (User Datagram Protocol) are used as upper transport protocols. To transfer the image captured by the imaging device 100, a higher-level application protocol such as RTP (Real-time Transport Protocol)/RTCP (RTP Control Protocol) or HTTP (Hyper Text Transfer Protocol) is used to control the transfer. uses RTSP (Real-Time Streaming Protocol) or the like. Either IPv4 or IPv6 may be used for IP. Also, communication between higher-level applications is made possible by using Web services using techniques such as HTTP and RTP described above. Also, although not shown, it is possible to connect to the Internet via a hub or router.

The controller device 200 can control multiple imaging devices 100 and can exchange information with other controller devices 200 .

The imaging system according to the present embodiment can be used for applications and services such as monitoring and entrance/exit management.

The imaging apparatus 100, which is a feature of this embodiment, includes a lens 11, an imaging sensor section 12, a signal processing section 13, a signal output control section 14, a communication control section 15, an IF section 16, an abnormality detection section 17, an illuminance monitoring section 18, and a , GPS 19 , clock 20 , memory 21 , maintenance IF section 22 , control section 23 and infrared LED 24 .

The lens 11 is an optical lens for photographing that forms an image of visible light 301 and infrared light (invisible light) 302 from an object at a predetermined focal length on the imaging sensor unit 12, and includes a plurality of lenses. be done.

The imaging sensor unit 12 is a unit that separates the visible light and infrared light imaged by the lens 11 using various filters and photoelectrically converts them, thereby outputting a plurality of pixel signals corresponding to predetermined wavelength components.

The signal processing unit 13 processes the output signal output from the imaging sensor unit 12 to perform interior processing, image processing to remove the influence of infrared light that has passed through the second wavelength band during color photography, This is a portion that performs image processing such as gamma correction, white balance, RGB matrix correction, etc. on the image signal and outputs a visible image output signal and an infrared image output signal.

The signal output control unit 14 outputs the visible image signal and the infrared image signal "that the object was photographed at the same timing" output from the signal processing unit 13, according to the instruction of the communication control unit 15 or the control unit 23, the IF unit 16 to a predetermined network-connected controller device 200 .

The communication control section 15 controls the image signal output from the signal output control section 14 via the IF section 16 and transmits and receives control signals to and from the controller device 200 via the IF section. It is also the part that executes the above-mentioned network protocol, application protocol, Web service, and the like.

The IF section 16 is a communication IF section that connects the imaging device 100 and the network 303 .

The abnormality detection unit 17 is a part that constantly or periodically monitors whether an abnormality has occurred in the hardware or software of the imaging apparatus 100 and detects an abnormality. For example, the imaging device 100 may be removed from a predetermined installation location, become unable to capture images, or become unable to perform network communication, or may be illegally accessed.

The illuminance monitoring unit 18 is a part that constantly or periodically monitors the brightness of the imaging range of the imaging device 100 using an illuminance sensor or the like. When it is detected that the illuminance is insufficient, it is notified to the control unit 23 and the infrared LED 24 is illuminated.

The GPS 19 is a part that acquires the current position of the imaging device 100 from position information received from satellites. The acquired position information can also be notified to the controller device 200 via the IF section 16 .

The clock 20 is a part that manages current time information and sets/cancels the timer. Time information is automatically adjusted using general-purpose techniques such as NTP (Network Time Protocol) and standard radio waves.

The memory 21 is a storage device (ROM (Read-Only Memory) area, FROM (Flash ROM) area) that stores programs, various setting information, and property information, and loads and temporarily stores these programs and data. , a storage device (RAM (Random Access Memory) area) for storing work data. Here, the recording device may use an external memory (USB memory or NAS (Network-Attached Storage)) or a portable medium (micro flash, SD card, magnetic tape, etc.) in addition to the built-in memory.

The maintenance IF section 22 is an IF section with which a maintenance person of the imaging device 100 communicates for updating the program or diagnosing when a failure occurs. Further, when the abnormality detection unit 17 detects an abnormality, it is possible to automatically notify the remote maintenance site of the content of the detected abnormality.

The control unit 23 is a part that controls the operation of each component described above.

On the other hand, the controller device 200 includes a user IF section 201, a display section 202, a clock 203, a memory 204, a recording/playback section 205, a communication control section 206, an IF section 207, a camera management section 208, a moving body area extraction section 209, and a face area detection section. 210 , a face feature point detection unit 211 , a face matching unit 212 , a face DB 213 and a control unit 214 .

A user IF section 201 is a section where a user operates the controller device 200 using a remote controller, touch panel, keyboard, mouse, buttons, and the like.

A display unit 202 displays an operation screen of the controller device 200, a visible image or an infrared image received via the network 303, a face authentication result, a warning screen, etc. on an external or built-in monitor.

A clock 203 is a part that manages current time information and sets/cancels a timer. Time information is automatically adjusted using general-purpose techniques such as NTP and standard radio waves.

The memory 204 includes a storage device (ROM area, FROM area) for storing programs, various setting information, and property information, a storage device (RAM area) for loading and temporarily storing these programs and data, and storing work data. area). Here, in addition to the built-in memory, the recording device uses external memory (USB memory and NAS) and portable media (micro flash, SD card, DVD, Blu-ray (registered trademark) Disc, magnetic tape, etc.) You can

The recording/reproducing unit 205 is a part that records and reproduces visible images and infrared images received via the network 303 and the IF unit 207 and metadata associated with these images in the memory 204 . The data to be recorded is encrypted/decrypted and compressed/decompressed as required.

A communication control unit 206 is a part that transmits and receives control signals between the imaging apparatus 100 via the network 303 and the IF unit 207 . It is also the part that executes the above-mentioned network protocol, application protocol, Web service, and the like.

The IF section 207 is a communication IF section that connects the controller device 200 and the network 303 .

The camera management unit 208 is a part that manages one or more imaging devices 100 managed by the control device 200 via the network 303 . It is a part that creates, holds, updates, and deletes information (for example, IP address, installation location, manufacturer name, model name, introduction time/operation time, function specifications, maintenance contact information, etc.) regarding the imaging device 100 to be managed. .

A moving body region extraction unit 209 extracts a moving body such as a person, an animal, or an object existing in a visible image or an infrared image received via the IF unit 207 or recorded by the recording/reproducing unit 205, and acquires position information thereof. is. As a method for extracting a moving object in an image, a difference image (for example, the first and second difference images, the second and third difference images, ) and comparing them to extract a moving object, or extracting a moving object by using a background subtraction method while generating a background image from a captured image.

The face area detection unit 210 detects a human face directly from the visible image or infrared image received via the IF unit 207 or recorded by the recording/reproducing unit 205, or from the moving object area extracted by the moving object area extraction unit 209. Detect areas that exist. Detection methods include Viola & Johns' high-speed face detection algorithm technology using an integral image and a cascade type classifier.

The face feature point detection unit 211 is a part that detects feature points such as the eyes, nose, and corners of the mouth in the face area detected by the face area detection unit 210 . This makes it possible to correct the image position so that facial features can be extracted accurately.

The face matching unit 212 is a part that selects the optimum feature for identifying an individual from the feature points detected by the face feature point detection unit 211 and performs matching using the face DB 213 . Here, as a feature for distinguishing a face, a method using the entire grayscale information in the face region (for example, an eigenmethod that applies principal component analysis), or a method using the interval and direction component of local grayscale changes in the face region. can be used as a feature amount (for example, Elastic Bunch Graph Matching), a method combining these two methods, and the like. Also, as a matching method, a nearest neighbor method, a linear discriminant analysis, or the like can be applied.

The face DB 213 is a part that stores data in which face images are registered in advance for matching by the face matching unit 212 in an internal or external storage medium. It is also possible to register an image that is artificially generated from these registered images with changes in lighting, face orientation, and the like. For example, in an entry/exit management system, facial images of users who are permitted to enter a specific area or users who are employees are registered. It is also possible to additionally register an image in which the identity of the individual is confirmed as a result of facial recognition performed at a specific location. Here, the face DB 213 may be an external DB accessible via the network 303 instead of the controller device 200 . For example, surveillance camera systems at airports and the like use face databases of suspects, terrorists, and the like provided by police and legal institutions. Also, the DB may be shared among a plurality of controller devices.

The control unit 214 is a part that controls the operation of each component described above. As a result of matching by the face matching unit 212, if the user is not a pre-registered user (such as a suspicious person) or if the match is with a suspect, a report based on a predetermined format is automatically played back, and an administrator or police report is generated. It is possible to notify the contact and send the report.

FIG. 2 shows a configuration example of the imaging sensor unit 12 of the camera unit 100. As shown in FIG.

The imaging sensor unit 12 is composed of a sensor main body 2, a color filter 3, a cover glass 4, and a DBPF5.

The sensor main body 2 is a CCD (Charge Coupled Device) image sensor, and is a portion in which a photodiode is arranged as a light receiving element in each pixel. A CMOS (Complementary Metal-Oxide Semiconductor) image sensor may be used instead of the CCD image sensor.

The color filter 3 is provided in the sensor main body 2, and each pixel of the sensor main body 2 has red (R), green (G), blue (B), and infrared (IR) regions. This is the part arranged in the array. FIG. 4 shows variations of color filters used in this embodiment.

A cover glass 4 covers and protects the sensor main body 2 and the color filter 3 .

A DBPF 5 is an optical filter formed on the cover glass 4 . The DBPF 5 has transmission characteristics in the visible light band, has cutoff characteristics in a first wavelength band adjacent to the long wavelength side of the visible light band, and has a second wavelength that is a part of the first wavelength band. It is an optical filter having transmission characteristics in the band. Incidentally, the arrangement position of the DBPF 5 is not limited, and it may be provided on the lens 11, for example.

FIG. 3 shows transmittance spectra of the R, G, B, and IR filters of the color filter 3, where the vertical axis represents the transmittance and the horizontal axis represents the wavelength. The wavelength range in the graph includes part of the visible light band and the near-infrared band, for example, the wavelength range of 300 nm to 1100 nm.

As shown by R (double line) in the graph, the R filter portion has a substantially maximum transmittance at a wavelength of 600 nm, and the transmittance on the long wavelength side is maintained at a substantially maximum state even beyond 1000 nm. state.

The G filter portion has a maximum transmittance peak at a wavelength of about 540 nm, and a transmittance peak at about 620 nm on the longer wavelength side, as indicated by G (dashed line with wide intervals) in the graph. There are parts that are extremely small. In addition, in the G filter portion, the transmittance tends to increase on the long wavelength side from the portion where the transmittance is minimal, and the transmittance becomes maximum at about 850 nm. On the longer wavelength side, the transmittance is maximized even if the wavelength exceeds 1000 nm.

The filter part B has a maximum transmittance peak at a wavelength of about 460 nm, as shown by B (a dashed line with a narrow interval) in the graph, and has a transmittance peak at a wavelength of about 630 nm on the longer wavelength side. There is a part where the rate is extremely small. In addition, there is an upward trend on the longer wavelength side, and the transmittance reaches a maximum at about 860 nm, and on the longer wavelength side, the transmittance reaches a maximum even when it exceeds 1000 nm.

The IR filter section blocks short-wavelength light from about 780 nm, blocks long-wavelength light from about 1020 nm, and has the maximum transmittance in the range of about 820 nm to 920 nm.

The transmittance spectrum of each of the R, G, B, and IR filter portions is not limited to that shown in FIG. spectrum. Note that 1 on the horizontal axis indicating the transmittance does not mean that 100% of light is transmitted, but indicates the maximum transmittance in the color filter 3, for example.

Here, as shown by DBPF (solid line) in the graph, the DBPF 5 used in the present embodiment has a visible light band indicated by DBPF(VR) and a position slightly distant from the visible light band on the long wavelength side. The transmittance is high in two bands of the infrared band (second wavelength band) indicated by DBPF(IR). DBPF (VR) as a band with high transmittance in the visible light band is, for example, a wavelength band of approximately 370 nm to 700 nm. DBPF (IR) as a second wavelength band having high transmittance in the infrared region is, for example, a band of approximately 830 nm to 970 nm.

In this embodiment, the relationship between the transmittance spectrum of each filter portion of the color filter 3 and the transmittance spectrum of the DBPF 5 is defined as follows. That is, in the DBPF (IR), which is the second wavelength band for transmitting infrared light in the transmittance spectrum of the DBPF 5, the R filter portion, the G filter portion, and the B filter portion all have substantially maximum transmittance. 2 in which the transmittance is substantially the same in each filter portion and within the wavelength band B in which light is transmitted with the maximum transmittance of the IR filter portion. It's becoming

Here, the wavelength band A in which the R, G, and B filter portions have the same transmittance is defined as a portion where the transmittance difference between the filter portions is 10% or less. On the short wavelength side of this wavelength band A, the transmittances of the G and B filter portions are lower than that of the R filter portion, which has substantially the maximum transmittance. In the DBPF 5, the portions with different transmittances of the R, G, and B filter portions are the DBPF (VR), which is a portion with a high transmittance in the visible light band, and the second wavelength band in the infrared light band. It corresponds to a portion where the transmittance of the DBPF 5 between the DBPF 5 and the DBPF (IR), which is a portion with a high transmittance, is minimal. That is, in the infrared region, transmission of light is cut in portions where the difference in transmittance between the R, G, and B filter portions is large, and the transmittance of each filter portion becomes maximum on the longer wavelength side. It transmits light in the wavelength band A in which the ratio becomes the same.

From the above, in the present embodiment, the DBPF 5 used in place of the infrared light cut filter has a region that transmits light not only in the visible light band but also in the second wavelength band on the infrared side. Therefore, color imaging with visible light is affected by light that has passed through the second wavelength band. However, as described above, the second wavelength band does not transmit the light of the portions where the transmittance is different in each of the R, G, and B filter portions, and the transmittance of each filter portion becomes the maximum and becomes the same transmittance. Only light in the wavelength band is transmitted.

In addition, in the second wavelength band of the DBPF 5, the IR filter section transmits light in a portion where the transmittance is maximum. Therefore, assuming that four pixels that are irradiated with the same light and are extremely close to each other are provided with R, G, B, and IR filter units, in the second wavelength band, the R filter units, Light passes through the G filter portion, the B filter portion, and the IR filter portion in the same manner. It will reach That is, the amount of light passing through the second wavelength band on the infrared side of the light passing through each of the R, G, and B filters is the same as the amount of light passing through the IR filter section. When assuming as described above, the pixel output signal assumed as described above from the sensor main body 2 that basically received the light that passed through each of the R, G, and B filters and the IR filter The difference between the output signal of the pixel assumed as described above from the sensor body 2 that received the light is the R, G, and B that cut the infrared-side light that has passed through the respective R, G, and B filter units. is the output signal of the visible light portion of

In fact, in the color filter 3, each pixel of the sensor main body 2 is provided with one of R, G, B, and IR filter portions, and each color of light irradiated to each pixel is There is a high possibility that the amount of light will be different. Therefore, for example, in each pixel, the brightness of each color of each pixel is obtained using a known interpolation method (interpolation method), and the brightness of each color of each pixel that has been interpolated and the brightness of R, G, and B that have been similarly interpolated It is possible to use the difference from the IR luminance as the luminance of R, G, and B, respectively. The image processing method for removing the infrared light component from the luminance of each color of R, G, and B is not limited to this. Any method may be used as long as the method can cut off the influence of light. In any method, since the DBPF 5 cuts a portion where the transmittance of the R, G, and B filter portions differs by more than 10% in the infrared region, that is, a portion where the transmittance differs from a predetermined ratio, in each pixel , processing to remove the influence of infrared light is facilitated.

As described above, by using the imaging sensor unit 12, the imaging apparatus 100 capable of both color imaging and infrared imaging can be realized. In general, it is conceivable that normal photographing is performed in color, and infrared photographing is performed using infrared light, which is difficult for humans to recognize, without using visible light illumination at night. For example, in various surveillance cameras and the like, it is conceivable to perform night photography with infrared light using infrared light illumination when night photography is performed in places where night lighting is not required or where night lighting is not desirable. It can also be used for daytime photography and nighttime photography for observing wild animals.

When infrared light photography is used for nighttime photography, infrared light illumination is required because the amount of light is insufficient at night, even with infrared light, as with visible light.

The transmittance spectra (A) and (B) of DBPF5 shown in FIG. It is determined in consideration of the emission spectrum of the LED.

FIG. 21 shows the transmittance spectra R, G, B, and IR of the respective color filters similar to FIG. 2, the transmittance spectrum DBPF of the DBPF 5, and the emission spectrum IR-light of the LED lighting.

The second wavelength band indicated by DBPF (IR), which is the portion of the DBPF shown in FIG. 21A that transmits infrared light, is similar to the DBPF shown in FIG. All of the B filter portions have substantially maximum transmittance, and each filter portion has substantially the same transmittance. It is included in the wavelength band B through which light is transmitted.

In addition, substantially the entire wavelength band that is the peak of the emission spectrum of infrared light illumination included in both the wavelength band A and the wavelength band B is included in the wavelength band of DBPF (IR). there is In addition, when infrared light photography is performed under infrared light illumination instead of natural light at night, the second wavelength band indicated by DBPF (IR) need not be wider than the peak width of the optical spectrum of infrared light illumination. When the spectrum of the infrared light illumination is included in both the wavelength band A and the wavelength band B described above, the peak width of the peak whose peak is, for example, about 860 in the emission spectrum of the infrared light illumination. A transmittance peak portion of the DBPF 5 indicated by DBPF(IR) may be provided as the second wavelength band.

That is, in FIG. 21A, the peak in the emission spectrum of the infrared light illumination indicated by IR-light is on the short wavelength side of the wavelength bands A and B described above, and the peak of the DBPF indicated by DBPF(IR) is on the short wavelength side. 2 overlaps with the peak of the emission spectrum in IR-light on the short wavelength side of the wavelength band A and the wavelength band B. FIG.

Similarly to (A), the graph shown in FIG. 21(B) also shows the graph shown in FIG. The second wavelength band indicated by DBPF(IR) is matched with the peak of the emission spectrum indicated by IR-light of the above infrared light illumination.

In FIG. 21(B), infrared light illumination having a longer peak wavelength in the emission spectrum than that in (A) is used, and this peak is included in the above-described wavelength band A and wavelength band B. , on the longer wavelength side of the wavelength band A and the wavelength band B, respectively. Correspondingly, the second wavelength band indicated by DBPF(IR) of the DBPF 5 is provided so as to overlap the peak indicated by IR-light of the infrared illumination in the wavelength bands A and B described above.

The second wavelength band of the DBPF 5 may be that shown in either FIG. 2 or FIG. 21, as long as the second wavelength band is included in both the wavelength band A and the wavelength band B described above. . In addition, when the wavelength band that is the peak of the emission spectrum of the infrared light illumination used for nighttime infrared photography is determined, the wavelength band should be included in both the above-described wavelength band A and wavelength band B. In addition, it is preferable to match the second wavelength band of the DBPF 5 with the peak of the emission spectrum of the infrared light illumination.

In such an image sensor, the second wavelength band in which light is transmitted on the infrared side of the DBPF 5 is on the infrared side of each of the R, G, B, and IR filter portions, and the transmittance of each filter portion is maximized. , is included in the wavelength band A when the transmittance of each filter unit is the same, and is included in the wavelength band B where the transmittance of the IR filter unit is maximum. In other words, on the longer wavelength side than the visible light band, the transmittance of each of the R, G, and B filters is maximized only in the R filter portion, and the transmittance of the G and B filter portions is not maximized. Therefore, the DBPF 5 cuts the light in the portions where the transmittances of the respective R, G, and B filter portions are not the same and are different.

That is, since the R, G, B, and IR filter portions transmit the light of the second wavelength band in the infrared region, the transmittance in the infrared region in each filter portion is the same. . If the light of the second wavelength band is irradiated with the same amount of light, the amount of light transmitted through each of the R, G, B, and IR filter portions will be the same. As a result, as described above, the colors based on the output signals from the pixels corresponding to the respective R, G, and B filter sections are corrected, and the influence of infrared light passing through the second wavelength band on the colors during color photography is corrected. can be easily obtained.

Further, by making the second wavelength band correspond to the peak of the emission spectrum of the infrared light illumination included in the wavelength band A and the wavelength band B described above, the light of the infrared light illumination can be efficiently used, By narrowing the width of the second wavelength band, it is possible to reduce the influence of infrared light passing through the second wavelength band during color photography.

FIG. 5 is a block diagram showing signal processing in the signal processing section 12. As shown in FIG.

An outline of processing for an output signal from the imaging sensor unit 12 equipped with the color filter shown in FIG. 4 will be described.

Output signals of pixels of R, G, B, and IR are sent to respective internal processing blocks 21r, 21g, 21b, and 21ir. In each of the internal processing blocks 21r, 21g, 21b, and 21ir, interpolation processing (interpolation processing) using a well-known method is performed so that all pixels in the image data of each frame of the color filter 3 are red. Image data 20r represented in R, image data 20g represented in all pixels in green G, image data 20b represented in all pixels in blue B, and image data 20b represented in all pixels in infrared IR. The R, G, B, and IR signals are converted to form image data 20ir.

Next, in the infrared light elimination signal generation blocks 22r, 22g, 22b, and 22ir, signals of each color of R, G, and B are generated in order to eliminate the influence of the infrared light received from the second wavelength band. A signal to be subtracted from is generated from the signal of IR. The signals generated for each of R, G, and B by the infrared light removal signal generation blocks 22r, 22g, and 22b are subtracted from the signals of each color of R, G, and B. In this case, in the same pixel as described above, the IR signal is basically removed from each of the R, G, and B signals, so processing is facilitated. Actually, since the sensitivity differs for each pixel of each color due to the characteristics of the filter section of each pixel, a signal to be subtracted from each of the R, G, and B signals is created from the IR signal for each of the R, G, and B images. do.

Next, the R, G, and B signals are processed in an image processing block 23 using well-known RGB matrix processing, which transforms the R, G, and B signals using a determinant to correct the colors, and white in the image. Well-known white balance processing for making the output values of the respective R, G, and B signals the same, and well-known gamma correction, which is correction for image output to a display, etc. Next, in the luminance matrix block 24, the R, G, and B signals are multiplied by a coefficient to generate a luminance Y signal. Further, by dividing the luminance Y signal from the blue B signal and the red R signal, the RY and BY color difference signals are calculated, and the Y, RY, and BY signals are output. do.

Also, the IR signal is basically output as a black-and-white gradation image.

FIG. 6 shows a communication flow for exchanging visible images, infrared images, and control commands between the imaging device 100 and the controller device 200 shown in FIG. Here, the communication flow may use a unique communication protocol, but may also use, for example, a protocol established by ONVIF (Open Network Video Interface Forum) as standard communication for surveillance cameras.

First, when the imaging device 100 is installed at a predetermined location and connected to the network 303 and then turned on, the imaging device 100 is activated, and the control unit 23 of the imaging device 100 executes initialization processing. For example, loading of a program stored in the memory 21, acquisition of the current location of the GPS 19, etc., mainly start-up of hardware, initial parameter setting processing of software, and the like. It should be noted that PoE (Power Over Ethernet) may be used, and the PoE compliant hub may be used to activate at the timing of connecting to the network 303 (step 601).

After completing the necessary initial setting processing, the control unit 23 of the imaging device 100 sets IP addresses used by the communication control unit 15 and the IF unit 16 . The IP address can be set by a general-purpose method such as a method in which a PC or pad terminal is directly connected to the maintenance IF unit 22 and a static IP address is set, or a method in which an IP address is automatically set using DHCP (Dynamic Host Configuration Protocol). network protocol (step 602).

After setting the IP address, the control unit 23 of the imaging device 100 instructs the communication control unit 15 to notify the controller device 200 of its own existence. Protocols such as UPnP (Universal Plug and Play) and WS-Discovery (Web Services Dynamic Discovery) may be used as methods for automatically discovering devices existing on the network. In addition, when notifying, the manufacturer name, model name, installation location, date and time, etc. may be included (step 603). At this time, the installation location may be preset information or information acquired from the GPS 19 . Also, information obtained by determining whether the location is outdoors or indoors using the GPS 19 or the illuminance monitoring unit 18 may be included.

The control unit 214 of the controller device 200 that has received the notification can recognize the existence of the imaging device 100 by acquiring the IP address of the imaging device 100 . The control unit 214 notifies the administrator via the display unit 202 that the new imaging device 100 has been connected, and waits for an instruction from the administrator as to whether or not to manage the imaging device 100 by itself. If there is an instruction from the administrator via the user IF unit 201, or if the number of imaging devices 100 currently managed by the controller device 200 is checked, and if the maximum number has not been reached, the The communication control unit 206 is instructed to transmit a request to acquire the installed function information to the imaging device 100 (step 605).

The control unit 23 of the imaging device 100 that has received the function information acquisition request acquires the function information stored in the memory 21 and instructs the communication control unit 15 to transmit the function information to the controller device 200 . Functional information includes, for example, device management information (support availability and parameter values related to networks, systems, and security), imaging device performance information (backlight correction, brightness, contrast, white balance, focus adjustment, parameters related to image quality such as wide dynamic range, etc.). values, media profile parameters such as resolution, frame rate, and codec type), PTZ (pan/tilt/zoom) function information (coordinate system definition, movable parameters, preset positions, etc.), analysis function information (supported analysis function, authentication type, analysis result format, etc.) (step 606).

Here, FIG. 7 shows one configuration of information regarding the imaging device 100 of the present embodiment. The imaging apparatus 100 of the present embodiment has “visible image” and “infrared image” as output image types 701 , and “output visible image only” and “infrared image” as output modes 702 to be transmitted to the controller device 200 . There are four types of output: "only output", "automatic switching between both images depending on the illuminance and time", and "simultaneous output of both visible and infrared images". In addition, as the access destination information 703 of the visible image, URI/URL information accessed by the controller device 200 to acquire information on the visible image from the imaging device 100 and the actual visible image. has URI/URL information that the controller device 200 accesses to acquire information about infrared images and actual infrared images from the imaging device 100 . The visible image information 705 includes the visible image codec, transfer rate, resolution, etc. Similarly, the infrared image information 706 includes the infrared image codec, transfer rate, resolution, etc. that can be output. This configuration is an example, and may include other information.

The control unit 214 of the controller device 200 that has received the function information of the imaging device 100 notifies or automatically confirms the content of the function information to the administrator via the display unit 202, and determines that the controller device 200 will manage the function information. If so, it is added to the camera management unit 208 as a management target. A camera management unit 208 stores and manages all or part of the function information in the memory 204 . Also, the control unit 214 confirms the analysis function and authentication function supported by the controller device 200 itself, and determines whether or not to use the image of the imaging device 100 . Alternatively/in addition, the analysis function information supported by the imaging device 100 may be checked to determine the authentication method and analysis method to be executed when using the imaging device 100 (step 607).

When the control unit 214 of the controller device 200 determines to use the imaging device 100, the control unit 214 performs communication in order to set parameters included in the function information acquired in step 606 that need to be changed or set. It instructs the control unit 206 to transmit a device setting request to the imaging apparatus 100 . For example, in this embodiment, the output mode 702 is set to "output both visible image and infrared image simultaneously" (step 608). Here, for example, the output mode 702 may be determined based on the installation location of the imaging device 100 .

The control unit 23 of the imaging device 100 that has received the device setting request confirms whether or not the received setting can be executed, and returns the execution result to the controller device 200 (step 609).

Next, the control unit 214 of the controller device 200 instructs the communication control unit 206 to transmit an access destination information acquisition request for acquiring the protocol and parameters necessary for actually acquiring the visible image and the infrared image. (Step 610).

Upon receiving the access destination information acquisition request, the control unit 23 of the imaging device 100 accesses the media including the access destination information 703 of the visible image and the access destination information 704 of the infrared image (for example, media type, port number, transfer protocol, payload number, etc.) to the communication control unit 15 (step 611).

After receiving the access destination information, the control unit 214 of the controller device 200 subsequently transmits an acquisition request for session information (DESCRIBE) necessary for image reception to the imaging device 100 (step 612).

Upon receiving the session information acquisition request, the control unit 23 of the imaging device 100 instructs the communication control unit 15 to generate session information described using SDP (Session Description Protocol) and transmit it to the controller device 200 ( step 613).

The control unit 214 of the controller device 200 that has received the session information instructs the communication control unit 206 to establish an RTSP session with the imaging device 100 . Here, separate RTSP sessions are typically established for visible image transfer and infrared image transfer (step 614).

After establishing the RTSP session, the controller device 200 prepares for reception of these images and face authentication (step 615), the imaging device 100 prepares for transmission of visible images and infrared images (step 616), Send the result (step 617).

After confirming that all preparations have been completed, the control unit 214 of the controller device 200 instructs the communication control unit 206 to transmit a streaming start request (PLAY) to the imaging device 100 (step 618).

The control unit 23 of the imaging device 100 that has received the streaming start request instructs the signal output control unit 14 to output the image as requested by the controller device 200 in step 608, and the signal output control unit 14 outputs The communication control unit 15 is instructed to transmit the image to the imaging device 100 using RTP on the session established in steps 612/613 (step 620).

Also, the control unit 214 of the controller device 200 starts receiving the image (step 621).

Thereafter, RTP transfer of the visible image and the infrared image captured by the imaging device 100 is executed (steps 621 and 622). Here, in order to reduce the processing load on the controller device side, the communication control unit 15 of the imaging device 100 may use the marker bit of the RTP header so that the breaks between frames can be recognized.

Further, the communication control unit 15 of the imaging device 100 transmits an RTCP transmission report to the controller device 200 each time a predetermined number of frames are transferred. The same time stamp, frame number, packet count, etc. are stored in the report to indicate that the visible and infrared images were taken at the same time (step 623).

The control unit 214 of the controller device 200 that receives the visible image and the infrared image from the imaging device 100 stores these images in the memory 204 via the recording/reproducing unit 205, and extracts the moving object region extraction unit 209 and the face region detection unit. 210, a face feature point detection unit 211, and a face matching unit 212 are used to perform face authentication. Then, interruption or stop of streaming is controlled as necessary (step 624).

The above is the basic communication flow between the controller device 200 and the imaging device 100 .

Here, although RTP communication is used in the above communication flow, HTTP communication or another original communication method may be used. Also, instead of transferring the visible image and the infrared image in separate streams, they may be superimposed and transferred in the same stream (for example, common header (including time stamp and sequence number) + 1st visible image + first infrared image + etc.). In addition, if both images are transferred at the same time, the usage rate of the communication band increases. Therefore, the infrared image may be transferred every one frame, and the visible image may be transferred every 30 frames. Also in this case, the same time stamp and frame number are used for the infrared image and the visible image for frames captured at the same timing.

Here, in step 623, the imaging device 100 sends a transmission report to the controller device 200. Likewise, the controller device 200 sends a reception report containing information about packet loss and transfer delay to the imaging device 100. can be sent.

Also, in step 623, in order to indicate that the visible image and the infrared image were captured at the same time, it was described that the transmission report in which the same time stamp and frame number are set is transmitted. There is also a method of setting the sequence number to the same value, and a method of setting the same time stamp and frame number in the extension header of the RTP header.

The control unit 214 of the controller device 200 instructs the recording/reproducing unit 205 to store the received visible image and infrared image in the memory 204, and extracts the moving object region extracting unit 209, the face region detecting unit 210, and the facial feature point detecting unit. 211, a face matching unit 212 is used to detect a person included in the image, and perform face authentication such as whether or not the person is a suspicious person. In this case, it is possible to acquire a visible image and an infrared image of the same object at the same timing, and the same time stamp and sequence number are added, making it easy to synchronize both images. If you want to perform face authentication using only one of the images (for example, infrared image only) and want to compare and check both images (such as background and color). There is a method of using the other image with the same sequence number when there is a part of the other image that you want to recognize additional information, or you want to recognize the face in the other image.

In the present embodiment, "simultaneous output of both visible and infrared images" was set as the output mode 702 in step 608, but during the daytime, "simultaneous output of both visible and infrared images" may be set, and the setting may be changed according to the time and surrounding environment, such as setting "only infrared image" at night. Alternatively, if one of the images is received and face recognition is performed at the same time, and as a result of matching by the face matching unit 212, if it is desired to acquire further information on a person who is suspected to be a suspicious person, both images are received on the way. You may make it switch automatically like this.

If the controller 214 of the controller device 200 determines that there is a suspicious person in the image as a result of matching by the face matching unit 212 or that there is a candidate for a suspicious person, the control unit 214 notifies the administrator via the display unit 202, or It is also possible to notify and share information with other controller devices 200 via the IF unit 207 and track the suspicious person among a plurality of imaging devices 100 .

(Embodiment 2)
Next, the configuration of an imaging system according to Embodiment 2 of the present invention will be described.

FIG. 8 shows a configuration example of an imaging system according to Embodiment 2 of the present invention. Note that the imaging device 100 of the first embodiment and the imaging devices 800 and 810 of the present embodiment can be installed together on the same imaging system, and the controller device 200 manages these imaging devices. It is possible to

An imaging device 800 of the present embodiment is configured by mounting a moving body region extraction section 801 having the same function as the moving body region extraction section 209 of the controller device 200 on the imaging device 100 of the first embodiment. Other components have the same configuration as the imaging device 100 .

The control unit 23 of the imaging device 800 inputs only the infrared image out of the visible image and the infrared image output by the signal processing unit 13 to the moving object area extraction unit 801 . Reasons for using only an infrared image include the ability to detect objects that cannot be detected with a visible image, and the fact that the contrast between a person and the background is greater than in a visible image, making it effective for human detection.

The moving object area extraction unit 801 extracts moving object areas in the image using the input infrared image, and outputs the number and position information thereof. These results are output to the control section 25 or the signal output control section 14 . It may be stored in the memory 22 .

As described above, the imaging device 800 always monitors the moving object region in the image using the infrared image among the visible image and the infrared image taken at the same timing, and visualizes the information on the moving object region extracted with high accuracy. It can be provided to the controller device 200 along with the image or the infrared image. Since the controller device 200 can acquire information about the moving object area together with the image, the image processing load can be reduced.

Here, in order to reduce the usage of the communication band on the network, the control unit 23 of the imaging device 800 outputs the signal from the signal output control unit 14 via the IF unit 16 only when the moving object region extracting unit 801 extracts the moving object region. An image may be output, and if the moving object region cannot be extracted, the image may not be output from the signal output control unit 14, or the frame rate of the image output from the signal output control unit 14 may be reduced.

In addition, the control unit 23 of the imaging device 800 combines the moving object region extracted by the moving object region extracting unit 801 with the visible image and/or the infrared image output by the signal processing unit 13 so that the moving object region is surrounded by a rectangle on the image. Alternatively, the signal output control unit 14 may be instructed to generate/process the image.

Similarly, imaging device 810 of the present embodiment has a configuration in which face region detection section 802 having the same function as face region detection section 210 of controller device 200 is mounted on imaging device 800 . Other components have the same configuration as the imaging device 100 .

The control unit 23 of the imaging device 810 inputs only the infrared image out of the visible image and the infrared image output by the signal processing unit 13 to the moving object area extraction unit 801 . The moving object area extraction unit 801 extracts moving object areas in the image using the input infrared image, and outputs the number and position information to the control unit 23 or the signal output control unit 14, and at the same time, the face area detection unit Enter 802 . The face area detection section 802 detects an area in which a human face exists from the input moving body area, and outputs it to the control section 23 or the signal output control section 14 .

As described above, the imaging device 810 can constantly monitor a moving object region in an image using an infrared image among visible images and infrared images taken at the same timing, and extract the moving object region with high accuracy. An area in which a human face is present can be detected from the moving object area, and information on the moving object area and information on the face area can be provided to the controller device 200 together with a visible image or an infrared image. Since the controller device 200 can acquire these pieces of information together with the image, the image processing load can be reduced.

Here, in order to reduce the usage of the communication band on the network, the control unit 23 of the imaging device 810 controls the IF unit 16 from the signal output control unit 14 only when the face area detection unit 802 detects a person's face area. If the human face region cannot be detected even after extracting the moving object region, the image is not output from the signal output control unit 14, or the frame rate of the image output from the signal output control unit 14 is reduced. You can do it. Similarly, in order to detect only an object, an image may be output from the signal output control section 14 via the IF section 16 only when a moving body region, in which a human face region cannot be detected, is detected.

Further, the control unit 23 of the imaging device 810 detects the face area extracted by the face area detection unit 802 (and the moving object area extracted by the moving object area extraction unit 801) and the visible image and/or infrared image output by the signal processing unit 13. may be combined to instruct the signal output control unit 14 to generate/process an image in which the face area is surrounded by a rectangle.

The communication flow between the imaging devices 800 and 810 and the controller device 200 is almost the same as the content described in FIG. 6 of Embodiment 1, but only the different points are described below.

First, in step 606 of FIG. 6, the function information provided by the imaging devices 800 and 810 to the controller device 200 includes the information shown in FIG. 7 as an example, and the information shown in FIG. offer. In other words, in the function information, the imaging device 800 has contents indicating that it has a "moving body region extraction function", and the imaging device 801 has a "moving body region extraction function" and a "face region detection function". Include content that indicates

In the present embodiment, these pieces of information are used as analysis metadata, and as function information (analysis function information), analysis metadata types 901 . Analysis metadata output mode 902, moving body region metadata access destination information 903, face region metadata access destination information 904, moving body region/face region metadata access destination information 905, moving body region metadata information 906, face Contains region metadata information 907 .

In step 607, the control unit 214 of the controller device 200 that has received the function information confirms the analysis function and authentication function supported by the controller device 200 itself, and determines whether or not to use the analysis metadata output by the imaging device 800. to judge whether For example, only the analysis metadata “moving object region position information” is used for both the imaging devices 800 and 810, or the analysis metadata of the imaging device 800 is not used, and the analysis metadata of the imaging device 810 “face information” is used. It is possible to make a selection such as using only "location information of the area".

FIG. 10 shows a communication flow for transmitting visible images, infrared images, and analysis metadata between imaging devices 800 and 810 and controller device 200 . In this description, it is assumed that the imaging device 800 transmits analysis metadata of both “moving body region position information” and the imaging device 810 transmits both “moving body region position information” and “face region position information”. It is also assumed that imaging devices 800 and 810 establish a session for transferring analysis metadata in addition to visible and infrared images in step 614 of FIG.

The imaging devices 800 and 810 start frame transfer of the visible image and the infrared image (steps 1001 and 1002), and every time a predetermined number of frames are transferred (steps 1003 and 1004), the moving object region extraction unit 801 and the face region The analysis metadata extracted by the detection unit 802 is transmitted (step 1005). Here, the analysis metadata may be sent at the timing when the moving object area or the face area is detected.

The control unit 214 of the controller device 200 that has received the analysis metadata 1200 checks whether or not the analysis metadata 1200 contains moving body area information or face area information (step 1006). If the moving object area information is not included, the moving object area extracting section 209 is used to execute the moving object area extraction processing (step 1007).

On the other hand, if the moving body area information or face area information is included, it is checked whether or not the face area information is included (step 1008). If no face region information is included (that is, only moving body region information is included), face region detection processing is performed using the received moving body region information and the own face region detection unit 210 . (step 1008).

On the other hand, if the face region information is included, the face feature points are extracted using the received face region information and the face feature point detection unit 211 (step 1010). (step 10100).

FIG. 11 shows images handled by imaging devices 800 and 810 . An image 1100 is an example of a visible image captured by imaging devices 800 and 810 . An image 1101 is obtained by removing the background from the image 1100 and extracting only the moving object region. In this image example, three regions (A), (B), and (C) (portions surrounded by dotted-line rectangles) can be extracted. An image 1102 is obtained by extracting a face area from the image 1101 . In this image example, two regions (a) and (b) (parts surrounded by solid-line squares) can be extracted.

FIG. 12 shows a configuration example of analysis metadata 1200 sent by the imaging devices 800 and 810 in step 1005 .

Analysis metadata 1200 is roughly composed of a communication header 1201 and a payload 1210 . The communication header 1201 is similar to, for example, an RTP header or HTTP header.

The payload 1210 stores analysis metadata. For example, the frame number 1211 of the infrared image used to extract the moving object area or the face area, the frame number 1212 of the visible image, the maximum number of moving object areas 1213 that can be extracted by the imaging devices 800 and 810, The number of extracted moving object regions 1214 (here, n pieces), the coordinate information 1 to n (1215 to 1216) of the extracted moving object regions, the maximum number of face regions that can be extracted by the imaging device 810 1217, the actual face region detection unit It consists of the number 1218 of extracted face areas (here, m≦n) extracted in 802 and coordinate information 1 to m (1219 to 1220) of the extracted moving body areas.

As described above, the imaging devices 800 and 810 according to the present embodiment can output the information on the moving object region and the information on the human region accurately extracted using the infrared image in addition to the visible image and the infrared image as required image output. can be provided to the controller device 200 at the same time.

On the other hand, the controller device 200 can omit the conventional procedure by immediately using the received information of the moving body region and the information of the human region, so that the face authentication execution time can be shortened. This is effective in reducing the processing load of the controller device 200 when managing many imaging devices with one controller device 200 .

Here, in the present embodiment, an example in which the imaging devices 800 and 810 transmit at least one of the visible image and the infrared image and the analysis parameters to the controller device 200 has been described. In order to reduce the amount of data, the analysis parameters and the image of only the portion indicated by the analysis parameters (moving body area, face area) may be sent.

Further, when the moving body region extracting unit 801 first detects a moving body region in the image, the control unit 23 of the imaging devices 800 and 810 retains the frame number of the corresponding image, The moving object area may be tracked until it no longer exists, and the frame number may be added as attached information to the coordinate information in the analysis metadata 1200 shown in FIG. Thereby, the controller device 200 can easily grasp the frame number in which the moving object region is included and calculate the time by referring to the analysis metadata 1200 .

(Embodiment 3)
Next, the configuration of an imaging system according to Embodiment 3 of the present invention will be described.

The imaging apparatuses of the first and second embodiments described above use one set of lens 11, imaging sensor unit 12, and signal processing unit 13 to capture a visible image and an infrared image. The imaging apparatus of the present embodiment has a configuration in which two sets of lenses 11, an imaging sensor unit 12, and a signal processing unit 13 are arranged on the left and right, and a stereo image ( distance image) can be taken.

FIG. 13 shows a configuration example of an imaging system according to this embodiment. This imaging system comprises one or more imaging devices 1300 and a controller device 1310 .

As described above, the imaging apparatus 1300 includes two sets of lenses 11, an imaging sensor unit 12, and a signal processing unit 13, and newly includes a correction parameter calculation unit 1301 and a distance calculation unit 1302. FIG. The two lenses 11(a) and 11(b) are arranged left and right so that their optical axes are parallel to each other. Other components are basically the same as those of the imaging devices 100, 800, and 810 of the first and second embodiments.

The correction parameter calculator 1301 calculates clip levels and signal level correction values (for example, By setting parameters such as visible image signals, infrared image signals, correction values that are added, subtracted, multiplied, or divided to signals such as infrared signals and each color signal, two visible image signals (two infrared images signals) should be similar. The amount of correction in the image signal correction processing unit 203 is adjusted according to the level of the image signal by checking the outputs from the two signal processing units 13(a) and 13(b) and setting them accordingly. The process of matching the levels of the left and right image signals can be performed on both the infrared image signal and the visible image signal.

That is, the correction parameter calculation unit 1301 determines the amount of correction based on the signal levels of the image signals output from the two signal processing units 13(a) and 13(b). ) so that the signal levels of the image signals output from are approximate to each other. As a result, it is possible to recognize different parts of the subject as the same parts (corresponding points) due to, for example, different luminance levels in the two image data, and to suppress errors in the measured distance. be able to.

The distance calculation unit 1302 calculates the distance to the object using two visible image signals or infrared image signals respectively input from the two signal processing units 13(a) and 13(b). At this time, the same subject (corresponding point) is determined from the two images, and the distance is obtained in the same manner as in the conventional art by detecting the parallax as the difference in the positions of these same subjects on the images. That is, the corresponding points for measuring the difference are determined by image recognition, and the distance is calculated based on the parallax, which is the difference in the positions of the corresponding points in the image. Based on the distance information corresponding to each pixel, a stereo image (distance image) is generated and output to the signal output control section 14 .

The signal output control unit 14 can provide the stereo image (distance image) generated by the distance calculation unit 1302 to the controller device 1310 in addition to the two visible images captured on the left and right sides and the infrared image.

As described above, the imaging apparatus 1300 can acquire a visible light image and an infrared image of the subject at the same time, and can calculate the distance from both images. At this time, since the positions of the visible image and the infrared image are aligned, it is possible to prevent the distance measured between the two images from changing.

Here, the distance calculation unit 1302 uses two visible images and two infrared images to calculate respective distances, then generates two stereo images (distance images) and outputs them as they are. can be Alternatively, the two generated stereo images are compared, and if the difference in the distance information is within a predetermined threshold range, one of the distance images is output, and if the difference exceeds the threshold, both distance images are output. Alternatively, output the distance image that was set in advance (for example, the distance image calculated with the infrared image is prioritized, the distance image showing the value of the close distance, etc.), or output the area part that exceeds it as analysis metadata You may make it output individually. FIG. 22 shows an example of an outline of processing of the distance calculation unit 1302. As shown in FIG.

In addition, the control unit 23 of the imaging device 1300 uses the signal output control unit 14 according to instructions from the controller device 200 to generate visible images and infrared images output from the two signal processing units 13(a) and 13(b). , the data to be output via the IF unit 16 out of the stereo image (distance image) output from the distance calculation unit 1302 is controlled. For example, when the imaging device 1300 is installed in a place where privacy protection is required (for example, a restroom or a changing room), only stereo images are output, and when it is installed in a place where high security is required, It is possible to use such as outputting all images.

On the other hand, the controller device 1310 replaces the moving body region extraction unit 209, the face region detection unit 210, the face feature point detection unit 211, the face verification unit 212, and the face DB 213 of the controller device 200 of the first and second embodiments. , a stereo image (distance image) in addition to a visible image and an infrared image, or only a stereo image to perform analysis processing and authentication processing. A feature point detection unit 1313, a face matching unit 1314, and a 3D face DB 1315 are installed. As a result, for example, in face recognition, by obtaining three-dimensional data on the unevenness of the face and using it, it is possible to accurately and easily detect a face region and face feature points.

Further, the controller device 1310 acquires a stereo image (distance image) from the imaging device 1300, refers to the distance of the moving body region extracted by the moving body region extraction unit 209, and executes face authentication if the distance is within a predetermined distance. It is possible to determine whether or not to execute other operations.

FIG. 23 shows an example in which the image pickup devices 1300(a), (b), and (c) in which the controller device 1310 is installed at different locations are displayed. The controller device 1310 performs 3D-based face recognition using the visible image or infrared image and the distance image received from the imaging device 1300(a) installed at the entrance of an office or building, and displays the execution result. This makes it easier to check visitors and suspicious persons, and helps to reduce congestion at reception. Also, using the distance image received from the imaging device 1300 (b) installed in a store such as a public facility or a commercial facility, the controller device 1310 can determine the number of people looking at the product shelf and information ( For example, gender, height, face direction, body direction, posture, etc.). This will help us to determine the level of interest from shoppers' purchasing demographics, line of sight and attitude, and strengthen marketing/sales capabilities related to products and displays. Using visible images, infrared images, and range images received from imaging devices 1300(c) installed outdoors in amusement parks, parks, etc., the controller device 1310 extracts a person from the image and performs a 3D-based Face authentication is performed, and if the person is confirmed to be a pre-registered person, that part is displayed as a distance image, and only the person who cannot be confirmed is displayed as a visible image or an infrared image. Alternatively, information that cannot identify a person (for example, gender, height, face direction, whether the person is with children, posture, etc.) is displayed. This helps ensure the safety of visitors and early detection of suspicious individuals.

Next, FIG. 14 shows another configuration example of the imaging system of this embodiment. This imaging system comprises one or more imaging devices 1400 and a controller device 1410 . Although not shown, the network 303 may include the imaging device 100 according to the first embodiment, the imaging devices 800 and 810 according to the second embodiment, and the imaging device 1300 described above. The controller device 1410 can manage any imaging device.

The image pickup device 1400 includes two moving object region extraction units 1401(a) and 1401(b) in the image pickup device 1300 described above. The moving body region extraction unit 1401 may be the same as the moving body region extraction unit 1311 of the controller device 1310 . Other components have the same configuration as the imaging device 1300 .

The moving object region extracting units 1401(a) and 1401(b) are portions that use infrared images output from the two signal processing units 13(a) and 13(b) to extract moving object regions from the respective images. Information on these extracted moving object areas can be output to the signal output control section 14 or the control section 23 and provided to the controller device 1410, as in the second embodiment. Further, the moving object region extracting units 1401(a) and 1401(b) extract a moving object region using the stereo image (distance image) output from the distance calculating unit 1302, and compare the result with the above-described method to obtain a A moving object region can be extracted with high accuracy. Alternatively, a stereo image (distance image) may be used first to extract a moving object area, and then an infrared image may be used to check only the extracted moving object area in more detail.

Here, the control unit 23 refers to the information of the two moving object regions output from the moving object region extracting units 1401(a) and 1401(b), compares the extracted numbers and positions, and stores the comparison result as analysis metadata. It can also be sent as The controller device 1410 can use the analysis metadata to select which of the right or left visible image or infrared image is used for face authentication. For example, as a result of extracting a moving object region by the moving object region extracting units 1401(a) and 1401(b), the results of extraction by the moving object region extracting unit 1401(a) (or moving object region extracting unit 1401(b)) are When the number is large, the control unit 23 of the imaging device 1400 combines the moving object region information extracted by the moving object region extracting unit 1401(a) (or the moving object region extracting unit 1401(b)) with the signal processing unit 13(a) (or A visible image or an infrared image output from the signal processing unit 13(b)) is sent.

On the other hand, in the controller device 1410, the controller device 1310 is equipped with the face area detection unit 210, the face feature point 211, the face matching unit 212, the face DB 213, and the comprehensive determination unit 1411 of the controller device 200 described in the first embodiment. do.

As a result, the imaging device 1400 performs face authentication processing (visible image and infrared image are used) using the face area detection unit 210, the face feature point 211, the face matching unit 212, and the face DB 213 described in Embodiment 1. , the above-described face region detection unit 1312, face feature point detection unit 1313, face matching unit 1314, and face authentication processing using the 3D face DB 1315 (using a visible image, an infrared image, and a stereo image). be able to. A comprehensive determination unit 1411 is a part that makes a final determination of the result of human authentication based on the results of executing both face authentication processes. By executing two different face recognition methods, face recognition with higher accuracy can be executed.

Similarly, FIG. 15 shows another configuration example of the imaging system of this embodiment. This imaging system comprises one or more imaging devices 1500 and a controller device 1510 . Although not shown, the network 303 may include the imaging device 100 of the first embodiment, the imaging devices 800 and 810 of the second embodiment, and the imaging devices 1300 and 1400 described above. The controller device 1510 can manage any imaging device.

The image pickup device 1500 includes two face area detection units 1502(a) and 1502(b) in the image pickup device 1400 described above. The face area detection section 1502 may be the same as the face area detection section 1312 of the controller device 1310 . Other components have the same configuration as the imaging device 1400 .

The face area detection units 1501(a) and 1501(b) are parts for extracting a human face area using the moving object area information output from the two moving object area extraction units 1401(a) and 1401(b). Information on these extracted face areas can be output to the signal output control section 14 or the control section 23 and provided to the controller device 1510, as in the second embodiment.

Here, the control unit 23 refers to the information of the two face regions output from the face region detection unit 1501(a) and (b), compares the number and positions of the extracted faces, and the direction of the face. Results can be sent as analysis metadata. The controller device 1510 can use the analysis metadata to select an image more suitable for face authentication, thereby performing face authentication with higher accuracy.

On the other hand, the controller device 1510 has an authentication method selection unit 1511 , an iris detection unit 1512 , an iris collation unit 1513 and an iris DB 1514 newly installed in the controller device 200 or the controller device 1410 .

An authentication method selection unit 1511 selects whether to perform face authentication or iris authentication using a visible image, an infrared image, a stereo image (distance image), analysis parameter information, etc. received from the imaging device 1500. is. For example, iris authentication is performed when an object enters a predetermined distance range, and face authentication is performed otherwise. Alternatively, face authentication is normally performed, and iris authentication is also performed when conditions for iris authentication are satisfied.

The iris detection unit 1512 detects the iris position of the human eye using the infrared image received from the imaging device 1500 and the analysis parameters including the face region extracted from the image, and further detects the boundary between the iris and the white of the eye. Then, the iris and accompanying boundary are detected, the iris area is specified, and the pupil code is generated. Any well-known technique may be applied to these methods.

Based on the information detected by the iris detection unit 1512, the iris matching unit 1513 performs matching using the iris DB 1514, similar to face authentication.

As a result, the controller device 1510 can select the optimum biometric authentication using the visible image, the infrared image, the stereo image (range image), the analysis parameter information, etc. received from the imaging device 1500. It is possible to perform personal authentication.

FIG. 24 shows an example in which a controller device 1410 processes an image captured by an imaging device 1500 installed at an airport, a building entrance, or the like. In this example, the controller device 1410 acquires a visible image and a face region and distance information as analysis parameters from the imaging device 1500, thereby performing 2D face authentication with a relatively light image processing load for a face region at a long distance. 3D face authentication, which requires a large image processing load, is performed for face regions that are close to each other. In addition, the controller device 1510 acquires a visible image, an infrared image, and a face region and distance information as analysis parameters from the imaging device 1500, and performs face authentication using the visible image for a face region at a long distance, Infrared images are used to perform iris authentication for face regions that are close to each other. In addition, in order to eliminate congestion caused by waiting for a turn, it is possible to determine persons who are close to a predetermined position in the captured image, and perform face authentication in that order.

FIG. 16 shows another configuration example of the imaging system of this embodiment. This imaging system shows an example in which it is installed in a mobile terminal such as a smart phone or a tablet.

17 and 18 show configuration examples of functional information and analysis parameters relating to imaging devices 1400 and 1500 used in this embodiment.

A stereo image (distance image) generated by the imaging device can be provided to the controller device by a transfer method similar to that for visible images and infrared images, as shown in FIG. 17 .

Alternatively, as shown in FIG. 18, it is also possible to provide by adding to the position information of the moving body area or the position information of the face area. In that case, only the distance information about the coordinate area corresponding to the moving body area or the face area is extracted and added.

FIG. 20 shows a configuration example for sending distance information as part of analysis parameters. Distance information is stored in the payload 1210. For example, the distance information on the extracted moving body region (number n) is stored immediately after the coordinate information of the moving body region (2001, 2002), and the distance information on the face region (number m) is stored on the face. Store immediately after the coordinate information of the area (2003, 2004). In addition to this configuration, a configuration in which coordinate information and distance information are alternately stored may be used.

FIG. 19 shows another configuration example of the imaging system of this embodiment. An imaging device 1900 of this imaging system includes one moving object region extraction unit 1901, and is configured to extract a moving object region using infrared images from either one of the left and right signal processing units 13(a) and 13(b). . Alternatively, one of the infrared images and the stereo image output from the distance calculation unit 1302 are used to extract the moving object region.

DESCRIPTION OF SYMBOLS 2... Sensor main body 3... Color filter 5... DBPF (optical filter) 11... Lens (optical system) 12... Imaging sensor part 13... Signal processing part 14... Signal output control part 15... Communication control part , 16... IF unit 23... control unit 100, 800, 810, 1300, 1400, 1500... imaging device 200, 1310, 1410, 1510... controller device 801, 1401... moving body region extraction unit 802, 1501... Face area detection unit 1301 Correction parameter calculation unit 1302 Distance calculation unit 1600 Portable terminal.

Claims (1)

two imaging elements;
Light that is provided corresponding to each of the two imaging elements, has a characteristic of transmitting at least a visible light wavelength region and an infrared wavelength region, and is incident on each of the two imaging elements based on the characteristics. two filter sections for filtering the
two signal processing units that process a signal obtained by imaging the light that has passed through the filter unit with the imaging element and output a visible light signal and an infrared signal;
a distance calculation unit that calculates a distance to an object using the two visible light signals or the two infrared signals output by the two signal processing units;
The first data based on the visible light signal or the third data based on the infrared signal output from the signal processing unit to the signal output control unit is combined with the third data based on the distance image generated by the distance calculation unit. a signal output control unit that adds or multiplies two data and transmits the added or multiplied first data or third data to the outside;
with
Depending on the time of day or the surrounding environment, the data to be transmitted to the outside by the signal output control unit is selected from the added or multiplied first data or the added or multiplied third data. switch automatically,
Imaging device.

Images