JP2023170389A - Information output device, imaging apparatus, information output method, and program - Google Patents

Abstract

To allow more useful information to be obtained by using a multi-spectral camera in a conventional information output device.SOLUTION: An information output device 10 comprises: an image acquisition unit 31 which acquires, as candidate images, three or more original images which are captured by a multi-spectral camera 90 capable of recording respective electromagnetic waves in a plurality of wavelength bands as images on a same optical axis and are mutually different in wavelength bands of electromagnetic waves to be recorded, or images based on the original images; a selection unit 34 which selects two or more images for use from among the three or more candidate images on the basis of a prescribed selection condition; an output information acquisition unit 35 which uses the two or more selected images for use to acquire output information; and an output unit 4 which outputs the output information. An electromagnetic wave to be recorded in at least one of the three or more original images is visible light, and electromagnetic waves to be recorded in at least two of the three or more original images are infrared light. By the information output device 10, more useful information can be obtained by using the multi-spectral camera.SELECTED DRAWING: Figure 2

Description

The present invention provides an information output device, a photographing device, and an information output device capable of outputting output information based on an original image photographed by a multi-wavelength spectroscopic camera capable of recording electromagnetic waves in a plurality of wavelength bands as images on the same optical axis. Regarding output methods and programs.

Various techniques have been considered for detecting a specific object within an imaging range by performing image processing based on a captured image signal. Such technology was used in surveillance cameras, vehicle cameras, etc. In addition to visible light cameras that detect visible light and take images, near-infrared cameras and far-infrared cameras that detect the temperature of the object being imaged were also used.

Conventionally, when performing image processing by arranging a visible light camera and a far-infrared camera side by side, there is a technique that requires parallax correction for the purpose of correcting the optical axis misalignment of the two types of cameras (for example, Patent Document 1 reference).

Furthermore, for the purpose of eliminating parallax, there is a technique (for example, see Patent Document 2) in which a visible light camera and a near-infrared camera are arranged on the same optical axis to perform high-speed imaging. There is a technique (for example, see Patent Document 3) in which a visible light camera of a first wavelength and a near-infrared camera or a far-infrared camera of a second wavelength are arranged on the same optical axis using an optical component that selects a wavelength.

Note that as techniques related to the same optical axis optical system, there are known techniques related to the same optical axis lens optical system (see, for example, Patent Document 4) and techniques related to the reflective optical system (see, for example, Patent Document 5).

In addition, there is a known technology that uses a NAM (non-additive mixing) circuit to output a pixel value with higher level priority among pixels at the same position in two original images to obtain a composite image ( For example, see Patent Document 6).

Japanese Patent Application Publication No. 2013-247492 Japanese Patent Application Publication No. 2017-156796 Japanese Patent Application Publication No. 2002-209126 Japanese Patent Application Publication No. 2011-232606 Japanese Patent Application Publication No. 2012-042949 Patent No. 6902150

With the spread of multi-wavelength spectroscopic cameras, it is desired to be able to obtain more convenient information using multi-wavelength spectroscopic cameras.

An object of the present invention is to provide an information output device, a photographing device, an information output method, and a program that can obtain more convenient information using a multi-wavelength spectroscopic camera.

The information output device of the first invention includes three or more different wavelength bands of electromagnetic waves to be recorded, which are photographed by a multi-wavelength spectroscopic camera capable of recording electromagnetic waves in a plurality of wavelength bands as images on the same optical axis. an image acquisition unit that acquires the original image or an image based on the original image as a candidate image for the original image; and a selection unit that selects two or more images to be used from among the three or more candidate images based on predetermined selection conditions. , an output information acquisition unit that acquires output information using the selected two or more used images, and an output unit that outputs the output information, and at least one of the three or more original images is the one to be recorded. The information output device is an information output device in which the electromagnetic waves are visible light, and the electromagnetic waves to be recorded are infrared rays for at least two of the three or more original images.

With this configuration, more convenient information can be obtained using a multi-wavelength spectroscopic camera.

Further, in the information output device of the second invention, in contrast to the first invention, the image acquisition unit is configured to acquire background difference images for each of three or more original images as candidate images. , the output information acquisition unit is an information output device that includes a recognition unit that performs recognition processing using two or more used images, and acquires the recognition result of the recognition processing as output information.

With this configuration, it is possible to improve the accuracy of recognizing a moving object.

Moreover, the information output device of the third invention is an information output device in which the selection unit selects two or more images to be used based on selection conditions regarding the gradation of the candidate images, in contrast to the second invention. be.

With this configuration, the accuracy of moving object recognition can be further improved.

Further, in the information output device of the fourth invention, in contrast to the first invention, the image acquisition unit acquires a background difference image by a background difference method as a candidate image for the original image in which at least one infrared ray is recorded. The selection conditions include selecting a candidate image that is an original image that records visible light and a candidate image that is a background difference image based on an original image that records infrared rays, and the output information acquisition unit is an information output device that includes an image synthesis section that synthesizes selected images to be used, and obtains a synthesized image synthesized by the image synthesis section as output information.

With this configuration, it is possible to output information as a composite image in which a region in the image area where the infrared radiation state has changed can be easily identified.

Further, in the information output device of the fifth invention, in contrast to the fourth invention, the image acquisition unit is configured to acquire, as a candidate image, at least a background difference image based on the original image in which far infrared rays are recorded. The selection condition is an information output device including selecting at least a candidate image that is an original image in which visible light is recorded, and a candidate image that is a background difference image based on the original image in which far infrared rays are recorded.

With this configuration, it is possible to output a composite image that allows easy identification of areas where the temperature has changed within the area shown in the image.

Further, in the information output device of the sixth invention, in contrast to the first invention, the selection condition is that the candidate image is an original image in which visible light is recorded and the candidate image is an original image in which far infrared rays are recorded. The output information acquisition unit is an information output device that includes an image synthesis unit that synthesizes the selected use images, and acquires the composite image synthesized by the image synthesis unit as output information.

With this configuration, it is possible to output a composite image that allows the location of heat generation in the subject to be easily understood.

Furthermore, in the information output device of the seventh invention, in contrast to the first invention, the image acquisition unit records each of visible light, near-infrared rays, and far-infrared rays while irradiating near-infrared rays in a predetermined range. It is configured to acquire at least three original images as candidate images, and the selection conditions are a candidate image that is an original image that records visible light, a candidate image that is an original image that records near-infrared rays, and a candidate image that is an original image that records near-infrared rays. The output information acquisition unit includes an image synthesis unit that synthesizes the selected use images, and outputs the synthesized image synthesized by the image synthesis unit as output information. This is an information output device that obtains information as follows.

With this configuration, in a dark environment, it is possible to output a composite image that can detect a person, etc., even in a place with insufficient illumination.

Further, the information output device of the eighth invention is provided with respect to any one of the fourth to seventh inventions, including a recognition unit that performs recognition processing using a composite image, and a recognition unit that performs recognition processing using a composite image, and , and a notification unit that notifies a user.

With this configuration, it is possible to appropriately notify the user depending on the composite image.

Furthermore, the photographing device of the ninth invention is provided with a multi-wavelength spectroscopic camera capable of recording electromagnetic waves in a plurality of wavelength bands as images on the same optical axis, and an information output It is a photographing device including a device.

With this configuration, more convenient information can be obtained using a multi-wavelength spectroscopic camera.

According to the present invention, it is possible to provide an information output device, a photographing device, an information output method, and a program that can obtain more convenient information using a multi-wavelength spectroscopic camera.

A diagram showing an outline of an imaging device using an information output device according to one embodiment of the present invention Block diagram of the imaging device Flowchart showing an example of the operation of the information output device Flowchart illustrating an example of output information acquisition processing in the first use of the information output device A table explaining an example of the operation of the information output device in the first application. A diagram explaining an example of the operation of the information output device in the first application. Flowchart illustrating an example of output information acquisition processing in the second use of the information output device A diagram illustrating an example of the operation of the information output device in the second use. A diagram illustrating an example of the operation of the information output device in the third application. A diagram explaining an example of the operation of the information output device in the fourth use. A block diagram showing the configuration of an imaging device according to a modified example of the present embodiment An overview diagram of the computer system in the above embodiment Block diagram of the computer system

Hereinafter, embodiments of the information output device and the like will be described with reference to the drawings. Note that in the embodiments, constituent elements with the same reference numerals perform similar operations, and therefore, repeated explanation may be omitted.

Note that the terms used below are generally defined as follows. Note that the meanings of these terms should not always be interpreted as shown here; for example, if they are individually explained below, they should also be interpreted in light of that explanation.

An identifier for a certain matter is a character, code, etc. that uniquely indicates the matter. The identifier is, for example, an ID, but any type of information may be used as long as it can identify the corresponding item. That is, the identifier may be the name of the thing it represents, or may be a combination of codes so as to uniquely correspond to each other.

The image is, for example, a still image or a moving image taken by a camera, but is not limited thereto. For example, it may be a still image extracted from a moving image captured by a camera.

Acquisition may include acquiring items input by the user, etc., or information stored in the own device or another device (which may be pre-stored information or information stored in the device itself). It may also include acquiring information generated by performing information processing in the process. Obtaining information stored in another device may include obtaining information stored in another device via an API or the like, or may include obtaining information stored in another device via an API or the like, or may include obtaining information stored in another device via an API. It may also include acquiring the content (including the content of a web page, etc.).

Furthermore, a so-called machine learning method may be used to obtain the information. For example, machine learning techniques can be used as follows. That is, a learning device that receives a specific type of input information and outputs the desired type of output information is configured using a machine learning method. For example, two or more sets of input information and output information are prepared in advance, and the two or more sets of information are given to a module for configuring a machine learning learning device to configure the learning device, and the configured learning device is Accumulate in storage. Note that the learning device can also be called a classifier. Note that the machine learning method may be, for example, deep learning, random forest, SVM, etc. Further, for machine learning, functions in various machine learning frameworks such as fastText, tinySVM, random forest, and TensorFlow, and various existing libraries can be used, for example.

Further, the learning device is not limited to one obtained by machine learning. The learning device may be, for example, a table showing the correspondence between input vectors based on input information and output information. In this case, the output information corresponding to the feature vector based on the input information may be obtained from the table, or the output information may be obtained using two or more input vectors in the table and parameters for weighting each input vector. A vector may be generated that approximates the feature vector based on the information, and final output information may be obtained using output information and parameters corresponding to each input vector used for generation. Further, the learning device may be, for example, a function representing a relationship between an input vector based on input information or the like and information for generating output information. In this case, for example, information corresponding to a feature vector based on input information may be obtained using a function, and output information may be obtained using the obtained information.

Outputting information means displaying it on a display, projecting it using a projector, printing it on a printer, outputting sound, sending it to an external device, storing it on a recording medium, processing it to other processing devices or other programs, etc. This is a concept that includes the delivery of results. Specifically, this includes, for example, displaying information on a web page, sending it as an e-mail, outputting information for printing, and the like.

Accepting information refers to accepting information input from input devices such as keyboards, mice, touch panels, etc., receiving information sent via wired or wireless communication lines from other devices, etc., receiving information from optical disks, magnetic disks, semiconductors, etc. This is a concept that includes receiving information read from a recording medium such as a memory.

(Embodiment)

In this embodiment, an information output device 10 having a function of outputting output information using an image captured by a multi-wavelength spectroscopic camera will be described.

FIG. 1 is a diagram showing an outline of a photographing device 100 using an information output device 10 according to one embodiment of the present invention.

In this embodiment, the photographing device 100 includes an information output device 10 and a multi-wavelength spectroscopic camera 90. The photographing device 100 is configured to be usable using, for example, a terminal device 600. In the photographing device 100, the number of multi-wavelength spectroscopic cameras 90 may be one, or two or more.

The information output device 10 and the terminal device 600 can communicate with each other via a network such as a local area network or the Internet.

The user of the photographing device 100 can use the photographing device 100 by transmitting information to the photographing device 100 and receiving information transmitted from the photographing device 100 using the terminal device 600. Can be done. The electronic computer used as the terminal device 600 is, for example, a personal computer (PC) such as a laptop computer, but is not limited thereto. For example, the terminal device 600, which is a portable information terminal device such as a smartphone, may be used, or a device other than these, such as a tablet type information terminal device, may be used. Although the following example will be described assuming that a personal computer is used as the terminal device 600, the present invention is not limited to this.

Note that the configuration of the imaging device 100 is not limited to this. The number of each device included in the photographing device 100 is not limited, and other devices may be included in the photographing device 100.

The multi-wavelength spectroscopic camera 90 is connected to the information output device 10 so as to be able to transmit information to the information output device 10. The connection method does not matter, as long as it is connected via a wireless or wired connection path. The multi-wavelength spectroscopic camera 90 may be connected to the information output device 10 via a network such as a local area network or the Internet.

In this embodiment, the multi-wavelength spectroscopic camera 90 is a camera that can record electromagnetic waves (including light rays) in a plurality of wavelength bands as images on the same optical axis. Examples of the electromagnetic waves in each of the plurality of wavelength bands include ultraviolet rays, visible light, and infrared rays. Here, infrared rays may be understood by classifying them into each wavelength band of near infrared rays, mid-infrared rays, and far infrared rays, for example. Note that the electromagnetic waves may be of frequency bands not included in the above, such as X-rays and microwaves.

The multi-wavelength spectroscopic camera 90 has an optical system that acquires optical signals or electromagnetic wave signals in each wavelength band of ultraviolet rays, visible light, near infrared rays, mid-infrared rays, and far infrared rays on the same optical axis, for example. It may be said that the multi-wavelength spectroscopic camera 90 has an optical signal acquisition section that has an optical system that acquires optical signals or electromagnetic wave signals in each of these wavelength bands along the same optical axis.

In this embodiment, the multi-wavelength spectroscopic camera 90 is configured to be able to record electromagnetic waves in three or more wavelength bands as images through the same optical axis optical system, for example, among all wavelengths from ultraviolet to far infrared. has been done. In other words, the multi-wavelength spectroscopic camera 90 is configured to image signals from the optical signal acquisition section using three or more types of multi-wavelength cameras. The multi-wavelength spectroscopic camera 90 can be configured using, for example, a known optical system with the same optical axis. Examples of the same optical axis optical system include a same optical axis lens optical system as shown in Patent Document 4 mentioned above. Further, for example, a reflective optical system as shown in the above-mentioned Patent Document 5 can be mentioned. The multi-wavelength spectroscopic camera 90 is capable of acquiring images in which electromagnetic waves of three or more wavelength bands are recorded using such a same optical axis optical system. In this embodiment, the images obtained in this manner are images of the same subject taken with the same optical axis and the same angle of view. The obtained image is transmitted to the information output device 10 and acquired by the information output device 10 as described later.

For example, the multi-wavelength spectroscopic camera 90 is composed of a component that separates incident electromagnetic waves into each wavelength band, an image sensor, etc., and is configured to be able to perform camera signal processing on the electromagnetic waves in each wavelength band. For example, for far infrared rays, images are recorded using a dedicated sensor. For other wavelength bands, for example, a prism that separates mid-infrared rays from ultraviolet rays is used to make electromagnetic waves in the target wavelength band incident on each sensor to record an image.

FIG. 2 is a block diagram of the imaging device 100.

As shown in FIG. 2, the information output device 10 includes a storage section 1, a reception section 2, a processing section 3, and an output section 4. The processing section 3 includes an image acquisition section 31 , a selection section 34 , an output information acquisition section 35 , and a notification section 36 . The output information acquisition section 35 includes an image composition section 351 and a recognition section 352. The information output device 10 is, for example, a server device, but is not limited to this.

The storage unit 1 stores various types of information, such as information used by the processing unit 3, for example. The storage unit 1 is preferably a non-volatile recording medium, but can also be realized by a volatile recording medium. Although acquired information and the like are stored in these, the process by which the information and the like are stored is not limited. For example, information etc. may be stored via a recording medium, information etc. transmitted via a communication line etc. may be stored, or input via an input device. information etc. may be stored.

In the present embodiment, the storage unit 1 stores, for example, original images, candidate images, etc. acquired by the image acquisition unit 31. Further, the storage unit 1 stores output information etc. acquired by the output information acquisition unit 35. Note that when a learning device is used in the processing in the processing section 3, the learning device may be stored in the storage section 1.

The original image is an image captured and acquired by the multi-wavelength spectroscopic camera 90. The original image is an image that has not been subjected to processing using other images, such as compositing or background subtraction. The original image can be said to be, for example, a spectral image obtained by spectrally dispersing an optical signal. The original image is, for example, an image obtained by performing one or more predetermined image processes on one spectral image. The original image is an image that can become a candidate image.

A composite image is an image obtained by combining two or more images. Note that the synthesis method does not matter. A composite image is an image obtained from two or more images. A background difference image is an image showing a background difference between two or more images. Such a background difference image may be interpreted as being included in the composite image, or may be interpreted as an image different from the composite image.

Note that an image identifier that identifies items related to images handled by the information output device 10, such as the original image and other images described later, may be used. The image identifier may be, for example, an identifier that identifies an individual image or an identifier that identifies the type of image. For example, it may be an identifier that can identify the wavelength band to be recorded, such as "visible light image," "far-infrared image," or "near-infrared image." Alternatively, it may be an identifier that identifies information regarding the origin or acquisition of the image, such as a "synthetic image" or "far-infrared background difference image." For example, the information may be information that can identify various types of information, such as original image identification information described in Patent Document 6.

The reception unit 2 receives various instructions, information, and the like. The various instructions, information, etc. are, for example, setting information. The setting information is information for specifying the image to be acquired by the selection unit 34.

Here, acceptance usually refers to acceptance of information input from an input device such as a touch panel, keyboard, or mouse. However, reception may be a concept that includes reception of information transmitted via a wired or wireless communication line, reception of information read from a recording medium such as an optical disk, a magnetic disk, or a semiconductor memory.

Any means for inputting the user's instructions may be used, such as a touch panel, keyboard, mouse, or menu screen.

The processing unit 3 performs various processes. The various processes are performed by the image acquisition section 31, the selection section 34, the output information acquisition section 35, and the notification section 36.

The image acquisition unit 31 acquires an original image or an image based on the original image as a candidate image for three or more original images photographed by the multi-wavelength spectroscopic camera 90 and having different wavelength bands of electromagnetic waves to be recorded. The image acquisition unit 31 stores the acquired candidate images in the storage unit 1 or the like. Note that acquiring as a candidate image means, for example, acquiring image data in a manner that allows it to be identified as a candidate image. For example, this may include, but is not limited to, recording a flag indicating that the image is a candidate image in association with the image, or storing the image in the storage unit 1 in a predetermined manner.

In this embodiment, the electromagnetic wave to be recorded in at least one of the three or more original images is visible light. Further, in at least two of the three or more original images, the electromagnetic waves to be recorded are infrared rays, and more specifically, one of them is far infrared rays and the other is near infrared rays. In addition, one may be far infrared rays, and the other may be mid-infrared rays. That is, it is sufficient if at least one of the three or more original images records far infrared rays. In other words, in this embodiment, the three or more original images with different wavelength bands of electromagnetic waves to be recorded include, for example, an original image in which visible light is recorded (visible light image), near-infrared rays or mid-infrared rays. Examples include an original image (near-infrared image or mid-infrared image) in which far-infrared rays are recorded, and an original image (far-infrared image) in which far-infrared rays are recorded. Note that instead of or in addition to these original images, original images recorded in other wavelength bands may be used. For example, in at least one of the three or more original images, the electromagnetic waves to be recorded are visible light, and in at least two of the three or more original images, the electromagnetic waves to be recorded are near infrared rays and mid-infrared rays. It's okay.

Further, in the present embodiment, an example of an image based on the original image is a background difference image obtained with respect to the original image. That is, the image acquisition unit 31 is configured to be able to acquire a background subtraction image of the original image using a background subtraction method using another original image that shares the same electromagnetic wave as the recording target and has a different shooting time. You can leave it there. Note that the image based on the original image may be an image obtained by performing other image processing or composition processing on the original image.

In the present embodiment, the image acquisition unit 31 is configured to acquire three original images recording each of visible light, near infrared rays, and far infrared rays as candidate images. Preferably, for the first application, the image acquisition unit 31 acquires as at least three original images in which visible light, near infrared rays, and far infrared rays are recorded while irradiating near infrared rays in a predetermined range. is configured to do so. Such an original image can be captured by the multi-wavelength spectroscopic camera 90, for example, by using near-infrared illumination together with the multi-wavelength spectroscopic camera 90.

The image acquisition unit 31 is configured to acquire, as a candidate image, a background subtraction image based on a background subtraction method for at least one original image in which infrared rays are recorded. Preferably, for one purpose, the image acquisition unit 31 is configured to acquire, as a candidate image, a background difference image based on at least an original image in which far-infrared rays are recorded. In the present embodiment, the image acquisition unit 31 is configured to acquire background difference images for each of three or more original images as candidate images. Note that the background difference image may be appropriately generated by the information output device 10 such as another part of the processing unit 3 (for example, the output information acquisition unit 35 described below), or may be generated by the multiwavelength spectroscopic camera 90. Good too. That is, the background difference image may be acquired by the information processing apparatus 10 based on the original image, or an image generated by the multi-wavelength spectroscopic camera 90 may be acquired by the information processing apparatus 10.

The selection unit 34 selects two or more images to be used from among the three or more candidate images based on predetermined selection conditions. Selecting a use image may mean, for example, that the information output device 10 makes it possible to identify that the candidate image has been selected as a use image. For example, this may include recording a flag indicating that it is a used image in association with a candidate image, or storing candidate images selected as used images in the storage unit 1 in a predetermined manner. Not limited.

Various selection conditions can be used. The selection conditions include, for example, the wavelength band in which the original image corresponding to the candidate image is to be recorded (for example, visible light, far infrared rays, etc.), or whether the candidate image is the original image or the background difference. Information regarding the type of image, such as whether it is an image, information regarding the conditions and environment at the time the original image was taken (e.g., time of day, date and time, presence or absence of lighting, etc.), information regarding the image quality of candidate images, etc., as appropriate. Configurable.

Further, the selection condition is not limited to one specific one. For example, when a plurality of selection conditions are set in advance and the reception unit 2 receives an instruction input regarding the usage made by the user, the selection unit 34 selects the image to be used according to the selection condition according to the instruction input. It may be configured to select. Regarding this, it may be interpreted that acceptance of a predetermined instruction input by the reception unit 2 is included in the selection condition. Note that the information processing device 10 may be configured to always function so as to be used only for a predetermined purpose without accepting input instructions or the like.

In this embodiment, a plurality of types of selection conditions are set for each of a plurality of uses, as shown below. That is, the selection conditions include a plurality of individual selection conditions corresponding to each of a plurality of uses.

For example, selection conditions regarding the gradation of candidate images are set for a predetermined purpose. That is, the selection unit 34 selects two or more images to be used based on selection conditions regarding the gradation of the candidate images. More specifically, the selection conditions may include conditions related to gradation and conditions that limit the image to be a background difference image. For example, among a plurality of candidate images that are background difference images, the selection condition is set to be the image with the largest gradation difference and the image with the second largest gradation difference. Note that the selection condition may be set to be an image with the smallest gradation difference among a plurality of candidate images that are background difference images.

Further, in this embodiment, selection conditions regarding a combination of a predetermined wavelength band and a predetermined type are set for a predetermined use, for example. For example, the selection condition is that the candidate image is an original image that records visible light, or that the candidate image is a background difference image based on an original image that records infrared rays. More specifically, for example, for one purpose, the candidate image is an original image recording visible light, or the candidate image is a background difference image based on an original image recording far infrared rays; is set as a selection condition. In addition, for another purpose, the candidate image is an original image that records visible light, the candidate image is a background difference image based on the original image that records near-infrared rays, and the candidate image is a candidate image that is a background difference image based on an original image that records near-infrared rays. The selection condition is that either the candidate image is a background difference image based on the recorded original image or the candidate image is satisfied.

The output information acquisition unit 35 acquires output information using the selected two or more used images. The output information may be, for example, an image acquired using the used image, or other information acquired using the used image. Other information may be information such as strings or flags that have meaning by themselves, may be strings such as hash values, or may be information that is a combination of these. . In the present embodiment, the output information acquisition unit 35 is configured to acquire instructions input by the user regarding usage and output information corresponding to a predetermined usage.

In this embodiment, for example, the output information acquisition section 35 includes an image synthesis section 351 and a recognition section 352, each of which is used for a corresponding purpose. That is, the output information acquisition unit 35 is configured to be able to perform, for example, an image composition function that combines images, and a recognition function that performs recognition processing based on images. Note that the output information acquisition section 35 may include a background subtraction function for obtaining a background subtraction image using a background subtraction method, and an image storage function for accumulating images in the storage section 1. Note that the output information acquisition unit 35 does not need to have any of these functions.

The image composition section 351 composes the use images selected by the selection section 34. It may be said that the image composition unit 351 obtains a composite image based on the used image. In some applications, the output information acquisition section 35 is configured to acquire a composite image acquired by the image composition section 351 as output information.

The recognition unit 352 performs recognition processing using the image and obtains a recognition result. The recognition unit 352 performs recognition processing using, for example, the use image selected by the selection unit 34. Further, the recognition unit 352 performs recognition processing using the composite image. The recognition result may be, for example, an image, segmentation information or label information indicating detected locations and contents of the image, or information such as a character string indicating the contents of the image. The recognition unit 352 is configured to perform image recognition using various known methods such as pattern matching, feature extraction, and machine learning. The recognition unit 352 may be configured to perform recognition processing of the image recognition device described in Patent Document 6 mentioned above, for example.

The notification unit 36 notifies the user when the recognition result of the recognition unit 352 satisfies a predetermined notification condition. For example, the notification unit 36 notifies the user based on the recognition result of the recognition unit 352 in the corresponding application. The notification condition may be, for example, that a recognition result indicating that a predetermined object has been detected or that a recognition result indicating that a predetermined state is obtained is obtained. Not limited. In addition, notifications to the user may include, for example, sending information to the terminal device 600 used by the user, sending information to contacts recorded in association with the user (e.g., sending an e-mail, This includes, but is not limited to, sending a message, etc. to a user), operating a device that can be confirmed by the user, etc.

Note that the notification unit 36 may perform the notification by outputting output information from the output unit 4. In other words, the notification unit 36 outputs the output information in a predetermined output mode when a predetermined notification condition is met, and otherwise outputs the output information in a different format from the predetermined output mode. You can also do this.

The output unit 4 outputs output information. The output unit 4 may or may not include an output device such as a display or a speaker. The output unit 4 can be realized by output device driver software, output device driver software and the output device, or the like. Further, the output unit 4 may output the output information by transmitting the output information to the terminal device 600 or the like.

Next, a basic operation example of the information output device 10 will be explained using a flowchart. The following processing is executed based on control by the processing unit 3.

FIG. 3 is a flowchart showing an example of the operation of the information output device 10.

(Step S1) The reception unit 2 receives instruction information regarding the usage input by the user. In other words, the processing unit 3 acquires the instruction information regarding the usage received by the reception unit 2. Note that the processing unit 3 may acquire instruction information that has been previously received by the reception unit 2 and stored in the storage unit 1.

(Step S2) The processing unit 3 acquires the original image as a candidate image.

(Step S3) The processing unit 3 obtains a background difference image of the original image as a candidate image.

(Step S4) The processing unit 3 selects two or more images to be used from among the three or more candidate images based on the selection conditions. The processing unit 3 selects images to be used, for example, based on selection conditions corresponding to instruction information regarding usage.

(Step S5) The processing unit 3 acquires output information. The processing unit 3 acquires the output information, for example, by an acquisition method according to the instruction information regarding the purpose.

Note that the selection of images to be used and the acquisition of output information may always be performed based on one selection condition. In this case, the process of step S1 may not be performed.

(Step S6) The processing unit 3 outputs output information.

Next, an example of the usage of the information output device 10 will be explained.

The first application is, for example, detecting a moving object within the photographing range of the multi-wavelength spectroscopic camera 90. In the photographing device 100 using the multi-wavelength spectroscopic camera 90, images of multiple wavelengths (multiple wavelength bands) without parallax taken on the same optical axis can be used, so moving object detection can be performed with high accuracy. Can be done.

In the first application, at least three types of background difference images: visible light, near-infrared light, and far-infrared light are used as candidate images. Further, as the selection condition, a condition regarding the gradation of the candidate image is used. For example, the selection condition is that the difference between the maximum and minimum gradation values among the candidate images is the smallest.

FIG. 4 is a flowchart illustrating an example of output information acquisition processing in the first application of the information output device 10.

In the first application, for example, the recognition unit 352 performs recognition processing on an image selected as a use image, and the result is output information. That is, for example, as shown in FIG. 4, the output information acquisition unit 35 acquires a used image (S11), and the recognition unit 352 performs recognition processing (S12). Then, when the recognition result satisfies the notification condition, the notification unit 36 acquires the corresponding notification content (S13), and the output information acquisition unit 35 acquires the recognition result as output information (S14). Note that when the notification content is acquired, the notification content may also be acquired as output information by the output information acquisition unit 35, but the present invention is not limited to this. Here, the notification conditions include, for example, the detection of some moving object or the detection of a portion to which a predetermined label (person, animal, etc.) is attached.

FIG. 5 is a table explaining an example of the operation of the information output device 10 in the first application. FIG. 6 is a diagram illustrating an example of the operation of the information output device 10 in the first application.

In Figure 5, the difference between the maximum and minimum values of 8-bit gradations (sometimes called gradation difference) is shown for background difference images created for typical images taken indoors and images taken outdoors. ). An image with the largest gradation difference has high detection sensitivity, but it can be said that it is likely to contain noise. On the other hand, since the detection sensitivity of an image with the smallest gradation difference is low, there is a high possibility that small objects will be difficult to detect. In this embodiment, since moving object detection is performed using a background difference image with a small gradation difference, large objects, which can be said to be relatively important, can be detected reliably by preventing the influence of noise. be able to.

FIG. 6 is an example of daytime outdoor imaging, and three types of background difference images of visible light, near-infrared light, and far-infrared light are shown for each of two scenes. Scene A includes a person within the angle of view and leaves are falling. In scene B, the shadow of a person outside the angle of view is within the angle of view, and fallen leaves are falling.

Looking at FIG. 6, it can be seen that in the far-infrared background subtraction image, which is a thermal image, the influence of fallen leaves is small and the influence of the shadow of a person is almost non-existent. When noise and the like are not generated, the difference between the maximum and minimum gradation values of such a far-infrared background difference image is relatively small. Therefore, as described above, the presence of a person can be detected with high accuracy while reducing the influence of fallen leaves, etc. and shadows.

That is, in the first application, recognition processing can be performed using an appropriate image depending on the situation among the background difference images of three types of wavelength bands. The accuracy of moving object recognition can be improved, and false alarms of moving object detection can be reduced.

Note that in the first application, the selection conditions and recognition method may be changed depending on the specific purpose. For example, first, the first judgment regarding moving object detection is made by focusing on the background difference image with the largest gradation difference or the background difference image with the second largest gradation difference, and then the first judgment is made with the background difference image with the smallest gradation difference. The first judgment may be modified (a second judgment may be made). Specifically, for example, in a region where a moving object was detected in the first judgment, if a moving object is also detected in the second judgment, a result that a moving object has been detected may be obtained. Reliable moving object detection becomes possible. In addition, when performing moving object detection with higher sensitivity than usual, the background difference image with the largest gradation difference or the second largest background difference image may be selected as the image to be used for recognition processing. .

The second use is, for example, to enable easy identification of heat traces within the photographing range of the multi-wavelength spectroscopic camera 90.

In the second application, at least the visible light and near-infrared original images and the far-infrared background difference image are used as candidate images. Also, as a selection condition, if the photographed area is bright, it must be the original image of visible light or a far-infrared background subtraction image, and if it is not bright, it must be the original image of near-infrared rays or the background of far-infrared rays. The fact that the image is a difference image is used. Note that it may be configured to always use either the visible light original image or the near-infrared original image, or to use both. Note that, for example, whether the photographed area is bright or not may be determined based on whether the gradation difference of the original image of visible light is greater than or equal to a predetermined threshold, or the multi-wavelength spectroscopic camera 90 The determination may be made based on the measurement results of an illuminance meter used with the above.

FIG. 7 is a flowchart illustrating an example of output information acquisition processing in the second application of the information output device 10.

In the second application, for example, a composite image synthesized by the image composition unit 351 with respect to an image selected as a use image is configured to be output information. That is, for example, as shown in FIG. 7, the output information acquisition section 35 acquires a used image (S21), and the image composition section 351 synthesizes the used image (S22). Then, the output information acquisition unit 35 acquires the composite image as output information (S23).

Note that in the second application, the heat trace recognition process by the recognition unit 352 may be performed based on a used image or a composite image that is a far-infrared background difference image. In that case, when a heat trace is detected (an example of a notification condition), a notification in a predetermined manner may be given to the user. For example, the output information may be an image in which a frame or the like indicating the location of the heat trace is combined with the composite image, or the output information may include information indicating that the heat trace has been detected, but the present invention is not limited to this.

FIG. 8 is a diagram illustrating an example of the operation of the information output device 10 in the second application.

In FIG. 8, an example of indoor imaging is shown in which heat traces are detected after a person touches a steel shelf for 3 seconds and leaves. Examples of a visible light original image (background image), a near-infrared original image, a far-infrared background difference image, and a composite image are shown. In the background subtraction image, the part where the temperature changed when the person touched the steel shelf for 3 seconds appears as a gradation difference.

In the example shown in FIG. 8, the ceiling light is on and the brightness is constant, so the visible light original image and the far-infrared background difference image are selected as the used images and combined. In the composite image, the portion where the temperature has changed is displayed superimposed on the visible light original image. In this way, the heat trace is displayed superimposed on the original visible light image or the original near-infrared image, so the user can easily grasp the area where the temperature has changed due to the touch of the person. . Note that if the far-infrared background difference image is stored in the storage unit 1 or the like for a long period of time, it is possible to display the location where the heat trace was present for a long period of time.

The results of an experiment to determine how long heat traces remain when a person touches each material are as follows.

That is, when contacting a copper kettle, a heat signature can be detected for only about 16 seconds after a 1 second contact, about 21 seconds after a 3 second contact, and for about 21 seconds after a 5 second contact.

Also, when contacting a steel shelf, a heat signature can be detected for only about 56 seconds after a 1 second contact, about 69 seconds after a 3 second contact, and about 79 seconds after a 5 second contact.

Further, when contacting the resin mat, heat traces can be detected for about 117 seconds after 1 second contact, about 211 seconds after 3 seconds contact, and for about 319 seconds after 5 seconds contact.

In the second application, by using a far-infrared background subtraction image during such a period, it is possible to detect heat traces that cannot be visually confirmed. Using a background difference image, a specific location touched by a person can be displayed in a state that is easier for the user to understand. This kind of thing is effective, for example, in calling attention to preventive measures against infectious diseases.

A third use is, for example, to enable easy identification of heat generating locations within the photographing range of the multi-wavelength spectroscopic camera 90.

In the third application, at least original images of visible light, near-infrared light, and far-infrared light are used as candidate images. Further, as a selection condition, if the photographed area is bright, it is an original image of visible light or far infrared rays, and if it is not bright, it is an original image of near infrared rays or far infrared rays. Note that it may be configured to always use either the original image of visible light or the original image of near-infrared rays, or to use both. Note that, for example, whether or not the photographed area is bright may be determined based on whether the gradation difference of the original image of visible light is greater than or equal to a predetermined threshold, or the multi-wavelength spectroscopic camera 90 The determination may be made based on the measurement results of an illuminance meter used with the above.

In the third application as well, as in the second application described above, the output information is, for example, a composite image synthesized by the image composition unit 351 for the images selected as the images to be used. Note that, in the third application as well, similarly to the above-described second application, the recognition unit 352 may perform the recognition process of the heat generation location based on the used image or the composite image. In that case, a notification in a predetermined manner may be provided to the user when a heat generation location is detected (an example of a notification condition). For example, the output information may be an image in which a frame or the like indicating the heat generation location is combined with the composite image, or the output information may include information indicating that the heat generation location has been detected, but the present invention is not limited to this.

FIG. 9 is a diagram illustrating an example of the operation of the information output device 10 in the third application.

In FIG. 9, an example is shown in which, when an indoor switchboard is the subject of photography, a heat-generating location on the switchboard is detected using the original image of far-infrared rays. In the example shown in FIG. 9, the image is captured indoors, and the gradation difference between the visible light original image and the near-infrared original image is approximately the same. Here, an example is shown in which a visible light original image and a far-infrared original image are combined. For example, in a dark scene such as at night, the near-infrared original image and the far-infrared original image may be combined. As shown in FIG. 9, in the composite image, the heat generating location is displayed superimposed on the visible light source image.

In this way, in the third application, a composite image that allows easy identification of the heat generating part of the subject can be used as output information. That is, since the heat generation location is shown superimposed on the visible light original image or the near-infrared light original image, the user can easily grasp the heat generation location. Since the location of heat generation is displayed superimposed on the original image of visible light or near-infrared rays, abnormal locations can be easily identified, which is useful for, for example, inspection work on electrical equipment. Note that, as a third use, it may be used to inspect solar panels. Solar panels can be inspected easily and power outage accidents can be easily prevented.

A fourth use is, for example, to make it possible to identify people who are far away and cannot be reached by illumination within the photographing range of the multi-wavelength spectroscopic camera 90 that requires illumination.

In the fourth application, at least original images of visible light, near-infrared light, and far-infrared light are used as candidate images. For example, an original image taken using visible light or near-infrared illumination can be used. Further, the selection conditions are set so that all original images of visible light, near infrared rays, and far infrared rays are selected as images to be used. For example, as a selection condition, the candidate image must be an original image that records visible light, a candidate image that is an original image that records near-infrared rays, or a candidate image that is an original image that records far-infrared rays. , is set.

Similarly to the above-described second use, the fourth use is also configured to output a synthesized image that is synthesized by the image synthesis unit 351 with respect to images selected as use images, for example. Note that in the fourth application as well, similarly to the above-mentioned second application, the recognition unit 352 may perform the recognition process of the heat generation location based on the used image or the composite image.

FIG. 10 is a diagram illustrating an example of the operation of the information output device 10 in the fourth application.

FIG. 10 is an example of nighttime outdoor imaging, and a plurality of people and a dog are included within the angle of view. Since it is nighttime and the environment is dark, visibility in visible light images is low. In the original near-infrared image taken using near-infrared illumination, the subject can only be captured within the range of the foreground illumination. On the other hand, the original far-infrared image also captures people far away, where light cannot reach. In the fourth application, these three types of original images are combined and the combined image is used as output information. In the output information, it is possible to include people in places where no illumination can reach so that the user can check them.

In this manner, in the fourth application, it is possible to output a composite image that can be detected, including a person in a place with insufficient lighting, in a dark environment. For example, by using it for monitoring as a more specific application, it is possible to obtain output information that allows the situation to be confirmed over a wider range even in a dark environment.

As explained above, according to the present embodiment, three or more types of images can be used as appropriate depending on the purpose, among images taken with the same optical axis for each of a plurality of wavelength bands, and images based on the images. This allows you to obtain highly convenient information.

Note that the processing in this embodiment may be realized by software. This software may then be distributed by software download or the like. Furthermore, this software may be recorded on a recording medium such as a CD-ROM and distributed. Note that the software that implements the information output device 10 in this embodiment is the following program. That is, this program is a program executed by the computer of the information output device 10, and is a program that connects the computer of the information output device 10 to a multi-wavelength spectroscopic camera capable of recording electromagnetic waves in a plurality of wavelength bands as images on the same optical axis. an image acquisition unit that acquires the original image or an image based on the original image as a candidate image for three or more original images shot by the user in different wavelength bands of electromagnetic waves to be recorded; a selection unit that selects two or more usage images from among the three or more candidate images; an output information acquisition unit that acquires output information using the selected two or more usage images; and an output that outputs the output information. This is a program to make it function as a section. Here, it is preferable that the electromagnetic waves to be recorded in at least one of the three or more original images are visible light, and the electromagnetic waves to be recorded in at least two of the three or more original images are infrared rays.

In the above-described embodiment, the photographing device 100 can be used by connecting the terminal device 600 to the information output device 10 connected to the multi-wavelength spectroscopic camera 90. The configuration of the device 100 is not limited to this. For example, a multi-wavelength spectroscopic camera 90 may constitute an imaging device 100B built into the information output device 10.

FIG. 11 is a block diagram showing the configuration of an imaging device 100B according to a modified example of the present embodiment.

As shown in FIG. 11, the photographing device 100B is the information output device 10 that includes a multi-wavelength spectroscopic camera 90 therein. In other words, the information output device 10 including an optical system functioning as a multi-wavelength spectroscopic camera 90 and an optical signal acquisition section is configured to be used as a photographing device 100B. This structure is constructed such that a storage section 1, a receiving section 2, a processing section 3, and an output section 4 are provided inside the multi-wavelength spectroscopic camera, and the multi-wavelength spectroscopic camera functions as an information output device 10. It may be considered that For example, the user can use the output result of the output information by using the photographing device 100B.

(others)

FIG. 12 is an overview diagram of the computer system 800 in the above embodiment. FIG. 13 is a block diagram of the computer system 800.

These figures show the configuration of a computer that executes the programs described in this specification to realize the information output device and the like of the embodiments described above. The embodiments described above may be implemented in computer hardware and computer programs executed thereon.

Computer system 800 includes a computer 801 including a CD-ROM drive, a keyboard 802, a mouse 803, and a monitor 804.

In addition to a CD-ROM drive 8012, the computer 801 is connected to an MPU 8013, a bus 8014 connected to the CD-ROM drive 8012, a ROM 8015 for storing programs such as a boot-up program, and connected to the MPU 8013. It includes a RAM 8016 for temporarily storing program instructions and providing temporary storage space, and a hard disk 8017 for storing application programs, system programs, and data. Although not shown here, the computer 801 may further include a network card that provides connection to a LAN.

A program that causes the computer system 800 to execute the functions of the information processing apparatus of the above-described embodiments may be stored in the CD-ROM 8101, inserted into the CD-ROM drive 8012, and further transferred to the hard disk 8017. Alternatively, the program may be transmitted to the computer 801 via a network (not shown) and stored on the hard disk 8017. The program is loaded into RAM 8016 during execution. The program may be loaded directly from CD-ROM 8101 or the network.

The program does not necessarily include an operating system (OS) that causes the computer 801 to execute the functions of the information processing apparatus of the above-described embodiment, a third party program, or the like. The program need only contain those parts of the instructions that call the appropriate functions (modules) in a controlled manner to achieve the desired results. How computer system 800 operates is well known and will not be described in detail.

Note that in the above program, in the transmission step for transmitting information and the reception step for receiving information, processing is performed by hardware, such as processing performed by the modem or interface card in the transmission step (processing that can only be performed by hardware). Processing that is not carried out) is not included.

Further, the number of computers that execute the above program may be one or more. That is, centralized processing or distributed processing may be performed.

Furthermore, in the embodiments described above, two or more components existing in one device may be physically realized by one medium.

Furthermore, in the above embodiments, each process (each function) may be realized by being centrally processed by a single device (system), or by being distributed by a plurality of devices. (In this case, the entire system made up of multiple devices that perform distributed processing can be understood as one “device”).

In addition, in the above embodiment, the information exchange performed between each component is performed by one component, for example, when the two components that exchange the information are physically different. This may be done by outputting information and receiving the information by the other component, or by one component if the two components passing the information are physically the same. This may be performed by moving from a phase of processing corresponding to the component to a phase of processing corresponding to the other component.

In the above embodiments, information related to processing executed by each component, for example, information accepted, acquired, selected, generated, transmitted, or received by each component. Information such as threshold values, formulas, addresses, etc. used by each component in processing may be stored temporarily or for a long period in a recording medium (not shown), even if not specified in the above description. Further, the information may be stored in the recording medium (not shown) by each component or by a storage unit (not shown). Further, each component or a reading section (not shown) may read information from the recording medium (not shown).

In addition, in the above-described embodiment, if the information used in each component, for example, information such as threshold values, addresses, various setting values, etc. used by each component in processing, may be changed by the user, the above-mentioned Even if it is not specified in the description, the user may or may not be able to change the information as appropriate. If the information can be changed by the user, the change is realized by, for example, a reception unit (not shown) that receives change instructions from the user, and a change unit (not shown) that changes the information in accordance with the change instruction. It's okay. The acceptance of the change instruction by the reception unit (not shown) may be, for example, acceptance from an input device, information transmitted via a communication line, or information read from a predetermined recording medium. .

The present invention is not limited to the embodiments described above, and various modifications can be made, which are also included within the scope of the present invention.

Some components and functions of the embodiments described above may be omitted.

As described above, the information output device etc. according to the present invention has the effect of being able to obtain more convenient information using a multi-wavelength spectroscopic camera, and is useful as an information output device etc.

1 Storage section 2 Reception section 3 Processing section 4 Output section 10 Information output device 31 Image acquisition section 34 Selection section 35 Output information acquisition section 36 Notification section 90 Multi-wavelength spectroscopic camera 100, 100B Photographing device 351 Image composition section 352 Recognition section

Claims (11)

Regarding three or more original images with different wavelength bands of electromagnetic waves to be recorded, which are taken by a multi-wavelength spectroscopic camera that can record electromagnetic waves in multiple wavelength bands as images on the same optical axis, the original image or the original an image acquisition unit that acquires an image based on the image as a candidate image;
a selection unit that selects two or more images to be used from among the three or more candidate images based on predetermined selection conditions;
an output information acquisition unit that acquires output information using the selected two or more used images;
and an output unit that outputs the output information,
At least one of the three or more original images has visible light as the electromagnetic wave to be recorded, and in at least two of the three or more original images, the electromagnetic wave to be recorded is infrared rays. The image acquisition unit is configured to acquire a background difference image for each of the three or more original images as the candidate image,
The information output device according to claim 1, wherein the output information acquisition unit includes a recognition unit that performs recognition processing using the two or more used images, and acquires a recognition result of the recognition processing as the output information. The information output device according to claim 2, wherein the selection unit selects two or more images to be used based on the selection condition regarding the gradation of the candidate image. The image acquisition unit is configured to acquire, as the candidate image, a background difference image based on a background subtraction method for at least one original image in which infrared rays are recorded;
The selection condition includes selecting a candidate image that is an original image recording visible light and a candidate image that is a background difference image based on the original image recording the infrared rays,
The information output device according to claim 1, wherein the output information acquisition section includes an image composition section that composes selected use images, and acquires a composite image composed by the image composition section as the output information. The image acquisition unit is configured to acquire, as the candidate image, a background difference image based on an original image in which at least far infrared rays are recorded;
5. The selection condition includes selecting at least a candidate image that is an original image in which visible light is recorded, and a candidate image that is a background difference image based on the original image in which far-infrared rays are recorded. Information output device. The selection conditions include selecting a candidate image that is an original image that records visible light and a candidate image that is an original image that records far infrared rays,
The information output device according to claim 1, wherein the output information acquisition section includes an image composition section that composes selected use images, and acquires a composite image composed by the image composition section as the output information. The image acquisition unit is configured to acquire, as at least the candidate images, three original images in which each of visible light, near infrared rays, and far infrared rays are recorded while irradiating near infrared rays in a predetermined range,
The selection conditions include selecting a candidate image that is an original image that records visible light, a candidate image that is an original image that records near-infrared rays, and a candidate image that is an original image that records far-infrared rays,
The information output device according to claim 1, wherein the output information acquisition section includes an image composition section that composes selected use images, and acquires a composite image composed by the image composition section as the output information. a recognition unit that performs recognition processing using the composite image;
The information output device according to any one of claims 4 to 7, further comprising a notification unit that notifies a user when the recognition result satisfies a predetermined notification condition. An imaging device comprising: a multi-wavelength spectroscopic camera capable of recording electromagnetic waves in a plurality of wavelength bands as images on the same optical axis; and an information output device according to claim 1. An information output method performed using an image acquisition unit, a selection unit, an output information acquisition unit, and an output unit, the method comprising:
Regarding three or more original images in which the image acquisition unit has different wavelength bands of electromagnetic waves to be recorded, which are taken by a multi-wavelength spectroscopic camera capable of recording electromagnetic waves in a plurality of wavelength bands as images on the same optical axis, an image acquisition step of acquiring the original image or an image based on the original image as a candidate image;
a selection step in which the selection unit selects two or more images to be used from among the three or more candidate images based on predetermined selection conditions;
an output information acquisition step in which the output information acquisition unit acquires output information using the selected two or more used images;
The output unit includes an output step of outputting the output information,
In at least one of the three or more original images, the electromagnetic wave to be recorded is visible light, and in at least two of the three or more original images, the electromagnetic wave to be recorded is infrared rays. computer,
Regarding three or more original images with different wavelength bands of electromagnetic waves to be recorded, which are taken by a multi-wavelength spectroscopic camera that can record electromagnetic waves in multiple wavelength bands as images on the same optical axis, the original image or the original an image acquisition unit that acquires an image based on the image as a candidate image;
a selection unit that selects two or more images to be used from among the three or more candidate images based on predetermined selection conditions;
an output information acquisition unit that acquires output information using the selected two or more used images;
An output unit that outputs the output information, and a program for functioning as an output unit,
In at least one of the three or more original images, the electromagnetic wave to be recorded is visible light, and in at least two of the three or more original images, the electromagnetic wave to be recorded is infrared rays.

Images