Classifications

• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
  • G01J3/02—Details
  • G01J3/0297—Constructional arrangements for removing other types of optical noise or for performing calibration
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
  • G01J3/02—Details
  • G01J3/10—Arrangements of light sources specially adapted for spectrometry or colorimetry
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
  • G01J3/28—Investigating the spectrum
  • G01J3/2823—Imaging spectrometer
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
  • G01J3/28—Investigating the spectrum
  • G01J3/40—Measuring the intensity of spectral lines by determining density of a photograph of the spectrum; Spectrography
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
  • G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
  • G01N21/27—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands using photo-electric detection ; circuits for computing concentration
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/80—Camera processing pipelines; Components thereof
  • H04N23/84—Camera processing pipelines; Components thereof for processing colour signals
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
  • G01J3/02—Details
  • G01J3/10—Arrangements of light sources specially adapted for spectrometry or colorimetry
  • G01J2003/102—Plural sources
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
  • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
  • G01J3/28—Investigating the spectrum
  • G01J2003/283—Investigating the spectrum computer-interfaced
  • G01J2003/2843—Processing for eliminating interfering spectra

Definitions

• the present disclosure relates to an imaging system.
• spectral information of a large number of narrow wavelength bands (hereinafter also simply referred to as “bands”), for example, several dozen bands, each has a narrow band, this is not possible with conventional RGB images that have only three bands of information. It becomes possible to understand the detailed physical properties of the object. Cameras that capture images in many such wavelength bands are called “hyperspectral cameras.” Hyperspectral cameras are used in various fields such as food testing, biological testing, drug development, and mineral component analysis.
• Patent Document 1 discloses an image analysis device for analyzing the distribution of substances in biological tissues.
• the image analysis device acquires a plurality of sample images by irradiating and photographing biological tissue with light in N wavelength bands selected from a predetermined wavelength range. Sample data based on a plurality of sample images and material teacher data are compared to generate material distribution data in the tissue.
• Patent Document 1 discloses normalizing the intensity of light reflected on the surface of a sample based on the intensity of light reflected on the surface of a reference member such as a white board.
• Patent Document 2 discloses an example of a hyperspectral imaging device that uses compressed sensing.
• Compressed sensing is a technology that restores more data than the observed data by assuming that the data distribution of the observed object is sparse in a certain space (for example, frequency space).
• the imaging device disclosed in Patent Document 2 includes an encoding element, which is an array of a plurality of optical filters having different spectral transmittances, on an optical path connecting an object and an image sensor.
• the imaging device can generate images of each of a plurality of wavelength bands in one imaging by performing a restoration calculation based on a compressed image obtained by imaging using an encoding element.
• the light source may be adjusted to satisfy good lighting conditions. Every time the light source was adjusted, it was necessary to obtain spectral information of the subject for calibration, such as a white board.
• the present disclosure provides an imaging system and method that can save labor in the imaging process after adjusting the light source.
• An imaging system includes a light source, an imaging device that photographs a subject illuminated with light from the light source and generates image data, and a processing device.
• the image data includes image information of each of four or more bands, or information of a compressed image obtained by compressing the image information of the four or more bands into one image.
• the processing device determines whether pixel values of a plurality of pixels in the image data satisfy a predetermined condition, and if the predetermined condition is not satisfied, the processing device determines whether or not the pixel values of a plurality of pixels in the image data satisfy a predetermined condition, and if the predetermined condition is not satisfied, the processing device
• the illumination conditions by the light source are changed under the condition that the spectral shape of the light source does not change.
• the general or specific aspects of the present disclosure may be implemented in a system, apparatus, method, integrated circuit, computer program or recording medium such as a computer readable recording disk, and the system, apparatus, method, integrated circuit, It may be realized by any combination of a computer program and a recording medium.
• the computer-readable recording medium includes, for example, a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory).
• a device may be composed of one or more devices. When the device is composed of two or more devices, the two or more devices may be arranged within one device, or may be arranged separately into two or more separate devices.
• the term "device" may refer not only to one device, but also to a system of multiple devices.
• FIG. 1 is a block diagram showing a schematic configuration of an imaging system.
• FIG. 2 is a flowchart illustrating an example of a processing method executed by a processing device.
• FIG. 3 is a diagram schematically illustrating an example configuration of an imaging system according to an exemplary embodiment of the present disclosure.
• FIG. 4A is a flowchart illustrating an example of a method for generating a hyperspectral image.
• FIG. 4B is a flowchart illustrating a variation of the method shown in FIG. 4A.
• FIG. 5 is a flowchart illustrating an example of a method for determining the specified range.
• FIG. 6A is a diagram illustrating an example of the data structure of a hyperspectral image.
• FIG. 6B is a diagram showing another example of the data structure of a hyperspectral image.
• FIG. 6C is a diagram showing still another example of the data structure of a hyperspectral image.
• FIG. 7 is a flowchart showing a modification of the method shown in FIG.
• FIG. 8 is a block diagram showing a configuration example of an imaging system.
• FIG. 9 is a flowchart illustrating another example of a hyperspectral image generation method.
• FIG. 10 is a flowchart illustrating another example of a method for determining the specified range.
• FIG. 11 is a block diagram showing another example of the configuration of the imaging system.
• FIG. 12A is a diagram schematically showing a configuration example of an imaging device.
• FIG. 12B is a diagram schematically showing another configuration example of the imaging device.
• FIG. 12A is a diagram schematically showing a configuration example of an imaging device.
• FIG. 12C is a diagram schematically showing still another configuration example of the imaging device.
• FIG. 12D is a diagram schematically showing still another configuration example of the imaging device.
• FIG. 13A is a diagram schematically showing an example of a filter array.
• FIG. 13B is a diagram illustrating an example of the spatial distribution of light transmittance of each of a plurality of wavelength bands included in the target wavelength range.
• FIG. 13C is a diagram showing an example of the spectral transmittance of area A1 included in the filter array shown in FIG. 13A.
• FIG. 13D is a diagram showing an example of the spectral transmittance of area A2 included in the filter array shown in FIG. 13A.
• FIG. 13A is a diagram schematically showing still another configuration example of the imaging device.
• FIG. 13A is a diagram schematically showing an example of a filter array.
• FIG. 13B is a diagram illustrating an example of the spatial distribution of light transmittance of each of a plurality of wavelength bands included
• FIG. 14A is a diagram for explaining an example of the relationship between a target wavelength range and a plurality of wavelength bands included therein.
• FIG. 14B is a diagram for explaining another example of the relationship between a target wavelength range and a plurality of wavelength bands included therein.
• FIG. 15A is a diagram for explaining the characteristics of spectral transmittance in a certain area of the filter array.
• FIG. 15B is a diagram showing the results of averaging the spectral transmittance shown in FIG. 13A for each wavelength band.
• all or part of a circuit, unit, device, member, or part, or all or part of a functional block in a block diagram may be, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration). ) may be implemented by one or more electronic circuits.
• An LSI or IC may be integrated into one chip, or may be configured by combining a plurality of chips.
• functional blocks other than the memory element may be integrated into one chip.
• it is called LSI or IC, but the name changes depending on the degree of integration, and it may also be called system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration).
• FPGAs Field Programmable Gate Arrays
• RLDs Reconfigurable Logic Devices
• circuit, unit, device, member, or section can be executed by software processing.
• the software is recorded on one or more non-transitory storage media such as ROM, optical disk, hard disk drive, etc., and when the software is executed by a processor, the functions specified in the software are executed. Executed by processing units and peripheral devices.
• a system or apparatus may include one or more non-transitory storage media on which software is recorded, a processing unit, and necessary hardware devices, such as interfaces.
• a collection of data or signals representing an image that is, data or signals representing the respective pixel values of a plurality of pixels in the image, may be simply referred to as an "image.”
• One method of adjusting the brightness at the position of the subject is to adjust control parameters such as current or voltage for driving the light source.
• control parameters such as current or voltage for driving the light source.
• a method may be considered in which the direction of the light source is changed to adjust the brightness at the position of the subject.
• the above operation can be performed by, for example, photographing a calibration object such as a white board and obtaining information on its brightness distribution and spectrum.
• the illumination condition may be a condition regarding the amount of light emitted to a subject to be inspected or analyzed or to a calibration subject such as a white board.
• the lighting conditions include, for example, the distance between the light source and the subject, the duty ratio of the current, voltage, or PWM (Pulse Width Modulation) signal used to drive the light source, or the ND (Neutral Neutral Control) placed between the light source and the subject.
• Density can be defined by parameters such as filter attenuation rate.
• the present inventors have discovered that there is a parameter range in which the spectral shape of light at the position of the subject does not change significantly and can be treated as approximately constant. For example, if the distance between the light source and the subject is changed without changing the direction of the light source, there is a range of distances where the spectral shape of the light at the subject position does not change significantly and can be treated as approximately constant.
• the range of parameters such as distance in advance and adjusting the parameters within that range, it is possible to adjust the brightness without having to retake a photo of a calibration subject such as a white board.
• spectral shape means the shape of a spectrum (i.e., wavelength distribution of light intensity) in which the intensity of each wavelength band is normalized by the intensity of a certain reference wavelength band.
• intensity of each wavelength band in an unstandardized spectrum is sometimes called “spectral intensity.” It is interpreted that the spectral shapes are the same between a certain spectrum and a spectrum in which the intensity of each wavelength band in the spectrum is uniformly multiplied by a constant.
• FIG. 1 is a block diagram showing a schematic configuration of an imaging system according to one aspect of the present disclosure.
• the imaging system includes a light source 50, an imaging device 100, and a processing device 200.
• the imaging device 100 photographs a subject illuminated with light from the light source 50 and generates image data.
• the image data includes image information of each of four or more bands, or information of a compressed image obtained by compressing the image information of each of four or more bands into one image.
• the processing device 200 determines whether the pixel values of a plurality of pixels in the image data satisfy a predetermined condition. If the predetermined condition is not met, the processing device 200 changes the illumination conditions by the light source 50 under the condition that the spectral shape of the light from the light source 50 at the position of the subject does not change.
• FIG. 2 is a flowchart illustrating an example of a processing method executed by the processing device 200.
• the processing device 200 acquires image data generated by the imaging device 100.
• the image data is data that includes image information of each of four or more bands, or data that includes information of a compressed image obtained by compressing the image information of four or more bands into one image.
• Each of the four or more bands may be, for example, a relatively narrow wavelength range included in a preset target wavelength range.
• the target wavelength range W can be set to various ranges depending on the application.
• the target wavelength range W may be, for example, a visible light wavelength range from about 400 nm to about 700 nm, a near-infrared wavelength range from about 700 nm to about 2500 nm, or another wavelength range. It's okay.
• Each band may be a wavelength range with a predetermined width, such as 5 nm, 10 nm, 20 nm, or 50 nm.
• the widths of the four or more bands may be the same or different.
• image data including image information of four or more bands may be referred to as a "hyperspectral image.”
• a hyperspectral image may include image information of 10 or more bands, 30 or more bands, or 50 or more bands, for example.
• Image data including compressed image information can be generated by imaging using an optical element called an encoding element described in Patent Document 2, for example. The entire disclosure content of Patent Document 2 is incorporated herein by reference.
• image data including compressed image information may be simply referred to as a "compressed image.”
• a compressed image is a monochrome image in which image information of four or more bands is compressed. As will be described later, the image of each band can be restored by a restoration calculation based on the compressed image and data indicating the spatial distribution of spectral transmittance of the encoding element.
• step S12 the processing device 200 determines whether the pixel values of a plurality of pixels in the acquired image data satisfy a predetermined condition. If the predetermined condition is not met, the process advances to step S13. If the predetermined conditions are met, the process advances to step S14. Specific examples of the conditions will be described later.
• the processing device 200 changes the illumination conditions by the light source 50 under the condition that the spectral shape of the light at the position of the subject does not change.
• the processing device 200 can change the illumination condition by changing the parameters that define the illumination condition within a specified range in which the spectral shape at the position of the subject does not change.
• the parameters include, for example, the distance between the light source 50 and the subject, the current, voltage, or duty ratio of the PWM signal for driving the light source 50, or the attenuation rate of an ND filter placed between the light source 50 and the subject. It can be a parameter to be determined. Thereby, the brightness at the position of the subject can be changed while maintaining the spectral shape.
• step S13 After step S13, return to step S12.
• the processing device 200 changes the lighting conditions until a predetermined condition is satisfied in step S12. If the predetermined condition is satisfied, the process proceeds to step S14, and the processing device 200 determines the illumination condition as the current condition.
• the spectral shape does not change does not mean that the spectral shape does not change at all, but means that the change in the spectral shape is within an allowable range depending on the purpose or use. Changes in the spectral shape can be evaluated based on the angle or inner product between vectors, assuming that the spectrum is a vector with dimensions of the number of bands N (N is an integer of 4 or more). For example, if the angle between two N-dimensional vectors representing two spectra is less than a reference value, it can be said that the shapes of both spectra are the same.
• the reference value may be a relatively small value, such as 1°, 3°, 5°, or 10°. Details of the method for evaluating the magnitude of change in spectral shape based on the angle between vectors will be described later.
• the imaging system may further include an adjustment device that adjusts the distance between the light source 50 and the subject. If the predetermined condition is not satisfied, the processing device 200 may change the illumination condition by causing the adjustment device to change the distance between the light source 50 and the subject within a specified range. In that case, the processing device 200 determines the distance between the light source 50 and the subject as a parameter that defines the illumination condition.
• the "specified range” is a parameter range in which the spectral shape of the light from the light source 50 can be considered to be approximately constant at the position of the subject.
• the prescribed range may be determined in advance by photographing using a calibration object such as a white board, for example.
• the prescribed range may be stored in advance in a storage device inside or outside the processing device 200.
• the processing device 200 may be configured to vary the lighting conditions by varying the parameter within the defined range based on the defined range stored in the storage device. For example, when changing the illumination condition by changing the distance between the light source 50 and the subject, the processing device 200 controls the adjustment device to change the distance within a specified range.
• the brightness can be adjusted while maintaining the spectral shape of the illumination light at the position of the subject. Therefore, it is possible to omit the work of reacquiring spectrum information of a calibration object such as a white board, which was conventionally necessary after adjusting the brightness. This makes it possible to more efficiently acquire spectrum information of the subject.
• the "predetermined condition” is, for example, that the pixel value of each of a plurality of pixels in the image data (hereinafter also simply referred to as "image") generated by the imaging device 100 is within a predetermined range.
• This may include conditions such as: That is, determining whether the pixel values of the plurality of pixels satisfy a predetermined condition may include determining whether the pixel value of each of the plurality of pixels is within a predetermined range.
• the predetermined range is, for example, a range in which pixel saturation of the image sensor included in the imaging device 100 does not occur (i.e., not too bright) and a good S/N ratio can be obtained (i.e., not too dark), and is set in advance. can be done.
• the pixel values of each of the plurality of pixels are within a predetermined range does not necessarily mean that the pixel values of all pixels in the image data are within the predetermined range. That is, "a plurality of pixels” does not necessarily mean all pixels in the image data, but may mean some pixels. For example, a condition may be set that the pixel values of a preset percentage (80%, 60%, 40%, etc.) of pixels are within a predetermined range.
• the “predetermined condition” is a condition in which a contrast value calculated from the pixel values of a plurality of pixels in image data generated by the imaging device 100 exceeds a threshold value, in place of or in addition to the above condition. May contain. That is, determining whether the pixel values of the plurality of pixels satisfy the above condition includes determining whether the contrast value calculated from the pixel values of the plurality of pixels exceeds the threshold value. Good too.
• the contrast value is, for example, an index value representing the degree of spread of pixel values in a histogram of an image (that is, a graph representing the relationship between pixel values and the frequency of pixel values).
• the contrast value can be quantitatively determined using, for example, the half-width in the histogram of the image, the difference between the maximum pixel value and the minimum pixel value, the variance, or the standard deviation.
• the contrast of a plurality of pixels included in a compressed image reflects randomness in encoding for each wavelength band during imaging. Therefore, by increasing the contrast, it is possible to improve the encoding performance, and it is possible to improve the convergence of the solution in the restoration calculation, which will be described later. Therefore, when determining whether or not a predetermined condition is satisfied for image data including compressed image information, images of each band are restored from the compressed image by imposing a condition that the contrast value exceeds a threshold value. Restoration errors in processing can be reduced.
• the processing device 200 repeats the operation of changing the parameter defining the illumination condition within the specified range until the predetermined condition is satisfied. It may be configured as follows. According to such a configuration, parameters such as the distance between the light source 50 and the subject can be automatically adjusted so that the subject is illuminated with appropriate brightness. For example, if a predetermined condition is not met, the processing device 200 is configured to cause the adjustment device to repeat an operation of changing the distance between the light source 50 and the subject within a predetermined range until the predetermined condition is met. may have been done.
• the adjustment device may be configured to change the distance between the light source 50 and the subject without changing the direction of the light source 50.
• the adjustment device may include an actuator such as a linear actuator that translates the light source 50 away from or closer to the subject.
• the distance between the light source 50 and the subject can be adjusted while the direction of the light source 50 is fixed. This makes it easy to adjust the brightness while maintaining the spectral shape of the light illuminating the subject.
• the processing device 200 may adjust the brightness at the subject position by changing the control parameters for driving the light source 50. That is, if the predetermined condition is not satisfied in step 12, the processing device 200 changes the illumination condition by changing the control parameter for driving the light source 50 within a predetermined range. It may be configured as follows.
• the control parameters for driving the light source 50 differ depending on the configuration of the light source 50. If the light source 50 is, for example, a light emitting diode (LED), the control parameter may be the current, voltage, or duty ratio of the PWM signal to drive the LED.
• the imaging system may further include a mechanism for inserting one ND filter selected from a plurality of ND filters between the light source 50 and the subject. If the predetermined condition is not met, the processing device 200 may change the illumination condition by causing the mechanism to change the ND filter inserted between the light source 50 and the subject.
• One ND filter to be used is selected from a plurality of ND filters having different light attenuation rates or transmittances. According to such a configuration, the brightness can be changed by replacing the ND filter without changing the spectral shape at the position of the subject.
• the imaging device 100 may be configured to generate image data that includes image information for each of four or more bands.
• the imaging device 100 may be configured to spatially separate light for each wavelength using a spectroscopic element such as a prism or a grating to obtain images of each band.
• the imaging device 100 may include a plurality of optical filters, each having a transmission wavelength range corresponding to one of a plurality of bands, in front of the image sensor.
• Such an imaging device 100 may be configured to generate images of each band based on intensity information of light transmitted through each optical filter.
• the imaging device 100 may include four or more image sensors each corresponding to four or more bands. In that case, each image sensor generates an image of a corresponding band.
• the imaging device 100 may be a hyperspectral camera that generates image data including image information for each of 10 or more or 100 or more bands, for example.
• the imaging device 100 may be configured to generate image data that includes compressed image information in which image information of four or more bands is compressed into one image.
• Such an imaging device 100 may have a configuration similar to that of the imaging device 100 disclosed in Patent Document 2, for example.
• the imaging device 100 includes an optical element that changes the spatial distribution of the intensity of light from a subject for each wavelength, and an image sensor that receives the light that has passed through the optical element and generates image data that includes compressed image information.
• the optical element may be, for example, a filter array including a plurality of optical filters arranged in a two-dimensional plane.
• the spectral transmittances of the plurality of optical filters in the filter array are different from each other, and the spectral transmittance of each of the plurality of optical filters can be designed to exhibit a plurality of maximum values. According to such a configuration, it is possible to generate a compressed image in which the image information of each of the four or more bands can be restored by a restoration process based on compressed sensing.
• the processing device 200 is configured to execute processing for generating images of each of the four or more bands based on the compressed image. can be done.
• the processing device 200 includes a compressed image, data indicating the spatial distribution of spectral transmittance of an optical element such as a filter array, and a spectrum of illumination light obtained by photographing a calibration object (for example, a white board) in advance. and spectral data indicating the spatial distribution of the four or more bands.
• the processing device 200 may generate other image data including image information of each of four or more bands based on the image data including information of the compressed image output from the image sensor. Further, the processing device 200 executes processing for generating images of each of the four or more bands based on the compressed image when the predetermined condition is satisfied, and The configuration may be such that the process is not executed if the process is not executed. According to such a configuration, an image of each band is generated when the pixel values of the compressed image satisfy favorable conditions. Therefore, images of each of four or more bands can be generated with high quality.
• the process of generating images of each of four or more bands based on the compressed image is not limited to the processing device 200, and may be executed by the processor of the imaging device 100.
• the imaging device 100 uses an optical element such as the filter array described above, an image sensor that receives light that has passed through the optical element, and images of each of four or more bands based on signals output from the image sensor. and a processor that generates image data including information.
• the processor first generates a compressed image based on the signal output from the image sensor.
• the processor performs processing based on the compressed image and data reflecting the spatial distribution of the spectral transmittance of the optical element and the spatial distribution of the spectrum of the light from the light source 50, thereby converting the four or more bands. may be configured to generate images of each of the.
• the predetermined condition is such that each pixel value of a plurality of pixels in each image of the four or more bands is It may also include a condition that it is within a predetermined range. Instead of or in addition to this condition, the predetermined condition includes a condition that a contrast value calculated from a plurality of pixel values in each image of the four or more bands exceeds a threshold value. Good too.
• the imaging system may further include a storage device that stores data indicating a defined range of the parameter that defines the illumination condition.
• the specified range represents a parameter range in which the spectral shape at the position of the subject does not change.
• the processing device 200 may be configured to change the illumination condition by changing the parameter within a specified range based on data indicating the specified range.
• the processing device 200 may be configured to cause the adjustment device to change the distance between the light source 50 and the subject within a specified range based on data indicating the specified range of the distance between the light source 50 and the subject.
• the imaging system may further include a stage having a support surface that supports the subject.
• the adjustment device may include a linear actuator that changes the distance between the light source 50 and the subject by moving the light source 50 in a direction perpendicular to the support surface of the stage.
• the processing device 200 calculates the parameters based on the relationship between the calibration image data generated by the imaging device 100 photographing the calibration subject illuminated by light from the light source 50 and the parameters that define the illumination conditions. A prescribed range may be determined.
• the processing device 200 causes the adjustment device to generate calibration image data in the imaging device 100 while changing parameters that define illumination conditions, and changes the spectral shape of the calibration subject specified based on the calibration image data.
• a range of parameters whose amount is smaller than a predetermined amount may be determined as the specified range.
• the processing device 200 may perform the following operations before photographing the subject. - Acquire calibration image data generated by the imaging device 100 photographing a calibration subject illuminated by light from the light source 50. - Determine whether the pixel values of a plurality of pixels in the calibration image data satisfy the above-mentioned predetermined conditions. - If a predetermined condition is satisfied, spectrum data of the subject for calibration is generated based on the image data for calibration, and is stored in the storage device. - If the predetermined conditions are not met, the parameters that define the illumination conditions are changed within the specified range.
• spectrum data of a calibration object (for example, a white board) can be acquired under good lighting conditions.
• the spectral data of the calibration object is used in the process of generating images of each of four or more bands based on the compressed image.
• the processing device 200 instructs the user to change the illumination condition instead of changing the illumination condition by the light source 50.
• a warning prompting the user may be output on a display device or an audio output device.
• a warning may be displayed on a display device or an audio output device to prompt you to change the distance between the light source 50 and the subject or the brightness of the light source 50, or to replace the ND filter inserted between the light source 50 and the subject. You may also output it.
• Such functionality could prompt the user to manually change the lighting conditions.
• a processing method is executed by one or more processors that execute a plurality of instructions recorded in one or more memories.
• the method includes an imaging apparatus 100 that generates image data by photographing a subject illuminated with light from a light source 50, wherein the image data includes image information of each of four or more bands, or image information of each of the four or more bands.
• the image data including the information of the compressed image obtained by compressing the image information of the above bands as one image is acquired from the imaging device 100, and the pixel values of the plurality of pixels in the image data satisfy a predetermined condition. and if the condition is not met, changing the illumination conditions by the light source 50 under conditions where the spectral shape of the light from the light source 50 at the position of the subject does not change. and, including.
• FIG. 3 is a diagram schematically illustrating an example configuration of an imaging system 1000 according to an exemplary embodiment of the present disclosure.
• the imaging system 1000 includes an imaging device 100, an illumination device 120, an adjustment device 130, a processing device 200, a stage 190, and a support 150.
• Stage 190 has a flat support surface (an upper surface in this example), on which object 270 is placed.
• the support body 150 has a structure that is fixed to the stage 190 and extends in a direction perpendicular to the support surface of the stage 190.
• the support body 150 supports the imaging device 100, the illumination device 120, and the adjustment device 130.
• the imaging device 100 includes an image sensor 160.
• the lighting device 120 includes two light sources 122.
• Adjustment device 130 includes a mechanism for adjusting the distance between light source 122 and subject 270.
• Processing device 200 is a computer that includes one or more processors and one or more memories. The processing device 200 controls the adjustment device 130 based on image data output from the imaging device 100.
• the adjustment device 130 in the example of FIG. 3 has a mechanism that moves each of the illumination device 120 and the imaging device 100 in a direction perpendicular to the support surface of the stage 190 (hereinafter sometimes referred to as the “height direction”). Be prepared. Adjustment device 130 may include an actuator (eg, a linear actuator) that includes one or more motors. The actuator may be configured to change the distance between the light source 122 and the subject 270 using, for example, an electric motor, hydraulic pressure, or pneumatic pressure. In the example shown in FIG. 3, the adjustment device 130 can change the position of not only the illumination device 120 but also the imaging device 100. The adjustment device 130 does not need to include a mechanism for changing the position of the imaging device 100. The adjustment device 130 also includes a measuring device that measures the distance between the stage 190 and the light source 122.
• an actuator eg, a linear actuator
• the actuator may be configured to change the distance between the light source 122 and the subject 270 using, for example, an electric motor, hydraulic pressure, or pneumatic pressure.
• the support body 150 in the example of FIG. 3 is provided with a scale indicating the height from the support surface of the stage 190. Based on the scale, the positions of the imaging device 100 and the light source 122 in the height direction can be known.
• the imaging device 100 may be, for example, a camera that generates image data including information on images of each of four or more bands.
• the imaging device 100 may be configured to generate a hyperspectral image that includes image information for each of ten or more bands, for example.
• the imaging device 100 may be a camera that generates image data that includes compressed image information in which image information of each of four or more bands is compressed into one image, as disclosed in Patent Document 2. There may be.
• Image data of each of four or more bands can be restored from compressed image data by the restoration process based on compressed sensing. A specific example of the restoration process based on compressed sensing will be described later.
• the imaging device 100 photographs a subject 270 illuminated with light from the light source 122, and compresses image information of each of four or more bands, or compresses image information of four or more bands into one image. Generate image data that includes image information.
• the imaging device 100 may acquire still images or moving images. A specific example of the configuration of the imaging device 100 will be described later.
• the illumination device 120 is a device that includes at least one light source 122 and illuminates the subject 270 on the stage 190.
• the light emitted from the light source 122 can be, for example, visible light, infrared light, or ultraviolet light.
• "light” refers to electromagnetic waves that include not only visible light but also infrared and ultraviolet light.
• two light sources 122 are arranged on both sides of the imaging device 100, but the number and arrangement of the light sources 122 are not limited to this example and can be changed as appropriate.
• the processing device 200 has a function as a controller that controls the light source 122, the imaging device 100, and the adjustment device 130.
• the processing device 200 instructs the light source 122 to turn on, and instructs the imaging device 100 to take a picture.
• the processing device 200 further causes the adjustment device 130 to adjust the distance between the light source 122 and the subject 270 based on the image data of the subject 270 output from the imaging device 100. Specifically, the processing device 200 first determines whether the pixel values of a plurality of pixels in the image data satisfy a predetermined condition. If the condition is not met, the processing device 200 causes the adjustment device 130 to change the distance between the light source 122 and the subject 270 within a specified range.
• the processing device 200 repeats the operation of causing the adjustment device 130 to change the distance between the light source 122 and the subject 270 by a certain distance within a specified range until the condition is met. In this manner, in this embodiment, the processing device 200 changes the illumination conditions by the light source 122 by changing the distance between the light source 122 and the subject 270.
• the specified range is determined based on calibration image data obtained by the imaging device 100 photographing a calibration subject such as a white board.
• the processing device 200 determines the specified range based on the relationship between the calibration image data and the distance between the light source 122 and the calibration subject. Specifically, the processing device 200 causes the adjustment device 130 to cause the imaging device 100 to generate calibration image data while changing the distance between the light source 122 and the calibration subject. For example, the processing device 200 determines, as the prescribed range, a range of distances in which the spectral shape of the calibration subject identified based on the calibration image data can be considered constant.
• the processing device 200 may determine, as the specified range, a distance range in which the spectral shape of the calibration subject can be considered constant and the luminance falls within a predetermined range.
• the processing device 200 stores the determined specified range in a storage device such as a memory within the processing device 200, for example.
• FIG. 4A is a flowchart illustrating an example of a method for generating a hyperspectral image using the imaging system 1000 in this embodiment.
• the imaging device 100 generates a compressed image in which image information of a plurality of bands (for example, 10 to 100 or more) constituting a hyperspectral image is compressed into one image.
• the imaging device 100 generates a compressed image by photographing through a filter array including a plurality of optical filters.
• Each of the plurality of optical filters has an individual spectral transmittance, and the spectral transmittance of each filter has a maximum value at a plurality of wavelengths.
• Spectral transmittance indicates the wavelength dependence of transmittance and is also referred to as transmission spectrum.
• the processing device 200 restores a hyperspectral image based on the generated compressed image, data on the spectral transmittance of each filter in the filter array, and data on the spectrum of light from the light source 122 at the position of the subject.
• data on the spectrum of light from the light source 122 at the position of the subject is obtained by photographing a white board, which is a calibration subject.
• the method shown in FIG. 4A includes step S100 of acquiring spectral data of a white board, which is a subject for calibration, and step S200 of generating a hyperspectral image of the subject using the spectral data of the white board.
• Step S100 includes steps S101 to S107.
• Step S200 includes steps S201 to S206.
• step S101 the imaging device 100 and the light source 122 are placed at an initial position a predetermined distance away from the white board placed on the stage 190.
• the arrangement of the imaging device 100 and the light source 122 to the initial position may be performed by the adjustment device 130 under the control of the processing device 200. Alternatively, the user may manually place the imaging device 100 and the light source 122 at the initial position.
• step S102 the imaging parameters of the imaging device 100 are determined. Specifically, parameters such as exposure time and gain that affect the brightness of the acquired image are determined. These parameters may be set, for example, according to input from the user, or may be automatically set to appropriate values.
• step S103 the imaging device 100 photographs the white board placed on the stage 190 to obtain a compressed image. Photographing may be performed, for example, according to a user's operation or an instruction from the processing device 200.
• step S104 the processing device 200 determines whether the pixel value of each of the plurality of pixels included in the obtained compressed image is within a predetermined range.
• the predetermined range is a range in which a good hyperspectral image can be restored based on the compressed image, and is recorded in advance in a storage device such as a memory of the processing device 200.
• the upper limit of the predetermined range may be determined, for example, based on the upper limit of the amount of light received by the photodetecting element of the image sensor. When the upper limit is exceeded, the pixel value becomes a constant value, and a difference in pixel values that exceeds the upper limit cannot be detected. In that case, it becomes difficult to obtain a correct restored image for the portion with pixel values above the upper limit.
• the lower limit of the predetermined range is the lower limit of pixel values at which a restored image with small errors can be obtained by the restoration calculation.
• the lower the pixel value is, the greater the influence of noise becomes, making it difficult to obtain an accurate restored image through restoration calculations.
• the upper limit of the predetermined range may be set, for example, to a value close to 100% of the maximum pixel value, and the lower limit may be set, for example, to a value from 20% to 50% or more of the maximum pixel value.
• the processing device 200 may determine whether the pixel values of all pixels included in the compressed image are within a predetermined range. Alternatively, the processing device 200 determines whether the pixel values of a predetermined percentage (for example, 80%, 70%, or 60%) of the pixels included in the compressed image are within a predetermined range. You may. Such a determination makes it possible to avoid a situation where the processing does not proceed further when the pixel values of a small number of pixels in the compressed image are outside a predetermined range.
• a predetermined percentage for example, 80%, 70%, or 60%
• step S104 determines whether the determination in step S104 is No. If the determination in step S104 is Yes, the process advances to step S106.
• step S105 the processing device 200 causes the adjustment device 130 to change the distance between the light source 122 and the white board within a specified range without changing the direction (that is, the angle) of the light source 122.
• the processing device 200 causes the adjustment device 130 to change the position of the light source 122 in the height direction upward or downward by a preset unit length.
• the process returns to step S103, and the compressed image of the white board is again acquired by the imaging device 100.
• the operations from steps S103 to S105 are repeated until it is determined Yes in step S104.
• the processing device 200 may return the light source 122 to the initial position and perform the same operation while moving the light source 122 by unit length in the opposite direction. If the determination in step S104 is not YES at any position within the specified range, the processing device 200 may stop the operation of the adjustment device 130 and output a warning to an external device such as a display.
• the processing device 200 changes the height of the light source 122 within a specified range, but if the determination in step S104 does not become Yes even if the height of the light source 122 reaches the upper or lower limit of the specified range, the height of the light source 122 is changed beyond the specified range.
• the light source 122 may be moved. If the determination in step S104 is Yes at a position where the light source 122 is outside the specified range, the processing device 200 may output a warning including information indicating the position.
• step S104 If it is determined in step S104 that the pixel value of each of the plurality of pixels included in the compressed image of the white board is within a predetermined range, the process proceeds to step S106.
• step S106 the processing device 200 restores the hyperspectral image from the compressed image of the white board. Details of the restoration process will be described later.
• This provides spectrum data of the white board.
• the spectrum data of the white board may be data indicating the reflected light intensity of each pixel in each image of a plurality of bands.
• the processing device 200 stores the spectrum data of the white board in the storage device.
• the storage device may be any storage device, such as a memory within the processing device 200 or a storage external to the processing device 200.
• spectrum data of the white board can be acquired.
• the spectral data of the white board is used to remove the influence of background light in the subsequent operation of acquiring a hyperspectral image of the subject. Note that although a white board is used as the calibration subject in this embodiment, a calibration subject other than the white board may be used.
• the distance between the light source 122 and the subject is adjusted within a specified range that is narrower than the movable range of the light source 122 by the adjustment device 130.
• a range in which the spectral shape of light at the position of the subject does not change significantly is determined in advance. Changes in the spectral shape can be evaluated based on the angle or inner product between vectors, assuming that the spectrum is a vector with dimensions equal to the number of wavelength band divisions N. For example, if the angle between two N-dimensional vectors representing spectra is less than a threshold value, the shapes of the spectra can be treated as being the same. A specific example of the method for determining the specified range will be described later.
• the spectral data stored in step S107 can be used in common without being changed. That is, when changing the distance between the light source and the subject within a specified range, there is no need to acquire spectrum data of the white board again. In contrast, in the conventional method, when the distance between the light source and the subject is changed, it is necessary to acquire spectrum data of the white board again.
• the work of reacquiring spectrum data of the white board (or other calibration object) is omitted. can. Thereby, it is possible to more efficiently acquire a hyperspectral image of the subject.
• step S200 the operation of acquiring a hyperspectral (HS) image of a subject in step S200 will be described.
• HS hyperspectral
• an operation can be performed to acquire a hyperspectral image of the object to be inspected.
• the brightness of the subject is adjusted. This is because objects have unique colors and reflection spectra vary depending on the object, so even if the brightness is appropriate for a white board, it may be too dark for the object.
• step S201 a subject to be inspected is placed on the stage 190. If a white board is placed on the stage 190, the subject is placed after removing the white board.
• step S202 the imaging device 100 photographs the subject and obtains a compressed image. Photographing may be performed, for example, according to a user's operation or an instruction from the processing device 200.
• step S203 the processing device 200 determines whether the pixel values of a plurality of pixels included in the obtained compressed image are within a predetermined range.
• the predetermined range is set in advance as a range in which a good hyperspectral image can be restored based on the compressed image.
• the processing device 200 determines that the pixel values of not all pixels included in the compressed image but a predetermined percentage (for example, 80%, 70%, or 60%) of the pixels are within a predetermined range. It may also be determined whether the pixel values of a plurality of pixels included in the obtained compressed image are within a predetermined range.
• a predetermined percentage for example, 80%, 70%, or 60%
• step S203 determines whether the determination in step S203 is No. If the determination in step S203 is Yes, the brightness adjustment is finished and the process proceeds to step S205.
• step S204 the processing device 200 causes the adjustment device 130 to change the distance between the light source 122 and the subject within a specified range.
• This operation is similar to the operation in step S105.
• the processing device 200 causes the adjustment device 130 to change the position of the light source 122 in the height direction upward or downward by a preset unit length.
• the process returns to step S202, and the compressed image of the subject is again acquired by the imaging device 100.
• the operations from steps S202 to S204 are repeated until it is determined Yes in step S203. Note that as a result of the light source 122 moving upward or downward by a unit length from the initial position, the upper or lower limit of the specified range may be reached.
• the processing device 200 may return the light source 122 to the initial position and perform the same operation while moving the light source 122 by unit length in the opposite direction. If the determination in step S203 is not YES at any position within the specified range, the processing device 200 may stop the operation of the adjustment device 130 and output a warning to an external device such as a display.
• the warning may include a message recommending starting over from the operation of acquiring spectrum data of the white board in step S100.
• step S203 determines whether the determination in step S203 is Yes within the specified range. If the determination in step S203 is not Yes within the specified range, the processing device 200 changes the control parameters such as the current or voltage for driving the light source 122 within a range in which the spectral shape does not change. Adjustments may be made so that the determination in step S203 is Yes.
• step S203 If it is determined in step S203 that the pixel value of the pixel included in the compressed image of the subject is within a predetermined range, the process advances to step S205.
• step S205 the processing device 200 restores a hyperspectral image from the compressed image of the subject. This process is similar to the process in step S106, and details of the restoration process will be described later.
• step S206 the processing device 200 reads the white board spectral data stored in step S107 from the storage device, and calculates the value of each pixel of each band in the hyperspectral image restored in step S205 by comparing the values of each pixel in each band with the corresponding values in the white board spectral data. Divide by the value. As a result, a hyperspectral image in which the influence of the luminance distribution of light from the light source 122 is removed is obtained. Processing device 200 stores this hyperspectral image in a storage device. The processing device 200 may display this hyperspectral image on a display.
• a hyperspectral image of the subject can be efficiently acquired.
• the conventional method it was necessary to obtain spectrum data of the white board each time the power of the light emitted from the light source 122 or the direction or position of the light source 122 was changed in order to adjust the brightness at the position of the subject.
• Such work is extremely troublesome, and improvements have been required.
• the thickness of the subject and the white board are generally different, so in order to make the distance from the light source the same, it is necessary to make adjustments to cancel the difference in thickness each time the subject and the white board are changed.
• step S203 if the determination is No in step S203, the distance between the light source and the subject is changed within a specified range in which the spectral shape does not change, so the task is to acquire the spectral data of the white board anew. can be omitted. Therefore, compared to conventional methods, it is possible to efficiently acquire a hyperspectral image of a subject.
• step S206 it is possible to continue acquiring the hyperspectral image of the subject without acquiring the spectrum data of the white board again. For example, it is also possible to generate a hyperspectral moving image by continuously capturing images.
• a hyperspectral image of the subject is generated by performing the operations included in step S200 after acquiring spectrum data of the white board through the operations included in step S100.
• a hyperspectral image of the subject may be generated by first performing the operations in step S200 except for step S206, and then spectrum data of the white board may be acquired by the operations included in step S100.
• the final hyperspectral image may then be generated by dividing each pixel value of each band image in the hyperspectral image by the corresponding value in the spectral data of the white board. Even in that case, the distance between the light source and the subject and the distance between the light source and the white board are changed within a specified range in which the spectral shape at the subject position does not change.
• FIG. 4B is a flowchart showing a modification of the method shown in FIG. 4A. In the example shown in FIG. 4B, whether the light amount is appropriate is evaluated based on the hyperspectral image rather than the compressed image.
• the method shown in FIG. 4B includes step S150 of acquiring spectral data of the white board, and step S250 of acquiring a hyperspectral image of the subject.
• step S150 and step S100 in FIG. 4A is that step S106 is performed after step S103, and step S154 is performed after step S106 instead of step S104.
• step S250 and step S200 in FIG. 4A is that step S205 is performed after step S202, and step S253 is performed after step S205 instead of step S203.
• steps S101 to S103 in the example of FIG. 4B are the same as the example of FIG. 4A.
• step S103 the process proceeds to step S106, and the processing device 200 restores a hyperspectral image from the compressed image of the white board.
• the restoration method is the same as the example shown in FIG. 4A, and details will be described later.
• the processing device 200 determines whether the pixel value of each of the plurality of pixels included in the hyperspectral image of the white board is within a predetermined range.
• This processing may be performed for each image of a plurality of bands that make up the hyperspectral image. That is, when a hyperspectral image includes a first image corresponding to the first band and a second image corresponding to the second band, the pixel value of each of the plurality of pixels included in the first image is set to a predetermined value. It can be determined whether the pixel value is within the range, and whether the pixel value of each of the plurality of pixels included in the second image is within the predetermined range.
• the predetermined range may be different for each band, that is, each wavelength range.
• the processing device 200 may determine whether the pixel values of all pixels in the image of each band included in the hyperspectral image of the white board are within a predetermined range. Alternatively, the processing device 200 may cause the pixel values of a predetermined percentage (for example, 80%, 70%, or 60%, etc.) of the pixels in each band image included in the hyperspectral image to fall within a predetermined range. It may be determined whether or not there is one. Alternatively, it may be determined whether the pixel values of some or all of the pixels in the image of some bands instead of all the bands included in the hyperspectral image are within a predetermined range.
• a predetermined percentage for example, 80%, 70%, or 60%, etc.
• step S154 determines whether the determination in step S154 is No. If the determination in step S154 is No, the process proceeds to step S105, and the distance between the light source 122 and the white board is changed within the specified range. After that, the process returns to step S103, and the compressed image of the white board is acquired again. The operations in steps S103, S106, S154, and S105 are repeated until it is determined Yes in step S154.
• step S154 If it is determined Yes in step S154, the process proceeds to step S107, and the processing device 200 stores the spectrum data of the white board in the storage device.
• step S107 it becomes possible to acquire a hyperspectral image of the subject in step S250.
• steps S201 and S202 are the same as the example in FIG. 4A.
• the process proceeds to step S205, and the processing device 200 restores a hyperspectral image from the compressed image of the subject.
• the processing device 200 determines whether the pixel value of each of the plurality of pixels included in the hyperspectral image of the subject is within a predetermined range. This processing may be performed for each image of a plurality of bands that make up the hyperspectral image. The predetermined range may be different for each band, that is, each wavelength range. This process is similar to the process in step S154. Similarly to step S154, the processing device 200 processes not all pixels in the image of each band included in the hyperspectral image, but a predetermined percentage (for example, 80%, 70%, or 60%) of the pixels. It may be determined whether the value is within a predetermined range. Alternatively, it may be determined whether the pixel values of some or all of the pixels in the image of some bands instead of all the bands included in the hyperspectral image are within a predetermined range.
• a predetermined percentage for example, 80%, 70%, or 60%
• step S253 If the determination in step S253 is No, the process proceeds to step S204, and the distance between the light source 122 and the subject is changed within the specified range. Thereafter, the process returns to step S202, and a compressed image of the subject is acquired again. The operations of steps S202, S205, S253, and S204 are repeated until it is determined Yes in step S253.
• step S253 If it is determined Yes in step S253, the process proceeds to step S206, and the processing device 200 divides the value of each pixel in each band in the hyperspectral image of the subject by the corresponding value in the spectrum data of the white board. As a result, a hyperspectral image in which the influence of the luminance distribution of light from the light source 122 is removed is obtained.
• step S150 and step S250 may be reversed. That is, a hyperspectral image of the subject may be generated by first performing the operations in step S250 except for step S206, and then spectral data of the white board may be acquired by the operations included in step S100. Then, the final hyperspectral image may be generated by dividing each pixel value of each band image in the hyperspectral image of the subject by the corresponding value in the spectral data of the white board.
• step S106 in the example shown in FIG. 4A
• step S154 shown in FIG.
• the process may proceed to S107.
• step S205 in the example shown in FIG. 4A
• step S253 shown in FIG. 4B is performed, and if it is determined No in step S253, the process proceeds to step S204, and if it is determined Yes in step S253, the operation is performed in step S253 shown in FIG. 4B.
• the process may proceed to step S206.
• the processing device 200 may determine whether the amount of light or the brightness is appropriate based on at least one of the compressed image and the hyperspectral image. That is, the processing device 200 may determine whether the amount of light or brightness is appropriate based on the compressed image, the hyperspectral image, or the compressed image and the hyperspectral image.
• step S203 or S253 it is determined whether the pixel value of each of the plurality of pixels included in the image of each band in the compressed image or hyperspectral image is within a predetermined range. In place of or in addition to this determination, a determination may be made based on the contrast of the image. For example, when the contrast value calculated from the pixel values of a plurality of pixels in the image of each band in the compressed image or the hyperspectral image exceeds a threshold value, the processing device 200 processes each of the images of each band in the hyperspectral image of the subject. A process of dividing the pixel value by a corresponding value in the spectrum data of the white board and outputting the result may be performed.
• the contrast value is an index value that represents the degree of spread of pixel values in a histogram that represents the relationship between pixel values and the frequency of pixel values.
• the contrast value can be quantitatively determined based on, for example, the half-width in the histogram of the image, the difference between the maximum pixel value and the minimum pixel value, the variance, or the standard deviation.
• a contrast value may be calculated in step S203 in FIG. 4A or step S253 in FIG. 4B. If the contrast value does not exceed the threshold value, the process proceeds to step S204, and the distance between the light source 122 and the white board may be adjusted within a specified range.
• the contrast for each wavelength band of the plurality of pixels included in the compressed image corresponds to randomness in encoding for each wavelength band. Therefore, by increasing the contrast, encoding performance can be improved and solution convergence can be improved. Therefore, when determining whether or not a predetermined condition is satisfied for image data including compressed image information, images of each band are restored from the compressed image by imposing a condition that the contrast value exceeds a threshold value. Restoration errors in processing can be reduced. This makes it possible to generate even better hyperspectral images.
• the imaging device 100 generates a compressed image
• the processing device 200 restores a hyperspectral image from the compressed image.
• the imaging device 100 itself may be configured to generate a hyperspectral image. That is, the imaging device 100 may be a hyperspectral camera. In that case, the imaging device 100 is not limited to a camera that generates a hyperspectral image through processing based on compressed sensing using the filter array described above. Imaging device 100 may be a camera that generates hyperspectral images in other ways.
• a hyperspectral image can be obtained, for example, by imaging using a spectroscopic element such as a prism or a grating.
• a hyperspectral image can be obtained by separating light from an object into bands using a prism or grating and detecting the separated light band by band.
• a hyperspectral image of the white board may be acquired by the imaging device 100 instead of the operations of steps S103 and S106 shown in FIG. 4B.
• the imaging device 100 may acquire a hyperspectral image of the subject.
• the prescribed range can be determined, for example, by searching for the upper and lower limits of a range in which the change in the spectral shape of the light from the light source 122 is sufficiently small.
• the defined range is determined before the operation shown in FIG. 4A or 4B is performed.
• FIG. 5 is a flowchart illustrating an example of a method for determining the specified range.
• the imaging device 100 acquires a compressed image from which a hyperspectral image can be restored, as in the example of FIG. 4A.
• Processing device 200 restores a hyperspectral image based on the compressed image.
• the method shown in FIG. 5 includes operations from steps S301 to S316. The operation of each step will be explained below.
• step S301 the imaging device 100 and the light source 122 are placed at a position a predetermined distance from the white board (hereinafter referred to as the "initial position"). This operation may be performed manually by the user, or may be performed by the adjustment device 130 based on a command from the processing device 200.
• step S302 the processing device 200 causes the imaging device 100 to perform photographing and obtain a compressed image of the white board.
• step S303 the processing device 200 restores the hyperspectral image from the compressed image acquired in step S302. This restoration process is similar to the process in step S106 shown in FIG. 4A.
• step S304 the processing device 200 causes the adjustment device 130 to increase the position of the light source 122 in the height direction by a predetermined amount (for example, 0.5 mm, 1 cm, 2 cm, etc.). As a result, the distance between the light source 122 and the white board increases by a predetermined amount (also referred to as "unit length").
• a predetermined amount for example, 0.5 mm, 1 cm, 2 cm, etc.
• step S305 the processing device 200 causes the imaging device 100 to perform photography again and obtain a compressed image of the white board.
• step S306 the processing device 200 restores the hyperspectral image from the compressed image acquired in step S305.
• This restoration process is also similar to the process in step S106 shown in FIG. 4A.
• step S307 the processing device 200 compares the hyperspectral image restored in step S303 and the hyperspectral image restored in step S306, and compares the state in which the light source 122 is at the initial position and the position after the change. It is determined whether there is a change in the spectral shape depending on the time.
• Whether there is a change in the spectral shape can be determined using, for example, the Spectral Angle Mapper (SAM) method.
• SAM Spectral Angle Mapper
• An example of a method for determining the presence or absence of a change in spectral shape using the SAM method will be described below.
• FIG. 6A is a diagram illustrating an example of the data structure of a hyperspectral image.
• the hyperspectral image is expressed as a collection of N images 20W 1 , 20W 2 , . . . , 20W N.
• Hyperspectral image data having such a structure is called a "hyperspectral data cube.”
• N is the total number of wavelength bands included in the target wavelength range, and is an integer of 4 or more.
• the k-th image 20W k corresponds to the k-th wavelength band ⁇ k .
• the center wavelength ⁇ k of the k-th wavelength band is used as a reference symbol representing the k-th wavelength band.
• the pixel value be pk ij .
• the hyperspectral image can be expressed by the following N n ⁇ m matrices.
• the hyperspectral image is not limited to a three-dimensional array data structure as shown in FIG. 6A, but also has a two-dimensional array data structure as shown in FIG. 6B, or a one-dimensional array data structure as shown in FIG. 6C. You can leave it there.
• FIG. 6B information on images of N wavelength bands are arranged in the horizontal direction, and pixel values of n ⁇ m pixels in the images of each wavelength band are arranged in the vertical direction.
• the pixel values of all pixels of images of all wavelength bands are arranged in one line. In this way, the data structure of the hyperspectral image is arbitrary.
• a hyperspectral image can be thought of as an image having pixel values for each of the N wavelength bands for each pixel.
• the pixel value of each pixel can be expressed as an N-dimensional vector.
• the N-dimensional vector is a vector having pixel values of each of N bands as components.
• the pixel value of the pixel in the i-th row and j-th column can be expressed by the following N-dimensional vector.
• a change in the spectrum at a certain pixel can be evaluated based on the angle between the vector before the change and the vector after the change at the pixel.
• the vectors before and after the change of the pixel in the i-th row and j-th column are the following vectors a ij and b ij , respectively.
• p1k ij represents the pixel value before the change in the k-th band of the pixel in the i-th row and j-th column
• p2k ij represents the pixel value after the change in the k-th band of the pixel in the i-th row and j-th column.
• the angle formed by vector a ij and vector b ij is represented by ⁇ ij below.
• the processing device 200 calculates the integrated value T or It may be determined that the spectrum shape has changed when the average value A is greater than or equal to a threshold value.
• the integrated value T and the average value A are expressed by the following formulas.
• the processing device 200 may determine that the spectral shape has changed when the absolute value of the spectral angle at one or more representative pixels in the hyperspectral image is greater than or equal to a threshold value.
• the threshold may be set to a value such as 1°, 3°, or 5°.
• the threshold value is set to an appropriate value depending on the purpose or use.
• step S307 If it is determined in step S307 that there is no change in the spectrum shape, the process proceeds to step S308.
• step S308 the processing device 200 stores the changed height position of the light source 122 in the storage device as being within the specified range. Thereafter, the process returns to step S304, and the processing device 200 causes the adjustment device 130 to increase the height position of the light source 122 by a predetermined amount again. Thereafter, the acquisition of the compressed image in step S305, the restoration of the hyperspectral image in step S306, and the determination in step S307 are performed again. The operations from steps S304 to S308 are repeated until it is determined in step S307 that there has been a change in the spectrum shape. If it is determined that there has been a change in the spectrum shape, the process advances to step S309.
• step S309 the processing device 200 causes the storage device to store the position in the height direction of the light source 122 immediately before the position is changed in step S304 as the upper limit of the specified range.
• step S309 the processing device 200 performs the same operation as above while moving the light source 122 downward from the initial position.
• step S310 the processing device 200 causes the adjustment device 130 to move the light source 122 to the initial position.
• step S311 the processing device 200 causes the adjustment device 130 to reduce the position of the light source 122 in the height direction by a predetermined amount. This reduces the distance between the light source 122 and the white board by a predetermined amount (ie, unit length).
• step S312 the processing device 200 causes the imaging device 100 to perform photographing and obtain a compressed image of the white board.
• step S313 the processing device 200 restores the hyperspectral image from the compressed image acquired in step S312.
• step S314 the processing device 200 compares the hyperspectral image restored in step S303 with the hyperspectral image restored in step S313, and compares the state in which the light source 122 is at the initial position and the position after the change. It is determined whether there is a change in the spectral shape depending on the time.
• the determination method is the same as the method in step S307.
• step S314 If it is determined in step S314 that there is no change in the spectrum shape, the process advances to step S315.
• step S315 the processing device 200 stores the changed height position of the light source 122 in the storage device as being within the specified range. Thereafter, the process returns to step S311, and the processing device 200 causes the adjustment device 130 to reduce the height position of the light source 122 by a predetermined amount again. Thereafter, the acquisition of the compressed image in step S312, the restoration of the hyperspectral image in step S313, and the determination in step S314 are performed again. The operations from steps S311 to S315 are repeated until it is determined in step S314 that there has been a change in the spectrum shape. If it is determined that there has been a change in the spectrum shape, the process advances to step S316.
• step S316 the processing device 200 causes the storage device to store the position in the height direction of the light source 122 immediately before the position is changed in step S311 as the lower limit of the specified range.
• the processing device 200 does not proceed to step S309, and even if the initial position of the light source 122 is changed and the processing shown in FIG. 5 is performed. good.
• the processing device 200 repeats the process of generating a hyperspectral image while changing the height direction position of the light source 122, and the spectral shape changes. You may also search for the lower and upper limits of the distance range.
• the processing device 200 first determines the upper limit of the specified range and then determines the lower limit, but conversely, it may first set the lower limit and then determine the upper limit. In that case, the order of the operations from steps S304 to S309 and the operations from steps S311 to S316 will be reversed.
• the imaging device 100 acquires a compressed image, and the processing device 200 generates a hyperspectral image based on the compressed image.
• the present invention is not limited to such a configuration; for example, the imaging device 100 may be configured to generate a hyperspectral image by itself. An example of a method for determining the prescribed range in this case will be described below with reference to FIG.
• FIG. 7 is a flowchart illustrating an example of a method for determining the prescribed range of the distance between the light source 122 and the subject when the imaging device 100 acquires a hyperspectral image.
• the flowchart shown in FIG. 7 differs from the flowchart shown in FIG. 5 in that steps S302 and S303 are replaced with step S352, steps S305 and S306 are replaced with step S355, and steps S312 and S313 are replaced with step S362.
• steps S352, S355, and S362 the imaging device acquires a hyperspectral image instead of a compressed image. Except for this point, the operation shown in FIG. 7 is the same as the operation shown in FIG. 5.
• an appropriate prescribed range can be determined.
• FIG. 8 is a block diagram illustrating an example configuration of an imaging system 1000 that executes the above method.
• An imaging system 1000 shown in FIG. 8 includes an imaging device 100, a lighting device 120, an adjustment device 130, a processing device 200, and a display device 300.
• Illumination device 120 includes one or more light sources 122.
• the adjustment device 130 includes a mechanism such as an actuator that adjusts the distance between the light source 122 and the subject.
• the display device 300 is, for example, a display (monitor) such as a liquid crystal or an organic LED (OLED), and displays the results of processing by the processing device 200.
• the imaging device 100 in this example is a camera that generates a compressed image that is a source for restoring a hyperspectral image.
• the processing device 200 restores a hyperspectral image from the compressed image output from the imaging device 100.
• the processing device 200 may be directly connected to the imaging device 100, the lighting device 120, the adjustment device 130, and the display device 300, or may be indirectly connected to them via a wired and/or wireless network. Good too.
• the functions of the processing device 200 may be distributed among multiple devices. For example, at least some of the functions of the processing device 200 may be performed by an external computer such as a cloud server.
• the processing device 200 shown in FIG. 8 includes a light source controller 202, an imaging controller 204, a first processing circuit 212, a second processing circuit 214, and a memory 216.
• the light source controller 202 controls turning on, turning off, and light emission intensity of the light source 122 in the lighting device 120.
• the imaging controller 204 controls the operation of the imaging device 100.
• the imaging controller 204 sets, for example, the exposure time and gain of the imaging device 100.
• the first processing circuit 212 determines whether the brightness of the subject is appropriate based on the image data output from the imaging device 100, and controls the adjustment device 130 based on the determination result.
• the adjustment device 130 adjusts the brightness of the subject to an appropriate range by changing the position of the light source 122 in accordance with control from the first processing circuit 212.
• the first processing circuit 212 also causes the display device 300 to display processing results such as a hyperspectral image generated based on the compressed image.
• the second processing circuit 214 performs restoration processing based on the compressed image and generates a hyperspectral image.
• the second processing circuit 214 is a circuit independent of the first processing circuit 212, but both may be realized by one circuit.
• One such circuit may also function as light source controller 202 and imaging controller 204.
• the memory 216 is a storage device that stores computer programs executed by the first processing circuit 212 and the second processing circuit 214 and various data generated in the process of processing.
• the memory 216 stores data indicating the above-mentioned prescribed distance range, data indicating the spectral transmittance of the filter array in the imaging device 100, spectral data indicating a hyperspectral image of the white board, and the like.
• parameters of the imaging device 100 are first set from an external input device via the input interface (I/F) 221.
• Parameters include, for example, exposure time and gain.
• the parameters can be set, for example, in response to a user's operation using an input device.
• a white board was set up on the stage and filming began.
• Photography may be performed in response to an instruction from an external input device via the input interface 221.
• compressed image data is output from the imaging device 100.
• the compressed image data is sent to a first processing circuit 212 and a second processing circuit 214.
• the second processing circuit 214 generates a hyperspectral image based on the compressed image.
• the generated hyperspectral image is sent to the first processing circuit 212.
• information indicating the distance between the light source 122 and the white board is sent from the adjustment device 130 to the first processing circuit 212 via the input interface 222 .
• the first processing circuit 212 evaluates the brightness distribution of the white board based on the input distance information and the compressed image and/or the hyperspectral image, for example, through the process of step S104 shown in FIG. 4A or step S154 shown in FIG. 4B. .
• the first processing circuit 212 causes the adjustment device 130 to change the distance between the light source 122 and the white board so that the brightness distribution is optimized.
• the first processing circuit 212 sends control signals to the regulating device 130 via the interface 223 .
• Adjustment device 130 changes the position of light source 122 within a preset prescribed range in response to the control signal. Thereby, the optimum distance between the light source 122 and the white board or the optimum position of the light source 122 is determined.
• a hyperspectral image of the whiteboard is acquired when the distance between the light source 122 and the whiteboard is at the optimal distance, it becomes possible to acquire a hyperspectral image of the subject.
• the user removes the white board from the stage and places the subject on the stage. The same operation as above is performed for the subject as well. That is, the distance between the light source 122 and the subject is adjusted so that the brightness is optimized. A compressed image is acquired with the optimal distance.
• the first processing circuit 212 may determine the optimal distance from the optimal values of the brightness of the white board and the subject based on the relational expression between distance and brightness.
• a white board was set up on the stage and filming began.
• Compressed image data is output from the imaging device 100 to the first processing circuit 212 and the second processing circuit 214.
• the second processing circuit 214 generates a hyperspectral image of the white board based on the compressed image.
• the hyperspectral image is sent to the first processing circuit 212.
• distance information between the light source 122 and the white board is sent from the adjustment device 130 to the first processing circuit 212 via the input interface 222.
• the first processing circuit 212 evaluates the change in the spectral shape from the spectral shape when the light source 122 is in the initial position based on the distance information and the compressed image and/or the hyperspectral image.
• the first processing circuit 212 determines the upper and lower limits of the specified range of distance within which the spectral shape does not change, using the procedure shown in FIG. 5 or 7.
• the position of the light source 122 is automatically adjusted by the processing device 200 and the adjustment device 130, but this adjustment may also be performed manually by the user.
• the imaging system may include a mechanism that can manually adjust the distance, instead of the adjustment device 130 that automatically adjusts the distance between the light source 122 and the white board.
• First processing circuit 212 may be configured to display the compressed image and/or hyperspectral image on display device 300. The user may visually judge the brightness of the displayed image and, if the brightness is not appropriate, manually adjust the distance between the light source 122 and the white board.
• the processing device 200 may display information on the prescribed range on the display device 300. If the pixel value of each of the plurality of pixels included in the compressed image or hyperspectral image of the white board or subject is not within the predetermined range, the user manually adjusts the pixel value so that it falls within the predetermined range displayed on the display device 300. The distance between the light source 122 and the white board or subject may be changed.
• the mechanism for manually adjusting the distance between the light source 122 and the subject may be designed in advance so that its movable range falls within a specified range. According to such a design, it is possible to avoid the distance from falling outside the specified range, making it easier to set the lighting conditions to a suitable state.
• the specified range may be determined in advance by the manufacturer of the imaging system, and instructions such as "Please perform white plate correction within this specified range" may be given in a manual or the like.
• the user can set the lighting conditions to the optimum state by adjusting the distance between the light source 122 and the subject within a specified range according to the instructions.
• These methods of manually adjusting illumination conditions are not limited to configurations in which the distance between the light source 122 and the subject is adjusted, but can be similarly applied to configurations in which illumination conditions are adjusted by adjusting other parameters described below. .
• the processing device 200 changes the illumination conditions by changing the distance between the light source 122 and the white board and the distance between the light source 122 and the subject, but the present invention is not limited thereto.
• lighting conditions may be changed by changing control parameters such as current or voltage for driving light source 122. Even in that case, the processing device 200 changes the illumination conditions so that the brightness distribution at the position of the subject does not change.
• FIG. 9 is a flowchart illustrating an example of a method for generating a hyperspectral image when illumination conditions are changed by changing control parameters for driving a light source.
• the control parameter is a current for driving the light source 122 (hereinafter sometimes referred to as "driving current").
• the control parameter is not limited to current, but may be a voltage for driving the light source 122.
• the control parameter may be the duty ratio of the PWM signal.
• the method shown in FIG. 9 includes step S170 of acquiring spectral data of the white board, and step S270 of acquiring a hyperspectral image of the subject.
• Step S170 differs from step S100 shown in FIG. 4A in that step S173 is added between step S102 and step S103, and step S105 is replaced with step S175.
• Step S270 differs from step S100 shown in FIG. 4A in that step S204 is replaced with step S274.
• the main points different from the example of FIG. 4A will be explained below.
• the illumination conditions are changed by changing the drive current of the light source 122 instead of changing the distance between the light source 122 and the subject.
• the processing device 200 sets the drive current of the light source 122 to an initial value. If the processing device 200 determines in step S104 that the pixel values of the plurality of pixels included in the compressed image of the white board do not satisfy the predetermined condition, the processing device 200 proceeds to step S175 and changes the drive current of the light source 122 within a specified range. . For example, the drive current is increased or decreased by a predetermined amount.
• step S203 determines in step S203 that the pixel values of the plurality of pixels included in the compressed image of the subject do not satisfy the predetermined condition
• the processing device 200 proceeds to step S274 and adjusts the drive current of the light source 122 within the specified range. Change with .
• the specified range in this example is a current range in which the shape of the spectrum of light from the light source 122 can be considered to be approximately constant at the position of the subject.
• Processing device 200 may be configured to vary the drive current by varying the voltage for driving light source 122, for example.
• the method shown in FIG. 9 can also efficiently acquire a hyperspectral image of a subject.
• the processing device 200 determines whether the illumination conditions are appropriate for the compressed image, but as in the example of FIG. 4B, the processing device 200 may also determine whether the illumination conditions are appropriate for the hyperspectral image. good. That is, even in the example of FIG. 9 in which control parameters such as the drive current or drive voltage of the light source 122 are adjusted, a modification similar to the modification from FIG. 4A to FIG. 4B is possible.
• FIG. 10 is a flowchart illustrating an example of a method for determining the specified range of the drive current of the light source 122.
• steps S301, S304, S308, S309, S310, S311, S315, and S316 in FIG. 5 are replaced with steps S501, S504, S508, S509, S510, S511, S515, and S516, respectively.
• the basic flow is the same as the example shown in FIG.
• the prescribed range is determined by increasing or decreasing the driving current of the light source 122 by a predetermined amount and determining whether the spectrum shape changes each time.
• the specified range of the drive current of the light source 122 is determined, but the specified range of other control parameters such as the drive voltage or the duty ratio of the PWM signal can also be determined in a similar manner.
• the illumination conditions may be changed by a method different from these methods under conditions where the spectral shape at the position of the subject does not change.
• the illumination conditions may be changed by changing a neutral density filter such as an ND filter that may be placed between the light source 122 and the subject (or a calibration subject such as a white board). In this case as well, the neutral density filter is changed so that the spectral shape at the subject position does not change before and after the change.
• the imaging system may include a mechanism 135 that inserts one neutral density filter selected from a plurality of neutral density filters 136 having different transmittances between the light source 122 and the subject.
• Mechanism 135 may include a device such as an actuator that inserts or removes each neutral density filter 136 from the optical path from light source 122 to the subject in response to commands from processing device 200.
• the processing device 200 causes the mechanism 135 to perform a dimming operation inserted between the light source 122 and the subject when the pixel values of a plurality of pixels in the compressed image or the hyperspectral image do not satisfy a predetermined condition.
• the filter 136 it may be configured to change the lighting conditions.
• Each of the plurality of neutral density filters 136 may be a filter whose transmittance has small wavelength dependence and small in-plane unevenness.
• the distance between the light source 122 and the subject (or the height of the light source 122) in the operation shown in FIG. 4A or 4B and FIG. 5 or 7 is changed. Instead of adjustment, the operation of replacing the neutral density filter 136 may be performed.
• the neutral density filter 136 may be changed automatically or manually.
• the processing device 200 changes the neutral density filter 136 to be a candidate for switching.
• the display device 300 may display information specifying the . This allows the user to know which neutral density filter to switch to.
• FIG. 12A is a diagram schematically showing a configuration example of the imaging device 100 that acquires a compressed image and an example of processing by the processing device 200.
• This imaging device 100 has the same configuration as the imaging device disclosed in Patent Document 2.
• the imaging device 100 includes an optical system 140, a filter array 110, and an image sensor 160.
• Optical system 140 and filter array 110 are arranged on the optical path of light incident from object 70, which is a subject.
• Filter array 110 in the example of FIG. 10A is placed between optical system 140 and image sensor 160.
• an apple is illustrated as an example of the target object 70.
• the object 70 is not limited to an apple, but may be any object.
• the image sensor 160 generates data of a compressed image 10 in which information of a plurality of wavelength bands is compressed as a two-dimensional monochrome image.
• the processing device 200 generates data representing a plurality of images corresponding one-to-one to a plurality of wavelength bands included in a predetermined target wavelength range, based on data of the compressed image 10 generated by the image sensor 160. As described above, the number of wavelength bands included in the target wavelength range is set to N (N is an integer of 4 or more).
• restored image 20W 1 N images generated based on the compressed image are referred to as restored image 20W 1 , restored image 20W 2 , ..., restored image 20W N , and these are collectively referred to as "hyperspectral image 20". There are things to do.
• the filter array 110 in this embodiment is an array of a plurality of light-transmitting filters arranged in rows and columns.
• the plurality of filters include a plurality of types of filters having different wavelength dependencies of spectral transmittance, that is, light transmittance.
• the filter array 110 modulates the intensity of incident light for each wavelength and outputs the modulated light. This process by filter array 110 is referred to as "encoding", and filter array 110 is sometimes referred to as "encoding element”.
• the filter array 110 is placed near or directly above the image sensor 160.
• “nearby” means that the image of light from the optical system 140 is close enough to be formed on the surface of the filter array 110 in a somewhat clear state.
• “Directly above” means that the two are so close that there is almost no gap between them.
• Filter array 110 and image sensor 160 may be integrated.
• the optical system 140 includes at least one lens. Although optical system 140 is shown as one lens in FIG. 12A, optical system 140 may be a combination of multiple lenses. Optical system 140 forms an image on the imaging surface of image sensor 160 via filter array 110.
• the filter array 110 may be placed apart from the image sensor 160.
• 12B to 12D are diagrams showing a configuration example of the imaging device 100 in which the filter array 110 is placed apart from the image sensor 160.
• filter array 110 is arranged between optical system 140 and image sensor 160 and at a position away from image sensor 160.
• FIG. 12C filter array 110 is placed between object 70 and optical system 140.
• the imaging device 100 includes two optical systems 140A and 140B, with the filter array 110 disposed between them.
• an optical system including one or more lenses may be placed between the filter array 110 and the image sensor 160.
• the image sensor 160 is a monochrome photodetection device having a plurality of photodetection elements (also referred to as "pixels" in this specification) arranged two-dimensionally.
• the image sensor 160 may be, for example, a CCD (Charge-Coupled Device), a CMOS (Complementary Metal Oxide Semiconductor) sensor, or an infrared array sensor.
• the photodetecting element includes, for example, a photodiode.
• Image sensor 160 does not necessarily have to be a monochrome type sensor.
• a color type sensor may be used.
• a color type sensor includes multiple red (R) filters that transmit red light, multiple green (G) filters that transmit green light, and multiple blue (B) filters that transmit blue light.
• the color type sensor may further include a plurality of IR filters that transmit infrared rays. Further, a color type sensor may include a plurality of transparent filters that transmit all red, green, and blue light. By using a color type sensor, the amount of information regarding wavelength can be increased, and the accuracy of reconstructing the hyperspectral image 20 can be improved.
• the wavelength range to be acquired may be arbitrarily determined, and is not limited to the visible wavelength range, but may be an ultraviolet, near-infrared, mid-infrared, or far-infrared wavelength range.
• Processing device 200 may be a computer with one or more processors and one or more storage media, such as memory.
• the processing device 200 generates data of a plurality of restored images 20W 1 , 20W 2 , . . . 20W N based on the compressed image 10 acquired by the image sensor 160.
• FIG. 13A is a diagram schematically showing an example of the filter array 110.
• Filter array 110 has a plurality of regions arranged two-dimensionally. In this specification, the area may be referred to as a "cell.”
• Optical filters having individually set spectral transmittances are arranged in each region.
• the spectral transmittance is expressed by a function T( ⁇ ), where the wavelength of incident light is ⁇ .
• the spectral transmittance T( ⁇ ) can take a value of 0 or more and 1 or less.
• the filter array 110 has 48 rectangular areas arranged in 6 rows and 8 columns. This is just an example, and in actual applications, more areas may be provided. The number may be approximately the same as the number of pixels of the image sensor 160, for example. The number of filters included in the filter array 110 is determined depending on the application, for example, in the range of several tens to tens of millions.
• FIG. 13B is a diagram showing an example of the spatial distribution of the transmittance of light in each of the wavelength bands W 1 , W 2 , . . . , W N included in the target wavelength range.
• the difference in shading in each region represents the difference in transmittance. The lighter the area, the higher the transmittance, and the darker the area, the lower the transmittance.
• the spatial distribution of light transmittance differs depending on the wavelength band.
• FIGS. 13C and 13D are diagrams showing examples of spectral transmittances of area A1 and area A2 included in filter array 110 shown in FIG. 13A, respectively.
• the spectral transmittance of area A1 and the spectral transmittance of area A2 are different from each other. In this way, the spectral transmittance of the filter array 110 differs depending on the region. However, the spectral transmittances of all regions do not necessarily have to be different. In the filter array 110, at least some of the plurality of regions have different spectral transmittances.
• Filter array 110 includes two or more filters with different spectral transmittances.
• the number of spectral transmittance patterns in the plurality of regions included in the filter array 110 may be equal to or greater than the number N of wavelength bands included in the target wavelength range.
• the filter array 110 may be designed such that half or more of the regions have different spectral transmittances.
• the target wavelength range W can be set to various ranges depending on the application.
• the target wavelength range W may be, for example, a visible light wavelength range from about 400 nm to about 700 nm, a near-infrared wavelength range from about 700 nm to about 2500 nm, or a near-ultraviolet wavelength range from about 10 nm to about 400 nm.
• the target wavelength range W may be a wavelength range such as mid-infrared or far-infrared. In this way, the wavelength range used is not limited to the visible light range.
• "light” refers to all radiation including not only visible light but also infrared and ultraviolet light.
• N is an arbitrary integer greater than or equal to 4, and each wavelength range obtained by dividing the target wavelength range W into N equal parts is defined as a wavelength band W 1 , W 2 , . . . , W N .
• the plurality of wavelength bands included in the target wavelength range W may be set arbitrarily.
• the bandwidth may be made non-uniform depending on the wavelength band.
• the bandwidth differs depending on the wavelength band, and there is a gap between two adjacent wavelength bands. In this way, the plurality of wavelength bands can be determined arbitrarily.
• FIG. 15A is a diagram for explaining the characteristics of spectral transmittance in a certain region of the filter array 110.
• the spectral transmittance has a plurality of maximum values P1 to P5 and a plurality of minimum values with respect to wavelengths within the target wavelength range W.
• the light transmittance within the target wavelength range W is normalized so that the maximum value is 1 and the minimum value is 0.
• the spectral transmittance has maximum values in wavelength ranges such as wavelength band W 2 and wavelength band W N-1 .
• the spectral transmittance of each region can be designed to have maximum values in at least two of the wavelength bands W 1 , W 2 , . . . W N .
• local maximum values P1, P3, P4, and P5 are 0.5 or more.
• the filter array 110 transmits a large amount of components in a certain wavelength range, and does not transmit as much components in other wavelength ranges. For example, for light in k wavelength bands out of N wavelength bands, the transmittance is greater than 0.5, and for light in the remaining N ⁇ k wavelength bands, the transmittance is 0.5. It can be less than k is an integer satisfying 2 ⁇ k ⁇ N. If the incident light is white light that evenly contains wavelength components of all visible light, the filter array 110 divides the incident light into areas that have multiple intensity peaks that are discrete with respect to wavelength. The multi-wavelength light is then superimposed and output.
• FIG. 15B is a diagram showing, as an example, the result of averaging the spectral transmittance shown in FIG. 15A for each wavelength band W 1 , W 2 , . . . , W N.
• the averaged transmittance is obtained by integrating the spectral transmittance T( ⁇ ) for each wavelength band and dividing it by the bandwidth of that wavelength band.
• the transmittance value averaged for each wavelength band is defined as the transmittance in that wavelength band.
• the transmittance is outstandingly high in three wavelength ranges with maximum values P1, P3, and P5. In particular, the transmittance exceeds 0.8 in two wavelength ranges with maximum values P3 and P5.
• a gray scale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1.
• a binary scale transmittance distribution may be adopted in which the transmittance of each region can take either a value of approximately 0 or approximately 1.
• each region transmits most of the light in at least two of the wavelength ranges included in the target wavelength range, and transmits most of the light in the remaining wavelength ranges. I won't let you.
• "most" refers to approximately 80% or more.
• a part of all the cells, for example half of the cells, may be replaced with a transparent area.
• Such a transparent region transmits each light of the wavelength bands W 1 , W 2 , . . . W N included in the target wavelength range W with a similar high transmittance, for example, 80% or more.
• the plurality of transparent regions may be arranged in a checkerboard pattern, for example. That is, in the two arrangement directions of the plurality of regions in the filter array 110, regions whose light transmittances differ depending on the wavelength and transparent regions may be arranged alternately.
• Such data indicating the spatial distribution of spectral transmittance of the filter array 110 is obtained in advance based on design data or actual measurement calibration, and is stored in a storage medium included in the processing device 200. This data is used for calculation processing described later.
• the filter array 110 may be configured using, for example, a multilayer film, an organic material, a diffraction grating structure, or a fine structure containing metal.
• a multilayer film for example, a dielectric multilayer film or a multilayer film including a metal layer can be used.
• at least one of the thickness, material, and lamination order of each multilayer film is formed to be different for each cell. This makes it possible to achieve different spectral characteristics depending on the cell.
• By using a multilayer film sharp rises and falls in spectral transmittance can be realized.
• a structure using organic materials can be realized by containing different pigments or dyes depending on the cell, or by stacking different types of materials.
• a configuration using a diffraction grating structure can be realized by providing diffraction structures with different diffraction pitches or depths for each cell.
• a fine structure containing metal it can be manufactured using spectroscopy based on the plasmon effect.
• the processing device 200 reconstructs a multi-wavelength hyperspectral image 20 based on the compressed image 10 output from the image sensor 160 and the spatial distribution characteristics of transmittance for each wavelength of the filter array 110.
• multi-wavelength means, for example, more wavelength ranges than the three color wavelength ranges of RGB acquired by a normal color camera.
• the number of wavelength ranges can be, for example, about 4 to 100.
• the number of wavelength ranges is referred to as the "number of bands.” Depending on the application, the number of bands may exceed 100.
• the data to be obtained is the data of the hyperspectral image 20, and this data is designated as f.
• f is data obtained by integrating data f 1 , f 2 , . . . , f N of N bands.
• Data f may be expressed in various formats, as shown in FIG. 6A, FIG. 6B, or FIG. 6C.
• the horizontal direction of the image is the x direction
• the vertical direction of the image is the y direction.
• each of the data f 1 , f 2 , ..., f N has n ⁇ m pixel values. . Therefore, the data f is data with the number of elements n ⁇ m ⁇ N.
• the number of elements of the data g of the compressed image 10 encoded and multiplexed by the filter array 110 is n ⁇ m.
• Data g can be expressed by the following equation (1).
• Equation (1) represents hyperspectral image data expressed as a one-dimensional vector, as shown in FIG. 6C.
• Each of f 1 , f 2 , . . . , f N has n ⁇ m elements. Therefore, the vector on the right side is a one-dimensional vector of n ⁇ m ⁇ N rows and 1 column.
• the compressed image data g is calculated as a one-dimensional vector of n ⁇ m rows and 1 column.
• the matrix H represents a transformation that encodes and intensity modulates each component f 1 , f 2 , . Therefore, H is a matrix with n ⁇ m rows and n ⁇ m ⁇ N columns. Equation (1) can also be expressed as follows.
• pg ij is the i-th row and j-th Represents the pixel value of a column.
• the processing device 200 uses the redundancy of the image included in the data f to find a solution using a compressed sensing technique. Specifically, the desired data f is estimated by solving Equation (2) below.
• f' represents the estimated data of f.
• the first term in parentheses in the above equation represents the amount of deviation between the estimation result Hf and the acquired data g, a so-called residual term.
• the second term in parentheses is a regularization term or stabilization term.
• Equation (2) means finding f that minimizes the sum of the first term and the second term.
• the function in parentheses in equation (2) is called an evaluation function.
• the processing device 200 can converge the solution through recursive iterative operations and calculate f that minimizes the evaluation function as the final solution f'.
• the first term in parentheses in Equation (2) means an operation to calculate the sum of squares of the difference between the acquired data g and Hf obtained by converting f in the estimation process using matrix H.
• the second term ⁇ (f) is a constraint in the regularization of f, and is a function reflecting the sparse information of the estimated data. This function has the effect of smoothing or stabilizing the estimated data.
• the regularization term may be represented by, for example, a discrete cosine transform (DCT), a wavelet transform, a Fourier transform, or a total variation (TV) of f. For example, when total variation is used, stable estimated data can be obtained that suppresses the influence of noise on the observed data g.
• DCT discrete cosine transform
• TV total variation
• the sparsity of the object 70 in the space of each regularization term differs depending on the texture of the object 70.
• a regularization term may be selected that makes the texture of the object 70 sparser in the space of regularization terms.
• multiple regularization terms may be included in the calculation.
• ⁇ is a weighting coefficient. The larger the weighting coefficient ⁇ , the greater the amount of redundant data to be reduced, and the higher the compression ratio. The smaller the weighting coefficient ⁇ , the weaker the convergence to a solution.
• the weighting coefficient ⁇ is set to an appropriate value that allows f to converge to some extent and not cause overcompression.
• the image encoded by the filter array 110 is acquired in a blurred state on the imaging surface of the image sensor 160. Therefore, the hyperspectral image 20 can be reconstructed by retaining this blur information in advance and reflecting the blur information in the matrix H described above.
• the blur information is expressed by a point spread function (PSF).
• PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, when a point image corresponding to one pixel on an image spreads to an area of k ⁇ k pixels around that pixel due to blur, the PSF is a group of coefficients that indicates the influence on the pixel value of each pixel in that area. , that is, can be defined as a matrix.
• the hyperspectral image 20 can be reconstructed by reflecting the effect of blurring of the coding pattern due to PSF on the matrix H.
• a position may be selected where the coding pattern of filter array 110 is too diffused and disappears.
• the hyperspectral image 20 can be restored from the compressed image 10 acquired by the image sensor 160.
• the processing device 200 restores the hyperspectral image 20 based on the data of the compressed image 10 output from the image sensor 160.
• a processor in the imaging device 100 may perform the process of restoring the hyperspectral image 20.
• a processor corresponding to the second processing circuit 214 in the processing device 200 shown in FIG. 8 is built into the imaging device 100.
• the processor generates hyperspectral image data 20 based on the compressed image data 10 output from the image sensor 160.
• a processor corresponding to the second processing circuit 214 shown in FIG. 8 may be installed in an external computer such as a cloud server that communicates with the imaging device 100 or the processing device 200 via a network. In that case, the external computer generates hyperspectral image 20 data based on the compressed image 10 data acquired from the imaging device 100 and transmits it to the processing device 200 .
• the compressed image and the restored image may be generated by imaging using a method different from imaging using the filter array 110 including the plurality of optical filters described above, that is, the encoding element.
• the light receiving characteristics of the image sensor 160 may be changed for each pixel by processing the image sensor 160.
• a compressed image can be generated by imaging using the image sensor 160 that has undergone the processing. That is, the compressed image may be generated by an imaging device configured such that the filter array 110 is built into the image sensor 160.
• the encoded information corresponds to the light receiving characteristics of the image sensor 160.
• the optical characteristics of the optical system 140 change spatially and wavelength-wise, thereby compressing the spectral information. It's okay.
• a compressed image can also be generated by an imaging device including this configuration.
• the encoded information is information corresponding to the optical characteristics of an optical element such as a metalens. In this way, the intensity of the incident light may be modulated for each wavelength using the imaging device 100 having a configuration different from the configuration using the filter array 110, and a compressed image and a restored image may be generated.
• the number of signals included in the compressed image also includes a configuration that generates a restored image that includes a larger number of signals than the number of pixels (for example, the number of pixels).
• the photoresponse characteristics may correspond to the light receiving characteristics of the image sensor, or may correspond to the optical characteristics of the optical element.
• the imaging device If the first data set does not satisfy a first predetermined condition, causing the light source to irradiate a second light under a second radiation condition, and causing the imaging device to photograph the subject irradiated with the second light; thereby causing the imaging device to generate four or more second data sets each including a plurality of pixel values, a first value indicating a magnitude of change in spectral shape determined based on the four or more first data sets and the four or more second data sets is smaller than a threshold;
• the four or more first data sets correspond to four or more wavelength ranges,
• the four or more second data sets correspond to the four or more wavelength ranges, Method.
• the method may include not causing the light source to irradiate the second light under the second radiation condition when the four or more first data sets satisfy the first predetermined condition.
• each of the four or more first data sets includes n ⁇ m pixel values;
• Each of the four or more second data sets includes n ⁇ m pixel values, The number of the first data sets and the number of the second data sets are N,
• the first value may be ⁇ (absolute value of the angle 11 )+...+(absolute value of the angle nm ) ⁇ /(n ⁇ m).
• the imaging device includes four or more image sensors I 1 , ... I N corresponding to the four or more wavelength ranges
• the four or more image sensors I 1 , ..., I N correspond to the four or more first data sets f 11 , ... f 1N , respectively
• the four or more image sensors I 1 , ..., I N respectively correspond to the four or more second data sets f 21 , ... f 2N
• the image sensor I1 includes pixels s1 11 , . . . , pixels s1 1m , . . . , pixels s1 n1 , .
• the image sensor IN includes pixels sN 11 , . . . , pixels sN 1m , . . .
• pixels sN n1 , . . . , pixels sN nm corresponds to the pixel value p11 11 and the pixel value p21 11 ,...
• the pixel s1 1m corresponds to the pixel value p11 1m and the pixel value p21 1m ,...
• the pixel s1 n1 corresponds to the pixel value p11 n1 and the pixel value p21 n1 ,...
• the pixel s1 nm corresponds to the pixel value p11 nm and the pixel value p21 nm , ...,
• the pixel sN 11 corresponds to the pixel value p1N 11 and the pixel value p2N 11 ,...
• the pixel sN 1m corresponds to the pixel value p1N 1m and the pixel value p2N 1m ,...
• the pixel sN n1 corresponds to the pixel value p1N n1 and the pixel value p2N n1 ,...
• the pixel sN nm may correspond to the pixel value p1N nm and the pixel value p2N nm .
• the four or more light transmittance characteristics corresponding to each of the four or more filters are different from each other, and the four or more light transmittance characteristics include a target wavelength range including four or more wavelength ranges.
• first data which is a light transmittance of the imaging device; and if the first data does not satisfy a first predetermined condition, causing the light source to emit second light under a second radiation condition; to photograph the subject irradiated with the second light, thereby causing the imaging device to generate second data including a plurality of pixel values, Changes in spectral shape determined based on four or more first data sets generated based on the first data and four or more second data sets generated based on the second data.
• the first value indicating the size is smaller than the threshold;
• the four or more first data sets correspond to the four or more wavelength ranges,
• the four or more second data sets correspond to the four or more wavelength ranges, Method.
• the method may include not causing the light source to emit the second light under the second radiation condition when the first data satisfies the first predetermined condition.
• each of the four or more first data sets includes n ⁇ m pixel values;
• Each of the four or more second data sets includes n ⁇ m pixel values, The number of the first data sets and the number of the second data sets are N,
• the four or more first data sets are f 11 ,..., f 1N ,
• the four or more second data sets are f 21 ,..., f 2N ,
• f 1N (p1N 11 ...p1N 1m ...p1N n1 ...p1N nm ) T
• f21 (p21 11 ...p21 1m ...p21 n1 ...p21 nm ) T
• f22 (p22 11 ...p22 1m ...p22 n1 ...p22 nm ) T ,...
• f 2N (p2N 11 ...p2N 1m ...p2N n1 ...p2N nm ) T ,
• Each of p11 11 , ..., p2N nm is a pixel value,
• the first value is the absolute value of the angle 11 formed by (p11 11 p12 11 ...p1N 11 ) and (p21 11 p22 11 ...p2N 11 ),..., (p11 nm p12 nm ...p1N It may be determined based on the absolute value of the angle nm formed by (p21 nm , p22 nm ...p2N nm ).
• the first value may be ⁇ (absolute value of the angle 11 )+...+(absolute value of the angle nm ) ⁇ /(n ⁇ m).
• the imaging device includes an image sensor, the image sensor includes n ⁇ m pixels p 11 ...p 1m ...p n1 ...p nm ;
• the pixel p11 corresponds to the pixel value pg1 11 and the pixel value pg2 11
• the pixel p1m corresponds to the pixel value pg1 1m and the pixel value pg21 1m
• the pixel p n1 may correspond to the pixel value pg1 n1 and the pixel value pg2 n1
• pixel p nm may correspond to the pixel value pg1 nm and the pixel value pg2 nm .
• FIG. 1 a light source and An imaging device that generates image data by photographing a subject illuminated with light from the light source, wherein the image data includes image information of each of four or more bands, or images of the four or more bands.
• an imaging device including information of a compressed image in which the information is compressed as one image;
• a processing device Equipped with The processing device includes: determining whether pixel values of a plurality of pixels in the image data satisfy a predetermined condition; If the predetermined condition is not met, changing the illumination conditions by the light source under conditions where the spectral shape of the light from the light source at the position of the subject does not change; Imaging system.
• the processing device causes the adjustment device to repeat an operation of changing the distance between the light source and the subject within the specified range until the condition is met.
• Imaging system according to technique 2.
• the processing device changes the illumination condition by changing a control parameter for driving the light source within a predetermined range when the predetermined condition is not satisfied. imaging system.
• Determining whether the pixel values of the plurality of pixels satisfy the predetermined condition includes determining whether the pixel value of each of the plurality of pixels is within a predetermined range.
• the imaging system according to any one of techniques 1 to 5.
• Determining whether the pixel values of the plurality of pixels satisfy the predetermined condition includes determining whether a contrast value calculated from the pixel values of the plurality of pixels exceeds a threshold value. , the imaging system according to any one of Techniques 1 to 5.
• the image data includes information about the compressed image
• the processing device executes processing for generating images of each of the four or more bands based on the compressed image if the predetermined condition is satisfied.
• Imaging system according to any one of techniques 1 to 7.
• the imaging device includes: an optical element that changes the spatial distribution of the intensity of light from the subject for each wavelength; an image sensor that receives light that has passed through the optical element and generates the image data that includes information about the compressed image;
• the imaging device includes: an optical element that changes the spatial distribution of the intensity of light from the subject for each wavelength; an image sensor that receives light that has passed through the optical element and outputs the image data including information about the compressed image; Equipped with The processing device according to any one of techniques 1 to 5, wherein the processing device generates other image data including image information of each of the four or more bands based on the image data output from the image sensor. Imaging system.
• Determining whether the pixel values of the plurality of pixels satisfy the predetermined condition includes determining whether the pixel value of each of the plurality of pixels in each image of the four or more bands is within a predetermined range.
• Determining whether or not the pixel values of the plurality of pixels satisfy the predetermined condition includes determining whether the contrast value calculated from the pixel values of the plurality of pixels in each image of the four or more bands exceeds a threshold value.
• the optical element includes a plurality of optical filters arranged in a two-dimensional plane, the spectral transmittances of the plurality of optical filters are different from each other, and the spectral transmittance of each of the plurality of optical filters has a plurality of maximum values.
• the imaging system according to technique 9 or 11, which shows.
• the adjustment device includes a linear actuator that changes the distance between the light source and the subject by moving the light source in a direction perpendicular to the support surface of the stage.
• the imaging system according to any one of techniques 2 to 4.
• the processing device is configured to determine the subject based on the relationship between the calibration image data generated by the imaging device photographing the calibration subject illuminated by light from the light source and the parameters that define the illumination conditions. determining a prescribed range of the parameter in which the spectral shape at the position does not change; changing the illumination condition by changing the parameter within the specified range; The imaging system according to any one of techniques 1 to 16.
• the processing device includes: causing the imaging device to generate the calibration image data while changing the parameters; The imaging system according to technique 17, wherein a range of the parameter in which an amount of change in the spectral shape of the calibration subject specified based on the calibration image data is smaller than a predetermined amount is determined as the specified range.
• the processing device before photographing the subject, obtaining calibration image data generated by the imaging device photographing a calibration subject illuminated by light from the light source; determining whether pixel values of a plurality of pixels in the calibration image data satisfy the predetermined condition; If the predetermined condition is met, generating spectrum data of the calibration subject based on the calibration image data and storing it in a storage device; If the predetermined condition is not met, changing the parameter defining the illumination condition within a specified range;
• the imaging system according to any one of techniques 1 to 18.
• FIG. 20 A method performed by one or more processors executing a plurality of instructions stored in one or more memories, the method comprising: An imaging device that generates image data by photographing a subject illuminated with light from a light source, the image data being image information of each of four or more bands, or image information of the four or more bands. obtaining the image data from the imaging device, including information on a compressed image that is compressed as one image; determining whether pixel values of a plurality of pixels in the image data satisfy a predetermined condition; If the predetermined condition is not met, changing the illumination conditions by the light source under conditions where the spectral shape of the light from the light source at the position of the subject does not change; method including.
• the technology of the present disclosure can be widely used in cameras and measuring instruments that acquire multi-wavelength images, such as hyperspectral cameras.
• the technology of the present disclosure can be applied, for example, to applications such as line inspection in a factory, where slight color changes are identified and quality is evaluated.
• Compressed image 20 Hyperspectral image 70 Object 100 Imaging device 110 Filter array 120 Illumination device 122 Light source 130 Adjustment device 140 Optical system 150 Support 160 Image sensor 170 Object 190 Stage 200 Processing device 270 Subject 300 Display device 1000 Imaging system

Landscapes

• Physics & Mathematics (AREA)
• Spectroscopy & Molecular Physics (AREA)
• General Physics & Mathematics (AREA)
• Engineering & Computer Science (AREA)
• Multimedia (AREA)
• Signal Processing (AREA)
• Health & Medical Sciences (AREA)
• Theoretical Computer Science (AREA)
• Mathematical Physics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Chemical & Material Sciences (AREA)
• Analytical Chemistry (AREA)
• Biochemistry (AREA)
• General Health & Medical Sciences (AREA)
• Immunology (AREA)
• Pathology (AREA)
• Investigating Or Analysing Materials By Optical Means (AREA)

Priority Applications (2)

Applications Claiming Priority (4)

Related Child Applications (1)

Publications (1)

Family

ID=88022999

Family Applications (1)

Country Status (3)

Cited By (2)

Families Citing this family (2)

Citations (9)

• 2023
  • 2023-03-03 JP JP2024507734A patent/JPWO2023176492A1/ja active Pending
  • 2023-03-03 WO PCT/JP2023/007936 patent/WO2023176492A1/ja not_active Ceased
• 2024
  • 2024-09-02 US US18/822,361 patent/US20240422443A1/en active Pending

Patent Citations (9)

Also Published As

Similar Documents

Legal Events