WO2024203282A1 - 撮像システムおよび方法 - Google Patents

撮像システムおよび方法 Download PDF

Info

Publication number
WO2024203282A1
WO2024203282A1 PCT/JP2024/009554 JP2024009554W WO2024203282A1 WO 2024203282 A1 WO2024203282 A1 WO 2024203282A1 JP 2024009554 W JP2024009554 W JP 2024009554W WO 2024203282 A1 WO2024203282 A1 WO 2024203282A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
illumination light
compressed
compressed image
light
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2024/009554
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
智 佐藤
孝行 清原
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Panasonic Intellectual Property Management Co Ltd
Original Assignee
Panasonic Intellectual Property Management Co Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Panasonic Intellectual Property Management Co Ltd filed Critical Panasonic Intellectual Property Management Co Ltd
Priority to JP2025510264A priority Critical patent/JPWO2024203282A1/ja
Publication of WO2024203282A1 publication Critical patent/WO2024203282A1/ja
Priority to US19/329,640 priority patent/US20260012711A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/95—Computational photography systems, e.g. light-field imaging systems
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/28—Investigating the spectrum
    • G01J3/30—Measuring the intensity of spectral lines directly on the spectrum itself
    • G01J3/36—Investigating two or more bands of a spectrum by separate detectors
    • G—PHYSICS
    • G01—MEASURING; TESTING
    • G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
    • G01J3/00—Spectrometry; Spectrophotometry; Monochromators; Measuring colours
    • G01J3/46—Measurement of colour; Colour measuring devices, e.g. colorimeters
    • G01J3/50—Measurement of colour; Colour measuring devices, e.g. colorimeters using electric radiation detectors
    • G01J3/51—Measurement of colour; Colour measuring devices, e.g. colorimeters using electric radiation detectors using colour filters
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/54—Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/56—Cameras or camera modules comprising electronic image sensors; Control thereof provided with illuminating means
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/667—Camera operation mode switching, e.g. between still and video, sport and normal or high- and low-resolution modes
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/68—Control of cameras or camera modules for stable pick-up of the scene, e.g. compensating for camera body vibrations
    • H04N23/682—Vibration or motion blur correction
    • H04N23/683—Vibration or motion blur correction performed by a processor, e.g. controlling the readout of an image memory

Definitions

  • This disclosure relates to imaging systems and methods.
  • spectral information from a large number of narrow wavelength bands hereafter simply referred to as “bands”
  • bands narrow wavelength bands
  • RGB images that have color information in three bands (red, green, and blue).
  • a camera that can capture images in such many wavelength bands is called a “hyperspectral camera.”
  • Hyperspectral cameras are used in a variety of fields, such as food inspection, biomedical testing, pharmaceutical development, and mineral composition analysis.
  • Patent Document 1 discloses an example of a hyperspectral imaging device that uses compressed sensing.
  • Compressive sensing is a technology that restores more data than the observed data by assuming that the data distribution of the observed object is sparse in a certain space (e.g., frequency space).
  • the imaging device disclosed in Patent Document 1 is equipped with an encoding element, which is an array of multiple optical filters with different spectral transmittances, on the optical path connecting the object and the image sensor.
  • the imaging device can generate images corresponding to each of multiple wavelength bands in a single imaging session by performing a restoration calculation based on a compressed image acquired by imaging using the encoding element.
  • the present disclosure provides a system and method capable of improving the quality of images in each wavelength band restored from a compressed image acquired by an imaging device using compressed sensing.
  • An imaging system includes an illumination device that emits illumination light to be irradiated onto an object, an imaging device that receives reflected light from the object caused by the illumination light and generates a compressed image in which image information of multiple wavelength bands included in a target wavelength range is compressed into a single piece of image information, and a processing device that generates multiple restored images based on the compressed image, each of which corresponds to a different one of the multiple wavelength bands.
  • the multiple wavelength bands include a first wavelength band and a second wavelength band. The intensity of the light of the first wavelength band included in the illumination light is different from the intensity of the light of the second wavelength band included in the illumination light.
  • a method is executed by a computer and includes causing an illumination device to emit illumination light to be irradiated onto an object, and causing an imaging device to receive reflected light from the object caused by the illumination light and generate a compressed image in which image information of multiple wavelength bands included in a target wavelength range is compressed into a single piece of image information, generating the compressed image.
  • the multiple wavelength bands include a first wavelength band and a second wavelength band. The intensity of the light of the first wavelength band included in the illumination light is different from the intensity of the light of the second wavelength band included in the illumination light.
  • a method is executed by a computer and includes acquiring first data on the spectral reflectance characteristics of an object, acquiring second data on the spectrum of illumination light irradiated on the object, acquiring third data on the spectral sensitivity characteristics of each pixel of an imaging device that images the object, and generating an estimated image that is estimated to be generated when the imaging device images the object irradiated with the illumination light, based on the first data, the second data, and the third data.
  • a comprehensive or specific aspect of the present disclosure may be realized in a system, an apparatus, a method, an integrated circuit, a computer program, or a computer-readable recording medium, or in any combination of a system, an apparatus, a method, an integrated circuit, a computer program, and a recording medium.
  • a computer-readable recording medium includes a non-volatile recording medium such as a CD-ROM (Compact Disc-Read Only Memory).
  • An apparatus may be composed of one or more devices. When an apparatus is composed of two or more devices, the two or more devices may be arranged in a single device, or may be arranged separately in two or more separate devices.
  • "apparatus" may mean not only one device, but also a system consisting of multiple devices.
  • the technology disclosed herein makes it possible to improve the quality of images of each wavelength band restored from a compressed image in which spectral information has been compressed.
  • FIG. 1A is a diagram illustrating a schematic configuration example of an imaging system.
  • FIG. 1B is a diagram illustrating a schematic configuration example of another imaging system.
  • FIG. 1C is a diagram illustrating a schematic configuration example of still another imaging system.
  • FIG. 1D is a diagram illustrating a schematic configuration example of still another imaging system.
  • FIG. 2A is a schematic diagram illustrating an example of a filter array.
  • FIG. 2B is a diagram showing an example of a spatial distribution of the light transmittance of each of a plurality of wavelength bands included in the target wavelength range.
  • FIG. 2C is a diagram showing an example of the spectral transmittance of the area A1 included in the filter array shown in FIG. 2A.
  • FIG. 2D is a diagram showing an example of the spectral transmittance of the area A2 included in the filter array shown in FIG. 2A.
  • FIG. 3 is a diagram for explaining an example of the relationship between a target wavelength range and a plurality of wavelength bands included therein.
  • FIG. 4A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of the filter array.
  • FIG. 4B is a diagram showing the results of averaging the spectral transmittance shown in FIG. 4A for each wavelength band.
  • FIG. 5A is a diagram illustrating an example of a data structure of a hyperspectral image.
  • FIG. 5B is a diagram showing another example of the data structure of the hyperspectral image.
  • FIG. 5C is a diagram showing yet another example of the data structure of a hyperspectral image.
  • FIG. 6A is a diagram showing a color swatch that includes multiple stripes extending vertically and horizontally.
  • FIG. 6B is a diagram showing an example of a compressed image obtained when the color sample shown in FIG. 6A is imaged by an imaging device.
  • FIG. 7A is a graph illustrating an example of non-uniform spectral intensity of illumination light.
  • FIG. 7B is a graph illustrating an example of non-uniform spectral intensity of illumination light.
  • FIG. 7C is a graph illustrating an example of non-uniform spectral intensity of illumination light.
  • FIG. 7D is a graph illustrating an example of non-uniform spectral intensity of illumination light.
  • FIG. 8 is a block diagram showing an example of the configuration of an imaging system.
  • FIG. 9 is a diagram showing an example of mask data.
  • FIG. 10 is a block diagram showing another example of the configuration of the imaging system.
  • FIG. 11 is a flowchart showing an example of the operation of the processing device.
  • FIG. 12 is a diagram showing an example of a method for generating an edge image.
  • FIG. 13 is a diagram showing another example of a method for generating an edge image.
  • FIG. 14 is a diagram showing yet another example of a method for generating an edge image.
  • FIG. 15 is a diagram showing an example of a superimposed display of a compressed image and an edge image.
  • FIG. 16 is a diagram showing an example in which a compressed image and an edge image are displayed side by side.
  • FIG. 16 is a diagram showing an example in which a compressed image and an edge image are displayed side by side.
  • FIG. 17 is a diagram showing yet another example of displaying a compressed image and an edge image.
  • FIG. 18 is a diagram showing yet another example of displaying a compressed image and an edge image.
  • FIG. 19 is a diagram showing yet another example of displaying a compressed image and an edge image.
  • FIG. 20 is a block diagram showing another example of the configuration of the imaging system.
  • FIG. 21 is a flowchart showing another example of the operation of the processing device.
  • FIG. 22 is a flowchart showing yet another example of the operation of the processing device.
  • FIG. 23 is a block diagram showing still another example of the configuration of the imaging system.
  • FIG. 24 is a flowchart showing yet another example of the operation of the processing device.
  • FIG. 21 is a flowchart showing another example of the operation of the processing device.
  • FIG. 22 is a flowchart showing yet another example of the operation of the processing device.
  • FIG. 23 is a block diagram showing still another example of the configuration of the imaging system.
  • FIG. 25 is a block diagram showing an example of the configuration of a system that determines an optimal illumination light spectrum according to an assumed target object.
  • FIG. 26 is a diagram showing an example of the spectral reflectance of an object.
  • FIG. 27 is a flowchart showing an example of processing executed by a processing circuit in the system shown in FIG.
  • all or part of a circuit, unit, device, member or part, or all or part of a functional block in a block diagram may be implemented by one or more electronic circuits including, for example, a semiconductor device, a semiconductor integrated circuit (IC), or an LSI (large scale integration).
  • the LSI or IC may be integrated into a single chip, or may be configured by combining multiple chips.
  • functional blocks other than memory elements may be integrated into a single chip.
  • LSI or IC are used here, the name may vary depending on the degree of integration, and may be called a system LSI, VLSI (very large scale integration), or ULSI (ultra large scale integration).
  • a field programmable gate array which is programmed after the LSI is manufactured, or a reconfigurable logic device (RLD), which can reconfigure the junctions within the LSI or set up circuit partitions within the LSI, may also be used for the same purpose.
  • FPGA field programmable gate array
  • RLD reconfigurable logic device
  • all or part of the functions or operations of a circuit, unit, device, member or part can be executed by software processing.
  • the software is recorded on one or more non-transitory recording media such as ROMs, optical disks, hard disk drives, etc., and when the software is executed by a processor, the functions specified in the software are executed by the processor and peripheral devices.
  • the system or device may include one or more non-transitory recording media on which the software is recorded, a processor, and necessary hardware devices, such as interfaces.
  • data or signals representing an image i.e., a collection of data or signals representing the pixel values of each of a number of pixels in an image, may be referred to simply as an "image.”
  • light refers to electromagnetic waves including not only visible light (wavelengths of about 400 nm to about 700 nm), but also ultraviolet light (wavelengths of about 10 nm to about 400 nm) and infrared light (wavelengths of about 700 nm to about 1 mm).
  • FIG. 1A is a diagram illustrating a schematic configuration example of an imaging system 1000 according to an exemplary embodiment of the present disclosure.
  • the imaging system 1000 illustrated in FIG. 1A includes an imaging device 100, a processing device 200, and an illumination device 300.
  • the imaging device 100 includes a configuration similar to that of the imaging device disclosed in Patent Document 1.
  • the imaging device 100 includes an optical system 140, a filter array 110, and an image sensor 160.
  • the optical system 140 and the filter array 110 are disposed on the optical path of light incident from an object 70, which is a subject.
  • the filter array 110 in the example of FIG. 1A is disposed between the optical system 140 and the image sensor 160.
  • FIG. 1A illustrates an apple as an example of the object 70.
  • the object 70 is not limited to an apple, and may be any object.
  • the object 70 is illuminated by illumination light from the illumination device 300, and is imaged by the imaging device 100.
  • the image sensor 160 generates data of a compressed image 10 in which image information of a plurality of wavelength bands included in a preset target wavelength range is compressed as two-dimensional monochrome image information.
  • the processing device 200 generates data indicating a plurality of images corresponding one-to-one to the plurality of wavelength bands included in the target wavelength range based on the data of the compressed image 10 generated by the image sensor 160.
  • the number of wavelength bands included in the target wavelength range is N (N is an integer of 4 or more).
  • restored images 20W 1 , 20W 2 , ..., 20W N are referred to as restored images 20W 1 , 20W 2 , ..., 20W N , and these may be collectively referred to as "hyperspectral image 20" or simply "spectral image”.
  • the filter array 110 is an optical element in which multiple light-transmitting filters are arranged in rows and columns.
  • the multiple filters include multiple types of filters with different spectral transmittances.
  • the spectral transmittance indicates the wavelength dependency of the transmittance, and is also called the transmission spectrum.
  • the filter array 110 modulates the intensity of the incident light for each wavelength and outputs it. This process performed by the filter array 110 is called “encoding.”
  • the filter array 110 is also called an "encoding element” or an "encoding mask.”
  • the filter array 110 is disposed near or directly above the image sensor 160.
  • “near” means close enough that a reasonably clear image of the light from the optical system 140 is formed on the surface of the filter array 110.
  • “Directly above” means that the two are so close that there is almost no gap between them.
  • the filter array 110 and the image sensor 160 may be integrated.
  • Optical system 140 includes at least one lens. In FIG. 1A, optical system 140 is shown as a single lens, but optical system 140 may be a combination of multiple lenses. Optical system 140 forms an image on the imaging surface of image sensor 160 through filter array 110.
  • FIGS. 1B to 1D are diagrams showing configuration examples of the imaging device 100 in which the filter array 110 is disposed away from the image sensor 160.
  • the filter array 110 is disposed between the optical system 140 and the image sensor 160 and at a position distant from the image sensor 160.
  • the filter array 110 is disposed between the object 70 and the optical system 140.
  • the imaging device 100 includes two optical systems 140A and 140B, and the filter array 110 is disposed between them.
  • an optical system including one or more lenses may be disposed between the filter array 110 and the image sensor 160.
  • the image sensor 160 is a monochrome type light detection device having a plurality of light detection elements (also referred to as "pixels" in this specification) arranged two-dimensionally.
  • the image sensor 160 may be, for example, a charge-coupled device (CCD), a complementary metal oxide semiconductor (CMOS) sensor, or an infrared array sensor.
  • the light detection elements include, for example, photodiodes.
  • the image sensor 160 does not necessarily have to be a monochrome type sensor.
  • a color type sensor may be used.
  • the color type sensor may include, for example, a plurality of red (R) filters that transmit red light, a plurality of green (G) filters that transmit green light, and a plurality of blue (B) filters that transmit blue light.
  • the color type sensor may further include a plurality of IR filters that transmit infrared light.
  • the color type sensor may also include a plurality of transparent filters that transmit all red, green, and blue light.
  • the processing device 200 may be a computer including one or more processors and one or more storage media such as a memory.
  • the processing device 200 generates data of a plurality of restored images 20W 1 , 20W 2 , . . . 20W N based on the compressed image 10 acquired by the image sensor 160.
  • FIG. 2A is a diagram showing a schematic example of a filter array 110.
  • the filter array 110 has a number of regions arranged two-dimensionally. In this specification, these regions are sometimes referred to as "cells.” In each region, an optical filter having an individually set spectral transmittance is arranged.
  • the spectral transmittance is expressed as a function T( ⁇ ), where ⁇ is the wavelength of the incident light.
  • the spectral transmittance T( ⁇ ) can take a value between 0 and 1.
  • the filter array 110 has 48 rectangular regions arranged in 6 rows and 8 columns. This is merely an example, and in actual applications, more regions may be provided. The number may be approximately the same as the number of pixels in the image sensor 160, for example. The number of filters included in the filter array 110 is determined according to the application and ranges from tens to tens of millions, for example.
  • FIG. 2B is a diagram showing an example of the spatial distribution of the light transmittance of each of the wavelength bands W1 , W2 , ..., WN included in the target wavelength range.
  • the difference in the shading of each region represents the difference in the transmittance. The lighter the region, the higher the transmittance, and the darker the region, the lower the transmittance.
  • the spatial distribution of the light transmittance differs depending on the wavelength band.
  • 2C and 2D are diagrams showing examples of the spectral transmittance of the region A1 and the region A2 included in the filter array 110 shown in FIG. 2A, respectively.
  • the spectral transmittance of the region A1 and the spectral transmittance of the region A2 are different from each other. In this way, the spectral transmittance of the filter array 110 varies depending on the region. However, it is not necessary that the spectral transmittance of all the regions is different.
  • the spectral transmittance of at least some of the multiple regions is different from each other.
  • the filter array 110 includes two or more filters having different spectral transmittances from each other.
  • the number of patterns of the spectral transmittance of the multiple regions included in the filter array 110 may be equal to or greater than the number N of wavelength bands included in the target wavelength range.
  • the filter array 110 may be designed so that the spectral transmittance of more than half of the regions is different.
  • FIG. 3 is a diagram for explaining the relationship between the target wavelength range W and the wavelength bands W 1 , W 2 , ..., W N included therein.
  • the target wavelength range W can be set to various ranges depending on the application.
  • the target wavelength range W can be, for example, a visible light wavelength range of about 400 nm to about 700 nm, a near-infrared wavelength range of about 700 nm to about 2500 nm, or a near-ultraviolet wavelength range of about 10 nm to about 400 nm.
  • the target wavelength range W may be a wavelength range such as mid-infrared or far-infrared. In this way, the wavelength range used is not limited to the visible light range.
  • radiation in general, including infrared and ultraviolet rays, as well as visible light is referred to as "light”.
  • N is an arbitrary integer equal to or greater than 4, and the wavelength bands obtained by equally dividing the target wavelength range W into N are wavelength bands W1 , W2 , ..., WN .
  • the multiple wavelength bands included in the target wavelength range W may be set arbitrarily.
  • the bandwidths of the wavelength bands may be made non-uniform.
  • the method of determining the multiple wavelength bands may be arbitrary.
  • FIG. 4A is a diagram for explaining the characteristics of the spectral transmittance in a certain region of the filter array 110.
  • the spectral transmittance has multiple maximum values P1 to P5 and multiple minimum values for wavelengths in the target wavelength range W.
  • the maximum value of the light transmittance in the target wavelength range W is normalized to 1 and the minimum value is 0.
  • the spectral transmittance has maximum values in wavelength ranges such as wavelength band W 2 and wavelength band W N-1 . In this way, the spectral transmittance of each region can be designed to have maximum values in at least two wavelength ranges among the wavelength bands W 1 , W 2 , ..., W N.
  • the maximum values P1, P3, P4, and P5 are 0.5 or more.
  • the filter array 110 transmits a large amount of components in a certain wavelength range among the incident light, and does not transmit components in other wavelength ranges very much.
  • the transmittance of light in k wavelength bands out of the N wavelength bands may be greater than 0.5, and the transmittance of light in the remaining N-k wavelength bands may be less than 0.5, where k is an integer satisfying 2 ⁇ k ⁇ N. If the incident light is white light that contains all visible light wavelength components evenly, the filter array 110 modulates the incident light into light having multiple discrete intensity peaks with respect to wavelength for each region, and outputs this multi-wavelength light by superimposing it.
  • FIG. 4B is a diagram showing, as an example, the result of averaging the spectral transmittance shown in FIG. 4A for each wavelength band W 1 , W 2 , ..., W N.
  • the averaged transmittance is obtained by integrating the spectral transmittance T( ⁇ ) for each wavelength band and dividing by the bandwidth of the wavelength band.
  • the transmittance value averaged for each wavelength band in this way is defined as the transmittance in that wavelength band.
  • the transmittance is remarkably high in three wavelength ranges having maximum values P1, P3, and P5. In particular, the transmittance exceeds 0.8 in two wavelength ranges having maximum values P3 and P5.
  • a grayscale transmittance distribution is assumed in which the transmittance of each region can take any value between 0 and 1 inclusive.
  • a binary scale transmittance distribution may be used in which the transmittance of each region can take a value of either approximately 0 or approximately 1.
  • each region transmits most of the light in at least two of the multiple wavelength ranges included in the target wavelength range, and does not transmit most of the light in the remaining wavelength ranges.
  • "most" refers to approximately 80% or more.
  • a part of all the cells may be replaced with a transparent region.
  • a transparent region transmits the light of each of the wavelength bands W 1 , W 2 , ..., W N included in the target wavelength range W with a similarly high transmittance, for example, a transmittance of 80% or more.
  • the multiple transparent regions may be arranged, for example, in a checkerboard pattern. That is, in two arrangement directions of the multiple regions in the filter array 110, regions whose light transmittance varies depending on the wavelength and transparent regions may be arranged alternately.
  • the data showing the spatial distribution of the spectral transmittance of the filter array 110 is acquired in advance based on design data or actual measurement calibration, and is stored in a storage medium provided in the processing device 200. This data is used in the calculation processing described below.
  • a configuration using a diffraction grating structure can be realized by providing a diffraction structure with a different diffraction pitch or depth for each cell.
  • a microstructure containing a metal When a microstructure containing a metal is used, it can be manufactured by utilizing spectrum due to the plasmon effect.
  • the processing device 200 reconstructs a multi-wavelength hyperspectral image 20 based on the compressed image 10 output from the image sensor 160 and the spatial distribution characteristics of the transmittance for each wavelength of the filter array 110.
  • multi-wavelength means more wavelength ranges than the wavelength ranges of the three colors RGB captured by a normal color camera, for example.
  • the number of wavelength ranges can be, for example, about 4 to 100. This number of wavelength ranges is referred to as the "number of bands.” Depending on the application, the number of bands may exceed 100.
  • the number of pixels in the horizontal direction of each of the images 20W 1 , 20W 2 , ..., 20W N is m
  • the number of pixels in the vertical direction is n
  • the pixel value of the pixel in the i-th row and j-th column in the kth image 20W k is pk ij .
  • the hyperspectral image 20 can be expressed by the following N n ⁇ m matrices.
  • Data representing the matrix H may be created before the reconstruction operation and stored in a storage device such as the memory of the processing device 200.
  • the processing device 200 utilizes the image redundancy contained in the data f to find a solution using a compressed sensing technique.
  • the desired data f is estimated by solving the following equation (2).
  • f' represents the estimated f data.
  • the first term in the parentheses in the above equation represents the amount of deviation between the estimated result Hf and the acquired data g, the so-called residual term.
  • the sum of squares is used as the residual term, but the absolute value or the square root of the sum of squares, etc. may also be used as the residual term.
  • the second term in the parentheses is a regularization term or stabilization term. Equation (2) means to find f that minimizes the sum of the first and second terms.
  • the function in the parentheses in equation (2) is called the evaluation function.
  • the processing device 200 can converge the solution by recursive iterative calculations and calculate f that minimizes the evaluation function as the final solution f'.
  • the first term in the parentheses in formula (2) means an operation to obtain the sum of squares of the difference between the acquired data g and Hf obtained by transforming f in the estimation process by the matrix H.
  • the second term ⁇ (f) is a constraint condition in the regularization of f, and is a function reflecting the sparse information of the estimated data. This function has the effect of smoothing or stabilizing the estimated data.
  • the regularization term can be expressed, for example, by the discrete cosine transform (DCT), wavelet transform, Fourier transform, or total variation (TV) of f. For example, when the total variation is used, stable estimated data that suppresses the influence of noise in the observed data g can be obtained.
  • the sparsity of the object 70 in the space of each regularization term differs depending on the texture of the object 70.
  • a regularization term that makes the texture of the object 70 sparser in the space of the regularization term may be selected.
  • multiple regularization terms may be included in the operation.
  • ⁇ is a weighting coefficient. The larger the weighting factor ⁇ , the more redundant data is reduced, and the higher the compression ratio. The smaller the weighting factor ⁇ , the weaker the convergence to a solution.
  • the weighting factor ⁇ is set to an appropriate value that allows f to converge to a certain extent, but does not result in over-compression.
  • the image encoded by the filter array 110 may be acquired in a blurred state on the imaging surface of the image sensor 160.
  • the blur information is stored in advance, and the hyperspectral image 20 can be reconstructed by reflecting the blur information in the above-mentioned matrix H.
  • the blur information is represented by a point spread function (PSF).
  • the PSF is a function that defines the degree of spread of a point image to surrounding pixels. For example, when a point image corresponding to one pixel on an image spreads to a region of k ⁇ k pixels around the pixel due to blurring, the PSF can be defined as a group of coefficients, i.e., a matrix, that indicates the influence on the pixel value of each pixel in the region.
  • the hyperspectral image 20 with reduced blurring can be reconstructed.
  • the position at which the filter array 110 is disposed is arbitrary, but a position at which the encoding pattern of the filter array 110 does not diffuse too much and disappear can be selected.
  • a hyperspectral image 20 can be restored from the compressed image 10 acquired by the image sensor 160. Details of the method for restoring the hyperspectral image 20 are disclosed in Patent Document 1. The entire disclosure of Patent Document 1 is incorporated herein by reference.
  • Compressed images and hyperspectral images may be generated by imaging using a filter array 110 including the above-mentioned multiple optical filters, i.e., a method other than imaging using an encoding mask.
  • the imaging device 100 may be configured such that the light receiving characteristics of the image sensor 160 are changed for each pixel by processing the image sensor 160.
  • a compressed image can be generated by imaging using the processed image sensor 160, as in the above example. That is, a compressed image may be generated by an imaging device configured such that the filter array 110 is built into the image sensor 160.
  • the encoded information corresponds to the light receiving characteristics of the image sensor 160.
  • a configuration may be adopted in which an optical element such as a metalens is introduced into at least a part of the optical system 140, causing the optical characteristics of the optical system 140 to vary spatially and wavelength-wise, thereby compressing the spectral information.
  • a compressed image can also be generated by an imaging device including such a configuration.
  • the encoded information corresponds to the optical characteristics of the optical element such as a metalens.
  • the intensity of the incident light may be modulated for each wavelength by using an imaging device 100 having a configuration different from that using the filter array 110, to generate the compressed image 10 and the hyperspectral image 20.
  • the present disclosure also includes a configuration for generating a restored image containing a greater number of signals (e.g., the number of pixels) than the number of signals contained in the compressed image 10, based on encoded information corresponding to the optical response characteristics of the imaging device 100, which includes multiple light-receiving regions having different optical response characteristics, and the compressed image 10 generated by the imaging device 100.
  • the optical response characteristics may correspond to the light-receiving characteristics of an image sensor, or may correspond to the optical characteristics of an optical element.
  • the color sample 70A shown in FIG. 6A includes six stripes of different colors extending vertically and six stripes of different colors extending horizontally.
  • the colors of the vertically extending stripes are, from left to right, yellow (Y), blue (B), magenta (M), green (G), cyan (C), and red (R).
  • the colors of the horizontally extending stripes are, from top to bottom, red (R), cyan (C), green (G), magenta (M), blue (B), and yellow (Y).
  • Each horizontally extending stripe of a different color is connected to a vertically extending stripe of the same color, and is located behind each vertically extending stripe of a different color.
  • a compressed image 10A as shown in FIG. 6B is obtained.
  • an enlarged view of the area surrounded by the dotted line frame in FIG. 6B is shown.
  • this compressed image 10A the difference in luminance value between the green (G) area and the magenta (M) area is small, and the boundary between these areas is unclear.
  • the boundary between the green (G) area and the blue (B) area is clear.
  • the boundary between two areas that should be of different colors may be unclear.
  • the accuracy of restoring the hyperspectral image based on the compressed image 10A decreases. For example, problems such as blurred color boundaries in the restored image may occur.
  • the inventors investigated the configuration of an imaging system to solve this problem.
  • the inventors discovered that the above problem could be solved by performing imaging using an illumination device 300 that emits illumination light with non-uniform spectral intensity.
  • the lighting device 300 in this embodiment is configured to emit illumination light with non-uniform spectral intensity.
  • non-uniform spectral intensity means that the light intensities of at least two wavelength bands among the multiple wavelength bands included in the target wavelength range are different. That is, when the multiple wavelength bands included in the target wavelength range include a first wavelength band and a second wavelength band, the light intensity of the first wavelength band included in the illumination light is different from the light intensity of the second wavelength band included in the illumination light.
  • the light intensity of the i-th wavelength band i is an integer of 1 or more
  • the light intensities of the two wavelength bands are different means that the difference in their light intensities is greater than 20% of the light intensity of the wavelength band with the higher intensity.
  • the light intensity of the first wavelength band and the light intensity of the second wavelength band may differ by 30% or more, or may differ by 40% or more.
  • the target wavelength range may be the wavelength range that is the subject of image restoration, or the wavelength range of light that is the subject of detection by the image sensor. For example, if each band has a bandwidth of 10 nm and 30 bands of images of 400 nm to 410 nm, 410 nm to 420 nm, ..., 690 nm to 700 nm are restored from a compressed image, the target wavelength range may be 400 nm to 700 nm.
  • the target wavelength range may be 500 nm to 800 nm.
  • FIG. 8 is a block diagram showing a more detailed configuration example of the imaging system 1000 in this embodiment.
  • the imaging system shown in FIG. 8 includes an imaging device 100, a processing device 200, a lighting control circuit 350, a lighting device 300, a display device 400, and an input user interface (UI) 500.
  • UI input user interface
  • the imaging device 100 includes an image sensor 160 and a sensor control circuit 150 that controls the image sensor 160. Although not shown in FIG. 8, the imaging device 100 also includes a filter array 110 and at least one optical system 140, as shown in FIGS. 1A to 1D. The filter array 110 and the optical system 140 may be arranged in any of the arrangements shown in FIGS. 1A to 1D.
  • the filter array 110 is an example of an optical element that includes multiple regions with different transmission spectra.
  • the image sensor 160 receives light whose intensity has been modulated by the filter array 110 according to the wavelength for each region, and obtains a monochrome image based on the light, i.e., a compressed image. The value of each pixel of the compressed image is superimposed with information of multiple wavelength bands included in the target wavelength range.
  • the processing device 200 includes a reconstruction calculation circuit 250 and a memory 210 such as a RAM and a ROM.
  • the reconstruction calculation circuit 250 is an integrated circuit that includes one or more processors such as a CPU or a GPU.
  • the reconstruction calculation circuit 250 performs a reconstruction calculation based on the compressed image output from the image sensor 160. This reconstruction calculation corresponds to the calculation shown in the above-mentioned formula (2).
  • the image capturing and restoration processing are performed according to the instructions input from the input UI 500.
  • the restoration calculation circuit 250 performs restoration calculation for all wavelength bands included in the target wavelength range, or for a portion of wavelength bands specified via the input UI.
  • the memory 210 stores a computer program executed by a processor included in the restoration calculation circuit 250, various data referenced by the restoration calculation circuit 250, and various data generated by the restoration calculation circuit 250.
  • the memory 210 stores mask data reflecting the spatial distribution of the spectral transmittance of the filter array 110 in the imaging device 100.
  • the mask data is data including information representing the matrix in the above formulas (1) and (2), or information for deriving the matrix (hereinafter also referred to as "mask matrix information").
  • the mask matrix information can be information in a matrix format or a format similar to a matrix having multiple elements corresponding to the spatial distribution of the transmittance of the filter array 110 for each of multiple unit bands included in the target wavelength range.
  • the mask data is created in advance and stored in the memory 210.
  • the illumination device 300 is a device that emits illumination light to be irradiated onto an object.
  • the illumination device 300 includes one or more light sources.
  • the illumination device 300 may be equipped with an optical filter that modulates the intensity of the light emitted from the light source according to the wavelength.
  • the illumination device 300 may be configured to emit illumination light having a non-uniform spectral intensity that enhances the edges of the object in the compressed image.
  • the illumination device 300 may be equipped with a configuration that changes the spectrum of the illumination light.
  • the illumination device 300 may be equipped with multiple types of light sources with different emission spectra.
  • the illumination device 300 may be equipped with multiple optical filters with different transmission spectra and a mechanical mechanism that inserts one optical filter selected from the optical filters into the optical path of the illumination light.
  • the lighting control circuit 350 is a circuit that controls the operation of the lighting device 300.
  • the lighting control circuit 350 causes the lighting device 300 to emit illumination light in synchronization with the image capture by the image sensor 160.
  • the lighting control circuit 350 causes the lighting device 300 to emit illumination light.
  • the lighting control circuit 350 is a circuit independent of the processing device 200, but may be included in the processing device 200.
  • the display device 400 includes an image processing circuit 420 and a display 430.
  • the image processing circuit 420 performs necessary processing on the image restored by the restoration calculation circuit 250 and displays it on the display 430.
  • the display 430 may be any type of display, such as a liquid crystal or organic LED.
  • the input UI 500 includes hardware and software for the user to instruct shooting and to set various conditions such as imaging conditions and restoration conditions.
  • the input UI 500 may include input devices such as a keyboard and a mouse.
  • the input UI 500 may be realized by a device capable of both input and output, such as a touch screen. In that case, the touch screen may also function as the display 430.
  • the imaging conditions may include conditions such as resolution, gain, and exposure time.
  • the restoration conditions may include conditions such as a condition for specifying multiple wavelength bands to be restored, and a condition such as the number of calculations.
  • the input imaging conditions are sent to the control circuit 150 of the imaging device 100.
  • the control circuit 150 causes the image sensor 160 to perform imaging according to the imaging conditions. As a result, the image sensor 160 generates a compressed image.
  • the input restoration conditions are also sent to the restoration calculation circuit 250.
  • the restoration calculation circuit 250 acquires mask data from the memory 210 according to the set restoration conditions, and performs restoration processing based on the mask data and the compressed image. This allows the reconstruction calculation circuit 250 to generate a spectral image (i.e., a reconstructed image) for the specified wavelength bands.
  • the generated spectral image is sent to the image processing circuit 420.
  • the image processing circuit 320 causes the display 330 to display the restored image for each of the multiple wavelength bands.
  • the image processing circuit 420 may display the spectral image on the display 430 after performing processing such as, for example, determining the layout on the screen, linking with band information, or coloring according to wavelength.
  • the mask data in this example includes information on a mask image for deriving a spatial distribution of transmittance for each of a plurality of unit bands included in the target wavelength range.
  • the mask data in this example includes a mask image for each of a large number of unit bands divided every 1 nm. Each unit band is specified by a lower limit wavelength and an upper limit wavelength.
  • Each mask image shown in FIG. 9 is acquired by capturing an image of a background having the color of the corresponding unit band by image sensor 120 through filter array 110. Such mask image data is recorded in advance for each unit band. Note that the width of each unit band is not limited to 1 nm and can be determined to any value.
  • Each element of the above matrix H is determined based on the pixel value corresponding to each of a plurality of pixels included in each mask image shown in FIG. 9.
  • Each pixel value may be normalized by the maximum bit number set for that pixel value and determined as each element of matrix H.
  • FIG. 10 is a block diagram showing another example of the configuration of the imaging system 1000.
  • the processing device 200 in this example further includes an edge detection circuit 270.
  • the edge detection circuit 270 generates an edge image in which the edges of the compressed image are emphasized, based on the compressed image output from the image sensor 160.
  • the processing device 200 causes the display device 400 to display the generated edge image.
  • FIG. 11 is a flowchart showing an example of the operation of the processing device 200 in this embodiment.
  • the processing device 200 acquires a compressed image of an object illuminated with non-uniform light from the illumination device 300 from the imaging device 100 (step S110).
  • the processing device 200 detects edges of the compressed image (step S120).
  • the edge detection can be performed using an edge detection algorithm that uses a filter such as a Sobel filter, a Laplacian filter, or a Canny filter on the compressed image.
  • the edge detection may be performed from the compressed image using a learned model that has been trained in advance by machine learning such as deep learning.
  • the processing device 200 Based on the result of the edge detection, the processing device 200 generates an edge image in which the edges in the compressed image are emphasized (step S130).
  • the processing device 200 causes the display 430 to display the compressed image and the edge image (step S140).
  • FIG. 12 is a diagram showing an example of a method for generating an edge image.
  • the processing device 200 generates an edge image based on mask data reflecting the spatial distribution of the transmission spectrum of the filter array 110 and the compressed image. Specifically, the processing device 200 generates an edge image based on an image converted from the compressed image based on the average pixel value of each pixel of the mask image for each band included in the mask data.
  • Part (a) of FIG. 12 shows a schematic representation of a plurality of mask images. For simplicity, a 4 ⁇ 4 pixel region in the mask image is shown schematically.
  • Part (b) of FIG. 12 shows an example of the average pixel value for each pixel of the plurality of mask images.
  • the average pixel value of each pixel corresponds to the average transmittance of the region of the filter array 110 corresponding to the pixel (i.e., the average transmittance for each band).
  • Part (c) of FIG. 12 shows a schematic representation of a compressed image output from the imaging device 100.
  • Part (d) of FIG. 12 shows a schematic representation of an image obtained by multiplying the pixel value of each pixel of the compressed image by the reciprocal of the average pixel value (i.e., the average transmittance) of the corresponding pixel in the mask image.
  • the processing device 200 may detect edges from an image in which the compressed image is weighted by the reciprocal of the average pixel value for each pixel of the mask image, and generate an edge image as shown in part (e) of FIG. 12. This method can reduce the influence of the transmission spectrum of the filter array 110 differing from region to region. This makes it possible to detect the edges of the object more accurately than a method of directly detecting edges from the compressed image.
  • FIG. 14 is a diagram showing yet another example of a method for generating an edge image.
  • the processing device 200 generates a single image by weighting and adding up the pixel values of each pixel of the restored images of all bands contained in the restored spectral image, as shown in parts (c) and (d) of FIG. 14, and generates an edge image from that image.
  • the weighting and addition may be performed by simply averaging the pixel values of each pixel (i.e., the weights are constant), or weighting may be performed according to the relative visual sensitivity of humans, for example by increasing the weight of the band corresponding to green.
  • the processing device 200 may generate an edge image based on mask data reflecting the spatial distribution of the transmission spectrum of the filter array 110 and one or more restored images included in the multiple restored images. This allows the edge of the object to be detected more accurately.
  • the processing device 200 can be configured to display the compressed image and edge image on the display device 400. This allows the user to determine whether the edges in the compressed image are clear or not, and if the edges are unclear, adjust the lighting device 300 to change the spectrum of the illumination light so that the edges become clearer. Such adjustment of the lighting device 300 may be performed manually or automatically by the processing device 200.
  • FIG. 16 is a diagram showing an example in which a compressed image 10 and an edge image 30 are displayed on the same screen.
  • the processing device 200 may display the compressed image 10 and the edge image 30 side by side on the display device 400 as shown in FIG. 16.
  • the compressed image 10 and the edge image 30 are displayed next to each other, but they may be displayed at positions apart on the screen.
  • the timing at which the compressed image 10 and the edge image 30 start to be displayed may be simultaneous or different.
  • the processing device 200 may be configured to display the compressed image 10 and the edge image 30 on the same screen.
  • the edge image 30 is an image in which the pixels of the edge region and the pixels of the other region are color-coded, as in the example of FIG. 15.
  • the edge image 30 may be an image in which the pixels of the edge region are expressed in a color that can be distinguished from the pixels of the other region, such as black, white, red, or blue.
  • FIG. 17 is a diagram showing yet another example of displaying a compressed image and an edge image.
  • imaging is performed multiple times using illumination light of different spectra.
  • the processing device 200 causes the display device 400 to display a plurality of compressed images 10A and 10B obtained by imaging multiple times and a plurality of edge images 30A and 30B generated based on them side by side.
  • the edge images 30A and 30B shown in FIG. 17 also display shading reflecting the luminance values of each pixel of the compressed images 10A and 10B together with solid lines indicating edges, but this shading does not have to be displayed.
  • the processing device 200 may display the compressed images 10A and 10B and the edge images 30A and 30B at separate positions on the same screen.
  • the display start timing of the compressed images 10A and 10B and the edge images 30A and 30B may be simultaneous or different.
  • the illumination device 300 is configured to emit a first illumination light and a second illumination light having a different spectral shape from the first illumination light.
  • spectral shape refers to the shape of a spectrum (i.e., a wavelength distribution of light intensity) in which the intensity of each wavelength band is normalized by the intensity of a certain reference wavelength band.
  • spectral intensity the intensity of each wavelength band in an unnormalized spectrum. It is interpreted that the spectral shape is the same between a spectrum and a spectrum in which the intensity of each wavelength band in the spectrum is uniformly multiplied by a constant.
  • the imaging device 100 is configured to generate a first compressed image 10A by receiving reflected light from an object due to the first illumination light, and to generate a second compressed image 10B by receiving reflected light from an object due to the second illumination light.
  • the processing device 200 is configured to generate a first edge image 30A based on the first compressed image 10A, and to generate a second edge image 30B based on the second compressed image 10B.
  • the processing device 200 causes the display device 400 to display, side by side, a plurality of compressed images 10A and 10B acquired by multiple imaging using illumination light of different spectra, and difference images 31A and 31B showing the difference between a plurality of edge images generated based on them.
  • the processing device 200 can generate the difference images 31A and 31B by deriving, for each pixel, the difference between the pixel value of a pixel included in the edge image 30A shown in FIG. 17 and the pixel value of a pixel included in the edge image 30B and corresponding 1:1 to each of the pixels included in the edge image 30A.
  • FIG. 17 the example of FIG.
  • the processing device 200 sequentially switches between a plurality of compressed images 10A and 10B acquired by multiple imaging using illumination light of different spectra, a plurality of edge images 30A and 30B generated based on them, and difference images 31A and 31B showing the difference between the edge images, and displays them on the display device 400.
  • the processing device 200 may sequentially switch between and display two or more images selected from a compressed image, a superimposed image, an edge image, and a difference image.
  • the processing device 200 may also cause the display device 400 to display one or more restored images contained in a hyperspectral image (i.e., multiple restored images) restored based on a compressed image.
  • a hyperspectral image i.e., multiple restored images
  • the illumination control circuit 350 can change the spectrum of the illumination light by switching the light source in the illumination device 300 or switching the optical filter that modulates the light emitted from the light source.
  • the functions of the edge detection circuit 270 and the edge determination circuit 272 may be realized by a single processing circuit.
  • the edge determination circuit 272 determines whether the edge image satisfies a predetermined condition, and if the condition is not satisfied, instructs the illumination control circuit 350 to change the spectrum of the illumination light.
  • the condition may be, for example, that the maximum length of the continuous, uninterrupted edges detected from the compressed image is equal to or greater than a certain value.
  • the condition may be that the similarity with a reference edge image generated from a reference image of an object prepared in advance is equal to or greater than a certain value.
  • the reference image may be an image of an object of the same type as the object to be inspected, or CAD data of the object.
  • the similarity between an edge image and a reference edge image can be calculated, for example, by the following method.
  • the luminance of the edge image is E(u, v)
  • the luminance of the reference edge image is R(u, v).
  • (u, v) represent the coordinates of a pixel
  • the luminance of a pixel that corresponds to an edge is 1, and the luminance of a pixel that does not correspond to an edge is 0.
  • the similarity S can be calculated by the following formula.
  • the similarity S has a value of 1 if there is a perfect match and 0 if there is a perfect difference.
  • FIG. 21 is a flowchart showing an example of the operation of the processing device 200.
  • the processing device 200 acquires a compressed image of an object illuminated with light from the illumination device 300 from the imaging device 100 (step S210).
  • the processing device 200 detects edges of the compressed image (step S220).
  • the edge detection method is as described above.
  • the processing device 200 Based on the result of the edge detection, the processing device 200 generates an edge image in which the edges in the compressed image are emphasized (step S230).
  • the processing device 200 determines whether or not the edge image satisfies a predetermined condition (step S240). If the condition is satisfied, the processing device 200 restores a hyperspectral (HS) image based on the compressed image (step S250).
  • HS hyperspectral
  • the processing device 200 determines, based on the compressed image acquired by the image sensor 160, whether or not it is necessary to switch from a mode in which the illumination device 300 emits illumination light having a certain spectrum to a mode in which the illumination device 300 emits illumination light having another spectrum. For example, the processing device 200 determines, based on the amount of edges contained in the compressed image, whether or not it is necessary to switch the illumination light emission mode. Alternatively, the processing device 200 may acquire a reference image in which the edges of the object are emphasized, which is generated by a method different from the imaging using the imaging device 100, and determine, based on a comparison between the edge image generated based on the compressed image and the reference image, whether or not it is necessary to switch the illumination light emission mode. Such an operation makes it possible to acquire a compressed image with clear edges, thereby improving the restoration accuracy of the hyperspectral image based on the compressed image.
  • the processing device 200 determines which of the multiple compressed images to use as a basis for generating a hyperspectral image, i.e., multiple restored images, based on multiple compressed images generated by sequentially capturing images using multiple illumination lights with different spectra. For example, the processing device 200 determines which of the multiple compressed images to use as a basis for generating multiple restored images, based on a comparison of the amount of edges contained in multiple edge images generated from the multiple compressed images. Through such processing, multiple restored images are generated based on compressed images with clear edges, thereby improving the restoration accuracy.
  • the processing device 200 generates multiple restored images based on compressed images acquired by imaging using illumination light with a non-uniform spectrum in the target wavelength range.
  • the processing device 200 may also generate multiple restored images using illumination light with a uniform spectrum.
  • imaging using illumination light with a non-uniform spectrum may be performed to determine optimal restoration parameters.
  • the illumination device 300 may be configured to emit a first illumination light and a second illumination light having different spectral shapes from each other, and a third illumination light having a constant light intensity in the target wavelength range.
  • illumination light with a constant light intensity may be expressed as "illumination light with a substantially uniform spectrum”.
  • the imaging device 100 may be configured to generate a first compressed image by imaging using the first illumination light, generate a second compressed image by imaging using the second illumination light, and generate a third compressed image by imaging using the third illumination light.
  • the processing device 200 may be configured to generate multiple decompressed images based on the first compressed image, the second compressed image, and the third compressed image.
  • the processing device 200 includes an edge image synthesis circuit 274 instead of the edge determination circuit 272 shown in FIG. 20.
  • the edge image synthesis circuit 274 generates one synthetic edge image from a plurality of edge images based on a plurality of compressed images captured using illumination light with a plurality of types of non-uniform spectrum.
  • the edge image synthesis circuit 274 may be configured to generate a synthetic edge image, for example, by performing an OR operation between the plurality of edge images.
  • Such a synthetic edge image includes all edges detected in at least one edge image.
  • the edge image synthesis circuit 274 may generate a synthetic edge image by synthesizing pixels detected as edges in a threshold number or more of the plurality of edge images as edge pixels.
  • multiple types of illumination light with different spectra are sequentially emitted to detect the respective edges, and multiple edge images are integrated to make it easier to detect edges throughout the entire image. Furthermore, since illumination light with a uniform spectrum is advantageous for image restoration, the processing device 200 improves restoration accuracy by switching between illumination light for edge detection and illumination light for image restoration.
  • step S230 the processing device 200 determines whether a predetermined number of edge images have been generated (step S440).
  • the predetermined number can be set to, for example, a number equal to or less than the number of types of spectra that the lighting device 300 can emit. If this determination is Yes, proceed to step S450. If this determination is No, proceed to step S260.
  • the processes of steps S210, S220, S230, S440, and S260 are repeated until a determination of Yes is made in step S440.
  • the processing device 200 When a predetermined number of edge images have been generated, the processing device 200 generates a composite edge image by synthesizing the generated edge images using the method described above (step S450). Next, the processing device 200 obtains a compressed image of the object illuminated with light having a substantially uniform spectrum in the target wavelength range. The processing device 200 generates a hyperspectral image (i.e., a plurality of restored images) based on the compressed image of the object illuminated with light having a substantially uniform spectrum and the synthetic edge image (step S470). As described above, the hyperspectral image can be generated by, for example, making the weighting coefficient ⁇ of the total variation in pixels corresponding to edges in the synthetic edge image smaller than the values in other pixels.
  • a hyperspectral image i.e., a plurality of restored images
  • edge regions can be detected with high accuracy from multiple compressed images acquired by imaging using illumination light with multiple non-uniform spectra, and the detection results can be used to generate high-quality hyperspectral images.
  • the process of detecting edges based on one or more compressed images and the process of generating multiple restored images are executed by one processing device 200.
  • the processing device for detecting edges and the processing device for generating restored images may be distributed.
  • the imaging system 1000 may include a processing device for detecting edges but not a processing device for generating restored images.
  • the process of generating restored images may be executed by an external device such as a cloud server connected to the imaging system 1000 via a network.
  • the imaging system 1000 may include a communication device for communicating with such external devices.
  • the present disclosure also includes a system that generates a compressed image and an edge image by simulation based on data prepared in advance, without actually preparing an illumination device 300 and an object, and determines the optimal illumination light spectrum for an expected object.
  • a system that generates a compressed image and an edge image by simulation based on data prepared in advance, without actually preparing an illumination device 300 and an object, and determines the optimal illumination light spectrum for an expected object.
  • FIG. 25 is a block diagram showing an example configuration of a system 2000 that determines the optimal illumination light spectrum for an assumed object.
  • This system 2000 includes a processing device 200, a display device 400, and an input UI 500.
  • the processing device 200 includes a memory 210 and a processing circuit 280.
  • the memory 210 stores second data related to the spectrum of the illumination light irradiated to the object, and third data related to the spectral sensitivity characteristics of each pixel of the imaging device 100 that images the object.
  • the second data may be, for example, data indicating the spectrum of multiple illumination lights that can be emitted by the aforementioned illumination device 300.
  • the third data may be, for example, mask data reflecting the spatial distribution of the spectral transmittance of the aforementioned filter array 110.
  • the second data and the third data may be input from the input UI 500.
  • each of the first data, second data, and third data can be data (e.g., a table) indicating a function that depends on the position coordinates (x, y) and the wavelength ⁇ .
  • the first data indicates the spectral reflectance ⁇ (x, y, ⁇ ) of the object
  • the second data indicates the intensity of the illumination light I(x, y, ⁇ )
  • the third data indicates the spectral sensitivity s(x, y, ⁇ ) of the imaging device 100.
  • the third data corresponds to the mask data described above.
  • FIG. 26 is a diagram showing an example of the spectral reflectance of an object 70.
  • FIG. 26 illustrates the spectral reflectance at two locations on the surface of the object 70.
  • the spectral reflectance can vary depending on the position on the object 70. Therefore, the first data indicating the spatial distribution of the spectral reflectance can be data having a reflectance value for the number of wavelengths for each pixel. Alternatively, the first data can be data having a reflectance value for the number of wavelengths for each voxel in the CAD data.
  • FIG. 27 is a flowchart showing an example of processing executed by the processing circuit 280.
  • the processing circuit 280 acquires the spectral reflectance data (first data) of the object, the spectral data (second data) of the illumination light, and the mask data (third data) from the input UI 500 or the memory 210 (step S510).
  • the processing circuit 280 generates an estimated image by the above method based on the spectral reflectance data of the object, the spectral data of the illumination light, and the mask data (step S520).
  • the processing circuit 280 detects edges from the estimated image (step S530). The process of detecting edges is the same as the process of step S120 shown in FIG. 11.
  • the processing circuit 280 generates an estimated edge image based on the result of the edge detection (step S540).
  • the process of generating the estimated edge image is the same as the process of step S130 shown in FIG. 11.
  • the processing circuit 280 causes the display 430 to display the estimated edge image (step S550).
  • the above operations allow the user to determine whether or not the illumination light spectrum is suitable for generating a spectral image of the object, based on the displayed estimated edge image. If the illumination light spectrum is not suitable, the user may operate the input UI 500 to select another illumination light spectrum and cause the processing circuit 280 to execute the operations shown in FIG. 27 again. By repeating such operations, the optimal illumination light spectrum can be determined.
  • the processing circuit 280 may perform a process of changing the spectral data of the illumination light until the estimated edge image satisfies a predetermined condition. That is, the generation of an estimated image and the generation of an estimated edge image may be repeated by changing the spectral data of the illumination light.
  • the predetermined condition is the same as the condition in step S240 shown in FIG. 21. In this case, the spectral data of multiple types of illumination light is stored in advance in the memory 210.
  • the processing circuit 280 may generate multiple estimated images based on the spectral data of multiple illumination lights, and determine the illumination light that most emphasizes edges as the optimal illumination light based on a comparison of the multiple estimated images, as in the example shown in FIG. 22.
  • the processing device 200 may display information indicating optimal illumination light on the display 430 instead of or in addition to the edge image. By displaying such information, the user can select optimal illumination light.
  • the illumination device is configured to emit the illumination light as a first illumination light, and further to emit a second illumination light having a spectral shape different from that of the first illumination light;
  • the imaging device is configured to generate the compressed image as a first compressed image, and further to receive reflected light from the object caused by the second illumination light and generate a second compressed image in which image information of the plurality of wavelength bands is compressed as a single piece of image information;
  • the processing device determines, based on the first compressed image, whether or not it is necessary to switch from a mode in which the illumination device emits the first illumination light to a mode in which the illumination device emits the second illumination light.
  • the processing device includes: acquiring a reference image of the object, the reference image having enhanced edges, the reference image being generated by a method other than the imaging using the imaging device; The imaging system described in Technology 10, wherein based on a comparison between the first edge image and the reference image, it is determined whether or not it is necessary to switch from a mode in which the illumination device emits the first illumination light to a mode in which the illumination device emits the second illumination light.
  • the illumination device is configured to emit the illumination light as a first illumination light, and further to emit a second illumination light having a spectral shape different from that of the first illumination light;
  • the imaging device is configured to generate the compressed image as a first compressed image, and further to receive reflected light from the object caused by the second illumination light and generate a second compressed image in which image information of the plurality of wavelength bands is compressed as a single piece of image information;
  • the processing device determines, based on the first compressed image and the second compressed image, whether the first compressed image or the second compressed image is used to generate the plurality of decompressed images;
  • the imaging system according to claim 1 or 2.
  • the processing device includes: generating a first edge image based on the first compressed image, the first edge image being enhanced with edges of the first compressed image; generating a second edge image based on the second compressed image, the second edge image being enhanced with edges of the second compressed image; determining whether the first compressed image or the second compressed image is to be used to generate the plurality of decompressed images based on a comparison between an amount of edges included in the first edge image and an amount of edges included in the second edge image; 13.
  • the imaging system according to claim 12.
  • the illumination device is configured to emit the illumination light as a first illumination light, and further emit a second illumination light having a spectral shape different from that of the first illumination light, and a third illumination light having a constant light intensity in the target wavelength range;
  • the imaging device is configured to generate the compressed image as a first compressed image, to receive reflected light from the object caused by the second illumination light and to generate a second compressed image in which image information of the plurality of wavelength bands is compressed as a single piece of image information, and to receive reflected light from the object caused by the third illumination light and to generate a third compressed image in which image information of the plurality of wavelength bands is compressed as a single piece of image information;
  • the processing device generates the plurality of decompressed images based on the first compressed image, the second compressed image, and the third compressed image.
  • the imaging system according to claim 1 or 2.
  • the processing device includes: generating a first edge image based on the first compressed image, the first edge image being enhanced with edges of the first compressed image; generating a second edge image based on the second compressed image, the second edge image being enhanced with edges of the second compressed image; determining restoration parameters for generating the plurality of restored images based on the first edge image and the second edge image; generating the plurality of decompressed images based on the decompression parameters and the third compressed image; 15.
  • a computer-implemented method comprising: Emitting illumination light to be irradiated onto an object to an illumination device; an imaging device that receives reflected light from the object due to the illumination light and generates a compressed image in which image information of a plurality of wavelength bands included in a target wavelength range is compressed into a single piece of image information, and generates the compressed image; Including, the plurality of wavelength bands includes a first wavelength band and a second wavelength band; an intensity of the light of the first wavelength band contained in the illumination light is different from an intensity of the light of the second wavelength band contained in the illumination light; method.
  • REFERENCE SIGNS LIST 10 Compressed image 20 Hyperspectral image 20W1 , 20W2 , ..., 20WN restored image 30 Edge image 40 Superimposed image 70 Object 100 Imaging device 110 Filter array 140, 140A, 140B Optical system 160 Image sensor 200 Processing device 210 Memory 220 Sensor control circuit 250 Restoration calculation circuit 270 Edge detection circuit 280 Processing circuit 300 Illumination device 350 Illumination control circuit 400 Output device 420 Image processing circuit 430 Display 500 Input UI 1000 Imaging System

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Physics & Mathematics (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Computing Systems (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Investigating Or Analysing Materials By Optical Means (AREA)
PCT/JP2024/009554 2023-03-31 2024-03-12 撮像システムおよび方法 Ceased WO2024203282A1 (ja)

Priority Applications (2)

Application Number Priority Date Filing Date Title
JP2025510264A JPWO2024203282A1 (https=) 2023-03-31 2024-03-12
US19/329,640 US20260012711A1 (en) 2023-03-31 2025-09-16 Imaging system and method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2023057197 2023-03-31
JP2023-057197 2023-03-31

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US19/329,640 Continuation US20260012711A1 (en) 2023-03-31 2025-09-16 Imaging system and method

Publications (1)

Publication Number Publication Date
WO2024203282A1 true WO2024203282A1 (ja) 2024-10-03

Family

ID=92904593

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2024/009554 Ceased WO2024203282A1 (ja) 2023-03-31 2024-03-12 撮像システムおよび方法

Country Status (3)

Country Link
US (1) US20260012711A1 (https=)
JP (1) JPWO2024203282A1 (https=)
WO (1) WO2024203282A1 (https=)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016130727A (ja) * 2015-01-08 2016-07-21 パナソニックＩｐマネジメント株式会社 撮像装置
WO2020080045A1 (ja) * 2018-10-15 2020-04-23 パナソニックＩｐマネジメント株式会社 物体認識方法、車両制御方法、情報表示方法、および物体認識装置
WO2021246192A1 (ja) * 2020-06-05 2021-12-09 パナソニックＩｐマネジメント株式会社 信号処理方法、信号処理装置、および撮像システム
WO2022075094A1 (ja) * 2020-10-05 2022-04-14 パナソニックＩｐマネジメント株式会社 処理装置、システム、方法、コンピュータプログラム
US20220201187A1 (en) * 2020-12-23 2022-06-23 Microsoft Technology Licensing, Llc Depth image sensor with always-depleted photodiodes

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2016130727A (ja) * 2015-01-08 2016-07-21 パナソニックＩｐマネジメント株式会社 撮像装置
WO2020080045A1 (ja) * 2018-10-15 2020-04-23 パナソニックＩｐマネジメント株式会社 物体認識方法、車両制御方法、情報表示方法、および物体認識装置
WO2021246192A1 (ja) * 2020-06-05 2021-12-09 パナソニックＩｐマネジメント株式会社 信号処理方法、信号処理装置、および撮像システム
WO2022075094A1 (ja) * 2020-10-05 2022-04-14 パナソニックＩｐマネジメント株式会社 処理装置、システム、方法、コンピュータプログラム
US20220201187A1 (en) * 2020-12-23 2022-06-23 Microsoft Technology Licensing, Llc Depth image sensor with always-depleted photodiodes

Also Published As

Publication number Publication date
US20260012711A1 (en) 2026-01-08
JPWO2024203282A1 (https=) 2024-10-03

Similar Documents

Publication Publication Date Title
JP7720532B2 (ja) 信号処理方法、信号処理装置、および撮像システム
JP7668452B2 (ja) 信号処理方法、信号処理装置、および撮像システム
JP2022002398A (ja) 撮像装置および分光システム
JP7829156B2 (ja) 画像処理装置、撮像システム、および復元画像の誤差を推定する方法
JP2016130727A (ja) 撮像装置
US20230419478A1 (en) Method and apparatus for detecting foreign object included in inspection target
US20240422443A1 (en) Imaging system
JP7570014B2 (ja) 撮像装置
US20240127574A1 (en) Imaging system, method used in imaging system, and storage medium storing computer program used in imaging system
WO2023282069A1 (ja) 信号処理装置および信号処理方法
WO2023286613A1 (ja) フィルタアレイ、光検出装置、および光検出システム
WO2024203282A1 (ja) 撮像システムおよび方法
US20240311971A1 (en) Signal processing method, non-volatile computer-readable recording medium, and system
WO2024043009A1 (ja) 信号処理方法および信号処理装置
US20230069927A1 (en) Filter array and light detection system
WO2024195499A1 (ja) 撮像システム、行列データ、および行列データの生成方法
US20250191161A1 (en) Information processing method and imaging system
WO2025100176A1 (ja) 画像を補正する方法およびシステム
WO2026088780A1 (ja) 情報処理方法、処理装置、およびコンピュータプログラム
JP2024020922A (ja) 復元画像の評価方法および撮像システム、
WO2025192485A1 (ja) 撮像システムおよび撮像方法
WO2025115710A1 (ja) 画像の補正に用いられる方法、およびその方法を実行する処理回路
WO2025084081A1 (ja) 撮像システムおよびそれに用いられる方法
WO2024142683A1 (ja) 光検出装置、光検出システム、およびフィルタアレイ
WO2025192316A1 (ja) 撮像装置、撮像システム、および撮像方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 24779412

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2025510264

Country of ref document: JP

Kind code of ref document: A

WWE Wipo information: entry into national phase

Ref document number: 2025510264

Country of ref document: JP

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 24779412

Country of ref document: EP

Kind code of ref document: A1