Classifications

• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/70—Arrangements for image or video recognition or understanding using pattern recognition or machine learning
  • G06V10/764—Arrangements for image or video recognition or understanding using pattern recognition or machine learning using classification, e.g. of video objects
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T11/00—Two-dimensional [2D] image generation
  • G06T11/10—Texturing; Colouring; Generation of textures or colours
• • G—PHYSICS
  • G01—MEASURING; TESTING
  • G01N—INVESTIGATING OR ANALYSING MATERIALS BY DETERMINING THEIR CHEMICAL OR PHYSICAL PROPERTIES
  • G01N21/00—Investigating or analysing materials by the use of optical means, i.e. using sub-millimetre waves, infrared, visible or ultraviolet light
  • G01N21/17—Systems in which incident light is modified in accordance with the properties of the material investigated
  • G01N21/25—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands
  • G01N21/27—Colour; Spectral properties, i.e. comparison of effect of material on the light at two or more different wavelengths or wavelength bands using photo-electric detection ; circuits for computing concentration
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T5/00—Image enhancement or restoration
  • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
  • G06V10/00—Arrangements for image or video recognition or understanding
  • G06V10/10—Image acquisition
  • G06V10/12—Details of acquisition arrangements; Constructional details thereof
  • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
  • G06V10/143—Sensing or illuminating at different wavelengths
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/50—Constructional details
  • H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
  • H04N23/60—Control of cameras or camera modules
• • H—ELECTRICITY
  • H04—ELECTRIC COMMUNICATION TECHNIQUE
  • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
  • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
  • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/10—Image acquisition modality
  • G06T2207/10032—Satellite or aerial image; Remote sensing
  • G06T2207/10036—Multispectral image; Hyperspectral image
• • G—PHYSICS
  • G06—COMPUTING OR CALCULATING; COUNTING
  • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
  • G06T2207/00—Indexing scheme for image analysis or image enhancement
  • G06T2207/20—Special algorithmic details
  • G06T2207/20212—Image combination
  • G06T2207/20224—Image subtraction

Definitions

• the technology of the present disclosure relates to an information processing device, an information processing method, and a program.
• the International Publication No. 2019/151029 pamphlet includes a white light source section that irradiates an object with white light, an imaging section that takes a multispectral image of the object, and a multispectral image of the object irradiated with white light.
• an imaging device includes an object identification unit that identifies a wavelength of light optimal for analyzing the object as an effective wavelength, and a variable wavelength light source that irradiates the object with light of the effective wavelength.
• JP2018-098341A discloses a first pixel including a metal thin film filter that transmits light in a first frequency band, and a first pixel that transmits light in a second frequency band that is wider than the first frequency band.
• An image sensor is disclosed that includes a second pixel that includes a color filter.
• JP 2021-135404A discloses a lens device including an optical system, an optical member, an irradiation device, and a control section.
• the optical system includes a lens that forms an optical image of a subject.
• the optical member is an optical member disposed at or near the pupil position of the optical system, and includes a frame body having a plurality of aperture regions, and a plurality of optical filters disposed in the plurality of aperture regions, A plurality of optical filters including two or more optical filters that transmit light in at least some different wavelength bands, and a plurality of polarization filters arranged in a plurality of aperture areas, the plurality of polarization filters having different polarization directions. and has.
• the irradiation device irradiates the subject with illumination light.
• the control unit controls at least one of the optical system, the optical member, and the irradiation device.
• the control unit changes spectral characteristics of light emitted from the optical system for the plurality of aperture regions.
• Japanese Unexamined Patent Publication No. 2010-025750 discloses an image processing device that processes a multiband image captured using a plurality of bandpass filters.
• the image processing device includes a sub-light source specifying means for specifying a sub-light source different from the main light source, a spectral estimation means for obtaining a spectral image from a multiband image, and a spectral estimation means for specifying a sub-light source specified by the sub-light source specifying means.
• the apparatus further includes an image separating means for separating the spectral image obtained by the spectral estimating means into an image under a main light source and an image under a sub-light source based on the spectral estimation means.
• One embodiment according to the technology of the present disclosure compared to a case where a channel image is generated based on an operation that only includes addition for a first spectral image among a plurality of different spectral images.
• An information processing device, an information processing method, and a program are provided that can realize color adjustment with a high degree of freedom in pseudo-coloring a plurality of spectral images.
• a first aspect of the technology of the present disclosure includes a processor, and the processor generates a channel image for each channel by performing assignment processing for assigning a plurality of different spectral images to different channels.
• An information processing device that generates a first pseudo-color image based on a channel image, and the allocation process includes a process that generates a channel image based on an operation including subtraction on a first spectral image of a plurality of spectral images. It is.
• a second aspect of the technology of the present disclosure is that in the information processing apparatus according to the first aspect, the calculation is a product-sum calculation including the first spectral image, and the subtraction is a product-sum calculation including the first spectral image in the product-sum calculation.
• This is an information processing device realized by including negative values in coefficients.
• a third aspect of the technology of the present disclosure is that in the information processing device according to the second aspect, the coefficient is such that the range of the first pseudo-color image falls within the expression range of the display medium that displays the first pseudo-color image.
• This is an information processing device that is set to a value.
• a fourth aspect of the technology of the present disclosure is the information processing apparatus according to any one of the first to third aspects, in which the plurality of spectral images include polarization information and/or first wavelength information.
• the information processing device includes a plurality of spectral images, and the number of the plurality of spectral images is greater than or equal to the number of channels.
• a fifth aspect of the technology of the present disclosure is the information processing device according to any one of the first to fourth aspects, in which the plurality of spectral images are captured by an image sensor having a polarizer.
• the processor performs alignment processing on the first spectral image, and the information processing device performs subtraction on the first spectral image that has undergone the alignment processing.
• a sixth aspect of the technology of the present disclosure is that in the information processing apparatus according to any one of the first to fifth aspects, the first pseudo-color image is processed by a second pseudo-color image whose calculation includes only addition.
• the information processing apparatus is an image in which second wavelength information has enhanced discrimination with respect to a pseudo-color image.
• the processor in the information processing apparatus according to any one of the first to sixth aspects, the processor generates a plurality of images generated based on the plurality of spectral images.
• This is an information processing device that divides each area into areas and performs calculations based on color information set for a plurality of areas.
• An eighth aspect according to the technology of the present disclosure is an information processing apparatus according to any one of the first to seventh aspects, in which the different channels are channels of three primary colors.
• a ninth aspect of the technology of the present disclosure is the information processing device according to any one of the first to eighth aspects, wherein the processor causes the display device to display the first pseudo-color image.
• This is an information processing device that outputs data for.
• a tenth aspect of the technology of the present disclosure is to generate a channel image for each channel by performing an assignment process for assigning a plurality of different spectral images to different channels, and to generate a channel image for each channel based on the plurality of channel images.
• generating a first pseudo-color image using a plurality of spectral images, and the allocation process includes a process of generating a channel image based on an operation including subtraction on the first spectral image of the plurality of spectral images. It is.
• An eleventh aspect of the technology of the present disclosure is a program for causing a computer to execute specific processing, and the specific processing includes assigning a plurality of different spectral images to different channels.
• the allocation process includes: generating a channel image for each channel; and generating a first pseudocolor image based on the plurality of channel images;
• This program includes processing for generating channel images based on operations including subtraction.
• FIG. 1 is a block diagram illustrating an example of the hardware configuration of an imaging device according to an embodiment.
• FIG. 1 is an exploded perspective view showing an example of a photoelectric conversion element according to an embodiment.
• FIG. 2 is a block diagram illustrating an example of a functional configuration for realizing image display processing according to the embodiment.
• FIG. 3 is a block diagram illustrating an example of the operation of an output value acquisition section and an interference removal processing section according to the embodiment.
• FIG. 3 is a block diagram illustrating an example of the operation of the alignment processing section according to the embodiment.
• FIG. 2 is a block diagram illustrating an example of the operation of an initial image generation unit according to the embodiment.
• FIG. 2 is a block diagram illustrating an example of the operation of an initial image output unit according to the embodiment.
• FIG. 1 is a block diagram illustrating an example of the hardware configuration of an imaging device according to an embodiment.
• FIG. 1 is an exploded perspective view showing an example of a photoelectric conversion element according to an embodiment.
• FIG. 2 is a block diagram illustrating an example of the operation of a region setting determination section and a region setting section according to the embodiment.
• FIG. 2 is a block diagram illustrating an example of the operation of a color setting determination section and a gain setting section according to the embodiment.
• FIG. 2 is a block diagram illustrating an example of the operation of a pseudo color image generation unit according to the embodiment.
• FIG. 2 is a block diagram showing an example of the operation of a pseudo color image output unit and a first example of a pseudo color image according to the embodiment.
• FIG. 7 is a block diagram showing an example of the operation of the pseudo color image output unit and a second example of the pseudo color image according to the embodiment.
• FIG. 7 is a block diagram showing an example of the operation of the pseudo color image output unit and a third example of the pseudo color image according to the embodiment.
• FIG. 7 is a block diagram showing an example of the operation of the pseudo color image output unit and a fourth example of the pseudo color image according to the embodiment.
• 3 is a flowchart illustrating an example of the flow of image display processing according to the embodiment.
• CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor.”
• CCD is an abbreviation for “Charge Coupled Device”.
• NVM is an abbreviation for “Non-Volatile Memory.”
• RAM is an abbreviation for "Random Access Memory.”
• CPU is an abbreviation for "Central Processing Unit.”
• GPU is an abbreviation for “Graphics Processing Unit.”
• EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory.”
• HDD is an abbreviation for "Hard Disk Drive.”
• LiDAR is an abbreviation for “Light Detection and Ranging.”
• TPU is an abbreviation for “Tensor processing unit”.
• SSD is an abbreviation for “Solid State Drive.”
• USB is an abbreviation for “Universal Serial Bus.”
• ASIC is an abbreviation for “Application Specific Integrated Circuit.”
• FPGA is an abbreviation for “Field-Programmable Gate Array.”
• IC is an abbreviation for "Integrated Circuit.”
• a straight line refers to not only a perfect straight line but also an error that is generally allowed in the technical field to which the technology of the present disclosure belongs, and that does not go against the spirit of the technology of the present disclosure. It refers to a straight line that includes the error of.
• the imaging device 10 is a multispectral camera capable of outputting a pseudo-color multispectral image, and includes an optical system 12, an image sensor 14, a control driver 16, an input/output I/F 18, It includes a computer 20, a reception device 22, and a display 24.
• the imaging device is an example of an "information processing device" according to the technology of the present disclosure.
• the optical system 12 includes a first lens 26, a pupil splitting filter 28, and a second lens 30.
• the first lens 26, the pupil splitting filter 28, and the second lens 30 are arranged along the optical axis OA of the imaging device 10 from the subject 4 side to the image sensor 14 side.
• the lenses 30 are arranged in this order.
• the first lens 26 transmits light obtained by the light emitted from the light source 2 reflecting off the subject 4 (hereinafter referred to as "subject light") to the pupil division filter 28.
• the second lens 30 forms an image of the subject light that has passed through the pupil splitting filter 28 onto a light receiving surface 48A of a photoelectric conversion element 48 provided in the image sensor 14.
• the pupil splitting filter 28 has a spectral filter 40 and a polarizing filter 42.
• the spectral filter 40 has filters 44A to 44H
• the polarizing filter 42 has polarizers 46A to 46H.
• the filters 44A to 44H are shown arranged in a straight line along the direction perpendicular to the optical axis OA, but the filters 44A to 44H are arranged in the direction around the optical axis OA. arranged along the
• Filter 44A has a first transmission wavelength band ⁇ 1
• filter 44B has a second transmission wavelength band ⁇ 2
• filter 44C has a third transmission wavelength band ⁇ 3
• filter 44D has a first transmission wavelength band ⁇ 3 . It has four transmission wavelength bands ⁇ 4 .
• the filter 44E has a fifth transmission wavelength band ⁇ 5
• the filter 44F has a sixth transmission wavelength band ⁇ 6
• the filter 44G has a seventh transmission wavelength band ⁇ 7
• the filter 44H has a fifth transmission wavelength band ⁇ 5.
• the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 are mutually different wavelength bands.
• the first transmission wavelength band ⁇ 1 is set to 435 nm
• the second transmission wavelength band ⁇ 2 is set to 495 nm
• the third transmission wavelength band ⁇ 3 is set to 555 nm.
• the fourth transmission wavelength band ⁇ 4 is set to 615 nm.
• the fifth transmission wavelength band ⁇ 5 is set to 675 nm
• the sixth transmission wavelength band ⁇ 6 is set to 735 nm
• the seventh transmission wavelength band ⁇ 7 is set to 795 nm
• the eighth transmission wavelength band ⁇ 8 is set to 855 nm.
• each wavelength band illustrated here is just an example, and the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 may be set to any wavelength band, and may be set to different wavelengths. Preferably, it is a band.
• the filters 44A to 44H will be referred to as "filters 44".
• the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 may be referred to as the “transmission wavelength band ⁇ 8 ” . ”.
• the polarizers 46A to 46H are overlapped with the filters 44A to 44H, respectively.
• the polarizer 46A is a polarizer whose transmission axis is set at an angle of 0°
• the polarizer 46B is a polarizer whose transmission axis is set at an angle of 20°
• the polarizer 46C is a polarizer whose transmission axis is set at an angle of 20°.
• the polarizer 46D is a polarizer with an angle of 40°
• the polarizer 46D is a polarizer with a transmission axis set at an angle of 60°.
• the polarizer 46E is a polarizer with a transmission axis set at an angle of 80°
• the polarizer 46F is a polarizer with a transmission axis set at an angle of 100°
• the polarizer 46G is a polarizer with a transmission axis set at an angle of 100°
• the polarizer 46H is a polarizer whose axis angle is set to 120°
• the polarizer 46H is a polarizer whose transmission axis angle is set to 140°.
• the polarizers 46A to 46H will be referred to as "polarizers 46", respectively.
• the number of filters 44 is eight, but the number of filters 44 may be any number as long as it is equal to or greater than the number of channels (see FIGS. 6 and 10) to be described later.
• the number of polarizers 46 is eight, but the number of polarizers 46 may be any number as long as it is the same as the number of filters 44.
• the image sensor 14 includes a photoelectric conversion element 48 and a signal processing circuit 50.
• the image sensor 14 is, for example, a CMOS image sensor.
• a CMOS image sensor is exemplified as the image sensor 14, but the technology of the present disclosure is not limited to this.
• the image sensor 14 may be another type of image sensor such as a CCD image sensor. The technology of the present disclosure is realized.
• FIG. 1 shows a schematic configuration of the photoelectric conversion element 48.
• FIG. 2 specifically shows the configuration of a part of the photoelectric conversion element 48.
• the photoelectric conversion element 48 has a pixel layer 52, a polarizing filter layer 54, and a spectral filter layer 56.
• the pixel layer 52 has a plurality of pixels 58.
• the plurality of pixels 58 are arranged in a matrix and form a light receiving surface 48A of the photoelectric conversion element 48.
• Each pixel 58 is a physical pixel having a photodiode (not shown), photoelectrically converts the received light, and outputs an electric signal according to the amount of received light.
• pixels 58 provided in the photoelectric conversion element 48 will be referred to as “physical pixels 58" in order to distinguish them from pixels forming a multispectral image. Furthermore, pixels forming an image displayed on the display 24 are referred to as “image pixels.”
• the photoelectric conversion element 48 outputs the electrical signals output from the plurality of physical pixels 58 to the signal processing circuit 50 as image data 120.
• the signal processing circuit 50 digitizes the analog imaging data 120 input from the photoelectric conversion element 48.
• a plurality of physical pixels 58 form a plurality of pixel blocks 60.
• Each pixel block 60 is formed by a total of four physical pixels 58, two in the vertical direction and two in the horizontal direction.
• four physical pixels 58 forming each pixel block 60 are shown arranged in a straight line along a direction perpendicular to the optical axis OA, but as an example, Thus, the four physical pixels 58 are arranged adjacent to the photoelectric conversion element 48 in the vertical and horizontal directions, respectively.
• the polarizing filter layer 54 has polarizers 62A to 62D.
• the polarizer 62A is a polarizer whose transmission axis has an angle of 0°.
• the polarizer 62B is a polarizer whose transmission axis is set at an angle of 45°.
• the polarizer 62C is a polarizer whose transmission axis has an angle of 90°.
• the polarizer 62D is a polarizer whose transmission axis has an angle of 135°.
• polarizers 62 will be referred to as "polarizers 62.”
• the spectral filter layer 56 includes a B filter 64A, a G filter 64B, and an R filter 64C.
• the B filter 64A is a blue band filter that transmits most of the light in the blue wavelength band among the plurality of wavelength bands.
• the G filter 64B is a green band filter that transmits the most light in the green wavelength band among the plurality of wavelength bands.
• the R filter 64C is a red band filter that transmits most of the light in the red wavelength band among the plurality of wavelength bands.
• a B filter 64A, a G filter 64B, and an R filter 64C are assigned to each pixel block 60.
• the B filter 64A, the G filter 64B, and the R filter 64C are shown arranged in a straight line along the direction perpendicular to the optical axis OA, but as an example, as shown in FIG.
• the B filter 64A, the G filter 64B, and the R filter 64C are arranged in a matrix in a predetermined pattern arrangement.
• the B filter 64A, the G filter 64B, and the R filter 64C are arranged in a matrix in a Bayer arrangement, as an example of a predetermined pattern arrangement.
• the predetermined pattern arrangement may be an RGB stripe arrangement, an R/G checkered arrangement, an X-Trans (registered trademark) arrangement, a honeycomb arrangement, or the like other than the Bayer arrangement.
• filters 64 the B filter 64A, G filter 64B, and R filter 64C
• the B filter 64A, the G filter 64B, and the R filter 64C will be referred to as "filters 64", respectively.
• a signal processing circuit 50, a control driver 16, a computer 20, a reception device 22, and a display 24 are connected to the input/output I/F 18.
• the computer 20 has a processor 70, an NVM 72, and a RAM 74.
• the processor 70 is an example of a "processor" according to the technology of the present disclosure.
• the processor 70 controls the entire imaging device 10 .
• the processor 70 is, for example, an arithmetic processing device including a CPU and a GPU, and the GPU operates under the control of the CPU and is responsible for executing processing related to images.
• an arithmetic processing unit including a CPU and a GPU is cited as an example of the processor 70, but this is just an example, and the processor 70 may be one or more CPUs with integrated GPU functions. , one or more CPUs without integrated GPU functionality.
• the processor 70, NVM 72, and RAM 74 are connected via a bus 76, and the bus 76 is connected to the input/output I/F 18.
• the NVM 72 is a non-temporary storage medium and stores various parameters and programs.
• NVM 72 is a flash memory (eg, EEPROM).
• EEPROM electrically erasable programmable read-only memory
• the RAM 74 temporarily stores various information and is used as a work memory.
• the processor 70 reads a necessary program from the NVM 72 and executes the read program in the RAM 74.
• the processor 70 controls the control driver 16 and the signal processing circuit 50 according to a program executed in the RAM 74.
• the control driver 16 controls the photoelectric conversion element 48 under the control of the processor 70 .
• the receiving device 22 includes, for example, a release button, a touch panel, hard keys (all not shown), and receives instructions from a user or the like.
• the display 24 is, for example, a liquid crystal display and displays various images.
• an image display program 80 is stored in the NVM 72.
• the image display program 80 is an example of a "program" according to the technology of the present disclosure.
• the processor 70 reads the image display program 80 from the NVM 72 and executes the read image display program 80 on the RAM 74.
• the processor 70 executes image display processing on the captured image data 120 according to an image display program 80 executed on the RAM 74 .
• the image display process is an example of "specific processing" according to the technology of the present disclosure.
• Image display processing is performed by the processor 70 in accordance with the image display program 80, including an output value acquisition section 82, an interference removal processing section 84, an alignment processing section 86, an initial image generation section 88, an initial image output section 90, an area setting determination section 92, This is realized by operating as a region setting section 94, a color setting determination section 96, a gain setting section 98, a pseudo color image generation section 100, and a pseudo color image output section 102.
• the image display process is started every time image data 120 is input from the image sensor 14 to the processor 70.
• the output value acquisition unit 82 acquires the output value Y of each physical pixel 58 based on the imaging data 120 input from the image sensor 14 to the processor 70.
• the output value Y of each physical pixel 58 corresponds to the brightness value of each pixel included in the captured image 122 indicated by the captured image data 120.
• the output value Y of each physical pixel 58 is a value that includes interference (that is, crosstalk). That is, since light in each of the transmission wavelength bands ⁇ of the first transmission wavelength band ⁇ 1 , the second transmission wavelength band ⁇ 2 , and the third transmission wavelength band ⁇ 3 is incident on each physical pixel 58, the output value Y is , a value corresponding to the light amount of the first transmission wavelength band ⁇ 1 , a value corresponding to the light amount of the second transmission wavelength band ⁇ 2 , and a value corresponding to the light amount of the third transmission wavelength band ⁇ 3 are mixed values.
• the processor 70 separates and extracts a value corresponding to each transmission wavelength band ⁇ from the output value Y for each physical pixel 58, that is, a process to remove interference. It is necessary to perform interference removal processing on the output value Y. Therefore, in this embodiment, the interference removal processing unit 84 performs interference removal processing on the output value Y of each physical pixel 58 acquired by the output value acquisition unit 82.
• the output value Y of each physical pixel 58 includes red, green, and blue brightness values as components of the output value Y.
• the output value Y of each physical pixel 58 is expressed by equation (1).
• YR is the red luminance value of the output value Y
• YG is the green luminance value of the output value Y
• YB is the blue luminance value of the output value Y. It is.
• the first spectral image 124A to the eighth spectral image 124H are images generated by performing interference removal processing on the captured image.
• the pixel value X of each image pixel included in the first spectral image 124A to the eighth spectral image 124H before pseudo-coloring, which will be described later, is the value of each light in the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8
• the luminance value of the pixel value X is included as a component of the pixel value X.
• the pixel value X of each image pixel is expressed by equation (2).
• the brightness value X ⁇ 1 is the brightness value of light in the first transmission wavelength band ⁇ 1 of the pixel value X
• the brightness value X ⁇ 2 is the brightness value of light in the second transmission wavelength band ⁇ 2 of the pixel value X
• the brightness value X ⁇ 3 is the brightness value of light in the third transmission wavelength band ⁇ 3 of the pixel value X
• the brightness value X ⁇ 4 is the brightness value of the light in the fourth transmission wavelength band ⁇ 3 of the pixel value This is the brightness value of light at ⁇ 4 .
• the brightness value X ⁇ 5 is the brightness value of light in the fifth transmission wavelength band ⁇ 5 of the pixel value X
• the brightness value X ⁇ 6 is the brightness value of light in the sixth transmission wavelength band ⁇ 6 of the pixel value X
• the brightness value X ⁇ 7 is the brightness value of light in the seventh transmission wavelength band ⁇ 7 of the pixel value X
• the brightness value X ⁇ 8 is the brightness value of light in the seventh transmission wavelength band ⁇ 7 of the pixel value This is the brightness value of light at ⁇ 8 .
• the brightness values X ⁇ 1 to X ⁇ 8 will be referred to as "brightness values X ⁇ ".
• the interference matrix A is defined based on the spectrum of the subject light, the spectral transmittance of the first lens 26, the spectral transmittance of the second lens 30, the spectral transmittance of the plurality of filters 44, and the spectral sensitivity of the image sensor 14. It's a queue.
• the interference removal matrix A + also includes the spectrum of the subject light, the spectral transmittance of the first lens 26, the spectral transmittance of the second lens 30, the spectral transmittance of the plurality of filters 44, and the image sensor 14. is a matrix defined based on the spectral sensitivity of .
• the interference cancellation matrix A + is set by a user or the like and stored in the NVM 72 in advance.
• the interference removal processing unit 84 acquires the interference cancellation matrix A + stored in the NVM 72 and the output value Y of each physical pixel 58 acquired by the output value acquisition unit 82 . Then, the interference removal processing unit 84 calculates and outputs the pixel value X of each image pixel based on the acquired interference removal matrix A + and the output value Y of each physical pixel 58 using equation (4).
• the pixel value X of each image pixel includes the luminance value of each light in the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 as a component of the pixel value X.
• the first spectral image 124A of the captured images 122 is an image corresponding to the luminance value X ⁇ 1 of light in the first transmission wavelength band ⁇ 1 (that is, an image based on the luminance value X ⁇ 1 ).
• the second spectral image 124B of the captured image 122 is an image corresponding to the luminance value X ⁇ 2 of light in the second transmission wavelength band ⁇ 2 (that is, an image based on the luminance value X ⁇ 2 ).
• the third spectral image 124C of the captured image 122 is an image corresponding to the luminance value X ⁇ 3 of light in the third transmission wavelength band ⁇ 3 (that is, an image based on the luminance value X ⁇ 3 ).
• the fourth spectral image 124D of the captured image 122 is an image corresponding to the luminance value X ⁇ 4 of light in the fourth transmission wavelength band ⁇ 4 (that is, an image based on the luminance value X ⁇ 4 ).
• the fifth spectral image 124E of the captured image 122 is an image corresponding to the luminance value X ⁇ 5 of light in the fifth transmission wavelength band ⁇ 5 (that is, an image based on the luminance value X ⁇ 5 ).
• the sixth spectral image 124F of the captured images 122 is an image corresponding to the luminance value X ⁇ 6 of light in the sixth transmission wavelength band ⁇ 6 (that is, an image based on the luminance value X ⁇ 6 ).
• the seventh spectral image 124G of the captured image 122 is an image corresponding to the brightness value X ⁇ 7 of light in the seventh transmission wavelength band ⁇ 7 (that is, an image based on the brightness value X ⁇ 7 ).
• the eighth spectral image 124H of the captured image 122 is an image corresponding to the luminance value X ⁇ 8 of light in the eighth transmission wavelength band ⁇ 8 (that is, an image based on the luminance value X ⁇ 8 ).
• the first spectral image 124A to the eighth spectral image 124H will be referred to as "spectral image 124.”
• the captured image 122 corresponds to the brightness value X ⁇ of each light in the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 . It is separated into a plurality of spectral images 124. That is, the captured image 122 is separated into spectral images 124 for each transmission wavelength band ⁇ of the plurality of filters 44.
• the plurality of spectral images 124 are images obtained by being captured by the image sensor 14 having a polarizer 46 and a filter 44, and include polarization information corresponding to the polarizer 46 and wavelength information corresponding to the filter 44. include.
• the number of spectral images 124 is greater than or equal to the number of channels (see FIGS. 6 and 10), which will be described later.
• the plurality of spectral images 124 are examples of "a plurality of different spectral images” and a “first spectral image” according to the technology of the present disclosure.
• the wavelength information is an example of "first wavelength information” according to the technology of the present disclosure.
• the alignment processing unit 86 performs alignment processing on the plurality of spectral images 124.
• the alignment process includes, for example, a process of correcting optical distortion and/or a process of geometrically correcting distortion in imaging.
• the process for correcting optical distortion include processes such as distortion correction (for example, correction of barrel aberration or pincushion aberration).
• geometrically correcting processing for correcting distortion in photographing include processing such as trapezoidal correction (that is, projective transformation, affine transformation, etc.).
• the initial image generation unit 88 performs allocation processing to allocate the R channel, G channel, and B channel to a plurality of different spectral images 124. and for each B channel, an R channel image 126A, a G channel image 126B, and a B channel image 126C are generated. The initial image generation unit 88 then generates the initial image 128 by combining the generated R channel image 126A, G channel image 126B, and B channel image 126C.
• the initial image generation unit 88 performs arithmetic processing on the plurality of spectral images 124.
• the calculation is a product-sum calculation including a plurality of spectral images 124. That is, as shown in equation (5), the image pixel component X R assigned to the R channel of each image pixel included in the plurality of spectral images 124 is the first gain G R1 set for the R channel. It is calculated by the sum of the products of the ⁇ 8th gain G R8 and the luminance values X ⁇ 1 to X ⁇ 8 of the transmission wavelength bands ⁇ 1 to ⁇ 8 .
• the first gain G R1 to the eighth gain G R8 are gains corresponding to the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 , respectively.
• the image pixel component X G assigned to the G channel among the image pixels included in the plurality of spectral images 124 is the first gain G set for the G channel. It is calculated by the sum of the products of the G1 to eighth gains GG8 and the luminance values X ⁇ 1 to X ⁇ 8 of the transmission wavelength bands ⁇ 1 to ⁇ 8 .
• the first gain G G1 to the eighth gain G G8 are gains corresponding to the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 , respectively.
• the image pixel component X B assigned to the B channel among the image pixels included in the plurality of spectral images 124 is the first gain G B1 set for the B channel. It is calculated by the sum of the products of the ⁇ 8th gain G B8 and the luminance values X ⁇ 1 to X ⁇ 8 of the transmission wavelength bands ⁇ 1 to ⁇ 8 .
• the first gain G B1 to the eighth gain G B8 are gains corresponding to the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 , respectively.
• the plurality of spectral images 124 are divided into R channel, G channel, and B channel. assigned to.
• Table 1 shows the first gain G R1 to the eighth gain G R8 set for the R channel and the first gain G G1 to the eighth gain G R8 set for the G channel in the allocation process by the initial image generation unit 88. 8 gains G G8 and an example of the first gain G B1 to the eighth gain G B8 set for the B channel are shown. The gains shown in Table 1 are for the initial image 128.
• the fifth gain G R5 corresponding to the fifth transmission wavelength band ⁇ 5, which is the red transmission wavelength band is set to “1”
• the fifth gain G R5 corresponding to the fifth transmission wavelength band ⁇ 5 which is the red transmission wavelength band
• an R channel image 126A including only the brightness value X ⁇ 5 of red light is obtained from the plurality of spectral images 124.
• the third gain G G3 corresponding to the third transmission wavelength band ⁇ 3 which is the green transmission wavelength band is set to "1”
• the third gain G G3 corresponding to the third transmission wavelength band ⁇ 3 which is the green transmission wavelength band is set to "1”
• a G channel image 126B including only the brightness value X ⁇ 3 of green light is obtained from the plurality of spectral images 124.
• the first gain G B1 corresponding to the first transmission wavelength band ⁇ 1 which is the transmission wavelength band of blue, is set to "1", and corresponds to the transmission wavelength band other than the first transmission wavelength band ⁇ 1.
• the remaining gain By setting the remaining gain to "0", a B channel image 126C including only the brightness value X ⁇ 1 of blue light is obtained from the plurality of spectral images 124.
• the initial image 128 generated by combining the R channel image 126A, G channel image 126B, and B channel image 126C obtained in this way is a normal RGB image that is not pseudo-colored.
• the initial image 128 is an example of "an image generated based on a plurality of spectral images" according to the technology of the present disclosure.
• the R channel, G channel, and B channel will be referred to as "channels".
• the R channel image 126A, G channel image 126B, and B channel image 126C are respectively referred to as "channel images 126.” to be called.
• the initial image output section 90 outputs initial image data indicating the initial image 128 generated by the initial image generation section 88 to the display 24.
• Display 24 displays initial image 128 based on the initial image data.
• an initial image 128 showing a person's left hand 140, a first object 144, a second object 146, and a background 148 is displayed on the display 24.
• Hand 140 includes blood vessels 142 .
• the processor 70 changes the operation mode of the imaging device 10 so that the user or the like can divide the initial image 128 into a plurality of regions 132 through the reception device 22.
• the user or the like gives an area division instruction to the reception device 22 to divide the initial image 128 into a plurality of areas 132.
• the area division instruction is given to, for example, a touch panel included in the reception device 22.
• the receiving device 22 When receiving the area partitioning instruction, the receiving device 22 outputs area partitioning instruction data indicating the area partitioning instruction to the processor 70.
• the area setting determination unit 92 determines whether or not area division instruction data has been input to the processor 70.
• the area setting unit 94 divides the initial image 128 into a plurality of areas 132 according to the area division instruction data.
• the process of dividing the initial image 128 into a plurality of regions 132 includes, for example, a process of detecting the boundaries of images included in the plurality of spectral images 124, and/or a process of dividing the plurality of regions 132 based on pre-stored spectral characteristics.
• Various types of processing such as segmentation processing are used.
• the area division instructions include a first area instruction, a second area instruction, a third area instruction, a fourth area instruction, and a fifth area instruction.
• the first region instruction is an instruction to specify the region 132 (hereinafter also referred to as "first region 132A”) corresponding to a portion of the hand 140 other than the blood vessel 142.
• the second region instruction is an instruction to specify the region 132 (hereinafter also referred to as "second region 132B”) corresponding to the blood vessel 142 in the hand 140.
• the third area instruction is an instruction to specify the area 132 (hereinafter also referred to as "third area 132C”) corresponding to the first object 144.
• the fourth area instruction is an instruction to specify the area 132 (hereinafter also referred to as "fourth area 132D”) corresponding to the second object 146.
• the fifth area instruction is an instruction to specify the area 132 (hereinafter also referred to as "fifth area 132E”) corresponding to the background 148.
• the initial image 128 is divided into a plurality of regions 132 in response to the first region designation, the second region designation, the third region designation, the fourth region designation, and the fifth region designation. It is separated. Specifically, the initial image 128 is divided into a first area 132A, a second area 132B, a third area 132C, a fourth area 132D, and a fifth area 132E. The first region 132A, the second region 132B, the third region 132C, the fourth region 132D, and the fifth region 132E can be classified because they reflect light in different wavelength bands.
• the multiple areas 132 are an example of "multiple areas" according to the technology of the present disclosure.
• the processor 70 controls the operation mode of the imaging device 10 so that the user or the like can select a color for each region 132 through the reception device 22.
• a color palette 134 from which a plurality of colors can be selected is displayed on the display 24.
• the color setting instruction includes an area specifying instruction that specifies the area 132 to set a color among the plurality of areas 132, and a color specifying instruction that specifies the color to be set for the area 132 corresponding to the area specifying instruction.
• the color setting instruction is given to, for example, a touch panel (not shown) included in the reception device 22.
• the receiving device 22 When receiving a color setting instruction, the receiving device 22 outputs color setting instruction data indicating the color setting instruction to the processor 70.
• the color setting instruction data includes area information corresponding to the area specifying instruction and color information corresponding to the color specifying instruction. Color information is an example of "color information" according to the technology of the present disclosure.
• the color setting determination unit 96 determines whether color setting instruction data has been input to the processor 70.
• the gain setting unit 98 determines whether the area 132 specified by the area specifying instruction is specified by the color specifying instruction according to the color setting instruction data.
• the gain G is set for the plurality of spectral images 124 so that the colors are set to the same color.
• y R1 is the red luminance value of the first region 132A
• y R2 is the red luminance value of the second region 132B
• y R3 is the red luminance value of the third region 132C
• yR4 is the red luminance value of the fourth region 132D
• yR5 is the red luminance value of the fifth region 132E.
• y G1 is the green luminance value of the first region 132A
• y G2 is the green luminance value of the second region 132B
• y G3 is the green luminance value of the third region 132C
• y G4 is the green luminance value of the fourth region 132D
• yG5 is the green luminance value of the fifth region 132E.
• y B1 is the blue luminance value of the first region 132A
• y B2 is the blue luminance value of the second region 132B
• y B3 is the blue luminance value of the third region 132C
• y B4 is the blue luminance value of the fourth region 132D
• yB5 is the blue luminance value of the fifth region 132E.
• the average brightness value x of each region 132 is expressed by equation (9).
• x ⁇ 1-1 is the average brightness value of light in the first transmission wavelength band ⁇ 1 in the image pixels included in the first region 132A.
• x ⁇ 1-2 is the average brightness value of light in the first transmission wavelength band ⁇ 1 in the image pixels included in the second region 132B.
• x ⁇ 1-3 is the average brightness value of light in the first transmission wavelength band ⁇ 1 in the image pixels included in the third region 132C.
• x ⁇ 1-4 is the average brightness value of light in the first transmission wavelength band ⁇ 1 in the image pixels included in the fourth region 132D.
• x ⁇ 1-5 is the average brightness value of light in the first transmission wavelength band ⁇ 1 in the image pixel included in the fifth region 132E.
• x ⁇ 2-1 is the average brightness value of light in the second transmission wavelength band ⁇ 2 in the image pixels included in the first region 132A.
• x ⁇ 2-2 is the average brightness value of light in the second transmission wavelength band ⁇ 2 in the image pixels included in the second region 132B.
• x ⁇ 2-3 is the average brightness value of light in the second transmission wavelength band ⁇ 2 in the image pixel included in the third region 132C.
• x ⁇ 2-4 is the average brightness value of light in the second transmission wavelength band ⁇ 2 in the image pixels included in the fourth region 132D.
• x ⁇ 2-5 is the average brightness value of light in the second transmission wavelength band ⁇ 2 in the image pixels included in the fifth region 132E.
• x ⁇ 3-1 is the average brightness value of light in the third transmission wavelength band ⁇ 3 in the image pixels included in the first region 132A.
• x ⁇ 3-2 is the average brightness value of light in the third transmission wavelength band ⁇ 3 in the image pixel included in the second region 132B.
• x ⁇ 3-3 is the average brightness value of light in the third transmission wavelength band ⁇ 3 in the image pixels included in the third region 132C.
• x ⁇ 3-4 is the average brightness value of light in the third transmission wavelength band ⁇ 3 in the image pixel included in the fourth region 132D.
• x ⁇ 3-5 is the average brightness value of light in the third transmission wavelength band ⁇ 3 in the image pixel included in the fifth region 132E.
• x ⁇ 4-1 is the average brightness value of light in the fourth transmission wavelength band ⁇ 4 in the image pixels included in the first region 132A.
• x ⁇ 4-2 is the average brightness value of light in the fourth transmission wavelength band ⁇ 4 in the image pixel included in the second region 132B.
• x ⁇ 4-3 is the average brightness value of light in the fourth transmission wavelength band ⁇ 4 in the image pixel included in the third region 132C.
• x ⁇ 4-4 is the average brightness value of light in the fourth transmission wavelength band ⁇ 4 in the image pixels included in the fourth region 132D.
• x ⁇ 4-5 is the average brightness value of light in the fourth transmission wavelength band ⁇ 4 in the image pixel included in the fifth region 132E.
• x ⁇ 5-1 is the average brightness value of light in the fifth transmission wavelength band ⁇ 5 in the image pixel included in the first region 132A.
• x ⁇ 5-2 is the average brightness value of light in the fifth transmission wavelength band ⁇ 5 in the image pixel included in the second region 132B.
• x ⁇ 5-3 is the average brightness value of light in the fifth transmission wavelength band ⁇ 5 in the image pixel included in the third region 132C.
• x ⁇ 5-4 is the average brightness value of light in the fifth transmission wavelength band ⁇ 5 in the image pixel included in the fourth region 132D.
• x ⁇ 5-5 is the average brightness value of light in the fifth transmission wavelength band ⁇ 5 in the image pixel included in the fifth region 132E.
• x ⁇ 6-1 is the average brightness value of light in the sixth transmission wavelength band ⁇ 6 in the image pixels included in the first region 132A.
• x ⁇ 6-2 is the average brightness value of light in the sixth transmission wavelength band ⁇ 6 in the image pixel included in the second region 132B.
• x ⁇ 6-3 is the average brightness value of light in the sixth transmission wavelength band ⁇ 6 in the image pixel included in the third region 132C.
• x ⁇ 6-4 is the average brightness value of light in the sixth transmission wavelength band ⁇ 6 in the image pixel included in the fourth region 132D.
• x ⁇ 6-5 is the average brightness value of light in the sixth transmission wavelength band ⁇ 6 in the image pixel included in the fifth region 132E.
• x ⁇ 7-1 is the average brightness value of light in the seventh transmission wavelength band ⁇ 7 in the image pixel included in the first region 132A.
• x ⁇ 7-2 is the average brightness value of light in the seventh transmission wavelength band ⁇ 7 in the image pixel included in the second region 132B.
• x ⁇ 7-3 is the average brightness value of light in the seventh transmission wavelength band ⁇ 7 in the image pixel included in the third region 132C.
• x ⁇ 7-4 is the average brightness value of light in the seventh transmission wavelength band ⁇ 7 in the image pixel included in the fourth region 132D.
• x ⁇ 7-5 is the average brightness value of light in the seventh transmission wavelength band ⁇ 7 in the image pixel included in the fifth region 132E.
• x ⁇ 8-1 is the average brightness value of light in the eighth transmission wavelength band ⁇ 8 in the image pixels included in the first region 132A.
• x ⁇ 8-2 is the average brightness value of light in the eighth transmission wavelength band ⁇ 8 in the image pixels included in the second region 132B.
• x ⁇ 8-3 is the average brightness value of light in the eighth transmission wavelength band ⁇ 8 in the image pixel included in the third region 132C.
• x ⁇ 8-4 is the average brightness value of light in the eighth transmission wavelength band ⁇ 8 in the image pixels included in the fourth region 132D.
• x ⁇ 8-5 is the average brightness value of light in the eighth transmission wavelength band ⁇ 8 in the image pixel included in the fifth region 132E.
• the pixel value y of each image pixel included in the pseudo-color image 130 is expressed by equation (10) using the gain G.
• the gain G is the first gain G R1 to the eighth gain G R8 set for the R channel, the first gain G G1 to the eighth gain G G8 set for the G channel, and the first gain G R1 to the eighth gain G G8 set for the B channel.
• the first gain G B1 to the eighth gain G B8 are set as follows.
• the first gain G R1 to the eighth gain G R8 are gains corresponding to the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 , respectively.
• the first gain G G1 to the eighth gain G G8 are gains corresponding to the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 , respectively.
• the first gain G B1 to the eighth gain G B8 are gains corresponding to the first transmission wavelength band ⁇ 1 to the eighth transmission wavelength band ⁇ 8 , respectively.
• Gain G is expressed by equation (11).
• the pixel value y of each image pixel included in the pseudo-color image 130 is set based on the area 132 and color specified by the area specification instruction and color specification instruction.
• the color of the area 132 that is not specified by the area specification instruction and the color specification instruction is set based on the initial image 128.
• the first gain G R1 to the eighth gain G R8 , the first gain G G1 to the eighth gain G G8 , and the first gain G B1 to the eighth gain G B8 is derived.
• the gain G is derived by the above calculation method when the number of regions 132 divided by the region division instruction is equal to or less than the number of transmission wavelength bands ⁇ .
• the gain G may be derived as an approximate solution by performing interpolation processing such as least square sum, for example. Further, the gain G is uniformly set for a plurality of image pixels included in each spectral image 124, but may be set individually for each image pixel included in each spectral image 124.
• Table 2 shows an example of the gain G derived in accordance with a color setting instruction (hereinafter referred to as "first color setting instruction") for setting the second region 132B to blue.
• first color setting instruction a color setting instruction for setting the second region 132B to blue.
• the gain G derived according to the first color setting instruction includes a negative gain in addition to a positive gain.
• FIG. 10 shows an example in which a pseudo color image 130 is generated based on the gain G set by the gain setting section 98.
• the pseudo-color image generation unit 100 performs an allocation process for assigning the R channel, G channel, and B channel to the plurality of different spectral images 124, thereby assigning the R channel to each of the R channel, G channel, and B channel.
• An image 126A, a G channel image 126B, and a B channel image 126C are generated.
• the R channel, G channel, and B channel are examples of "different channels” according to the technology of the present disclosure.
• the allocation process is an example of "allocation process” according to the technology of the present disclosure.
• the R channel image 126A, the G channel image 126B, and the B channel image 126C are examples of "channel images” according to the technology of the present disclosure.
• the pseudocolor image 130 is an example of a "first pseudocolor image” according to the technology of the present disclosure.
• the pseudo color image generation unit 100 performs arithmetic processing on the plurality of spectral images 124.
• calculation is performed using the gain G set by the gain setting section 98 described above.
• the calculation is a product-sum calculation including a plurality of spectral images 124.
• the image pixel component X R assigned to the R channel of each image pixel included in the plurality of spectral images 124 is the first gain G R1 set for the R channel. It is calculated by the sum of the products of the ⁇ 8th gain G R8 and the luminance values X ⁇ 1 to X ⁇ 8 of the transmission wavelength bands ⁇ 1 to ⁇ 8 .
• the image pixel component X G assigned to the G channel among the image pixels included in the plurality of spectral images 124 is the first gain G set for the G channel. It is calculated by the sum of the products of the G1 to eighth gains GG8 and the luminance values X ⁇ 1 to X ⁇ 8 of the transmission wavelength bands ⁇ 1 to ⁇ 8 .
• the image pixel component X B assigned to the B channel among the image pixels included in the plurality of spectral images 124 is the first gain G B1 set for the B channel. It is calculated by the sum of the products of the ⁇ 8th gain G B8 and the luminance values X ⁇ 1 to X ⁇ 8 of the transmission wavelength bands ⁇ 1 to ⁇ 8 .
• the gain G includes a negative gain (that is, a negative value) in addition to a positive gain, calculations including subtraction are performed in the calculation process.
• the image pixel components X R , X G , and X B of each image pixel included in the plurality of spectral images 124 are calculated, so that the plurality of spectral images 124 are divided into R channel, G channel, and B channel.
• the pseudo color image generation unit 100 generates a pseudo color image 130 by combining the generated R channel image 126A, G channel image 126B, and B channel image 126C.
• the pseudo color image output section 102 outputs pseudo color image data representing the pseudo color image 130 generated by the pseudo color image generation section 100 to the display 24.
• the pseudo-color image data is an example of "data for displaying a first pseudo-color image" according to the technology of the present disclosure.
• Display 24 displays a pseudocolor image 130 represented by pseudocolor image data.
• a pseudo-color image 130 in which the second area 132B is set to blue is displayed on the display 24.
• the color of the second region 132B is set to blue, so that the color of the second region 132B is saturated at the fingertip of the hand 140.
• the upper limit of the range of the pseudo-color image 130 exceeds the expression range of the display 24.
• the expression range of the display 24 refers to the range of brightness that can be displayed on the display 24.
• the range of the pseudo-color image 130 refers to the range of brightness of image pixels included in the pseudo-color image 130.
• the display 24 is an example of a "display medium" and a "display device" according to the technology of the present disclosure.
• FIG. 12 shows an example in which a pseudo-color image 130 is generated according to a color setting instruction (hereinafter referred to as "second color setting instruction") for setting the second area 132B to light blue instead of blue.
• second color setting instruction a color setting instruction for setting the second area 132B to light blue instead of blue.
• the color of the second area 132B may be changed from blue to light blue. good.
• the color of the second region 132B is changed from blue to light blue, thereby preventing the color of the second region 132B from becoming saturated.
• Table 3 shows an example of the gain G derived according to the second color setting instruction.
• the gain G derived according to the second color setting instruction includes a negative gain in addition to a positive gain.
• the calculation process includes subtraction.
• a pseudo-color image 130 is generated in which the second area 132B is set to light blue.
• the gain G is set to a value that allows the range of the pseudo-color image 130 to fit within the expression range of the display 24 that displays the pseudo-color image 130. This prevents the color of the second region 132B from becoming saturated.
• the gain setting unit 98 sets the pseudo-color image 130 in the expression range of the display 24.
• the gain G may be adjusted so that the range of .
• FIG. 13 shows a color setting instruction (hereinafter referred to as "color setting instruction”) to set the first region 132A to yellow, which is easily visible to light blue, or red, which is a complementary color to light blue, while leaving the second region 132B set to light blue.
• color setting instruction a color setting instruction to set the first region 132A to yellow, which is easily visible to light blue, or red, which is a complementary color to light blue, while leaving the second region 132B set to light blue.
• Table 4 shows an example of the gain G derived according to the third color setting instruction.
• the gain G derived according to the third color setting instruction includes a negative gain in addition to a positive gain.
• the gain G since the gain G includes a negative gain (that is, a negative value) in addition to a positive gain, in the calculation process (see FIG. 10), calculations including subtraction are performed.
• a pseudo-color image 130 is generated in which the second region 132B is set to light blue and the first region 132A is set to yellow or red. Note that if the same color is set in different regions 132, the colors in the different regions 132 may become saturated, so it is preferable that different colors are set in different regions 132.
• the second area 132B corresponding to the blood vessel 142 is set to light blue
• the first area 132A corresponding to the part of the hand 140 other than the blood vessel 142 is set to yellow or red.
• the blood vessel 142 is emphasized with respect to the portion of the hand 140 other than the blood vessel 142.
• FIG. 14 shows a color setting instruction (hereinafter referred to as "fourth color An example is shown in which a pseudo-color image 130 is generated according to the "Setting Instructions"). Note that the colors of the fourth area 132D and the fifth area 132E remain the same as the colors of the fourth area 132D and the fifth area 132E in the initial image 128.
• Table 5 shows an example of the gain G derived according to the fourth color setting instruction.
• the gain G derived according to the fourth color setting instruction includes a negative gain in addition to a positive gain.
• the gain G includes a negative gain (that is, a negative value) in addition to a positive gain, in the calculation process (see FIG. 10), calculations including subtraction are performed.
• a pseudo-color image 130 is generated in which the second region 132B is set to light blue, the first region 132A is set to yellow or red, and the third region 132C is set to white.
• the third region 132C corresponding to the background 148 is set to white, so that the hand 140 and the blood vessel 142 are emphasized against the background 148.
• the pseudo-color image 130 is an image in which the discrimination of wavelength information is enhanced, for example, with respect to a pseudo-color image in which only addition is included in calculations (hereinafter referred to as a "comparison target pseudo-color image").
• An example of an image in which the discriminability of wavelength information is enhanced is an image that can be distinguished from a comparison target pseudocolor image by color difference and/or brightness difference.
• the color difference refers to, for example, complementary colors on the hue wheel.
• the other color out of the two colors in the relationship in the hue wheel is used in the pseudo-color image 130. It will be done.
• the color used for the pseudo color may be set for the comparison target pseudo color.
• the brightness difference for example, if the pseudo color image to be compared is expressed within the brightness range of "150 to 255", the pseudo color image 130 may be expressed within the brightness range of "0 to 50". .
• the comparison target pseudo-color image is an example of a "second pseudo-color image" according to the technology of the present disclosure.
• the gain is an example of a "coefficient of the first spectral image” according to the technology of the present disclosure.
• the wavelength information is an example of "second wavelength information” according to the technology of the present disclosure.
• FIG. 15 shows an example of the flow of image display processing according to this embodiment.
• step ST10 the output value acquisition unit 82 acquires the output value Y of each physical pixel 58 based on the imaging data 120 input from the image sensor 14 to the processor 70. (See Figure 4). After the process of step ST10 is executed, the image display process moves to step ST12.
• step ST12 the interference removal processing unit 84 performs interference removal processing on the output value Y of each physical pixel 58 obtained in step ST10 (see FIG. 4). Thereby, the captured image 122 represented by the captured image data 120 is separated into spectral images 124 for each transmission wavelength band ⁇ of the plurality of filters 44 . After the process of step ST12 is executed, the image display process moves to step ST14.
• step ST14 the alignment processing unit 86 performs alignment processing on the plurality of spectral images 124 generated in step ST12 (see FIG. 5). After the process of step ST14 is executed, the image display process moves to step ST16.
• step ST16 the initial image generation unit 88 performs an allocation process to allocate the R channel, G channel, and B channel to the plurality of spectral images 124 subjected to the alignment process in step ST14. , G channel, and B channel, an R channel image 126A, a G channel image 126B, and a B channel image 126C are generated (see FIG. 6). After the process of step ST16 is executed, the image display process moves to step ST18.
• step ST18 the initial image generation unit 88 generates the initial image 128 by combining the R channel image 126A, the G channel image 126B, and the B channel image 126C generated in step ST16 (see FIG. 6). After the process of step ST18 is executed, the image display process moves to step ST20.
• step ST20 the initial image output unit 90 outputs initial image data indicating the initial image 128 generated in step ST18 to the display 24 (see FIG. 7). As a result, the initial image 128 is displayed on the display 24. After the process of step ST20 is executed, the image display process moves to step ST22.
• step ST22 the area setting determination unit 92 determines whether or not area division instruction data has been input to the processor 70 (see FIG. 8). If the area segmentation instruction data is not input to the processor 70, the determination is negative and the image display process moves to step ST36. If the area segmentation instruction data is input to the processor 70, the determination is affirmative and the image display process moves to step ST24.
• step ST24 the region setting unit 94 divides the initial image 128 into a plurality of regions 132 according to the region division instruction data (see FIG. 8). After the process of step ST24 is executed, the image display process moves to step ST26.
• step ST26 the color setting determination unit 96 determines whether color setting instruction data has been input to the processor 70 (see FIG. 9). If the color setting instruction data is not input to the processor 70, the determination is negative and the image display process moves to step ST36. If the color setting instruction data is input to the processor 70, the determination is affirmative and the image display process moves to step ST28.
• step ST28 the gain setting unit 98 sets a gain to the plurality of spectral images 124 in accordance with the color setting instruction data so that the region 132 specified by the area specification instruction is set to the color specified by the color specification instruction.
• Set G (see Figure 9).
• step ST30 the pseudo-color image generation unit 100 performs arithmetic processing on the plurality of spectral images 124 based on the gain G set in step ST28, so that each of the R channel, G channel, and B channel is , an R channel image 126A, a G channel image 126B, and a B channel image 126C (see FIG. 10). Thereby, multiple spectral images 124 are assigned to the R channel, G channel, and B channel.
• the image display process moves to step ST32.
• step ST32 the pseudo color image generation unit 100 generates a pseudo color image 130 by combining the R channel image 126A, the G channel image 126B, and the B channel image 126C generated in step ST30 (see FIG. 10). .
• the image display process moves to step ST34.
• step ST34 the pseudo color image output unit 102 outputs pseudo color image data representing the pseudo color image 130 generated in step ST32 to the display 24 (see FIG. 11). As a result, a pseudo-color image 130 is displayed on the display 24.
• step ST36 the image display process moves to step ST36.
• step ST36 the processor 70 determines whether a condition for terminating the image display process (ie, an terminating condition) is satisfied.
• a condition for terminating the image display process ie, an terminating condition
• An example of the termination condition is a condition that a user or the like has given an instruction to the imaging device 10 to terminate the image display process.
• the determination is negative and the image display process moves to step ST26.
• the termination condition is satisfied, the determination is affirmative and the image display process is terminated.
• the image display processing method described as the function of the imaging device 10 described above is an example of the "information processing method" according to the technology of the present disclosure.
• the processor 70 generates a channel image 126 for each channel by performing an assignment process for assigning a plurality of different spectral images 124 to different channels. , generates a pseudocolor image 130 based on the plurality of channel images 126 (see FIG. 10).
• the assignment process includes generating channel images 126 based on operations including subtraction on multiple spectral images 124. Therefore, for example, compared to the case where the channel image 126 is generated based on an operation including only addition for a plurality of different spectral images 124, color adjustment with a higher degree of freedom is possible when pseudo-coloring the plurality of spectral images 124. can be realized. As a result, it is possible to generate a pseudo-color image 130 that is visually distinguishable from the comparison target pseudo-color image generated by combining the channel images 126 generated based on calculations involving only addition.
• the calculation is a product-sum calculation including a plurality of spectral images 124, and the subtraction is realized by including a negative value in the gain of the plurality of spectral images 124 in the product-sum calculation. Therefore, by adjusting the value of the gain containing a negative value and/or the number of gains containing a negative value, the color of the pseudocolor image 130 can be adjusted.
• the gain G is set to a value that allows the range of the pseudo-color image 130 to fit within the expression range of the display 24 that displays the pseudo-color image 130. Therefore, it is possible to avoid saturation of the color set for the pseudo-color image 130.
• the multiple spectral images 124 include polarization information and wavelength information, and the number of multiple spectral images 124 is greater than or equal to the number of channels. Therefore, for example, the degree of freedom in adjusting the color of the pseudo-color image 130 can be increased compared to when the number of multiple spectral images 124 is smaller than the number of channels.
• the processor 70 performs alignment processing on the plurality of spectral images 124 obtained by imaging by the image sensor 14 having the plurality of polarizers 46, and subtraction is performed on the plurality of spectra on which the alignment processing has been performed. This is done for image 124.
• the plurality of spectral images 124 are images obtained by being imaged by the image sensor 14 having a plurality of polarizers 46. Therefore, for example, misregistration occurs in the plurality of spectral images 124. Therefore, by performing the alignment process on the plurality of spectral images 124, the initial image 128 (see FIG. 6) and the pseudocolor image 130 (see FIG. ) image quality can be ensured.
• the pseudo-color image 130 is an image in which the discrimination of wavelength information is enhanced compared to the comparison pseudo-color image in which only addition is included in the calculation. Therefore, a user or the like can visually distinguish the pseudo-color image 130 from the comparison target pseudo-color image.
• the processor 70 divides the image generated based on the plurality of spectral images 124 into a plurality of regions 132, and calculations are performed based on color information set for the plurality of regions 132. Therefore, the user or the like can set a color for each designated area 132 among the plurality of areas 132.
• the different channels are the three primary color channels. Therefore, a pseudocolor image 130 can be generated based on the three primary color channels.
• the processor 70 outputs pseudocolor image data for displaying the pseudocolor image 130 on the display 24. Therefore, the user or the like can confirm the subject image included in the pseudo-color image 130 displayed on the display 24 in a color different from the original color of the subject image.
• the plurality of spectral images 124 are assigned to the R channel, G channel, and B channel corresponding to the three primary colors of color, but the plurality of spectral images 124 are assigned to the channels corresponding to the three primary colors of light. Good too.
• the calculation including subtraction is performed on all the spectral images 124, but even if the calculation including subtraction is performed on some of the spectral images 124 among the plurality of spectral images 124, good.
• a plurality of spectral images 124 including polarization information and wavelength information are generated, but a plurality of spectral images 124 including only one of polarization information and wavelength information may be generated. .
• the initial image 128 and the pseudo color image 130 are displayed on the display 24 provided in the imaging device 10, but the initial image 128 and the pseudo color image 130 are displayed on a display medium or a display device provided in an external device other than the imaging device 10.
• An initial image 128 and a pseudocolor image 130 may be displayed.
• the technology according to the embodiment described above may be applied to types of devices other than the imaging device 10 (hereinafter referred to as "other devices").
• processor 70 is illustrated, but instead of the processor 70 or together with the processor 70, at least one other CPU, at least one GPU, and/or at least one TPU may be used. It's okay.
• the image display program 80 may be stored in a portable non-transitory computer-readable storage medium (hereinafter simply referred to as a "non-transitory storage medium") such as an SSD or a USB memory.
• a non-transitory storage medium such as an SSD or a USB memory.
• the image display program 80 stored in a non-temporary storage medium may be installed on the computer 20 of the imaging device 10.
• the image display program 80 is stored in a storage device such as another computer or a server device connected to the imaging device 10 via a network, and the image display program 80 is downloaded in response to a request from the imaging device 10. It may be installed on the computer 20.
• image display program 80 it is not necessary to store the entire image display program 80 in another computer or storage device such as a server device connected to the imaging device 10, or in the NVM 72, but only a part of the image display program 80 can be stored. You can stay there.
• the computer 20 is built into the imaging device 10, the technology of the present disclosure is not limited to this, and for example, the computer 20 may be provided outside the imaging device 10.
• the computer 20 including the processor 70, NVM 72, and RAM 74 is illustrated, but the technology of the present disclosure is not limited to this, and instead of the computer 20, ASIC, FPGA, and/or PLD You may also apply a device that includes. Further, instead of the computer 20, a combination of hardware configuration and software configuration may be used.
• processors can be used as hardware resources for executing the various processes described in the above embodiments.
• the processor include a CPU, which is a general-purpose processor that functions as a hardware resource that executes various processes by executing software, that is, a program.
• the processor include a dedicated electronic circuit such as an FPGA, a PLD, or an ASIC, which is a processor having a circuit configuration specifically designed to execute a specific process.
• Each processor has a built-in memory or is connected to it, and each processor uses the memory to perform various processes.
• Hardware resources that execute various processes may be configured with one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a CPU and FPGA). Furthermore, the hardware resource that executes various processes may be one processor.
• one processor is configured by a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes various processes.
• a and/or B has the same meaning as “at least one of A and B.” That is, “A and/or B” means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, even when three or more items are expressed by connecting them with “and/or”, the same concept as “A and/or B" is applied.

Landscapes

• Engineering & Computer Science (AREA)
• Theoretical Computer Science (AREA)
• Physics & Mathematics (AREA)
• General Physics & Mathematics (AREA)
• Multimedia (AREA)
• General Health & Medical Sciences (AREA)
• Health & Medical Sciences (AREA)
• Signal Processing (AREA)
• Medical Informatics (AREA)
• Databases & Information Systems (AREA)
• Evolutionary Computation (AREA)
• Computing Systems (AREA)
• Computer Vision & Pattern Recognition (AREA)
• Software Systems (AREA)
• Artificial Intelligence (AREA)
• Mathematical Physics (AREA)
• Spectroscopy & Molecular Physics (AREA)
• Life Sciences & Earth Sciences (AREA)
• Chemical & Material Sciences (AREA)
• Analytical Chemistry (AREA)
• Biochemistry (AREA)
• Immunology (AREA)
• Pathology (AREA)
• Color Television Image Signal Generators (AREA)

Priority Applications (2)

Applications Claiming Priority (2)

Related Child Applications (1)

Publications (1)

Family

ID=88199948

Family Applications (1)

Country Status (3)

Cited By (1)

Families Citing this family (1)

Citations (5)

• 2022
  • 2022-11-09 JP JP2024511193A patent/JPWO2023188513A1/ja active Pending
  • 2022-11-09 WO PCT/JP2022/041773 patent/WO2023188513A1/ja not_active Ceased
• 2024
  • 2024-07-22 US US18/779,091 patent/US20240371052A1/en active Pending

Patent Citations (5)

Also Published As

Similar Documents

Legal Events