WO2024042783A1

WO2024042783A1 - Image processing device, image processing method, and program

Info

Publication number: WO2024042783A1
Application number: PCT/JP2023/017159
Authority: WO
Inventors: 高志椚瀬; 和佳岡田; 慶延岸根; 睦川中子; 友也平川
Original assignee: 富士フイルム株式会社
Priority date: 2022-08-22
Filing date: 2023-05-02
Publication date: 2024-02-29

Abstract

This image processing device is to be used for an image outputted from an imaging device comprising an optical system. The optical system has a plurality of filters provided around an optical axis. This image processing device comprises a processor. The processor performs, on the image, a process for correcting image deviation due to positional deviation of an optical image caused by light diffraction by means of the plurality of filters.

Description

Image processing device, image processing method, and program

The technology of the present disclosure relates to an image processing device, an image processing method, and a program.

International Publication No. 2021/085368 pamphlet discloses an imaging device including an imaging optical system, an imaging element, and a signal processing section. The imaging optical system has a first pupil region that passes light in a first wavelength band and a second pupil region that passes light in a second wavelength band different from the first wavelength band. be. The imaging optical system combines the axial chromatic aberration of the imaging optical system due to the difference between the first wavelength band and the second wavelength band with the aberration other than the axial chromatic aberration of the imaging optical system and the first pupil region of the imaging optical system and It is reduced based on the relationship with the position of the second pupil area. The image sensor includes a first pixel that receives light passing through a first pupil region of an imaging optical system, and a second pixel that receives light that passes through a second pupil region. The signal processing unit processes the signal output from the image sensor, and generates a first image in a first wavelength band and a second wavelength band based on the output signal of the first pixel and the output signal of the second pixel. , respectively.

JP 2022-063720A discloses an image correction device including a band image acquisition means, a high-resolution image acquisition means, a position difference acquisition means, a correction band image creation means, and a correction band image output means. ing. The band image acquisition means acquires a plurality of band images obtained by imaging the subject. The high-resolution image acquisition means acquires a high-resolution image, which is obtained by imaging the subject and has a higher resolution than the band image. The positional difference acquisition means sets one of the plurality of band images as a reference band image, sets at least one of the remaining band images as a target band image, and acquires a positional difference between the target band image and the reference band image. The correction band image creation means takes a pixel of the target band image as a target pixel, and for each target pixel, divides the imaging region of the target pixel into a plurality of regions, and calculates the pixel value of each partial region created by dividing the image capturing region of the target pixel into a plurality of regions. The target pixel at the pixel position of the reference band image is determined based on the value and the pixel value relationship between multiple pixels of the high-resolution image corresponding to the target pixel, and from the determined pixel value and position difference for each partial region. A corrected band image is created that retains the pixel values of light related to the band image. The correction band image output means outputs a correction band image.

International Publication No. 2021/153473 pamphlet discloses a lens device including an imaging optical system and an optical member. The imaging optical system includes a lens that forms an optical image of a subject. The optical member includes a frame body having a plurality of aperture regions, and a plurality of optical filters disposed in at least one of the plurality of aperture regions, the two or more optical filters transmitting light having at least some different wavelength bands. It has a plurality of optical filters including an optical filter, and a plurality of polarization filters arranged in at least one of the plurality of aperture regions, and the plurality of polarization filters having different polarization directions. The amount of light emitted from the imaging optical system can be changed for a plurality of aperture regions.

JP 2014-045275A discloses an image processing device including an image acquisition section, a phase difference image generation section, and a high resolution processing section. The image acquisition unit acquires a captured image obtained by capturing a first subject image and a second subject image having a parallax with respect to the first subject image. The phase difference image generation unit generates a first image corresponding to the first subject image and a second image corresponding to the second subject image based on the captured image. The high-resolution processing section performs high-resolution processing on the captured image based on the first image and the second image.

As an example, one embodiment of the technology of the present disclosure can obtain a multispectral image with higher image quality than when a multispectral image is generated based on a spectral image in which image shift has occurred. Provides an image processing device, an image processing method, and a program.

A first aspect of the technology of the present disclosure is an image processing device applied to an image output from an imaging device including an optical system, wherein the optical system includes a plurality of The image processing device has a filter, and the image processing device includes a processor, and the processor is an image processing device that performs processing on the image to correct an image shift due to a position shift of an optical image caused by being separated by a plurality of filters. be.

A second aspect of the technology of the present disclosure is the image processing apparatus according to the first aspect, in which the optical system has a plurality of apertures, each aperture is provided with a filter, and the image shift is at least The image processing apparatus includes image shift based on the characteristics of each aperture, and processing is performed based on the characteristics.

A third aspect according to the technology of the present disclosure is an image processing apparatus according to the second aspect, which is characterized by including a position of the center of gravity of the opening.

A fourth aspect according to the technology of the present disclosure is an image processing apparatus according to the third aspect, in which the center of gravity position is determined based on the position and/or shape of the opening.

A fifth aspect of the technology of the present disclosure is that in the image processing apparatus according to any one of the first to fourth aspects, positional deviation of the optical image is caused by at least a characteristic of the optical system. This is an image processing device that includes positional deviation.

A sixth aspect of the technology of the present disclosure is that in the image processing apparatus according to any one of the first to fifth aspects, processing is performed on a partial area of the image. It is an image processing device.

A seventh aspect of the technology of the present disclosure is an image processing apparatus according to any one of the first to sixth aspects, wherein the image is a spectral image for generating a multispectral image. It is a processing device.

An eighth aspect according to the technology of the present disclosure is that in the image processing device according to the seventh aspect, the image is obtained by performing interference removal processing on imaged data obtained by imaging with the imaging device. This is an image processing device that is a generated image.

A ninth aspect of the technology of the present disclosure is the image processing device according to any one of the first to eighth aspects, wherein the plurality of filters have different wavelength bands. .

A tenth aspect of the technology of the present disclosure is the image processing device according to any one of the first to ninth aspects, wherein the plurality of filters are arranged in line around an optical axis. It is an image processing device.

An eleventh aspect according to the technology of the present disclosure is an image processing apparatus according to the ninth aspect, in which processing is performed based on a combination of wavelength bands.

A twelfth aspect of the technology of the present disclosure is an image processing apparatus according to any one of the first to eleventh aspects, in which processing is performed based on design values regarding the optical system. It is.

A thirteenth aspect according to the technology of the present disclosure is an image processing apparatus according to the ninth aspect, in which processing is performed based on a correction value for each wavelength band.

A fourteenth aspect according to the technology of the present disclosure is an image processing apparatus according to any one of the ninth aspect, the eleventh aspect, and the thirteenth aspect, in which the image is processed using wavelengths corresponding to each wavelength band. This is an image processing device that includes band images.

A fifteenth aspect according to the technology of the present disclosure is the image processing apparatus according to the thirteenth aspect, wherein the image includes a wavelength band image corresponding to each wavelength band, and the correction value is determined according to the position of the wavelength band image. They are different image processing devices.

A 16th aspect according to the technology of the present disclosure is the image processing apparatus according to the 13th aspect, wherein the image includes a wavelength band image corresponding to each wavelength band, and the correction value is a wavelength band image with respect to a reference position in the image. This is an image processing device that is determined based on the direction and/or amount of positional shift of an image.

A seventeenth aspect according to the technology of the present disclosure is an image processing apparatus according to the fourteenth aspect, in which the wavelength band image is an image showing a characteristic part of a subject.

An 18th aspect according to the technology of the present disclosure is the image processing apparatus according to the 16th aspect, wherein the wavelength band image is an image showing a characteristic part of a subject, and the reference position is a position corresponding to the characteristic part. It is an image processing device.

A nineteenth aspect of the technology of the present disclosure is an image processing apparatus according to the sixteenth aspect, wherein the reference position is a position of any one of the wavelength band images. be.

A 20th aspect according to the technology of the present disclosure is an image processing apparatus according to the 17th aspect, in which the subject has a point and the characteristic part is a point.

A twenty-first aspect according to the technology of the present disclosure is an image processing apparatus according to the seventeenth aspect, in which the subject has a checkered pattern and the characteristic portion is an intersection included in the checkered pattern.

A twenty-second aspect according to the technology of the present disclosure is an image processing apparatus according to the seventeenth aspect, in which the subject is an image processing apparatus that is a calibration member.

A twenty-third aspect of the technology of the present disclosure is the image processing apparatus according to the seventeenth aspect, wherein the subject has a plurality of characteristic parts, and the plurality of characteristic parts are arranged in a straight line. It is.

A twenty-fourth aspect according to the technology of the present disclosure is the image processing apparatus according to the seventeenth aspect, wherein the subject has a plurality of characteristic parts, and the plurality of characteristic parts are arranged at equal intervals. It is.

A twenty-fifth aspect according to the technology of the present disclosure is an image processing apparatus according to the seventeenth aspect, in which the subject is an image processing apparatus including an object for inspection.

A twenty-sixth aspect according to the technology of the present disclosure is an image processing apparatus according to the twenty-fifth aspect, in which the subject is an image processing apparatus including a plurality of objects for inspection.

A twenty-seventh aspect of the technology of the present disclosure is the image processing device according to any one of the first to twenty-sixth aspects, wherein the optical system includes polarizing filters provided corresponding to each filter. The plurality of polarizing filters have mutually different polarization axes, and the imaging device includes an image sensor having a plurality of pixel blocks, and each pixel block includes a plurality of types of polarizers having mutually different polarization axes. This is an image processing device provided.

A twenty-eighth aspect of the technology of the present disclosure is an image processing method applied to an image output from an imaging device including an optical system, the optical system comprising a plurality of The image processing method includes performing a process on an image to correct an image shift due to a position shift of an optical image caused by being separated by a plurality of filters.

A twenty-ninth aspect of the technology of the present disclosure is a program for causing a computer to perform image processing on an image output from an imaging device including an optical system, wherein the optical system is provided around an optical axis. The program includes a plurality of filters, and the image processing includes processing for correcting an image shift due to a position shift of an optical image caused by dispersing an image by a plurality of filters.

FIG. 1 is a perspective view showing an example of an imaging device. It is an exploded perspective view showing an example of a pupil division filter. FIG. 2 is a block diagram showing an example of the hardware configuration of an imaging device. FIG. 2 is an explanatory diagram showing an example of the configuration of a photoelectric conversion element. FIG. 2 is a block diagram illustrating an example of a manner in which a multispectral image is generated based on a plurality of spectral images. FIG. 3 is a front view showing an example of a multispectral image generated based on a plurality of spectral images having image shifts. FIG. 2 is a schematic diagram showing a first example of the relationship between an aperture formed in a pupil splitting filter and an optical image. FIG. 6 is a schematic diagram showing a second example of the relationship between an aperture formed in a pupil splitting filter and an optical image. FIG. 2 is a block diagram illustrating an example of a functional configuration for executing multispectral image generation processing. FIG. 2 is a block diagram illustrating an example of the operation of an output value acquisition section and an interference removal processing section. FIG. 3 is a block diagram illustrating an example of the operation of a correction processing section. FIG. 2 is a block diagram illustrating an example of the operation of a multispectral image generation section. FIG. 2 is a block diagram illustrating an example of a hardware configuration of an imaging device and an example of a functional configuration for executing correction value derivation processing. FIG. 2 is a block diagram showing an example of an optical image formed on a light-receiving surface when a dot chart is imaged by an imaging device. FIG. 2 is a block diagram illustrating an example of the operation of a spectral image acquisition unit. FIG. 3 is a block diagram illustrating an example of the operation of a correction value deriving section. 3 is a flowchart illustrating an example of the flow of multispectral image generation processing. 7 is a flowchart illustrating an example of the flow of correction value derivation processing. It is a block diagram showing an example of operation of a correction value calculation part concerning a 1st modification. It is a block diagram showing an example of operation of a correction value calculation part concerning a 2nd modification. FIG. 2 is a block diagram illustrating an example of an optical image formed on a light receiving surface when a lattice chart is imaged by an imaging device. It is a block diagram showing an example of operation of a spectral image acquisition part concerning a 3rd modification. It is a block diagram showing an example of operation of a correction value derivation part concerning a 3rd modification. It is a block diagram showing an example of operation of a correction value derivation part concerning a 4th modification. It is a block diagram showing an example of operation of a correction value derivation part concerning a 5th modification. It is a block diagram showing an example of operation of a correction value derivation part concerning a 6th modification. It is a block diagram showing an example of operation of a correction value derivation part concerning a 7th modification. It is a block diagram showing an example of operation of a correction value derivation part concerning the 8th modification. It is a block diagram showing an example of operation of a correction value derivation part concerning a 9th modification.

An example of an embodiment of an image processing device, an image processing method, and a program according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

First, the words used in the following explanation will be explained.

LED is an abbreviation for "light emitting diode." CMOS is an abbreviation for "Complementary Metal Oxide Semiconductor." CCD is an abbreviation for “Charge Coupled Device”. I/F is an abbreviation for "Interface". RAM is an abbreviation for "Random Access Memory." CPU is an abbreviation for "Central Processing Unit." GPU is an abbreviation for “Graphics Processing Unit.” EEPROM is an abbreviation for "Electrically Erasable and Programmable Read Only Memory." HDD is an abbreviation for "Hard Disk Drive." EL is an abbreviation for "Electro Luminescence". TPU is an abbreviation for "Tensor processing unit". SSD is an abbreviation for "Solid State Drive." USB is an abbreviation for "Universal Serial Bus." ASIC is an abbreviation for “Application Specific Integrated Circuit.” FPGA is an abbreviation for "Field-Programmable Gate Array." PLD is an abbreviation for “Programmable Logic Device”. SoC is an abbreviation for "System-on-a-Chip." IC is an abbreviation for "Integrated Circuit."

In the description of this specification, the term "center" refers to an error that is generally allowed in the technical field to which the technology of the present disclosure belongs, in addition to the perfect "center," and is contrary to the spirit of the technology of the present disclosure. Refers to the ``center'' in the sense that it includes a certain degree of error. In the description of this specification, "same" means not only "the same" but also an error that is generally allowed in the technical field to which the technology of the present disclosure belongs, and which is contrary to the spirit of the technology of the present disclosure. It refers to "the same" in the sense that it includes a certain degree of error. In the description of this specification, "orthogonal" means not only complete "orthogonal" but also an error that is generally allowed in the technical field to which the technology of the present disclosure belongs, and is contrary to the spirit of the technology of the present disclosure. This refers to "orthogonal" in the sense that it includes a degree of error that does not occur. In the description of this specification, "perpendicular" refers to an error that is generally allowed in the technical field to which the technology of the present disclosure belongs, in addition to being perfectly perpendicular, to the extent that it does not go against the spirit of the technology of the present disclosure. It refers to vertical in the sense of including the error of. In the description of this specification, a "straight line" refers to not only a perfect straight line but also an error that is generally allowed in the technical field to which the technology of the present disclosure belongs, and that does not go against the spirit of the technology of the present disclosure. It refers to a straight line that includes the error of. In the description of this specification, "equal spacing" refers to errors that are generally allowed in the technical field to which the technology of the present disclosure belongs, in addition to perfectly equal spacing, and is contrary to the spirit of the technology of the present disclosure. It refers to equal intervals that include a certain degree of error.

As shown in FIG. 1 as an example, the imaging device 10 includes a lens device 12 and an imaging device body 14. The imaging device 10 is an example of an "imaging device" and an "image processing device" according to the technology of the present disclosure. The lens device 12 includes a pupil division filter 16 that separates incident light into a plurality of wavelength bands. The imaging device 10 is a multispectral camera that generates and outputs a multispectral image 74 by capturing light that has been split into multiple wavelength bands by the pupil splitting filter 16.

In the present embodiment, a multispectral image generated based on light separated into three wavelength bands will be described as an example of the multispectral image 74. The three wavelength bands are just an example, and there may be four or more wavelength bands. That is, the imaging device 10 may be a multispectral camera that can image a subject with higher wavelength resolution than a multispectral camera that can image light separated into three wavelength bands.

Further, the multispectral image 74 may include an image obtained by capturing light in the visible light band, and may include a wavelength band that cannot be perceived by the human eye (for example, a near-infrared band and/or The image may include an image in which light in the ultraviolet band, etc.) is visualized.

Examples of uses of the multispectral image 74 include measurement of a subject as an object to be observed in various fields such as medicine, agriculture, and industry, inspection of a subject, analysis of a subject, and evaluation of a subject. .

As shown in FIG. 2 as an example, the pupil division filter 16 includes a frame 18, spectral filters 20A to 20C, and polarization filters 22A to 22C.

The frame 18 has openings 24A to 24C. The openings 24A to 24C have the same shape. As an example, each of the openings 24A to 24C has a fan shape. Note that the shape of each of the openings 24A to 24C may be a shape other than a fan shape (for example, a square shape or a circular shape). The openings 24A to 24C are arranged at equal intervals around the optical axis OA. The center of gravity G of each aperture 24A to 24C is located off the optical axis OA. Each center of gravity position G is the center of the geometry of each aperture 24A-24C. Hereinafter, unless it is necessary to separately explain the openings 24A to 24C, each opening 24A to 24C will be referred to as an "opening 24." The opening 24 is an example of an "opening" according to the technology of the present disclosure.

The spectral filters 20A to 20C are provided in the apertures 24A to 24C, respectively, so that they are arranged at equal intervals around the optical axis OA. Each of the spectral filters 20A to 20C is a bandpass filter that transmits light in a specific wavelength band. The spectral filters 20A to 20C have different wavelength bands. Specifically, the spectral filter 20A has a first wavelength band _λ1 , the spectral filter 20B has a second wavelength band _λ2 , and the spectral filter 20C has a third wavelength band _λ3. has.

Hereinafter, unless it is necessary to separately explain the spectral filters 20A to 20C, each of the spectral filters 20A to 20C will be referred to as a "spectral filter 20." The spectral filter 20 is an example of a "filter" according to the technology of the present disclosure. In addition, if it is not necessary to separately explain the first wavelength band λ ₁ , the second wavelength band λ ₂ , and the third wavelength band λ ₃ , the first wavelength band λ ₁ , the second wavelength band λ ₂ , and Each of the third wavelength bands λ ₃ is referred to as a “wavelength band λ”.

The polarizing filters 22A to 22C are provided corresponding to the spectral filters 20A to 20C, respectively. Specifically, the polarizing filter 22A is provided in the aperture 24A, and is overlapped with the spectral filter 20A. The polarizing filter 22B is provided in the aperture 24B and overlapped with the spectral filter 20B. The polarizing filter 22C is provided in the aperture 24C and is overlapped with the spectral filter 20C.

Each of the polarizing filters 22A to 22C is an optical filter that transmits light vibrating in a specific direction. The polarizing filters 22A to 22C have polarization axes OA with mutually different polarization angles. Specifically, polarizing filter 22A has a first polarizing angle α 1, polarizing filter 22B has a second polarizing angle _{α 2} _, and polarizing filter 22C has a third polarizing angle α ₃ . has. Note that the polarization axis may also be referred to as the transmission axis. As an example, the first polarization angle α ₁ is set to 0°, the second polarization angle α ₂ is set to 45°, and the third polarization angle α ₃ is set to 90°. .

Hereinafter, unless it is necessary to explain the polarizing filters 22A to 22C separately, each of the polarizing filters 22A to 22C will be referred to as a "polarizing filter 22." The polarizing filter 22 is an example of the "polarizing filter 22" according to the technology of the present disclosure. Furthermore, if it is not necessary to separately explain the first polarization angle α ₁ , the second polarization angle α ₂ , and the third polarization angle α ₃ , the first polarization angle α ₁ , the second polarization angle α ₂ , and Each of the third polarization angles α ₃ is referred to as a “polarization angle α”.

In the example shown in FIG. 2, the number of apertures 24 is three, corresponding to the number of wavelength bands λ, but the number of apertures 24 is three, corresponding to the number of wavelength bands λ. (i.e., the number of the plurality of spectral filters 20). Furthermore, unused openings 24 among the plurality of openings 24 may be covered by a shielding member (not shown). Further, in the example shown in FIG. 2, the plurality of spectral filters 20 have different wavelength bands λ, but the plurality of spectral filters 20 may include spectral filters 20 having the same wavelength band λ. .

As shown in FIG. 3 as an example, the lens device 12 includes an optical system 26, and the imaging device body 14 includes an image sensor 28. The optical system 26 is an example of an "optical system" according to the technology of the present disclosure, and the image sensor 28 is an example of an "image sensor" according to the technology of the present disclosure.

The optical system 26 includes a pupil splitting filter 16, a first lens 30, and a second lens 32. The first lens 30, the pupil splitting filter 16, and the second lens 32 are arranged along the optical axis OA of the lens device 12 from the subject 4 side to the image sensor 28 side. The lenses 32 are arranged in this order. In the following description, the subject 4 side may be referred to as the "object side" and the image sensor 28 side may be referred to as the "image side".

The first lens 30 causes light obtained by the light emitted from the light source 2 being reflected by the subject 4 (hereinafter referred to as "subject light") to enter the pupil division filter 16. The subject light is an example of "light" according to the technology of the present disclosure. The second lens 32 forms an image of the subject light that has passed through the pupil splitting filter 16 onto a light receiving surface 34A of a photoelectric conversion element 34 provided in the image sensor 28.

The light source 2 is, for example, an LED light source, a laser light source, or an incandescent light bulb. The light emitted from the light source 2 is unpolarized. The light source 2 may be included in the imaging device body 14 and/or the lens device 12. Furthermore, the light emitted from the light source may be natural light.

The pupil splitting filter 16 is placed at the pupil position of the optical system 26. The pupil position refers to the aperture surface that limits the brightness of the optical system 26. The pupil position here includes a nearby position, and the nearby position refers to the range from the entrance pupil to the exit pupil. The configuration of the pupil division filter 16 is as described using FIG. 2. In FIG. 3, for convenience, a plurality of spectral filters 20 and a plurality of polarizing filters 22 are shown arranged in a straight line along a direction perpendicular to the optical axis OA.

The image sensor 28 includes a photoelectric conversion element 34 and a signal processing circuit 36. The image sensor 28 is, for example, a CMOS image sensor. In this embodiment, a CMOS image sensor is exemplified as the image sensor 28, but the technology of the present disclosure is not limited to this. For example, the image sensor 28 may be another type of image sensor such as a CCD image sensor. The technology of the present disclosure is realized.

As an example, FIG. 3 shows a schematic configuration of the photoelectric conversion element 34. Furthermore, as an example, FIG. 4 specifically shows the configuration of a part of the photoelectric conversion element 34. The photoelectric conversion element 34 includes a pixel layer 38, a polarizing filter layer 40, and a spectral filter layer 42. Note that the configuration of the photoelectric conversion element 34 shown in FIG. 3 is an example, and the technology of the present disclosure is valid even if the photoelectric conversion element 34 does not include the spectral filter layer 42.

The pixel layer 38 has a plurality of pixels 44. The plurality of pixels 44 are arranged in a matrix and form a light receiving surface 34A of the photoelectric conversion element 34. Each pixel 44 is a physical pixel having a photodiode (not shown), photoelectrically converts the received light, and outputs an electrical signal according to the amount of received light.

Hereinafter, in order to distinguish from the pixels forming the multispectral image 74, the pixels 44 provided in the photoelectric conversion element 34 will be referred to as "physical pixels 44." Moreover, the pixels forming the multispectral image 74 are referred to as "image pixels."

The photoelectric conversion element 34 outputs the electrical signals output from the plurality of physical pixels 44 to the signal processing circuit 36 as image data. The signal processing circuit 36 digitizes the analog imaging data input from the photoelectric conversion element 34. The image data is image data indicating a captured image 70.

A plurality of physical pixels 44 form a plurality of pixel blocks 46. Each pixel block 46 is formed by a total of four physical pixels 44, two in the vertical direction and two in the horizontal direction. In FIG. 3, for convenience, the four physical pixels 44 forming each pixel block 46 are shown arranged in a straight line along the direction perpendicular to the optical axis OA, but the four physical pixels 44 are arranged adjacent to the photoelectric conversion element 34 in the vertical and horizontal directions (see FIG. 4). The physical pixel 44 is an example of a "pixel" according to the technology of the present disclosure, and the pixel block 46 is an example of a "pixel block" according to the technology of the present disclosure.

The polarizing filter layer 40 has multiple types of polarizers 48A to 48D. Each polarizer 48A to 48D is an optical filter that transmits light vibrating in a specific direction. The polarizers 48A to 48D have polarization axes OA with mutually different polarization angles θ. Specifically, polarizer 48A has a first polarization angle θ ₁ , polarizer 48B has a second polarization angle θ ₂ , and polarizer 48C has a third polarization angle θ ₃ . , and the polarizer 48D has a fourth polarization angle _θ4 . As an example, the first polarization angle θ ₁ is set to 0°, the second polarization angle θ ₂ is set to 45°, and the third polarization angle θ ₃ is set to 90°. , the fourth polarization angle θ ₄ is set to 135°.

Hereinafter, unless it is necessary to explain the polarizers 48A to 48D separately, each of the polarizers 48A to 48D will be referred to as a "polarizer 48." The polarizer 48 is an example of a "polarizer" according to the technology of the present disclosure. In addition, if it is not necessary to separately explain the first polarization angle θ ₁ , the second polarization angle θ ₂ , the third polarization angle θ ₃ , and the fourth polarization angle θ ₄ , the first polarization angle θ ₁ , the The second polarization angle θ ₂ , the third polarization angle θ ₃ , and the fourth polarization angle θ ₄ are each referred to as “polarization angle θ”.

The spectral filter layer 42 includes a B filter 50A, a G filter 50B, and an R filter 50C. The B filter 50A is a blue band filter that transmits most of the light in the blue wavelength band among the plurality of wavelength bands. The G filter 50B is a green band filter that transmits the most light in the green wavelength band among the plurality of wavelength bands. The R filter 50C is a red band filter that transmits most of the light in the red wavelength band among the plurality of wavelength bands. A B filter 50A, a G filter 50B, and an R filter 50C are assigned to each pixel block 46.

In FIG. 3, for convenience, the B filter 50A, the G filter 50B, and the R filter 50C are shown arranged in a straight line along the direction perpendicular to the optical axis OA, but as an example, as shown in FIG. The B filter 50A, the G filter 50B, and the R filter 50C are arranged in a matrix in a predetermined pattern arrangement. In the example shown in FIG. 4, the B filter 50A, the G filter 50B, and the R filter 50C are arranged in a matrix in a Bayer arrangement, as an example of a predetermined pattern arrangement. Note that the predetermined pattern arrangement may be an RGB stripe arrangement, an R/G checkered arrangement, an X-Trans (registered trademark) arrangement, a honeycomb arrangement, or the like other than the Bayer arrangement.

Hereinafter, unless it is necessary to explain the B filter 50A, G filter 50B, and R filter 50C separately, the B filter 50A, the G filter 50B, and the R filter 50C will be referred to as "filters 50", respectively.

As shown in FIG. 3 as an example, the imaging device body 14 includes a control driver 52, an input/output I/F 54, a computer 56, and a display 58 in addition to the image sensor 28. A signal processing circuit 36, a control driver 52, a computer 56, and a display 58 are connected to the input/output I/F 54.

The computer 56 has a processor 60, a storage 62, and a RAM 64. The processor 60 controls the entire imaging device 10 . The processor 60 is, for example, an arithmetic processing device including a CPU and a GPU, and the GPU operates under the control of the CPU and is responsible for executing processing regarding images. Here, an arithmetic processing unit including a CPU and a GPU is cited as an example of the processor 60, but this is just an example, and the processor 60 may be one or more CPUs with integrated GPU functions. , one or more CPUs without integrated GPU functionality.

The processor 60, storage 62, and RAM 64 are connected via a bus 66, and the bus 66 is connected to the input/output I/F 54. Processor 60 is an example of a "processor" according to the technology of the present disclosure. The computer 56 is an example of a "computer" according to the technology of the present disclosure.

The storage 62 is a non-temporary storage medium and stores various parameters and programs. For example, storage 62 is flash memory (eg, EEPROM). However, this is just an example, and an HDD or the like may be used as the storage 62 in addition to a flash memory. The RAM 64 temporarily stores various information and is used as a work memory. Examples of the RAM 64 include DRAM and/or SRAM.

The processor 60 reads a necessary program from the storage 62 and executes the read program on the RAM 64. Processor 60 controls control driver 52 and signal processing circuit 36 according to a program executed on RAM 64. The control driver 52 controls the photoelectric conversion element 34 under the control of the processor 60.

The display 58 is, for example, a liquid crystal display or an EL display, and displays various images including the multispectral image 74. Note that the display 58 may be included in an external device (not shown) that is communicably connected to the imaging device 10.

As an example, as shown in FIG. 5, the imaging device 10 generates spectral images 72A to 72C corresponding to each wavelength band λ based on the captured image 70, and generates a multispectral image 74 based on the spectral images 72A to 72C. be done. The spectral image 72A is a spectral image corresponding to the first wavelength band _λ1 , the spectral image 72B is a spectral image corresponding to the second wavelength band _λ2 , and the spectral image 72C is a spectral image corresponding to the third wavelength band _λ3. This is the corresponding spectral image.

The specific details of the process of generating the spectral images 72A to 72C based on the captured image 70 and the process of generating the multispectral image 74 based on the spectral images 72A to 72C will be described later. Hereinafter, unless it is necessary to explain the spectral images 72A to 72C separately, each of the spectral images 72A to 72C will be referred to as a "spectral image 72." The spectral image 72 is an example of a "spectral image" and an "image" according to the technology of the present disclosure.

Here, FIG. 6 shows an example of the multispectral image 74. Multispectral image 74 includes multiple spectral images 72. Each spectral image 72 includes an object having the shape of the letter "F" as an image.

In the example shown in FIG. 6, an image shift (hereinafter referred to as "image shift") has occurred in each spectrum image 72. Each image shift differs for each spectrum image 72. The reason why the image shift differs for each spectral image 72 is that the spectral filter 20 (see FIG. 2) corresponding to each spectral image 72 is provided in the aperture 24 formed at different positions with respect to the optical axis OA, An example of this is that parallax occurs between the apertures 24 of. When parallax occurs between the plurality of apertures 24, a positional shift occurs in the optical image formed on the light receiving surface 34A for each wavelength band λ due to the parallax. The positional shift of the optical image has a different magnitude and direction for each wavelength band λ.

As described above, the cause of different image shifts for each spectral image 72 is, for example, that the parallax generated in the optical system 26 is different for each wavelength band λ due to the plurality of apertures 24 being formed at different positions. Can be mentioned. In a state where an image shift occurs in each spectral image 72, the image quality of the multispectral image 74 deteriorates compared to a case where no image shift occurs.

Further, the positional shift of the optical image is affected not only by the position of the aperture 24 but also by the shape of the aperture 24. Here, FIG. 7 shows an example of a mode in which the light emitted from the object point 76 is imaged on the light receiving surface 34A when the shape of the aperture 24 is fan-shaped, and FIG. An example of a mode in which light emitted from the object point 76 is imaged on the light receiving surface 34A when the shape of the object point 24 is rectangular is shown.

As an example, as shown in FIGS. 7 and 8, since the optical system 26 generally has an aberration, the light does not converge on one point on the light receiving surface 34A, but spreads out corresponding to the shape of the aperture 24. Therefore, the optical image 78 formed on the light receiving surface 34A has a shape corresponding to the shape of the aperture 24. When the positional deviation of the optical image 78 is defined by the center of gravity 78A of the shape of the optical image 78 (that is, the center of the geometric shape), the positional deviation of the optical image 78 differs depending on the shape of the optical image 78. In this way, the positional shift of the optical image 78 is affected not only by the position of the aperture 24 but also by the shape of the aperture 24.

In other words, the positional shift of the optical image 78 differs depending on the center of gravity position G of the aperture 24, which is defined by the position and shape of the aperture 24. For example, the center of gravity position G of the aperture 24 is defined by its relative position to the optical axis OA. Further, the positional shift of the optical image 78 also differs depending on the characteristics of the optical system 26 other than the center of gravity position G of the aperture 24. The characteristics of the optical system 26 other than the center of gravity position G of the aperture 24 include, for example, the arrangement position of the pupil division filter 16.

In addition to the parallax that differs for each wavelength band λ, factors that cause image shift include distortion occurring in the optical system 26 and/or trapezoidation caused by imaging a subject surface that is not perpendicular to the optical axis OA. Examples include distortion.

In this embodiment, in order to obtain a multispectral image 74 with higher image quality than in the case where the multispectral image 74 is generated based on the spectral image 72 in which image shift has occurred, the processor 60 performs the following process. A multispectral image generation process is performed.

As an example, as shown in FIG. 9, a multispectral image generation program 80 is stored in the storage 62. The multispectral image generation program 80 is an example of a "program" according to the technology of the present disclosure. The processor 60 reads the multispectral image generation program 80 from the storage 62 and executes the read multispectral image generation program 80 on the RAM 64. Processor 60 executes multispectral image generation processing to generate multispectral image 74 according to multispectral image generation program 80 executed on RAM 64 .

The multispectral image generation process is realized by the processor 60 operating as an output value acquisition section 82, an interference removal processing section 84, a correction processing section 86, and a multispectral image generation section 88 according to the multispectral image generation program 80. .

As an example, as shown in FIG. 10, when the imaging data output from the image sensor 28 is input to the processor 60, the output value acquisition unit 82 calculates the output value Y of each physical pixel 44 based on the imaging data. get. The output value Y of each physical pixel 44 corresponds to the luminance value of each pixel included in the captured image 70 indicated by the captured image data.

Here, the output value Y of each physical pixel 44 is a value that includes interference (that is, crosstalk). That is, since light in each wavelength band λ of the first wavelength band λ ₁ , the second wavelength band λ ₂ , and the third wavelength band λ ₃ is incident on each physical pixel 44, the output value Y is The value is a mixture of a value corresponding to the light amount in the band λ ₁ , a value depending on the light amount in the second wavelength band λ ₂ , and a value depending on the light amount in the third wavelength band λ ₃ .

In order to obtain the multispectral image 74, the processor 60 performs a process of separating and extracting a value corresponding to each wavelength band λ from the output value Y for each physical pixel 44, that is, a process of removing crosstalk. It is necessary to perform removal processing on the output value Y. Therefore, in this embodiment, in order to obtain the multispectral image 74 (see FIG. 7), the interference removal processing unit 84 applies interference to the output value Y of each physical pixel 44 acquired by the output value acquisition unit 82. Execute the removal process.

Here, the interference removal process will be explained. The output value Y of each physical pixel 44 includes each brightness value for each polarization angle θ for red, green, and blue as a component of the output value Y. The output value Y of each physical pixel 44 is expressed by equation (1).

However, Y _{θ1_R} is the brightness value of the red component of the output value Y whose polarization angle is the first polarization angle θ ₁ , and Y _{θ2_R} is the brightness value of the component of the output value Y that is red and whose polarization angle is the second polarization angle θ. ₂ , Y _{θ3_R} is the brightness value of the red component of the output value Y whose polarization angle is the third polarization angle θ 3, and Y _{θ4_R} is the brightness value of the red component of the output value Y whose polarization angle is the third polarization angle θ ₃ . is the brightness value of the component whose fourth polarization angle is _θ4 .

Further, Y _{θ1_G} is the luminance value of the green component of the output value Y whose polarization angle is the first polarization angle θ ₁ , and Y _{θ2_G} is the luminance value of the component of the output value Y that is green and whose polarization angle is the second polarization angle θ. ₂ , Y _{θ3_G} is the luminance value of the component of the output value Y that is green and has a polarization angle of the third polarization angle θ 3, and Y _{θ4_G} is the luminance value of the component that is green and has the polarization angle of the third polarization angle θ ₃ of the output value Y. is the brightness value of the component whose fourth polarization angle is _θ4 .

Further, Y _{θ1_B} is the luminance value of the blue component of the output value Y whose polarization angle is the first polarization angle θ ₁ , and Y _{θ2_B} is the luminance value of the blue component of the output value Y whose polarization angle is the second polarization angle θ. ₂ , Y _{θ3_B} is the luminance value of the blue component of the output value Y whose polarization angle is the third polarization angle θ ₃ , and Y _{θ4_B} is the polarization angle of the blue component of the output value Y. is the brightness value of the component whose fourth polarization angle is _θ4 .

The pixel value X of each image pixel forming the multispectral image 74 is the brightness value X λ1 of polarized light in a first wavelength band λ ₁ having a first polarization angle α ₁ (hereinafter referred to as "first wavelength band polarized light" ₎ . , a brightness value X _λ2 of polarized light in a second wavelength band λ ₂ having a second polarization angle α ₂ (hereinafter referred to as “second wavelength band polarized light”), and a third wavelength band having a third polarization angle α ₃ A luminance value X _λ3 of polarized light of λ ₃ (hereinafter referred to as “third wavelength band polarized light”) is included as a component of the pixel value X. The pixel value X of each image pixel is expressed by equation (2).

The output value Y of each physical pixel 44 is expressed by equation (3).

In equation (3), A is an interference matrix. The interference matrix A (not shown) is a matrix indicating characteristics of interference. The interference matrix A includes a plurality of known information such as the spectrum of the subject light, the spectral transmittance of the first lens 30, the spectral transmittance of the second lens 32, the spectral transmittance of the plurality of spectral filters 20, and the spectral sensitivity of the image sensor 28. predefined based on the value of .

When the interference cancellation matrix, which is the general inverse of the interference matrix A, is A ⁺ , the pixel value X of each image pixel is expressed by equation (4).

Similar to the interference matrix A, the interference removal matrix A ⁺ also includes the spectrum of the subject light, the spectral transmittance of the first lens 30, the spectral transmittance of the second lens 32, the spectral transmittance of the plurality of spectral filters 20, and the image sensor. This is a matrix defined based on the spectral sensitivity, etc. of No. 28. The interference cancellation matrix A ⁺ is stored in the storage 62 in advance.

The interference cancellation processing unit 84 acquires the interference cancellation matrix A ⁺ stored in the storage 62 and the output value Y of each physical pixel 44 acquired by the output value acquisition unit 82, and uses the acquired interference cancellation matrix A ⁺ Based on the output value Y of each physical pixel 44, the pixel value X of each image pixel is output using equation (4).

Here, as described above, the pixel value X of each image pixel is the brightness value X _λ1 of the first wavelength band polarized light, the brightness value X _λ2 of the second wavelength band polarized light, and the brightness value X _λ3 of the third wavelength band polarized light. are included as components of the pixel value X.

The spectral image 72A of the captured image 70 is an image corresponding to the luminance value X _λ1 of light in the first wavelength band λ ₁ (that is, an image based on the luminance value X _λ1 ). The spectral image 72B of the captured image 70 is an image corresponding to the brightness value X _λ2 of light in the second wavelength band λ ₂ (that is, an image based on the brightness value X _λ2 ). The spectral image 72C of the captured image 70 is an image corresponding to the brightness value X _λ3 of light in the third wavelength band λ ₃ (that is, an image based on the brightness value X _λ3 ).

In this way, by performing the interference removal process by the interference removal processing unit 84, the captured image 70 is divided into the spectrum image 72A corresponding to the brightness value X _λ1 of the first wavelength band polarized light and the brightness of the second wavelength band polarized light. It is separated into a spectral image 72B corresponding to the value X _λ2 and a spectral image 72C corresponding to the luminance value X _λ3 of the third wavelength band polarized light. That is, the captured image 70 is separated into spectral images 72 for each wavelength band λ of the plurality of spectral filters 20. As described above, image shift occurs in each spectrum image 72 (see FIG. 6). The image shift differs for each spectrum image 72.

As an example, as shown in FIG. 11, the correction processing unit 86 performs a process of correcting image shift (hereinafter referred to as "correction process") on each spectrum image 72. The correction process is an example of "image processing" and "calibration process" according to the technology of the present disclosure. In the storage 62, correction value groups 90A to 90C are stored in advance. The correction value group 90A includes a plurality of correction values 92A for correcting image shift of the spectral image 72A. The correction value group 90B includes a plurality of correction values 92B for correcting image shift of the spectral image 72B. The correction value group 90C includes a plurality of correction values 92C for correcting image shift of the spectral image 72C. The process of deriving each of the correction values 92A to 92C (hereinafter referred to as "correction value derivation process") will be described in detail later.

Hereinafter, when there is no need to distinguish between the correction values 92A to 92C, each correction value 92A to 92C will be referred to as a "correction value 92." Each correction value 92 may be determined for each image pixel included in the spectral image 72, or may be determined for each image region of the spectral image 72. By performing a correction process based on each correction value 92 on each spectral image 72, the image shift of each spectral image 72 is corrected.

As an example, as shown in FIG. 12, the multispectral image generation unit 88 generates a multispectral image 74 by combining a plurality of spectral images 72 whose image deviations have been corrected by the correction processing unit 86. For example, multispectral image data representing multispectral image 74 is output to display 58. Display 58 displays multispectral image 74 based on the multispectral image data. Note that the multispectral image data may be output to a device other than the display 58 (not shown).

FIG. 13 shows an example of a processing device 100 for deriving the correction values 92 corresponding to each of the above-mentioned spectral images 72. The processing device 100 includes a computer 102. Computer 102 includes a processor 104, storage 106, and RAM 108. The processor 104, storage 106, and RAM 108 are realized by the same hardware as the above-described processor 60, storage 62, and RAM 64 (see FIG. 2).

A correction value derivation program 110 is stored in the storage 106. The processor 104 reads the correction value derivation program 110 from the storage 106 and executes the read correction value derivation program 110 on the RAM 108. The processor 104 executes a correction value derivation process for deriving each correction value 92 according to a correction value derivation program 110 executed on the RAM 108.

The correction value derivation process is realized by the processor 104 operating as the spectral image acquisition unit 112 and the correction value derivation unit 114 according to the correction value derivation program 110. Note that although the processing device 100 is described here as a separate device from the imaging device 10, the processing device 100 may be the imaging device 10. That is, the correction value derivation process for deriving the correction value 92 may be performed by the imaging device 10.

FIGS. 14 to 16 show an example of how the correction value 92 is derived. In the examples shown in FIGS. 14 to 16, a dot chart 120 is used as the subject for deriving the correction value 92. The dot chart 120 has a plurality of dots 122. Each dot 122 is a circular point. The size of the dots 122 may be set arbitrarily. For example, the plurality of dots 122 are arranged on a pair of diagonal lines (not shown) of the dot chart 120. The dot chart 120 is an example of a "subject" and a "calibration member" according to the technology of the present disclosure. The dot 122 is an example of a "characteristic part" and a "point" according to the technology of the present disclosure.

FIG. 14 shows an example of optical images 130A to 130C, optical images 132A to 132C, and optical images 134A to 134C that are formed on the light receiving surface 34A when the dot chart 120 is imaged by the imaging device 10. There is. Optical images 130A to 130C are optical images corresponding to the first dot 122A located at the upper right of the plurality of dots 122, and optical images 132A to 132C are optical images corresponding to the first dot 122A located at the lower right of the plurality of dots 122. This is an optical image corresponding to the two dots 122B, and the optical images 134A to 134C are optical images corresponding to the third dot 122C located at the center of the plurality of dots 122.

The optical image 130A, the optical image 132A, and the optical image 134A are optical images corresponding to the first wavelength band λ ₁ , and the optical image 130B, the optical image 132B, and the optical image 134B are the optical images corresponding to the second wavelength band λ ₂ . The optical image 130C, the optical image 132C, and the optical image 134C are optical images corresponding to the third wavelength band λ ₃ . The first reference position 136A is a position corresponding to the first dot 122A within the light receiving surface 34A, the second reference position 136B is a position corresponding to the second dot 122B within the light receiving surface 34A, and the third reference position is a position corresponding to the second dot 122B within the light receiving surface 34A. 136C is a position corresponding to the third dot 122C within the light receiving surface 34A.

The optical images 130A to 130C are shifted from the first reference position 136A. The direction and amount of positional deviation of the optical images 130A to 130C are different from each other. Similarly, the optical images 132A to 132C are shifted from the second reference position 136B, and the optical images 134A to 134C are misaligned from the third reference position 136C. The directions and amounts of positional deviations of optical images 132A to 132C are different from each other, and the directions and amounts of positional deviations of optical images 134A to 134C are different from each other.

Further, for example, the first dot 122A and the second dot 122B are arranged at symmetrical positions in the vertical direction of the dot chart 120, but the direction of each positional shift of the optical images 130A to 130C and the optical images 132A to 132C and The amounts differ from each other depending on the location and shape of the opening 24 (see FIG. 2). Therefore, the optical images 130A to 130C and the optical images 132A to 132C are not symmetrical in the vertical direction of the light receiving surface 34A, but are asymmetrical. Further, as in the optical images 134A to 134C corresponding to the third dot 122C, positional deviation also occurs at the center of the light receiving surface 34A.

As an example, as shown in FIG. 15, the spectral image acquisition unit 112 acquires each of the spectral images 72A to 72C obtained by capturing the dot chart 120. For example, the spectral image 72A includes a wavelength band image 140A corresponding to the optical image 130A, the spectral image 72B includes a wavelength band image 140B corresponding to the optical image 130B, and the spectral image 72C includes a wavelength band image 140B corresponding to the optical image 130B. includes a wavelength band image 140C corresponding to the optical image 130C.

The reference position 142 is a position corresponding to the first dot 122A in each spectrum image 72. The wavelength band images 140A to 140C are shifted from the reference position 142. The direction and amount of positional shift of the wavelength band images 140A to 140C are different from each other. Hereinafter, unless it is necessary to separately explain the wavelength band images 140A to 140C, the wavelength band images 140A to 140C will be referred to as "wavelength band images 140." Moreover, hereinafter, the wavelength band image (not shown) corresponding to the dots 122 other than the first dot 122A may be referred to as the "wavelength band image 140." The wavelength band image 140 is an example of a "wavelength band image" according to the technology of the present disclosure.

As an example, as shown in FIG. 16, the correction value derivation unit 114 derives the correction value 92 in the following manner. For example, the correction value deriving unit 114 acquires each of the spectral images 72A to 72C acquired by the spectral image acquiring unit 112. Further, the correction value deriving unit 114 calculates the position of the wavelength band image 140A in the spectral image 72A and the position of the reference position 142 in the spectral image 72A to the position of the first dot 122A in the dot chart 120 (see FIG. 15). It is specified based on the position and design values regarding the optical system 26 (see FIG. 2).

Similarly, the correction value deriving unit 114 converts the position of the wavelength band image 140B in the spectral image 72B and the position of the reference position 142 in the spectral image 72B to the position of the first dot 122A in the dot chart 120 and the optical It is specified based on the design value etc. regarding the system 26. Further, the correction value deriving unit 114 converts the position of the wavelength band image 140C in the spectral image 72C and the position of the reference position 142 in the spectral image 72C to the position of the first dot 122A in the dot chart 120 and the optical system. It is specified based on the design value etc. regarding 26. The position of the first dot 122A in the dot chart 120 and the design values regarding the optical system 26 are both known values.

The design values related to the optical system 26 may include values related to specifications given to the imaging device 10 as a product, and may include values related to specifications. Further, the design values may include values related to the characteristics of the optical system 26. The characteristics of the optical system 26 may include, for example, the arrangement position of the pupil splitting filter 16 and/or the characteristics of the aperture 24 (see FIG. 2). The characteristics of the opening 24 may include the center of gravity position G of the opening 24 (see FIGS. 7 and 8). The center of gravity position G of the opening 24 may be determined based on the position and/or shape of the opening 24, for example.

Further, the design values regarding the optical system 26 may include values regarding the combination of wavelength bands λ of each spectral filter 20 (see FIG. 2). The value regarding the combination of wavelength bands λ may be a value indicating the wavelength band of each spectral filter 20 itself, or may be a value indicating the relationship between the wavelength bands of each spectral filter 20.

For example, the correction value deriving unit 114 derives the direction and amount of positional deviation of the wavelength band image 140A with respect to the reference position 142 based on the spectral image 72A, and based on the derived direction and amount, the correction value deriving unit 114 A correction value 92A is derived for the image pixel corresponding to the wavelength band image 140A among the plurality of image pixels included in the image. The correction value 92A is a correction value for correcting the positional shift of the wavelength band image 140A. For example, the correction value 92A includes a value indicating a direction opposite to the direction of positional deviation of the wavelength band image 140A with respect to the reference position 142 (that is, a direction of the reference position 142 with respect to the wavelength band image 140A), and a value indicating the derived amount. including.

Similarly, the correction value deriving unit 114 derives the direction and amount of positional deviation of the wavelength band image 140B with respect to the reference position 142 based on the spectral image 72B, and based on the derived direction and amount, the correction value deriving unit 114 A correction value 92B for the image pixel corresponding to the wavelength band image 140B among the plurality of image pixels is derived. The correction value 92B is a correction value for correcting the positional shift of the wavelength band image 140B. Further, the correction value deriving unit 114 derives the direction and amount of positional deviation of the wavelength band image 140C with respect to the reference position 142 based on the spectral image 72C, and based on the derived direction and amount, A correction value 92C for the image pixel corresponding to the wavelength band image 140C among the image pixels of is derived. The correction value 92C is a correction value for correcting the positional shift of the wavelength band image 140C. Each correction value 92 differs depending on the position of the wavelength band image 140.

As an example, FIGS. 15 and 16 show an example in which the correction value 92 for the image pixel corresponding to the wavelength band image 140 is derived based on the wavelength band image 140 corresponding to the first dot 122A. The direction and amount of positional deviation are also derived for the wavelength band image 140 corresponding to the dots 122 other than the first dot 122A, and based on the derived direction and amount, the correction value for the image pixel corresponding to the wavelength band image 140 is calculated. 92 is derived.

For image pixels (hereinafter referred to as "non-corresponding image pixels") other than the image pixels corresponding to the wavelength band image 140 (hereinafter referred to as "corresponding image pixels"), for example, the correction value 92 of the corresponding image pixel The correction value 92 may be derived based on . Further, the correction value 92 for the non-corresponding image pixel may be derived based on the position of the non-corresponding image pixel with respect to the corresponding image pixel.

Furthermore, each correction value 92 may include a correction value for correcting image shift due to distortion and/or trapezoidal distortion. Further, the correction value 92 may be derived based on only one of the direction and amount of positional deviation. Further, the correction value 92 may be derived based on a value obtained through experiment. Moreover, although an example is given here in which each correction value 92 is derived by the processing device 100, each correction value 92 may be experimentally derived by a developer of the imaging device 10 or the like. The correction value 92 derived in the above manner is stored in the storage 62 of the imaging device 10.

Next, the operation of the imaging device 10 according to this embodiment will be explained. FIG. 17 shows an example of the flow of multispectral image generation processing executed by the imaging device 10.

In the multispectral image generation process shown in FIG. 10). After the process of step ST10 is executed, the multispectral image generation process moves to step ST12.

In step ST12, the interference cancellation processing unit 84 acquires the interference cancellation matrix A ⁺ stored in the storage 62 and the output value Y of each physical pixel 44 acquired in step ST10, and obtains the interference cancellation matrix A The pixel value X of each image pixel is output based on ⁺ and the output value Y of each physical pixel 44 (see FIG. 10). By executing the interference removal process in step ST12, the captured image 70 is divided into a spectrum image 72A corresponding to the luminance value X _λ1 of the first wavelength band polarized light and a spectrum corresponding to the luminance value X _λ2 of the second wavelength band polarized light. It is separated into an image 72B and a spectrum image 72C corresponding to the luminance value X _λ3 of the third wavelength band polarized light. After the process of step ST12 is executed, the multispectral image generation process moves to step ST14.

In step ST14, the correction processing unit 86 performs a correction process to correct image shift on each of the spectral images 72A to 72C (see FIG. 11). After the process of step ST14 is executed, the multispectral image generation process moves to step ST16.

In step ST16, the multispectral image generation unit 88 generates the multispectral image 74 by combining the spectral images 72A to 72C that have been corrected in step ST14 (see FIG. 12). After the process of step ST16 is executed, the multispectral image generation process moves to step ST18.

In step ST18, the processor 60 determines whether a condition for terminating the multispectral image generation process (ie, an terminating condition) is satisfied. An example of the termination condition is a condition that the user has given an instruction to the imaging device 10 to terminate the multispectral image generation process. In step ST18, if the end condition is not satisfied, the determination is negative and the multispectral image generation process moves to step ST10. In step ST18, if the termination condition is satisfied, the determination is affirmative and the multispectral image generation process is terminated. Note that the image processing method described as the function of the imaging device 10 described above is an example of the "image processing method" according to the technology of the present disclosure.

Next, the operation of the processing device 100 according to this embodiment will be explained. FIG. 18 shows an example of the flow of the correction value derivation process executed by the processing device 100.

In the correction value derivation process shown in FIG. 18, first, in step ST20, the spectral image acquisition unit 112 acquires each of the spectral images 72A to 72C obtained by the imaging device 10 (see FIG. 15). After the process of step ST20 is executed, the correction value derivation process moves to step ST22.

In step ST22, the correction value deriving unit 114 derives the direction and amount of positional shift of the wavelength band images 140A to 140C based on the respective spectral images 72A to 72C, and based on the derived direction and amount, the correction value deriving unit 114 A correction value 92 for image pixels included in 72C is derived (see FIG. 16). The derived correction value 92 is stored in the storage 62 of the imaging device 10. After the process of step ST22 is executed, the correction value derivation process moves to step ST24.

In step ST24, the processor 104 determines whether a condition for terminating the correction value derivation process (ie, an terminating condition) is satisfied. An example of the termination condition is a condition that correction values 92 for a plurality of image pixels included in each of the spectral images 72A to 72C have been derived. In step ST24, if the termination condition is not satisfied, the determination is negative and the correction value derivation process moves to step ST22. In step ST24, if the termination condition is satisfied, the determination is affirmative and the correction value derivation process is terminated.

Next, the effects of this embodiment will be explained.

In this embodiment, the optical system 26 includes a plurality of spectral filters 20 provided around the optical axis OA (see FIGS. 2 and 3). The processor 60 performs a correction process on each spectral image 72 to correct image deviation due to positional deviation of the optical image caused by being separated by the plurality of spectral filters 20 (see FIG. 11). Therefore, it is possible to obtain a multispectral image 74 with higher image quality than when the multispectral image 74 is generated based on the spectral image 72 in which image shift has occurred.

The correction process is performed on a plurality of spectral images 72 to generate a multispectral image 74. Therefore, before the multispectral image 74 is generated, image shifts occurring in the plurality of spectral images 72 can be corrected.

The correction process is performed, for example, based on design values regarding the optical system 26. Therefore, it is possible to correct image shift according to design values regarding the optical system 26.

The design values regarding the optical system 26 include, for example, the characteristics of the optical system 26. Therefore, by performing the correction process based on the characteristics of the optical system 26, it is possible to correct the image shift according to the characteristics of the optical system 26.

The characteristics of the optical system 26 include, for example, the arrangement position of the pupil splitting filter 16. Therefore, by performing the correction process based on the arrangement position of the pupil division filter 16, it is possible to correct image shift according to the arrangement position of the pupil division filter 16.

The characteristics of the optical system 26 include, for example, the characteristics of the aperture 24. Therefore, by performing the correction process based on the characteristics of the aperture 24, it is possible to correct the image shift according to the characteristics of the aperture 24.

The characteristics of the opening 24 include, for example, the center of gravity position G of the opening 24. Therefore, by performing the correction process based on the center of gravity position G of the aperture 24, it is possible to correct the image shift according to the center of gravity position G of the aperture 24.

The center of gravity position G of the opening 24 is, for example, a position determined based on the position and/or shape of the opening 24. Therefore, by performing the correction process based on the position and/or shape of the aperture 24, it is possible to correct image shift according to the position and/or shape of the aperture 24.

The design values regarding the optical system 26 include, for example, values regarding the combination of wavelength bands λ of each spectral filter 20. Therefore, by performing the correction process based on the value related to the combination of wavelength bands λ of each spectral filter 20, it is possible to correct image shift according to the combination of wavelength bands λ of each spectral filter 20.

The correction process is performed based on the correction value 92 for each wavelength band λ. Therefore, even if the image shift differs between the spectral images 72 corresponding to each wavelength band λ, the image shift can be corrected for each spectral image 72 corresponding to each wavelength band λ.

Each spectral image 72 includes a wavelength band image 140. The positional shift of the wavelength band image 140 differs for each spectrum image 72. Therefore, by correcting the positional deviation of the wavelength band image 140 for each spectral image 72 through the correction process, the image deviation of each spectral image 72 can be corrected.

The correction value 92 used in the correction process differs depending on the position of the wavelength band image 140. Therefore, by using the correction value 92 according to the position of the wavelength band image 140 in the correction process, it is possible to correct positional deviations that differ for each wavelength band image 140.

The correction value 92 is determined based on the direction and/or amount of positional shift of the wavelength band image 140 with respect to the reference position 142 within the spectrum image 72. Therefore, by using the correction value 92 corresponding to the direction and/or amount of positional deviation of the wavelength band image 140 in the correction process, the positional deviation based on the direction and/or amount of positional deviation of the wavelength band image 140 is corrected. be able to.

A dot chart 120 having dots 122 is used in the correction value derivation process for deriving the correction value 92. Therefore, in the correction value derivation process, for example, the position of the wavelength band image 140 can be specified using a known value such as the position of the dot 122 in the dot chart 120.

The dot chart 120 has dots 122 as a characteristic part. Therefore, the position of the wavelength band image 140 can be specified more easily than when a calibration member including a characteristic portion having a more complicated shape than the dots 122 is used.

In the correction value derivation process, the position corresponding to the dot 122 in the spectrum image 72 is set as the reference position 142. Therefore, in each spectrum image 72, the correction value 92 can be derived based on the positional shift of each wavelength band image 140 with respect to the reference position 142.

In the imaging device 10, interference removal processing is performed on the imaging data. Therefore, even if the output value Y of each physical pixel 44 corresponding to the imaging data is a value that includes interference (that is, crosstalk), by performing the interference removal process, the output value Y corresponds to each wavelength band λ. The values can be separated and extracted. Thereby, a spectrum image 72 corresponding to each wavelength band λ can be obtained from the captured image 70.

The plurality of spectral filters 20 have mutually different wavelength bands λ. Therefore, a spectrum image 72 corresponding to each wavelength band λ can be obtained.

The plurality of spectral filters 20 are arranged side by side around the optical axis OA. Therefore, the number of pupil divisions can be secured in a smaller space than, for example, when a plurality of spectral filters 20 are arranged concentrically with the optical axis OA.

Next, a modification of this embodiment will be described.

In the first modified example shown in FIG. 19 as an example, the correction value derivation unit 114 derives the correction value 92 in the following manner. For example, the correction value deriving unit 114 selects any one of the plurality of wavelength band images 140 and sets the selected wavelength band image 140 as the reference position 142. Here, a case where the wavelength band image 140C is set as the reference position 142 will be described as an example.

The correction value deriving unit 114 derives the correction value 92 in the same manner as in the above embodiment, except that the wavelength band image 140C is set as the reference position 142. For example, the correction value deriving unit 114 derives the direction and amount of positional deviation of the wavelength band image 140A with respect to the reference position 142 based on the spectral image 72A, and based on the derived direction and amount, the correction value deriving unit 114 derives the direction and amount of positional deviation of the wavelength band image 140A with respect to the reference position 142. A correction value 92A for the image pixel is derived.

Similarly, the correction value deriving unit 114 derives the direction and amount of positional deviation of the wavelength band image 140B with respect to the reference position 142 based on the spectrum image 72B, and corresponds to the wavelength band image 140A based on the derived direction and amount. A correction value 92B for the image pixel is derived. Note that the correction value deriving unit 114 sets the correction value 92C to 0 for the image pixel corresponding to the wavelength band image 140C set as the reference position 142.

In the first modification, the position of any one of the plurality of wavelength band images 140 is set as the reference position 142. Therefore, the correction value 92 can be derived based on the positional deviation of the remaining wavelength band images 140 with respect to the position of the wavelength band image 140 set as the reference position 142.

Note that each correction value 92 does not need to include a correction value for correcting image shift due to distortion and/or trapezoidal distortion. In this way, the amount of correction by the correction value 92 can be reduced compared to the case where the correction value 92 includes a correction value for correcting image shift due to distortion and/or trapezoidal distortion.

In the second modified example shown in FIG. 20 as an example, the correction value derivation unit 114 derives the correction value 92 in the following manner. For example, the correction value deriving unit 114 selects any one of the plurality of spectral images 72. Here, the case where the spectrum image 72C is selected will be described as an example. Further, the correction value deriving unit 114 extracts the wavelength band image 140C corresponding to the first dot 122A from the spectrum image 72C by performing image processing on the spectrum image 72C.

Then, the correction value deriving unit 114 sets the extracted wavelength band image 140C as the reference position 142. Here, an example will be described in which the wavelength band image 140C corresponding to the first dot 122A in the spectrum image 72C is set as the reference position 142.

Further, the correction value deriving unit 114 extracts the wavelength band image 140A corresponding to the first dot 122A from the spectrum image 72A by performing image processing on the spectrum image 72A. Similarly, the correction value deriving unit 114 extracts the wavelength band image 140B corresponding to the first dot 122A from the spectrum image 72B by performing image processing on the spectrum image 72B.

Further, the correction value deriving unit 114 identifies the position of the wavelength band image 140A within the spectrum image 72A by image processing. Similarly, the correction value deriving unit 114 identifies the position of the wavelength band image 140B within the spectrum image 72B by image processing. Further, the correction value deriving unit 114 specifies the position of the wavelength band image 140C within the spectrum image 72C by image processing, and sets the specified position of the wavelength band image 140C as the position of the reference position 142.

Then, similarly to the first modification, the correction value deriving unit 114 derives the correction value 92A based on the direction and amount of positional deviation of the wavelength band image 140A with respect to the reference position 142. Further, the correction value deriving unit 114 derives the correction value 92B based on the direction and amount of positional deviation of the wavelength band image 140B with respect to the reference position 142. Note that also in the second modification, the correction value 92C is set to 0.

In the second modification, the position of the wavelength band image 140 within the spectral image 72 and the position of the reference position 142 within the spectral image 72 are specified by image processing. Therefore, for example, the position of the wavelength band image 140 in the spectral image 72 and the reference position 142 in the spectral image 72 can be determined without using the position of the dot 122 in the dot chart 120 and the design values regarding the optical system 26. can be located.

As an example, a third modification shown in FIGS. 21 to 23 is an example in which a grid chart 150 is used instead of the dot chart 120 in the second modification. The grid chart 150 has a grid pattern. The lattice pattern may be a check pattern in which each region 152 surrounded by the lattice has a different color in adjacent regions 152, or a pattern composed only of lattice-like lines. In the third modification, a check pattern is used as an example of a check pattern. The adjacent areas 152 may have any combination of colors.

When the grid chart 150 is used, the intersection points 154 included in the grid pattern correspond to characteristic parts. The plurality of regions 152 are arranged linearly in the vertical and horizontal directions of the grid chart 150, and the plurality of intersections 154 are also linearly arranged in the vertical and horizontal directions of the grid chart 150. As an example, each region 152 has a square shape, and the plurality of intersection points 154 are arranged at equal intervals in the vertical and horizontal directions of the grid chart 150. The grid chart 150 is an example of a "subject" and a "calibration member" according to the technology of the present disclosure. The intersection 154 is an example of a “characteristic portion” and an “intersection” according to the technology of the present disclosure.

FIG. 21 shows an example of optical images 160A to 160C, optical images 162A to 162C, and optical images 164A to 164C that are formed on the light receiving surface 34A when the lattice chart 150 is imaged by the imaging device 10. There is. The optical images 160A to 160C are optical images corresponding to the first intersection point 154A located in the second row from the top and the second column from the right among the plurality of intersection points 154, and the optical images 162A to 162C are the optical images corresponding to the first intersection point 154A located in the second row from the top and the second column from the right among the plurality of intersection points 154. 154, which corresponds to the second intersection 154B located in the second row from the bottom and second column from the right, and the optical images 164A to 164C correspond to the third intersection 154B located at the center of the plurality of intersections 154. This is an optical image corresponding to the intersection 154C.

The optical image 160A, the optical image 162A, and the optical image 164A are optical images corresponding to the first wavelength band λ ₁ , and the optical image 160B, the optical image 162B, and the optical image 164B are the optical images corresponding to the second wavelength band λ ₂ . The optical image 160C, the optical image 162C, and the optical image 164C are optical images corresponding to the third wavelength band λ ₃ . The first reference position 166A is a position corresponding to the first intersection 154A in the light receiving surface 34A, the second reference position 166B is a position corresponding to the second intersection 154B in the light receiving surface 34A, and the third reference position 166C is a position corresponding to the third intersection 154C within the light receiving surface 34A.

The optical images 160A to 160C are shifted from the first reference position 166A. The direction and amount of positional deviation of the optical images 160A to 160C are different from each other. Similarly, the optical images 162A to 162C are shifted from the second reference position 166B, and the optical images 164A to 164C are misaligned from the third reference position 166C. The directions and amounts of positional deviations of optical images 162A to 162C are different from each other, and the directions and amounts of positional deviations of optical images 164A to 164C are different from each other.

As an example, as shown in FIG. 22, the spectral image acquisition unit 112 acquires each of the spectral images 72A to 72C obtained by capturing the grid chart 150. For example, the spectral image 72A includes a wavelength band image 170A corresponding to the optical image 160A, the spectral image 72B includes a wavelength band image 170B corresponding to the optical image 160B, and the spectral image 72C includes a wavelength band image 170B corresponding to the optical image 160B. includes a wavelength band image 170C corresponding to the optical image 160C. The direction and amount of positional shift of the wavelength band images 170A to 170C are different from each other. Hereinafter, unless it is necessary to separately explain the wavelength band images 170A to 170C, the wavelength band images 170A to 170C will be referred to as "wavelength band images 170."

As shown in FIG. 23 as an example, the correction value derivation unit 114 derives the correction value 92 in the following manner. For example, the correction value deriving unit 114 selects any one of the plurality of spectral images 72. Here, the case where the spectrum image 72C is selected will be described as an example. Further, the correction value deriving unit 114 extracts the wavelength band image 170C from the spectrum image 72C by performing image processing on the spectrum image 72C.

Then, the correction value deriving unit 114 sets the extracted wavelength band image 170C as the reference position 172. Here, an example will be described in which a wavelength band image 170C corresponding to the first intersection 154A in the spectrum image 72C is set as the reference position 172.

Further, the correction value deriving unit 114 extracts the wavelength band image 170A corresponding to the first intersection point 154A from the spectrum image 72A by performing image processing on the spectrum image 72A. Similarly, the correction value deriving unit 114 extracts the wavelength band image 170B corresponding to the first intersection point 154A from the spectrum image 72B by performing image processing on the spectrum image 72B.

Further, the correction value deriving unit 114 identifies the position of the wavelength band image 170A within the spectrum image 72A by image processing. Similarly, the correction value deriving unit 114 identifies the position of the wavelength band image 170B within the spectrum image 72B by image processing. Further, the correction value deriving unit 114 specifies the position of the wavelength band image 170C within the spectrum image 72C by image processing, and sets the specified position of the wavelength band image 170C as the position of the reference position 172.

Then, similarly to the second modification, the correction value deriving unit 114 derives the correction value 92A based on the direction and amount of positional deviation of the wavelength band image 170A with respect to the reference position 172. Further, the correction value deriving unit 114 derives the correction value 92B based on the direction and amount of positional deviation of the wavelength band image 170B with respect to the reference position 172. Note that also in the third modification, the correction value 92C is set to 0.

In the third modification, the grid chart 150 has a grid pattern as a characteristic part. Therefore, the position of the wavelength band image 170 can be specified more easily than when a calibration member having a characteristic portion having a more complicated shape than a checkered pattern is used.

As an example, in a fourth modification example shown in FIG. 24, the lattice chart 150 described in the third modification example is used. Similar to the third modification, the correction value deriving unit 114 derives the direction and amount of positional deviation of the wavelength band image 170 with respect to the reference position 172 based on the spectrum image 72, and calculates the wavelength based on the derived direction and amount. A correction value 92 for the image pixel corresponding to the band image 170 is derived.

However, in the fourth modification, the correction value deriving unit 114 derives each correction value 92 based on, for example, that the intersection points 154 of each row of the lattice chart 150 are lined up in a straight line in the horizontal direction of the lattice chart 150. For example, although the wavelength band images 170 corresponding to the intersection points 154 of each row of the grid chart 150 have different amounts of positional shift in the vertical direction of the spectrum image 72, the correction values 92 correspond to the intersection points 154 of each row of the grid chart 150. A correction value that aligns the positions of the wavelength band images 170 in the vertical direction of the spectrum image 72 is derived. As an example, FIG. 24 shows a mode in which each correction value 92 is derived based on the fact that the intersection points 154 in the second row from the top of the grid chart 150 are lined up in a straight line in the horizontal direction of the grid chart 150.

In the fourth modification, for example, each correction value 92 is derived based on the fact that the intersection points 154 of each row of the grid chart 150 are lined up in a straight line in the horizontal direction of the grid chart 150. Therefore, for example, as the correction value 92, a correction value can be obtained in which the positions of the wavelength band image 170 corresponding to the intersection points 154 of each row of the lattice chart 150 are aligned in the vertical direction of the spectrum image 72. As a result, for example, compared to a case where a correction value in which the position of the wavelength band image 170 corresponding to the intersection point 154 of each row of the grid chart 150 is shifted in the vertical direction of the spectral image 72 is derived as the correction value 92, the spectral image 72 can be suppressed in the vertical direction.

As an example, as in a fifth modified example shown in FIG. A value of 92 may be derived. For example, the wavelength band images 170 corresponding to the intersection points 154 of each column of the lattice chart 150 have different amounts of positional deviation in the lateral direction of the spectral image 72. A correction value may be derived that aligns the wavelength band image 170 corresponding to the intersection point 154 in the horizontal direction of the spectrum image 72. As an example, FIG. 25 shows a mode in which each correction value 92 is derived based on the fact that the intersection points 154 in the second column from the right of the grid chart 150 are lined up in a straight line in the vertical direction of the grid chart 150.

In the fifth modification, for example, a correction value can be obtained as the correction value 92 such that the positions of the wavelength band images 170 corresponding to the intersections 154 of each column of the lattice chart 150 are aligned in the horizontal direction of the spectrum image 72. As a result, for example, compared to the case where a correction value 92 is derived in which the position of the wavelength band image 170 corresponding to the intersection point 154 of each column of the lattice chart 150 is shifted in the horizontal direction of the spectrum image 72, Lateral distortion of the spectrum image 72 can be suppressed.

As an example, as in the sixth modified example shown in FIG. Each correction value 92 may be derived based on the fact that the intersection points 154 of each column are lined up in a straight line in the vertical direction of the grid chart 150. As an example, FIG. 26 shows that the intersection points 154 in the second row from the top of the grid chart 150 are lined up in a straight line in the horizontal direction of the grid chart 150, and that the intersection points 154 in the second column from the right of the grid chart 150 are arranged in a straight line in the horizontal direction of the grid chart 150. A mode in which each correction value 92 is derived based on the vertical alignment of 150 is shown.

In the sixth modification, for example, as the correction value 92, the positions of the wavelength band images 170 corresponding to the intersections 154 of each row of the lattice chart 150 are aligned in the vertical direction of the spectrum image 72, and the intersections 154 of each column of the lattice chart 150 It is possible to obtain a correction value that aligns the positions of the wavelength band images 170 corresponding to the spectrum image 72 in the horizontal direction. As a result, for example, the correction value 92 may be a correction value that shifts the position of the wavelength band image 170 corresponding to the intersection 154 of each row of the lattice chart 150 in the vertical direction of the spectral image 72, and/or a correction value for each column of the lattice chart 150. Compared to the case where a correction value in which the position of the wavelength band image 170 corresponding to the intersection point 154 is shifted in the horizontal direction of the spectrum image 72 is derived, vertical and/or horizontal distortion of the spectrum image 72 can be suppressed. can.

In the sixth modification, each correction value 92 may include a correction value for correcting image shift due to trapezoidal distortion. In this way, the trapezoidal distortion of the spectral image 72 can be suppressed compared to the case where the correction value 92 does not include a correction value for correcting image shift due to trapezoidal distortion.

In the fourth to sixth modifications, the correction value deriving unit 114 calculates each correction value 92 based on, for example, that the intersection points 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150. may be derived. In this way, each correction value 92 is derived regardless of whether the intersection points 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150. Distortion can be suppressed.

In the fourth to sixth modifications, the correction value deriving unit 114 calculates each correction value based on, for example, that the intersection points 154 of each column of the grid chart 150 are arranged at equal intervals in the vertical direction of the grid chart 150. 92 may be derived. In this way, each correction value 92 is derived regardless of whether the intersection points 154 of each column of the grid chart 150 are arranged at regular intervals in the vertical direction of the grid chart 150. distortion can be suppressed.

Further, in the fourth to sixth modifications, the correction value deriving unit 114 determines, for example, that the intersection points 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150; Each correction value 92 may be derived based on the fact that the intersection points 154 of each column are arranged at equal intervals in the vertical direction of the grid chart 150. In this way, each correction value 92 is derived regardless of whether the intersection points 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150, and/or Compared to the case where each correction value 92 is derived regardless of whether the intersection points 154 are arranged at regular intervals in the vertical direction of the grid chart 150, vertical and/or horizontal distortion of the spectral image 72 can be suppressed. can.

Further, based on the fact that the intersection points 154 of each row of the grid chart 150 are arranged at equal intervals in the horizontal direction of the grid chart 150, and that the intersection points 154 of each column of the grid chart 150 are arranged at equal intervals in the vertical direction of the grid chart 150, In the case where each correction value 92 is derived, each correction value 92 may include a correction value for correcting image shift due to trapezoidal distortion. In this way, the trapezoidal distortion of the spectral image 72 can be suppressed compared to the case where the correction value 92 does not include a correction value for correcting image shift due to trapezoidal distortion.

For example, the intersection points 154 of each row of the grid chart 150 may be arranged at equal intervals in the horizontal direction of the grid chart 150, and/or the intersection points 154 of each column of the grid chart 150 may be arranged at equal intervals in the vertical direction of the grid chart 150. When each correction value 92 is derived based on the alignment, the correction value 92 may be derived in the following manner.

That is, as the correction value 92, the interval between the wavelength band images 170 corresponding to the intersection 154 located in an area other than the central area of the lattice chart 150 is set as the interval between the wavelength band images 170 corresponding to the intersection 154 located in the central area of the lattice chart 150. A correction value may be derived to match the interval between 170 and 170. In this way, compared to the case where the correction value 92 is derived regardless of the interval between the wavelength band images 170 corresponding to the intersection 154 located in the central area of the lattice chart 150, the vertical direction of the spectral image 72 and/or Lateral distortion can be suppressed.

As an example, a seventh modification shown in FIG. 27 is an example in which inspection objects 180A to 180D are used as the subject 4 instead of the dot chart 120 in the second modification. As an example, each of the inspection objects 180A to 180D is a subject. Each inspection object 180A to 180D may be of any type. The inspection objects 180A to 180D are different types of objects. Hereinafter, unless it is necessary to explain the inspection objects 180A to 180D separately, each of the inspection objects 180A to 180D will be referred to as an "inspection object 180." Each test object 180 has a feature 182. Here, an example is given in which the number of the plurality of inspection objects 180 is four, but the number of the plurality of inspection objects 180 may be any number. The inspection object 180 is an example of a "subject" and "inspection object" according to the technology of the present disclosure.

FIG. 27 shows an example of optical images 190A to 190C formed on the light receiving surface 34A when the inspection objects 180A to 180D are imaged by the imaging device 10. As an example, the optical images 190A to 190C are optical images corresponding to a characteristic portion 182 (hereinafter referred to as “first characteristic portion 182A”) of the first inspection object 180A. The optical image 190A is an optical image corresponding to the first wavelength band λ ₁ , the optical image 190B is an optical image corresponding to the second wavelength band λ ₂ , and the optical image 190C is an optical image corresponding to the third wavelength band λ ₃ . The corresponding optical image. The reference position 192 is a position corresponding to the first characteristic portion 182A within the light receiving surface 34A. The optical images 190A to 190C are shifted from the reference position 192. The direction and amount of positional deviation of the optical images 190A to 190C are different from each other.

As an example, as shown in FIG. 27, the spectral image 72A includes a wavelength band image 200A corresponding to the optical image 190A, and the spectral image 72B includes a wavelength band image 200B corresponding to the optical image 190B. The spectral image 72C includes a wavelength band image 200C corresponding to the optical image 190C. The direction and amount of positional shift of the wavelength band images 200A to 200C are different from each other. Hereinafter, unless it is necessary to separately explain the wavelength band images 200A to 200C, the wavelength band images 200A to 200C will be referred to as "wavelength band images 200."

The correction value deriving unit 114 derives the correction value 92 in the following manner. For example, the correction value deriving unit 114 selects any one of the plurality of spectral images 72. Here, the case where the spectrum image 72C is selected will be described as an example. Further, the correction value deriving unit 114 extracts the wavelength band image 200C from the spectrum image 72C by performing image processing on the spectrum image 72C.

Then, the correction value deriving unit 114 sets the extracted wavelength band image 200C as the reference position 202. Here, an example will be described in which a wavelength band image 200C corresponding to the first characteristic portion 182A in the spectrum image 72C is set as the reference position 202.

Further, the correction value deriving unit 114 extracts the wavelength band image 200A corresponding to the first characteristic portion 182A from the spectrum image 72A by performing image processing on the spectrum image 72A. Similarly, the correction value deriving unit 114 extracts the wavelength band image 200B corresponding to the first characteristic portion 182A from the spectral image 72B by performing image processing on the spectral image 72B.

Further, the correction value deriving unit 114 identifies the position of the wavelength band image 200A within the spectrum image 72A by image processing. Similarly, the correction value deriving unit 114 identifies the position of the wavelength band image 200B within the spectrum image 72B by image processing. Further, the correction value deriving unit 114 specifies the position of the wavelength band image 200C within the spectrum image 72C by image processing, and sets the specified position of the wavelength band image 200C as the position of the reference position 202.

Then, similarly to the second modification, the correction value deriving unit 114 derives the correction value 92A based on the direction and amount of positional shift of the wavelength band image 200A with respect to the reference position 202. Further, the correction value deriving unit 114 derives the correction value 92B based on the direction and amount of positional deviation of the wavelength band image 200B with respect to the reference position 202. Note that in the seventh modification as well, the correction value 92C is set to 0.

In the seventh modification, a plurality of inspection objects 180 are used as the subject 4 when deriving the correction value 92. Therefore, the correction value 92 can be derived without using a calibration member (that is, a dedicated member for deriving the correction value 92).

Note that the correction value deriving process according to the seventh modification may be incorporated into the correction process in the imaging device 10. Then, the correction value 92 derived in the correction value derivation process according to the seventh modification may be used in the correction process.

As an example, in the eighth modification shown in FIG. 28, a plurality of inspection objects 180A to 180E of the same type are used as the subject 4 compared to the seventh modification. Hereinafter, unless there is a need to explain the inspection objects 180A to 180E separately, each of the inspection objects 180A to 180E will be referred to as an "inspection object 180." Each inspection object 180 has a first feature 184 and a second feature 186.

For example, the correction value deriving unit 114 sets the wavelength band image 200 corresponding to the first characteristic portion 184 as the reference position 202, and calculates the direction and amount of positional deviation of the wavelength band image 200 with respect to the reference position 202 based on the spectrum image 72. Based on the derived direction and amount, a correction value 92 for the image pixel corresponding to the wavelength band image 200 is derived.

However, in the eighth modification, the correction value deriving unit 114 derives each correction value 92 based on, for example, that the position of the first characteristic portion 184 with respect to the second characteristic portion 186 in each inspection object 180 is the same. do. For example, the wavelength band image 200 corresponding to the first characteristic portion 184 of each inspection object 180 may be different in direction and/or amount from the wavelength band image 200 corresponding to the second characteristic portion 186 of each inspection object 180. Although there is a positional shift, a correction value is derived as the correction value 92 in which the direction and/or amount of the positional shift of the wavelength band images 200 corresponding to the first characteristic portions 184 of each inspection object 180 are the same.

In the eighth modification, each correction value 92 is derived, for example, based on the fact that the position of the first characteristic portion 184 with respect to the second characteristic portion 186 in each inspection object 180 is the same. Therefore, for example, it is possible to obtain, as the correction value 92, a correction value in which the directions and/or amounts of positional deviations of the wavelength band images 200 corresponding to the first characteristic portions 184 of each inspection object 180 are aligned. As a result, for example, compared to the case where the correction value 92 is derived regardless of the fact that the position of the first characteristic portion 184 with respect to the second characteristic portion 186 is the same in each inspection object 180, the vertical direction of the spectral image 72 is Directional and/or lateral distortion can be suppressed.

Note that, for example, when each correction value 92 is derived based on the fact that the position of the first characteristic part 184 with respect to the second characteristic part 186 is the same in each inspection object 180, the correction value 92 is calculated according to the following procedure. It may be derived as follows.

That is, as the correction value 92, a wavelength band image corresponding to the first characteristic portion 184 and the second characteristic portion 186 of the inspection objects 180A to 180D located in a region other than the central region of the imaging target region 210 by the imaging device 10 is used. A correction value is derived that adjusts the interval between the wavelength band images 200 to the interval between the wavelength band images 200 corresponding to the first characteristic portion 184 and the second characteristic portion 186 of the inspection object 180E located in the central region of the imaging target region 210. may be done. In this way, compared to the case where the correction value 92 is derived regardless of the interval between the wavelength band images 200 corresponding to the first characteristic portion 184 and the second characteristic portion 186 of the inspection object 180E located in the central region. Thus, vertical and/or horizontal distortion of the spectral image 72 can be suppressed.

Note that the correction value deriving process according to the eighth modification may be incorporated into the correction process in the imaging device 10. Then, the correction value 92 derived in the correction value derivation process according to the eighth modification may be used in the correction process.

As an example, in the ninth modification example shown in FIG. 29, two subjects 220A and 220B are used as the subject 4, compared to the eighth modification example. As an example, the imaging target area 210 by the imaging device 10 includes a subject area 210A where the subject 220A is placed, a subject area 210B where the subject 220B is placed, and an empty area where the subject 220A and the subject 220B are not placed. 210C and a free area 210D.

The correction value deriving unit 114 divides each spectrum image 72 obtained by imaging the imaging target region 210 by the imaging device 10 into regions 212A to 212D. The area 212A corresponds to the subject area 210A, the area 212B corresponds to the subject area 210B, the area 212C corresponds to the free area 210C, and the area 212D corresponds to the free area 210D. There is. In each spectral image 72, a region 212A includes a wavelength band image 230A corresponding to a subject 220A, and a region 212B includes a wavelength band image 230B corresponding to a subject 220B.

The correction value deriving unit 114 may divide each spectral image 72 into regions 212A to 212D based on an instruction received by the imaging device 10 from a user or the like, and performs image processing on each spectral image 72. By doing so, each spectral image 72 may be divided into a plurality of regions 212A to 212D based on the presence or absence of a wavelength band image. Here, each spectral image 72 is divided into four areas 212A to 212D corresponding to two subjects 220A and 220B, but the number of subjects included in the imaging target area 210 and each spectral image 72 are The number of regions to be divided may be any number. Then, the correction value deriving unit 114 derives the correction value 92 for the area 212A and the area 212B, but does not derive the correction value 92 for the area 212C and the area 212D.

In the ninth modification, the correction value 92 is derived for the region 212A and the region 212B, and the correction value 92 is not derived for the region 212C and the region 212D. Therefore, the load on the processor 104 of the processing device 100 can be reduced compared to the case where the correction value 92 is also derived for the region 212C and the region 212D.

Furthermore, when the correction value 92 derived according to the ninth modification is used, a correction process is performed on a part of the spectral image 72 in the imaging device 10. Therefore, the load on the processor 60 of the imaging device 10 can be reduced compared to the case where the correction process is performed on the entire region of the spectral image 72.

Further, in the embodiment described above, the correction process on the spectral image 72 is executed in the imaging device 10, but the spectral image 72 is input from the image capturing device 10 to an external device, and the correction process on the spectral image 72 is executed in the external device. Good too. In this case, the external device is an example of an "image processing device" related to the technology of the present disclosure.

Further, in the above embodiment, the processor 60 is illustrated in the imaging device 10, but instead of the processor 60, or together with the processor 60, at least one other CPU, at least one GPU, and/or at least one TPU may also be used.

Further, in the above embodiment, the processor 104 is illustrated as an example of the processing device 100, but instead of or together with the processor 104, at least one other CPU, at least one GPU, and/or at least one TPU may also be used.

Furthermore, in the above embodiment, the imaging device 10 has been described using an example in which the multispectral image generation program 80 is stored in the storage 62, but the technology of the present disclosure is not limited to this. For example, the multispectral image generation program 80 may be stored in a portable non-transitory computer-readable storage medium (hereinafter simply referred to as "non-transitory storage medium") such as an SSD or a USB memory. A multispectral image generation program 80 stored on a non-transitory storage medium may be installed on the computer 56 of the imaging device 10.

Further, the multispectral image generation program 80 is stored in a storage device such as another computer or a server device connected to the imaging device 10 via a network, and the multispectral image generation program 80 is generated in response to a request from the imaging device 10. may be downloaded and installed on the computer 56 of the imaging device 10.

Further, it is not necessary to store the entire multispectral image generation program 80 in a storage device such as another computer or server device connected to the imaging device 10, or in the storage 62, but a part of the multispectral image generation program 80. may be stored.

Furthermore, in the above embodiment, the processing device 100 has been described using an example in which the correction value derivation program 110 is stored in the storage 106, but the technology of the present disclosure is not limited to this. For example, the correction value derivation program 110 may be stored in a non-temporary storage medium. The correction value derivation program 110 stored in a non-transitory storage medium may be installed on the computer 102 of the processing device 100.

Further, the correction value derivation program 110 is stored in a storage device such as another computer or server device connected to the processing device 100 via a network, and the correction value derivation program 110 is downloaded in response to a request from the processing device 100. may be installed on the computer 102 of the processing device 100.

Further, it is not necessary to store all of the correction value derivation program 110 in a storage device such as another computer or server device connected to the processing device 100, or in the storage 106, but only a part of the correction value derivation program 110 is stored. You can leave it.

Further, although the imaging device 10 has a built-in computer 56, the technology of the present disclosure is not limited to this, and for example, the computer 56 may be provided outside the imaging device 10.

Further, although the processing device 100 has a built-in computer 102, the technology of the present disclosure is not limited to this, and for example, the computer 102 may be provided outside the processing device 100.

Further, in the above embodiment, the computer 56 including the processor 60, the storage 62, and the RAM 64 is illustrated as an example of the imaging device 10, but the technology of the present disclosure is not limited to this, and instead of the computer 56, an ASIC, A device including an FPGA and/or a PLD may also be applied. Further, instead of the computer 56, a combination of hardware configuration and software configuration may be used.

Further, in the above embodiment, the computer 102 including the processor 104, the storage 106, and the RAM 108 is illustrated as the processing device 100, but the technology of the present disclosure is not limited to this, and instead of the computer 102, an ASIC, A device including an FPGA and/or a PLD may also be applied. Further, in place of the computer 102, a combination of hardware configuration and software configuration may be used.

Additionally, the following various processors can be used as hardware resources for executing the various processes described in the above embodiments. Examples of the processor include a CPU, which is a general-purpose processor that functions as a hardware resource that executes various processes by executing software, that is, a program. Examples of the processor include a dedicated electronic circuit such as an FPGA, a PLD, or an ASIC, which is a processor having a circuit configuration specifically designed to execute a specific process. Each processor has a built-in memory or is connected to it, and each processor uses the memory to perform various processes.

Hardware resources that execute various processes may be configured with one of these various processors, or a combination of two or more processors of the same type or different types (for example, a combination of multiple FPGAs, or a CPU and FPGA). Furthermore, the hardware resource that executes various processes may be one processor.

As an example of a configuration using one processor, firstly, one processor is configured by a combination of one or more CPUs and software, and this processor functions as a hardware resource that executes various processes. Second, there is a form of using a processor, as typified by an SoC, in which a single IC chip realizes the functions of an entire system including a plurality of hardware resources that execute various processes. In this way, various types of processing are realized using one or more of the various types of processors described above as hardware resources.

Furthermore, as the hardware structure of these various processors, more specifically, an electronic circuit that is a combination of circuit elements such as semiconductor elements can be used. Further, the above line of sight detection processing is just an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged without departing from the main idea.

The descriptions and illustrations described above are detailed explanations of the parts related to the technology of the present disclosure, and are merely examples of the technology of the present disclosure. For example, the above description regarding the configuration, function, operation, and effect is an example of the configuration, function, operation, and effect of the part related to the technology of the present disclosure. Therefore, unnecessary parts may be deleted, new elements may be added, or replacements may be made to the written and illustrated contents described above without departing from the gist of the technology of the present disclosure. Needless to say. In addition, in order to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the descriptions and illustrations shown above do not include parts that require particular explanation in order to enable implementation of the technology of the present disclosure. Explanations regarding common technical knowledge, etc. that do not apply are omitted.

In this specification, "A and/or B" has the same meaning as "at least one of A and B." That is, "A and/or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, even when three or more items are expressed by connecting them with "and/or", the same concept as "A and/or B" is applied.

All documents, patent applications, and technical standards mentioned herein are incorporated herein by reference to the same extent as if each individual document, patent application, and technical standard was specifically and individually indicated to be incorporated by reference. Incorporated by reference into this book.

Regarding the above embodiments, the following additional notes are further disclosed.

(Additional note 1)
A member for deriving correction values used in calibration processing,
The calibration process is a calibration process for an image output from an imaging device including an optical system,
The optical system is provided around an optical axis and includes a plurality of filters having mutually different wavelength bands,
The correction value is a correction value for correcting an image shift due to a positional shift of an optical image that occurs when the image is separated by the plurality of filters,
The image includes a wavelength band image corresponding to each of the wavelength bands,
The member includes a characteristic portion corresponding to the reference position when the correction value is derived based on the direction and/or amount of positional shift of the wavelength band image with respect to the reference position in the image.
(Additional note 2)
A device for deriving correction values used in calibration processing,
The calibration process is a calibration process for an image output from an imaging device including an optical system,
The optical system is provided around an optical axis and includes a plurality of filters having mutually different wavelength bands,
The correction value is a correction value for correcting an image shift due to a positional shift of an optical image that occurs when the image is separated by the plurality of filters,
The image includes a wavelength band image corresponding to each of the wavelength bands,
The device includes a processor;
The apparatus includes: the processor deriving the correction value based on the direction and/or amount of displacement of the wavelength band image with respect to a reference position within the image.
(Additional note 3)
A method for deriving a correction value used in a calibration process, the method comprising:
The calibration process is a calibration process for an image output from an imaging device including an optical system,
The optical system is provided around an optical axis and includes a plurality of filters having mutually different wavelength bands,
The correction value is a correction value for correcting an image shift due to a positional shift of an optical image that occurs when the image is separated by the plurality of filters,
The image includes a wavelength band image corresponding to each of the wavelength bands,
The method comprises deriving the correction value based on the direction and/or amount of displacement of the wavelength band image with respect to a reference position within the image.
(Additional note 4)
A program for causing a computer to execute processing for deriving correction values used in calibration processing,
The calibration process is a calibration process for an image output from an imaging device including an optical system,
The optical system is provided around an optical axis and includes a plurality of filters having mutually different wavelength bands,
The correction value is a correction value for correcting an image shift due to a positional shift of an optical image that occurs when the image is separated by the plurality of filters,
The image includes a wavelength band image corresponding to each of the wavelength bands,
The processing includes deriving the correction value based on the direction and/or amount of positional shift of the wavelength band image with respect to a reference position within the image.

Claims

An image processing device applied to an image output from an imaging device including an optical system,
The optical system includes a plurality of filters provided around an optical axis,
The image processing device includes a processor,
The processor performs processing on the image to correct an image shift due to a positional shift of an optical image caused by being separated by the plurality of filters.
The optical system has a plurality of apertures,
Each of the openings is provided with each of the filters,
The image shift includes at least an image shift based on the characteristics of each of the openings,
The image processing device according to claim 1, wherein the processing is performed based on the characteristics.
The image processing device according to claim 2, wherein the feature includes a center of gravity position of the opening.
The image processing device according to claim 3, wherein the center of gravity position is a position determined based on the position and/or shape of the opening.
The image processing device according to any one of claims 1 to 4, wherein the positional deviation of the optical image includes at least a positional deviation of the optical image caused by a characteristic of the optical system.
The image processing device according to any one of claims 1 to 5, wherein the processing is performed on a partial area of the image.
The image processing device according to any one of claims 1 to 6, wherein the image is a spectral image for generating a multispectral image.
The image processing device according to claim 7, wherein the image is an image generated by performing interference removal processing on imaged data obtained by imaging by the imaging device.
The image processing device according to any one of claims 1 to 8, wherein the plurality of filters have wavelength bands different from each other.
The image processing device according to any one of claims 1 to 9, wherein the plurality of filters are arranged side by side around the optical axis.
The image processing device according to claim 9, wherein the processing is performed based on a combination of the wavelength bands.
The image processing device according to any one of claims 1 to 11, wherein the processing is performed based on design values regarding the optical system.
The image processing device according to claim 9, wherein the processing is performed based on a correction value for each wavelength band.
The image processing device according to any one of claims 9, 11, and 13, wherein the image includes a wavelength band image corresponding to each of the wavelength bands.
The image includes a wavelength band image corresponding to each of the wavelength bands,
The image processing device according to claim 13, wherein the correction value differs depending on the position of the wavelength band image.
The image includes a wavelength band image corresponding to each of the wavelength bands,
The image processing device according to claim 13, wherein the correction value is determined based on the direction and/or amount of positional shift of the wavelength band image with respect to a reference position within the image.
The image processing device according to claim 14, wherein the wavelength band image is an image showing a characteristic part of a subject.
The wavelength band image is an image showing a characteristic part of the subject,
The image processing device according to claim 16, wherein the reference position is a position corresponding to the characteristic portion.
The image processing device according to claim 16, wherein the reference position is a position of any one of the plurality of wavelength band images.
The object has a point,
The image processing device according to claim 17, wherein the characteristic portion is the point.
The subject has a checkered pattern,
The image processing device according to claim 17, wherein the characteristic portion is an intersection included in the checkered pattern.
The image processing device according to claim 17, wherein the subject is a calibration member.
The subject has a plurality of the characteristic parts,
The image processing device according to claim 17, wherein the plurality of characteristic portions are arranged in a straight line.
The subject has a plurality of the characteristic parts,
The image processing device according to claim 17, wherein the plurality of characteristic portions are arranged at equal intervals.
The image processing device according to claim 17, wherein the subject includes an inspection object.
The image processing device according to claim 25, wherein the subject includes a plurality of the inspection objects.
The optical system has a polarizing filter provided corresponding to each of the filters,
The plurality of polarizing filters have mutually different polarization axes,
The imaging device includes an image sensor having a plurality of pixel blocks,
The image processing device according to any one of claims 1 to 26, wherein each of the pixel blocks is provided with a plurality of types of polarizers having mutually different polarization axes.
An image processing method applied to an image output from an imaging device including an optical system, the method comprising:
The optical system includes a plurality of filters provided around an optical axis,
The image processing method includes performing a process on the image to correct an image shift due to a position shift of an optical image caused by being separated by the plurality of filters.
A program for causing a computer to perform image processing on an image output from an imaging device equipped with an optical system, the program comprising:
The optical system includes a plurality of filters provided around an optical axis,
The image processing includes processing for correcting an image shift due to a positional shift of an optical image caused by the image being separated by the plurality of filters.