WO2023047794A1 - 情報処理方法、情報処理装置、情報処理プログラム、及び情報処理システム - Google Patents

情報処理方法、情報処理装置、情報処理プログラム、及び情報処理システム Download PDF

Info

Publication number
WO2023047794A1
WO2023047794A1 PCT/JP2022/029055 JP2022029055W WO2023047794A1 WO 2023047794 A1 WO2023047794 A1 WO 2023047794A1 JP 2022029055 W JP2022029055 W JP 2022029055W WO 2023047794 A1 WO2023047794 A1 WO 2023047794A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
parameter
information processing
interference
processor
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/029055
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
友也 平川
和佳 岡田
慶延 岸根
高志 椚瀬
康一 田中
達郎 岩▲崎▼
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Fujifilm Corp
Original Assignee
Fujifilm Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Fujifilm Corp filed Critical Fujifilm Corp
Priority to CN202280061734.3A priority Critical patent/CN117957849A/zh
Priority to JP2023549396A priority patent/JP7824307B2/ja
Publication of WO2023047794A1 publication Critical patent/WO2023047794A1/ja
Priority to US18/595,461 priority patent/US20240212107A1/en
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B5/00—Optical elements other than lenses
    • G02B5/30—Polarising elements
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B11/00—Filters or other obturators specially adapted for photographic purposes
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B15/00—Special procedures for taking photographs; Apparatus therefor
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V10/00—Arrangements for image or video recognition or understanding
    • G06V10/10—Image acquisition
    • G06V10/12—Details of acquisition arrangements; Constructional details thereof
    • G06V10/14—Optical characteristics of the device performing the acquisition or on the illumination arrangements
    • G06V10/141—Control of illumination
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10032—Satellite or aerial image; Remote sensing
    • G06T2207/10036—Multispectral image; Hyperspectral image

Definitions

  • the present invention relates to an information processing method, an information processing apparatus, an information processing program, and an information processing system for processing multispectral images.
  • Patent Documents 1 and 2 describe suppressing the influence of ghosts.
  • An embodiment according to the technology of the present disclosure provides an information processing method, an information processing device, an information processing program, and an information processing system for acquiring a multispectral image with good image quality.
  • An information processing method is an information processing method for acquiring an interference cancellation parameter of a split-pupil multispectral camera, which is executed by an information processing apparatus including a processor, wherein the split-pupil multispectral The camera includes a plurality of aperture regions arranged at or near the pupil, a plurality of optical filters arranged in the plurality of aperture regions and transmitting a plurality of lights having at least some different wavelength bands, a plurality of light and an image sensor that outputs a plurality of image signals corresponding to, the processor obtains a first parameter acquisition step of acquiring a first interference cancellation parameter used for interference cancellation of the plurality of image signals; an information acquisition step of acquiring a first image signal, which is a plurality of image signals for a plurality of lights, as information indicating the wavelength characteristics of the subject; a second parameter acquisition step of acquiring a second interference cancellation parameter used for interference cancellation of the second image signal that is the image signal, and the processor acquires the information acquired by the first imaging in the second
  • the first shooting is the preliminary shooting
  • the second shooting is the main shooting
  • the focus position of the first shooting is the same as the focus position of the second shooting. is.
  • the processor in the information acquisition step, calculates the wavelength intensity of the subject in a state in which light does not pass through some of the plurality of aperture regions. get.
  • the processor shields some of the plurality of aperture regions and blocks the remaining aperture regions.
  • a first image signal output by photographing a subject whose wavelength characteristics are known is acquired as information.
  • the processor in the information acquiring step, physically blocks the partial opening region by arranging a light shielding member that does not transmit light to the partial opening region.
  • a first image signal as information is acquired in a state of being shielded from the target.
  • the split-pupil multispectral camera comprises a plurality of first polarizing members arranged in a plurality of aperture regions and transmitting light with different polarization angles
  • the processor disposes a second polarizing member having a polarization angle orthogonal to the first polarizing member arranged in the partial aperture region in the partial aperture region. is optically shielded, a first image signal is acquired as information.
  • the processor selects an optical filter that transmits one of the plurality of lights and does not transmit the remaining light than the plurality of optical filters.
  • a first image signal is acquired by photographing a subject with known wavelength characteristics while the apparatus is placed on the side of the subject.
  • the processor in a state in which the plurality of optical filters are not arranged in the plurality of aperture regions, By arranging an optical filter that transmits one of the light beams but does not transmit the remaining light beams closer to the subject than the plurality of optical filters, the information of the subject whose wavelength characteristics are unknown is converted to the first information. , and in a state in which a plurality of optical filters are arranged in a plurality of aperture regions, an optical filter that transmits one of the plurality of lights and does not transmit the remaining light is positioned closer to the subject than the previous number of optical filters.
  • a parameter for correcting the second information is determined based on the first information. Obtained as the second interference cancellation parameter.
  • a ninth aspect of the information processing method wherein the split-pupil multispectral camera is arranged in a plurality of aperture regions and transmits light having different polarization angles. and a plurality of second polarizing members arranged in the image sensor and transmitting light having polarization angles corresponding to the polarization angles of the plurality of first polarizing members, wherein the processor performs an information acquisition step a plurality of image signals corresponding to the polarization angles of the plurality of first polarizing members are acquired as information.
  • An information processing apparatus uses a plurality of aperture regions arranged at or near the pupil, and a plurality of light beams arranged in the plurality of aperture regions and having at least part of different wavelength bands.
  • An information processing device for acquiring interference elimination parameters of a split-pupil multispectral camera comprising a plurality of optical filters that transmit light and an image sensor that outputs a plurality of image signals corresponding to a plurality of lights, the information processing device comprising a processor , the processor performs a first parameter acquisition process of acquiring a first interference elimination parameter used for interference elimination of a plurality of image signals, and a first photographing process to obtain a plurality of image signals for a plurality of lights and to determine the wavelength characteristics of the subject.
  • Information acquisition processing acquired as information to be shown and information acquired by the first imaging are referred to acquire a second interference elimination parameter used for interference elimination of the second image signal, which is a plurality of image signals of the second imaging. and a second parameter acquisition process, in which the processor calculates the difference between the information acquired by the first imaging and the second image signal interference-removed with the second interference-removal parameter as the first A second interference elimination parameter is acquired that is smaller than the difference between the information acquired by photographing and the second image signal that has been interference eliminated with the first interference elimination parameter.
  • An information processing program provides an information processing apparatus comprising a processor, a plurality of aperture regions arranged at or near a pupil, and at least a part of the aperture regions arranged in the plurality of aperture regions.
  • Information for obtaining interference elimination parameters of a split-pupil multispectral camera comprising a plurality of optical filters that transmit a plurality of lights with different wavelength bands, and an image sensor that outputs a plurality of image signals corresponding to the plurality of lights
  • An information processing program causing a processing method to be executed, wherein the processor performs a first parameter acquisition step of acquiring a first interference elimination parameter used for interference elimination of a plurality of image signals, An information acquisition step of acquiring a plurality of image signals as information indicating wavelength characteristics of a subject; and a second parameter acquisition step of acquiring a second interference elimination parameter used in the second parameter acquisition step, and in the second parameter acquisition step, the processor causes the information acquired by the first imaging and the interference eliminated by the second interference elimination parameter A second interference elimination parameter is
  • An information processing system provides a plurality of aperture regions arranged at or near the pupil, and a plurality of lights having at least a part of different wavelength bands arranged in the plurality of aperture regions.
  • a split-pupil multispectral camera including a plurality of transmitting optical filters and an image sensor outputting a plurality of image signals corresponding to a plurality of lights; and an information processing apparatus according to the tenth aspect.
  • the processor performs interference elimination of the plurality of image signals using the interference elimination parameter, and outputs the plurality of image signals in a state where the interference has been eliminated to the output device. output.
  • FIG. 1 is a diagram showing a schematic configuration of an imaging system 10 according to the first embodiment.
  • FIG. 2 is a perspective view showing the configuration of the lens device.
  • FIG. 3 is a cross-sectional view showing the configuration of the lens device.
  • FIG. 4 is an external view showing an example of a filter unit.
  • FIG. 5 is an external view showing another example of the filter unit.
  • FIG. 6 is a diagram showing how the filter sets are arranged on the frame.
  • FIG. 7 is a diagram showing the polarization direction of a polarizing filter.
  • FIG. 8 is a diagram showing the configuration of an image sensor.
  • FIG. 1 is a diagram showing a schematic configuration of an imaging system 10 according to the first embodiment.
  • FIG. 2 is a perspective view showing the configuration of the lens device.
  • FIG. 3 is a cross-sectional view showing the configuration of the lens device.
  • FIG. 4 is an external view showing an example of a filter unit.
  • FIG. 5 is an external view showing another example of the
  • FIG. 9 is a diagram showing a functional configuration of a processor; 10A and 10B are diagrams showing how the light shielding member shields a part of the openings and shields the other openings.
  • FIG. 11 is a diagram showing a state of preliminary photography for obtaining interference cancellation parameters.
  • 12A and 12B are diagrams showing the state of actual photographing in the first mode.
  • FIG. 13 is a diagram showing how the wavelength band of illumination light is changed to optically shield.
  • FIG. 14 is a diagram showing a state in which the diffusion plate 99A is arranged in front of the light source 99.
  • FIG. 15 is a diagram showing the state of preliminary photography in the third mode.
  • FIG. 16 is a diagram showing how the actual photographing is performed in the third mode.
  • FIG. 17 is a diagram showing a state in which the color filters are arranged closer to the subject than the filter units.
  • FIG. 18 is a diagram showing a state in which the color filters are attached to another frame and inserted closer to the subject than the filter unit.
  • FIG. 19 is a diagram showing a state in which the color filter is attached closer to the subject than the first lens.
  • FIG. 20 is a diagram showing a state in which the color filters are arranged closer to the subject than the frame.
  • Imaging devices that capture multispectral images are known to use polarizing elements.
  • wavelength information mixed by each polarization pixel (polarization direction is, for example, 0deg, 45deg, 90deg, 135deg) is acquired, and interference is eliminated based on the mixing ratio (computation using an inverse matrix). to generate an image for each wavelength.
  • interference cancellation is performed using a theoretically calculated interference cancellation matrix, differences between the development environment and the actual environment (environment where images are actually taken and image processing is performed using an imaging device) (for example, due to refraction changes in the degree of polarization, and differences in ghost and/or flare), it is not possible to correctly generate multispectral images.
  • interference is removed for each pixel as an example, but the present invention can be applied to a wide range of situations, such as removing only pixels or areas where the influence of interference is large.
  • FIG. 1 is a diagram showing a schematic configuration of an imaging system 10 (imaging system, imaging device, information processing system, information processing device) according to the first embodiment.
  • the imaging system 10 includes a lens device 100 (pupil-splitting multispectral camera), an imaging device body 200 (pupil-splitting multispectral camera), a display device 300 (liquid crystal display, etc.), a storage device 310 (magneto-optical recording device, semiconductor memory, etc.). ), and an operation device 320 (keyboard, mouse, button, switch, dial, etc.), which can capture a light source 99 (object) to obtain a multispectral image.
  • the configuration of each of these units will be described in detail below.
  • the imaging system 10 is capable of calculating interference cancellation parameters based on the acquired multispectral image and performing interference cancellation using the calculated interference cancellation parameters.
  • FIG. 2 is a perspective view showing the configuration of the lens device 100
  • FIG. 3 is a sectional view showing the configuration of the lens device 100.
  • the lens device 100 has an optical system including a first lens 110 and a second lens 120 arranged in a lens barrel 102. These lenses rotate a first lever 104 and a second lever 106, respectively. By moving it forward and backward in the direction of the optical axis L, the focal length and/or the image magnification are adjusted.
  • the first lens 110 and the second lens 120 may be a lens group composed of a plurality of lenses.
  • a slit 108 is formed in the lens barrel 102 at or near the pupil of the lens apparatus 100.
  • a filter unit 134 is inserted into the slit 108 so that the optical axis of the imaging optical system (first lens 110 , and the optical axis L of the second lens 120).
  • FIG. 4 is an external view showing an example of the filter unit 134.
  • the filter unit 134 has a frame 135, and the frame 135 has four opening areas (open areas 135A to 135D; a plurality of opening areas).
  • the center of gravity of these open areas 135A-135D is the center of gravity 135G.
  • a filter set 137 (filter set 137A-137D; multiple optical filters, multiple firsts) may be positioned in the aperture regions 135A-135D. The configuration of the filter set 137 will be described later.
  • FIG. 5 is an external view showing another example of the frame (filter unit). As shown in the figure, part of the openings may be blocked according to the number of images to be acquired, or frames having different numbers of opening regions may be used. For example, when acquiring images corresponding to three aperture regions, one of the aperture regions (here, aperture region 135D) may be shielded with a light shielding member 135E as shown in part (a) of FIG. Alternatively, a frame 133 having three opening regions 133A to 133C may be used, as shown in part (b) of FIG.
  • FIG. 6 is a diagram showing how the filter sets are arranged on the frame.
  • the filter set 137 includes color filters and polarizing filters and is arranged in the aperture regions 135A to 135D, respectively.
  • filter set 137A includes color filter 138A and polarizing filter 139A
  • filter set 137B includes color filter 138B and polarizing filter 139B
  • filter set 137C includes color filter 138C and polarizing filter 139C
  • filter set 137D includes a color filter 138D and a polarizing filter 139D.
  • the color filters 138A to 138D are preferably a plurality of optical filters that transmit a plurality of lights with different wavelength bands at least partially.
  • the polarizing filters 139A to 139D are preferably polarizing filters that transmit light with different polarization angles.
  • FIG. 7 is a diagram showing the polarizing directions of the polarizing filters 139A to 139D.
  • the polarization directions of the polarizing filters 139A to 139D are 0deg, 45deg, 90deg and 135deg, respectively.
  • the first polarizing member may be a filter that polarizes with a polarizing film, or a filter that polarizes with a wire grid or a plurality of slits.
  • the image sensor 138 is a CMOS (Complementary Metal-Oxide Semiconductor) type image sensor (imaging device), and outputs a plurality of image signals corresponding to a plurality of lights transmitted by the color filters 138A to 138D.
  • the image sensor 138 is a monochrome imaging device having a pixel array layer 211, a polarizing filter element array layer 213, and a microlens array layer 215.
  • the layers are arranged in the order of pixel array layer 211, polarizing filter element array layer 213, and microlens array layer 215 from the image plane side to the object side.
  • the image sensor 138 is not limited to the CMOS type, and may be an XY address type or CCD (Charge Coupled Device) type image sensor.
  • the pixel array layer 211 is configured by two-dimensionally arranging a large number of photodiodes 211A (a plurality of pixel groups). One photodiode 211A constitutes one pixel. Each photodiode 211A is regularly arranged along the horizontal direction (x direction) and the vertical direction (y direction).
  • the polarizing filter element array layer 213 is configured by two-dimensionally arranging four types of polarizing filter elements 214A, 214B, 214C, and 214D having different polarization directions (polarization directions of light to be transmitted).
  • the polarization directions of the polarizing filter elements 214A, 214B, 214C, 214D can be, for example, 0°, 45°, 90°, 135°. Also, these polarization directions can correspond to the polarization directions of the polarization filters 139A to 139D (see FIG. 7) in the filter unit 134 described above.
  • the image sensor 138 includes a plurality of image groups receiving any of the light transmitted through the plurality of aperture regions by these polarizing filter elements 214A-214D. These polarizing filter elements 214A and 214B are arranged at the same intervals as the photodiodes 211A and provided for each pixel.
  • the microlens array layer 215 includes microlenses 216 arranged in each pixel.
  • the image sensor 138 includes an analog amplifier, an A/D converter (Analog-to-Digital Converter), and an imaging device driver (not shown).
  • FIG. 9 is a diagram showing the configuration of the processor 230 (processor).
  • the processor 230 includes an imaging control unit 232, an image acquisition unit 234, a parameter acquisition unit 236, an interference elimination unit 238, a display control unit 240, and a recording control unit 242.
  • An information acquisition process (information acquisition process) for acquiring a plurality of image signals, a parameter acquisition process (parameter acquisition process) for acquiring interference elimination parameters, an interference elimination process (interference elimination process) for eliminating interference, and the like are executed.
  • processors include, for example, a CPU (Central Processing Unit), which is a general-purpose processor that executes software (programs) to realize various functions.
  • CPU Central Processing Unit
  • GPUs Graphics Processing Units
  • the various processors described above also include a PLD (Programmable Logic Device), which is a processor whose circuit configuration can be changed after manufacturing, such as an FPGA (Field Programmable Gate Array).
  • PLD Programmable Logic Device
  • FPGA Field Programmable Gate Array
  • a dedicated electric circuit which is a processor having a circuit configuration exclusively designed for executing specific processing such as ASIC (Application Specific Integrated Circuit), is included in the various processors described above.
  • Each function of the processor 230 may be implemented by one processor or may be implemented by a plurality of processors. Also, one processor may correspond to a plurality of functions. Furthermore, each function of processor 230 may be implemented by a circuit, or a part of each function may be implemented by a circuit and the rest by a processor.
  • processor When the processor or electric circuit described above executes software (program), processor (computer) readable code of the software to be executed and data necessary for executing the software are stored in a non-temporary memory such as flash memory 244 (Flash Memory).
  • the software and data are stored in a physical recording medium, and the processor refers to the software and data.
  • the software stored in the non-temporary recording medium includes an adjustment program for executing the adjustment method according to this embodiment.
  • codes and data may be recorded in a non-temporary recording medium using various magneto-optical recording devices, semiconductor memories, or the like.
  • semiconductor memory includes ROM (Read Only Memory) and EEPROM (Electronically Erasable and Programmable ROM) in addition to flash memory.
  • ROM Read Only Memory
  • EEPROM Electrically Erasable and Programmable ROM
  • the parameter acquisition unit 236 acquires interference elimination parameters (second interference elimination parameters) according to each of the following modes, and obtains them by shooting in a development environment or the like in a state in which noise or the like in the actual environment is not taken into account. Based on the plurality of image signals obtained, an interference removal parameter (first interference removal parameter) used for interference removal of the plurality of image signals is obtained (first parameter obtaining step, first parameter obtaining process).
  • the wavelength is adjusted while physically shielding some (two) of the three aperture regions 135A to 135C (a plurality of aperture regions) and leaving the remaining (one) aperture region open.
  • a subject whose characteristics are known is photographed (preliminary photographing, first photographing). Images are obtained by changing the opening to be opened and photographing is repeated, and interference elimination parameters are obtained based on these images.
  • the focal position for preliminary photography is equivalent to the focal position for main photography (actual photography; second photography).
  • the focus position being "equal" includes not only the case where the focus position is completely the same, but also the case where there is a deviation to the extent that the influence on the elimination of interference is acceptable.
  • FIG. 10 is a diagram showing how the light shielding member 131 shields a part of the openings and opens the rest of the openings.
  • a light shielding member 131A corresponding to the opening region 135A is arranged on the object side of the frame 135 to shield the opening region other than the opening region 135A.
  • a light shielding member 131B corresponding to 135B is arranged on the subject side of the frame 135 to shield the opening area other than the opening area 135B, and a light shielding member 131C corresponding to the opening area 135C is arranged on the subject side of the frame 135 to close the opening.
  • the open areas other than the area 135C are shielded.
  • Part (b) of the figure shows a state in which the opening region 135A is opened by the light blocking member 131A.
  • the light blocking member 131 it is possible to use a member that does not transmit light (light in the wavelength band used for image acquisition) at all or substantially does not transmit (the influence of transmission is within an allowable range in terms of accuracy of eliminating interference). preferable.
  • filter sets 137A to 137C are arranged on the imaging device main body 200 side of the frame 135 (see FIG. 6). Specifically, a color filter 138A of a wavelength band ⁇ 1 of transmitted light and a polarizing filter 139A of a polarization angle of 0 deg are arranged for the aperture region 135A, and a polarizing filter 139A of a wavelength band ⁇ 2 of transmitted light is arranged for the aperture region 135B.
  • a color filter 138B and a polarization filter 139B with a polarization angle of 45 degrees are arranged, and a color filter 138C with a wavelength band ⁇ 3 of transmitted light and a polarization filter 139C with a polarization angle of 90 degrees are arranged for the aperture region 135C.
  • FIG. 11 is a diagram showing a state of preliminary photography (first photography) for acquiring interference cancellation parameters in the first mode.
  • Part (a) of FIG. 11 shows a state in which an aperture region 135A that transmits light in the wavelength band ⁇ 1 (polarization angle of 0 deg) is opened and the remaining aperture region is shielded by a light shielding member 131A.
  • the imaging control unit 232 controls readout of an image signal from the image sensor 210 (image sensor) in response to an imaging instruction operation on the operation device 320 (shutter button or the like), and converts the image signal output by shooting into Information indicating the wavelength characteristics of the light source 99 (object) is obtained (information obtaining step, information obtaining process). It is assumed that the light source 99 has known wavelength characteristics. Examples of "subjects with known wavelength characteristics" include white paper and color charts.
  • image signals output from the four types of pixels (pixels corresponding to the polarizing filter elements 214A to 214D) of the image sensor 210 in this state are assumed to be x0, x45, x135, and x90.
  • part (b) of FIG. 11 shows a state in which an aperture region 135B that transmits light in the wavelength band ⁇ 2 (polarization angle of 45 degrees) is opened and the remaining aperture region is shielded by a light shielding member 131B.
  • the image signals output from the image sensor 210 in this state are y0, y45, y135, and y90.
  • a process (processing) in which the imaging control unit 232 acquires these image signals is an information acquisition process (information acquisition process).
  • Part (c) of FIG. 11 shows a state in which an aperture region 135C that transmits light in the wavelength band ⁇ 3 (polarization angle of 90 degrees) is opened and the remaining aperture region is shielded by a light shielding member 131C.
  • the image signals output from the image sensor 210 in this state are z0, z45, z135, and z90.
  • a process in which the imaging control unit 232 acquires these image signals is also an information acquisition process.
  • the opening of the aperture regions need not be performed in the order of the aperture regions 135A to 135C.
  • the imaging control unit 232 (processor) generates four image signals (plurality of image signals; (object) is acquired as information indicating wavelength characteristics (information acquisition step, information acquisition processing).
  • opening one of the three opening areas opening areas 135A to 135C
  • shielding the remaining two opening areas has been described.
  • the number of open areas to be covered is not limited to two, and at least one open area may be shielded.
  • FIG. 12 is a diagram showing the state of main photography (second photography) in the first mode.
  • the imaging control unit 232 (processor) acquires an image signal in a state in which no light shielding member is arranged.
  • the elements of the matrix (x0, x45, x135, x90) T on the left side are the sensor intensities of the pixels with the polarization angles of 0 deg, 45 deg, 135 deg, and 90 deg of the image sensor 210, respectively.
  • image signal the components of the matrix (I ⁇ 1, 0, 0) T on the right side are the intensities of the light in the wavelength bands ⁇ 1, ⁇ 2, and ⁇ 3 that have passed through the filter unit 134, respectively.
  • Equation 1 is such that when interference is eliminated from the sensor output intensity when only the first aperture region is opened, the intensity of light in the wavelength band ⁇ 1 is I ⁇ 1, and the intensity of light in the other wavelength bands ⁇ 2 and ⁇ 3 is 0. It means that it should be Note that the matrix (x0, x45, x135, x90) T on the left side may be described as "matrix X" below.
  • the elements of the matrix (y0, y45, y135, y90) T on the left side are the sensor intensities of the pixels with the polarization angles of 0 deg, 45 deg, 135 deg, and 90 deg of the image sensor 210 (first image signal), and the components of the matrix (0, I ⁇ 2, 0) T on the right side are the intensities of the light in the wavelength bands ⁇ 1, ⁇ 2, and ⁇ 3 that have passed through the filter unit 134, respectively.
  • Equation 2 states that "when interference is eliminated from the sensor output intensity when only the second aperture region is opened, the intensity of light in the wavelength band ⁇ 2 is I ⁇ 2, and the intensity of light in the other wavelength bands ⁇ 1 and ⁇ 3 is 0. It means that it should be Note that the matrix (y0, y45, y135, y90) T on the left side may be described as "matrix Y" below.
  • the elements of the matrix (z0, z45, z135, z90) T on the left side are the sensor intensities of the pixels with the polarization angles of 0 deg, 45 deg, 135 deg, and 90 deg of the image sensor 210 (first image signal), and the components of the matrix (0, 0, I ⁇ 3) T on the right side are the intensities of the light in the wavelength bands ⁇ 1, ⁇ 2, and ⁇ 3 that have passed through the filter unit 134, respectively.
  • Equation 3 states that "when interference is eliminated from the sensor output intensity when only the third aperture region is opened, the intensity of light in the wavelength band ⁇ 3 is I ⁇ 3, and the intensity of light in the other wavelength bands ⁇ 1 and ⁇ 2 is 0. It means that it should be In the following, the matrix (z0, z45, z135, z90) T on the left side may be referred to as "matrix Z".
  • the parameter acquisition unit 236 refers to the first image signal (information indicating the wavelength characteristics of the subject) acquired for the first to third aperture regions based on the above (formula 1) to (formula 3). ) to acquire a second interference cancellation parameter used for interference cancellation of the second image signal (a plurality of image signals in the second imaging (main imaging)) (second parameter acquisition step, second parameter acquisition process).
  • the parameter acquisition unit 236 obtains the pseudo-inverse matrix A second interference elimination parameter (interference elimination matrix A) is acquired by the following (Equation 5) using a formula (second parameter acquisition step, second parameter acquisition process).
  • Equation 6 states that "when interference is eliminated from the sensor output intensity when only the first aperture region is opened, the intensity of light in the wavelength band ⁇ 1 is 1, and the intensity of light in the other wavelength bands ⁇ 2 and ⁇ 3 is 0. It means that it should be
  • Equation 7 states that "when interference is eliminated from the sensor output intensity when only the second aperture region is opened, the intensity of light in the wavelength band ⁇ 2 is 1, and the intensity of light in the other wavelength bands ⁇ 1 and ⁇ 3 is 0. It means that it should be
  • Equation 8 states that "when interference is eliminated from the sensor output intensity when only the third aperture region is opened, the intensity of light in the wavelength band ⁇ 3 is 1, and the intensity of light in the other wavelength bands ⁇ 1 and ⁇ 2 is 0. It means that it should be
  • the parameter acquisition unit 236 obtains the second interference cancellation parameter ( Interference cancellation matrix A) is acquired (second parameter acquisition step, second parameter acquisition process).
  • the interference removing unit 238 (processor) removes interference as shown in (Equation 12) below (interference removing step, interference removing process).
  • the above-described processing yields (Intensity of light passing through the first to third aperture regions) is (1, 2, 3), and interference can be eliminated correctly. That is, the difference between the information acquired by the first imaging (the first image signal, information indicating the wavelength characteristics of the object) and the second image signal eliminated by the second interference elimination parameter is the information acquired by the first imaging. is smaller than the difference between the second image signal and the interference-removed second image signal with the first interference-removal parameter.
  • the display control unit 240 (processor) outputs an image (a plurality of image signals) corresponding to the image signal after the interference removal (the image signal (1, 2, 3) in the above example) to the display device 300 (output device). can be displayed.
  • the recording control unit 242 (processor) can cause the storage device 310 (output device) to store an image (a plurality of image signals) corresponding to the image signal after interference removal.
  • Equation 13 is such that when interference is eliminated from the sensor output intensity when only the first aperture region is opened, the intensity of light in the wavelength band ⁇ 1 is 1, and the intensity of light in the other wavelength bands ⁇ 2 and ⁇ 3 is 0. It means that it should be
  • Equation 14 states that "when interference is eliminated from the sensor output intensity when only the second aperture region is open, the intensity of light in the wavelength band ⁇ 2 is 1, and the intensity of light in the other wavelength bands ⁇ 1 and ⁇ 3 is 0. It means that it should be
  • Equation 15 states that "when interference is eliminated from the sensor output intensity when only the third aperture region is opened, the intensity of light in the wavelength band ⁇ 3 is 1, and the intensity of light in the other wavelength bands ⁇ 1 and ⁇ 2 is 0. It means that it should be
  • the parameter acquisition unit 236 obtains the interference cancellation parameter (interference cancellation Matrix A) is acquired (parameter acquisition step, parameter acquisition process).
  • the interference removal unit 238 (processor) performs interference removal as shown in (Equation 19) below (interference removal step, interference removal processing).
  • a light-shielding member that does not transmit light is used to shield (light-shield) areas other than a part of the opening area.
  • (Second polarizing member) may be used to block light. For example, light can be blocked by disposing a polarizing filter with a polarization angle of 90 degrees in the first opening area (opening area 135A, polarization angle of polarizing filter 139 is 0 degrees). The same applies to the second and third opening areas.
  • optical shielding is performed by changing the wavelength band of illumination light as shown in FIG. 13 without using a light shielding member as in the first mode.
  • Preliminary photography is performed on a subject whose wavelength characteristics are known in an optically shielded state.
  • the first aperture region here, the aperture region 135A in which the color filter 138A of the transmitted light wavelength band ⁇ 1 is arranged
  • the second and third aperture regions aperture region 135A in which the color filter 138A of the wavelength band ⁇ 2 is arranged
  • aperture region 135C in which the color filter 138C of the wavelength band ⁇ 3 is arranged.
  • the second aperture region (here, the aperture region 135B in which the color filter 138B of the wavelength band ⁇ 2 of the transmitted light is arranged) transmits 1
  • the light of the wavelength band that does not pass through the th and third aperture regions (the aperture region 135A in which the color filter 138A of the wavelength band ⁇ 1 is arranged and the aperture region 135C in which the color filter 138C of the wavelength band ⁇ 3 is arranged) is irradiated.
  • the illumination in these pre-shooting shadows can be performed using, for example, a laser light source or an LED (Light-Emitting Diode) light source that emits monochromatic light such as red, green, and blue.
  • a diffusion plate 99A may be placed in front of the light source 99 (between the light source 99 and the lens device 100) to uniformly disperse the light from the light source.
  • a device such as a monochromator or a monochromator
  • light in a desired wavelength band is extracted from light containing multiple wavelength bands using an optical element such as a prism or a diffraction grating, and used for illumination in preliminary photography. good too.
  • the imaging control unit 232, the image acquisition unit 234, and the parameter acquisition unit 236 acquire the wavelength characteristics of the subject acquired in preliminary imaging in the same manner as described above for the first aspect.
  • An interference cancellation parameter can be acquired based on the information (wavelength intensity of the object) indicated (information acquisition step, information acquisition process, second parameter acquisition process, second parameter acquisition process).
  • the image sensor 210 is used, which has a polarizing filter arranged in the aperture region and a polarizing filter element array layer 213 .
  • the third mode is a mode in which a color sensor is used without using a polarizing filter.
  • FIG. 15 is a diagram showing a state of preliminary photography (first photography) in the third mode.
  • the filter unit 134A has color filters but no polarizing filters.
  • the image sensor 210A does not have a polarizing filter element array layer, and is provided with color filters 212A to 212C having transmission wavelength bands corresponding to the transmission wavelength bands of the color filters 138A to 138C (see FIG. 6) of the filter unit 134A. ing.
  • These color filters 212A to 212C constitute the color filter array layer 212.
  • FIG. 1 shows a polarizing filter arranged in the aperture region and a polarizing filter element array layer 213 .
  • the third mode is a mode in which a color sensor is used without using a polar
  • the image acquisition unit 234 blocks the aperture regions 135A and 135B with the light shielding member 140A and opens the aperture region 135C as shown in part (a) of FIG.
  • Image signals for light in the wavelength bands ⁇ 1 to ⁇ 3 are acquired as "information indicating the wavelength characteristics of the subject" in a state of being transmitted through the unit 134 (information acquisition step, information acquisition process).
  • the image acquisition unit 234 blocks the aperture regions 135A and 135C with the light shielding member 140B and opens the aperture region 135B, that is, only the light in the wavelength band ⁇ 2 is filtered by the filter unit.
  • image signals for light in the wavelength bands ⁇ 1 to ⁇ 3 are acquired as "information indicating the wavelength characteristics of the subject" (information acquisition step, information acquisition process). Furthermore, as shown in part (c) of FIG. 15, the image acquisition unit 234 blocks the aperture regions 135B and 135C with the light shielding member 140C and opens the aperture region 135A. Image signals for light in the wavelength bands ⁇ 1 to ⁇ 3 (a plurality of image signals for a plurality of lights) are acquired as "information indicating the wavelength characteristics of the subject" in a state of being transmitted through the unit 134 (information acquisition step, information acquisition process).
  • the parameter acquisition unit 236 can acquire the interference cancellation parameters (parameter acquisition step, parameter acquisition process) in the same manner as described above for the first and second aspects, and the interference cancellation unit 238 (processor) can perform interference elimination using the acquired interference elimination parameters (interference elimination processing, interference elimination step).
  • FIG. 16 is a diagram showing the state of actual photography (second photography) in the third mode. As shown in the figure, in actual photographing, photographing is performed without using the light shielding members 140A to 140C.
  • the color filters 138A to 138C arranged in the aperture regions 135A to 135C and the color filters of the transmission wavelength band (wavelength band selection filters) are arranged on the object side (light source side) from the filter unit 134, and the wavelength characteristics Preliminary photography (first photography) is performed for a subject with a known .
  • Parts (a) to (c) of FIG. 17 show how the color filters 142A to 142C having transmitted light wavelength bands of ⁇ 3, ⁇ 2, and ⁇ 1 are arranged closer to the object than the filter unit 134. FIG. .
  • the color filters are not directly attached to the filter unit 134 as in the example shown in FIG. may be separately attached to the frame 132 different from the frame 135 and inserted closer to the subject than the filter unit 134 .
  • the color filter 144 may be mounted closer to the subject than the first lens 110 (in FIG. 19, the lens device 100 closest to the subject) and replaced.
  • the same transmission wavelength band as that of the color filters 138A to 138C is transmitted while the color filters 138A to 138C are not attached to the frame 135 or the filter unit 134 is not inserted into the slit 108.
  • the color filter (wavelength selection filter) provided is arranged closer to the subject than the frame 135 . That is, the color filters are arranged closer to the subject than the color filters 138A to 138C when the color filters 138A to 138C are attached to the frame 135. FIG. Alternatively, the color filters are arranged closer to the subject than the frame 135 when the filter unit 134 (frame 135) is inserted into the slit 108 .
  • the image acquisition unit 234 acquires an image signal (first information) about a subject whose wavelength characteristics are unknown by preliminary imaging (first imaging) in this state (information acquisition step, information acquisition process).
  • Parts (a) to (c) of FIG. 20 are diagrams showing a state in which color filters 146A to 146C having transmitted light wavelength bands of ⁇ 3, ⁇ 2, and ⁇ 1 are arranged closer to the subject than the frame 135.
  • the image acquisition unit 234 acquires the first image signal in the state shown in each part of the drawing.
  • the parameter acquisition unit 236 acquires a parameter for correcting the second information as an interference cancellation parameter based on the first information (parameter acquisition step, parameter acquisition process).
  • the interference elimination unit 238 can eliminate interference using the acquired interference elimination parameters (interference elimination process, interference elimination process). According to the fifth aspect, it is possible to obtain highly accurate interference elimination parameters even for an object whose wavelength characteristics are unknown.
  • Imaging system 99 Light source 99A Diffusion plate 100 Lens device 102 Lens barrel 104 First lever 106 Second lever 108 Slit 110 First lens 120 Second lens 131 Light shielding member 131A Light shielding member 131B Light shielding member 131C Light shielding member 132 Frame 133 Frame Body 133A Opening area 133B Opening area 133C Opening area 134 Filter unit 134A Filter unit 135 Frame body 135A Opening area 135B Opening area 135C Opening area 135D Opening area 135E Light blocking member 135G Center of gravity 137 Filter set 137A Filter set 137B Filter set 137C Filter set 137D Filter Set 138 Image sensor 138A Color filter 138B Color filter 138C Color filter 138D Color filter 139 Polarization filter 139A Polarization filter 139B Polarization filter 139C Polarization filter 139D Polarization filter 140A Light blocking member 140B Light blocking member 140C Light blocking member 142A Color filter 142B Color filter 142C Color filter 144 Color filter 146A Color filter

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • Optics & Photonics (AREA)
  • Studio Devices (AREA)
PCT/JP2022/029055 2021-09-22 2022-07-28 情報処理方法、情報処理装置、情報処理プログラム、及び情報処理システム Ceased WO2023047794A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
CN202280061734.3A CN117957849A (zh) 2021-09-22 2022-07-28 信息处理方法、信息处理装置、信息处理程序及信息处理系统
JP2023549396A JP7824307B2 (ja) 2021-09-22 2022-07-28 情報処理方法、情報処理装置、情報処理プログラム、及び情報処理システム
US18/595,461 US20240212107A1 (en) 2021-09-22 2024-03-05 Information processing method, information processing apparatus, information processing program, and information processing system

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2021154616 2021-09-22
JP2021-154616 2021-09-22

Related Child Applications (1)

Application Number Title Priority Date Filing Date
US18/595,461 Continuation US20240212107A1 (en) 2021-09-22 2024-03-05 Information processing method, information processing apparatus, information processing program, and information processing system

Publications (1)

Publication Number Publication Date
WO2023047794A1 true WO2023047794A1 (ja) 2023-03-30

Family

ID=85720504

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/029055 Ceased WO2023047794A1 (ja) 2021-09-22 2022-07-28 情報処理方法、情報処理装置、情報処理プログラム、及び情報処理システム

Country Status (4)

Country Link
US (1) US20240212107A1 (https=)
JP (1) JP7824307B2 (https=)
CN (1) CN117957849A (https=)
WO (1) WO2023047794A1 (https=)

Families Citing this family (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117999793A (zh) * 2021-09-29 2024-05-07 富士胶片株式会社 数据处理装置、方法及程序以及多光谱相机

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021153473A1 (ja) * 2020-01-31 2021-08-05 富士フイルム株式会社 レンズ装置、撮像装置、光学部材、撮像方法、及び撮像プログラム
JP2021135404A (ja) * 2020-02-27 2021-09-13 富士フイルム株式会社 レンズ装置、撮像装置、撮像方法、及び撮像プログラム

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6145079B2 (ja) * 2014-08-29 2017-06-07 富士フイルム株式会社 撮像装置及び撮像方法
TWI552603B (zh) * 2015-05-19 2016-10-01 晶相光電股份有限公司 影像校正系統以及方法
WO2020250774A1 (ja) * 2019-06-11 2020-12-17 富士フイルム株式会社 撮像装置
WO2021172284A1 (ja) * 2020-02-28 2021-09-02 富士フイルム株式会社 撮像装置及び方法
CN117999793A (zh) * 2021-09-29 2024-05-07 富士胶片株式会社 数据处理装置、方法及程序以及多光谱相机
CN117999792A (zh) * 2021-09-30 2024-05-07 富士胶片株式会社 信息处理装置、方法及程序以及摄像装置

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2021153473A1 (ja) * 2020-01-31 2021-08-05 富士フイルム株式会社 レンズ装置、撮像装置、光学部材、撮像方法、及び撮像プログラム
JP2021135404A (ja) * 2020-02-27 2021-09-13 富士フイルム株式会社 レンズ装置、撮像装置、撮像方法、及び撮像プログラム

Also Published As

Publication number Publication date
JPWO2023047794A1 (https=) 2023-03-30
JP7824307B2 (ja) 2026-03-04
US20240212107A1 (en) 2024-06-27
CN117957849A (zh) 2024-04-30

Similar Documents

Publication Publication Date Title
JP4483951B2 (ja) 撮像装置
JP4941332B2 (ja) 撮像装置
JP7441313B2 (ja) レンズ装置、撮像装置、及び撮像方法
US20240019762A1 (en) Lens device, imaging apparatus, imaging method, and imaging program
JP7261192B2 (ja) レンズ装置、撮像装置、撮像方法、及び撮像プログラム
JP7238121B2 (ja) 撮像装置
CN115152205A (zh) 摄像装置及方法
JP4968527B2 (ja) 撮像装置
US20230319385A1 (en) Optical member, lens device, and imaging apparatus
WO2023047794A1 (ja) 情報処理方法、情報処理装置、情報処理プログラム、及び情報処理システム
US12197109B2 (en) Lens device and imaging apparatus
JP2017208585A (ja) 撮像装置および画像データ生成プログラム
US20090102939A1 (en) Apparatus and method for simultaneously acquiring multiple images with a given camera
CN108605083B (zh) 摄像装置及图像数据生成方法
JPWO2023047794A5 (https=)
WO2017126242A1 (ja) 撮像装置、及び、画像データ生成方法
JP7399194B2 (ja) 画像処理装置、撮像装置、画像処理方法及び画像処理プログラム
WO2025047424A1 (ja) 撮像装置
JP2025034968A (ja) 情報処理装置、方法及びプログラム並びにマルチスペクトルカメラ
JP2006003247A (ja) テレビジョンカメラ装置

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 2023549396

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 202280061734.3

Country of ref document: CN

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 22872550

Country of ref document: EP

Kind code of ref document: A1