US11870969B2 - Imaging apparatus, portable terminal apparatus, and imaging method - Google Patents
Imaging apparatus, portable terminal apparatus, and imaging method Download PDFInfo
- Publication number
- US11870969B2 US11870969B2 US17/329,689 US202117329689A US11870969B2 US 11870969 B2 US11870969 B2 US 11870969B2 US 202117329689 A US202117329689 A US 202117329689A US 11870969 B2 US11870969 B2 US 11870969B2
- Authority
- US
- United States
- Prior art keywords
- imaging condition
- correction coefficient
- camera
- imaging
- image
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active, expires
Links
- 238000003384 imaging method Methods 0.000 title claims abstract description 417
- 230000003595 spectral effect Effects 0.000 claims abstract description 77
- 230000008859 change Effects 0.000 claims abstract description 13
- 238000012937 correction Methods 0.000 claims description 233
- 238000011156 evaluation Methods 0.000 claims description 51
- 230000003287 optical effect Effects 0.000 claims description 33
- 230000035945 sensitivity Effects 0.000 claims description 27
- 238000000034 method Methods 0.000 claims description 13
- 238000002834 transmittance Methods 0.000 claims description 6
- 238000012545 processing Methods 0.000 description 25
- 238000010586 diagram Methods 0.000 description 8
- 238000012986 modification Methods 0.000 description 7
- 230000004048 modification Effects 0.000 description 7
- 230000005856 abnormality Effects 0.000 description 6
- 238000010801 machine learning Methods 0.000 description 6
- 230000006866 deterioration Effects 0.000 description 5
- 239000000758 substrate Substances 0.000 description 5
- 230000006870 function Effects 0.000 description 4
- 239000004065 semiconductor Substances 0.000 description 4
- 230000003245 working effect Effects 0.000 description 4
- 238000005286 illumination Methods 0.000 description 3
- 230000007246 mechanism Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 2
- 230000000295 complement effect Effects 0.000 description 2
- 229910044991 metal oxide Inorganic materials 0.000 description 2
- 150000004706 metal oxides Chemical class 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 238000012360 testing method Methods 0.000 description 2
- 239000003086 colorant Substances 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000001771 impaired effect Effects 0.000 description 1
- 238000012417 linear regression Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/257—Colour aspects
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
- H04N13/106—Processing image signals
- H04N13/15—Processing image signals for colour aspects of image signals
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N13/20—Image signal generators
- H04N13/296—Synchronisation thereof; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/10—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from different wavelengths
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/54—Mounting of pick-up tubes, electronic image sensors, deviation or focusing coils
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/62—Control of parameters via user interfaces
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
- H04N23/63—Control of cameras or camera modules by using electronic viewfinders
- H04N23/631—Graphical user interfaces [GUI] specially adapted for controlling image capture or setting capture parameters
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/70—Circuitry for compensating brightness variation in the scene
- H04N23/73—Circuitry for compensating brightness variation in the scene by influencing the exposure time
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/90—Arrangement of cameras or camera modules, e.g. multiple cameras in TV studios or sports stadiums
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
- H04N2013/0074—Stereoscopic image analysis
- H04N2013/0077—Colour aspects
Definitions
- the present disclosure relates to an imaging apparatus, a portable terminal apparatus, and an imaging method.
- an imaging apparatus including an RGB camera for capturing a color image and a spectroscopic camera for capturing a spectroscopic image, is known (see, for example, JP-T-2015-502058).
- the imaging apparatus described in JP-T-2015-502058 is an imaging system including an RGB sensor which is an RGB camera and a NIR sensor which is a spectroscopic camera.
- the spectroscopic image captured by the NIR sensor can be superimposed and displayed on the color image captured by the RGB sensor.
- An imaging apparatus includes a first camera which includes a color filter portion including a red color filter that transmits light in a red wavelength region, a green color filter that transmits light in a green wavelength region, and a blue color filter that transmits light in a blue wavelength region, and a first light receiving portion receiving light transmitted through the color filter portion, and which captures a color image based on a gradation value of red light transmitted through the red color filter, a gradation value of green light transmitted through the green color filter, and a gradation value of blue light transmitted through the blue color filter, a second camera which includes a spectroscopic element configured to disperse light having a predetermined spectral wavelength from incident light and to change the spectral wavelength to four or more wavelengths, and a second light receiving portion receiving the light dispersed by the spectroscopic element, and which captures a spectroscopic image of each wavelength dispersed by the spectroscopic element, a first imaging condition setting portion which sets a first imaging condition which is
- a portable terminal apparatus includes the imaging apparatus of the first aspect as described above and a housing in which the imaging apparatus is incorporated.
- An imaging method is an imaging method of an imaging apparatus including a first camera which includes a color filter portion including a red color filter that transmits light in a red wavelength region, a green color filter that transmits light in a green wavelength region, and a blue color filter that transmits light in a blue wavelength region, and a first light receiving portion receiving light transmitted through the color filter portion, and which captures a color image based on a gradation value of red light transmitted through the red color filter, a gradation value of green light transmitted through the green color filter, and a gradation value of blue light transmitted through the blue color filter, and a second camera which includes a spectroscopic element configured to disperse light having a predetermined spectral wavelength from incident light and to change the spectral wavelength to four or more wavelengths, and a second light receiving portion receiving the light dispersed by the spectroscopic element, and which captures a spectroscopic image of each wavelength dispersed by the spectroscopic element, including setting a first imaging condition
- FIG. 1 is a side sectional view illustrating a schematic configuration of an electronic device according to a first embodiment.
- FIG. 2 is a block diagram illustrating a schematic configuration of the electronic device according to the first embodiment.
- FIG. 3 is a flowchart illustrating an imaging method of a spectroscopic image of the first embodiment.
- FIG. 4 is a diagram illustrating a light receiving sensitivity characteristic of a second light receiving portion, a spectral characteristic of a spectroscopic element, a light source characteristic of a light source, and a camera optical characteristic of a second camera obtained by multiplying these previous characteristics, in the first embodiment.
- FIG. 5 is a diagram illustrating an overview of a correction coefficient of a third embodiment.
- FIG. 6 is a block diagram illustrating a schematic configuration of an electronic device according to a fourth embodiment.
- FIG. 7 is a flowchart illustrating an imaging method of a spectroscopic image according to the fourth embodiment.
- FIG. 1 is a view illustrating an example of an electronic device 1 of the present embodiment, and is a side sectional view illustrating a schematic configuration of the electronic device 1 .
- FIG. 2 is a block diagram illustrating a schematic configuration of the electronic device 1 .
- the electronic device 1 is a portable terminal apparatus that can be carried by a user, and for example, a smartphone, a tablet terminal, a mobile phone, a head-mounted display, or the like can be exemplified.
- a smartphone is exemplified as an example of the electronic device 1 .
- the electronic device 1 includes a housing 10 , a first camera 21 disposed inside the housing 10 , a second camera 22 , and a control portion 30 (see FIG. 2 ).
- the first camera 21 , the second camera 22 , and the control portion 30 constitute the imaging apparatus according to the present disclosure.
- the housing 10 is formed in a thin box shape, for example, and is provided with a touch panel 11 on one side, which displays an image and accepts an input operation from a user.
- the electronic device 1 may include an input button or the like that accepts an input operation from the user.
- first window 12 and a second window 13 are provided on a surface opposite to the surface on which the touch panel 11 of the housing 10 is provided.
- the first window 12 is provided to face the first camera 21
- a second window 13 is provided to face the second camera 22 .
- the electronic device 1 may include a light source or the like that irradiates a target object with illumination light when the first camera 21 and the second camera 22 capture the target object, and may be provided with a light amount detection sensor or the like for automatically setting a first imaging condition which is an imaging condition by the first camera 21 .
- the first camera 21 is an RGB camera disposed to face the first window 12 .
- the RGB camera is a camera for imaging a color image which is widely used in a portable terminal such as a general smartphone.
- the first camera 21 includes a color filter portion 211 , a first light receiving portion 212 , and a first imaging optical system 213 .
- the color filter portion 211 includes a red color filter (R filter) which transmits light in the red wavelength region, a green color filter (G filter) which transmits light in the green wavelength region, and a blue color filter (B filter) which transmits light in the blue wavelength region.
- R filter red color filter
- G filter green color filter
- B filter blue color filter
- the R filter, the G filter, and the B filter may be collectively referred to as a color filter.
- a plurality of these R filters, G filters, and B filters are provided, and for example, is disposed in a Bayer array.
- the first light receiving portion 212 is an imaging element constituted with a general charge-coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor.
- the first light receiving portion 212 has an imaging pixel corresponding to each color of the color filter portion 211 , and receives light transmitted through the corresponding color filter.
- the first light receiving portion 212 acquires an RGB gradation value for one pixel of image data by receiving light transmitted through one R filter, one B filter, and two G filters at the imaging pixel corresponding to each filter. Accordingly, the first camera 21 acquires the RGB gradation value of each pixel of the image data from each imaging pixel of the first light receiving portion 212 to generate a color image.
- the first imaging optical system 213 is an optical system which guides the light incident from the first window 12 to the first light receiving portion 212 , and is constituted with, for example, a plurality of lenses or the like. Further, as illustrated in FIG. 2 , the first imaging optical system 213 is provided with a first aperture mechanism 213 A capable of adjusting the incident range of the light incident on the first light receiving portion 212 , and adjusts the light receiving amount of the first light receiving portion 212 by adjusting the incident range.
- the second camera 22 is a spectroscopic camera disposed to face the second window 13 , and captures each spectroscopic image for a plurality of wavelengths.
- the second camera 22 includes a spectroscopic element 221 , a second light receiving portion 222 , and a second imaging optical system 223 .
- the spectroscopic element 221 is an element capable of spectroscopically transmitting light having a predetermined spectral wavelength from incident light and switching the spectral wavelength to be dispersed to an optional wavelength.
- the spectroscopic element 221 uses a variable wavelength Fabry-Perot element provided with a pair of reflective films facing each other and an electrostatic actuator which changes the gap between the pair of reflective films by applying a voltage.
- the spectroscopic element 221 using such a variable wavelength Fabry-Perot element can be made smaller and thinner in element size, and can be suitably incorporated particularly for an electronic device 1 having a limited disposition space such as a portable terminal apparatus.
- the spectroscopic element 221 is not limited to the Fabry-Perot element as described above, and for example, acousto-optic tunable filter (AOTF), liquid crystal tunable filter (LCTF), or the like may be used.
- AOTF acousto-optic tunable filter
- LCTF liquid crystal tunable filter
- the spectroscopic element 221 of the present embodiment can switch the spectral wavelength of the transmitted light into four or more wavelengths, and can capture an image in which the color of a target object can be analyzed with higher accuracy than in the first camera 21 which captures a color image in three colors of RGB, in the visible light region.
- the second light receiving portion 222 is provided on the same substrate as a camera substrate 23 on which the first light receiving portion 212 is provided.
- the first light receiving portion 212 and the second light receiving portion 222 are disposed on the same camera substrate 23 , but the circuit substrate provided with the first light receiving portion 212 and the circuit substrate provided with the second light receiving portion 222 may be separate bodies.
- the second light receiving portion 222 is an imaging element constituted with a general charge-coupled device (CCD) sensor and a complementary metal oxide semiconductor (CMOS) sensor. That is, the second light receiving portion 222 has a plurality of imaging pixels, and each imaging pixel outputs an electric signal according to the intensity of the received light. Accordingly, the second camera 22 generates a spectroscopic image in which the light intensity value output from each imaging pixel is the data of one pixel.
- a general RGB camera such as the first light receiving portion 212 interpolates data from the data of a peripheral pixel, and for example, in the pixel corresponding to the R filter, calculates blue and green data from the data of the pixels corresponding to the peripheral B filter and G filter.
- the second light receiving portion 222 which is a spectroscopic camera can directly acquire data for a plurality of wavelengths at each pixel, so that the above interpolation is unnecessary and the high-definition vivid image can be generated compared to a general RGB camera.
- the second imaging optical system 223 is an optical system which guides the light incident from the second window 13 to the second light receiving portion 222 , and is constituted with, for example, a plurality of lenses. Further, as illustrated in FIG. 2 , the second imaging optical system 223 is provided with a second aperture mechanism 223 A capable of adjusting the incident range of the light incident on the second light receiving portion 222 , and adjusts the light receiving amount of the second light receiving portion 222 by adjusting the incident range.
- a second aperture mechanism 223 A capable of adjusting the incident range of the light incident on the second light receiving portion 222 , and adjusts the light receiving amount of the second light receiving portion 222 by adjusting the incident range.
- the second imaging optical system 223 is provided on the side opposite to the second light receiving portion 222 of the spectroscopic element 221 , but an optical component such as a lens constituting the second imaging optical system 223 may also be provided between the spectroscopic element 221 and the second light receiving portion 222 .
- the control portion 30 is constituted with various electronic circuits such as a central processing unit (CPU) mounted on a main board 14 disposed inside the housing 10 and a semiconductor memory, and includes a storage 31 and a processor 32 .
- CPU central processing unit
- the storage 31 is constituted with, for example, a semiconductor memory or the like, and stores various programs and various data.
- the storage 31 functions as a correction coefficient recording portion according to the present disclosure, and stores a correction coefficient ⁇ for setting the second imaging condition.
- the correction coefficient ⁇ will be described later.
- the processor 32 executes various processing in the electronic device 1 by reading and executing the program stored in the storage 31 . Specifically, the processor 32 functions as a first imaging condition setting portion 321 , a second imaging condition setting portion 322 , a first imaging control portion 323 , a second imaging control portion 324 , an image evaluation portion 325 , and a correction coefficient updating portion 326 .
- the first imaging condition setting portion 321 acquires the first imaging condition which is the imaging condition of the first camera when performing the imaging processing using the first camera 21 .
- the first imaging condition includes an exposure time (shutter speed), an aperture value (F value), a focus distance, an ISO sensitivity, or the like.
- the first imaging condition setting portion 321 automatically sets the first imaging condition based on the distance and brightness (light receiving amount) from the first light receiving portion 212 to the imaging target object. Further, the first imaging condition can be appropriately changed by the operation of the user, and for example, when the user inputs the first imaging condition from the touch panel 11 , the first imaging condition setting portion 321 sets the input first imaging condition.
- the second imaging condition setting portion 322 sets the second imaging condition which is the imaging condition of the second camera 22 when performing the imaging processing using the second camera 22 .
- the second imaging condition includes the exposure time, the aperture value, the focus distance, the ISO sensitivity, or the like, as in the first imaging condition.
- the second imaging condition setting portion 322 sets the second imaging condition based on the first imaging condition set by the first imaging condition setting portion 321 . That is, the second imaging condition setting portion 322 corrects the first imaging condition by using the correction coefficient ⁇ stored in the storage 31 to obtain the second imaging condition. More detailed processing of the second imaging condition setting portion 322 will be described later.
- the first imaging control portion 323 controls the first camera 21 based on the first imaging condition, and causes the first camera 21 to image a color image (RGB image).
- the second imaging control portion 324 controls the second camera 22 based on the second imaging condition, and causes the second camera 22 to image a spectroscopic image for a plurality of spectral wavelengths.
- the image evaluation portion 325 evaluates whether the spectroscopic image of each wavelength captured by the second camera 22 is an appropriate spectroscopic image. For example, the image evaluation portion 325 determines whether each spectroscopic image causes halation, in other words, whether each imaging pixel of the second light receiving portion 222 receives light exceeding the detectable light amount. Specifically, in the spectroscopic image, it is determined whether the gradation value is not the maximum value in a predetermined number or more (for example, one or more) of pixels.
- the image evaluation portion 325 may determine whether the gradation value of each pixel is within a predetermined reference range in the spectroscopic image of each wavelength when a predetermined test target is imaged.
- the test target may include, for example, a white reference plate or the like having a reflectance of 99% or more with respect to light of each wavelength.
- the image evaluation portion 325 can evaluate not only the presence or absence of the halation but also whether the S/N ratio of the spectroscopic image is appropriate.
- the correction coefficient updating portion 326 updates the correction coefficient when setting the second imaging condition based on the evaluation result of the image evaluation portion 325 . That is, in the evaluation result by the image evaluation portion 325 , the correction coefficient is updated so that halation does not occur in the spectroscopic image.
- imaging processing in a plurality of imaging modes can be performed such as imaging processing of the color image using the first camera 21 , imaging processing of the spectroscopic image using the second camera 22 , and imaging processing of the color image and the spectroscopic image using both the first camera 21 and the second camera 22 .
- the spectroscopic image can be superimposed on the color image and displayed on the touch panel 11 , or the result from performing analysis processing based on the spectroscopic image (for example, component analysis or the like) can be superimposed on the color image and displayed on the touch panel 11 .
- imaging processing of the color image using the first camera 21 is the same as the imaging processing in related art by a general smartphone or the like.
- the control portion 30 activates the first camera 21 . Further, the first imaging condition setting portion 321 automatically sets the first imaging condition such as the exposure time, the aperture value, the focus distance, and the ISO sensitivity based on the light receiving amount of light received by the first light receiving portion 212 and the distance to the target object. When it is input to change the first imaging condition by the input operation of the user, the first imaging condition set and input by the user may be acquired.
- the imaging command for commanding imaging is input by the input operation of the user, so that the first imaging control portion 323 controls the first camera 21 to capture a color image.
- the electronic device 1 of the present embodiment sets the second imaging condition based on the first imaging condition.
- the imaging processing of the spectroscopic image will be described in detail.
- FIG. 3 is a flowchart illustrating an imaging method of a spectroscopic image of the present embodiment.
- control portion 30 activates the first camera 21 and the second camera 22 (step S 1 ).
- the first imaging condition setting portion 321 sets the first imaging condition of the first camera 21 (step S 2 ).
- This step S 2 is the same as the setting of the first imaging condition performed in the imaging processing of the color image by the first camera 21 described above. That is, the first imaging condition setting portion 321 automatically sets the first imaging condition such as the exposure time, the aperture value, the focus distance, and the ISO sensitivity based on the light receiving amount of light received by the first light receiving portion 212 and the distance to the target object, or sets the first imaging condition input by the user.
- the first imaging control portion 323 images a color image of the target object by the first camera 21 based on the first imaging condition and displays the color image on the touch panel 11 (step S 3 ). That is, the real-time image of the target object is displayed on the touch panel 11 . Accordingly, the user can adjust the imaging position of the spectroscopic image while checking the target object which is the subject.
- the second imaging condition setting portion 322 sets the second imaging condition based on the first imaging condition (step S 4 ).
- the correction coefficient ⁇ is recorded in the storage 31 .
- the correction coefficient ⁇ includes an exposure correction coefficient ⁇ t for the exposure time, an aperture correction coefficient ⁇ a for the aperture value, and an ISO correction coefficient ⁇ i for the ISO sensitivity.
- each of the initial value of the correction coefficient ⁇ and the update value are stored, and in the step S 4 , which is initially executed after the application is executed in the step S 1 , the initial value of the correction coefficient ⁇ is used, and in the second and subsequent steps S 4 , the update value of the correction coefficient ⁇ is used. That is, in the present embodiment, the correction coefficient ⁇ is sequentially updated by the correction coefficient updating portion 326 , so that the second imaging condition is optimized every time a spectroscopic image is captured.
- the initial value of the correction coefficient ⁇ is set based on the light receiving sensitivity characteristic of the second light receiving portion 222 and the spectral characteristic of the spectroscopic element.
- the electronic device 1 includes a light source portion that irradiates a target object with illumination light, it is set based on the light receiving sensitivity characteristic of the second light receiving portion 222 , the spectral characteristic of the spectroscopic element 221 , and the light source characteristic of the light source portion.
- FIG. 4 is a diagram illustrating the light receiving sensitivity characteristic, the spectral characteristic, the light source characteristic, and the characteristic of the second camera (hereinafter referred to as a camera optical characteristic) obtained by multiplying these previous characteristics.
- the light receiving sensitivity characteristic of the second light receiving portion 222 is a characteristic indicating the sensitivity of the second light receiving portion to each wavelength.
- the spectral characteristic of the spectroscopic element 221 is a characteristic indicating the transmittance of light of each wavelength when a predetermined spectral wavelength is transmitted by the spectroscopic element 221 .
- the spectroscopic element 221 can change the wavelength of the transmitted light within a predetermined wavelength range, and the transmittance characteristic when the spectroscopic element 221 transmits the light of a predetermined spectral wavelength, exhibits that the transmittance peaks at the corresponding spectral wavelength as illustrated in a line P of FIG. 4 .
- the light source characteristic is a characteristic indicating the light amount (light intensity) of each wavelength of the illumination light.
- the camera characteristic value is a characteristic obtained by multiplying these light receiving sensitivity characteristic, spectral characteristic, and light source characteristic.
- the initial value of the correction coefficient ⁇ is set based on the maximum characteristic value in the camera optical characteristic as described above. That is, when a spectroscopic image of a wavelength ⁇ M having the maximum characteristic value is captured by the second camera 22 , a correction coefficient ⁇ in which the gradation value of each pixel is less than the maximum gradation value, is set. Accordingly, it is possible to image a spectroscopic image in which the occurrence of halation is suppressed for any spectral wavelength.
- the second imaging control portion 324 controls the second camera 22 to capture a spectroscopic image (step S 5 ).
- the first camera 21 may be controlled to simultaneously image a color image.
- the second imaging control portion 324 sets the wavelength of the light transmitted from the spectroscopic element 221 to the designated spectral wavelength, and adjusts the second aperture mechanism 223 A based on the aperture value A 2 of the second imaging condition.
- the second imaging control portion 324 detects the light receiving amount of light in the second light receiving portion 222 at the exposure time T 2 of the second imaging condition, and amplifies a light receiving signal according to the light receiving amount output from each imaging element based on the ISO sensitivity I 2 of the second imaging condition. Accordingly, a spectroscopic image based on the second imaging condition is captured.
- the wavelength of the light transmitted from the spectroscopic element are sequentially switched, and imaging processing of the spectroscopic image may be performed by the second camera 22 according to the second imaging condition in the same manner as above.
- the image evaluation portion 325 determines whether the spectroscopic image captured in the step S 5 is an appropriate image (step S 6 ). That is, the image evaluation portion 325 determines whether the gradation value of each pixel in the spectroscopic image for each spectral wavelength is less than the preset maximum gradation value.
- the second imaging control portion 324 determines whether to perform re-imaging by the second camera 22 (step S 7 ). For example, when re-imaging is instructed by the input operation of a user, YES may be determined in the step S 7 . Further, when the re-imaging try setting number is set in advance and the number of times continuously determined as NO in the step S 6 becomes the re-imaging try setting number which has been set, it may be determined as NO in the step S 7 , and when the number of times continuously determined as NO is less than the re-imaging try setting number, it may be determined as YES in the step S 7 .
- the correction coefficient updating portion 326 updates the update value of the correction coefficient ⁇ stored in the storage 31 (step S 8 ). After the step S 4 is executed for the first time, the correction coefficient updating portion 326 calculates a new correction coefficient ⁇ from the initial value of the correction coefficient ⁇ to record the new correction coefficient as an update value. After the step S 4 is executed for the second time, a new correction coefficient ⁇ is calculated from the update value of the previous correction coefficient ⁇ to update the update value.
- the correction coefficient updating portion 326 may calculate the update value by subtracting a preset value from the correction coefficient ⁇ used in the step S 4 and may calculate the update value by multiplying a predetermined value less than 1. Further, the correction coefficient updating portion 326 may change the amount of decrease in the correction coefficient ⁇ according to the number of pixels causing halation.
- the correction coefficient updating portion 326 may update only one or two of the exposure correction coefficient ⁇ t, the aperture correction coefficient ⁇ a, and the ISO correction coefficient ⁇ i. For example, only the exposure correction coefficient ⁇ t may be updated, or the exposure correction coefficient ⁇ t and the aperture correction coefficient ⁇ a may be updated.
- the processing returns to the step S 4 , the second imaging condition setting portion 322 resets the second imaging condition based on the updated correction coefficient ⁇ , and the second imaging control portion 324 causes the second camera 22 to capture a spectroscopic image based on the second imaging condition.
- the electronic device 1 displays the spectroscopic image captured by the second camera 22 on the touch panel 11 (step S 9 ).
- the display method of the spectroscopic image is performed based on the imaging mode selected by the user, and for example, only the spectroscopic image may be displayed on the touch panel 11 , or the spectroscopic image may be superimposed on the color image and displayed. Alternatively, the analysis result analyzed based on the spectroscopic image may be superimposed on the color image and displayed.
- the electronic device 1 of the present embodiment includes an imaging apparatus constituted with the first camera 21 , the second camera 22 , and the control portion 30 .
- the first camera 21 includes a color filter portion 211 including the R filter which transmits light in the red wavelength region, the G filter which transmits light in the green wavelength region, and the B filter which transmits light in the blue wavelength region, and the first light receiving portion 212 which receives light transmitted through the color filter portion 211 .
- the first camera 21 captures a color image based on the gradation value of red light transmitted through the R filter, the gradation value of green light transmitted through the G filter, and the gradation value of blue light transmitted through the B filter.
- the second camera 22 includes the spectroscopic element 221 capable of dispersing light having a predetermined spectral wavelength from the incident light and changing the spectral wavelength to four or more wavelengths, and the second light receiving portion 222 that receives the light dispersed by the spectroscopic element 221 .
- the second camera 22 captures a spectroscopic image of each wavelength dispersed by the spectroscopic element 221 .
- the control portion 30 includes the storage 31 and the processor 32 , and the processor 32 reads and executes a program stored in the storage 31 to function as the first imaging condition setting portion 321 , the second imaging condition setting portion 322 , or the like.
- the first imaging condition setting portion 321 sets the first imaging condition, which is the imaging condition of the first camera 21 .
- the second imaging condition setting portion 322 sets the second imaging condition, which is the imaging condition of the second camera 22 , based on the first imaging condition.
- the user does not need to set the second imaging condition by himself/herself with respect to the second camera 22 which captures the spectroscopic image, and the user is not required to have specialized knowledge, either. Accordingly, it is possible to improve the convenience when using the second camera 22 by the electronic device 1 . Further, since the second imaging condition is set based on the first imaging condition, it is not necessary to perform a preliminary operation for capturing a spectroscopic image a plurality of times to set the second imaging condition, or the like, and it is possible to quickly set the second imaging condition.
- the second imaging condition setting portion 322 sets at least the exposure time as the second imaging condition.
- the second light receiving portion 222 When the second light receiving portion 222 receives light and the light receiving amount exceeds the permissible value, halation occurs in the spectroscopic image, and an appropriate spectroscopic image cannot be captured.
- the exposure time is most related to the occurrence of such halation, and in the present embodiment, the image abnormality due to halation can be effectively suppressed by appropriately setting the second imaging condition including the exposure time T 2 .
- the second imaging condition setting portion 322 sets the second imaging condition by multiplying the first imaging condition by the correction coefficient ⁇ .
- the second imaging condition setting portion 322 can easily set the second imaging condition by simple processing of only multiplying the first imaging condition by the correction coefficient ⁇ , and the time required to image the spectroscopic image can be shortened.
- the correction coefficient ⁇ is set based on the camera optical characteristic obtained by multiplying a spectral characteristic indicating the transmittance of the spectroscopic element 221 for each spectral wavelength and a light receiving sensitivity characteristic indicating the light receiving sensitivity of the second light receiving portion 222 for each wavelength.
- the correction coefficient ⁇ is set based on the maximum value of the characteristic value of the camera optical characteristic as described above.
- the wavelength at which the specific value becomes the maximum value is a wavelength at which halation is likely to occur in the captured spectroscopic image. Accordingly, when a correction coefficient is set so that halation does not occur at the corresponding wavelength, the occurrence of halation can be suppressed for each of the other spectral wavelengths.
- the electronic device 1 of the present embodiment further includes the image evaluation portion 325 which evaluates a spectroscopic image obtained by imaging a target object with the second camera 22 under the second imaging condition, the storage 31 which records the correction coefficient ⁇ , and the correction coefficient updating portion which updates the update value of the correction coefficient ⁇ stored in the storage 31 based on the evaluation result by the image evaluation portion 325 .
- the correction coefficient updating portion 326 updates the correction coefficient ⁇ , so that it is possible to set the correction coefficient ⁇ capable of imaging an appropriate spectroscopic image. Accordingly, it is possible to capture an appropriate spectroscopic image in which halation or the like is not seen.
- the initial correction coefficient ⁇ is set based on the wavelength of the maximum characteristic value in the camera optical characteristic of the second camera.
- the second embodiment differs from the first embodiment in that it is set based on the minimum characteristic value.
- the electronic device 1 of the second embodiment has the same configuration as the configuration of the first embodiment, and as illustrated in FIGS. 1 and 2 , includes the first camera 21 , the second camera 22 , the control portion 30 , or the like.
- the electronic device 1 of the second embodiment captures a spectroscopic image
- the spectroscopic image is captured by the imaging method as illustrated in FIG. 3 , substantially the same as the method of the first embodiment.
- the method of setting the initial correction coefficient ⁇ which is used by the second imaging condition setting portion 322 when setting the second imaging condition is different from the method of the first embodiment.
- the initial value of the correction coefficient ⁇ is set based on the minimum value of the characteristic value among the camera optical characteristics of the second camera as illustrated in FIG. 4 .
- the correction coefficient ⁇ is set so that the gradation value of each pixel of the spectroscopic image becomes equal to or higher than a preset threshold value. Accordingly, deterioration of the S/N ratio of each pixel of the spectroscopic image can be suppressed.
- the image evaluation portion 325 evaluates whether a clear image is captured as a spectroscopic image. For example, the image evaluation portion 325 determines whether the maximum gradation value among the gradation values of each pixel of the spectroscopic image is equal to or greater than the threshold value.
- the correction coefficient updating portion 326 updates the correction coefficient ⁇ .
- the correction coefficient updating portion 326 may calculate the update value of the correction coefficient ⁇ used in the step S 4 by adding a preset value or multiplying the correction coefficient by a predetermined value greater than 1.
- the correction coefficient ⁇ is set based on the minimum value of the characteristic value of the camera optical characteristic.
- the camera optical characteristic it is possible to specify the spectral wavelength at which the light receiving amount of the second light receiving portion 222 tends to decrease when a spectroscopic image is captured. That is, in the camera optical characteristic, the wavelength at which the specific value becomes the minimum is the wavelength at which the gradation value of each pixel of the captured spectroscopic image is low and the image tends to be unclear (the S/N ratio tends to deteriorate). Accordingly, by setting a correction coefficient which suppresses the deterioration of the S/N ratio at the corresponding wavelength, the deterioration of the S/N ratio can be suppressed for each of the other spectral wavelengths.
- the third embodiment is different from the first embodiment in that the second imaging condition is set independently for each spectral wavelength dispersed by the spectroscopic element 221 .
- FIG. 5 is a diagram illustrating an overview of the correction coefficient stored in the storage 31 in the third embodiment.
- each of correction coefficients ⁇ _ ⁇ (an exposure correction coefficient ⁇ t_ ⁇ , an aperture correction coefficient ⁇ a_ ⁇ , and an ISO correction coefficient ⁇ i_ ⁇ ) is associated with each spectral wavelength which can be dispersed by the spectroscopic element 221 , and is stored in the storage 31 .
- an exposure correction coefficient ⁇ t_ 400 , an aperture correction coefficient ⁇ a_ 400 , and an ISO correction coefficient ⁇ i_ 400 are associated with a wavelength of 400 nm
- an exposure correction coefficient ⁇ t_ 700 , an aperture correction coefficient ⁇ a_ 700 , and an ISO correction coefficient ⁇ i_ 700 are associated with a wavelength of 700 nm.
- correction coefficients ⁇ _ ⁇ are set based on the camera optical characteristic of each wavelength.
- the initial value of the correction coefficient ⁇ _ ⁇ is set such that when a spectroscopic image of a spectral wavelength ⁇ is captured, the gradation value of each pixel of the spectroscopic image is set to be less than an upper limit value and greater than a predetermined threshold value.
- the second imaging condition setting portion 322 sets each of the second imaging conditions for the spectral wavelengths of the spectroscopic image to be captured. For example, when the spectral wavelength of the spectroscopic image to be captured is ⁇ a, the second imaging condition setting portion 322 reads a correction coefficient ⁇ _ ⁇ a corresponding to the wavelength ⁇ a, that is, each of an exposure correction coefficient ⁇ t_ ⁇ a , an aperture correction coefficient ⁇ a_ ⁇ a , and an ISO correction coefficient ⁇ i_ ⁇ a .
- the second imaging condition setting portion 322 sets each of the second imaging conditions for each spectral wavelength.
- the image evaluation portion 325 determines whether each spectroscopic image is appropriate, and for a spectral wavelength determined to be inappropriate, the update of the correction coefficient ⁇ is performed by the correction coefficient updating portion 326 in the step S 8 .
- the correction coefficient updating portion 326 calculates an update value obtained by reducing a correction coefficient ⁇ _ 400 by a predetermined correction amount, calculates an update value obtained by increasing a correction coefficient ⁇ _ 550 by a predetermined correction amount, and does not update a correction coefficient ⁇ _ 700 .
- the second imaging condition setting portion 322 sets the second imaging condition for every spectral wavelength based on each of the correction coefficients ⁇ _ ⁇ s which is set for each spectral wavelength which can be changed by the spectroscopic element 221 .
- a different correction coefficient is set for every spectral wavelength, and a second imaging condition for every spectral wavelength is set based on the corresponding correction coefficient. Accordingly, the probability that an abnormality such as halation will occur in the spectroscopic image is reduced, and an appropriate spectroscopic image can be captured.
- the correction coefficient updating portion 326 updates the correction coefficient stored in the storage 31 based on the evaluation result of the spectroscopic image by the image evaluation portion 325 .
- the fourth embodiment is different from the first embodiment in that a model for calculating the correction coefficient is generated based on the evaluation result of the spectroscopic image by the image evaluation portion 325 .
- FIG. 6 is a block diagram illustrating a schematic configuration of an electronic device 1 A according to the present embodiment.
- the processor 32 of the present embodiment reads and executes the program stored in the storage 31 , thereby, as illustrated in FIG. 6 , functioning as the first imaging condition setting portion 321 , a second imaging condition setting portion 322 A, the first imaging control portion 323 , the second imaging control portion 324 , the image evaluation portion 325 , the correction coefficient updating portion 326 , and a model generation portion 327 .
- the second imaging condition is set based on the first imaging condition and the correction coefficient
- the image evaluation portion 325 evaluates whether the spectroscopic image captured under the second imaging condition is appropriate, that is, whether an abnormality such as halation has occurred.
- the model generation portion 327 uses the evaluation result by the image evaluation portion 325 , the first imaging condition, and the correction coefficient ⁇ to generate a correction coefficient output model by machine learning in which the first imaging condition is used as an input and the correction coefficient ⁇ is used as an output.
- the first imaging condition and the correction coefficient ⁇ used at the time of model generation are the first imaging condition and the correction coefficient ⁇ used for setting the second imaging condition when the spectroscopic image to be evaluated is captured.
- machine learning for example, linear regression, least squares method, various algorithms such as k-NN, or deep learning can be used.
- the model generation portion 327 may further generate a correction coefficient calculation model by using the spectral wavelength of the spectroscopic image as teacher data.
- the model generation portion 327 can generate a correction coefficient output model in which the first imaging condition and the spectral wavelength ⁇ of the spectroscopic image to be captured are used as input and the correction coefficient ⁇ _ ⁇ for the corresponding spectral wavelength ⁇ is used as an output.
- the model generation portion 327 may generate a correction coefficient calculation model by adding the color image captured by the first camera 21 as teacher data.
- a correction coefficient output model can be generated using statistics data such as the average gradation value, the maximum gradation value, and the minimum gradation value of each color component of R (red), G (green), and B (blue) in the color image, as teacher data, and a highly accurate correction coefficient can be calculated.
- FIG. 7 is a flowchart illustrating an imaging method of the present embodiment.
- a spectroscopic image is captured by a method substantially similar to the method of the first embodiment.
- the user operates the touch panel 11 or the like to select an application for capturing a spectroscopic image, so that the processing of the step S 1 is performed, and the control portion 30 activates the first camera 21 and the second camera 22 .
- the processing of the step S 2 is performed, so that the first imaging condition setting portion 321 sets the first imaging condition of the first camera 21 , and in the step S 3 , the first imaging control portion 323 causes the color image captured by the first camera 21 to be displayed on the touch panel 11 .
- the second imaging condition setting portion 322 A inputs the acquired first imaging condition to the correction coefficient output model generated by the model generation portion 327 , and sets the second imaging condition based on the output correction coefficient ⁇ (step S 4 A).
- the method of calculating the second imaging condition is the same as the method of the first embodiment or the like, and for example, the second imaging condition is calculated by multiplying the first imaging condition by the correction coefficient ⁇ .
- each correction coefficient ⁇ _ ⁇ corresponding to a plurality of spectral wavelengths may be output.
- the second imaging control portion 324 controls the second camera 22 under the set second imaging condition to image a spectroscopic image, and the step S 6 is performed so that the image evaluation portion 325 evaluates whether the captured spectroscopic image is an appropriate image.
- the correction coefficient updating portion 326 updates the correction coefficient ⁇ output from the correction coefficient output model in the step S 4 A, and further, the second imaging condition setting portion 322 A sets the second imaging condition based on the updated correction coefficient ⁇ (step S 8 A).
- the method of updating of the correction coefficient updating portion 326 is the same as the method of the fourth embodiment, and when halation occurs in the spectroscopic image, the correction coefficient ⁇ is reduced by a predetermined amount, and when the spectroscopic image is unclear, the correction coefficient ⁇ is increased by a predetermined amount. After that, the processing returns to the step S 5 .
- step S 6 if it is determined as YES in the step S 6 , the processing of the step S 9 is performed, and the captured spectroscopic image or an image based on the spectroscopic image is displayed on the touch panel 11 .
- the model generation portion 327 accumulates the first imaging condition, the correction coefficient ⁇ , and the evaluation result obtained in a series of processing in the storage 31 (step S 10 ).
- the model generation portion 327 accumulates the historical data which associates with the first imaging condition, the correction coefficient ⁇ obtained in the step S 4 A, and the evaluation result of the spectroscopic image for each spectral wavelength, in the storage 31 .
- the model generation portion 327 further accumulates, in addition to the historical data which associates with the first imaging condition, the correction coefficient ⁇ obtained in step S 4 A, and the evaluation result of the spectroscopic image, the historical data which associates with the first imaging condition, the correction coefficient updated in the step S 8 A, and the evaluation result in the step S 6 for the spectroscopic image captured under the second imaging condition based on the corresponding correction coefficient, in the storage 31 .
- the model generation portion 327 reconstructs and updates the correction coefficient output model based on the accumulated historical data (step S 11 ).
- the step S 11 does not have to be immediately after the spectroscopic image is captured. For example, it may be performed at a fixed cycle every time a spectroscopic image is captured, or it may be performed every time a fixed amount of historical data is accumulated.
- the correction coefficient output from the correction coefficient output model can be made more accurate by updating the correction coefficient output model based on the accumulated historical data, and it is possible to set the second imaging condition of the above more appropriately when a spectroscopic image is captured.
- the processor 32 further functions as the image evaluation portion 325 and the model generation portion 327 .
- the image evaluation portion 325 evaluates a spectroscopic image obtained by imaging a target object with the second camera 22 under the second imaging condition.
- the model generation portion 327 uses the first imaging condition, the correction coefficient and the evaluation result of the image evaluation portion 325 for the spectroscopic image captured under the second imaging condition set by using the correction coefficient ⁇ , as teacher data and generates a correction coefficient output model in which the first imaging condition is used as an input and the correction coefficient ⁇ is used as an output.
- the second imaging condition setting portion 322 A acquires the correction coefficient ⁇ by inputting the first imaging condition into the correction coefficient output model, and sets the second imaging condition based on the correction coefficient ⁇ .
- the model generation portion 327 can generate a correction coefficient output model which outputs an optimum correction coefficient for the first imaging condition by machine learning. Accordingly, the second imaging condition setting portion 322 A can obtain an appropriate correction coefficient ⁇ according to the first imaging condition simply by inputting the first imaging condition into the correction coefficient output model, and can easily calculate the second imaging condition based on the corresponding correction coefficient ⁇ .
- the model generation portion 327 can perform machine learning with the accumulated historical data as teacher data and improve the accuracy of the correction coefficient output model.
- the present disclosure is not limited to the embodiment described above, and a modification, an improvements, or the like within the range in which the object of the present disclosure can be achieved are included in the present disclosure.
- the second imaging condition setting portion 322 sets the second imaging condition by using each of the exposure correction coefficient, the aperture correction coefficient, and the ISO correction coefficient, but the present disclosure is not limited thereto.
- the second imaging condition setting portion 322 may set at least an exposure correction coefficient and set an exposure time based on the exposure correction coefficient.
- the aperture correction coefficient and the ISO correction coefficient may be set to “1”, and the aperture value and the ISO sensitivity of the first imaging condition may be used as they are.
- the spectroscopic image is captured with setting the second imaging condition by using the initial value of the correction coefficient ⁇ set in advance for the first imaging condition, and when the image evaluation portion determines that the spectroscopic image is not appropriate, control is performed to update the correction coefficient ⁇ so that the appropriate spectroscopic image is captured.
- the second imaging condition setting portion 322 may change the initial value of the correction coefficient ⁇ to be initially used based on the first imaging condition and the color image captured by the first camera 21 . For example, when the color image captured by the first camera is dark and the gradation value of each pixel is low, large values are set for the exposure time, the aperture value, the focus distance, and the ISO sensitivity, which are the first imaging conditions. In this case, the correction coefficient ⁇ may be set to “1”, and the second imaging condition may be the same as the first imaging condition to image the spectroscopic image by the second camera 22 .
- the second imaging condition setting portion 322 may display the second imaging condition on the touch panel 11 before imaging the spectroscopic image by the second camera 22 . Further, the second imaging condition setting portion 322 may accept the instruction of the user to change the second imaging condition and change the set second imaging condition or correction coefficient according to the instruction of the user.
- the user can confirm the second imaging condition before imaging the spectroscopic image, and further, can finely adjust the second imaging condition and the correction coefficient. Accordingly, the spectroscopic image can be captured under the condition desired by the user.
- the model generation portion 327 may accumulate the correction coefficient set by the user and the first imaging condition when the corresponding correction coefficient is set as historical data. Accordingly, the condition according to the preference of the user can be learned, and the correction coefficient output model optimized for every user can be generated.
- the second imaging condition setting portion 322 calculates the second imaging condition by multiplying the correction coefficient ⁇ by the first imaging condition, but the present disclosure is not limited thereto.
- the second imaging condition setting portion 322 may calculate the second imaging condition by adding or subtracting a correction coefficient with respect to the first imaging condition.
- the model generation portion 327 generates a correction coefficient output model in which the first imaging condition is used as an input and the correction coefficient ⁇ is used as an output, but the present disclosure is not limited thereto.
- a correction coefficient output model may be generated in which the first imaging condition is used as an input and the second imaging condition calculated based on the correction coefficient ⁇ is used as an output.
- the model generation port ion 327 may generate a correction coefficient output model in which the first imaging condition and the color image captured by the first camera 21 are used as input and the second imaging condition is used as an output.
- the output second imaging condition may be the second imaging condition for each spectral wavelength.
- An imaging apparatus includes a first camera which includes a color filter portion including a red color filter that transmits light in a red wavelength region, a green color filter that transmits light in a green wavelength region, and a blue color filter that transmits light in a blue wavelength region, and a first light receiving portion receiving light transmitted through the color filter portion, and which captures a color image based on a gradation value of red light transmitted through the red color filter, a gradation value of green light transmitted through the green color filter, and a gradation value of blue light transmitted through the blue color filter, a second camera which includes a spectroscopic element configured to disperse light having a predetermined spectral wavelength from incident light and to change the spectral wavelength to four or more wavelengths, and a second light receiving portion receiving the light dispersed by the spectroscopic element, and which captures a spectroscopic image of each wavelength dispersed by the spectroscopic element, a first imaging condition setting portion which sets an imaging condition of the first camera
- the user since the user does not need to set the imaging condition with respect to the second camera which captures the spectroscopic image and the user is not required to have specialized knowledge, either, the convenience when using the second camera is improved. Further, since the second imaging condition is set based on the first imaging condition, it is not necessary to perform a preliminary operation for capturing a spectroscopic image a plurality of times to set the second imaging condition, and it is possible to quickly set the second imaging condition.
- the second imaging condition setting portion sets at least an exposure time as the second imaging condition.
- the second light receiving portion receives light and the light receiving amount exceeds the permissible value, halation occurs in the spectroscopic image, and an appropriate spectroscopic image cannot be captured.
- the exposure time is most related to the occurrence of such halation, and as in the present aspect, the image abnormality due to halation can be effectively suppressed by appropriately setting the exposure time as the second imaging condition.
- the second imaging condition setting portion sets the second imaging condition by multiplying the first imaging condition by a predetermined correction coefficient.
- the second imaging condition setting portion can easily set the second imaging condition by only multiplying the first imaging condition by the correction coefficient, and the time required to image the spectroscopic image can be shortened.
- the second imaging condition setting portion sets the second imaging condition for the spectral wavelengths based on each of the correction coefficients set for each of the spectral wavelengths that can be changed by the spectroscopic element.
- a different correction coefficient is set for every spectral wavelength, and a second imaging condition for every spectral wavelength is set based on the corresponding correction coefficient. Accordingly, the probability that an abnormality such as halation will occur in the spectroscopic image is reduced, and an appropriate spectroscopic image can be captured.
- the correction coefficient is set based on a camera optical characteristic obtained by multiplying a spectral characteristic indicating a transmittance of the spectroscopic element for each wavelength and a light receiving sensitivity characteristic indicating light receiving sensitivity of the second light receiving portion for each wavelength.
- the correction coefficient is set based on a maximum value of a characteristic value of the camera optical characteristic.
- the wavelength at which the specific value becomes the maximum value is a wavelength at which halation is likely to occur in the captured spectroscopic image, and the occurrence of halation can be suppressed for each of the other spectral wavelengths by setting a correction coefficient which can calculate the second imaging condition under which halation does not occur at the corresponding wavelength.
- the correction coefficient may be set based on a minimum value of a characteristic value of the camera optical characteristic.
- the wavelength at which the specific value becomes the minimum value is the wavelength at which the gradation value of each pixel of the captured spectroscopic image is low and the image tends to be unclear (the S/N ratio tends to deteriorate). Accordingly, by setting a correction coefficient which suppresses the deterioration of the S/N ratio at the corresponding wavelength, the deterioration of the S/N ratio can be suppressed for each of the other spectral wavelengths.
- an image evaluation portion which evaluates the spectroscopic image obtained by imaging a target object with the second camera under the second imaging condition
- a correction coefficient recording portion which stores the correction coefficient
- a correction coefficient updating portion which updates the correction coefficient stored in the correction coefficient recording portion based on an evaluation result by the image evaluation portion.
- the spectroscopic image captured by the second camera is evaluated by the image evaluation portion, and when the evaluation result determines that the spectroscopic image is not appropriate, the correction coefficient updating portion updates the correction coefficient. Accordingly, it is possible to set a correction coefficient capable of imaging an appropriate spectroscopic image by repeating the imaging processing of the spectroscopic image a plurality of times and the evaluation of the spectroscopic image by the image evaluation portion.
- an image evaluation portion which evaluates the spectroscopic image obtained by imaging a target object with the second camera under the second imaging condition
- a model generation portion which generates a correction coefficient output model in which the first imaging condition, the correction coefficient, and the evaluation result of the image evaluation portion for the spectroscopic image captured under the second imaging condition set by using the correction coefficient, are used as teacher data, and the first imaging condition is used as an input and the correction coefficient is used as an output, in which the second imaging condition setting portion acquires the correction coefficient by inputting the first imaging condition into the correction coefficient output model and sets the second imaging condition based on the correction coefficient.
- the model generation portion can generate a correction coefficient output model which outputs an optimum correction coefficient for the first imaging condition by machine learning. Accordingly, the second imaging condition setting portion can obtain an appropriate correction coefficient ⁇ according to the first imaging condition by inputting the first imaging condition into the correction coefficient output model, and can easily calculate the second imaging condition based on the corresponding correction coefficient ⁇ .
- the model generation portion can perform machine learning with the accumulated historical data as teacher data and improve the accuracy of the correction coefficient output model.
- a portable terminal apparatus of a second aspect of the present disclosure includes the imaging apparatus of the first aspect and a housing in which the imaging apparatus is incorporated.
- the portable terminal apparatus as in the present aspect is, for example, a terminal apparatus such as a smartphone, a tablet terminal, and a head-mounted display, and is required to improve user convenience.
- a terminal apparatus such as a smartphone, a tablet terminal, and a head-mounted display
- the imaging apparatus mentioned above By incorporating the imaging apparatus mentioned above in such a portable terminal apparatus, the convenience of the user can be improved.
- An imaging method of a third aspect of the present disclosure is an imaging method of an imaging apparatus including a first camera which includes a color filter portion including a red color filter that transmits light in a red wavelength region, a green color filter that transmits light in a green wavelength region, and a blue color filter that transmits light in a blue wavelength region, and a first light receiving portion receiving light transmitted through the color filter portion, and which captures a color image based on a gradation value of red light transmitted through the red color filter, a gradation value of green light transmitted through the green color filter, and a gradation value of blue light transmitted through the blue color filter, and a second camera which includes a spectroscopic element configured to disperse light having a predetermined spectral wavelength from incident light and to change the spectral wavelength to four or more wavelengths, and a second light receiving portion receiving the light dispersed by the spectroscopic element, and which captures a spectroscopic image of each wavelength dispersed by the spectroscopic element, including setting
- the user since the user does not need to set the imaging condition with respect to the second camera which captures the spectroscopic image and the user is not required to have specialized knowledge, either, the convenience when using the second camera is improved.
- the second imaging condition is set based on the first imaging condition, it is not necessary to perform a preliminary operation for capturing a spectroscopic image a plurality of times to set the second imaging condition, and it is possible to quickly set the second imaging condition.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Human Computer Interaction (AREA)
- Color Television Image Signal Generators (AREA)
- Studio Devices (AREA)
- Blocking Light For Cameras (AREA)
- Transforming Light Signals Into Electric Signals (AREA)
- Exposure Control For Cameras (AREA)
- Cameras In General (AREA)
Abstract
Description
Claims (8)
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2020092970A JP2021190785A (en) | 2020-05-28 | 2020-05-28 | Imaging device, portable terminal apparatus and imaging method |
JP2020-092970 | 2020-05-28 |
Publications (2)
Publication Number | Publication Date |
---|---|
US20210377510A1 US20210377510A1 (en) | 2021-12-02 |
US11870969B2 true US11870969B2 (en) | 2024-01-09 |
Family
ID=78704426
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US17/329,689 Active 2041-07-18 US11870969B2 (en) | 2020-05-28 | 2021-05-25 | Imaging apparatus, portable terminal apparatus, and imaging method |
Country Status (3)
Country | Link |
---|---|
US (1) | US11870969B2 (en) |
JP (1) | JP2021190785A (en) |
CN (1) | CN113747004B (en) |
Families Citing this family (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
KR102661114B1 (en) * | 2020-11-10 | 2024-04-25 | 삼성전자주식회사 | Camera module test apparatus, camera module test method and image generating device |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20130250123A1 (en) | 2011-11-04 | 2013-09-26 | Qualcomm Incorporated | Multispectral imaging system |
US20140078285A1 (en) * | 2011-05-17 | 2014-03-20 | Olympus Corporation | Imaging apparatus and microscope system having the same |
CN103888681A (en) | 2014-04-18 | 2014-06-25 | 四川华雁信息产业股份有限公司 | Automatic exposure method and device |
US20140340515A1 (en) * | 2013-05-14 | 2014-11-20 | Panasonic Corporation | Image processing method and system |
CN108322670A (en) * | 2018-04-27 | 2018-07-24 | Oppo广东移动通信有限公司 | A kind of control method of multi-camera system, mobile terminal and storage medium |
US20200193597A1 (en) * | 2018-12-14 | 2020-06-18 | Spectral Md, Inc. | Machine learning systems and methods for assessment, healing prediction, and treatment of wounds |
US20210144287A1 (en) * | 2017-08-01 | 2021-05-13 | Sony Corporation | Imaging apparatus, imaging method, and program |
-
2020
- 2020-05-28 JP JP2020092970A patent/JP2021190785A/en active Pending
-
2021
- 2021-05-25 US US17/329,689 patent/US11870969B2/en active Active
- 2021-05-27 CN CN202110584275.4A patent/CN113747004B/en active Active
Patent Citations (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US20140078285A1 (en) * | 2011-05-17 | 2014-03-20 | Olympus Corporation | Imaging apparatus and microscope system having the same |
US20130250123A1 (en) | 2011-11-04 | 2013-09-26 | Qualcomm Incorporated | Multispectral imaging system |
JP2015502058A (en) | 2011-11-04 | 2015-01-19 | クゥアルコム・インコーポレイテッドQualcomm Incorporated | Multispectral imaging system |
US20140340515A1 (en) * | 2013-05-14 | 2014-11-20 | Panasonic Corporation | Image processing method and system |
CN103888681A (en) | 2014-04-18 | 2014-06-25 | 四川华雁信息产业股份有限公司 | Automatic exposure method and device |
US20210144287A1 (en) * | 2017-08-01 | 2021-05-13 | Sony Corporation | Imaging apparatus, imaging method, and program |
CN108322670A (en) * | 2018-04-27 | 2018-07-24 | Oppo广东移动通信有限公司 | A kind of control method of multi-camera system, mobile terminal and storage medium |
US20200193597A1 (en) * | 2018-12-14 | 2020-06-18 | Spectral Md, Inc. | Machine learning systems and methods for assessment, healing prediction, and treatment of wounds |
Non-Patent Citations (1)
Title |
---|
Machine translation of CN 108322670 A. (Year: 2018). * |
Also Published As
Publication number | Publication date |
---|---|
JP2021190785A (en) | 2021-12-13 |
CN113747004A (en) | 2021-12-03 |
CN113747004B (en) | 2023-05-23 |
US20210377510A1 (en) | 2021-12-02 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10419685B2 (en) | Image processing apparatus, image processing method, and computer-readable recording medium | |
US20070177004A1 (en) | Image creating method and imaging device | |
US11297259B2 (en) | Finder apparatus, imaging apparatus, and method of controlling finder apparatus | |
US8773551B2 (en) | Signal processing apparatus, solid-state image capturing apparatus, electronic information device, signal processing method, control program and storage medium | |
US10531058B2 (en) | Method for enhancing color images, and an assembly performing such method | |
US9654668B2 (en) | Image composition apparatus and image composition method | |
KR20020032331A (en) | Apparatus and method for defective pixel concealment of image sensor | |
US11290657B2 (en) | Finder apparatus, imaging apparatus, and method of controlling finder apparatus | |
US11870969B2 (en) | Imaging apparatus, portable terminal apparatus, and imaging method | |
CN113848044B (en) | Method for detecting brightness and chrominance consistency of display screen | |
US11290634B2 (en) | Imaging apparatus, imaging method, and program | |
US9900492B2 (en) | Imaging device and photographing apparatus | |
US20100245590A1 (en) | Camera sensor system self-calibration | |
JP5480672B2 (en) | Imaging apparatus and control method thereof | |
EP2146501B1 (en) | Image processing apparatus, image-capturing apparatus, and image processing program | |
US9906705B2 (en) | Image pickup apparatus | |
US20030077080A1 (en) | Photometric apparatus and method for evaluating backlight | |
US20220114760A1 (en) | Image analysis apparatus, image analysis method, and non-transitory computer-readable storage medium storing image analysis program | |
US20230326163A1 (en) | Imaging apparatus | |
US20220397456A1 (en) | Autofocusing method, spectroscopic camera, and computer program | |
US11245844B2 (en) | Processing apparatus, imaging apparatus, and imaging system | |
JP7410621B2 (en) | Imaging device, its control method, and program | |
US20240015411A1 (en) | Imaging apparatus, imaging method, and storage medium | |
US20240080568A1 (en) | Electronic Device and Method for Dynamically Adjusting Exposure Parameter of Spectral Sensor | |
JP2023087762A (en) | Spectroscopic imaging method, spectroscopic imaging apparatus, and computer program |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
AS | Assignment |
Owner name: SEIKO EPSON CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNOR:NISHIMURA, TERUYUKI;REEL/FRAME:056343/0830 Effective date: 20210319 |
|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NON FINAL ACTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: FINAL REJECTION MAILED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |