US10999481B2 - Image processing apparatus, imaging system, image processing method, and recording medium - Google Patents
Image processing apparatus, imaging system, image processing method, and recording medium Download PDFInfo
- Publication number
- US10999481B2 US10999481B2 US16/681,701 US201916681701A US10999481B2 US 10999481 B2 US10999481 B2 US 10999481B2 US 201916681701 A US201916681701 A US 201916681701A US 10999481 B2 US10999481 B2 US 10999481B2
- Authority
- US
- United States
- Prior art keywords
- image
- fresnel zone
- zone pattern
- image processing
- pattern
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000012545 processing Methods 0.000 title claims abstract description 49
- 238000003384 imaging method Methods 0.000 title claims description 85
- 238000003672 processing method Methods 0.000 title claims description 24
- 230000009466 transformation Effects 0.000 claims abstract description 30
- 238000010586 diagram Methods 0.000 description 27
- 230000006870 function Effects 0.000 description 22
- 238000005516 engineering process Methods 0.000 description 20
- 238000002834 transmittance Methods 0.000 description 15
- 238000003860 storage Methods 0.000 description 14
- 101100391560 Oryza sativa subsp. japonica FZP gene Proteins 0.000 description 12
- 230000014509 gene expression Effects 0.000 description 12
- 230000005540 biological transmission Effects 0.000 description 9
- 230000000903 blocking effect Effects 0.000 description 9
- 230000000052 comparative effect Effects 0.000 description 9
- 238000000034 method Methods 0.000 description 9
- 230000015654 memory Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000008859 change Effects 0.000 description 5
- 238000004519 manufacturing process Methods 0.000 description 5
- 230000003247 decreasing effect Effects 0.000 description 4
- 238000011161 development Methods 0.000 description 4
- 230000003287 optical effect Effects 0.000 description 4
- 230000015572 biosynthetic process Effects 0.000 description 3
- 238000004364 calculation method Methods 0.000 description 3
- 238000005520 cutting process Methods 0.000 description 3
- 230000000694 effects Effects 0.000 description 3
- 238000011156 evaluation Methods 0.000 description 3
- 239000000758 substrate Substances 0.000 description 3
- 238000012937 correction Methods 0.000 description 2
- 230000006866 deterioration Effects 0.000 description 2
- 230000002093 peripheral effect Effects 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 239000004065 semiconductor Substances 0.000 description 2
- 230000002123 temporal effect Effects 0.000 description 2
- 238000006243 chemical reaction Methods 0.000 description 1
- 230000007547 defect Effects 0.000 description 1
- 238000005286 illumination Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 238000005259 measurement Methods 0.000 description 1
- 238000012986 modification Methods 0.000 description 1
- 230000004048 modification Effects 0.000 description 1
- 238000012544 monitoring process Methods 0.000 description 1
- 230000004044 response Effects 0.000 description 1
- 238000004088 simulation Methods 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 238000012546 transfer Methods 0.000 description 1
- 238000013519 translation Methods 0.000 description 1
Images
Classifications
-
- H04N5/2254—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T5/00—Image enhancement or restoration
- G06T5/50—Image enhancement or restoration using two or more images, e.g. averaging or subtraction
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/50—Constructional details
- H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B3/00—Simple or compound lenses
- G02B3/02—Simple or compound lenses with non-spherical faces
- G02B3/08—Simple or compound lenses with non-spherical faces with discontinuous faces, e.g. Fresnel lens
-
- G—PHYSICS
- G02—OPTICS
- G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
- G02B5/00—Optical elements other than lenses
- G02B5/18—Diffraction gratings
- G02B5/1876—Diffractive Fresnel lenses; Zone plates; Kinoforms
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/60—Control of cameras or camera modules
-
- H04N5/232—
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20048—Transform domain processing
- G06T2207/20056—Discrete and fast Fourier transform, [DFT, FFT]
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20212—Image combination
- G06T2207/20221—Image fusion; Image merging
Definitions
- the present invention relates to an image processing apparatus, an imaging system, an image processing method, and a non-transitory recording medium and particularly, to a technology for acquiring an image of a subject without a lens.
- a general type of technology for acquiring an image of a subject is a technology for forming an optical image of a subject using a lens.
- a technology for acquiring the image of the subject without a lens has been developed. For example, in “Development of lensless camera technology enabling easy focus adjustment after motion picture imaging”, [online], Nov.
- a Fresnel zone plate is arranged near an imaging element, and the image of the subject can be acquired without a lens by performing Fourier transformation on a moire fringe generated by superimposing a projected image formed on the imaging element by light from the subject with a projected pattern corresponding to the Fresnel zone plate.
- An effect such as a decrease in size of an apparatus is expected.
- a lensless imaging technology for using the Fresnel zone plate in the mask pattern like in “Development of lensless camera technology enabling easy focus adjustment after motion picture imaging”, [online], Nov. 15, 2016, Hitachi, Ltd., [searched on May 8, 2017], Internet (http://www.hitachi.cojpNew/cnews/month/2016/11/1115.html) and Yusuke Nakamura, Takeshi Shimano, Kazuyuki Tajima, Mayu Sao, and Taku Hoshizawa (Hitachi, Ltd.) “Lensless Light-field Imaging with Fresnel Zone Aperture”, The Institute of Image Information and Television Engineers Technical Report, vol. 40, no. 40, IST2016-51, pp.
- WO2016/203573A an image of a subject is reconstructed by performing Fourier transformation on a moire fringe that is formed by light incident on two grating patterns (Fresnel zone plates) arranged opposite to each other from the subject.
- FIGS. 17A to 17E are diagrams illustrating reconstruction of the image in a case where light of a point light source present at infinity is incident on the Fresnel zone plate from an inclined direction. In this case, a projected image illustrated in FIG. 17A is formed. In a case where the projected image is multiplied with a Fresnel zone pattern illustrated in FIG.
- FIG. 17B a moire fringe in the vertical direction and a noise component having a concentric shape are superimposed in the post-multiplication image as illustrated in FIG. 17C .
- FIG. 17D a signal intensity in an X direction in the image illustrated in FIG. 17C is illustrated in FIG. 17D .
- the signal intensity is subjected to Fourier transformation, two solutions are obtained at positions symmetric about the center as illustrated in FIG. 17E , and it is not possible to specify which solution is the true solution.
- FIGS. 18A to 18D are diagrams illustrating a simulation result of the reconstructed image based on the method of acquiring the projected image by multiplying one Fresnel zone pattern with the Fresnel zone plate used in imaging.
- FIG. 18A illustrates the subject.
- an image illustrated in FIG. 18B is obtained.
- the image obtained by reconstruction is an unclear image in which two subject images overlap.
- the reconstruction result illustrated in FIG. 18D is acquired, and an imaging range is reduced.
- WO2016/203573A overlapping of the images is avoided by cutting the reconstructed image in half and displaying the cut reconstructed image.
- an angle of view imaging range
- the image quality of the reconstructed image may be decreased.
- plates (substrates) corresponding to two patterns are maintained.
- An object of the present invention is to provide an image processing apparatus, an imaging system, an image processing method, and a non-transitory recording medium capable of obtaining a clear image having a wide angle of view without using a lens.
- an image processing apparatus comprises a projected image input unit that inputs a projected image formed by light incident on a Fresnel zone plate from a subject, a complex image generation unit that generates a complex image consisting of an image of a real part and an image of an imaginary part by multiplying the projected image with each of a first Fresnel zone pattern and a second Fresnel zone pattern having the same local spatial frequency in each region and a different phase of the local spatial frequency with respect to the first Fresnel zone pattern, and a Fourier transformation unit that reconstructs an image of a spatial domain by performing two-dimensional complex Fourier transformation on the complex image.
- the image of the spatial domain is reconstructed by performing two-dimensional complex Fourier transformation on the complex image consisting of the image of the real part and the image of the imaginary part which are obtained by multiplying the projected image with each of the first Fresnel zone pattern and the second Fresnel zone pattern having different phases of the local spatial frequencies.
- the position of the subject can be specified from a signal component of a moire fringe by removing a noise component.
- a clear image not having overlapping of subject images can be obtained without a lens.
- the image since it is not necessary to hide a part of the subject, the image can be obtained at a wide angle of view.
- the first and second Fresnel zone patterns may be maintained as electronic data.
- the “Fresnel zone plate” includes a zone plate of which the transmittance of the subject light continuously changes depending on a distance from the center, and a zone plate of which the transmittance non-continuously (transmission or non-transmission) changes by setting a threshold value for the transmittance of the subject light incident on the zone plate.
- the projected image used in reconstruction of the image may be acquired by an imaging unit including the Fresnel zone plate and an image sensor, or the projected image that is separately recorded may be acquired through a non-transitory recording medium or a network.
- the image processing apparatus is such that in the first aspect, the phases of the local spatial frequencies of the first Fresnel zone pattern and the second Fresnel zone pattern are shifted positively or negatively in a range of greater than or equal to 70° and smaller than or equal to 110°.
- the second aspect defines the range of the shift in phase in which a clear image can be reconstructed.
- the image processing apparatus is such that in the first or second aspect, the phase of the local spatial frequency of the first Fresnel zone pattern or the phase of the local spatial frequency of the second Fresnel zone pattern is the same as a phase of a local spatial frequency of the Fresnel zone plate.
- the image processing apparatus is such that in any one of the first to third aspects, the complex image generation unit generates the image of the real part by multiplying the projected image with the first Fresnel zone pattern and generates the image of the imaginary part by multiplying the projected image with the second Fresnel zone pattern.
- the image processing apparatus is such that in any one of the first to fourth aspects, the complex image generation unit generates the complex image by using the first Fresnel zone pattern and the second Fresnel zone pattern that have different enlargement ratios depending on a subject distance in focus.
- the projected image formed by the Fresnel zone plate has a different size depending on the distance of the subject.
- the enlargement ratio of the multiplied Fresnel zone pattern is different depending on the distance of the subject.
- the fifth aspect by generating the complex image using the first Fresnel zone pattern and the second Fresnel zone pattern having different enlargement ratios depending on the subject distance in focus, a less blurred clear image can be obtained.
- the image processing apparatus further comprises, in any one of the first to fifth aspects, an information input unit that inputs information of a local spatial frequency of the Fresnel zone plate, in which the complex image generation unit multiplies the projected image with each of the first Fresnel zone pattern and the second Fresnel zone pattern that have the local spatial frequency based on the input information.
- an information input unit that inputs information of a local spatial frequency of the Fresnel zone plate, in which the complex image generation unit multiplies the projected image with each of the first Fresnel zone pattern and the second Fresnel zone pattern that have the local spatial frequency based on the input information.
- the “information of the local spatial frequency” includes information of a pitch of the Fresnel zone plate.
- the first Fresnel zone pattern and the second Fresnel zone pattern having the same pitch as the input pitch of the Fresnel zone plate can be used.
- an imaging system comprises an imaging unit that includes a Fresnel zone plate on which light from a subject is incident, and an image sensor in which a projected image is formed by the light transmitted through the Fresnel zone plate, and that acquires the projected image from the image sensor, and the image processing apparatus according to any one of the first to sixth aspects, in which the projected image input unit inputs the projected image acquired by the imaging unit.
- the projected image input unit inputs the projected image acquired by the imaging unit.
- an image processing method comprises a step of inputting a projected image formed by light incident on a Fresnel zone plate from a subject, a step of generating a complex image consisting of an image of a real part and an image of an imaginary part by multiplying the projected image with each of a first Fresnel zone pattern and a second Fresnel zone pattern having the same local spatial frequency in each region and a different phase of the local spatial frequency with respect to the first Fresnel zone pattern, and a step of reconstructing an image of a spatial domain by performing two-dimensional complex Fourier transformation on the complex image.
- a clear image can be obtained at a wide angle of view without using a lens in the same manner as the first and seventh aspects.
- the image processing method is such that in the eighth aspect, the phases of the local spatial frequencies of the first Fresnel zone pattern and the second Fresnel zone pattern are shifted positively or negatively in a range of greater than or equal to 70° and smaller than or equal to 110°.
- the ninth aspect defines the range of the shift in phase in which a clear image can be reconstructed.
- the image processing method is such that in the eighth or ninth aspect, the phase of the local spatial frequency of the first Fresnel zone pattern or the phase of the local spatial frequency of the second Fresnel zone pattern is the same as a phase of a local spatial frequency of the Fresnel zone plate.
- the image processing method is such that in any one of the eighth to tenth aspects, in the step of generating the complex image, the image of the real part is generated by multiplying the projected image with the first Fresnel zone pattern, and the image of the imaginary part is generated by multiplying the projected image with the second Fresnel zone pattern.
- the image processing method is such that in any one of the eighth to eleventh aspects, in the step of generating the complex image, the complex image is generated using the first Fresnel zone pattern and the second Fresnel zone pattern that have different enlargement ratios depending on a subject distance in focus. According to the twelfth aspect, a less blurred clear image can be obtained in the same manner as the fifth aspect.
- the image processing method further comprises, in any one of the eighth to twelfth aspects, a step of acquiring the projected image from an image sensor by the Fresnel zone plate on which the light from the subject is incident, and the image sensor in which the projected image is formed by the light transmitted through the Fresnel zone plate, in which in the step of inputting the projected image, the acquired projected image is input.
- the image processing method further comprises, in any one of the eighth to thirteenth aspects, a step of inputting information of a local spatial frequency of the Fresnel zone plate, in which in the step of generating the complex image, the projected image is multiplied with each of the first Fresnel zone pattern and the second Fresnel zone pattern that have the local spatial frequency based on the input information.
- the reconstruction of the image can be quickly and easily performed in the same manner as the sixth aspect.
- a non-transitory recording medium is a non-transitory recording medium on which a computer-readable code of an image processing program is recorded.
- the image processing program causes a computer to implement a function of inputting a projected image formed by light incident on a Fresnel zone plate from a subject, a function of generating a complex image consisting of an image of a real part and an image of an imaginary part by multiplying the projected image with each of a first Fresnel zone pattern and a second Fresnel zone pattern having the same local spatial frequency in each region and a different phase of the local spatial frequency with respect to the first Fresnel zone pattern, and a function of reconstructing an image of a spatial domain by performing two-dimensional complex Fourier transformation on the complex image.
- a clear image can be obtained at a wide angle of view without using a lens in the same manner as the first and eighth aspects.
- the image processing program recorded on the non-transitory recording medium in the fifteenth aspect may be a program that further causes the computer to implement the same features (functions) as the image processing method according to the ninth to fourteenth aspects.
- a clear image can be obtained at a wide angle of view without using a lens.
- FIGS. 1A to 1I are diagrams illustrating reconstruction of an image in an embodiment of the present invention.
- FIGS. 2A to 2H are diagrams illustrating the reconstruction of the image in the embodiment of the present invention.
- FIG. 3 is a diagram illustrating translation of a projected image caused by incidence of inclined light.
- FIG. 4 is a block diagram illustrating a configuration of an imaging system in a first embodiment.
- FIG. 5 is a diagram illustrating a configuration of an image processing unit.
- FIG. 6 is a diagram illustrating images and information stored in a storage unit.
- FIGS. 7A and 7B are diagrams illustrating examples of a Fresnel zone plate.
- FIGS. 8A and 8B are diagrams illustrating examples of Fresnel zone patterns having different phases.
- FIG. 9 is a flowchart illustrating an image processing method according to the first embodiment.
- FIG. 10 is a diagram illustrating enlargement or reduction of the Fresnel zone pattern depending on a focal distance.
- FIG. 11 is a diagram illustrating conditions of examples.
- FIG. 12 is a diagram illustrating examples of Fresnel zone patterns having different phases.
- FIGS. 13A to 13G are diagrams illustrating results of examples and comparative examples.
- FIGS. 14A to 14G are other diagrams illustrating the results of the examples and the comparative examples.
- FIGS. 15A to 15G are still other diagrams illustrating the results of the examples and the comparative examples.
- FIGS. 16A to 16E are still other diagrams illustrating the results of the examples and the comparative examples.
- FIGS. 17A to 17E are diagrams illustrating reconstruction of an image in the technology of the related art.
- FIGS. 18A to 18D are other diagrams illustrating the reconstruction of the image in the technology of the related art.
- FIGS. 1A to 1I are diagrams for describing a summary of image processing in the embodiment of the present invention. For simplification, a case of imaging a point light source at infinite distance in one dimension and reconstructing an image will be described.
- FIG. 1A is a projected image formed in an image sensor by light incident on a Fresnel zone plate from a subject. The projected image is shifted depending on the direction of the point light source.
- An image obtained by multiplying the projected image with a first Fresnel zone pattern (the phase of the local spatial frequency at the center is 0°) illustrated in FIG. 1B is the image illustrated in FIG. 1C .
- This image will be referred to as an image of a real part.
- a signal intensity of the image of the real part in an X direction is illustrated in FIG. 1F .
- FIG. 1E an image obtained by multiplying the projected image illustrated in FIG. 1A with a second Fresnel zone pattern (Fresnel zone pattern that has the same local spatial frequency in each region as the first Fresnel zone pattern and has a shift of 90° in phase of the local spatial frequency with the first Fresnel zone pattern) illustrated in FIG. 1D is the image illustrated in FIG. 1E .
- This image will be referred to as an image of an imaginary part.
- FIG. 1G is an example of the signal intensity of the image illustrated in FIG. 1E in the X direction.
- the local spatial frequency of the Fresnel zone pattern corresponds to a pattern having a shape of streaks configured with transmission regions and light blocking regions. The detail of the shape of streaks is referred to as a pitch.
- a complex image is configured with the image of the real part illustrated in FIG. 1C and the image of the imaginary part illustrated in FIG. 1E .
- a graph plotted in a Y direction denoting the signal intensity of the image of the real part illustrated in FIG. 1F and a Z direction denoting the signal intensity of the image of the imaginary part illustrated in FIG. 1G with respect to the X direction is illustrated in FIG. 1H .
- a signal having one peak is obtained as illustrated in FIG. 1I . This signal corresponds to an image of a spatial domain, and the position of the peak corresponds to the position of the point light source which is the subject.
- FIG. 2A illustrates a text type subject.
- FIG. 2B is a projected image formed in the image sensor by light incident on the Fresnel zone plate from the subject.
- An image obtained by multiplying the projected image with a first Fresnel zone pattern (the phase of the local spatial frequency at the center is 0°) illustrated in FIG. 2C is the image illustrated in FIG. 2D .
- This image will be referred to as an image of a real part.
- FIG. 2E with a second Fresnel zone pattern (Fresnel zone pattern that has the same local spatial frequency in each region as the first Fresnel zone pattern and has a shift of 90° in phase of the local spatial frequency with the first Fresnel zone pattern) illustrated in FIG. 2E is the image illustrated in FIG. 2F .
- This image will be referred to as an image of an imaginary part.
- a graph plotted in the Y direction denoting the signal intensity of the image of the real part illustrated in FIG. 2D and the Z direction denoting the signal intensity of the image of the imaginary part illustrated in FIG. 2F with respect to the X direction is illustrated in FIG. 2G
- the image of the subject is reconstructed as illustrated in FIG. 2H .
- a clear image not having overlapping of the subject images can be obtained without a lens unlike a case of a technology of the related art described with FIGS. 17A to 17E and FIGS. 18A to 18D .
- the imaging range is not reduced, and the image can be acquired at a wide angle of view.
- a pattern I(r) of a coded aperture (Fresnel zone plate) is represented by Expression (1).
- I ( r ) cos ⁇ r 2 (1)
- I(r) As the value of I(r) is increased, the transmittance of light in a predetermined wavelength range is increased.
- the radius of the Fresnel zone plate is denoted by r.
- a constant determining the detail (pitch) of the pattern is denoted by ⁇ (>0).
- I2(r) that falls in a range of 0 to 1 by applying an offset as in Expression (2) will be considered.
- I 2( r ) 1 ⁇ 2(1+cos ⁇ r 2 (2)
- I2(r) and S(r) are originally two-dimensional images and are functions of two variables. However, for simplification, only one-dimensional images on a cross section acquired by cutting by a plane including the centers of the two-dimensional images and the incidence light source will be focused.
- a captured shadow image (projected image) is subjected to image restoration (reconstruction) in a computer and is output.
- image restoration reconstruction
- the shadow image is multiplied with a Fresnel zone aperture image (Fresnel zone pattern) that is not positionally shifted.
- a case of two functions represented by Expressions (5) and (6) below will be considered.
- Mc ( r ) e j ⁇ r 2 (6)
- Mr(r) is the same real number function as I(r). However, the offset (direct current component) is removed in Mr(r).
- Reconstruction of the image in the technology of the related art (“Development of lensless camera technology enabling easy focus adjustment after motion picture imaging”, [online], Nov. 15, 2016, Hitachi, Ltd., [searched on May 8, 2017], Internet (http://www.hitachi.co.jpNew/cnews/month/2016/11/1115.html), Yusuke Nakamura, Takeshi Shimano, Kazuyuki Tajima, Mayu Sao, and Taku Hoshizawa (Hitachi, Ltd.) “Lensless Light-field Imaging with Fresnel Zone Aperture”, The Institute of Image Information and Television Engineers Technical Report, vol.
- the real part and the imaginary part correspond to Fresnel zone patterns of which the phases are shifted by ( ⁇ /2), that is, 90°.
- Mc(r) has the same real number part (cos ⁇ r2) as Mr(r).
- the complex image including the image of the real part and the image of the imaginary part is generated by multiplying the projected image with two Fresnel zone patterns (first and second Fresnel zone patterns) of different phases corresponding to the real part and the imaginary part of the complex number function, respectively.
- the first term is a component that can be removed by offset correction and the like.
- the second term is a moire interference fringe from which a “frequency of difference” (corresponds to cos ( ⁇ ) in a case where two apertures are represented by cos ⁇ and cos ⁇ ) between superimposed Fresnel zone apertures is extracted and matches the basis of Fourier transformation.
- the second term is a component that is transformed into a delta function and changed into a “point” by applying Fourier transformation and contributes to image formation.
- the third term corresponds to a “frequency of sum” (corresponds to cos ( ⁇ + ⁇ ).
- the third term is a component that does not contribute to image formation and acts as a noise even in a case where Fourier transformation is performed.
- Fr2(r) and Fc2(r) The images in a state where the first term is removed by applying appropriate offset correction to Fr(r) and Fc(r) are denoted by Fr2(r) and Fc2(r).
- Fr2(r) and Fc2(r) Fourier transformation of Fr(r) and Fc(r) is denoted by fr(k) and fc(k) and is represented by Expressions (9) and (10).
- ⁇ (k, ⁇ , ⁇ r) is a real number polynomial.
- a restored image can be obtained using the absolute value of a complex number with respect to fr(k).
- fr(k) in the case of the technology of the related art
- the first term and the second term generate two points that are symmetric about an origin.
- a defect is present in that the restored image has point symmetry (refer to the examples in FIGS. 17A to 17E and FIGS. 18A to 18D ).
- fc(k) in the case of the embodiment of the present invention
- an image is normally reconstructed without posing such a problem.
- a common point in both cases is that the third term of fr(r) and the second term of fc(r) act as a noise. Due to the effect of these terms, a modulation transfer function (MTF) of an optical system cannot be 100% (meaning that the MTF cannot be 100% even in a case where a noise caused by the sensor is not present). However, this noise is decreased in a case where the value of ⁇ is increased. Thus, the effect can be reduced by increasing the value of ⁇ (making the pattern more detailed).
- MTF modulation transfer function
- a phase rotates by depending on the incidence angle of light.
- the absolute value of the complex number is used with respect to the first term of fc(k) (in the case of the embodiment of the present invention)
- FIG. 4 is a block diagram illustrating a configuration of an imaging system 10 (imaging system) according to a first embodiment.
- the imaging system 10 comprises an imaging module 100 (imaging unit) and an imaging apparatus main body 200 (image processing apparatus).
- the imaging system 10 can be applied to a digital camera, a smartphone, a tablet terminal, a monitoring camera, and the like.
- the imaging module 100 comprises a Fresnel zone plate 110 (Fresnel zone plate) and an imaging element 120 (image sensor). A projected image formed by light transmitted through the Fresnel zone plate 110 from the subject is acquired by the imaging element 120 .
- the Fresnel zone plate 110 is arranged on a light-receiving surface side of the imaging element 120 in a state where the center of the Fresnel zone plate 110 matches the center of the imaging element 120 , and the Fresnel zone plate 110 is parallel to the light-receiving surface of the imaging element 120 .
- the imaging module 100 may be replaceable with respect to the imaging apparatus main body 200 .
- the Fresnel zone plate 110 may be replaceable with respect to the imaging module 100 .
- the Fresnel zone plate 110 may be referred to as “FZP”.
- FIG. 7A is a diagram illustrating FZP 1 that is an example of the Fresnel zone plate 110 .
- FZP 1 the transmittance of incident light continuously changes depending on a distance from the center.
- a region (transmission region) that is more similar to white has a higher transmittance of light.
- a region (light blocking region) that is more similar to black has a lower transmittance of light.
- the transmission region and the light blocking region are alternately arranged in the concentric shape as a whole.
- the transmission regions and the light blocking regions constitute the Fresnel zone plate.
- the interval between concentric circles is decreased from the center to the periphery of FZP 1 .
- Such a pattern (change in local spatial frequency) of the shape of concentric circles is represented by Expressions (1), (2), and (6) and the like.
- the detail of the concentric circles in Expressions (1), (2), and (6) is referred to as a “pitch”.
- the pitch is determined by the value of ⁇ . In a case where ⁇ is small, the pattern is coarse. In a case where ⁇ is large, the pattern is detailed.
- a memory may be disposed in the imaging module 100 , and information of the pitch (value of ⁇ ) may be stored in the memory.
- An image processing unit 210 (information input unit: refer to FIG. 5 ) may acquire and use the information.
- An optical axis L (refer to FIGS. 1A to 1I ) of the Fresnel zone plate 110 is an axis that passes through the centers of FZP and the imaging element 120 and is perpendicular to FZP and the light-receiving surface of the imaging element 120 .
- FZP is arranged near (for example, at approximately 1 mm) the imaging element 120 .
- the projected image may be blurred due to diffraction of light depending on the distance between FZP and the imaging element 120 . Thus, it is preferable that FZP is not excessively separated from the imaging element 120 .
- FIG. 7B is a diagram illustrating FZP 2 that is another example of the Fresnel zone plate.
- a threshold value is set for the transmittance of FZP 1 .
- a region in which the transmittance exceeds the threshold value is the transmission region (white part) having a transmittance of 100%.
- a region in which the transmittance is smaller than or equal to the threshold value is the light blocking region (black part) having a transmittance of 0%.
- the transmittance non-continuously (in two levels of 0% and 100%) changes depending on the distance from the center.
- the transmission region and the light blocking region are alternately arranged in the concentric shape as a whole.
- the transmission regions and the light blocking regions constitute the Fresnel zone plate.
- the “Fresnel zone plate” in the embodiment of the present invention has the aspect of FZP 1 and the aspect of FZP 2 . Accordingly, the “Fresnel zone pattern” in the embodiment of the present invention also has a pattern in which the transmittance continuously changes and a pattern in which the transmittance non-continuously changes.
- a light blocking unit (a region in which light is not transmitted like the light blocking region) may be disposed in the peripheral part of the Fresnel zone plate illustrated in FIGS. 7A and 7B , and incidence of unnecessary light on the peripheral part of the imaging element 120 may be prevented.
- the imaging element 120 is an image sensors that includes a plurality of pixels configured with photoelectric conversion elements arranged in two-dimensional directions (in a two-dimensional shape). Light condensing efficiency may be increased by disposing a microlens in each pixel.
- a color image may be reconstructed by arranging a color filter (for example, red, blue, and green) in each pixel. In this case, an interpolation process corresponding to the arrangement pattern of the color filters is performed in the acquisition of the first and second projected image like a demosaicing process (referred to as demosaicing) in color image generation in a typical digital camera.
- demosaicing demosaicing
- a signal of a color insufficient in each pixel is generated, and a signal of each color (for example, red, blue, and green) is obtained in all pixels.
- a process can be performed by the image processing unit 210 (projected image input unit 210 A).
- the imaging apparatus main body 200 comprises the image processing unit 210 , a storage unit 220 , a display unit 230 , and an operation unit 240 .
- the imaging apparatus main body 200 performs image restoration and the like of the subject based on the projected image acquired by the imaging module 100 .
- FIG. 5 is a diagram illustrating a configuration of the image processing unit 210 .
- the image processing unit 210 includes a projected image input unit 210 A (projected image input unit), a complex image generation unit 210 B (complex image generation unit), a Fourier transformation unit 210 C (Fourier transformation unit), the information input unit 210 D (information input unit), and a display control unit 210 E.
- the projected image input unit 210 A acquires, from the imaging element 120 , the projected image formed in the imaging element 120 by light incident on FZP from the subject by controlling the imaging module 100 .
- the complex image generation unit 210 B generates the complex image including the image of the real part and the image of the imaginary part by multiplying the projected image with a plurality of Fresnel zone patterns (first and second Fresnel zone patterns) that have the same local spatial frequencies and have different phases of the local spatial frequencies.
- the Fourier transformation unit 210 C reconstructs the image of the spatial domain by performing two-dimensional complex Fourier transformation on the complex image.
- the information input unit 210 D acquires information (information of the pitch) of the Fresnel zone plate 110 used in the acquisition of the projected image.
- the display control unit 210 E controls display of the projected image, the complex image, the reconstructed image, and the like on the display unit 230 .
- Computer (processor)-readable codes of various programs for operating the imaging system 10 like an image processing program for executing the image processing method according to the embodiment of the present invention are recorded in a read only memory (ROM) 210 F (non-transitory recording medium).
- ROM read only memory
- the function of the image processing unit 210 can be implemented using various processors.
- the various processors include a central processing unit (CPU) that is a general-purpose processor implementing various functions by executing software (program).
- the various processors include a programmable logic device (PLD) that is a processor such as a field programmable gate array (FPGA) of which the circuit configuration can be changed after manufacturing.
- the various processors include a dedicated electric circuit or the like that is a processor such as an application specific integrated circuit (ASIC) having a circuit configuration dedicatedly designed to execute a specific process.
- ASIC application specific integrated circuit
- each unit may be implemented by one processor or may be implemented by combining a plurality of processors.
- a plurality of functions may be implemented by one processor.
- a first form is configuring one processor with a combination of one or more CPUs and software and implementing a plurality of functions by the processor as represented by a computer such as a client and a server.
- a second form is using a processor that implements the function of the whole system by one integrated circuit (IC) chip as represented by a system on chip (SoC) and the like.
- IC integrated circuit
- SoC system on chip
- Various functions are configured using one or more of the various processors as a hardware structure.
- the hardware structure of the various processors is more specifically an electric circuit (circuitry) in which circuit elements such as semiconductor elements are combined.
- a computer-readable code of the software (including the image processing program according to the embodiment of the present invention) to be executed is stored in a non-transitory recording medium such as the ROM 210 F (refer to FIG. 5 ), and the processor refers to the software.
- a random access memory (RAM) is used as a temporary storage region.
- RAM random access memory
- EEPROM electronically erasable and programmable read only memory
- the storage unit 220 is configured with a non-transitory recording medium such as a compact disk (CD), a digital versatile disk (DVD), a hard disk, and various semiconductor memories.
- the storage unit 220 stores images and information illustrated in FIG. 6 in association with each other.
- a projected image 220 A is the projected image acquired from the imaging module 100 .
- Fresnel zone plate information 220 B is information (including pitch information such as the value of ⁇ ) of the local spatial frequency of the Fresnel zone plate 110 .
- the Fresnel zone plate information 220 B may be information acquired from the imaging module 100 or may be information input through the operation unit 240 .
- Fresnel zone pattern information 220 C is information indicating the Fresnel zone pattern.
- a complex image 220 D is a complex image that includes the image of the real part and the image of the imaginary part and is obtained by multiplying the Fresnel zone patterns (first and second Fresnel zone patterns) indicated by the Fresnel zone pattern information 220 C with the projected image.
- a reconstructed image 220 E is an image of the spatial domain obtained by performing two-dimensional complex Fourier transformation on the complex image 220 D.
- the display unit 230 is configured to include a display apparatus such as a liquid crystal display, not illustrated.
- the display unit 230 displays the projected image, the complex image, the reconstructed image, and the like and is also used for displaying a user interface (UI) screen at the time of an instruction input through the operation unit 240 .
- the operation unit 240 is configured with devices such as a keyboard, a mouse, and a button, not illustrated. Using these devices, a user can input a projected image acquisition instruction, an image reconstruction instruction, a focal distance condition, information (the pitch and the phase) of the local spatial frequency, and the like.
- the display apparatus of the display unit 230 may be configured with a touch panel and may be used as the operation unit 240 in addition to the image display.
- FIG. 9 is a flowchart illustrating a procedure of the image processing method according to the present embodiment.
- step S 100 the image processing unit 210 (projected image input unit 210 A) acquires the projected image of the subject from the imaging element 120 by controlling the imaging module 100 .
- the acquired projected image is the projected image formed in the imaging element 120 by light incident on the Fresnel zone plate 110 from the subject.
- step S 110 the image processing unit 210 (information input unit 210 D) inputs information (pitch of the Fresnel zone plate 110 ) of the local spatial frequency of the Fresnel zone plate 110 used in the acquisition of the projected image.
- This information may be input from a memory, not illustrated, of the imaging module 100 or may be input in response to an operation performed on the operation unit 240 by the user.
- the projected image acquired in step S 100 may be analyzed and input by the information input unit 210 D.
- the pitch is determined by the value of ⁇ in Expressions (1) to (3) and (6) and the like. Thus, the value of ⁇ may be specifically input.
- the pitch (value of ⁇ ) can be acquired by analyzing the captured image.
- a value with which a clear image is obtained may be obtained by repeating the reconstruction of the image by changing the pitch (value of ⁇ ).
- step S 120 the image processing unit 210 (complex image generation unit 210 B) generates the complex image including the image of the real part and the image of the imaginary part by multiplying the projected image with each of the first and second Fresnel zone patterns.
- the Fresnel zone patterns multiplied in step S 120 patterns that are selected from patterns (Fresnel zone pattern information 220 C) stored in the storage unit 220 depending on the pitch (value of ⁇ ) input in step S 110 can be used.
- patterns acquired by changing (may be enlargement or reduction as necessary) the patterns stored in the storage unit 220 depending on the pitch (value of ⁇ ) can be used.
- the image processing unit 210 (complex image generation unit 210 B) stores the generated complex image in the storage unit 220 as the complex image 220 D.
- the first Fresnel zone pattern can be the pattern (phase at the center is 0°; “°” denotes “degree” which is the unit of angles) illustrated in FIG. 8A .
- the image of the real part is obtained by multiplying the first Fresnel zone pattern with the projected image.
- the second Fresnel zone pattern can be the pattern (has the same pitch and a shift of 90° in phase with the first Fresnel zone pattern) illustrated in FIG. 8B .
- the image of the imaginary part is obtained by multiplying the second Fresnel zone pattern with the projected image.
- the shift in phase between the first and second Fresnel zone patterns is 90°.
- phase of the local spatial frequency of the first Fresnel zone pattern or the second Fresnel zone pattern may be the same as the phase of the Fresnel zone plate 110 .
- Fresnel zone pattern information 220 C data of a plurality of Fresnel zone patterns having different phases may be stored in the storage unit 220 as the Fresnel zone pattern information 220 C, and a desired pattern can be selected and used.
- the image processing unit 210 (complex image generation unit 210 B) may generate a desired pattern based on information of the pitch and the phase (refer to FIG. 12 for the Fresnel zone pattern of each phase). Since the Fresnel zone patterns are stored in the storage unit 220 as the Fresnel zone pattern information 220 C that is electronic data, selection and generation of a desired pattern can be quickly and easily performed.
- the subject In a case where the subject (light source) is present at infinity, parallel light is incident on the Fresnel zone plate 110 , and the projected image formed in the imaging element 120 has the same size as the Fresnel zone plate 110 .
- the subject In a case where the subject is present at a finite distance, light that spreads is incident, and the projected image is increased as the distance is decreased. Accordingly, an image that is in focus at a desired distance can be obtained using patterns having different enlargement ratios as the first and second Fresnel zone patterns depending on a subject distance in focus. For example, a plurality of patterns corresponding to the subject distance can be stored in the storage unit 220 as the Fresnel zone pattern information 220 C and can be used by reading the patterns.
- one Fresnel zone pattern may be stored as a reference pattern and may be enlarged at different enlargement ratios depending on the subject distance.
- a pattern that corresponds to the infinite distance and has the same size as the Fresnel zone plate can be used as a reference.
- FIG. 10 is a diagram illustrating different enlargement ratios of the Fresnel zone pattern depending on the subject distance.
- the generation of the complex image (step S 120 ) and the reconstruction of the image (step S 130 ) may be repeated by changing the enlargement ratio, and a clear image may be acquired by maximizing an in-focus evaluation value (for example, the integral value of a brightness signal in a focus evaluation region set in the image) of the reconstructed image.
- an in-focus evaluation value for example, the integral value of a brightness signal in a focus evaluation region set in the image
- step S 130 the image processing unit 210 (Fourier transformation unit 210 C) reconstructs the image of the subject (image of the spatial domain) by performing two-dimensional complex Fourier transformation on the complex image as illustrated in Expression (10).
- the image processing unit 210 (display control unit 210 E) displays the reconstructed image on the display unit 230 (step S 140 ).
- the image processing unit 210 (Fourier transformation unit 210 C) stores the reconstructed image in the storage unit 220 as the reconstructed image 220 E.
- FIG. 11 is a table illustrating the phase of the Fresnel zone plate for acquisition of the projected image and the phase of the Fresnel zone pattern for acquisition of the complex image (a column with “real part” corresponds to the phase of the first Fresnel zone pattern, and a column with “imaginary part” corresponds to the phase of the second Fresnel zone pattern).
- Condition 8 (Example 1) to Condition 14 (Example 7) and Condition 22 (Example 8) to Condition 25 (Example 11) illustrate a preferable numerical value range of the shift in phase between the first and second Fresnel zone pattern in the embodiment of the present invention.
- FIG. 12 is a diagram illustrating Fresnel zone patterns having different phases.
- the first row illustrates Fresnel zone patterns having phases of 0°, 10°, 20°, 30°, 40°, and 50° at the center.
- the second row illustrates Fresnel zone patterns having phases of 60°, 70°, 75°, 80°, 90°, and 100° at the center.
- the third row illustrates Fresnel zone patterns having phases of 105°, 110°, 120°, 130°, 140°, and 150° at the center.
- the fourth row illustrates Fresnel zone patterns having phases of 160°, 170°, 180°, 190°, 200°, and 210° at the center.
- the fifth row illustrates Fresnel zone patterns having phases of 220°, 230°, 240°, 250°, 260°, and 270° at the center.
- the sixth row illustrates Fresnel zone patterns having phases of 280°, 290°, 300°, 310°, 320°, and 330° at the center.
- the seventh row illustrates Fresnel zone patterns having phases of 340°, 350°, and 360° at the center.
- FIGS. 13A to 13G to FIGS. 16A to 16E are diagrams illustrating images reconstructed under the above conditions.
- FIGS. 13A to 13G correspond to Conditions 1 to 7, respectively.
- FIGS. 14A to 14G correspond to Conditions 8 to 14 (preferable numerical value range of the shift in phase), respectively.
- FIGS. 15A to 15G correspond to Conditions 15 to 21, respectively.
- FIGS. 16A to 16D correspond to Conditions 22 to 25 (preferable numerical value range of the shift in phase), respectively.
- FIG. 16E corresponds to Condition 26 (technology of the related art).
- Condition 8 Example 1 to Condition 14 (Example 7) and Condition 22 (Example 8) to Condition 25 (Example 11) that satisfy a preferable range (70° to 110°) of the shift in phase between the first and second Fresnel zone patterns in the embodiment of the present invention
- overlapping of the images is completely or almost not present in the reconstructed image, and a clear image is obtained.
- it is not necessary to restrict the angle of view for obtaining a clear image.
- the clearest image is obtained in the case of Condition 11 (shift in phase is 90°).
- Condition 17 (Comparative Example 10) and Condition 22 (Example 8) will be reviewed by comparison.
- Condition 17 in which the phase of the Fresnel zone plate for imaging and the phase of the first Fresnel zone pattern (for the image of the real part) is equal, and the value of the shift in phase deviates from the preferable range, overlapping of the images occurs.
- Condition 22 in which the phase of the Fresnel zone plate for imaging and the phase of the first Fresnel zone pattern (for the image of the real part) are different, and the range of the shift in phase is in the preferable range, overlapping of the images does not occur.
- the phase of the Fresnel zone plate for imaging and the phase of the first Fresnel zone pattern do not need to be the same, and the shift in phase between the first and second Fresnel zone patterns may be approximately 90° (greater than or equal to 70° and smaller than or equal to 110°) like Conditions 8 to 14 (Examples 1 to 7) and Condition 22 (Example 8).
- Condition 11 Example 4
- Condition 25 Example 11
Landscapes
- Physics & Mathematics (AREA)
- Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Optics & Photonics (AREA)
- Theoretical Computer Science (AREA)
- Image Processing (AREA)
- Studio Devices (AREA)
- Stereoscopic And Panoramic Photography (AREA)
- Diffracting Gratings Or Hologram Optical Elements (AREA)
- Length Measuring Devices By Optical Means (AREA)
Abstract
Description
I(r)=cos βr 2 (1)
I2(r)=½(1+cos βr 2 (2)
S(r)=I2(r−Δr)=½{1+cos β(r−Δr)2} (3)
r=√{square root over (x 2 +y 2)} (4)
Mr(r)=I(r)=cos βr 2 (5)
Mc(r)=e jβr
-
- 10: imaging system
- 100: imaging module
- 110: Fresnel zone plate
- 120: imaging element
- 200: imaging apparatus main body
- 210: image processing unit
- 210A: projected image input unit
- 210B: complex image generation unit
- 210C: Fourier transformation unit
- 210D: information input unit
- 210E: display control unit
- 210F: ROM
- 220: storage unit
- 220A: projected image
- 220B: Fresnel zone plate information
- 220C: Fresnel zone pattern information
- 220D: complex image
- 220E: reconstructed image
- 230: display unit
- 240: operation unit
- F: Fresnel zone plate
- FZP1: Fresnel zone plate
- FZP2: Fresnel zone plate
- L: optical axis
- SD: shadow
- S100 to S140: step of image processing method
- d: distance
- θ: incidence angle
Claims (20)
Applications Claiming Priority (4)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2017-109269 | 2017-06-01 | ||
JP2017109269 | 2017-06-01 | ||
JPJP2017-109269 | 2017-06-01 | ||
PCT/JP2018/014981 WO2018221019A1 (en) | 2017-06-01 | 2018-04-10 | Image processing device, imaging system, image processing method, and recording medium |
Related Parent Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2018/014981 Continuation WO2018221019A1 (en) | 2017-06-01 | 2018-04-10 | Image processing device, imaging system, image processing method, and recording medium |
Publications (2)
Publication Number | Publication Date |
---|---|
US20200084350A1 US20200084350A1 (en) | 2020-03-12 |
US10999481B2 true US10999481B2 (en) | 2021-05-04 |
Family
ID=64454630
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
US16/681,701 Active US10999481B2 (en) | 2017-06-01 | 2019-11-12 | Image processing apparatus, imaging system, image processing method, and recording medium |
Country Status (5)
Country | Link |
---|---|
US (1) | US10999481B2 (en) |
EP (1) | EP3633973B1 (en) |
JP (1) | JP6773903B2 (en) |
CN (1) | CN110720207B (en) |
WO (1) | WO2018221019A1 (en) |
Families Citing this family (9)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP6772380B2 (en) * | 2017-06-01 | 2020-10-21 | 富士フイルム株式会社 | Image pickup device, image processing device, image pickup system, image processing method, and recording medium |
WO2019171691A1 (en) * | 2018-03-06 | 2019-09-12 | ソニー株式会社 | Image processing device, imaging device, and image processing method |
US11399134B2 (en) * | 2018-03-14 | 2022-07-26 | Sony Corporation | Image processing apparatus, imaging apparatus, and image processing method |
JP6924163B2 (en) * | 2018-03-27 | 2021-08-25 | 株式会社日立製作所 | Imaging device, imaging module and imaging method |
JP7065761B2 (en) * | 2018-12-26 | 2022-05-12 | 株式会社日立製作所 | Distance measuring device and distance measuring method |
CN109474818B (en) * | 2019-01-03 | 2024-05-10 | Oppo广东移动通信有限公司 | Image sensor and imaging module |
CN114208144A (en) * | 2019-07-31 | 2022-03-18 | 麦克赛尔株式会社 | Imaging device, portable terminal, and exposure control method |
WO2021024452A1 (en) * | 2019-08-08 | 2021-02-11 | マクセル株式会社 | Imaging device and method |
WO2023171358A1 (en) * | 2022-03-07 | 2023-09-14 | ソニーグループ株式会社 | Mask unit, imaging device, and imaging-device operating method |
Citations (7)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6246796B1 (en) * | 1997-05-28 | 2001-06-12 | Nippon Telegraph And Telephone Corporation | Method and apparatus for transmitting or processing images |
US20100060962A1 (en) * | 2007-01-29 | 2010-03-11 | Celloptic, Inc. | System, apparatus and method for extracting image cross-sections of an object from received electromagnetic radiation |
US8559014B2 (en) * | 2009-09-25 | 2013-10-15 | Hwan J. Jeong | High-resolution, common-path interferometric imaging systems and methods |
US20140049451A1 (en) * | 2011-10-20 | 2014-02-20 | Panasonic Corporation | Display device and display system |
WO2016203573A1 (en) | 2015-06-17 | 2016-12-22 | 日立マクセル株式会社 | Image capture device |
JP2018055831A (en) | 2016-09-26 | 2018-04-05 | 東芝ライテック株式会社 | Lighting device |
US20190278006A1 (en) * | 2016-09-26 | 2019-09-12 | Hitachi, Ltd. | Imaging device |
Family Cites Families (8)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
CN101641049A (en) * | 2007-03-21 | 2010-02-03 | 光谱辨识公司 | Biometrics based on locally consistent features |
CN101150402B (en) * | 2007-11-01 | 2011-02-16 | 北京理工大学 | A dual graph encryption method based on fraction rank Fourier conversion |
CN101430428B (en) * | 2008-11-25 | 2011-01-26 | 中国科学院微电子研究所 | Super-resolution Fresnel zone plate |
GB0822149D0 (en) * | 2008-12-04 | 2009-01-14 | Univ Sheffield | Provision of image data |
US20120267549A1 (en) * | 2009-05-07 | 2012-10-25 | President And Fellows Of Havard College | Methods and apparatus for fluorescence sensing employing fresnel zone plates |
US9823486B2 (en) * | 2013-03-11 | 2017-11-21 | Stc. Unm | Rotating point-spread function (PSF) design for three-dimensional imaging |
KR102441587B1 (en) * | 2015-08-03 | 2022-09-07 | 삼성전자주식회사 | Method and apparatus for processing holographic image |
CN105931196B (en) * | 2016-04-11 | 2018-10-19 | 天津大学 | Coding aperture camera image restoration methods based on Fourier Optics modeling |
-
2018
- 2018-04-10 CN CN201880035494.3A patent/CN110720207B/en active Active
- 2018-04-10 EP EP18810056.4A patent/EP3633973B1/en active Active
- 2018-04-10 JP JP2019522000A patent/JP6773903B2/en active Active
- 2018-04-10 WO PCT/JP2018/014981 patent/WO2018221019A1/en active Application Filing
-
2019
- 2019-11-12 US US16/681,701 patent/US10999481B2/en active Active
Patent Citations (11)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US6246796B1 (en) * | 1997-05-28 | 2001-06-12 | Nippon Telegraph And Telephone Corporation | Method and apparatus for transmitting or processing images |
US20100060962A1 (en) * | 2007-01-29 | 2010-03-11 | Celloptic, Inc. | System, apparatus and method for extracting image cross-sections of an object from received electromagnetic radiation |
US8559014B2 (en) * | 2009-09-25 | 2013-10-15 | Hwan J. Jeong | High-resolution, common-path interferometric imaging systems and methods |
US20140049451A1 (en) * | 2011-10-20 | 2014-02-20 | Panasonic Corporation | Display device and display system |
WO2016203573A1 (en) | 2015-06-17 | 2016-12-22 | 日立マクセル株式会社 | Image capture device |
US20180136480A1 (en) | 2015-06-17 | 2018-05-17 | Maxell, Ltd. | Imaging apparatus |
US10423002B2 (en) * | 2015-06-17 | 2019-09-24 | Maxell, Ltd. | Imaging apparatus capable of generating an image using moire without a lens |
US20190361257A1 (en) | 2015-06-17 | 2019-11-28 | Maxell, Ltd. | Imaging apparatus |
JP2018055831A (en) | 2016-09-26 | 2018-04-05 | 東芝ライテック株式会社 | Lighting device |
US20190278006A1 (en) * | 2016-09-26 | 2019-09-12 | Hitachi, Ltd. | Imaging device |
US10649118B2 (en) * | 2016-09-26 | 2020-05-12 | Hitachi, Ltd. | Imaging device |
Non-Patent Citations (7)
Title |
---|
"International Search Report (Form PCT/ISA/210) of PCT/JP2018/014981", dated Jul. 3, 2018, with English translation thereof, pp. 1-3. |
"Search Report of Europe Counterpart Application", dated Mar. 6, 2020, pp. 1-5. |
"Written Opinion of the International Searching Authority (Form PCT/ISA/237) of PCT/JP2018/014981", dated Jul. 3, 2018, with English translation thereof, pp. 1-8. |
Hitachi Ltd., "Development of lensless camera technology enabling easy focus adjustment after motion picture imaging", with English concise description of relevance, Nov. 15, 2016, Available at: "http://www.hitachi.co.jp/New/cnews/month/2016/11/1115.html". |
Kazuyuki Tajima et al., "Lensless light-field imaging with multi-phased fresnel zone aperture" , 2017 IEEE International Conference on Computational Photography (ICCP) , May 12, 2017 , pp. 1-7. |
Office Action of Japan Counterpart Application, with English translation thereof, dated Jul. 1, 2020, pp. 1-5. |
Yusuke Nakamura et al., "Lensless Light-field Imaging with Fresnel Zone Aperture", The Institute of Image Informationand Television Engineers Technical Report, vol. 40, No. 40, Nov. 17, 2016, pp. 1-4. |
Also Published As
Publication number | Publication date |
---|---|
CN110720207B (en) | 2021-04-27 |
EP3633973B1 (en) | 2020-11-11 |
CN110720207A (en) | 2020-01-21 |
EP3633973A4 (en) | 2020-04-08 |
WO2018221019A1 (en) | 2018-12-06 |
JPWO2018221019A1 (en) | 2020-03-19 |
US20200084350A1 (en) | 2020-03-12 |
EP3633973A1 (en) | 2020-04-08 |
JP6773903B2 (en) | 2020-10-21 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
US10999481B2 (en) | Image processing apparatus, imaging system, image processing method, and recording medium | |
US10911668B2 (en) | Imaging apparatus, image processing apparatus, imaging system, image processing method, and recording medium employing first and second Fresnel Zone plates with corresponding image sensors | |
US11272098B2 (en) | Imaging apparatus and imaging module | |
US11288780B2 (en) | Imaging apparatus | |
JP2018098623A (en) | Imaging device | |
TWI639976B (en) | Image processing device | |
US10488561B2 (en) | Imaging device | |
JP6646619B2 (en) | Imaging device | |
US11388431B2 (en) | Imaging apparatus and imaging system | |
JP2023016864A (en) | Imaging apparatus and method | |
JP6947891B2 (en) | Mobile information terminal | |
WO2019131191A1 (en) | Wide conversion lens and image capture device | |
JP6770164B2 (en) | Imaging device | |
JP7389195B2 (en) | Image generation method |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
FEPP | Fee payment procedure |
Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY |
|
AS | Assignment |
Owner name: FUJIFILM CORPORATION, JAPAN Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:KISHINE, YASUNOBU;NARUSE, YOSUKE;SIGNING DATES FROM 20190912 TO 20190917;REEL/FRAME:051043/0775 |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: AWAITING TC RESP., ISSUE FEE NOT PAID |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT RECEIVED |
|
STPP | Information on status: patent application and granting procedure in general |
Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED |
|
STCF | Information on status: patent grant |
Free format text: PATENTED CASE |