WO2019054304A1 - 撮像装置 - Google Patents

撮像装置 Download PDF

Info

Publication number
WO2019054304A1
WO2019054304A1 PCT/JP2018/033266 JP2018033266W WO2019054304A1 WO 2019054304 A1 WO2019054304 A1 WO 2019054304A1 JP 2018033266 W JP2018033266 W JP 2018033266W WO 2019054304 A1 WO2019054304 A1 WO 2019054304A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
imaging
unit
distance
pixel
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2018/033266
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
征志 中田
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Interactive Entertainment Inc
Original Assignee
Sony Interactive Entertainment Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Interactive Entertainment Inc filed Critical Sony Interactive Entertainment Inc
Priority to US16/644,260 priority Critical patent/US11064182B2/en
Publication of WO2019054304A1 publication Critical patent/WO2019054304A1/ja
Anticipated expiration legal-status Critical
Priority to US17/341,807 priority patent/US11438568B2/en
Priority to US17/875,981 priority patent/US12439015B2/en
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/204—Image signal generators using stereoscopic image cameras
    • H04N13/207—Image signal generators using stereoscopic image cameras using a single two-dimensional [2D] image sensor
    • H04N13/218—Image signal generators using stereoscopic image cameras using a single two-dimensional [2D] image sensor using spatial multiplexing
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B3/00—Simple or compound lenses
    • G02B3/0006—Arrays
    • G02B3/0037—Arrays characterized by the distribution or form of lenses
    • G02B3/0056—Arrays characterized by the distribution or form of lenses arranged along two different directions in a plane, e.g. honeycomb arrangement of lenses
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B5/00—Optical elements other than lenses
    • G02B5/20—Filters
    • G02B5/201—Filters in the form of arrays
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
    • G02B7/28—Systems for automatic generation of focusing signals
    • G02B7/30—Systems for automatic generation of focusing signals using parallactic triangle with a base line
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B7/00—Mountings, adjusting means, or light-tight connections, for optical elements
    • G02B7/28—Systems for automatic generation of focusing signals
    • G02B7/34—Systems for automatic generation of focusing signals using different areas in a pupil plane
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/257—Colour aspects
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/20—Image signal generators
    • H04N13/296—Synchronisation thereof; Control thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/45—Cameras or camera modules comprising electronic image sensors; Control thereof for generating image signals from two or more image sensors being of different type or operating in different modes, e.g. with a CMOS sensor for moving images in combination with a charge-coupled device [CCD] for still images
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/50—Constructional details
    • H04N23/55—Optical parts specially adapted for electronic image sensors; Mounting thereof
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/67—Focus control based on electronic image sensor signals
    • H04N23/672—Focus control based on electronic image sensor signals based on the phase difference signals
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/10—Circuitry of solid-state image sensors [SSIS]; Control thereof for transforming different wavelengths into image signals
    • H04N25/11—Arrangement of colour filter arrays [CFA]; Filter mosaics
    • H04N25/13—Arrangement of colour filter arrays [CFA]; Filter mosaics characterised by the spectral characteristics of the filter elements
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N25/00—Circuitry of solid-state image sensors [SSIS]; Control thereof
    • H04N25/70—SSIS architectures; Circuits associated therewith
    • H04N25/703—SSIS architectures incorporating pixels for producing signals other than image signals
    • H04N25/704—Pixels specially adapted for focusing, e.g. phase difference pixel sets
    • H—ELECTRICITY
    • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H10F—INORGANIC SEMICONDUCTOR DEVICES SENSITIVE TO INFRARED RADIATION, LIGHT, ELECTROMAGNETIC RADIATION OF SHORTER WAVELENGTH OR CORPUSCULAR RADIATION
    • H10F39/00—Integrated devices, or assemblies of multiple devices, comprising at least one element covered by group H10F30/00, e.g. radiation detectors comprising photodiode arrays
    • H10F39/10—Integrated devices
    • H10F39/12—Image sensors
    • H—ELECTRICITY
    • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H10F—INORGANIC SEMICONDUCTOR DEVICES SENSITIVE TO INFRARED RADIATION, LIGHT, ELECTROMAGNETIC RADIATION OF SHORTER WAVELENGTH OR CORPUSCULAR RADIATION
    • H10F39/00—Integrated devices, or assemblies of multiple devices, comprising at least one element covered by group H10F30/00, e.g. radiation detectors comprising photodiode arrays
    • H10F39/80—Constructional details of image sensors
    • H10F39/805—Coatings
    • H10F39/8053—Colour filters
    • H—ELECTRICITY
    • H10—SEMICONDUCTOR DEVICES; ELECTRIC SOLID-STATE DEVICES NOT OTHERWISE PROVIDED FOR
    • H10F—INORGANIC SEMICONDUCTOR DEVICES SENSITIVE TO INFRARED RADIATION, LIGHT, ELECTROMAGNETIC RADIATION OF SHORTER WAVELENGTH OR CORPUSCULAR RADIATION
    • H10F39/00—Integrated devices, or assemblies of multiple devices, comprising at least one element covered by group H10F30/00, e.g. radiation detectors comprising photodiode arrays
    • H10F39/80—Constructional details of image sensors
    • H10F39/806—Optical elements or arrangements associated with the image sensors
    • H10F39/8063—Microlenses
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B5/00—Optical elements other than lenses
    • G02B5/30—Polarising elements
    • G02B5/3025—Polarisers, i.e. arrangements capable of producing a definite output polarisation state from an unpolarised input state
    • G—PHYSICS
    • G02—OPTICS
    • G02B—OPTICAL ELEMENTS, SYSTEMS OR APPARATUS
    • G02B5/00—Optical elements other than lenses
    • G02B5/30—Polarising elements
    • G02B5/3025—Polarisers, i.e. arrangements capable of producing a definite output polarisation state from an unpolarised input state
    • G02B5/3058—Polarisers, i.e. arrangements capable of producing a definite output polarisation state from an unpolarised input state comprising electrically conductive elements, e.g. wire grids, conductive particles
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B11/00—Filters or other obturators specially adapted for photographic purposes
    • G—PHYSICS
    • G03—PHOTOGRAPHY; CINEMATOGRAPHY; ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ELECTROGRAPHY; HOLOGRAPHY
    • G03B—APPARATUS OR ARRANGEMENTS FOR TAKING PHOTOGRAPHS OR FOR PROJECTING OR VIEWING THEM; APPARATUS OR ARRANGEMENTS EMPLOYING ANALOGOUS TECHNIQUES USING WAVES OTHER THAN OPTICAL WAVES; ACCESSORIES THEREFOR
    • G03B35/00—Stereoscopic photography
    • G03B35/08—Stereoscopic photography by simultaneous recording
    • G03B35/10—Stereoscopic photography by simultaneous recording having single camera with stereoscopic-base-defining system
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/60—Control of cameras or camera modules
    • H04N23/66—Remote control of cameras or camera parts, e.g. by remote control devices

Definitions

  • the present invention relates to an imaging device.
  • the conventional methods for capturing 3D images have the following problems. (1) When viewing a video on a head mounted display, if the line connecting the user's eyes becomes oblique or vertical due to tilting the neck, etc., the camera position at the time of shooting will differ and it will be difficult to obtain appropriate parallax. Become. (2) If the optical size of each of the two cameras is increased to obtain high image quality, the overall size of the camera system increases. (3) A blind spot occurs at a close distance between lenses of a large camera.
  • the present invention has been made in view of these problems, and one object is to provide an improved technique for providing a suitable three-dimensional image.
  • an imaging device is a first imaging unit that images a subject, and a second imaging unit installed at a peripheral position of the first imaging unit and that images the subject. Generating data for output based on a plurality of second imaging units having a smaller optical size than the first imaging unit, an image captured by the first imaging unit, and an image captured by the plurality of second imaging units And an image processing unit.
  • FIG. 6 It is a figure which shows notionally the structure of the imaging device in related technology. It is a figure which shows the structural example of the pixel with which the related art imaging device is provided. It is a figure which illustrates the pixel arrangement in the related art imaging device. It is a figure which shows the outline of the structure of the image sensor in related technology. In related art, it is a figure for demonstrating the principle which acquires distance information by phase difference. It is a figure for demonstrating the relationship between the image acquired by related technology, and a focal distance. It is a figure which shows typically the relationship of the focal distance in the case of FIG. 6, and a phase difference. It is a figure which shows the functional block of the image processing part in related technology.
  • FIG. 21 (a) is a view schematically showing the arrangement of cameras in the prior art
  • FIG. 21 (b) is a view schematically showing the arrangement of the cameras in the first embodiment
  • FIG. 22 (a) is a view schematically showing an arrangement of cameras in the related art
  • FIG. 22 (b) is a view schematically showing an arrangement of cameras in the first embodiment. It is a block diagram which shows the function structure of the imaging device of 1st Example.
  • FIG. 1 is a diagram conceptually showing the structure of the imaging device in the present embodiment.
  • the imaging device 12 includes an imaging optical system 14, an aperture 18, an imaging element 20, and an image processing unit 22.
  • the imaging optical system 14 has a general configuration including a focusing lens for forming an image of a subject on an imaging surface of the imaging device 20. In the drawing, one lens is shown as representative.
  • the diaphragm 18 has an opening and has a general configuration for adjusting the amount of incident light by changing its aperture.
  • the imaging device 20 includes a two-dimensional array of pixels, converts the intensity of incident light into a charge, and outputs the charge to the image processing unit 22.
  • the pixel in this embodiment mode has a structure in which at least a microlens, a polarizer, and a photodiode are integrally stacked.
  • a phase difference image obtained by dividing incident light into two images is obtained.
  • a region corresponding to one microlens is referred to as one pixel region.
  • a plurality of photodiodes are provided for one pixel.
  • the photodiode is a representative example of a mechanism for converting the intensity of incident light into charge
  • the present invention is not limited to this. That is, even if any photoelectric conversion mechanism is employed instead of the photodiode, the present embodiment can be realized similarly, and one unit mechanism for converting light into charge can be used instead of each photodiode.
  • the polarizer may be provided to all the pixels, or may be provided discretely to some of the pixels.
  • the image processing unit 22 performs image processing using the two-dimensional distribution of the luminance of light output from the imaging device 20, and generates a general color image and a distance image representing the distance to the subject as a pixel value.
  • the imaging device 12 may be further provided with an operation unit by the user and a mechanism for executing an imaging operation, an adjustment operation of imaging conditions, and the like according to the content of the operation.
  • the imaging device 12 establishes a communication with an external information processing device such as a game machine by wire or wireless, and has a mechanism for transmitting generated data and receiving control signals such as a data transmission request. Good. However, since these mechanisms may be similar to those of a general imaging device, the description thereof is omitted.
  • FIG. 2 shows an example of the structure of a pixel included in the imaging device 12.
  • the figure schematically shows the functional structure of the element cross section, and detailed structures such as interlayer insulating films and wirings are omitted. Further, in the drawing, the cross-sectional structure of two adjacent pixels is illustrated.
  • the pixel 110 includes a microlens layer 112, a color filter layer 114, a polarizer layer 116, and a photoelectric conversion layer 118.
  • the microlens layer 112 is provided for each pixel and condenses the light incident through the diaphragm 18.
  • the color filter layer 114 transmits light of a different color for each pixel.
  • the polarizer layer 116 includes a wire grid type polarizer in which a plurality of linear conductor members, for example, members (wires) such as tungsten and aluminum are arranged in stripes at intervals smaller than the wavelength of incident light.
  • Polarized luminance is obtained by converting the transmitted polarization component into a charge in the photoelectric conversion layer 118.
  • An image acquisition technique using a wire grid type polarizer as illustrated is disclosed, for example, in Japanese Patent Application Laid-Open No. 2012-80065 and the like.
  • the element structure of the imaging device 12 in the present embodiment is not limited to that illustrated.
  • the polarizer is not limited to the wire grid type, and may be any practical one such as a linear dichroism polarizer.
  • a cross section of the wire extending in the depth direction of the drawing is shown as a polarizer, but the principal axis angle of the polarizer is four, and the direction of the wire is different accordingly.
  • the polarizer layer 116 may have regions with and without a polarizer depending on the pixel. In the region where the polarizer is not provided, the light transmitted through the color filter layer 114 is incident on the photoelectric conversion layer 118 as it is.
  • the photoelectric conversion layer 118 includes a general photodiode and outputs incident light as a charge. As described above, in this embodiment, by providing a plurality of photodiodes for one microlens, light transmitted through different regions of the focusing lens is separately converted into charges.
  • a technique for performing focus detection based on the phase difference of light detected in this manner is put to practical use as a method of phase difference autofocus (see, for example, Japanese Patent Application Laid-Open No. 2013-106194).
  • the distance to the subject is acquired using the phase difference. If the detection values of a plurality of photodiodes provided in one pixel are summed, the luminance for one pixel in a general imaging device can be obtained. That is, according to the configuration of the pixels shown in FIG. 2, a general color image, a distance image, and a polarization image can be obtained simultaneously.
  • FIG. 3 exemplifies a pixel array in the imaging device 20.
  • the figure schematically shows a combination of layers when a partial region of the imaging device 20 is viewed from the top, and a vertically long rectangle indicates one photodiode (for example, the photodiode 120).
  • the pair of left and right two photodiodes correspond to one pixel (for example, pixel 122).
  • the color filters in the color filter layer 114 are arranged in a Bayer pattern, and any of red, green and blue light is detected for each pixel. In the figure, they are indicated by the letters "R", "G” and "B" respectively.
  • polarizers are provided in the pixels 124 a and 124 b shown by thick lines.
  • the thick diagonal lines in these pixels 124a and 124b indicate the wires that make up the polarizer. That is, the pixels 124a, 124b are provided with polarizers of different principal axis angles. Although two types of polarizers whose main axis angles are orthogonal to each other are illustrated in the figure, another pixel is used to provide four types of polarizers having main axis angles every 45 degrees.
  • Each polarizer transmits polarization components in a direction orthogonal to the direction of the wire.
  • the photodiode provided in the lower layer outputs a charge representing the luminance of the polarized light component in four directions at 45 ° intervals.
  • detection values from two photodiodes provided in one pixel may be summed.
  • the pixel provided with the polarizer is a green pixel.
  • the pixels provided with the polarizer can be made relatively close to each other, and it is possible to obtain polarization luminance of a plurality of azimuths of the same color with high resolution.
  • a polarization image of four directions can be obtained by separating and interpolating this for each polarization direction.
  • the polarization image it is possible to obtain the normal vector of the object surface.
  • the normal vector represents the inclination of a minute area on the surface of the subject, and this can be used to interpolate the distance value at the feature point obtained based on the phase difference. Since the distance value and the normal vector due to the phase difference are simultaneously obtained from the photographed images of the same viewpoint by the same imaging device 12, accurate interpolation can be realized without the need for alignment and the like.
  • the color filter layer 114 may be removed from the pixel 110.
  • the color filter may be a dye-based filter such as cyan or magenta.
  • the arrangement shown in FIG. 3 is merely an example, and the pixel arrangement of this embodiment is not limited to this.
  • the density of pixels provided with a polarizer may be further increased, or polarizers may be provided in all the pixels.
  • FIG. 4 shows the outline of the structure of the image sensor in the present embodiment.
  • the image sensor 170 includes a pixel unit 172, a row scanning unit 174 as a peripheral circuit, a horizontal selection unit 176, a column scanning unit 180, and a control unit 178.
  • the pixel section 172 is formed by arranging the pixels as shown in FIG. 2 in a matrix.
  • Each photodiode in the photoelectric conversion layer 118 is connected to the row scanning unit 174 for each row, the horizontal selection unit 176 for each column, and the column scanning unit 180.
  • the row scanning unit 174 is configured by a shift register, an address decoder, and the like, and drives each pixel row by row.
  • the signal output from the pixel selectively scanned by the row scanning unit 174 is supplied to the horizontal selection unit 176.
  • the horizontal selection unit 176 is configured by an amplifier, a horizontal selection switch, and the like.
  • the column scanning unit 180 is configured of a shift register, an address decoder, and the like, and drives in order while operating each horizontal selection switch of the horizontal selection unit 176.
  • the signal from each pixel supplied to the horizontal selection unit 176 is output to the outside by the selective scanning by the column scanning unit 180.
  • the control unit 178 generates a timing signal, and controls the drive timing of the horizontal selection unit 176, the column scanning unit 180, and the like.
  • the peripheral circuit as illustrated may be divided into two depending on the presence or absence of a polarizer so that the timing and interval of data reading can be controlled independently.
  • the frame rate of the pixel including the polarizer may be increased.
  • it is possible to increase the detection sensitivity of the movement of the surface of the subject by obtaining the distribution of the normal vector at a high frequency using the luminance distribution of polarized light output at a high rate. How to control the timing of data reading may be determined according to the processing content of the subsequent stage, the required detection sensitivity, and the like.
  • FIG. 5 is a diagram for explaining the principle of acquiring distance information by phase difference.
  • This figure shows a state in which the light from the subject 130 enters the imaging surface 134 of the imaging device 20 through the focusing lens 132 of the imaging optical system 14 as viewed from the upper side of the imaging space.
  • the states (a), (b) and (c) it is assumed that the distances from the imaging surface 134 to the subject 130 are different, and the subject 130 in the state (b) is at the in-focus position, that is, the focusing surface 138.
  • the light emitted from one point of the subject 130 forms an image at one point on the imaging surface 134 as illustrated. Therefore, one point of the subject 130 corresponds to one pixel, and even if two photodiodes are provided in one pixel, the luminous flux detected by them is from substantially the same point of the subject 130.
  • the position where the light forms an image is the imaging surface 134 It slips away.
  • the photodiode on the left side for example, the photodiode 138a
  • the photodiode on the right side for example, the photodiode 138b
  • phase difference images two images in which the luminances detected by the left photodiode and the right photodiode are pixel values are referred to as “phase difference images”, and the shift amount of the image of the same object in both is referred to as “phase difference”.
  • FIG. 6 is a diagram for explaining the relationship between an image acquired in the present embodiment and a focal length.
  • the figure schematically shows a phase difference image when the space in which the face and the cube exist is photographed.
  • the left and right images the left is detected by the left photodiode and the right is detected by the right photodiode.
  • a phase difference of (A'-A) occurs in the cube image.
  • (B) is the case where the cube is in focus. In this case, the cube image is at a distance A from the left edge of the image in both of the phase difference images, and there is no phase difference.
  • there is a B'-B phase difference in the image of the face there is a B'-B phase difference in the image of the face.
  • the phase difference may take a negative value because the direction in which the object deviates is reversed depending on whether the object is closer or farther than the focal distance.
  • FIG. 7 schematically shows the relationship between the focal length and the phase difference in the case of FIG.
  • the solid line in the figure shows the phase difference of the face, and the broken line shows the phase difference of the cube as a change with respect to the focal length.
  • the characteristics of the phase difference are not limited to those illustrated, due to various factors of the optical system.
  • the focal length is F1
  • the phase difference of the face is 0, and the cube has a phase difference of A'-A.
  • the focal length is F2
  • the phase difference of the cube is 0, and the face has a phase difference of B'-B.
  • the phase difference is uniquely determined by the focal length.
  • the focal length can be obtained similarly to the focusing function in a general imaging device.
  • the relationship between the distance of the subject from the focal distance (focus plane) and the phase difference is prepared in advance as a table that is experimentally obtained from an image obtained by actually shooting a subject at a known distance.
  • the distance from the imaging surface to the subject can be calculated by obtaining the distance from the focal plane based on the observed phase difference and further adding the focal length.
  • the brightness of light observed through the polarizer changes with the main axis angle ⁇ pol of the polarizer as in the following equation.
  • I max and I min are the maximum value and the minimum value of the observed luminance, respectively, and ⁇ is the polarization phase. If to shaft angle theta pol four types as described above were obtained polarization image, the brightness I of pixels at the same position will satisfy the formula 1 for each spindle angle theta pol. Therefore, I max , I min , and ⁇ can be obtained by approximating a curve passing through those coordinates (I, ⁇ pol ) to a cosine function using a least squares method or the like. The degree of polarization ⁇ is determined by the following equation using I max and I min thus determined.
  • the normal to the object surface can be expressed by an azimuth angle ⁇ that represents the angle of the light incident surface (emission surface in the case of diffuse reflection) and a zenith angle ⁇ that represents the angle on the surface.
  • the spectrum of reflected light is represented by a linear sum of the spectrum of specular reflection and diffuse reflection.
  • specular reflection is light that is specularly reflected on the surface of an object
  • diffuse reflection is light that is scattered by pigment particles that make up the object.
  • the above-mentioned azimuth angle ⁇ is a principal axis angle that gives the minimum luminance I min in Equation 1 in the case of specular reflection, and is a principal axis angle that gives the maximum luminance I max in Equation 1 in the case of diffuse reflection.
  • the zenith angle ⁇ has the following relationship with the degree of polarization s s in the case of specular reflection and the degree of polarization d d in the case of diffuse reflection, respectively.
  • n is the refractive index of the object.
  • the zenith angle ⁇ can be obtained by substituting the degree of polarization ⁇ obtained in Equation 2 into either ⁇ s or d d in Equation 3. From the azimuth angle ⁇ and the zenith angle ⁇ thus obtained, the normal vectors (p x , p y , p z ) are obtained as follows.
  • the normal vector of the object shown in the pixel can be determined from the relationship between the luminance I represented by each pixel of the polarization image and the principal axis angle ⁇ pol of the polarizer, and the normal vector distribution can be obtained as the entire image it can.
  • the normal can be determined with higher accuracy by adopting an appropriate model of specular reflection and diffuse reflection based on the color and material.
  • specular reflection and diffuse reflection since various techniques for separating specular reflection and diffuse reflection have been proposed, such techniques may be applied to obtain the normal more strictly.
  • FIG. 8 shows functional blocks of the image processing unit 22 in the present embodiment.
  • the functional blocks shown in FIG. 17 and FIG. 17 and FIG. 20 described later can be realized as hardware in the configuration of an imaging device, various arithmetic circuits, a microprocessor, a buffer memory, etc. Is realized by the program to be Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any of them.
  • the image processing unit 22 acquires two-dimensional data of luminance from the imaging device 20 and performs predetermined preprocessing, a focal length acquisition unit 30 which acquires an actual focal length, and a distance image based on the phase difference.
  • Feature point acquiring unit 34 that generates the normal point image generating unit 36 that generates a normal image from polarization luminance of a plurality of azimuths, and complements the distance value based on the phase difference using a normal vector to generate a distance image
  • the pixel value acquisition unit 32 acquires the luminance signal detected by the imaging device 20 as two-dimensional data, and performs predetermined preprocessing such as A / D conversion and clamping processing.
  • the focal length acquisition unit 30 reads out from the memory the focal length acquired in a focusing function (not shown) included in the imaging device 2 and the like. In an environment where focal length adjustment is possible, the data is read out each time the focal length changes. In an apparatus in which the focal length is fixed, the setting value is obtained in the beginning.
  • the feature point distance acquisition unit 34 includes a phase difference detection unit 46, a distance value acquisition unit 48, and a distance correspondence table 50.
  • the phase difference detection unit 46 separates pixel values detected by the left photodiode and the right photodiode among the pixel values acquired by the pixel value acquisition unit 32, and generates a phase difference image. At this time, all pixels in the Bayer arrangement may be processed, or only green pixels may be processed. Then, the feature points of both are extracted, and the phase difference is acquired for each feature point by specifying the position representing the same feature point in the subject.
  • the distance correspondence table 50 stores a distance correspondence table in which the distance from the focus plane is associated with the phase difference.
  • the distance value acquisition unit 48 refers to the distance correspondence table based on the phase difference acquired by the phase difference detection unit 46, and acquires a distance value corresponding to the phase difference. Then, the absolute value of the distance from the imaging surface is acquired for each feature point by adding to the focal length acquired from the focal length acquisition unit 30.
  • the normal-line image generation unit 36 extracts the value of the pixel provided with the polarizer among the pixel values acquired by the pixel value acquisition unit 32, and further separates and interpolates for each principal axis angle of the polarizer to obtain a plurality of azimuths. Generate a polarized image of At this time, detection values by two photodiodes provided in one pixel are summed up to form one pixel value. Further, by interpolating the polarization luminance of each azimuth, polarization luminances of a plurality of azimuths are acquired for the same position coordinate on the image plane. Then, the normal vector is calculated using Equations 1 to 4 based on the change in polarization luminance with respect to the azimuth.
  • the normal image generation unit 36 generates a normal image having three elements of the normal vector obtained for each pixel as pixel values. This image can basically have the same resolution as the captured image. On the other hand, depending on the normal vector and the resolution required for the subsequent distance image, the normal image may be generated at a lower resolution than the captured image.
  • the distance image generation unit 38 complements the distance value to the feature point generated by the feature point distance acquisition unit 34 using the normal image generated by the normal image generation unit 36 to obtain the pixel of the distance of the object surface. Generate a distance image represented as a value. That is, although the feature point distance acquiring unit 34 can acquire distance values for feature points such as the contour of the image of the subject and the surface pattern whose phase difference is known, it extracts feature points such as a monochrome smooth object surface. It is difficult to calculate the distance of the difficult area.
  • the normal image generation unit 36 can obtain the inclination of the object surface in detail for each minute area. Therefore, the normal vector is obtained by sequentially giving the slope based on the normal vector acquired by the normal image generation unit 36, starting from the distance value at the feature point acquired by the feature point distance acquisition unit 34. The distance can be determined with the same resolution as in.
  • the defect correction unit 40 corrects the pixel value of the pixel provided with the polarizer among the pixel values acquired by the pixel value acquisition unit 32.
  • a pixel provided with a polarizer reflects a polarization component in the same direction as the principal axis angle of the polarizer, light reaching the photodiode has lower intensity than light incident on the imaging surface. Therefore, by correcting the luminance of the pixel to a level similar to that of the surrounding pixels, it is possible to prevent some pixels of the color image from becoming a black point.
  • interpolation may be performed using peripheral pixel values, or the reduction rate of the light amount due to the polarizer may be obtained by experiment etc., and a constant based on that may be multiplied by the corresponding pixel value.
  • the detection values of the pair of photodiodes are summed and handled as one pixel value.
  • the color image generation unit 42 demosaic-processes the image after defect correction to generate a color image in which one pixel has three color values. That is, by interpolating the pixel values obtained in the Bayer arrangement as shown in FIG. 3 for each color, all the pixels have three elements. General demosaicing techniques can be applied to this process.
  • the output unit 44 acquires at least the data of the distance image generated by the distance image generation unit 38 and the data of the color image generated by the color image generation unit 42, and sequentially transmits the data to an external device.
  • the output unit 44 may temporarily store the data in a memory, a recording medium, or the like, and transmit the data to an external device at an appropriate timing according to a user operation or the like, or the user can carry it out.
  • various information processing can be performed accurately using them. For example, since the position of the subject in the three-dimensional space is known along with the color, they can be once arranged in the virtual space, and the display image can be reconstructed according to the viewpoint of the user wearing the head mounted display. At this time, virtual reality and augmented reality can be realized by generating an image for the left viewpoint and an image for the right viewpoint and displaying the display screen of the head mounted display in the left and right areas divided into two.
  • the output unit 44 may further output the normal image generated by the normal image generation unit 36.
  • the information on the normal line can be used for motion detection because it represents a change in the posture of the subject with higher sensitivity than the image of the subject itself.
  • the image processing unit 22 of the imaging device 12 can generate the distance image together with the color image, the load of the information processing apparatus that performs various processes using it can be suppressed, and Power consumption can be reduced.
  • At least one of the distance image generation unit 38, the feature point distance acquisition unit 34, and the normal image generation unit 36 may be provided in an information processing apparatus other than the imaging apparatus 12.
  • a logic circuit having at least a part of the functions as illustrated may be provided in the lower layer of the pixel array to be a stacked image sensor.
  • FIG. 9 schematically shows the transition of a photographed image in the image processing unit 22.
  • the pixel value acquisition unit 32 acquires data of a captured image such as the image 220.
  • a cube is shown as a subject.
  • the data to be acquired strictly includes information on the brightness of natural light or polarized light detected by the left photodiode and the right photodiode.
  • the feature point distance acquisition unit 34 acquires the phase difference of the feature point as described above, and generates data 222 of the distance value for the feature point from it and the focal length.
  • the data 222 shown in the figure is expressed in the form of a distance image in which the higher the distance value is, the higher the brightness is, and the place where the distance value is not obtained is the lowest brightness.
  • FIG. 3 when a pair of photodiodes are disposed on the left and right with respect to the area of one pixel, the phase difference appears in the horizontal direction of the image plane. Therefore, as shown in data 222, an accurate phase difference can not be specified for the edge in the horizontal direction, and the distance value is also indefinite.
  • the normal image generation unit 36 generates a normal image 224 using polarization images of a plurality of directions.
  • a part of the distribution of normal vectors of the cube surface is indicated by arrows, but in practice the normal vectors can be determined in pixel units.
  • the distance image generation unit 38 applies the inclination of the surface based on the normal vector in pixel units, starting from the distance of the edge portion obtained by the data 222 of the distance value based on the phase difference.
  • the distance between the edges in the data 222 is a plane, and the distance value of the surface including the horizontal edge portion where the distance value can not be obtained.
  • position information 226 in the world coordinate system can be acquired for a portion of the cube surface that is viewed as a captured image.
  • the distance image generation unit 38 may generate information related to the position coordinates of the object surface in such a three-dimensional space, or may generate a distance image in which the distance value is represented on the image plane.
  • FIG. 10 is a flowchart showing a processing procedure in which the image processing unit 22 in the present embodiment generates and outputs various data from the captured image.
  • the pixel value acquisition unit 32 acquires, from the imaging device 20, data of luminance detected by each photodiode (S10).
  • the obtained luminance data is supplied to the feature point distance acquisition unit 34, the normal image generation unit 36, and the defect correction unit 40.
  • the feature point distance acquisition unit 34 separates the luminance detected by the left photodiode and the right photodiode to generate a phase difference image, and acquires the phase difference by correlating the feature points (S14). Then, based on the phase difference and the focal length, the distance value for the pixels constituting the feature point is specified (S16).
  • the normal image generation unit 36 generates polarization images of a plurality of azimuths by extracting values of pixels detecting polarization and separating and interpolating the values for each principal axis angle of the polarizer (S18). Then, by acquiring the azimuth dependency of the polarization luminance at the same position, a normal vector is calculated for each pixel or in a unit larger than that, and a normal image is generated (S20).
  • the distance image generation unit 38 generates a distance image in which the distance value is complemented by obtaining the distance value using the normal image at a position where the distance value is not obtained by the phase difference (S22).
  • the defect correction unit 40 performs defect correction to amplify the luminance level of the pixel whose polarization is detected so as to be the same level as the other pixels (S24).
  • the color image generation unit 42 generates a color image by demosaicing the image of the corrected Bayer array (S26).
  • the output unit 44 sequentially outputs the data of the color image and the distance image to an external device or a memory (S28). At this time, data of the normal image may be output simultaneously.
  • the output target may be switched according to the request from the output destination device. If it is not necessary to end photographing or data output by a user operation or the like, the processing from S10 to S28 is repeated for each image frame (N in S30). If it is necessary to end the process, all the processes are ended (Y in S30).
  • the distance image generation unit 38 When the distance image generation unit 38 generates a distance image in S22, the distance images generated for a predetermined number of plural image frames are accumulated, and data obtained by averaging them is used as the distance image at that time.
  • the output may be performed at time intervals corresponding to a plurality of image frames. As a result, the ratio of noise components included in the distance image generated from one image frame can be reduced, and a distance image with high accuracy can be output.
  • the optimum number of frames for storing the distance image is determined by experiment or the like in consideration of the required accuracy and time resolution. Alternatively, the number of frames may be adaptively changed according to the luminance level of the actual captured image or the like.
  • the imaging device of this embodiment includes a structure in which a polarizer is provided on the upper layer of the pair of photodiodes.
  • a polarizer is provided on the upper layer of the pair of photodiodes.
  • FIG. 11 is a diagram for explaining the positional relationship between the polarizer and the photodiode. The figure shows a cross section of the laminated structure of the polarizer layers 230a, 230b, 230c and the photodiode pairs 232a, 232b, 232c and their positional relationship 234a, 234b, 234c viewed from the top.
  • FIG. 12 schematically shows a wire arrangement of polarizers at different positions on the imaging surface.
  • the shape of the polarizer is changed according to the position on the imaging device 20 to make the detection sensitivity in the photodiode uniform.
  • the loss of incident light is reduced by narrowing the wire width of the polarizer of the pixels 240b and 240c in the peripheral portion as compared to the pixel 240a in the central portion.
  • the wire width is gradually narrowed according to the distance from the center.
  • the wire height may be lowered or both the width and height may be changed according to the distance from the center.
  • the entire arrangement of the wires may be shifted by a small amount in line symmetry with respect to the center line.
  • the pixel 240b on the left side on the imaging device 20 shifts the entire wire array to the left
  • the pixel 240c on the right side shifts the entire wire array to the right.
  • the amount of incident light can be increased according to the angle.
  • the width, height, and shift amount of the wire optimize the values so as to minimize the in-plane distribution in the actual photographed image. At this time, as described in FIG. 11, the sensitivity difference between the pair of photodiodes is also minimized.
  • FIG. 13 and FIG. 14 are diagrams for explaining the unit of data and the generation path of various information in the case of making the reading unit different depending on the presence or absence of the polarizer.
  • each detection value is summed up and used as a pixel unit value 266 (S54) to be used for generating a polarized image or to determine the color value of the pixel in a color image (S56, S58) ).
  • polarization information and phase difference information are acquired only from the pixel provided with the polarizer.
  • the detection values summed up in the pixel 262 provided with a polarizer are used as they are for generating a polarization image or for determining the color value of the pixel in a color image (S66, S68).
  • the phase difference information is acquired from other than the pixel provided with the polarizer.
  • the sensitivity of the phase difference information can be increased as compared with the case of FIG. 13, the accuracy can be maintained even in an environment where the illuminance is low. By thus changing the reading unit depending on the presence or absence of the polarizer, it is possible to reduce the reading time while acquiring necessary information.
  • the value of a pixel provided with a polarizer may not be used to generate a color image, and pixel values in the periphery without a polarizer may be interpolated.
  • the reading unit and the data generation path may be optimized according to the accuracy and resolution required for various data, the surrounding illumination environment, the limitation of processing time, and the like.
  • the illuminance may be measured or the communication environment for data transmission may be measured, and switching may be performed according to the result.
  • FIG. 15 shows variations of polarizers at pixel values for which polarizers are provided.
  • polarizers with one main axis angle are provided for one pixel as described above. And, by changing the main axis angle every 45 °, four types of pixels as illustrated are obtained. These pixels are arranged at equal intervals or in proximity to each other on the imaging device 20. Since dispersing pixels where a polarizer is provided disperses pixels whose luminance level is lowered due to reflection, when a phase difference image or a color image is generated by pixels which are not provided with a polarizer, holes are accurately filled by interpolation. Can.
  • the entire imaging device 20 with pixels as shown in (b).
  • polarizers having different principal axis angles are provided in each of the regions corresponding to the pair of photodiodes.
  • two types of pixels are illustrated, each of which is a pair of polarizers whose principal axis angles differ by 90 °.
  • the process of acquiring the distance value from the phase difference is based on comparing the distributions of detection values by the left and right photodiodes.
  • the process of acquiring a normal vector from polarization information includes a process of comparing detected values of light transmitted through polarizers having different principal axis angles. Therefore, in the case of using a polarizer as shown in (c), the processing for acquiring the phase difference and the processing for acquiring the normal can be shared in that the detection values of the left and right photodiodes are compared.
  • FIG. 16 shows a variation of the photodiode provided in one pixel.
  • the photodiodes are arranged one by one in the left and right areas obtained by dividing the pixel area in the vertical direction.
  • the phase difference appears only in the horizontal direction of the image plane. Therefore, as described in FIG. 9, the phase difference becomes indeterminate with respect to some feature points such as an edge in the horizontal direction, and the distance value can not be obtained. Therefore, as illustrated, one photodiode may be disposed in each of four regions formed by dividing one pixel (for example, the pixel 280) into two in the vertical and horizontal directions.
  • the detection values of two vertically adjacent photodiodes are summed, the same phase difference image as that of the left photodiode and the right photodiode described above can be obtained, and the phase difference of the horizontal direction component can be acquired.
  • the detection values of two adjacent photodiodes are summed, a phase difference image can be obtained by the upper photodiode and the lower photodiode, and the phase difference of the vertical direction component can be acquired. As a result, the distance value can be obtained regardless of the direction of the feature point.
  • a polarizer with one principal axis angle is provided in each pixel, but as shown in FIG. 15, the principal axis angle of the polarizer is made different for each photodiode, and the presence or absence of the polarizer is controlled.
  • a photodiode may be provided for each of the four divided regions also for the pixel without a polarizer. Such non-polarizer pixels and certain pixels may be periodically arranged.
  • the color of the color filter may be the same in the pixel, or may be different for each photodiode.
  • one photodiode may be provided in each of the upper and lower areas divided in the lateral direction, or the pixel area may be divided into smaller than two rows and two columns, and the photodiodes may be arranged in each area.
  • FIG. 17 shows the configuration of functional blocks of the system when the imaging device is configured by a stereo camera.
  • This system includes an imaging device 300 and an information processing device 302.
  • the imaging device 300 includes a first imaging unit 12a and a second imaging unit 12b.
  • the first imaging unit 12a and the second imaging unit 12b correspond to the imaging device 12 shown in FIG. 1, respectively, and arrange them in the left and right so as to have a predetermined interval, thereby forming an imaging device 300.
  • the first imaging unit 12a is a left viewpoint
  • the second imaging unit 12b is a right viewpoint camera.
  • Each of the first imaging unit 12a and the second imaging unit 12b includes an image processing unit having the function shown in FIG. Therefore, the first imaging unit 12a of the imaging device 300 outputs the data of the distance image and the color image of the left viewpoint, and the second imaging unit 12b outputs the data of the distance image and the color image of the right viewpoint.
  • the information processing apparatus 302 acquires an image data acquisition unit 304 that acquires image data from the imaging apparatus 300, a subject information generation unit 306 that integrates the information to generate comprehensive information related to the position and orientation of the subject, and the information And an output data generation unit 308 that generates output data using the
  • the image data acquisition unit 304 acquires, from the imaging device 300, data of distance images and color images acquired for at least each of the left and right viewpoints.
  • the subject information generation unit 306 generates final information on the position, posture, shape, and the like of the subject by integrating the distance images acquired from the imaging device 300. That is, with respect to a portion of the subject which can not be seen from one of the viewpoints of the first imaging unit 12a and the second imaging unit 12b, data is compensated using the other distance image to minimize the portion where the distance is indefinite. .
  • the subject information generation unit 306 may further generate and integrate a distance image separately according to the principle of triangulation using color images of left and right viewpoints or luminance images of left and right viewpoints.
  • the output data generation unit 308 generates data to be output, such as a display image, using the color image and the distance image of the left and right viewpoints.
  • general processing such as linear matrix (color matrix) and gamma correction is performed at the time of output, and output to the display device.
  • FIG. 18 is a diagram for describing processing in which the subject information generation unit 306 integrates distance images of left and right viewpoints. As shown in the upper part of the figure, when the three-dimensional space 320 in which two cubes 322a and 322b exist is photographed from the left and right viewpoints L and R, a left viewpoint image 324a and a right viewpoint image 324b are obtained.
  • the area where the first imaging unit 12a and the second imaging unit 12b can independently obtain distance values is limited to a portion appearing as an image in the left viewpoint image 324a and the right viewpoint image 324b, respectively.
  • the left side of the cube 322b is seen only from the left viewpoint L
  • the right side of the cube 322a is seen only from the right viewpoint R, so their distance values are included in only one of the distance images. Therefore, the subject information generation unit 306 reduces the area where the distance value is indeterminate by applying the value of the other distance image to the area on the subject whose value is not obtained in one of the distance images.
  • the subject information generation unit 306 can generate information related to the position of the subject in the world coordinate system whose number of viewpoints is not limited to one by integrating the distance images of the plurality of viewpoints. The position is obtained for each minute area on the surface of the subject, and as a result, the posture and the shape of the subject are also obtained.
  • the accuracy can be enhanced by using the average value of them as the distance value.
  • the subject information generation unit 306 may itself generate a distance image using color images of left and right viewpoints, and may further integrate the results. In this case, further distance values can be obtained for regions viewed from both viewpoints, and as a result, three distance values can be obtained for the regions. If the average value of them is used as the distance value, the accuracy can be further improved. However, depending on the required accuracy, the processing time can be shortened by omitting the generation of a distance image using a color image.
  • the subject information generation unit 306 may further fill in the hole of the distance value by another means or further improve the accuracy.
  • a deep learning technique is being put to practical use as machine learning using a neural network.
  • the subject information generation unit 306 is made to learn so that the distance value and the change thereof can be derived from the color in the color image and the change thereof, the shape of the image, and the like. Then, using the color image that is actually acquired, the distance value of the region that can not be seen from the viewpoint of the imaging device may be estimated, or the distance value of the viewed region may be corrected to improve the accuracy.
  • this method is provided with a subject information generation unit having the same function in an information processing apparatus (not shown) connected to the imaging device 12. It is also good. This function is particularly effective in expanding the area where the distance value can be obtained or enhancing the accuracy in the case where the viewpoint of the imaging device is limited or the photographing environment where the luminance is not sufficient.
  • the function of the information processing apparatus 302 including the subject information generation unit 306 or a part of the functions of the imaging apparatus 300 may be provided to another apparatus connected to the network or a plurality of apparatuses may share the operation. You may do it. At this time, the information processing apparatus 302 or a display apparatus (not shown) may sequentially acquire the results, and appropriately perform its own processing or display an image accordingly.
  • FIG. 19 is a diagram for describing a method of acquiring state information such as the position, posture, and shape of a subject in a three-dimensional space by shooting while moving the imaging device 12.
  • the illustrated example shows how the imaging device 12 is moved along a circular orbit centered on a cube that is a subject.
  • an acceleration sensor is provided in the imaging device 12, and the imaging time, the captured image, and the position and orientation of the imaging device 12 in a three-dimensional space are associated with each other and recorded. Then, based on the color image and the distance image obtained for the viewpoint at each shooting time of a predetermined rate, a model space such as the three-dimensional space 320 of FIG. 18 is filled with the acquired data.
  • Such processing may be performed by the distance image generation unit 38 inside the imaging device 12 or may be performed by the subject information generation unit 306 of the information processing device 302.
  • the processing load on the information processing apparatus 302 can be reduced, and an increase in processing time can be suppressed.
  • how to move the imaging device 12 is not limited to that illustrated.
  • the imaging device 12 may be moved in a range corresponding to the movable range of the virtual viewpoint with respect to the image to be finally displayed.
  • the photographed image may be obtained in all directions by rotating the imaging device 12. Further, among the data acquired while moving the imaging device 12 as described above, memory consumption can be suppressed by devising that only the value for the feature point is accumulated for the distance value.
  • the imaging device 12 instead of moving the imaging device 12, similar information can be obtained by arranging three or more photographed images. Also in this case, the plurality of imaging devices 12 are installed to face each other so that the optical axis converges in the vicinity of the subject, as illustrated. Alternatively, the imaging device 12 may be installed in the opposite direction so that the optical axis diverges outward. In these cases, color images and range images at the same time can be obtained in a plurality of fields of view. It may be connected by stitching processing to obtain wide-angle information. At this time, only a part of the plurality of installed imaging devices may be the imaging device 12 having the functional block shown in FIG.
  • a function of generating a distance image is provided, and the other imaging apparatuses generate only a color image.
  • processing resources can be concentrated on necessary targets, such as processing at a later stage such as processing and superimposing virtual objects with high accuracy.
  • FIG. 20 shows functional blocks of an imaging apparatus having a function of focusing using a phase difference of polarization.
  • the imaging device 400 includes a pixel value acquisition unit 402 for acquiring detection values by each photodiode, a polarization phase difference detection unit 404 for detecting a phase difference of a polarization image from detection values by two photodiodes of pixels provided with a polarizer. And a focusing unit 406 that adjusts the position of the lens based on the phase difference of polarization and focuses on the appropriate position.
  • the pixel value acquisition unit 402 reads out a detection value by a photodiode in at least a pixel provided with a polarizer, and performs predetermined preprocessing such as A / D conversion and clamping processing.
  • the polarization phase difference detection unit 404 separates the polarization luminance distribution detected by the left photodiode and the right photodiode, and generates polarization images of four directions for each. Then, a polarization degree image representing the polarization degree obtained using Expression 2 on the image plane or a normal image representing the normal vector obtained from the polarization degree on the image plane are generated as a phase difference image.
  • the figure shows the phase contrast images 410a, 410b with polarization generated as such.
  • general natural light phase difference images 412 a and 412 b are shown for comparison.
  • a disk-shaped object is shown.
  • the outlines of the subject are obtained as feature points in the general natural light phase difference images 412a and 412b, while the information on the subject surface is scarce.
  • the change in luminance may be small and may not be regarded as a feature point. Therefore, when specifying the position of the feature point of these images 412a and 412b as shown by the arrow and focusing from the phase difference, it is conceivable that accurate adjustment can not be performed due to the lack of information.
  • the phase difference images 410a and 410b representing the degree of polarization or the normal vector represent the unevenness of the object surface, so they have higher sensitivity to the shape than the image of natural light and are less susceptible to illumination. Therefore, even if it looks like a uniform image, changes corresponding to the shape appear as an image as shown. Therefore, as indicated by the arrows, more positions of feature points on which the phase difference is based can be obtained. If these positional relationships are integrated to derive a phase difference and focusing is performed based thereon, more accurate and quick adjustment can be realized.
  • the focusing unit 406 derives an appropriate position of the lens based on the phase difference and performs adjustment, as in general focusing processing.
  • the illustrated imaging device 400 shows a functional block focusing only on the focusing function, but by combining with the image processing unit 22 shown in FIG. 8, a distance based on luminance data obtained by focusing with high accuracy It may be possible to output an image or a color image.
  • the image pickup device a plurality of photodiodes are provided for one microlens, and a polarizer is provided in an intermediate layer of at least a part of the microlens and the photodiode. Do. Thereby, the polarization image and the phase difference image can be simultaneously acquired. Then, the distance at the feature point of the subject is obtained based on the phase difference, and the distance between the feature points is complemented using the normal vector obtained from the polarization to obtain a distance value with respect to a wide area of the photographed image You can get
  • the distance on the object on the subject which is captured in the image of one viewpoint but not captured in the image of the other viewpoint becomes indefinite.
  • the distance can be derived if it appears in the captured image, so in some cases more distance data can be obtained using a stereo camera. Therefore, it can be used as a substitute for a stereo camera, and the imaging device having a distance measuring function can be miniaturized.
  • the results of the left and right photographed images can be integrated, so that a wider range of distance values can be obtained, and the position and orientation of the subject in three-dimensional space can be accurately reproduced.
  • the accuracy of distance information can be further improved by obtaining and integrating distance images as in the prior art using color images of left and right viewpoints. Since these methods do not depend on light of a specific wavelength band such as infrared light, information can be similarly obtained outdoors.
  • the present invention can be applied to all types of information processing without restriction on the processing of the latter stage.
  • acquisition of distance values based on phase difference generation of a normal image based on polarization, and processing of integrating them to generate a distance image can basically be performed in row units or several units of image planes.
  • the line buffer can be implemented by an arithmetic circuit in the imaging apparatus. Therefore, it is possible to share the function with an apparatus that performs information processing and display processing using various data, and to cope with photographing and display at a high frame rate.
  • phase difference of polarization since it is possible to acquire the phase difference of polarization, it is possible to extract the change of the shape as the feature point with high sensitivity even if the object surface is a rough surface that is not extracted as the feature point in the natural light image. Therefore, it is possible to obtain much information as the basis of the phase difference, and it is possible to further improve the accuracy of the conventional focusing function. Even in the case of a stereo camera, more feature points can be obtained than the luminance image of natural light by using the polarization degree image and the normal image, and thus a distance image by acquiring corresponding points from images of left and right viewpoints The generation accuracy of can also be enhanced.
  • the detection subject is not limited to the photodiode as long as it is a mechanism for converting light into charge.
  • some or all of the photodiodes may be used as the organic photoelectric conversion film.
  • the material and structure of the organic photoelectric conversion film can be appropriately determined by using a known technique described in WO 2014/156659 and the like.
  • a distance measurement technology by irradiating light of a predetermined wavelength band such as infrared light. That is, a mechanism for irradiating the reference light to the imaging device 12 is provided, and the reflected light is detected by the photodiode. By irradiating the reference light in a random pattern, it is possible to create feature points even on the surface of an object with a few feature points.
  • the processing in the image processing unit is the same as that of the present embodiment, but there are many feature points that are the basis of the phase difference, so distance values based on the phase difference can be acquired at more locations. Therefore, the accuracy of the complementation using the normal vector is improved, and the distance information can be obtained more accurately.
  • An illuminance sensor may be further provided in the imaging device 12 to irradiate the reference light when the illuminance is lower than a predetermined value to prevent the deterioration of analysis accuracy due to the illuminance decrease.
  • the imaging device in the present embodiment may be realized by a general camera whose main function is acquisition of a color image, or may be provided in another device having an imaging function.
  • it may be provided in a high-performance mobile phone, a portable terminal, a personal computer, a capsule endoscope, a wearable terminal and the like.
  • the functions of the defect correction unit 40 and the color image generation unit 42 may be omitted, and only the distance image may be output.
  • the color filter layer of the imaging device may be omitted.
  • all pixel regions are divided into partial regions, and photodiodes are arranged respectively.
  • one photodiode is made to correspond to one microlens. May be included.
  • one photodiode may be provided for a pixel provided with a polarizer. In this case, the phase difference image is acquired from the other pixels.
  • a plurality of photodiodes may be provided only for pixels provided with a polarizer. In any case, it is possible to obtain the same effect by omitting the process of summing the detection values described in the present embodiment.
  • Embodiment The embodiment of the present invention can be appropriately combined with the above-described related art.
  • first to fifth examples will be described as embodiments of the present invention.
  • FIGS. 21 (a) and (b) and FIGS. 22 (a) and (b) show the arrangement of cameras in the related art and the arrangement of cameras in the first embodiment in comparison.
  • 21 (a) and (b) and FIGS. 22 (a) and (b) show the appearance of the camera viewed from the front (in other words, the object side), that is, the lens arrangement of the camera is schematically shown. It shows.
  • FIG. 21A schematically shows an arrangement of cameras in the prior art.
  • the two large cameras 502 have an inter-pupil distance IPD (typically 60 mm). ⁇ 70 mm) may have been set apart.
  • IPD inter-pupil distance
  • the size of the imaging device indicated by the broken line in other words, the size of the camera system may be increased, which may cause a problem.
  • FIG. 21B schematically shows the arrangement of the cameras in the first embodiment.
  • a plurality of small cameras 504 for imaging the same subject are provided on the left and right of one large camera 502 for imaging a certain subject.
  • the large camera 502 has a relatively large optical size
  • the small camera 504 has a relatively small optical size.
  • large camera 502 has a larger lens size than small camera 504.
  • images captured by two small cameras 504 provided apart from each other by IPD can be acquired as parallax images, and the image quality of parallax images can be improved using images captured by the large camera 502. it can.
  • FIG. 22 (a) also schematically shows the arrangement of cameras in the prior art.
  • the large cameras 502 need to be arranged in the vertical direction, and the size of the imaging device is further increased.
  • the vertical inter-camera distance may exceed the IPD, ie, problems with size and IPD limitations may have occurred.
  • FIG. 22B also schematically shows the arrangement of the cameras in the first embodiment.
  • a plurality of small cameras 504 are provided on the left, right, upper and lower sides of the large camera 502. According to this aspect, it is possible to obtain upper and lower parallax images in addition to the left and right parallax images while suppressing an increase in the size of the imaging device 500.
  • the imaging apparatus 500 generates a parallax image and / or a wide-angle image to be displayed on a head mounted display (hereinafter, also referred to as “HMD”).
  • FIG. 23 is a block diagram showing a functional configuration of the imaging device 500 of the embodiment.
  • the imaging device 500 includes one large-sized imaging unit 510, two small-sized imaging units 512, and an image processing unit 514.
  • each element described as a functional block that performs various processes can be configured by hardware as a circuit block, a memory, or another LSI, and software can be configured as a memory. It is realized by the program etc. which were loaded to. Therefore, it is understood by those skilled in the art that these functional blocks can be realized in various forms by hardware only, software only, or a combination thereof, and is not limited to any of them.
  • a computer program including a plurality of modules corresponding to a plurality of functional blocks in FIG. 23 may be installed in the imaging device 500 storage.
  • the CPU or GPU of the imaging device 500 may exert the function of each functional block by reading out the computer program to the main memory and executing it.
  • the large image capturing unit 510 corresponds to the large camera 502 in FIG. 21B and captures an image of an object present in front of the image capturing apparatus 500.
  • the subject includes, for example, a three-dimensional space and an object present there.
  • the compact imaging unit 512 corresponds to the compact camera 504 in FIG. 21B and captures an image of the subject.
  • the optical size of the large image pickup unit 510 is larger than the optical size of the small image pickup unit 512, and the large image pickup unit 510 can obtain an image with higher image quality than the small image pickup unit 512.
  • two small imaging units 512 are disposed in proximity to the left and right of the large imaging unit 510.
  • a plurality of small-sized imaging units 512 may be disposed above and below (or at an oblique position) the large-sized imaging unit 510.
  • three or more compact imaging units 512 may be disposed in the direction in which parallax should be obtained.
  • the image processing unit 514 generates data to be output to an external device based on the image captured by the large-sized imaging unit 510 and the image captured by the small-sized imaging unit 512.
  • the image processing unit 514 includes a characteristic storage unit 520, a pixel data acquisition unit 522, an adjustment unit 524, a generation unit 526, and an output unit 528.
  • the characteristic storage unit 520 stores data (hereinafter, also referred to as “characteristic data”) regarding the characteristic of the image captured by the large-size imaging unit 510.
  • the characteristic storage unit 520 stores characteristic data of an image captured by each of the plurality of small-sized imaging units 512.
  • the characteristic data can also be said to be data indicating the characteristic of the pixel value acquired from the imaging element of each imaging unit.
  • the characteristic data may also include data indicating at least one feature or tendency of hue, lightness, and saturation.
  • the characteristic data is data indicating the content or degree of difference when the characteristic of the image picked up by the large image pickup unit 510 and the characteristic of the image picked up by each of the plurality of small image pickup units 512 are different. May be included.
  • the characteristic data stored in the characteristic storage unit 520 may indicate the characteristic calculated from the data after shooting, but the camera (each of the large image pickup unit 510 and the small image pickup unit 512) or the image pickup element In the test process at the time of manufacturing the (image sensor), the characteristics of the image may be measured and stored in the ROM of the camera or the imaging device (characteristic storage unit 520).
  • characteristic measurement in a test process in a test process after manufacturing a single camera or an imaging device, signals from pixels, noises, and the like obtained when lenses, illuminance, wavelengths, and the like are changed may be measured.
  • the deviation of the stitching position, the contrast, and the color difference after combining a plurality of cameras (for example, the large imaging unit 510 and the small imaging unit 512) It may be measured.
  • the pixel data acquisition unit 522 acquires data of an image captured by the large image capturing unit 510, that is, a value of each pixel. In addition, the pixel data acquisition unit 522 acquires data of an image captured by each of the plurality of small-sized imaging units 512, that is, a value of each pixel.
  • the characteristics of the image picked up by the large image pickup unit 510 may be different from the characteristics of the image picked up by each of the plurality of small image pickup units 512.
  • an image captured by the large-sized imaging unit 510 may be reddish, while an image captured by the small-sized imaging unit 512 may be bluish.
  • the color of the image obtained by combining the two images may be unnatural. Therefore, the adjustment unit 524 determines the characteristics of the image captured by the large-sized imaging unit 510 and the characteristics of the images captured by the plurality of small-sized imaging units 512 based on the characteristic data stored in the characteristic storage unit 520. Adjust to match.
  • the adjustment unit 524 matches the characteristics of the image captured by each of the plurality of small-sized imaging units 512 with the characteristics of the image captured by the large-sized imaging unit 510.
  • the images taken by each of the above are corrected.
  • the property storage unit 520 may store property data indicating a difference between the property of the image captured by the large-size imaging unit 510 and the property of the image captured by each of the plurality of small-size imaging units 512.
  • the adjustment unit 524 may adjust the pixel values acquired from each of the plurality of small-sized imaging units 512 so as to offset the difference indicated by the characteristic data.
  • the image of the large-sized imaging unit 510 and the image of the small-sized imaging unit 512 it is possible to generate a natural-looking image based on both images.
  • the image quality is degraded when the image is corrected. Therefore, the high-quality image captured by the large-size imaging unit 510 is corrected by correcting the low-quality image captured by the small-size imaging unit 512 without touching the high-quality image captured by the large-size imaging unit 510. It is possible to suppress the image quality deterioration of the image. As a result, it is possible to suppress the deterioration of the image quality also for the image obtained by combining the image of the large image pickup unit 510 and the image of the small image pickup unit 512. In particular, it is suitable for displaying the image of the large-sized imaging unit 510 inside the field of view of the user (that is, in a region where the resolution of the user's eyes is high).
  • the adjustment unit 524 inputs, to the generation unit 526, data of the image captured by the large-size imaging unit 510 and data of the image captured by each of the plurality of small-size imaging units 512 (data after the above correction).
  • the generation unit 526 generates data to be output to an external device based on the input data of the plurality of types of images. In the embodiment, a VR image, an AR image, or a wide angle image is generated.
  • the output unit 528 transmits the data generated by the generation unit 526 to a predetermined external device (such as an information processing device).
  • the output unit 528 of the embodiment transmits the VR image, the AR image, or the wide-angle image generated by the generation unit 526 to the HMD, and displays the image on the screen of the HMD.
  • the output unit 528 may store the data generated by the generation unit 526 in a predetermined storage device or recording medium.
  • the generation unit 526 may combine the image captured by the large image capturing unit 510 with the image captured by the plurality of small image capturing units 512, and may output a wide-angle image as a result of combining to the output unit 528.
  • FIG. 24 shows an example of image composition.
  • the central image 530 is a relatively high quality image captured by the large image capturing unit 510.
  • the left image 532 a is an image of relatively low image quality captured by the compact imaging unit 512 installed on the left side of the large imaging unit 510.
  • the right image 532 b is an image of relatively low image quality captured by the small imaging unit 512 installed on the right side of the large imaging unit 510.
  • low quality areas are hatched.
  • the generation unit 526 may generate the combined image 534 by combining the left image 532a on the left side of the center image 530 and combining the right image 532b on the right side of the center image 530.
  • a composite image 534 a wide-angle image in which an occlusion area that can not be captured by the large-size imaging unit 510 is complemented.
  • the resolution of the human eye is higher at the center and lower at the periphery.
  • the central portion of the composite image 534 is high quality, ie, it can provide a wide angle image suitable for human eyes.
  • the generation unit 526 may change the combining ratio (in other words, the blending ratio) according to the position at which the image is combined. For example, the generation unit 526 may increase the reflection ratio of the pixel value of the left image 532a as approaching the peripheral region in the overlapping portion of the central image 530 and the left image 532a (the same applies to the right image 532b). Conversely, as the central region is approached, the reflection ratio of the pixel values of the central image 530 may be increased. Thereby, it is possible to suppress the user from feeling uncomfortable with the composite image 534 due to the difference in the image quality between the center image 530 and the left image 532a (the same applies to the right image 532b).
  • the combining ratio in other words, the blending ratio
  • the generation unit 526 is based on at least two of the image captured by the large image capturing unit 510 and the plurality of images captured by the plurality of small image capturing units 512, data relating to parallax and data relating to the distance to the subject. And / or may be generated. Data on distance can be said to be depth information.
  • the generating unit 526 causes the HMD to display an image generated by each of the two small imaging units 512. It may be determined as For example, the generation unit 526 generates an image captured by the left small-size imaging unit 512 (referred to as “left image”) as an image for the left eye and an image captured by the right small-sized imaging unit 512 (referred to as the “right image”). ) May be determined as an image for the right eye.
  • left image an image captured by the left small-size imaging unit 512
  • right image an image captured by the right small-sized imaging unit 512
  • the generation unit 526 may store the correspondence between the image (referred to as “center image”) captured by the large-size imaging unit 510 and the left image. This correspondence may be, for example, the correspondence between the pixels of one image and the pixels of the other image, or may be the correspondence between pixels imaging the same portion (position) of the subject.
  • the generation unit 526 may correct the value of each pixel in the left image based on the value of the corresponding pixel in the center image, or may replace it, for example.
  • the generation unit 526 may store the correspondence between the right image and the center image, and correct the value of each pixel in the right image based on the value of the corresponding pixel in the center image.
  • the image quality of the left eye image and the right eye image can be improved.
  • the generation unit 526 may newly generate an image for the left eye and an image for the right eye based on the data of the center image, the data of the left image, and the data of the right image.
  • the distance between the left small-sized imaging unit 512 and the right small-sized imaging unit 512 may be different from the IPD, and may be, for example, 100 mm or more.
  • FIG. 25 is a flowchart showing processing of the generation unit 526 in the first embodiment.
  • the generation unit 526 detects the parallax between the left image and the right image (in other words, the size of the deviation of the imaging position of the subject). In addition, the generation unit 526 detects the parallax between the left image and the center image. The generation unit 526 also detects the parallax between the right image and the center image (S10). The generation unit 526 estimates the distance to the subject based on the parallax detected in S10 (S12).
  • the generation unit 526 may generate first distance data indicating the distance to the subject by triangulation based on the parallax between the left image and the right image. Further, the generation unit 526 may generate second distance data indicating the distance to the subject by triangulation based on the parallax between the left image and the center image. The generation unit 526 may also generate third distance data indicating the distance to the subject by triangulation based on the parallax between the right image and the center image. The generation unit 526 may generate a distance image representing the distance to the subject as a pixel value as the first to third distance data. The generation unit 526 may generate a final estimated value of distance by smoothing the first to third distance data.
  • the generation unit 526 obtains RGB data of each of the left image, the center image, and the right image (S14), and generates a composite image (for example, the composite image 534 in FIG. 24) obtained by combining the left image, the center image, and the right image S16).
  • a composite image for example, the composite image 534 in FIG. 24
  • the generation unit 526 generates parallax information conforming to the position of the eye of the user wearing the HMD by inverse calculation of triangulation based on the distance to the subject (S18).
  • the generation unit 526 generates an image for the left eye and an image for the right eye based on the parallax information obtained in S18 (S20). For example, the image for the left eye and the image for the right eye having a shift indicated by the parallax information generated in S18 may be extracted from the composite image generated in S16.
  • the generation unit 526 outputs the left-eye image and the right-eye image to the output unit 528, and causes the HMD to display the image (S22).
  • the generation unit 526 may output the distance image to the output unit 528, and the output unit 528 may transmit the distance image to a predetermined external device.
  • the imaging apparatus 500 of the first embodiment it is possible to obtain parallax images and wide-angle images in various directions including the vertical direction while suppressing the enlargement of the apparatus. In addition, it is easy to secure the image quality of the image while suppressing the increase in size of the device.
  • FIG. 26 schematically illustrates the configuration of an imaging device 500 according to the second embodiment.
  • the figure shows the configuration when the imaging device 500 is viewed from the top.
  • the imaging device 500 includes a plurality of large-sized imaging units 510 (three in FIG. 26) and a plurality of small-sized imaging units 512 (two in FIG. 26).
  • the plurality of large-size imaging units 510 capture subjects in directions different from one another, and in FIG.
  • At least one small-sized imaging unit 512 (all small-sized imaging units 512 in the second embodiment) is provided between a plurality of large-sized imaging units 510.
  • the generation unit 526 generates a wide-angle image (an image of 180 ° in FIG. 26) obtained by combining the plurality of images captured by the plurality of large-sized imaging units 510 and the plurality of images captured by the plurality of small-sized imaging units 512. May be
  • the angle of view of the large image pickup unit 510 is indicated by a broken line
  • the angle of view of the small image pickup unit 512 is indicated by an alternate long and short dash line.
  • the lens end portions 542 of the plurality of large image pickup units 510 are configured to be closer to the subject than the lens end portions 544 of the plurality of small image pickup units 512.
  • the circle connecting the lens front ends 544 of the plurality of small imaging units 512 is configured to be inside (the radius becomes smaller) than the circle connecting the lens front ends 542 of the plurality of large imaging units 510 . This prevents the small imaging unit 512 from being included in the angle of view of the large imaging unit 510, in other words, prevents the small imaging unit 512 from appearing in the high-quality image imaged by the large imaging unit 510. it can.
  • FIG. 27 schematically illustrates the configuration of an imaging device 500 according to the third embodiment.
  • the figure schematically shows the configuration when the imaging device 500 is viewed from the front.
  • the imaging apparatus 500 includes a plurality of large imaging units (large imaging units 510a and 510b) and a plurality of small imaging units (small imaging units 512a to 512f).
  • the large image pickup unit 510a and the large image pickup unit 510b image the same direction, in other words, an image of an object existing in the same direction.
  • the small imaging unit 512a, the small imaging unit 512b, and the small imaging unit 512c are disposed around the large imaging unit 510a, and the small imaging unit 512d, the small imaging unit 512e, and the small imaging unit 512f are disposed around the large imaging unit 510b. Be done.
  • Each of the small size imaging unit 512a to the small size imaging unit 512f captures an image in the same direction as the large size imaging unit 510a and the large size imaging unit 510b.
  • the small imaging unit may be further disposed at an oblique position of the large imaging unit.
  • the large image pickup unit 510a and the large image pickup unit 510b are provided at separate positions of the IPD.
  • the generation unit 526 generates an image for the right eye based on the high quality image captured by the large image pickup unit 510a, and generates an image for the left eye based on the high quality image captured by the large image pickup unit 510b.
  • the distance between the large imaging unit 510 a and the large imaging unit 510 b may be different from that of the IPD. In that case, the distance to the subject is determined based on the high-quality images captured by the plurality of large-sized imaging units and the low-quality images captured by the plurality of small-sized imaging units. An image for the left eye may be generated.
  • the imaging device 500 is an HMD based on an output signal of a sensor (acceleration sensor, gyro sensor, etc.) mounted on the HMD and / or an image of the appearance of the HMD captured by a predetermined camera. It further comprises an attitude detection unit that detects an attitude.
  • the posture detection unit detects, as the posture of the HMD, the gaze direction of the user wearing the HMD and the inclination of the gaze of the user (in other words, the inclination of a line connecting both eyes).
  • the generation unit 526 of the imaging device 500 selects a part of the plurality of images captured by the plurality of imaging units according to the gaze direction detected by the posture detection unit and / or the inclination of the gaze. May be
  • the generation unit 526 may generate the right-eye image and the left-eye image based on the selected image. For example, when the line of sight of the user is inclined obliquely to the left, the upper side, and the right, the lower side, the generation unit 526 generates an image for the left eye using an image captured by the compact imaging unit 512a.
  • the captured image may be used to generate an image for the right eye.
  • the generation unit 526 may improve the image quality of the image for the left eye and the image for the right eye using images captured by the large image capturing unit 510a and the large image capturing unit 510b. .
  • the generation unit 526 may generate an image for the left eye using a captured image by the small-size imaging unit 512b when the line of sight direction of the user has moved further to the left. For example, the generation unit 526 derives the distance data to the subject for generating the left-eye image and the color data of the subject using the image captured by the large image capturing unit 510a and the image captured by the small image capturing unit 512b. May be
  • the generation unit 526 may generate an image for the right eye using an image captured by the compact imaging unit 512e when the line of sight direction of the user has moved further to the right.
  • the small image pickup units at the upper, lower, left, and right positions of each of the plurality of large image pickup units, appropriate parallax according to the change in the gaze direction of the user and the change in the inclination of the gaze. It becomes easy to present the image to the user.
  • a plurality of images obtained by imaging the subject by a plurality of imaging units are required.
  • the occlusion of parallax generated in the large image pickup unit is interpolated by the image pickup data by the small image pickup unit.
  • the generation unit 526 sets the distance to the subject in the occlusion area not included in the image captured by the large image capturing unit 510 b in the image captured by the large image capturing unit 510 a as the distance from the large image capturing unit 510 a. Derivation is performed based on an image captured by the small imaging unit 512a, the small imaging unit 512b, or the small imaging unit 512c installed in the periphery.
  • the distance to the subject in the area included in at least one of the image captured by the large image capturing unit 510a and the image captured by the large image capturing unit 510b can be obtained without omission.
  • the imaging apparatus 500 of the fourth embodiment enhances the quality of output data by using the result of machine learning (deep learning or the like).
  • the imaging apparatus 500 of the fourth embodiment further includes a learning result storage unit (not shown) that stores the result of machine learning.
  • the learning result storage unit includes (1) a first learning result for obtaining a distance to a subject based on a plurality of images obtained from a plurality of imaging units, and (2) an image captured by a small imaging unit 512.
  • the second learning result to be corrected by the image captured by the large image capturing unit 510 is stored.
  • the first learning result is a result of machine learning based on a combination of the parallax between the plurality of images obtained from the plurality of imaging units and the distance to the subject, and the parallax between the plurality of images is input May be a program that outputs the distance to the subject. Further, the first learning result may be data indicating a correspondence between parallax between a plurality of images and a distance to a subject. The first learning result may be a program that outputs the distance to the subject based on the RGB information of the captured image as disclosed in “Japanese Patent Application Laid-Open No. 2016-157188”.
  • the second learning result when the subject is an object moving from the imaging range of the large imaging unit 510 to the imaging range of the small imaging unit 512, the shape of the object indicated by the captured image (for example, central image) of the large imaging unit 510 And the program for setting the image of the object with respect to the captured image (for example, the left image or the right image) of the compact imaging unit 512. Also, the second learning result identifies the object based on the shape of the object of the center image 530 when the same object appears across the center image, the left image, and the right image, and the shape that the object should have May be a program that reflects the left image and the right image.
  • the technology related to the former second learning result is also disclosed in “Japanese Patent Laid-Open No. 2012-203439”.
  • the techniques related to the latter second learning result are also disclosed in "Japanese Patent Application Laid-Open No. 2005-128959” and "Japanese Patent Application Laid-Open No. 2005-319018".
  • the generation unit 526 of the imaging apparatus 500 is based on at least two of the image captured by the large-sized imaging unit 510 and the image captured by the plurality of small-sized imaging units 512. Generate data on the distance to the subject. In the fourth embodiment, the generation unit 526 further corrects data related to the distance to the subject based on the first learning result stored in the learning result storage unit. In addition, the generation unit 526 corrects the image captured by the small-size imaging unit 512 based on the image captured by the large-size imaging unit 510 and the second learning result stored in the learning result storage unit.
  • FIG. 28 is a flowchart showing processing of the generation unit in the fourth embodiment. Since S30 and S32 of FIG. 28 are the same as S10 and S12 of FIG. 24, the description will be omitted.
  • the generation unit 526 corrects the distance to the subject estimated in S32 according to the first learning result (S34). For example, the average value of the distance obtained in S32 and the distance obtained according to the first learning result may be used as the corrected distance. Since S36 of FIG. 28 is the same as S14 of FIG. 24, the description is omitted.
  • the generation unit 526 corrects the RGB data acquired in S36 according to the second learning result (S38). For example, RGB data indicating an object identified in the center image 530 may be reflected on RGB data of the left image 532a or the right image 532b.
  • the following S40 to S46 are the same as S16 to S22 in FIG. Note that one of the correction processing of S34 and the correction processing of S38 may be executed.
  • the image pickup elements (image sensors) of the respective image pickup units may be formed on the same substrate.
  • a shielding member partition
  • an imaging element which should detect light transmitted through a lens of a certain imaging unit does not detect light transmitted through a lens of another imaging unit.
  • the pixel data acquisition unit 522 of the imaging device 500 acquires the pixel value of the region corresponding to the large imaging unit 510 in the imaging device as the pixel value of the image imaged by the large imaging unit 510.
  • the pixel data acquisition unit 522 is a pixel of an image captured by the compact imaging unit 512a (or compact imaging unit 512b) in the pixel value of the region corresponding to the compact imaging unit 512a (or compact imaging unit 512b) in the imaging device. Get as a value.
  • the plurality of imaging units share the imaging device on a single substrate, the number of parts of the imaging device 500 can be reduced, and the manufacturing cost of the imaging device 500 can be reduced.
  • the image processing unit 514 may be provided in the imaging element.
  • the imaging device 500 is a stacked image sensor in which a logic circuit (and / or a control circuit) mounting the function of the image processing unit 514 is provided in the lower layer of the pixel array. May be configured as As a result, many image processings are completed in the image sensor, so that the processing speed can be increased, the processing in the latter stage can be reduced in weight, and the processing load on the external device can be reduced.
  • At least one of the one or more large-sized imaging units 510 and the plurality of small-sized imaging units 512 may be provided with a polarizer that transmits a polarization component in a predetermined direction among the light transmitted through the lens.
  • each of the large-sized imaging unit 510 and the small-sized imaging unit 512 may include an imaging optical system 14, an aperture 18, and an imaging device 20 as shown in FIG. 1 of the related art.
  • the imaging device 20 may include a two-dimensional array of pixels, and the pixels may have a structure in which a microlens, a polarizer, and a photodiode are integrally stacked.
  • a plurality of types of polarizers having a plurality of types of principal axis angles may be provided in a plurality of imaging units (or a pixel unit in a single imaging unit). According to the first modification, it is possible to obtain a polarized image (or a plurality of types of polarized images corresponding to a plurality of directions). Thereby, it is possible to obtain the normal vector of the object surface using the polarization image.
  • a second modified example related to the first modified example will be described.
  • a photodiode photoelectric conversion unit which is a unit for converting light transmitted through a microlens into electric charge
  • a photodiode may be provided in each of a plurality of partial areas obtained by dividing a pixel area corresponding to one microlens.
  • At least one of the one or more large-sized imaging units 510 and the plurality of small-sized imaging units 512 may be provided with pixels capable of detecting four or more wavelength bands (herein referred to as “special pixels”).
  • the four or more wavelength bands include, for example, a first wavelength band (red), a second wavelength band (green), a third wavelength band (blue), and other wavelength bands (for example, yellow, magenta, etc.) May be.
  • wavelength bands infrared, ultraviolet etc.
  • Data on four or more wavelength bands is also called multispectral data and is also called hyperspectral data.
  • An imaging apparatus 500 stores a spectrum data storage unit that stores data indicating detection results of four or more wavelength bands (that is, the features of the spectrum) and a predetermined correspondence with an object that is a subject. You may provide further.
  • the generation unit 526 obtains the detection results of four or more types of wavelength bands output from the imaging unit including the special pixel with reference to the correspondence stored in the spectrum data storage unit, and an object corresponding to the detection result. May be identified.
  • the generation unit 526 may generate an output image in which data (pixel values) of colors associated in advance with the specified object is set in the area of the specified object.
  • the spectrum data storage unit may store data indicating a correspondence between detection results of four or more types of wavelength bands and types of light sources.
  • the type of light source may include, for example, a sun, a fluorescent lamp, and an LED.
  • the generation unit 526 may acquire detection results of four or more types of wavelength bands output from the imaging unit including the special pixel, and specify a light source corresponding to the detection result. Then, the generation unit 526 may generate an output image in which a color corresponding to the specified light source is set. According to the third modification, it becomes easy to identify an object (for example, water and alcohol) which is difficult to distinguish by visible light. In addition, estimation of the light source is facilitated.
  • the image captured by the compact imaging unit 512 includes noise such as light shot noise. Since the small-sized imaging unit 512 has a small optical size, the signal-to-noise ratio of the image captured by the small-sized imaging unit 512 tends to decrease when the surroundings become dark. In other words, the small imaging unit 512 is more likely to deteriorate the signal noise ratio of the captured image than the large imaging unit 510.
  • the imaging apparatus 500 further includes a measurement unit that measures the signal-to-noise ratio of the image captured by each of the plurality of small-sized imaging units 512.
  • the generation unit 526 performs pixel addition on the images captured by each of the plurality of small-size imaging units 512 in accordance with the signal-to-noise ratio.
  • a known method may be adopted for pixel addition.
  • the generation unit 526 sets a plurality of adjacent pixels (for example, two pixels) as one virtual pixel with respect to a captured image of the small-sized imaging unit 512 in which the signal noise ratio is less than a predetermined threshold.
  • the captured image of the compact imaging unit 512 may be corrected by setting the sum of values to a pixel value of one virtual pixel. According to the fourth modified example, it is possible to suppress the deterioration of the signal-to-noise ratio caused by the change in the surrounding environment of the image captured by the small size imaging unit 512.
  • the imaging apparatus 500 including the plurality of imaging units and the image processing unit 514 has been described.
  • a plurality of imaging devices independent of each other corresponding to the large imaging unit 510 and the small imaging unit 512, and an information processing apparatus including the function of the image processing unit 514 are provided.
  • a cooperative camera system may be constructed. The techniques described in the embodiments and the modifications are also applicable to the camera system.
  • 500 imaging apparatus 510 large-sized imaging unit, 512 small-sized imaging unit, 514 image processing unit, 520 characteristic storage unit, 522 pixel data acquisition unit, 524 adjustment unit, 526 generation unit, 528 output unit.
  • the present invention is applicable to an apparatus or system for processing an image.

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • General Physics & Mathematics (AREA)
  • Optics & Photonics (AREA)
  • Human Computer Interaction (AREA)
  • Spectroscopy & Molecular Physics (AREA)
  • Studio Devices (AREA)
  • Automatic Focus Adjustment (AREA)
  • Blocking Light For Cameras (AREA)
  • Focusing (AREA)
  • Cameras In General (AREA)
  • Stereoscopic And Panoramic Photography (AREA)
  • Exposure Control For Cameras (AREA)
  • Camera Bodies And Camera Details Or Accessories (AREA)
  • Image Processing (AREA)
  • Color Television Image Signal Generators (AREA)
  • Closed-Circuit Television Systems (AREA)
  • Testing, Inspecting, Measuring Of Stereoscopic Televisions And Televisions (AREA)
  • Length Measuring Devices By Optical Means (AREA)
PCT/JP2018/033266 2017-09-15 2018-09-07 撮像装置 Ceased WO2019054304A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US16/644,260 US11064182B2 (en) 2017-09-15 2018-09-07 Imaging apparatus
US17/341,807 US11438568B2 (en) 2017-09-15 2021-06-08 Imaging apparatus
US17/875,981 US12439015B2 (en) 2017-09-15 2022-07-28 Imaging apparatus

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
JP2017-178302 2017-09-15
JP2017178302A JP6951917B2 (ja) 2017-09-15 2017-09-15 撮像装置

Related Child Applications (2)

Application Number Title Priority Date Filing Date
US16/644,260 A-371-Of-International US11064182B2 (en) 2017-09-15 2018-09-07 Imaging apparatus
US17/341,807 Continuation US11438568B2 (en) 2017-09-15 2021-06-08 Imaging apparatus

Publications (1)

Publication Number Publication Date
WO2019054304A1 true WO2019054304A1 (ja) 2019-03-21

Family

ID=65723294

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2018/033266 Ceased WO2019054304A1 (ja) 2017-09-15 2018-09-07 撮像装置

Country Status (3)

Country Link
US (3) US11064182B2 (https=)
JP (4) JP6951917B2 (https=)
WO (1) WO2019054304A1 (https=)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115917394A (zh) * 2020-08-31 2023-04-04 索尼集团公司 医疗成像系统、医疗成像装置和操作方法

Families Citing this family (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6951917B2 (ja) 2017-09-15 2021-10-20 株式会社ソニー・インタラクティブエンタテインメント 撮像装置
JP7028814B2 (ja) * 2019-02-07 2022-03-02 ファナック株式会社 外形認識装置、外形認識システム及び外形認識方法
KR102099232B1 (ko) * 2019-07-03 2020-04-08 주식회사 레티널 소형 반사부를 이용한 카메라 모듈 및 이를 이용한 증강 현실용 광학 장치
US11233963B2 (en) * 2019-12-27 2022-01-25 Omnivision Technologies, Inc. Devices and methods for obtaining three-dimensional shape information using polarization and phase detection photodiodes
WO2021192814A1 (ja) * 2020-03-27 2021-09-30 ソニーグループ株式会社 情報処理装置と情報処理方法および情報処理システム
CN116391139B (zh) * 2020-10-12 2025-06-27 日本电信电话株式会社 光学元件、摄像元件以及摄像装置
JP7696742B2 (ja) * 2021-03-30 2025-06-23 キヤノン株式会社 画像処理装置、画像処理装置の制御方法
JPWO2023008102A1 (https=) * 2021-07-29 2023-02-02
JP2023115632A (ja) * 2022-02-08 2023-08-21 ソニーセミコンダクタソリューションズ株式会社 信号処理装置、プログラム
KR102478304B1 (ko) * 2022-02-23 2022-12-19 주식회사 코비스테크놀로지 중공사막 카트리지 검사 장치 및 방법
WO2025115562A1 (ja) * 2023-11-30 2025-06-05 富士フイルム株式会社 画像データ処理装置、画像データ処理方法、画像データ処理プログラム、撮像装置及び学習装置

Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06141246A (ja) * 1992-04-09 1994-05-20 Olympus Optical Co Ltd 撮像装置
JP2000112019A (ja) * 1998-10-08 2000-04-21 Olympus Optical Co Ltd 電子３眼カメラ装置
JP2007295113A (ja) * 2006-04-21 2007-11-08 Matsushita Electric Ind Co Ltd 撮像装置
JP2009048033A (ja) * 2007-08-22 2009-03-05 Panasonic Corp 立体画像撮像装置
WO2011114572A1 (ja) * 2010-03-19 2011-09-22 富士フイルム株式会社 撮像装置、方法およびプログラム並びにこれに用いる記録媒体
JP2012156775A (ja) * 2011-01-26 2012-08-16 Toshiba Corp カメラモジュール
WO2013099169A1 (ja) * 2011-12-27 2013-07-04 パナソニック株式会社 ステレオ撮影装置
WO2014196216A1 (ja) * 2013-06-05 2014-12-11 株式会社 東芝 イメージセンサ装置及びその製造方法

Family Cites Families (54)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US6205259B1 (en) 1992-04-09 2001-03-20 Olympus Optical Co., Ltd. Image processing apparatus
US20010054989A1 (en) * 1993-10-22 2001-12-27 Matthew Zavracky Color sequential display panels
JP2763000B2 (ja) * 1997-07-08 1998-06-11 ソニー株式会社 再生装置
JP3839180B2 (ja) 1999-02-19 2006-11-01 三菱電機株式会社 撮像装置及び撮像方法
US20050175083A1 (en) * 2000-09-01 2005-08-11 Gutierrez Novelo Manuel R. Stereoscopic video capturing device and dual receiver with viewer for three-dimension display, and method thereof
JP2003298920A (ja) 2002-03-29 2003-10-17 Fuji Photo Film Co Ltd デジタルカメラ
JP2004241825A (ja) 2003-02-03 2004-08-26 Konica Minolta Holdings Inc 携帯端末
US20050088515A1 (en) * 2003-10-23 2005-04-28 Geng Z. J. Camera ring for three-dimensional (3D) surface imaging
US7180602B2 (en) * 2003-12-11 2007-02-20 Nuonics, Inc. Agile spectral interferometric microscopy
US7639208B1 (en) * 2004-05-21 2009-12-29 University Of Central Florida Research Foundation, Inc. Compact optical see-through head-mounted display with occlusion support
JP4379295B2 (ja) 2004-10-26 2009-12-09 ソニー株式会社 半導体イメージセンサー・モジュール及びその製造方法
JP2006333132A (ja) * 2005-05-26 2006-12-07 Sony Corp 撮像装置及び撮像方法、プログラム、プログラム記録媒体並びに撮像システム
US7566855B2 (en) * 2005-08-25 2009-07-28 Richard Ian Olsen Digital camera with integrated infrared (IR) response
US8254712B2 (en) * 2007-03-30 2012-08-28 Fujifilm Corporation Image processing apparatus, image processing method, image managing apparatus, image managing method, computer program product, and image order sheet
JP5003529B2 (ja) 2008-02-25 2012-08-15 株式会社ニコン 撮像装置および対象物の検出方法
JP4984257B2 (ja) * 2008-02-28 2012-07-25 大日本印刷株式会社 会場設営シミュレーション装置、プログラム、媒体、簡易な画像変形合成方法
JP4448889B2 (ja) 2008-04-01 2010-04-14 富士フイルム株式会社 撮像素子及び撮像装置
WO2011078244A1 (ja) 2009-12-24 2011-06-30 シャープ株式会社 多眼撮像装置および多眼撮像方法
JP5204349B2 (ja) * 2010-08-31 2013-06-05 パナソニック株式会社 撮影装置、再生装置、および画像処理方法
JP5682437B2 (ja) 2010-09-07 2015-03-11 ソニー株式会社 固体撮像素子、固体撮像装置、撮像機器、及び、偏光素子の製造方法
JP5695395B2 (ja) * 2010-11-19 2015-04-01 インターナショナル・ビジネス・マシーンズ・コーポレーションＩｎｔｅｒｎａｔｉｏｎａｌ Ｂｕｓｉｎｅｓｓ Ｍａｃｈｉｎｅｓ Ｃｏｒｐｏｒａｔｉｏｎ 立体画像生成方法及びその装置
JP2013061560A (ja) 2011-09-14 2013-04-04 Ricoh Co Ltd 測距装置および撮像装置
JP5874334B2 (ja) 2011-11-15 2016-03-02 株式会社ニコン 画像処理装置、撮像装置、画像処理プログラムおよび撮像装置の制御プログラム
EP2597483B8 (de) * 2011-11-25 2017-06-07 Safran Vectronix AG Entfernungsmesser
JP5943613B2 (ja) * 2012-01-16 2016-07-05 キヤノン株式会社 画像処理装置及び画像処理装置の制御方法
CA2861212A1 (en) 2012-03-07 2013-09-12 Sony Corporation Image processing device and method, and program
JP5998862B2 (ja) * 2012-11-09 2016-09-28 株式会社ソシオネクスト 動画像処理装置
JP2014165855A (ja) 2013-02-27 2014-09-08 Nikon Corp 電子機器
JP6429440B2 (ja) 2013-09-18 2018-11-28 キヤノン株式会社 撮像装置、その制御方法、および制御プログラム
WO2015045785A1 (ja) * 2013-09-30 2015-04-02 富士フイルム株式会社 画像処理装置、撮像装置、画像処理方法及び画像処理プログラム
US9870879B2 (en) 2013-12-20 2018-01-16 Vizimax Inc. Controlled switching devices and method of using the same
US10402665B2 (en) * 2014-05-14 2019-09-03 Mobileye Vision Technologies, Ltd. Systems and methods for detecting traffic signs
KR102250192B1 (ko) * 2014-05-19 2021-05-10 삼성전자주식회사 이종 화소 구조를 갖는 이미지 센서
US10257494B2 (en) * 2014-09-22 2019-04-09 Samsung Electronics Co., Ltd. Reconstruction of three-dimensional video
JP5893713B1 (ja) 2014-11-04 2016-03-23 オリンパス株式会社 撮像装置、撮像方法、処理プログラム
KR101580585B1 (ko) * 2014-12-02 2015-12-28 서울시립대학교 산학협력단 전정색영상과 적외선영상의 융합 방법 및 장치
US20160165211A1 (en) * 2014-12-08 2016-06-09 Board Of Trustees Of The University Of Alabama Automotive imaging system
JP6447121B2 (ja) 2014-12-26 2019-01-09 株式会社リコー 画像処理装置、画像処理方法、撮像システム、画像処理システム、およびプログラム
CN107533210A (zh) * 2015-01-14 2018-01-02 因维萨热技术公司 相位检测自动聚焦
JP6611477B2 (ja) 2015-06-08 2019-11-27 キヤノン株式会社 撮像装置、発光制御方法、プログラム
JP6488203B2 (ja) 2015-07-01 2019-03-20 株式会社ソニー・インタラクティブエンタテインメント 画像処理装置、画像処理システム、多視点カメラ、および画像処理方法
JP6552315B2 (ja) 2015-07-27 2019-07-31 キヤノン株式会社 撮像装置
JP6584237B2 (ja) 2015-09-01 2019-10-02 キヤノン株式会社 制御装置、制御方法、およびプログラム
US10217189B2 (en) * 2015-09-16 2019-02-26 Google Llc General spherical capture methods
US20170094251A1 (en) * 2015-09-30 2017-03-30 Faro Technologies, Inc. Three-dimensional imager that includes a dichroic camera
JP6005246B2 (ja) 2015-11-05 2016-10-12 キヤノン株式会社 撮像装置、ヒストグラムの表示方法、プログラム、画像処理装置
JP6808409B2 (ja) 2015-11-25 2021-01-06 キヤノン株式会社 イメージセンサおよび撮像装置
US9674504B1 (en) * 2015-12-22 2017-06-06 Aquifi, Inc. Depth perceptive trinocular camera system
US10313657B2 (en) * 2015-12-25 2019-06-04 Boe Technology Group Co., Ltd. Depth map generation apparatus, method and non-transitory computer-readable medium therefor
US10638920B2 (en) * 2016-06-13 2020-05-05 Capsovision Inc Method and apparatus of lens alignment for capsule
US10242442B2 (en) * 2016-10-27 2019-03-26 International Business Machines Corporation Detection of outlier lesions based on extracted features from skin images
US10810721B2 (en) * 2017-03-14 2020-10-20 Adobe Inc. Digital image defect identification and correction
US10401872B2 (en) * 2017-05-23 2019-09-03 Gopro, Inc. Method and system for collision avoidance
JP6951917B2 (ja) 2017-09-15 2021-10-20 株式会社ソニー・インタラクティブエンタテインメント 撮像装置

Patent Citations (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH06141246A (ja) * 1992-04-09 1994-05-20 Olympus Optical Co Ltd 撮像装置
JP2000112019A (ja) * 1998-10-08 2000-04-21 Olympus Optical Co Ltd 電子３眼カメラ装置
JP2007295113A (ja) * 2006-04-21 2007-11-08 Matsushita Electric Ind Co Ltd 撮像装置
JP2009048033A (ja) * 2007-08-22 2009-03-05 Panasonic Corp 立体画像撮像装置
WO2011114572A1 (ja) * 2010-03-19 2011-09-22 富士フイルム株式会社 撮像装置、方法およびプログラム並びにこれに用いる記録媒体
JP2012156775A (ja) * 2011-01-26 2012-08-16 Toshiba Corp カメラモジュール
WO2013099169A1 (ja) * 2011-12-27 2013-07-04 パナソニック株式会社 ステレオ撮影装置
WO2014196216A1 (ja) * 2013-06-05 2014-12-11 株式会社 東芝 イメージセンサ装置及びその製造方法

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN115917394A (zh) * 2020-08-31 2023-04-04 索尼集团公司 医疗成像系统、医疗成像装置和操作方法

Also Published As

Publication number Publication date
US20200204786A1 (en) 2020-06-25
JP2024019210A (ja) 2024-02-08
US11438568B2 (en) 2022-09-06
US20220385879A1 (en) 2022-12-01
US12439015B2 (en) 2025-10-07
US20210306610A1 (en) 2021-09-30
JP7393498B2 (ja) 2023-12-06
JP2023014082A (ja) 2023-01-26
JP7170810B2 (ja) 2022-11-14
JP2019054463A (ja) 2019-04-04
JP2022003817A (ja) 2022-01-11
JP6951917B2 (ja) 2021-10-20
JP7664351B2 (ja) 2025-04-17
US11064182B2 (en) 2021-07-13

Similar Documents

Publication Publication Date Title
JP7170810B2 (ja) 撮像装置、画像生成方法およびコンピュータプログラム
JP6821028B2 (ja) 撮像装置および画像データ読み出し方法
WO2019082820A1 (ja) カメラシステム
KR102777120B1 (ko) Vr/ar 응용에서 심도 증강을 위한 다중-기선 카메라 어레이 시스템 아키텍처
KR101843994B1 (ko) 이종 이미저를 구비한 모놀리식 카메라 어레이를 이용한 이미지의 캡처링 및 처리
US8885067B2 (en) Multocular image pickup apparatus and multocular image pickup method
JP5649091B2 (ja) 画像キャプチャ装置および画像キャプチャ方法
JP6422924B2 (ja) 撮像装置および被写体情報取得方法
JP2015521411A (ja) πフィルタ群を用いてパターン化されたカメラモジュール
CN107924572A (zh) 使用阵列相机执行高速视频捕获和深度估计的系统和方法
WO2013027504A1 (ja) 撮像装置
US10122990B2 (en) Imaging system and method of producing context and focus images
WO2013005489A1 (ja) 撮像装置、画像処理装置
TWI599809B (zh) 鏡頭模組陣列、影像感測裝置與數位縮放影像融合方法
JP5982907B2 (ja) 画像処理装置、および画像処理方法、並びにプログラム
JP2012156818A (ja) 立体撮像システム、補正装置およびそのプログラム
WO2013005602A1 (ja) 撮像装置、画像処理装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18856997

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18856997

Country of ref document: EP

Kind code of ref document: A1