EP1764835B1 - Multifunctional optical sensor comprising a matrix of photodetectors coupled microlenses - Google Patents
Multifunctional optical sensor comprising a matrix of photodetectors coupled microlenses Download PDFInfo
- Publication number
- EP1764835B1 EP1764835B1 EP05425654A EP05425654A EP1764835B1 EP 1764835 B1 EP1764835 B1 EP 1764835B1 EP 05425654 A EP05425654 A EP 05425654A EP 05425654 A EP05425654 A EP 05425654A EP 1764835 B1 EP1764835 B1 EP 1764835B1
- Authority
- EP
- European Patent Office
- Prior art keywords
- photodetectors
- microlenses
- matrix
- optical sensor
- view
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Not-in-force
Links
- 239000011159 matrix material Substances 0.000 title claims abstract description 124
- 230000003287 optical effect Effects 0.000 title claims abstract description 88
- 238000012544 monitoring process Methods 0.000 claims abstract description 44
- 238000000926 separation method Methods 0.000 claims abstract description 22
- 230000007613 environmental effect Effects 0.000 claims abstract description 14
- 239000007787 solid Substances 0.000 claims abstract description 11
- 230000005855 radiation Effects 0.000 claims abstract description 8
- 230000006870 function Effects 0.000 claims description 70
- 238000005286 illumination Methods 0.000 claims description 10
- 238000007781 pre-processing Methods 0.000 claims description 10
- 239000011248 coating agent Substances 0.000 claims description 2
- 238000000576 coating method Methods 0.000 claims description 2
- 238000005516 engineering process Methods 0.000 claims description 2
- 239000000463 material Substances 0.000 claims description 2
- 239000000243 solution Substances 0.000 description 16
- 238000001514 detection method Methods 0.000 description 15
- 238000005192 partition Methods 0.000 description 12
- 238000012545 processing Methods 0.000 description 8
- 238000005259 measurement Methods 0.000 description 6
- 238000013459 approach Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 230000010354 integration Effects 0.000 description 3
- 238000004519 manufacturing process Methods 0.000 description 3
- 238000000034 method Methods 0.000 description 3
- 230000003044 adaptive effect Effects 0.000 description 2
- 238000004378 air conditioning Methods 0.000 description 2
- 230000003213 activating effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 230000008021 deposition Effects 0.000 description 1
- 238000006073 displacement reaction Methods 0.000 description 1
- 230000008030 elimination Effects 0.000 description 1
- 238000003379 elimination reaction Methods 0.000 description 1
- 239000011259 mixed solution Substances 0.000 description 1
- 230000004297 night vision Effects 0.000 description 1
- 230000002093 peripheral effect Effects 0.000 description 1
- 230000001105 regulatory effect Effects 0.000 description 1
- 230000003595 spectral effect Effects 0.000 description 1
- XLYOFNOQVPJJNP-UHFFFAOYSA-N water Substances O XLYOFNOQVPJJNP-UHFFFAOYSA-N 0.000 description 1
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
- H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J1/00—Photometry, e.g. photographic exposure meter
- G01J1/02—Details
- G01J1/04—Optical or mechanical part supplementary adjustable parts
- G01J1/0407—Optical elements not provided otherwise, e.g. manifolds, windows, holograms, gratings
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J1/00—Photometry, e.g. photographic exposure meter
- G01J1/02—Details
- G01J1/04—Optical or mechanical part supplementary adjustable parts
- G01J1/0407—Optical elements not provided otherwise, e.g. manifolds, windows, holograms, gratings
- G01J1/0411—Optical elements not provided otherwise, e.g. manifolds, windows, holograms, gratings using focussing or collimating elements, i.e. lenses or mirrors; Aberration correction
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J1/00—Photometry, e.g. photographic exposure meter
- G01J1/42—Photometry, e.g. photographic exposure meter using electric radiation detectors
- G01J1/4204—Photometry, e.g. photographic exposure meter using electric radiation detectors with determination of ambient light
-
- G—PHYSICS
- G01—MEASURING; TESTING
- G01J—MEASUREMENT OF INTENSITY, VELOCITY, SPECTRAL CONTENT, POLARISATION, PHASE OR PULSE CHARACTERISTICS OF INFRARED, VISIBLE OR ULTRAVIOLET LIGHT; COLORIMETRY; RADIATION PYROMETRY
- G01J1/00—Photometry, e.g. photographic exposure meter
- G01J1/42—Photometry, e.g. photographic exposure meter using electric radiation detectors
- G01J1/4228—Photometry, e.g. photographic exposure meter using electric radiation detectors arrangements with two or more detectors, e.g. for sensitivity compensation
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
- H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
- H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
- H01L27/144—Devices controlled by radiation
- H01L27/146—Imager structures
- H01L27/14601—Structural or functional details thereof
- H01L27/14625—Optical elements or arrangements associated with the device
-
- H—ELECTRICITY
- H01—ELECTRIC ELEMENTS
- H01L—SEMICONDUCTOR DEVICES NOT COVERED BY CLASS H10
- H01L27/00—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate
- H01L27/14—Devices consisting of a plurality of semiconductor or other solid-state components formed in or on a common substrate including semiconductor components sensitive to infrared radiation, light, electromagnetic radiation of shorter wavelength or corpuscular radiation and specially adapted either for the conversion of the energy of such radiation into electrical energy or for the control of electrical energy by such radiation
- H01L27/144—Devices controlled by radiation
- H01L27/146—Imager structures
- H01L27/14601—Structural or functional details thereof
- H01L27/14625—Optical elements or arrangements associated with the device
- H01L27/14627—Microlenses
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
- H04N23/95—Computational photography systems, e.g. light-field imaging systems
- H04N23/957—Light-field or plenoptic cameras or camera modules
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/10—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used
- B60R2300/108—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of camera system used using 'non-standard' camera systems, e.g. camera sensor used for additional purposes i.a. rain sensor, camera sensor split in multiple image areas
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/30—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing
- B60R2300/304—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images
- B60R2300/305—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the type of image processing using merged images, e.g. merging camera image with stored images merging camera image with lines or icons
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
- B60R2300/8053—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for bad weather conditions or night vision
-
- B—PERFORMING OPERATIONS; TRANSPORTING
- B60—VEHICLES IN GENERAL
- B60R—VEHICLES, VEHICLE FITTINGS, OR VEHICLE PARTS, NOT OTHERWISE PROVIDED FOR
- B60R2300/00—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle
- B60R2300/80—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement
- B60R2300/8093—Details of viewing arrangements using cameras and displays, specially adapted for use in a vehicle characterised by the intended use of the viewing arrangement for obstacle warning
Definitions
- the present invention relates to a multifunctional optical sensor, in particular for automotive use, comprising a matrix of photodetectors of the CCD or CMOS type having a sensitive area divided into sub-areas, which, individually or combined together, are designated to specific functions of monitoring the scene or measuring environmental parameters.
- the scene monitoring functions include monitoring the scene in front, behind or laterally to the vehicle.
- the frontal monitoring detects, for example, the presence of a vehicle coming from the opposite direction, the presence of a curve or the movement of the vehicle towards the longitudinal demarcation lines of the lane.
- the monitoring behind the vehicle can, for example, aid parking manoeuvres.
- the lateral monitoring detects, for example, the vehicles that arrive laterally and that are not visible with the external rear-view mirror, since they are in the so-called "blind angle".
- the measurement of environmental parameters comprises, for example, the measurement of fog, rain, window fogging, illumination and solar irradiation conditions.
- each function is associated to a single lens (or to multiple lenses positioned on different matrices) and said lens is associated to a subgroup of photodetectors.
- Figure 1 shows a perspective view of an embodiment of the sensor according to the invention of the document EP-A-1 418 089 .
- Use of single aperture optics limits the possibility of reducing the size of the opto-mechanical system as a whole.
- the complexity of the process for the construction and assembly of the system does not allow significantly to lower the costs of the sensor for large volumes.
- the object of the present invention is to provide an optical sensor of the type defined above which enables to:
- said object is achieved by a multifunctional optical sensor having the characteristics set out in claim 1.
- the present invention relates to a multifunctional optical sensor comprising a matrix of photodetectors of the CCD or CMOS type and a matrix of microlenses, in which each microlens is coupled to a subgroup (cluster) of photodetectors (pixels) or to a single photodetector.
- the microlenses are grouped in subgroups, each of which, individually or combined with others, is dedicated to a specific function of monitoring the scene or measuring environmental parameters.
- the present invention is directed, in particular, to the application on a motor vehicle of a multifunctional optical sensor of the type mentioned above, which can be positioned for example:
- optical sensor according to the present invention can also be used in other applications, such as: road infrastructures, robotics, domotics, agriculture, etc.
- the sum of the individual IFOV 16 determines the global field of view FOV.
- Each microlens 14 can also be separated from the contiguous microlens by a "baffle" 18' whose function is to prevent the radiation coming from a contiguous microlens from reaching the non corresponding photodetector.
- Figure 5 schematically shows a lateral view of a portion of an optical sensor 10 [p1] which comprises a photodetectors matrix 12 and a microlenses matrix 14 in which the distance between the centres of the diaphragms 18 is different from the distance between the centres of the microlenses 14.
- Each microlens 14 is associated to a cluster of photodetectors 12 or to a single photodetector. In the example of Figure 5, the microlenses 14 are mutually identical.
- Each microlens 14 is set to focalise the radiation coming from a portion of solid angle IFOV 16 on the cluster of photodetectors 12 or on the single photodetector 12 associated to the microlens 14.
- Figure 5 shows a matrix of diaphragms 18 positioned between the photodetectors matrix 12 and the microlenses matrix 14. The diaphragm matrix 18 enables to select for each photodetector 12 or cluster of photodetectors 12 the portion of solid angle IFOV 16.
- the microlenses must be calculated adding a prismatic component to the spherical component in order to have microlenses operating with optical axis not coinciding their axis of symmetry (off-axis).
- the IFOV 16 of each microlens will have such central directions 20 that ⁇ 1 ⁇ 2 ⁇ 3. As in the case shown in Figure 6, the total FOV can be sub-sampled.
- FIG. 7 shows a microlens 22 operating by total internal reflection associated to refractive off-axis microlenses 14.
- Figure 8 shows an optical sensor with a reflexive lens 24 associated to refractive off-axis microlenses 14.
- micro-objectives 15 are composed by at least two superposed micro-objectives.
- the doublet optical solution allows to vary effective focal length (e.f.l.) maintaining the back focal length (b.f.l.) equal for all micro-objectives.
- the distance between the photodetectors matrix 12 and the micro-objectives matrix 15 is constant (b.f.l.) whilst the possibility of varying e.f.l. allows to have a non constant separation between the central directions 20, i.e. ⁇ 1 ⁇ 2 ⁇ 3. In this case, the IFOV are varied at the same time.
- a possible alternative to the use of the diaphragms 18 consists of using a matrix with photo-detectors 12 having smaller size than the diaphragms 18.
- a high resolution photodetectors matrix 12 it is possible to have either a high resolution photodetectors matrix 12, or a photodetectors matrix 12 with larger size and hence smaller resolution.
- a diaphragms matrix 18 is used to vary the central direction 20 associated to each microlens 14 (case described in the previous optical solutions).
- the disadvantages are due to the fact that the displacement of the central directions 20 occurs by discrete steps (the minimum pitch is equal to the size of the photodetector 12) and not continuously as when using the diaphragms 18, and that to visualise the images requires pre-processing for addressing the active photodetectors.
- the microlenses of the optical sensor according to the present invention can be constituted by GRIN (gradient index) material.
- GRIN gradient index
- On the lower plane of some microlens or of some subgroup of microlenses can be deposited a selective interferential coating operating as a filter to transmit only the wavelengths of interest.
- a NIR (near infrared) LED illuminator can be used, the related spectral band has to be selected with respect to the background.
- the photodetectors matrix is in CCD or CMOS technology, standard or with parallel architecture (pre-processing at the photodetector level).
- Figure 11 is a schematic view showing the object and image planes of two optical systems, the first one constituted by a single aperture optics 8 ( Figure 11a) and the second one constituted by a 2D matrix of microlenses 14 ( Figure 11b), said optical systems being coupled to matrices 2D of photodetectors 12, said matrices having the same format mxn.
- the photodetectors 12 subtend the IFOV according to the laws of geometric optics applied to the lens 8: adjacent portions of the plane of the objects are subtended in the image plane by mutually contiguous photodetectors 12.
- the central direction 20 of the field of view IFOV of each individual microlens 14 can be established independently. Therefore in Figure 12 the microlenses 14 are so positioned that the previous rule no longer applies and thus the angular separation between the central directions 20 of the fields of view relating to adjacent photodetectors is not constant within the matrix. However, in this case a pre-processing for addressing the photodetectors is necessary in order to visualise the images.
- a 2D matrix of mxn photodetectors 12 having a single aperture lens 8 with field of view of x horizontal degrees and y vertical degrees can be redesigned as a 1D linear matrix of mxn photodetectors 12, in such a way that each microlens 14 associated to the corresponding photodetector 12 has a field of view of x/m horizontal degrees and y/n vertical degrees with central direction 20 such as to cover a portion of the global field of view of x horizontal degrees and y vertical degrees ( Figure 13).
- central direction 20 of the field of view IFOV of each individual microlens 14 can be established independently, can be used also to optimise the partition of the photodetectors matrix into sub-areas dedicated to specific functions and in particular to exploit the entire sensitive area of the matrix.
- the format of the TV camera used in systems with single aperture lens for monitoring the scene in front of the vehicle, depends mainly on two parameters: field of view FOV and resolution R needed in the areas of the scene in which some objects have to be discriminated with higher precision (e.g., horizontal signs on the road surface). This means that in the other areas of the scene in which there are no objects of interest the previous resolution R is wholly redundant.
- the format of the camera must be at least CIF (320x256 pixels) or VGA (640x480 pixels).
- the frontal scene monitoring functions are manifold (vehicle crossing, Lane Warning, curve detection, vertical signs detection, pedestrian monitoring, etc.).
- An alternative that reduces the format of the TV camera entails the use of an optical zoom.
- the size and complexity of an optical zoom make it difficult to integrate it with other optical systems dedicated to the environmental parameters measuring functions (note the complexity of the optical sensor shown in Figure 1, according to the invention of the document EP-A-1 418 089 ).
- the optical zoom increase the fabrication costs of the sensor.
- the sub-area dedicated to frontal scene monitoring can be optimised, increasing resolution in the areas where details need to be discriminated (horizontal signs, obstacle recognition, etc.) and reducing it in the areas where the necessary information is more qualitative (road edges, horizon, etc.). This is equivalent to processing the images with the optimal resolution, as is made possible by an optical zoom.
- microlenses matrix defining the direction and amplitude of the field of view for each of them allows to simplify image processing.
- a possible optical pre-processing function consists of applying optical filters in order to pre-transform the image for subsequent processing.
- a high resolution is required even in non significant areas in order to have sufficient resolution to identify some areas of the images.
- an additional possibility consists of defining on the sensitive area of the matrix k groups of j photodetectors, each able to create (by means of a single microlens or a matrix of j microlenses) the image of the same portion of scene or of different portions of scene.
- On each group of j photodetectors are positioned diaphragms with different shapes. When a group of photodetectors frames a portion of scene that matches the shape of the diaphragm, the signal is the highest.
- This approach can be used, for example, for the Lane Warning function, as shown in Figure 17, in which there are k/2 groups of j photodetectors which view the left part of the road scenario (type 1 region of interest) and k/2 groups of j photodetectors which view the right part of the road scenario (type 2 region of interest).
- the microlenses 14 are so positioned that adjacent portions of the object plane are not subtended, in the image plane, by mutually contiguous photodetectors 12 and, therefore, the angular separation between the central directions 20 of the fields of view relating to adjacent photodetectors 12 is not constant within the matrix.
- Said k subgroups of microlenses are positioned horizontally on the photodetectors matrix: the first subgroup starting from the top left corner of the matrix and proceeding rightwards, the second one starting from the end of the first subgroup and proceeding rightwards and so on; reaching the right edge of the matrix, the subsequent line is started.
- Figure 18a shows the enlargement of both the subgroup of photodetectors relating to an ROI and of the photodetectors actually exposed to the radiation that passes through diaphragm.
- Figure 18b shows the positioning of the k th subgroup of microlenses as described above.
- the partition of the sensitive area of the photodetectors matrix can have different configurations according to the number and type of integrated functions.
- Figure 19 shows, by way of example, a first possible partition of the sensitive area of the photodetectors matrix, already mentioned in the document EP-A-1 418 089 by the same Applicant.
- the same functions can be integrated differently according to the inventive elements of the present patent application.
- the so-called "twilight” function is performed by a sub-area of the matrix that has to measure environmental illumination.
- the number of photodetectors dedicated to this function can even be reduced to just one and there are no constraints in terms of positioning on the sensitive area of the matrix.
- a central photodetector (or a few photodetectors) is surrounded (are surrounded) by eight or more photodetectors that have different fields of view, the central one(s) larger and the lateral ones smaller, in order to have information both about the intensity of environmental lighting (central photodetector(s)) and about the intensity and direction of solar irradiation (lateral photodetectors).
- the information about environmental illumination enables automatically to turn on/off the headlights of the vehicles in conditions of poor illumination.
- the information on the direction of the solar illumination enables to optimise the air conditioning system of the vehicle, e.g. for activating and regulating the air flows of multi-zone air conditioning systems.
- some photodetectors are oriented towards the dashboard of the vehicle to measure the radiation directed thereon (diffused radiation on the photodetectors).
- the photodetectors dedicated to the illumination and solar irradiation function can be positioned separately from each other, i.e. in non contiguous positions.
- the number of photodetectors can even be reduced to just one and there are no constraints in terms of positioning on the sensitive area of the matrix.
- a photodetector (a few photodetectors) has (have) a frontal field of view of about 20° and a second photodetector (a few photodetectors) a smaller field of view, e.g. about 10°.
- the sub-area of the sensitive matrix marked with "frontal monitoring” performs the so-called "Lane Warning” function.
- the area of interest i.e. the area that is used effectively for image processing
- the field of view of the photodetectors are smaller (higher resolution) in the areas of the images in which the lane demarcation lines could be located. This enables to reduce the number of photodetectors dedicated to this functions.
- the area called "frontal monitoring”, alternatively or in addition to the "Lane Warning” function, can be dedicated to the vehicle crossing detection function.
- the area of interest i.e. the area that is used effectively for image processing
- the number of photodetectors dedicated to this function is reduced from the one described in the document EP-A-1 418 089 by the same Applicant.
- the fields of view of the photodetectors are smaller (higher resolution) in the areas of the image where the potential indicators of the presence of headlights of a crossed vehicle or of taillights of a vehicle that precedes the reference vehicle (the one whereon the sensor is mounted) could be located.
- the photodetectors of the first subgroup have fields of view that assure long range monitoring of the scene to detect vehicles that arrive from the opposite lane
- the photodetectors of the second subgroup have fields of view that assure short range monitoring of the scene to detect the vehicles that precede the reference vehicle.
- a set of photodetectors positioned in the unused areas of the sub-area for the frontal monitoring function can be dedicated to lateral monitoring i.e. to the detection of the presence of a curve for commanding the adaptive headlights of the vehicle.
- the portion of sensitive area called "frontal monitoring” can serve a combination of multiple functions, e.g. Lane Warning, vehicle crossing, curve detection, etc.
- the sub-area dedicated to such functions is preferably constituted by photodetectors whose microlenses have optimised directions and fields of view: high resolution only in the areas where the objects of interest for the processing algorithms could be located, low resolution in non interesting areas (e.g. the horizon). The result is comparable to the one that would be obtained with an optical zoom.
- Figure 20 shows a second example of partition of the sensitive area of the photodetectors matrix, already described in the document EP-A-1 418 089 by the same Applicant. The same functions can be integrated according to the inventive elements of the present patent application.
- the use of microlenses allows to maintain the optical axis of the microlenses matrix parallel to the road plane. It is possible to compensate for the different distance between the photodetectors matrix and the windshield whereon the rain drops lie by designing microlenses with different focal length, as shown in the schematic representation of Figure 21.
- the windshield of the vehicle is designated by the number 30.
- the number 32 schematically indicates water drops deposited on the outer surface of the windshield.
- the number 36 designates device able to perform the function of monitoring the vehicle occupant.
- the photodetectors matrix designated by the reference 38, is oriented towards the front of the vehicle.
- Some photodetectors 40 e.g. positioned in the bottom left and right corner of the matrix 38, are used to determine the position of the driver and the presence, type and position of the passenger. Since this function does not require optics capable of creating the image of the entire vehicle but only of monitoring and discriminating the presence of passengers, a number of photodetectors equal, for example, to 9 is indicated.
- an optical system 42 is used, able to receive the image positioned to the rear of the photodetectors matrix.
- the optical system 42 can be a wave guide element as shown in Figure 22.
- the optical system 42 can comprise prismatic elements (not shown).
- the field of view of the optical system is designated by the reference number 44.
- Figure 23 shows an additional possibility of partition of the useful surface of the photodetectors matrix. This partition enables to integrate a higher number of functions on a matrix with a reduced format (e.g., CIF).
- CIF reduced format
- Figure 24 shows the possible arrangement of sensors 10 according to the present invention for performing the function of viewing the blind angle.
- each sensor 10 can have two different fields of view to cover different directions and distances, so that the arriving vehicle crosses the two beams at different times, generating a stepped signal that can be used to signal the danger.
Abstract
Description
- The present invention relates to a multifunctional optical sensor, in particular for automotive use, comprising a matrix of photodetectors of the CCD or CMOS type having a sensitive area divided into sub-areas, which, individually or combined together, are designated to specific functions of monitoring the scene or measuring environmental parameters.
- The scene monitoring functions include monitoring the scene in front, behind or laterally to the vehicle. The frontal monitoring detects, for example, the presence of a vehicle coming from the opposite direction, the presence of a curve or the movement of the vehicle towards the longitudinal demarcation lines of the lane. The monitoring behind the vehicle can, for example, aid parking manoeuvres. The lateral monitoring detects, for example, the vehicles that arrive laterally and that are not visible with the external rear-view mirror, since they are in the so-called "blind angle".
- The measurement of environmental parameters comprises, for example, the measurement of fog, rain, window fogging, illumination and solar irradiation conditions.
- The documents
EP-A-1 418 089 and genericEP-A-1 521 226 by the same Applicant describe multifunctional optical sensors, but refer to multifunctional integration solutions on matrix of photodetectors of the CCD or CMOS type by means of single aperture optics or matrices of lenses positioned in front of the optical window of the sensor. In the documentEP-A-1 521 226 , each function is associated to a single lens (or to multiple lenses positioned on different matrices) and said lens is associated to a subgroup of photodetectors. - Figure 1 shows a perspective view of an embodiment of the sensor according to the invention of the document
EP-A-1 418 089 . Use of single aperture optics limits the possibility of reducing the size of the opto-mechanical system as a whole. Moreover, the complexity of the process for the construction and assembly of the system does not allow significantly to lower the costs of the sensor for large volumes. - The object of the present invention is to provide an optical sensor of the type defined above which enables to:
- optimise the partition of the matrix: there is more freedom in defining the shape (rectangular, trapezoidal, linear) and the co-ordinates of the sub-areas of the photodetectors matrix associated to each function, the photodetectors used solely for the separation of the sub-areas are reduced or eliminated, the entire sensitive area is used and it is possible to assign different directions and fields of view to photodetectors belonging to the same sub-area/function (as will be illustrated farther on);
- miniaturise the sensor from the optics and chip viewpoint: using microlenses matrices, the typical dimensions of single aperture optical systems are eliminated, by optimising the partition of the matrix its format is reduced; high miniaturisation simplifies integration on the vehicle, enabling to insert the optical sensor in the rear-view mirrors, near the roof, in the ceiling lamp, etc.;
- simplify image processing: every photodetector or group of photodetectors has its field of view and direction optimised in such a way as to achieve a sort of optical "pre-processing";
- reduce costs thanks to optimised formats of the matrix, low cost microlens fabrication processes, deposition of interference filters on the surfaces of the microlens matrices adjacent to the photodetectors.
- According to the present invention, said object is achieved by a multifunctional optical sensor having the characteristics set out in
claim 1. - The present invention shall now be described in detail with reference to the accompanying drawings, provided purely by way of non limiting example, in which:
- Figure 1 shows a perspective view of an embodiment of the sensor according to document
EP-A-1 418 089 , - Figure 2 is a schematic view showing the principle of operation of an optical solution with single aperture,
- Figure 3 is a schematic view showing the principle of operation of an optical solution based on microlens matrices,
- Figure 4 is a schematic view showing the principle of operation of an optical system based on microlens matrices that constitutes prior art,
- Figure 5 is a schematic view showing the principle of operation of the optical system of the present invention according to a first embodiment,
- Figure 6 is a schematic view showing the principle of operation of the optical system of the present invention according to a second embodiment,
- Figures 7 and 8 are schematic views illustrating the principle of operation of two variants of the optical system of Figure 6,
- Figure 9 is a schematic view showing the principle of operation of the optical system of the present invention according to a third embodiment,
- Figure 10 shows the principle of operation of a microlens matrix with high resolution, not operating with the aid of diaphragms,
- Figure 11 is a schematic view showing the spaces of the objects and of the images of two optical systems, constituted by a single aperture optics (Fig. 11a) and by a 2D microlens matrix (Fig. 11b), said optical systems being coupled to 2D photodetector matrices with the same format,
- Figure 12 is a variant of Figure 11b,
- Figure 13 is a schematic view showing the spaces of the objects and of the images of an optical system constituted by a 1D microlens matrix,
- Figure 14 is another variant of Figure 11b,
- Figure 15 shows an example of an optical sensor with a matrix of photodetectors with constant dimensions or pitch and microlenses with different field of view on a road scenario,
- Figure 16 is an example of application of the optical sensor of Figure 15 on a road scenario,
- Figure 17 is an example of a matrix of subgroups of photodetectors that frame the same portion of scene or different portions of scene,
- Figure 18 is a variant of the use of the diaphragms of Figure 17 for optical pre-processing functions,
- Figures 19 and 20 show two examples of partition in sub-areas of the sensitive area of the photodetector matrix, previously described in the document
EP-A-1 418 089 by the same Applicant, - Figure 21 schematically shows an optical solution for detecting rain based on a matrix of microlenses with different focal length,
- Figure 22 shows an example of light wave guide coupled to the photodetector matrix to perform the function of monitoring the occupant of the vehicle,
- Figure 23 shows an example of partition into sub-areas of the sensitive area of the photodetector matrix according to a preferred characteristic of the present invention, and
- Figure 24 shows the possible positioning on the vehicle of multifunction sensors for the "blind angle" function according to the present invention.
- The present invention relates to a multifunctional optical sensor comprising a matrix of photodetectors of the CCD or CMOS type and a matrix of microlenses, in which each microlens is coupled to a subgroup (cluster) of photodetectors (pixels) or to a single photodetector. The microlenses are grouped in subgroups, each of which, individually or combined with others, is dedicated to a specific function of monitoring the scene or measuring environmental parameters.
- The present invention is directed, in particular, to the application on a motor vehicle of a multifunctional optical sensor of the type mentioned above, which can be positioned for example:
- in proximity to the windshield, e.g. in the interior rear-view mirror, to carry out, for example, the following functions: measurement of environmental illumination or entry into a gallery, measurement of solar irradiation, detection of the presence of raindrops on the windshield, detection of conditions of internal or external fogging of the windshield, detection of the presence of ice, detection of fog conditions and monitoring the scene in front of the vehicle (vehicle crossing); additional functions which can be integrated in addition or alternatively to the vehicle crossing function are the functions: levelling, curve or lane detection (for commanding adaptive headlights or for Lane Warning), night vision (viewing scene in the near-infrared or NIR), detection of vertical road signs, detection of pedestrians, black box (storing images relating to frontal monitoring in a circular memory buffer which can be used in case of accident);
- near the rear window, to carry out the following functions: measurement of environmental illumination, rain/fogging (internal and external), ice, fog, rear monitoring (parking); additional functions which can be integrated in addition or alternatively to rear monitoring are: blind angle monitoring , levelling, lane detection, black box;
- in the side mirrors to carry out the functions: measurement of environmental illumination, fog, rear monitoring (blind angle); additional functions which can be integrated in addition or alternatively to blind angle monitoring are: parking, levelling, lane detection, black box;
- in the uprights of the windshield for the functions of: user identification, occupant monitoring for the air-bag system.
- The optical sensor according to the present invention can also be used in other applications, such as: road infrastructures, robotics, domotics, agriculture, etc.
- The present invention shall now be described in detail with reference to the operating principles of optical systems.
- Figure 2 shows the single aperture optical solutions whereon are based the documents
EP-A-1 418 089 andEP-A-1 521 226 : thelens 8 has a field of view FOV=2·arctan (d/2f) where d=n·dpixel is the total dimension of thephotodetector matrix 12, dpixel is the dimension or pitch of thephotodetector 12 and f is the focal length of the optical system. - In general, to reduce the dimensions of the optical system, and in particular the focal length f by a factor n,
microlenses 14 can be used, each coupled to aphotodetector 12, with linear dimension of d/n=dpixel and focal length f1=f/n (Figure 3). In this case, each of themicrolenses 14 has a field of view that coincides with the global field of view, FOV1=FOV=2·arctan(dpixel/2f1). - To have a global field of view FOV that is the resultant of the individual fields of view of each of the
microlenses 14 it is necessary to reduce the size of the active area of thephotodetectors 12 in such a way that their dimensions are dpixel/n or equivalently to positiondiaphragms 18 having aperture with dimension dpixel/n in front of thephotodetectors 12. - If the distance between the centres of the
diaphragms 18 is different from the dimensions of themicrolenses 14, as shown in Figure 4 which constitutes prior art, in front of thephotodetectors matrix 12 is positioned a matrix ofdiaphragms 18 such that anymicrolens 14 has anIFOV 16 withcentral direction 20 and a constant angular separation αi between the central directions 20 (α1= α2= α3= α4). The sum of theindividual IFOV 16 determines the global field of view FOV. Eachmicrolens 14 can also be separated from the contiguous microlens by a "baffle" 18' whose function is to prevent the radiation coming from a contiguous microlens from reaching the non corresponding photodetector. - If the angular separation αi between the
central directions 20 is not to be constant, as contained inclaim 1 and shown in Figure 5, it is necessary to change the distances between the centres of the diaphragms. Figure 5 schematically shows a lateral view of a portion of anoptical sensor 10[p1] which comprises aphotodetectors matrix 12 and amicrolenses matrix 14 in which the distance between the centres of thediaphragms 18 is different from the distance between the centres of themicrolenses 14. Eachmicrolens 14 is associated to a cluster ofphotodetectors 12 or to a single photodetector. In the example of Figure 5, themicrolenses 14 are mutually identical. Eachmicrolens 14 is set to focalise the radiation coming from a portion ofsolid angle IFOV 16 on the cluster ofphotodetectors 12 or on thesingle photodetector 12 associated to themicrolens 14. Figure 5 shows a matrix ofdiaphragms 18 positioned between thephotodetectors matrix 12 and themicrolenses matrix 14. Thediaphragm matrix 18 enables to select for eachphotodetector 12 or cluster ofphotodetectors 12 the portion ofsolid angle IFOV 16. - A difference with respect to the case of Figure 4 is that the angular separation αi between the
central directions 20 of the portions ofsolid angle 16 subtended by themicrolenses 14 is not constant. In the example of Figure 5, the angles between thecentral directions 20 are designated α1, α2, α3, and they are such that α1≠α2≠α3. The reason for this is that the distance between the centres of thediaphragms 18 is not constant. Another difference is that the total FOV is not the sum of theIFOV 16 and therefore the FOV can be sub-sampled. To prevent the radiation coming from acontiguous microlens 14 from reaching the noncorresponding photodetector 12, each microlens 14 can be separated from thecontiguous microlens 14 by a "baffle" 18'. - If the angular separation αi between the
central directions 20 is not to be constant, but the distance between thediaphragms 18 is to be equal to the dimensions of themicrolenses 14, as shown in Figure 6, the microlenses must be calculated adding a prismatic component to the spherical component in order to have microlenses operating with optical axis not coinciding their axis of symmetry (off-axis). TheIFOV 16 of each microlens will have suchcentral directions 20 that α1≠α2≠α3. As in the case shown in Figure 6, the total FOV can be sub-sampled. - In order to cover a field of view larger than what is possible with a refractive optical solution as described above, solutions with microlenses of the kind with total internal reflection, reflexive solutions and mixed solutions can be considered. For example, Figure 7 shows a
microlens 22 operating by total internal reflection associated to refractive off-axis microlenses 14. Figure 8 shows an optical sensor with areflexive lens 24 associated to refractive off-axis microlenses 14. - An additional possibility for obtaining a non constant angular separation αi between the
central directions 20 is to use a matrix of micro-objectives, as shown in Figure 9. The micro-objectives 15 are composed by at least two superposed micro-objectives. The doublet optical solution allows to vary effective focal length (e.f.l.) maintaining the back focal length (b.f.l.) equal for all micro-objectives. In this way, the distance between thephotodetectors matrix 12 and themicro-objectives matrix 15 is constant (b.f.l.) whilst the possibility of varying e.f.l. allows to have a non constant separation between thecentral directions 20, i.e. α1≠α2≠α3. In this case, the IFOV are varied at the same time. - For the optical systems described above, a possible alternative to the use of the
diaphragms 18 consists of using a matrix with photo-detectors 12 having smaller size than thediaphragms 18. Lastly for equal sizes of the sensitive area of thephotodetectors matrix 12 it is possible to have either a highresolution photodetectors matrix 12, or aphotodetectors matrix 12 with larger size and hence smaller resolution. As shown in Figure 10a, using a lowresolution photodetectors matrix 12, adiaphragms matrix 18 is used to vary thecentral direction 20 associated to each microlens 14 (case described in the previous optical solutions). Instead, using a highresolution photodetectors matrix 12, Figure 10b, only somephotodetectors 12, corresponding to thecentral direction 20 of themicrolens 14 to be obtained, are activated, and the others are rendered inactive, with no need to use adiaphragms matrix 18. The advantages of using a high resolution matrix consist of eliminating thediaphragms 18 and being able to reconfigure (also while acquiring the images from the matrix) the active photodetectors in such a way as to change thecentral directions 20 of the fields of view. The disadvantages are due to the fact that the displacement of thecentral directions 20 occurs by discrete steps (the minimum pitch is equal to the size of the photodetector 12) and not continuously as when using thediaphragms 18, and that to visualise the images requires pre-processing for addressing the active photodetectors. - The microlenses of the optical sensor according to the present invention can be constituted by GRIN (gradient index) material. On the lower plane of some microlens or of some subgroup of microlenses can be deposited a selective interferential coating operating as a filter to transmit only the wavelengths of interest. For some functions, for example, a NIR (near infrared) LED illuminator can be used, the related spectral band has to be selected with respect to the background.
- The photodetectors matrix is in CCD or CMOS technology, standard or with parallel architecture (pre-processing at the photodetector level).
- The integration of multiple functions on a photodetectors matrix coupled to a microlenses matrix according to
claim 1 is in accordance with the following rules: - each function is associated to a single microlens or to multiple microlenses, not mutually contiguous, or to a single subgroup of mutually contiguous microlenses or to multiple, not mutually contiguous subgroups of microlenses.
- each microlens is associated to a single photodetector or to a subgroup (cluster) of photodetectors;
- the contiguous photodetectors able to be associated to a function define a sub-area (ROI or Region Of Interest);
- some photodetectors can be used only for separating the sub-areas;
- the angular separation between the central directions of the fields of view (IFOV) relating to photodetectors or clusters of adjacent photodetectors is not constant within the matrix.
- The condition whereby the angular separation between the central directions of the fields of view (IFOV) relating to adjacent photodetectors or clusters of photodetectors is not constant within the matrix occurs in the following cases:
- there are adjacent sub-areas dedicated to as many functions in which the fields of view of the sub-areas are different;
- at least one of the functions integrated on the photodetectors matrix is associated to a single subgroup of microlenses which subtends a solid angle FOV, but, thanks to the fact that the central direction of the field of view IFOV of each individual microlens can be established independently, contiguous photodetectors or clusters or photodetectors, associated to the subgroup of microlenses, do not always have mutually adjacent IFOV (this case will be described and illustrated more extensively in the subsequent paragraph "Matrix shape");
- the microlenses of at least one subgroup have different and mutually contiguous fields of view, in such a way as to obtain a different resolution inside the global field of view of said subgroup (this case will be described and illustrated more extensively in the paragraph "Frontal monitoring");
- the microlenses of at least one subgroup have equal but not mutually contiguous fields of view, in such a way as to sample in non continuous fashion the global field of view of said subgroup of microlenses and therefore obtain a different resolution inside the global field of view of said subgroup of microlenses (this case will be described and illustrated more extensively in the paragraph "Frontal monitoring");
- The variation in angular separation between the central directions of the fields of view relating to photodetectors or clusters of photodetectors can be obtained:
- modifying the distance between the centres of the diaphragms positioned in front of the photodetector or cluster of photodetectors (Figure 5);
- modifying the prismatic component in the case of off-axis microlenses matrices (Figure 6).
- The variation in the field of view of an individual microlens can be obtained:
- modifying the diameter of the diaphragms positioned in front of the photodetector or cluster of photodetectors;
- using the solution with micro-objectives (Figure 9).
- Based on the above rules, additional innovative elements can be identified, which will be described individually hereafter, relating to:
- 1. matrix shape;
- 2. frontal monitoring;
- 3. zoom;
- 4. optical pre-processing;
- 5. matrix partition.
- Figure 11 is a schematic view showing the object and image planes of two optical systems, the first one constituted by a single aperture optics 8 (Figure 11a) and the second one constituted by a 2D matrix of microlenses 14 (Figure 11b), said optical systems being coupled to matrices 2D of
photodetectors 12, said matrices having the same format mxn. - In Figure 11a the
photodetectors 12 subtend the IFOV according to the laws of geometric optics applied to the lens 8: adjacent portions of the plane of the objects are subtended in the image plane by mutuallycontiguous photodetectors 12. - In Figure 11b, which constitutes prior art like the previous Figure 11a, the single aperture lens is replaced by a
microlenses matrix 14 and the previous considerations continues to apply: adjacent portions of the object plane are subtended in the image plane by mutuallycontiguous photodetectors 12. - However, the
central direction 20 of the field of view IFOV of eachindividual microlens 14 can be established independently. Therefore in Figure 12 themicrolenses 14 are so positioned that the previous rule no longer applies and thus the angular separation between thecentral directions 20 of the fields of view relating to adjacent photodetectors is not constant within the matrix. However, in this case a pre-processing for addressing the photodetectors is necessary in order to visualise the images. - The previous example can be considered as a generalisation of particular cases, two of which are illustrated below.
- A 2D matrix of
mxn photodetectors 12 having asingle aperture lens 8 with field of view of x horizontal degrees and y vertical degrees (Figure 11a) can be redesigned as a 1D linear matrix ofmxn photodetectors 12, in such a way that each microlens 14 associated to the correspondingphotodetector 12 has a field of view of x/m horizontal degrees and y/n vertical degrees withcentral direction 20 such as to cover a portion of the global field of view of x horizontal degrees and y vertical degrees (Figure 13). - This can be useful, for example, when it is necessary to perform the 2D monitoring of a scene and the surface available for integrating the sensor is sufficient only for a 1D linear matrix of mxn photodetectors and not for a 2D matrix of mxn photodetectors.
- The fact that the
central direction 20 of the field of view IFOV of eachindividual microlens 14 can be established independently, can be used also to optimise the partition of the photodetectors matrix into sub-areas dedicated to specific functions and in particular to exploit the entire sensitive area of the matrix. - Figure 14 shows, by way of example, the case in which there is a need to view a portion of scene with a field of view of x horizontal degrees and y vertical degrees, where x=y, and the sub-area available on the
sensor 10, constituted by a matrix 2D ofphotodetectors 12, is rectangular: if the same resolution has to be maintained along the two axes x and y of the object plane, and therefore the same field of view has to be maintained for eachmicrolens 12, the microlenses can be positioned on the rectangular sub-area as shown in Figure 14. - The format of the TV camera, used in systems with single aperture lens for monitoring the scene in front of the vehicle, depends mainly on two parameters: field of view FOV and resolution R needed in the areas of the scene in which some objects have to be discriminated with higher precision (e.g., horizontal signs on the road surface). This means that in the other areas of the scene in which there are no objects of interest the previous resolution R is wholly redundant.
- For most of the frontal monitoring functions, the format of the camera must be at least CIF (320x256 pixels) or VGA (640x480 pixels).
- These formats are not compatible with the optical solutions based on micro-optics matrices proposed above, where the size of the photodetector is in the order of tens of microns, i.e. far larger than that of the photodetectors (less than 10 microns) of the standard matrices used today for consumer or automotive applications. The use of photodetectors, with dimensions in the order of tens of microns, combined with high resolution means excessively expanding the total area of the chip and consequently raising fabrication costs.
- In the case of optical solutions based on micro-optics matrices, it is necessary to design the subgroup of microlenses, dedicated to the frontal monitoring function, so that the fields of view IFOV of the individual photodetectors (or clusters of photodetectors), mutually contiguous, are not kept constant for the whole field of view FOV of the microlenses subgroup, but they are defined on the basis of the resolutions actually required in the different areas of the scene as shown in Figure 15. Consequently, the angular separation between the central directions of the fields of view IFOV of the individual mutually contiguous photodetectors (or clusters of photodetectors) is not constant.
- This approach enables to define a higher resolution in the point of escape of the images relative to that of the peripheral area, as shown in Figure 16. Proceeding from the point of escape towards the outer edges of the matrix, both the field of view IFOV of the individual photodetectors (or of the photodetectors clusters) and the angular separation between the central direction of the IFOV increase, or else only the angular separation between the central directions of the IFOV increases, whilst the IFOV are instead maintained constant, in order to sample non continuously the portion of the scene that requires a lower resolution.
- The frontal scene monitoring functions are manifold (vehicle crossing, Lane Warning, curve detection, vertical signs detection, pedestrian monitoring, etc.).
- To integrate all these functions on a same photodetectors matrix, coupled with a single aperture lens, it is first of all necessary to evaluate the functional specifications in terms of field of view, minimum and maximum range, resolution of a reference obstacle at the maximum distance. Combining these specifications enables to define the format of the matrix, which will obviously be sufficient for some functions and redundant for others. With this approach, the format of the camera will definitely be greater than VGA.
- An alternative that reduces the format of the TV camera entails the use of an optical zoom. However, the size and complexity of an optical zoom make it difficult to integrate it with other optical systems dedicated to the environmental parameters measuring functions (note the complexity of the optical sensor shown in Figure 1, according to the invention of the document
EP-A-1 418 089 ). Moreover, the optical zoom increase the fabrication costs of the sensor. - If a matrix of microlenses is used instead of single aperture optical systems, the sub-area dedicated to frontal scene monitoring can be optimised, increasing resolution in the areas where details need to be discriminated (horizontal signs, obstacle recognition, etc.) and reducing it in the areas where the necessary information is more qualitative (road edges, horizon, etc.). This is equivalent to processing the images with the optimal resolution, as is made possible by an optical zoom.
- The solutions for varying resolutions have already been discussed in the previous paragraph "Frontal monitoring".
- The ability to design the microlenses matrix defining the direction and amplitude of the field of view for each of them allows to simplify image processing.
- A possible optical pre-processing function consists of applying optical filters in order to pre-transform the image for subsequent processing. With a single aperture optics, a high resolution is required even in non significant areas in order to have sufficient resolution to identify some areas of the images. Instead, using different fields of view for each microlens or subgroups of microlenses, it is possible to define the sub-areas of the matrix with appropriate resolution and field of view, in order to simplify the image processing operation.
- With reference to figure 17, an additional possibility consists of defining on the sensitive area of the matrix k groups of j photodetectors, each able to create (by means of a single microlens or a matrix of j microlenses) the image of the same portion of scene or of different portions of scene. On each group of j photodetectors are positioned diaphragms with different shapes. When a group of photodetectors frames a portion of scene that matches the shape of the diaphragm, the signal is the highest. This approach can be used, for example, for the Lane Warning function, as shown in Figure 17, in which there are k/2 groups of j photodetectors which view the left part of the road scenario (
type 1 region of interest) and k/2 groups of j photodetectors which view the right part of the road scenario (type 2 region of interest). - The example shown in Figure 12 can now be analysed from a different viewpoint. As explained above, the
microlenses 14 are so positioned that adjacent portions of the object plane are not subtended, in the image plane, by mutuallycontiguous photodetectors 12 and, therefore, the angular separation between thecentral directions 20 of the fields of view relating toadjacent photodetectors 12 is not constant within the matrix. Based on this general example, it is possible to design k subgroups of microlenses with such field of view as to view k portions of the scene which exactly match the shape of the diaphragms of Figure 17, to select k positions of the horizontal demarcation lines. Said k subgroups of microlenses, however, are positioned horizontally on the photodetectors matrix: the first subgroup starting from the top left corner of the matrix and proceeding rightwards, the second one starting from the end of the first subgroup and proceeding rightwards and so on; reaching the right edge of the matrix, the subsequent line is started. - Figure 18a shows the enlargement of both the subgroup of photodetectors relating to an ROI and of the photodetectors actually exposed to the radiation that passes through diaphragm. Figure 18b shows the positioning of the kth subgroup of microlenses as described above.
- The advantages of this solution are: the removal of the diaphragms of Figure 17, the use of a matrix with smaller format, the elimination of any form of pre-processing for addressing the photodetectors, mentioned in the example of Figure 12.
- The partition of the sensitive area of the photodetectors matrix can have different configurations according to the number and type of integrated functions.
- Figure 19 shows, by way of example, a first possible partition of the sensitive area of the photodetectors matrix, already mentioned in the document
EP-A-1 418 089 by the same Applicant. The same functions can be integrated differently according to the inventive elements of the present patent application. - The so-called "twilight" function is performed by a sub-area of the matrix that has to measure environmental illumination. The number of photodetectors dedicated to this function can even be reduced to just one and there are no constraints in terms of positioning on the sensitive area of the matrix. According to a preferred characteristic, a central photodetector (or a few photodetectors) is surrounded (are surrounded) by eight or more photodetectors that have different fields of view, the central one(s) larger and the lateral ones smaller, in order to have information both about the intensity of environmental lighting (central photodetector(s)) and about the intensity and direction of solar irradiation (lateral photodetectors). The information about environmental illumination enables automatically to turn on/off the headlights of the vehicles in conditions of poor illumination. The information on the direction of the solar illumination enables to optimise the air conditioning system of the vehicle, e.g. for activating and regulating the air flows of multi-zone air conditioning systems. According to an additional preferred characteristic, some photodetectors are oriented towards the dashboard of the vehicle to measure the radiation directed thereon (diffused radiation on the photodetectors). The photodetectors dedicated to the illumination and solar irradiation function can be positioned separately from each other, i.e. in non contiguous positions.
- With regard to the fog detection function (based on active technique), the number of photodetectors can even be reduced to just one and there are no constraints in terms of positioning on the sensitive area of the matrix.
- For the tunnel function, the number of photodetectors can even be reduced to just one and there are no constraints in terms of positioning on the sensitive area of the matrix. According to a preferred characteristic, a photodetector (a few photodetectors) has (have) a frontal field of view of about 20° and a second photodetector (a few photodetectors) a smaller field of view, e.g. about 10°.
- The sub-area of the sensitive matrix marked with "frontal monitoring" performs the so-called "Lane Warning" function. Preferably, the area of interest (i.e. the area that is used effectively for image processing) is a trapezoid and therefore the number of photodetectors dedicated to this function is reduced from the one described in the document
EP-A-1 418 089 by the same Applicant. Preferably, the field of view of the photodetectors are smaller (higher resolution) in the areas of the images in which the lane demarcation lines could be located. This enables to reduce the number of photodetectors dedicated to this functions. - The area called "frontal monitoring", alternatively or in addition to the "Lane Warning" function, can be dedicated to the vehicle crossing detection function. According to a preferred characteristic, the area of interest (i.e. the area that is used effectively for image processing) is a trapezoid and therefore the number of photodetectors dedicated to this function is reduced from the one described in the document
EP-A-1 418 089 by the same Applicant. The fields of view of the photodetectors are smaller (higher resolution) in the areas of the image where the potential indicators of the presence of headlights of a crossed vehicle or of taillights of a vehicle that precedes the reference vehicle (the one whereon the sensor is mounted) could be located. According to a preferred characteristic, two subgroups of photodetectors are provided to perform this function: the photodetectors of the first subgroup have fields of view that assure long range monitoring of the scene to detect vehicles that arrive from the opposite lane, and the photodetectors of the second subgroup have fields of view that assure short range monitoring of the scene to detect the vehicles that precede the reference vehicle. - According to a preferred characteristic, a set of photodetectors positioned in the unused areas of the sub-area for the frontal monitoring function can be dedicated to lateral monitoring i.e. to the detection of the presence of a curve for commanding the adaptive headlights of the vehicle.
- The portion of sensitive area called "frontal monitoring" can serve a combination of multiple functions, e.g. Lane Warning, vehicle crossing, curve detection, etc. The sub-area dedicated to such functions is preferably constituted by photodetectors whose microlenses have optimised directions and fields of view: high resolution only in the areas where the objects of interest for the processing algorithms could be located, low resolution in non interesting areas (e.g. the horizon). The result is comparable to the one that would be obtained with an optical zoom.
- Figure 20 shows a second example of partition of the sensitive area of the photodetectors matrix, already described in the document
EP-A-1 418 089 by the same Applicant. The same functions can be integrated according to the inventive elements of the present patent application. - For the rain/fogging function, there are no constraints in terms of positioning on the sensitive area of the matrix. In the solution described in the document
EP-A-1 418 089 , in order to have the rain drops on a same image plane, the optical axis had to be perpendicular to the windshield. According to the present invention, the use of microlenses allows to maintain the optical axis of the microlenses matrix parallel to the road plane. It is possible to compensate for the different distance between the photodetectors matrix and the windshield whereon the rain drops lie by designing microlenses with different focal length, as shown in the schematic representation of Figure 21. In this figure, the windshield of the vehicle is designated by thenumber 30. Thenumber 32 schematically indicates water drops deposited on the outer surface of the windshield. The references 14', 14'', 14''' designate microlenses with different focal length positioned in such a way that the respective focalisation points fall on the plane of the photodetectors independently of the different distance of the microlenses relative to the windshield. - With reference to Figure 22, the
number 36 designates device able to perform the function of monitoring the vehicle occupant. The photodetectors matrix, designated by thereference 38, is oriented towards the front of the vehicle. Somephotodetectors 40, e.g. positioned in the bottom left and right corner of thematrix 38, are used to determine the position of the driver and the presence, type and position of the passenger. Since this function does not require optics capable of creating the image of the entire vehicle but only of monitoring and discriminating the presence of passengers, a number of photodetectors equal, for example, to 9 is indicated. Since the interior of the vehicle is positioned to the rear of the active side of thephotodetectors matrix 38, to view the scene anoptical system 42 is used, able to receive the image positioned to the rear of the photodetectors matrix. For instance, theoptical system 42 can be a wave guide element as shown in Figure 22. Alternatively, theoptical system 42 can comprise prismatic elements (not shown). The field of view of the optical system is designated by thereference number 44. - Figure 23 shows an additional possibility of partition of the useful surface of the photodetectors matrix. This partition enables to integrate a higher number of functions on a matrix with a reduced format (e.g., CIF).
- Figure 24 shows the possible arrangement of
sensors 10 according to the present invention for performing the function of viewing the blind angle. According to a preferred characteristic of the invention, to perform this function eachsensor 10 can have two different fields of view to cover different directions and distances, so that the arriving vehicle crosses the two beams at different times, generating a stepped signal that can be used to signal the danger.
Claims (23)
- A multifunctional optical sensor, comprising a matrix of photodetectors (12) of the COD or CMOS type, having a sensitive area divided into sub-areas, each of which, individually or combined with others, is dedicated to a specific function of monitoring the scene observed by the sensor or measuring environmental parameters, wherein:- said optical sensor comprises a matrix of microlenses (14, 15), each microlens being set to focus the radiation coming from a portion of solid angle (16) on the associated photodetector (12) or cluster of mutually contiguous photodetectors (12),- each function is associated to a single microlens or to a single subgroup of mutually contiguous microlenses or to multiple, not mutually contiguous microlenses or to not mutually contiguous subgroups of microlenses,- the contiguous photodetectors are able to be associated to a function of defining a sub-area, which is a region of interest (ROI), and- the angular separation between the central directions of the portions of solid angle (16) subtended by adjacent photodetectors or adjacent clusters of photodetectors is not constant within the matrix,characterised in that
said not constant angular separation is provided either by:- the sensor comprising a matrix of diaphragms (18) being positioned between the photodetectors matrix (12) and the microlenses matrix (14), the distances between the centres of the diaphragms being different from the dimensions of the microlenses, whereby said matrix of diaphragms (18) by changing the distances between the centres of the diaphragms enables to select for each photodetector (12) or cluster of photodetectors a portion of the solid angle subtended by each microlens (14),
or by- the sensor comprising a matrix of diaphragms (18) being positioned between the photodetectors matrix (12) and the microlenses matrix (14), the distances between the centres of the diaphragms being equal to the dimensions of the microlenses, and the microlenses (14) being centred on a respective photodetector (12) or cluster of photodetectors, each microlens (14) being designed in such a way as to select for each photodetector (12) or cluster of photodetectors a portion of the solid angle subtended by each microlens,
or by- the sensor comprising a matrix of diaphragms (18) being positioned between the photodetectors matrix (12) and the microlenses matrix (14), the distances between the centres of the diaphragms being different from the dimensions of the microlenses, the microlenses being a matrix of micro-objectives (15) each of which is constituted by at least two microlenses, each of said micro-objectives (15) being associated to a respective photodetector (12) or group of photodetectors (12) and related diaphragms, the microobjectives (15) allowing to vary effective focal length while maintaining the back focal length and thereby allow for a non-constant separation between the central directions
or by- said sensor not comprising a matrix of diaphragms and the matrix of photo-detectors (12) being a high resolution matrix including active photo-detectors (12) and inactive photo-detectors (12), the active photo-detectors (12) defining the central direction of the field of view subtended by a respective microlens. - Optical sensor as claimed in claim 1, characterised in that a separation baffle is present between contiguous microlenses.
- Optical sensor as claimed in claim 1, characterised in that said microlenses (14, 22, 24) are refractive microlenses, with total internal reflection, reflection or a combination of the above.
- Optical sensor as claimed in claim 1, characterised in that there are adjacent sub-areas dedicated to as many functions in which the fields of view of the sub-areas are different.
- Optical sensor as claimed in claim 1, characterised in that at least one of the functions integrated on the photodetectors matrix is associated to a single microlenses subgroup which subtends a solid angle field of view, and contiguous photodetectors or photodetector clusters, associated to the microlenses subgroup, which do not always have mutually adjacent individual fields of view.
- Optical sensor as claimed in claim 1, characterised in that the microlenses of at least one subgroup have different and mutually contiguous fields of view.
- Optical sensor as claimed in claim 1, characterised in that the microlenses of at least one subgroup have equal, but not contiguous fields of view, in such a way as to sample in non continuous fashion the global field of view of the microlenses subgroup.
- Optical sensor as claimed in claim 1, characterised in that there are photodetectors used solely for separating the adjacent sub-areas of the sensitive area of the photodetectors matrix.
- Optical sensor as claimed in claim 1, characterised in that said microlenses (14) are constituted by a gradient index, GRIN, material.
- Optical sensor as claimed in claim 1, characterised in that the microlenses of at least one sub-group have on their surface oriented towards the photodetectors matrix (12) a selective interferential optical coating operating as a filter to transmit only the wavelengths of interest.
- Optical sensor as claimed in claim 1, characterised in that the photodetectors matrix (12) is in CCD or CMOS technology, standard or with parallel architecture provided with pre-processing at the photodetector level (12).
- Optical sensor as claimed in claim 1, characterised in that at least one sub-area of the matrix having mxn, photodetectors is coupled to a subgroup of microlenses in such a way that adjacent portions of the objects plane are subtended in the image plane by photodetectors (12) that are not mutually contiguous.
- Optical sensor as claimed in claim 1, characterised in that a sub-area having mxn=t photodetectors which subtends a field of view of x horizontal degrees and y vertical degrees is remapped on the matrix as a sub-area having pxq=t photodetectors, wherein each photodetector has a field of view of x/m horizontal degrees and y/n vertical degrees and central direction which covers a portion of the global field of view of x horizontal degrees and y vertical degrees.
- Optical sensor as claimed in claim 1, characterised in that it comprises at least one subgroup of microlenses (14) dedicated to the frontal scene monitoring function, in which in the microlenses belonging to said subgroup the angular separation between the central directions subtended by the adjacent microlenses is not constant and the individual fields of view subtended by the microlenses are not constant, in such a way as to vary the resolution in specific areas of interest of the scene.
- Optical sensor as claimed in claim 1, characterised in that it comprises at least one subgroup of microlenses (14) dedicated to the frontal scene monitoring function, in which in the microlenses belonging to said subgroup the angular separation between the central directions subtended by the adjacent microlenses is not constant and the individual fields of view subtended by the microlenses are constant, in such a way as to sample in non continuous fashion the portion of the scene for which a lower resolution is required.
- Optical sensor as claimed in claim 1, characterised in that in the aforesaid photodetectors matrix (12) are defined k groups of j photodetectors, each group able to create the image of the same portion of scene or of different portions of scene, on said groups being positioned diaphragms with different shapes so that when a group creates the image of the portion of scene that matches the shape of the diaphragm the signal is strongest.
- Optical sensor as claimed in claim 1, characterised in that in the aforesaid photodetectors matrix (12) are defined k groups of photodetectors positioned horizontally, said groups being coupled with the respective microlenses able to create the image of k different portions of the scene.
- An optical sensor as claimed in claim 1, characterised in that it comprises at least one photodetector serving the function of measuring environmental illumination, positioned in any point of the sensitive area of the photodetectors matrix, said at least one photodetector being associated to a central microlens surrounded by a certain number of microlenses, in such a way as to obtain information both about the intensity of environmental lighting by means of said central microlens, and on the intensity and direction of solar irradiation by means of said lateral microlenses.
- Optical sensor as claimed in claim 18, characterised in that at least one of said lateral microlenses is oriented towards the dashboard of the vehicle to measure the solar irradiation directed thereon.
- Optical sensor as claimed in claim 1, characterised in that the sub-area of the photodetectors matrix destined to perform the frontal monitoring function has the shape of a trapezoid.
- Optical sensor as claimed in claim 1, characterised in that for the rain function said microlenses have their optical axis parallel to the road plane and such focal distance as to compensate for the curvature of the windshield.
- Optical sensor as claimed in claim 1, characterised in that to the occupant monitoring function in said photodetectors matrix are dedicated one or more pixels coupled to a prismatic or wave guide system able to receive the image positioned to the rear of said photodetectors matrix.
- Optical sensor as claimed in claim 1, characterised in that for the blind angle monitoring function in said photodetectors matrix are dedicated one or more subgroups of photodetectors, said subgroups of photodetectors having different fields of view in order to cover different directions with different resolutions.
Priority Applications (8)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
AT05425654T ATE385044T1 (en) | 2005-09-19 | 2005-09-19 | MULTIFUNCTIONAL OPTICAL SENSOR WITH A MATRIX OF PHOTODETECTORS COUPLED TO MICRO LENSES |
DE602005004544T DE602005004544T2 (en) | 2005-09-19 | 2005-09-19 | Multifunctional optical sensor with a matrix of photodetectors coupled to microlenses |
EP05425654A EP1764835B1 (en) | 2005-09-19 | 2005-09-19 | Multifunctional optical sensor comprising a matrix of photodetectors coupled microlenses |
CNB2006101398863A CN100573891C (en) | 2005-09-19 | 2006-09-19 | The multifunctional optical sensor that comprises the photo-detector matrix that is coupled to matrix of microlenses |
US11/533,089 US7518099B2 (en) | 2005-09-19 | 2006-09-19 | Multifunctional optical sensor comprising a photodetectors matrix coupled to a microlenses matrix |
JP2006252768A JP2007086068A (en) | 2005-09-19 | 2006-09-19 | Multi-functional photosensor equipped with photodetector matrix connected to microlens matrix |
US12/212,811 US20090236505A1 (en) | 2005-09-19 | 2008-09-18 | Multifunctional optical sensor comprising a photodetectors matrix coupled to a microlenses matrix |
JP2011001465U JP3168177U (en) | 2005-09-19 | 2011-03-18 | Multifunctional optical sensor with a photodetector matrix coupled to a microlens matrix |
Applications Claiming Priority (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP05425654A EP1764835B1 (en) | 2005-09-19 | 2005-09-19 | Multifunctional optical sensor comprising a matrix of photodetectors coupled microlenses |
Publications (2)
Publication Number | Publication Date |
---|---|
EP1764835A1 EP1764835A1 (en) | 2007-03-21 |
EP1764835B1 true EP1764835B1 (en) | 2008-01-23 |
Family
ID=35954120
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
EP05425654A Not-in-force EP1764835B1 (en) | 2005-09-19 | 2005-09-19 | Multifunctional optical sensor comprising a matrix of photodetectors coupled microlenses |
Country Status (6)
Country | Link |
---|---|
US (2) | US7518099B2 (en) |
EP (1) | EP1764835B1 (en) |
JP (2) | JP2007086068A (en) |
CN (1) | CN100573891C (en) |
AT (1) | ATE385044T1 (en) |
DE (1) | DE602005004544T2 (en) |
Families Citing this family (55)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US7502474B2 (en) * | 2004-05-06 | 2009-03-10 | Advanced Micro Devices, Inc. | Network interface with security association data prefetch for high speed offloaded security processing |
WO2005115820A1 (en) | 2004-05-25 | 2005-12-08 | Siemens Aktiengesellschaft | Monitoring unit in addition to an assist system for motor vehicles |
US7701636B2 (en) | 2008-03-06 | 2010-04-20 | Aptina Imaging Corporation | Gradient index microlenses and method of formation |
CN102177442B (en) | 2008-10-10 | 2014-10-22 | 皇家飞利浦电子股份有限公司 | Light directionality sensor |
DE102008061760A1 (en) * | 2008-12-12 | 2010-06-17 | Daimler Ag | Device for monitoring an environment of a vehicle |
DE102008062977A1 (en) * | 2008-12-23 | 2010-06-24 | Adc Automotive Distance Control Systems Gmbh | Optical module with multifocal optics for the detection of distance and near range in one image |
JP5320180B2 (en) * | 2009-06-19 | 2013-10-23 | ナイルス株式会社 | Vehicle rain / occupant detection sensor |
DE102009027372A1 (en) * | 2009-07-01 | 2011-01-05 | Robert Bosch Gmbh | Camera for a vehicle |
EP2452213B1 (en) | 2009-07-06 | 2015-10-21 | Conti Temic microelectronic GmbH | Optical module for simultaneously focusing on two fields of view |
DE102009053452A1 (en) * | 2009-11-17 | 2011-06-16 | Conti Temic Microelectronic Gmbh | Camera system for use in vehicle for mounting on inclined plate, has image sensor with sensitive surface for detecting electromagnetic radiation and micro lens arrangement for projecting electromagnetic radiation on image sensor |
EP2336749B1 (en) | 2009-12-21 | 2015-09-02 | C.R.F. Società Consortile per Azioni | Optical detection system for motor-vehicles having multiple functions, including detection of the condition of the road surface |
US8836848B2 (en) * | 2010-01-26 | 2014-09-16 | Southwest Research Institute | Vision system |
WO2011111079A1 (en) | 2010-03-11 | 2011-09-15 | Datalogic Scanning Group Sr.L. | Image capturing device |
US8942964B2 (en) | 2010-06-08 | 2015-01-27 | Southwest Research Institute | Optical state estimation and simulation environment for unmanned aerial vehicles |
JP5638328B2 (en) * | 2010-09-21 | 2014-12-10 | Jx日鉱日石エネルギー株式会社 | Lubricating oil deterioration monitoring device |
GB201020024D0 (en) | 2010-11-25 | 2011-01-12 | St Microelectronics Ltd | Radiation sensor |
GB201020023D0 (en) | 2010-11-25 | 2011-01-12 | St Microelectronics Ltd | Radiation sensor |
EP2646802A1 (en) | 2010-11-30 | 2013-10-09 | Conti Temic Microelectronic GmbH | Detection of raindrops on a pane by means of a camera and lighting |
GB2485996A (en) | 2010-11-30 | 2012-06-06 | St Microelectronics Res & Dev | A combined proximity and ambient light sensor |
GB2486000A (en) | 2010-11-30 | 2012-06-06 | St Microelectronics Res & Dev | Optical proximity detectors with arrangements for reducing internal light propagation from emitter to detector |
GB2485998A (en) | 2010-11-30 | 2012-06-06 | St Microelectronics Res & Dev | A single-package optical proximity detector with an internal light baffle |
DE102011017355B4 (en) * | 2011-04-16 | 2022-08-11 | Continental Autonomous Mobility Germany GmbH | Multifocal imaging system with optical filters |
US8466406B2 (en) | 2011-05-12 | 2013-06-18 | Southwest Research Institute | Wide-angle laser signal sensor having a 360 degree field of view in a horizontal plane and a positive 90 degree field of view in a vertical plane |
US9928707B2 (en) * | 2011-05-16 | 2018-03-27 | Garrett Thermal Systems Limited | Surveillance system |
DE102011103302A1 (en) | 2011-06-03 | 2012-12-06 | Conti Temic Microelectronic Gmbh | Camera system for a vehicle |
CN103139470A (en) * | 2011-11-30 | 2013-06-05 | 索尼公司 | Digital imaging system |
DE102011056051A1 (en) | 2011-12-05 | 2013-06-06 | Conti Temic Microelectronic Gmbh | Method for evaluating image data of a vehicle camera taking into account information about rain |
US20130169606A1 (en) * | 2011-12-30 | 2013-07-04 | Qualcomm Mems Technologies, Inc. | Light direction distribution sensor |
DE102012103873A1 (en) | 2012-05-03 | 2013-11-21 | Conti Temic Microelectronic Gmbh | Detecting raindrops on a glass by means of a camera and lighting |
DE102012105434B4 (en) * | 2012-06-22 | 2014-06-05 | Conti Temic Microelectronic Gmbh | Camera system for a vehicle for acquiring image data from a plurality of distance ranges |
DE102013208677A1 (en) * | 2013-05-13 | 2014-11-13 | Robert Bosch Gmbh | Image recorder for taking a two-dimensional optical image |
US10203399B2 (en) | 2013-11-12 | 2019-02-12 | Big Sky Financial Corporation | Methods and apparatus for array based LiDAR systems with reduced interference |
CN103715215B (en) * | 2013-12-30 | 2018-10-16 | 上海集成电路研发中心有限公司 | A kind of image pixel elements to angular-sensitive |
CN103745984B (en) * | 2013-12-31 | 2018-06-22 | 上海集成电路研发中心有限公司 | It can measure pel array, imaging sensor and the method for incident light angle |
US9360554B2 (en) | 2014-04-11 | 2016-06-07 | Facet Technology Corp. | Methods and apparatus for object detection and identification in a multiple detector lidar array |
DE102014108239A1 (en) * | 2014-06-12 | 2015-12-17 | Hella Kgaa Hueck & Co. | Method for adaptively controlling a high-resolution headlight system |
US9506803B2 (en) | 2014-09-17 | 2016-11-29 | Delphi Technologies, Inc. | Vehicle optical sensor system |
US10036801B2 (en) | 2015-03-05 | 2018-07-31 | Big Sky Financial Corporation | Methods and apparatus for increased precision and improved range in a multiple detector LiDAR array |
US9992477B2 (en) | 2015-09-24 | 2018-06-05 | Ouster, Inc. | Optical system for collecting distance information within a field |
US9866816B2 (en) | 2016-03-03 | 2018-01-09 | 4D Intellectual Properties, Llc | Methods and apparatus for an active pulsed 4D camera for image acquisition and analysis |
US10023120B2 (en) * | 2016-03-30 | 2018-07-17 | Delphi Technologies, Inc. | Multi-purpose camera device for use on a vehicle |
FR3050831B1 (en) * | 2016-04-29 | 2018-04-27 | Silios Technologies | MULTISPECTRAL IMAGING DEVICE |
US10274599B2 (en) * | 2016-06-01 | 2019-04-30 | Toyota Motor Engineering & Manufacturing North America, Inc. | LIDAR systems with expanded fields of view on a planar substrate |
EP3487175A1 (en) * | 2017-11-21 | 2019-05-22 | Reliance Core Consulting LLC | Methods and systems for detecting motion corresponding to a field of interest |
JP6922705B2 (en) * | 2017-12-06 | 2021-08-18 | トヨタ自動車株式会社 | Sensor mounting structure |
US10953797B2 (en) * | 2018-04-05 | 2021-03-23 | Toyota Motor Engineering & Manufacturing North America, Inc. | Cloaking devices with converging lenses and coherent image guides and vehicles comprising the same |
CN108387974B (en) * | 2018-05-02 | 2023-11-07 | 中国人民解放军国防科技大学 | High-power optical fiber laser receiving and transmitting integrated end cap |
KR102079349B1 (en) * | 2018-10-29 | 2020-04-07 | 니덱모빌리티코리아 주식회사 | Sun Sensor Module |
CN110027458B (en) * | 2019-05-08 | 2020-12-22 | 嘉兴市万荣电器股份有限公司 | Self-balancing new energy automobile protection small table |
CN112882356B (en) * | 2019-11-29 | 2021-11-12 | 上海微电子装备(集团)股份有限公司 | Photoelectric detector, focusing and leveling device and photoetching projection equipment |
CN113534403B (en) * | 2020-03-31 | 2023-03-10 | 华为技术有限公司 | Camera module and electronic equipment |
CN112584015B (en) * | 2020-12-02 | 2022-05-17 | 达闼机器人股份有限公司 | Object detection method, device, storage medium and electronic equipment |
CN112600994B (en) * | 2020-12-02 | 2023-04-07 | 达闼机器人股份有限公司 | Object detection device, method, storage medium, and electronic apparatus |
DE102021132724A1 (en) * | 2021-12-10 | 2023-06-15 | Valeo Schalter Und Sensoren Gmbh | OPTICAL VEHICLE ENVIRONMENT SENSOR, VEHICLE AND MANUFACTURING PROCESS |
CN114283602A (en) * | 2021-12-17 | 2022-04-05 | 一汽奔腾轿车有限公司 | V2X road condition projection system based on microlens array |
Family Cites Families (14)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
US5550677A (en) * | 1993-02-26 | 1996-08-27 | Donnelly Corporation | Automatic rearview mirror system using a photosensor array |
JPH08107194A (en) * | 1994-10-03 | 1996-04-23 | Fuji Photo Optical Co Ltd | Solid state image sensor |
US5751492A (en) * | 1996-06-14 | 1998-05-12 | Eastman Kodak Company | Diffractive/Refractive lenslet array incorporating a second aspheric surface |
GB9618717D0 (en) * | 1996-09-07 | 1996-10-16 | Philips Electronics Nv | Image sensor |
JPH10239777A (en) * | 1997-02-28 | 1998-09-11 | Kuraray Co Ltd | Rear projection type video display device |
US6381072B1 (en) * | 1998-01-23 | 2002-04-30 | Proxemics | Lenslet array systems and methods |
DE19909986C2 (en) * | 1999-03-06 | 2002-08-29 | Kostal Leopold Gmbh & Co Kg | Optoelectronic monitoring device for a motor vehicle |
EP1107316A3 (en) * | 1999-12-02 | 2004-05-19 | Nikon Corporation | Solid-state image sensor, production method of the same and digital camera |
US6995800B2 (en) * | 2000-01-27 | 2006-02-07 | Canon Kabushiki Kaisha | Image pickup apparatus utilizing a plurality of converging lenses |
ITTO20020950A1 (en) * | 2002-11-05 | 2004-05-06 | Fiat Ricerche | MULTIFUNCTIONAL INTEGRATED VISION SYSTEM, WITH DIE |
ITTO20030770A1 (en) * | 2003-10-02 | 2005-04-03 | Fiat Ricerche | LONG-DETECTION DETECTOR LONG ONE |
US7081998B2 (en) * | 2003-10-23 | 2006-07-25 | Sanyo Electric Co., Ltd. | Solid-state imaging apparatus |
DE102004003013B3 (en) * | 2004-01-20 | 2005-06-02 | Fraunhofer-Gesellschaft zur Förderung der angewandten Forschung e.V. | Optical imaging system for timepiece, portable computer, mobile telephone, spectacles, clothing item, chip card or sticker using array of optical channels with relatively angled optical axes |
US7187502B2 (en) * | 2005-06-09 | 2007-03-06 | Microalign Techologies, Inc. | Compact optical assembly for imaging a remote object |
-
2005
- 2005-09-19 EP EP05425654A patent/EP1764835B1/en not_active Not-in-force
- 2005-09-19 AT AT05425654T patent/ATE385044T1/en not_active IP Right Cessation
- 2005-09-19 DE DE602005004544T patent/DE602005004544T2/en active Active
-
2006
- 2006-09-19 JP JP2006252768A patent/JP2007086068A/en active Pending
- 2006-09-19 CN CNB2006101398863A patent/CN100573891C/en not_active Expired - Fee Related
- 2006-09-19 US US11/533,089 patent/US7518099B2/en not_active Expired - Fee Related
-
2008
- 2008-09-18 US US12/212,811 patent/US20090236505A1/en not_active Abandoned
-
2011
- 2011-03-18 JP JP2011001465U patent/JP3168177U/en not_active Expired - Lifetime
Also Published As
Publication number | Publication date |
---|---|
EP1764835A1 (en) | 2007-03-21 |
US20070096010A1 (en) | 2007-05-03 |
JP3168177U (en) | 2011-06-02 |
JP2007086068A (en) | 2007-04-05 |
CN100573891C (en) | 2009-12-23 |
DE602005004544D1 (en) | 2008-03-13 |
ATE385044T1 (en) | 2008-02-15 |
US7518099B2 (en) | 2009-04-14 |
US20090236505A1 (en) | 2009-09-24 |
DE602005004544T2 (en) | 2008-04-30 |
CN1937236A (en) | 2007-03-28 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
EP1764835B1 (en) | Multifunctional optical sensor comprising a matrix of photodetectors coupled microlenses | |
DE102016215618B4 (en) | VEHICLE CAMERA DEVICE FOR IMAGE CAPTURE IN FRONT OF VEHICLE AND FOR VEHICLE CONTROL | |
US10960822B2 (en) | Vehicular rearview vision system with A-pillar display | |
US8203443B2 (en) | Vehicle vision system | |
US6396397B1 (en) | Vehicle imaging system with stereo imaging | |
US7385680B2 (en) | Camera module | |
US10137842B2 (en) | Camera system for a vehicle | |
EP2022676B1 (en) | Camera for mounting in motor vehicle | |
US20130096777A1 (en) | Vehicular image sensing system | |
CN104039610B (en) | Camera chain, the camera chain particularly for vehicle | |
US20120314069A1 (en) | Vehicle optical sensor system | |
CN106488092B (en) | Integrated camera, ambient light detection and rain sensor assembly | |
US10112552B2 (en) | Vehicle vision system with camera viewing through windshield | |
US8525881B2 (en) | Imaging system for a vehicle and mirror assembly comprising an imaging system | |
EP2869021B1 (en) | Multiple imager vehicle optical sensor system | |
CN103648854A (en) | Camera system for use in a vehicle and vehicle with such a camera system | |
US20210306537A1 (en) | Vehicular windshield-mounted forward viewing camera with coated stray light shield region of housing | |
EP3185179A1 (en) | Multiple imager vehicle optical sensor system | |
US20160107576A1 (en) | Multiple imager vehicle optical sensor system | |
US7491939B2 (en) | Photosensitive sensor and applications in the automotive field | |
CN115917424A (en) | Semiconductor device and optical structure |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
PUAI | Public reference made under article 153(3) epc to a published international application that has entered the european phase |
Free format text: ORIGINAL CODE: 0009012 |
|
AK | Designated contracting states |
Kind code of ref document: A1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR |
|
AX | Request for extension of the european patent |
Extension state: AL BA HR MK YU |
|
17P | Request for examination filed |
Effective date: 20070213 |
|
17Q | First examination report despatched |
Effective date: 20070316 |
|
GRAP | Despatch of communication of intention to grant a patent |
Free format text: ORIGINAL CODE: EPIDOSNIGR1 |
|
GRAS | Grant fee paid |
Free format text: ORIGINAL CODE: EPIDOSNIGR3 |
|
AKX | Designation fees paid |
Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR |
|
GRAA | (expected) grant |
Free format text: ORIGINAL CODE: 0009210 |
|
AK | Designated contracting states |
Kind code of ref document: B1 Designated state(s): AT BE BG CH CY CZ DE DK EE ES FI FR GB GR HU IE IS IT LI LT LU LV MC NL PL PT RO SE SI SK TR |
|
REG | Reference to a national code |
Ref country code: GB Ref legal event code: FG4D |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: EP |
|
REG | Reference to a national code |
Ref country code: IE Ref legal event code: FG4D |
|
REF | Corresponds to: |
Ref document number: 602005004544 Country of ref document: DE Date of ref document: 20080313 Kind code of ref document: P |
|
NLV1 | Nl: lapsed or annulled due to failure to fulfill the requirements of art. 29p and 29m of the patents act | ||
ET | Fr: translation filed | ||
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IS Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080523 Ref country code: FI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 Ref country code: ES Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080504 Ref country code: CH Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 Ref country code: LI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 |
|
REG | Reference to a national code |
Ref country code: CH Ref legal event code: PL |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: BG Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080423 Ref country code: AT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SI Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 Ref country code: LV Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 Ref country code: PL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 Ref country code: PT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080623 Ref country code: BE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: SE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080423 Ref country code: DK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 Ref country code: CZ Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 Ref country code: NL Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 Ref country code: SK Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: RO Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 |
|
PLBE | No opposition filed within time limit |
Free format text: ORIGINAL CODE: 0009261 |
|
STAA | Information on the status of an ep patent application or granted ep patent |
Free format text: STATUS: NO OPPOSITION FILED WITHIN TIME LIMIT |
|
26N | No opposition filed |
Effective date: 20081024 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LT Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: EE Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 Ref country code: MC Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20080930 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: CY Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 Ref country code: IE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20080919 |
|
GBPC | Gb: european patent ceased through non-payment of renewal fee |
Effective date: 20090919 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: LU Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20080919 Ref country code: HU Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080724 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: TR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080123 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GR Free format text: LAPSE BECAUSE OF FAILURE TO SUBMIT A TRANSLATION OF THE DESCRIPTION OR TO PAY THE FEE WITHIN THE PRESCRIBED TIME-LIMIT Effective date: 20080424 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: GB Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20090919 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 12 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 13 |
|
REG | Reference to a national code |
Ref country code: FR Ref legal event code: PLFP Year of fee payment: 14 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: IT Payment date: 20190912 Year of fee payment: 15 Ref country code: FR Payment date: 20190927 Year of fee payment: 15 |
|
PGFP | Annual fee paid to national office [announced via postgrant information from national office to epo] |
Ref country code: DE Payment date: 20191129 Year of fee payment: 15 |
|
REG | Reference to a national code |
Ref country code: DE Ref legal event code: R119 Ref document number: 602005004544 Country of ref document: DE |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: DE Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20210401 Ref country code: FR Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200930 |
|
PG25 | Lapsed in a contracting state [announced via postgrant information from national office to epo] |
Ref country code: IT Free format text: LAPSE BECAUSE OF NON-PAYMENT OF DUE FEES Effective date: 20200919 |