WO2018020424A1

WO2018020424A1 - A method for image recording and an optical device for image registration

Info

Publication number: WO2018020424A1
Application number: PCT/IB2017/054516
Authority: WO
Inventors: Przemyslaw SEKALSKI; Kamil GRABOWSKI; Dariusz Makowski; Marcin CHOJNACKI; Wojciech JALMUZNA; Henryk Blasinski; Piotr AMROZIK; Jakub CLAPA; Bartosz Sakowicz
Original assignee: Politechnika Lodzka
Priority date: 2016-07-29
Filing date: 2017-07-25
Publication date: 2018-02-01
Also published as: PL227466B1; PL418138A1

Abstract

The object of the invention is a method for recording an image from a matrix of photosensitive elements above which an optical element is arranged to direct the radiation incident from a wide range of angles to this matrix (1) of photosensitive elements. The method comprises a step of recording points of the deformed image by reading them from the matrix of photosensitive elements and storing in a memory. In the method according to the invention an adjustable distortion element (2) is used as the optical element. The image recording step is preceded by a calibration step in which for at least a number of distortion states of the adjustable distortion element (2) an image of an object having predefined shape is recorded. Based on the recorded image for each distortion state of the adjustable distortion element (2) a specific area of the matrix on which the useful image is formed is determined, and for each distortion state of the adjustable distortion element (2) and the corresponding specific area an image points position correction function is determined. In an acquisition step distortion of the adjustable distortion element (2) is adjusted, and reading from only the determined specific area of the matrix is performed. Thus obtained deformed image is subjected to the image points position correction according to the image points position correction function. The invention also provides an optical device for image recording having an optical element adapted to direct the radiation incident from a wide range of angles to a matrix of photosensitive elements placed under the optical element and a reading block (202) being in communication with the matrix and also with a control block (200) and a memory (205). The device according to the invention is characterized in that the optical element is an adjustable optical distortion element (2). The device is provided with a correction block (206) being in communication with the control block (200) and the memory (205). The matrix comprises at least 25 000 000 photosensitive elements.

Description

Ά method for image recording and an optical device for image registration

[ 0001 ] The invention relates to an optical device for image recording and a method for image recording.

[ 0002 ] The problem of recording images in a wide range of angles has occurred almost from the very beginning of photography and image recording techniques. Observation over a wide range of angles is nearly always associated with significant deformation caused by distortion of the optical path used. In case of digitally processed images such distortion can sometimes be corrected.

[ 0003 ] US Patent Application Publication No. 20020063802 discloses a solution employing a fisheye lens to record an image, which is next subjected to correction using digital processing. The correction allows to obtain a panoramic image from the image with deformed perspective resulting from direct recording of radiation passing through such a lens.

[0004 ] US Patent Application Publication No. 6130783 discloses a solution employing a camera facing upwardly and a convex mirror mounted over the camera. Also in this case the recorded image is strongly deformed and requires correction.

[ 0005 ] Canadian patent application No. CA2324802 discloses an imaging device comprising a CCD or CMOS matrix placed under an inverted cone shaped mirror.

[ 0006 ] American patent US 6412961 Bl states that only mirrors with a surface being a piece of a paraboloid enable faithful reconstruction of the undeformed image from an image being recorded after reflection from a mirror providing wide-angle recording. This document also discloses a mirror with an arbitrary surface selected to produce as little deformation as possible and to reduce the need for correction.

[0007 ] U.S. Patent Application Publication US20140009571 discloses a solution in which the image is divided into scenes from the "areas" of interest and for that areas separate correction is performed. A general method of correcting an image reflected from any mirror by determining a transform function is disclosed. However, this document raises the problem of insufficient use of photosensitive elements (pixels) of the matrix in some parts of images recorded after reflection from convex mirrors or refracted by fisheye lenses. In particular, due to correction of image deformations resulting from distortion of the optical element some points of the recorded image are missing and the corrected image has too low resolution in certain areas. For this reason, one cannot make enough zoom-in. This drawback could be overcame by using a larger matrix, but the use of too large matrix prevents recording from being sufficiently fast for real-time observation.

[0008 ] There is no solution that would allow continuous observation and continuous recording of a wide-angle image, as well as dynamic change of its resolution in areas of particular interest. The demand for this type of solution is quite significant, especially in monitoring applications, where it is advisable to provide the ability of performing close-ups in higher resolutions without interrupting recording of the whole area.

[ 0009 ] The aim of the invention is to solve the problem mentioned above.

[ 0010 ] A method according to the invention is for recording an image from a matrix of photosensitive elements in front of which an optical element is arranged to direct the radiation incident from a wide range of angles to the matrix of photosensitive elements. The method comprises a step of recording points of the deformed image by reading them from the matrix of photosensitive elements. The particular feature of the method according to the invention is that an adjustable distortion element is used as the optical element and the image recording step is preceded by a calibration step in which for at least a number of distortion states of the adjustable distortion element an image of an object having predefined shape is recorded. Based on thus recorded image, for each distortion state of the adjustable distortion element a specific area of the matrix on which the useful image is formed is determined, and for each distortion state of the adjustable distortion element and the corresponding specific area an image points position correction function is determined. Then in an acquisition step distortion of the adjustable distortion element is adjusted, next, reading from only the determined specific area of the matrix is performed, and the obtained deformed image is subjected to the image points position correction according to the image points position correction function. With this approach, distortion of the adjustable distortion element can be selected to obtain greater density of image points in the area of interest. Because then the image moves on the matrix, a larger matrix must be used, that is, a matrix with higher resolution. Real-time recording can be done by reading only a fragment of the matrix. Such solution enables observation in a wide range of angles, as well as performing close- ups without rotating the whole system. The distortion correction function can be selected to obtain the undeformed image or the image with a specific type of distortion, such as barrel distortion.

[0011 ] Advantageously, in the acquisition step distortion of the adjustable distortion element is adjusted also beyond the states for which the calibration was performed and values of the image points position correction function are determined based on at least one closest state.

[ 0012 ] The invention provides an optical device for image recording having an optical element adapted to direct the radiation incident from a wide range of angles to a matrix of photosensitive elements placed under the optical element and a reading block being in communication with the matrix and also with a control block and a block for transferring an image to a memory. The optical element is an adjustable distortion element. The device is provided with a correction block being in communication with the control block and with the memory via a transferring block, whereas the matrix comprises at least 25 000 000 photosensitive elements.

[ 0013 ] Advantageously, the adjustable distortion element is a mirror having a reflecting surface with adjustable shape.

[ 0014 ] Advantageously, the surface with adjustable shape is mechanically coupled with at least one piezoelectric element, with a group of electromechanical transducers, especially MEMS, or is a ferromagnetic surface and remains in a magnetic field generated by at least one coil.

[ 0015 ] Advantageously, the adjustable distortion optical element is an adaptive lens, especially a lens with variable shape or a movable fisheye lens with a servomechanism arrangement moving the lens in a plane substantially parallel to the matrix.

[0016 ] The object of the invention was shown by embodiments in the drawing in which: Fig. la schematically illustrates an embodiment of the device according to the invention with a mirror having a reflecting surface with adjustable shape, Fig. lb schematically illustrates an embodiment of the device according to the invention with a movable lens with adjustable shape, Fig. 2 presents a block diagram of an electronic system used for signal processing in the invention, Fig . 3a presents an exemplary undeformed image, Fig. 3b presents the same image deformed due to barrel distortion, Fig. 3c shows the effect of the correction function, and Fig. 3d shows the effect of the correction function for an image deformed by an optical element having no axial symmetry.

[0017 ] In the first embodiment illustrated schematically in Fig. 1, a mirror 2a in the form of an inverted quadrangular pyramid is centered over a matrix 1 of the photosensitive elements. The side walls of the mirror 2a are made of reflective plates combined with piezoelectric plates. Such a solution for adjusting a mirror is disclosed in the Polish patent PL173408B1. Changing position of the plates locally adjusts the shape of the mirror so that the rays 3d, 3e, 3f incident from a given range of directions are focused, whereas the rays 3a, 3b, 3c incident from other directions remain unchanged - as it is schematically shown in Fig. la. This allows to obtain greater density of points of the deformed image being recorded, corresponding to a particular area of interest.

[0018 ] Using this capability requires a high resolution matrix comprising at least 25 000 000 photosensitive elements. In this embodiment a 70 MPx CMOS matrix comprising 70 000 000 photosensitive elements was used. Reading such large matrices available on the market today takes too long, around 300 ms, so it is not possible to use the total resolution for real-time observation with typical refresh rates exceeding 25 fps. However, according to the invention, the whole matrix is not read, but only a fragment on which the image that requires correction is projected, the fragment being not smaller than the area of interest.

[0019] In the second embodiment illustrated in Fig. lb, a liquid lens 2b with an electrically-adjustable shape is used. Alternatively, one can use also lenses with variable shapes adjusted with piezoelectric elements, electromechanical transducers, especially MEMS, or magnetically.

[0020 ] In the third embodiment a fisheye lens located above the matrix 1 is used. Such solutions are known. Corresponding correction methods have also been widely described. In this embodiment the lens is movable and it can be moved above the matrix using a servomechanism in at least two directions, in the XY-plane parallel to the matrix. Such a solution results in a change of distortion in a given area.

[0021 ] Fig. 2 presents a block diagram of a digital system used in the device according to the invention. Digital representation of an image deformed by the mirror is read from the matrix 1. At least a fragment corresponding to the user's area of interest is read. If the matrix 1 used has analog outputs, analog-to-digital converters 201 are employed, the outputs of which are connected to the reading block 202. If the matrix 1 used has digital outputs, the signal is fed directly to the reading block 202 providing synchronization with the matrix as well as organizing information related to tags indicating end of pixel, end of block, end of line, end of frame, and other ones depending on the digital circuit that contains the matrix 1. Both the matrix 1 and the reading block 202 are in communication with a control block 200 that coordinates their operation. The whole system can be implemented in a single FPGA circuit .

[0022 ] The control block 200 determines the area of interest, so- called ROI (region of interest) . ROI is calculated taking into account user settings such as azimuth angle, horizontal angle, Pan- Tilt-Zoom, as well as rotation of the observation plane relative to the normal indicated by the settings. Because the adjustable distortion optical element 2a, 2b introduces deformation preventing the user from accurate and direct indicating the area of interest on the matrix 1, the user inputs the parameters corresponding to the undeformed image or required settings such as viewing angle and magnification, which together with the calibration allow to determine ROI of the matrix or the smallest area including ROI of the matrix in the deformed image. Thus the image read from the matrix 1 can be corrected.

[0023] he output of the reading block is in communication with a memory 205. Optionally, if the matrix 1 does not provide block reading capability, an additional image transfer block 204 is used as an interface between the reading block and the memory 205. The memory 205 is also in communication with a mosaic removing block 203. The mosaic removing block 203 is matched to the selected matrix 1 and implements the standard mechanisms for determining the color of the image points based on the recorded signals from the matrix. The mosaic removing block 203 and the reading block 202 together constitute the acquisition system. The mosaic removing block is in bidirectional communication with the memory 205 via Video Direct Memmory Access (VDMA) 208. In addition, the mosaic removing block 203 is in communication with the control block 200. The mosaic removing block 203 is used to combine signals from the photosensitive elements sensitive to particular colors, read from the memory 205, into triads representing individual color dots of the image. The mosaic removing block 203 must operate in a manner compatible with the matrix 1 used, according to the manufacturer's guidelines, and according to the distribution of color dots in the matrix. The output of the mosaic removing block is stored in the memory 205. This output is a fragment of the deformed image. Alternatively, the mosaic removing block 203 can be in unidirectional communication with the memory 205. In such a case the output of the reading block 202 is connected directly to the input of the mosaic removing block 203.

[ 0024 ] The memory 205 is in communication with a correction block

206 via an optional image transfer block 204. The correction block performs correction by means of the correction function determined in the calibration process discussed below and preceding actual image recording. The correction function assigns to each deformed image point (xl, yl) either the coordinates (x2, y2 ) that the point should have in the undeformed image, or a vector representing the displacement of the point to be applied to obtain the undeformed image. The correction block is also in communication with the control block 200. Alternatively, correction functions providing an image with distortion of a particular type can be used.

[ 0025 ] The memory 205 is in communication also with an output block

207 - e.g. a screen or image transmitting device, via an access block 208.

[ 0026 ] Owing to such a configuration, the control block 200 synchronizes operation of the matrix 1, the reading block 202, the mosaic removing block 203, and the correction block 206, making them work in simultaneous and sequential manner, to proces subsequent image blocks arranged in the memory to obtain the whole undeformed image ready to be presented to the user on the output device . [0027 ] Instead of the CMOS matrix one can use a CCD matrix or other one providing resolution not less than 25 000 000 elements and acquisition times close to those specified above.

[0028 ] In the present embodiment, CHR70M CMOSIS was used as the matrix. It is a CMOS matrix with maximum resolution of 70 MPx and 8 analog outputs. Its maximum operating frequency is 30 MHz. This matrix required the use of the analog-to-digital converter 201.

[0029] In further embodiment CMOS Python 25k OnSemi matrix was used, having resolution of 26,2 MPx and a digital LVDS output which supports rate of 32 x 720 Mbps and is connected directly to the reading block 202.

[0030 ] The invention can also be embodied with connection of the reading block 202 to memory realized by means of the mosaics removing block 203. Such approach requires synchronization of the operation of the recording block 202 with operation of the mosaics removing block by means of the control block 200, yet it allows to save access time to the memory 205.

[0031 ] The method according to the invention is performed by means of the device according to the invention. The method comprises a calibration step and an image points recording step.

[0032 ] The purpose of calibrating wide-angle optics is to determine transformation - the correction function - that has to be applied to the coordinates of the points of the deformed image in order to obtain the original image, free of deformation. This transformation is of type f (xl, yl) → (x2, y2) . An example of the effect of the correction function is shown in the drawing, in which Fig. 3a presents an undeformed image, Fig. 3b presents the recorded image deformed by the fisheye lens, and Fig. 3c illustrates how the correction function works by showing effect of subjecting undeformed image to the correction function. Fig. 3d shows correction of an image obtained from an optical element having no axial symmetry.

[0033] In the calibration step, images of the reference charts are recorded, for example according to ISO 12233 or I3A standards, however, one can use also other well-known and commercially available charts, such as charts from Imatest or ImageEngineering, as well as other kinds of reference charts comprising dots or other shapes of known distribution.

[0034] The recorded deformed image is compared to the image of the chart, and on this basis a displacement map of image points is created. Such a map is a table of size corresponding to the deformed image in which each point is assigned a vector representing the point's displacement relative to the corresponding point in the image of the reference chart.

[ 0035 ] Such a solution is valid for optical systems with constant and known geometric distortion. Due to the use of the adjustable distortion optical element, in the present invention the calibration process is more complex. The adjustment range is divided into states and for each state the correction function is determined. In addition, for each state the area in which the useful image is located is determined, since due to the adjustment some fragments of the matrix remain unused. The useful image is indicated by the user .

[ 0036 ] In some applications, the correction function is just a table of points' displacements. If range of distortion settings has to be represented by many states, this approach is ineffective because a very large amount of data has to be stored in the memory. In addition, every time the settings of the optical system are changed, i.e. distortion is adjusted, the table has to be reloaded and the new one has to be applied. For this reason sometimes the correction function is not tabulated in its entirety. Approximate values for whole blocks of image points are used. As a result, the output image almost always is slightly deformed. Another way is to determine for each calibration state the analytical functions of the form (x2, y2) = f (xl, yl, n) , where n is the state number. Because point-by- point image processing is very time-consuming, one can determine displacements for whole blocks of image points.

[0037 ] The present invention employs an optical element providing a wide range of viewing angles. It may be troublesome as element of this kind requires characterization in the full range of angles that is not filled with sharp reference image.

[ 0038 ] In a certain embodiment, the angle range problem is solved by performing recording with the optical device according to the invention around which a set of printed reference charts is spaced or around which the reference chart is moved so as to obtain the reference chart image for the full range of observation angles of the optical device.

[ 0039 ] In an alternative embodiment of the method according to the invention, the optical device according to the invention is placed on a rotary table and rotated keeping the constant distance between the center of the chart and the plane of the matrix 1.

[ 0040 ] In a further alternative embodiment of the method according to the invention, the optical device according to the invention is placed inside a solid of revolution, on the surface of which the calibration pattern is present.

[ 0041 ] In an acquisition step, the distortion of the adjustable distortion element 2a, 2b is adjusted to a predetermined adjustment state providing the desired density distribution of the image points. Subsequently, reading from the matrix area determined by the calibration process is performed for this adjustment state. The obtained deformed image is subjected to the image points position correction according to the image points position correction function determined in the calibration step. To do this, interpolation is used, the interpolation being applied during mosaic removing and indicating the positions of the pixels that are used for the target image. Polynomial and/or trigonometric and/or other equations used in the deformation correction can be approximated linearly so that the full pixel position is obtained as a result of calculations. Consequently, the pixel position error can be fractional. Additionally, in order to accelerate the calculations linear interpolation for a given calculation block is used. The resulting image is stored in the memory and displayed to the user.

[ 0042 ] In some specific applications, it is advisable to store the image in the memory and to display the image with specific distortions to the user. To achieve this effect, it is enough to modify the correction function resulting from the calibration.

Claims

1. A method of recording an image with a matrix (1) of photosensitive elements, in front of which an optical element is arranged to direct the radiation incident from a wide range of angles to the matrix (1) of photosensitive elements, the method comprising a step of recording points of the deformed image by reading them from the matrix (1) of photosensitive elements, the method being characterized in that

the optical element is an adjustable distortion element (2a, 2b) , the image recording step is preceded by a calibration step, in which for at least a number of distortion states of the adjustable distortion element (2a, 2b) an image of an object having predefined shape is recorded, and based on the recorded image, for each distortion state of the adjustable distortion element (2a, 2b) a specific area of the matrix (1) on which the useful image is formed is determined,

for each distortion state of the adjustable distortion element (2a, 2b) and the corresponding specific area an image points position correction function is determined, and in an acquisition step

distortion of the adjustable distortion element (2a, 2b) is adjusted,

reading only from the determined specific area of the matrix is performed, and thus obtained deformed image is subjected to the image points position correction according to the image points position correction function and next the image is stored in the memory (205) .

2. The method according to claim 1, characterized in that in the acquisition step distortion of the adjustable distortion element (2a, 2b) is adjusted also beyond the states for which the calibration was performed and values of the image points position correction function are determined based on at least one closest state .

3. An optical device for image recording provided with an optical element adapted to direct the radiation incident from a wide range of angles to a matrix (1) of photosensitive elements and a reading block (202) being in communication with the matrix and also with a control block (200) and a memory (205), characterized in that the optical element is an adjustable distortion element (2a, 2b) and in that

the device is provided with a correction block (206) being in communication with the control block (200) and the memory (205), whereas the matrix (1) comprises at least 25 000 000 photosensitive elements .

4. The device according to claim 3 characterized in that the adjustable distortion element is a mirror (2a) having a reflecting surface with adjustable shape (4a, 4b) .

5. The device according to claim 4 characterized in that the surface with adjustable shape (4a, 4b) is mechanically coupled with at least one piezoelectric element.

6. The device according to claim 4 characterized in that the surface with adjustable shape (4a, 4b) is mechanically coupled with a group of electromechanical transducers.

7. The device according to claim 4 characterized in that the surface with adjustable shape (4a, 4b) is a ferromagnetic surface and remains in a magnetic field generated by at least one coil.

8. The device according to claim 2 characterized in that the adjustable distortion optical element is an adaptive lens.

9. The device according to claim 8 characterized in that the adaptive lens is a fisheye lens with a servomechanism arrangement moving the lens in a plane substantially parallel to the matrix (1) .

10. The device according to claim 8 characterized in that The adaptive lens is a lens with variable shape (2b) .