WO2013087450A2

WO2013087450A2 - Estimation of global vertical shift for stereoscopic images

Info

Publication number: WO2013087450A2
Application number: PCT/EP2012/074319
Authority: WO
Inventors: Dennis Harres; Yoshihiro Murakami; Tohru Kurata
Original assignee: Sony Corporation; Sony Deutschland Gmbh
Priority date: 2011-12-13
Filing date: 2012-12-04
Publication date: 2013-06-20
Also published as: WO2013087450A3

Abstract

The present invention relates to a method to estimate global shift between a first image and a second image, preferably a left stereoscopic image and a right stereoscopic image, comprising providing the first image sub-sampled by a factor of N providing the second image sub-sampled by the factor of N estimating a shift between the first and second sub-sampled images by image registration of both sub-sampled images, and up-scaling said shift by the factor of N as an estimate of global shift.

Description

Estimation of Global Vertical Shift for Stereoscopic Images

FIELD OF INVENTION

[0001] The present invention relates to a method to estimate global shift between a first image and a second image, preferably a left stereoscopic image and a right stereoscopic image, preferably of 3D motion pictures. The invention also relates to an image shift estimation device, a multi stage image shift estimation system, a stereoscopic camera system, a computer program and a computer readable non-transitory medium. BACKGROUND OF THE INVENTION

[0002] The comfort of stereoscopic viewing highly depends on the proper configuration of the stereoscopic images, particularly of 3D motion pictures. Most severe physiological issues with stereoscopy origin in the vertical misalignment of the left and right eye content, for which an unusual ocular muscular activity is required to compensate for that problem optically. This causes high tension and strain followed by such symptoms as headache.

[0003] To avoid the described viewing discomfort much effort has been undertaken recently to assume correct alignment of the final stereoscopic content before being displayed to the viewer. In general, two common ways exist to solve the problem of stereoscopic alignment and misalignment, respectively.

[0004] First, a mechanical or optical calibration of the acquisition equipment enables correct stereoscopic reproduction which can be sometimes assisted by electronic tools and/or processing. The goal of this approach is mainly to preserve the quality of digitized content to the highest possible extent and avoid any information loss in the image border areas. This approach is successful on statically parameterized stereoscopic configurations. The main drawback is the necessity for manual setup and validation time of the proper stereoscopic alignment. To some degree of success this approach can be extended to dynamically changing parameters such as focal lens, focus distance, interaxial distance or convergence. In the latter case the quality of the components plays a big role in achieving good calibration results. Traditionally this type of preparation and associated costs origin in the professional sector of video and film industry and is unacceptable for the consumer market.

[0005] During stereoscopic acquisition process, however, proper mechanical and/or optical alignment of the equipment cannot be guaranteed, deviates due to manufacturing tolerances or i.e. is not achievable by design/costs constraints. Thus, an electronic adjustment or correction of the image signal must solve some of the stereoscopic issues after the scene has been captured by the image sensor of a camera, e.g. a 3D camera. The common practice is to apply a post-processing step to already recorded material at a later time frame. This tendency to offline post-processing solutions is based on the fact that the problem of stereoscopic correspondence is not trivial by nature. A complex multiple degree of freedom calibration of captured images is a high computational task. Real-time capable solutions do not yet exist.

[0006] An additional constraint arises that the algorithm's performance must not worsen the initial stereo alignment. The robustness of the computation algorithm or its reliability becomes a part of the solution. In a post-processing environment, user supervision is a commodity thus the user becomes part of the reliability process which cannot be applied in a real-time in-camera processing.

[0007] In a recent study it has been shown that the vertical misalignment is the issue with the biggest impact on stereoscopic viewing comfort. Thus, a proper solution for this specific issue is most desirable.

BRIEF SUMMARY OF INVENTION

[0008] It is an object of the present invention to provide a method allowing to estimate a global shift of two images which overcomes the constraints of prior art solutions and particularly allows a realization of real-time processing.

[0009] It is a further object of the present invention to provide an image shift estimation device, a multi stage image shift estimation system, a stereoscopic camera system comprising such an image shift estimation device as well as a corresponding computer program for implementing the afore-mentioned method. [0010] According to an aspect of the present invention there is provided a method to estimate global shift between a first image and a second image, preferably a left stereoscopic image and a right stereoscopic image of a 3D motion picture, comprising providing the first image sub-sampled by a factor of N providing the second image sub-sampled by the factor of N estimating a shift between the first and second sub-sampled images by image registration of both sub-sampled images, and upscaling said shift by the factor of N as an estimate of the global shift.

[0011] Such an inventive method is robust and computationally inexpensive. This method could be for example the basis to estimate the vertical alignment for a stereoscopic image pair. It could also be used in the detection of most dominant horizontal disparity for a proper convergence setting or for a global motion estimation in a codec environment or alike.

[0012] The inventive method is computationally inexpensive due to the use of sub-sampled images. It is applicable in a real-time environment in both hardware (for example ASIC) and software on DSPs, embedded systems or other processors based on a straightforward implementation of the algorithm. The invented method is further robust against local image dissimilarities and able to detect unreliable input in an unsupervised manner.

[0013] According to a further aspect of the present invention there is provided an image shift estimation device comprising an image registration unit provided with a first image sub-sampled by a factor of N and a second image sub-sampled by the factor of N, said image registration unit being adapted to estimate a global sub-sample shift between both images as a best match estimation; and an offset refinement unit receiving the global sub-sample shift estimation and adapted to upscale the sub-sampled shift by the factor of N and to refine the up-scaled shift by a refinement value, preferably between -N/2 and N/2.

[0014] According to a further aspect of the present invention there is provided a multi stage image shift estimation system comprising at least two image shift estimation devices as mentioned before and an image region separation unit adapted to provide at least one predetermined region of an image, wherein each image shift estimation device is provided with a different region of the image.

[0015] According to a further aspect of the present invention there is provided a stereoscopic camera system comprising an image shift estimation device as mentioned above or a multi stage image shift estimation system as mentioned above for vertical misalignment correction between left and right images, preferably in real time.

[0016] According to still further aspects a computer program comprising program code means for causing a processor to perform the steps of said inventive method when said computer program is carried out on a processor, as well as a computer readable non-transitory medium having instruction stored thereon which when carried out on a processor cause the processor to perform the steps of the inventive method are provided.

[0017] Preferred embodiments of the invention are defined in the dependent claims. It shall be understood that the claimed device, the claimed system, the claimed computer program and the claimed computer readable medium have similar and/or identical preferred embodiments as the claimed method and as defined in the dependent claims. [0018] It is to be understood that the features mentioned in the dependent claims and those yet to be explained below can be used not only in the respective combinations indicated, but also in other combinations or in isolation, without leaving the scope of the present invention.

[0019] The present invention is based on the idea to use "pixel-reduced" images for the image registration process necessary for estimating a shift between both pixel- reduced images. The estimated shift is then in a further step upscaled to gain the global shift. Due to the pixel reduction, the registration process is much faster and allows realtime processing. Preferably, the estimated shift is refined in a further process step according to a predetermined rule and thereby considering registration results in a neighborhood of the estimated shift value.

[0020] BRIEF DESCRIPTION OF DRAWINGS

[0021] These and other aspects of the present invention will be apparent from and explained in more detail below with reference to the embodiments described hereinafter. In the following drawings

Fig. 1 shows a schematic block diagram of an image shift estimation device;

Fig. 2 shows a schematic block diagram of the image shift estimation device in more detail;

Fig. 3 shows a schematic diagram of a correlation surface;

Fig. 4 shows a diagram to estimate a sub-pixel shift;

Fig. 5 shows diagrams used to explain the refinement step; shows a diagram used for reliability detection;

Fig. 7 shows a correlation surface to further explain the reliability detection;

Fig. 8 shows a diagram of a correlation surface used for disparity compensation;

Fig. 9 shows a diagram of a correlation surface used for deriving reliability from disparity;

Fig. 10 shows a schematic block diagram of a multi stage shift estimation system; and

Fig. 11 shows a flow diagram representing the inventive method.

DETAILED DESCRIPTION OF THE INVENTION

[0022] Fig. 1 shows a schematic block diagram of an image shift estimation device 10. This device 10 comprises a first sub-sample element 12 and a second sub-sample element 14. The first sub-sample element 12 receives a first input image A and the second sub-sample element 14 a second input image B. In the present embodiment, input image A is a right image and input image B is a left image of a stereoscopic video/motion picture. Left and right stereoscopic images may be supplied by a 3D camera system, as to mention one example.

[0023] Each sub-sample element 12, 14 is coupled with an image registration unit 16. The image registration unit 16 in turn is coupled with an offset refinement unit 18 and optionally with a reliability unit 20. [0024] The offset refinement unit 18 provides an output signal S which presents a global shift S between both input images A and B. In the present embodiment, the global shift S is just the shift in a vertical direction between both images A, B. This vertical shift indicates a vertical misalignment between both images A, B which may be caused for example by misalignment of both cameras of the 3D camera system.

[0025] The global shift in general, and the vertical shift in particular allow to correct the alignment between both images A, B in a following image correction process, which, however, is not part of the present invention and will not be described here.

[0026] Fig. 1 further shows that both sub-sample elements 12, 14 as well as the offset refinement unit 18 receive a value N as further input. This value (factor) indicates the degree of sub-sampling the input images A, B. For example, if the value N is 4 and the input images A, B have 1920 x 1080 pixels, the output sub-sampled image of both sub- sample elements 12, 14 is a 480 x 270 pixel image.

[0027] It is to be noted here that the sub-sample elements 12, 14 do not separate an image region from the input image but reduce the resolution of the input image by the factor of N so that the output sub-sampled image contains information from the entire input image.

[0028] A preferred value N is for example 16 for a 1920 x 1080 pixel image, which gives a good trade-of between speed and accuracy. However, in a particular embodiment, the value N could be 1.

[0029] In Fig. 2, the image registration unit 16 and the offset refinement unit 18 are shown in more detail. [0030] The image registration unit 16 comprises a first matching element 22 and a second matching element 24 and a correlation surface fusion element 26 receiving output signals from the matching elements 22 and 24.

[0031] The correlation surface fusion element 27 is coupled with a peak search element 28 and the reliability unit 20.

[0032] The offset refinement unit 18 comprises a sub-pixel shift element 30 coupled with the peak search element 28 and a periodic form compensation element 32. This compensation element in turn is coupled with a shift upscale element 34 receiving a position value (x, y) of the peak search element 28, a reliability value from the reliability unit 20 and the value or factor N. The output of the shift upscale element 34 is then the shift S.

[0033] It is without saying that the afore-mentioned functional components i.e. elements and units, may be provided in hardware or software or a combination thereof. The components may for example be realized in a micro processor or as a controller circuit. Further, the described components may be combined or integrated into a single multi functional component, for example a controller circuit.

[0034] As already mentioned before, the input images A, B may be supplied by a camera system. Hence, the image shift estimation device 10 may be integrated into such a 3D camera system. It would enable the realization of a real-time left/right image pair misalignment correction, particularly a misalignment correction in a vertical direction.

[0035] The functionality of this image shift estimation device 10 will now be described in detail with reference to Figs. 3 to 11.

[0036] In a very general consideration, the method (shown as flow diagram in Fig. 11) carried out by the image shift estimation device comprises three major blocks, namely an image registration 100, a global, preferably a vertical shift computation 102 and a reliability detection 104.

[0037] Image registration comprises matching 110, 112 of input images A and B, i.e. sub-sampled images A, B supplied to the image registration unit 16. The matching process can be achieved by a correlation function in a preferred embodiment or any other similarity measure. The preferred embodiment makes use of a normalized cross-correlation (NCC) function. The cross-correlation function is generally known and will therefore not be described in detail here. In general, the normalized cross-correlation is a measure of how well two images match each other. During correlation using NCC, a first image as the target is moved over a second image as a reference. The result of a cross-correlation is a triplet of a correlation value or score and an x, y position. An exemplary result of a cross- correlation is for example shown in Fig. 3, wherein the x-axis represents the horizontal offset from center in pixels, the y-axis the vertical offset from center in pixels and the correlation values are indicated by different colors within the xy-plane. In the following this diagram is called "correlation surface".

[0038] In the present embodiment which describes a stereoscopic case, the matching accuracy and precision could be improved by applying the similarity measure, here the cross-correlation function, in both directions due to present disparity and occlusions in both left and right views of the scene. That is the image registration comprises in a first path 110 the search of image A inside B and in a second parallel path 112 the search of image B in image A.

[0039] As shown in Fig. 2, the first matching element receives sub-sampled image B as reference and sub-sampled image A as target. The second matching element 24 receives the second sub-sampled image B as target and the sub-sampled first image A as reference. Hence, matching element 22 searches sub-sampled image A inside sub-sampled image B and second matching element 24 searches sub-sampled image B in sub-sampled image A. [0040] Both correlation results can be easily combined 114 by the correlation surface fusion element 26 using the symmetry property into one final correlation surface. The advantage of this combination is that it removes noise in the result and the symmetry effect of bidirectional matching improves the error curve of the sub-pixel shift estimation, as described below, to become symmetric, too.

[0041] An example of the result of the correlation surface fusion is shown in

Fig. 3.

[0042] In the next step 102, the global shift, in the present embodiment the vertical shift, is computed. The basis for this computation is the correlation surface provided by the correlation surface fusion element 26. In this surface, the peak search element 28 searches the global maximum which indicates the best match between sub-sampled images A, B (step 116). In the correlation surface shown in Fig. 3, this global maximum, in the following also called peak, is at an exposition x = 1 and a y position y = 0. In Fig. 3, the global maximum or peak is indicated.

[0043] The peak search element 28 supplies the position (x, y) of the peak to the shift upscale element 34. Further, the peak search element 28 supplies the peak value, that is the correlation value at the peak position (x=l, y=0) to the sub-pixel shift element 30 together with the correlation values of the direct neighbors (in a straight line, that is the top and bottom neighbors or the left and right neighbors dependent on whether y and/or x shift is calculated). In the present embodiment and hence for computing the shift in the vertical (y) direction, the correlation values of the top (x=l, y=-l) and bottom (x=l, y=l) neighbor of the peak is submitted to the sub-pixel shift element 30. Both correlation values are also indicated by rectangles in Fig. 3. As a result, the sub-pixel shift element 30 receives three correlation values which are then used for a sub-pixel shift estimation (step 118). In the present embodiment, this estimation may be carried out by an interpolation of a suitable approximation of the corresponding peak and neighboring values. The present preferred embodiment solves this problem with a parabola fitting through three or more points for a one-dimensional shift (x or y shift) or five or more points for a two-dimensional shift estimation (both directions y and x). The position of the parabola extreme is the desired sub-pixel shift. Fig. 4 exemplifies this interpolation. In the shown diagram, the correlation values of the peak and both neighbors are indicated by circles. Then a parabola is fitted through these three points and then the sub-pixel value of the maximum is determined and used for further processing, particularly for refining the shift value.

[0044] This refinement step benefits from the property of the correlation surface that the closer the registration of the both images is to the position of the perfect match, the higher is the correlation value. Especially in the direct neighborhood of the correlation peak, these correlation values will contain the proximity of the correlation to the best match. If both neighbors are equally far from the peak then the best match is exactly at the correlation peak and the sub-pixel shift is zero. If one of the neighbor values is closer to the peak then the optimal solution is between these neighbors and the peak.

[0045] The result of the sub-pixel shift estimation done by the sub-pixel shift element 30 needs not necessarily provide a linear output for the sub-pixel shift as the values of the correlation surface may change in a non-linear fashion and create a periodic bias between each 0 sub-pixel positions. Further, the choice of the above fitting function may cause a similar effect, too.

[0046] This periodic bias or error form can be studied from a designated set of corresponding images. The compensation for the periodic bias can be achieved by a conversion from values of original representation to the values of desired linear shape. In the preferred embodiment, a cubic function is estimated which describes the response curve of the sub-pixel estimation. The correction is then applied by means of a derived look-up table to remove this periodic bias. Fig. 5 shows the respective diagrams indicating this periodic bias effect. [0047] In other words, the periodic form compensation element 32 comprises a lookup table containing predefined values, namely compensated sub-pixel shift values. The periodic form compensation element 32 outputs the compensated sub-pixel value assigned to the sub-pixel value received as input (step 120).

[0048] In the next step 122, the shift upscale element 34 receives the compensated sub-pixel value and the xy value of the peak. In the present embodiment as already mentioned before, the vertical shift is calculated. Therefore, the supplied y value of the peak position is corrected by the compensated sub-pixel value and then the result is upscaled by the factor N. In other words, the refined y value representing the shift of the sub-sampled images is multiplied by the factor of N. This result is then output as the desired estimated global shift, here in the present embodiment the vertical shift between both input images A, B.

[0049] The final performance of the afore-mentioned method for estimating the global shift S and its reliability depends on the quality of the correlation surface. To avoid unreliable sub-pixel estimation, the image shift estimation device comprises the reliability element 20 providing functionality for correlation surface assessment. This function further enables options like to issue a warning of malfunction (i.e. in a camera) if the unreliable state persists;

to validate the calibration process;

to assess the stereoscopic quality of current scene (such as a user interface icon if critical);

to temporarily track the changes and derive consistency conclusions.

[0050] The reliability element evaluates the shape of the correlation surface to predict unreliable results (step 104). As soon as some of the expected criteria about the correlation shape or form are violated the reliability block prevents usage of unreliable sub-pixel results and potential introduction of more severe errors.

[0051] For this purpose the reliability element 20 carries out a dimensionality reduction for the problem of two-dimensional surface evaluation. For the dominant direction (i.e. horizontal or vertical; in the present embodiment the vertical direction), the correlation surface is reduced to one-dimensional curves based on a) maximum projection along selected dimension; and b) a cut at the position of the global peak along selected dimension. Additionally, the positions of the maxima from a) are collected to capture the two-dimensional essence of the original representation.

[0052] The diagrams shown in Figs. 6 and 7 illustrate this reliability detection.

[0053] If the assessment or evaluation of the correlation surface yields the result that the estimated shift is unreliable, the shift upscale element 34 receives a respective signal and in response thereto prevents/rejects the output of the shift value S (step 108). In the reliable case, the shift upscale element 34 outputs the estimated shift value S (step 106).

[0054] In a special case of stereoscopic input, the correlation surface reflects the disparity present in the stereoscopic image pair, that is the input images A, B. In a broader sense, the correlation surface can be regarded as a mix of correlation results of objects at each disparity level. Due to the horizontal orientation of the stereoscopic disparity, the peak of the correlation surface is prolonged in horizontal direction. Depending on contents of the scene and the object distribution in the depth, multiple peaks with very similar values may exist in the horizontal extension. Their corresponding neighbour stability depends on the quality for the particular disparity layer and may be not an optimal solution for the whole disparity range. The method described above improves the disparity-related instability by replacing the immediate neighbours of the peak with the local maxima of the neighbouring rows. This is illustrated in Fig. 9. [0055] Based on similar principles of disparity influence the reliability of the correlation surface may also be assessed by comparing the distance between a) the global shift S 1 from the global peak; b) the global shift from the second highest peak S2 which is not in the same column of the surface as the global one.

[0056] In a stable matching environment both the "first" and "second" peak should produce similar results SI ~ S2 as the most dominant disparity should exhibit similar global motion in vertical direction same as the less dominant disparity layer. High deviation between those values, i.e. |S1 - S2| > p means an unreliable matching result in the correlation surface or a highly distorted input. A respective illustration of this principle is shown in Fig. 9.

[0057] Of course as already mentioned above, if this evaluation step results in an unreliable matching result, a respective signal is supplied by the reliability element 20 to the shift upscale element 34.

[0058] The method described above and the respective device 10 for carrying out this method can be further integrated into a multi stage system 40, as shown in Fig. 10. This multi stage system 40 consists of multiple instances of the global shift estimation method described above. Each of the instances runs a region-of-interest of the input image. Such regions-of-interest (ROI) can be defined and are not limited to: full image (full), left top (LT), right top (RT), center, left bottom (LB) and right bottom (RB) image regions. The result of each instance is then combined into a final decision.

[0059] In Fig. 10, this multi stage system 40 comprises multiple image shift estimation devices 10 each working on a particular region-of-interest of the input image. These regions-of-interest are provided by a region-of interest unit 42 which receives as input images A, B. A region-of-interest is selected from the sub-sampled original image and/or a region-of-interest is first selected from the original image and then sub-sampled. [0060] Each instance 44 carries out a similarity function, here a normalized cross-correlation (NCC) for computing a correlation surface. The correlation surface is then searched for the global maximum value (peak) which is refined by a sub-pixel value before supplying it to a fusion unit 46. In the event that the reliability check results in an unreliable correlation surface, the shift value of the region-of-interest is not supplied to the fusion unit 46.

[0061] This multi stage embodiment has the following advantages: improvement of the overall accuracy;

ability to cope with rotation and zoom;

ability to detect rotation and zoom and thus extend the degree of freedom of the global motion/shift;

derive useful information from detected rotation and zoom to assess stereoscopic quality.

[0062] To sum up, the described shift estimation method has the following advantages: computationally inexpensive due to input sub-sampling, applicable in a real-time environment, in both hardware (i.e. ASIC) and software on DSPs (digital signal processors), embedded systems or other processors based on a straightforward implementation of the algorithm,

robust against local image dissimilarities, and

able to detect unreliable input in an unsupervised manner.

[0063] The described method and device can be employed in the field of global motion estimation, can be part of a bigger system to perform an initial estimation of the global motion, thus reducing the search space of the main system, or can be a global motion estimator inside a codex system. [0064] The invention has been illustrated and described in detail in the drawings and foregoing description, but such illustration and description are to be considered illustrative or exemplary and not restrictive. The invention is not limited to the disclosed embodiments. Other variations to the disclosed embodiments can be understood and effected by those skilled in the art in practicing the claimed invention, from a study of the drawings, the disclosure, and the appended claims.

[0065] In the claims, the word "comprising" does not exclude other elements or steps, and the indefinite article "a" or "an" does not exclude a plurality. A single element or other unit may fulfill the functions of several items recited in the claims. The mere fact that certain measures are recited in mutually different dependent claims does not indicate that a combination of these measures cannot be used to advantage.

[0066] A computer program may be stored / distributed on a suitable non- transitory medium, such as an optical storage medium or a solid-state medium supplied together with or as part of other hardware, but may also be distributed in other forms, such as via the Internet or other wired or wireless telecommunication systems.

[0067] Any reference signs in the claims should not be construed as limiting the scope.

Claims

1. Method to estimate global shift between a first image and a second image, preferably a left stereoscopic image and a right stereoscopic image, comprising

providing the first image sub-sampled by a factor of N

providing the second image sub-sampled by the factor of N

estimating a shift between the first and second sub-sampled images by image registration of both sub-sampled images, and

up-scaling said shift by the factor of N as an estimate of global shift.

2. Method of claim 1, comprising refining said estimated shift before up-scaling.

3. Method of claim 1, comprising

detecting the reliability of said estimated shift.

4. Method of claim 1, 2 or 3, wherein said image registration comprises

matching first and second sub-sampled images by a similarity function, preferably a correlation function.

5. Method of claim 4, wherein said correlation function is a normalized cross- correlation (NCC) function.

6. Method of claim 4 or 5, wherein said correlation function is applied in both directions, so that the first sub-sampled image is searched in the left sub-sampled image and the second sub-sampled image is searched in the first sub-sampled image.

7. Method of claim 6, wherein both correlation results are combined into one final correlation surface.

8. Method of claim 7, comprising

determining the peak of the correlation surface corresponding to the best match between said first and second sub-sampled images.

9. Method of claim 8, comprising

refining the position of the peak by considering the correlation values of at least two neighbors of the peak.

10. Method of claim 9, comprising determining said refined position of the peak by interpolating an extreme between said peak and both neighbors, wherein said extreme is the sub-sample shift.

11. Method of claim 10, wherein said interpolating comprises fitting a parabola through said peak and said both neighbors and determining the extreme of the parabola.

12. Method of claim 11, comprising compensating a periodic bias to linearize the sub- sample shift.

13. Method of claim 12, wherein a look-up table with compensation values is provided, and a respective compensation value from the look-up table is applied to the sub-sample shift.

14. Method of claims 3 and 6 and any other of the preceding claims, wherein detecting said reliability comprises evaluating the shape of said correlation surface and comparing it with at least one predefined criterion to predict unreliable shift results.

15. Method of claim 14, wherein unreliable shift results are blocked from usage.

16. Method of claim 14 or 15, wherein evaluating said shape of said correlation surface comprises evaluating the surface in a dominant direction only to reduce the dimensionality of the two dimensional surface to a one dimensional curve.

17. Method of any of the preceding claims, wherein said first and second images are left and right stereoscopic images.

18. Method of claim 18, wherein said shift is a vertical shift defining a vertical alignment between the left stereoscopic image and the right stereoscopic image.

19. Method of any of claims 1 to 16, wherein said shift is used for a motion estimation.

20. Method of any of claims 1 to 16, wherein said shift is used to detect the most dominant horizontal disparity for convergence setting.

21. Method of any of the preceding claims, wherein said factor N equals 1.

22. Method of any of the preceding claims, wherein said factor of N is greater than 1, preferably at least 4.

23. Image shift estimation device comprising

an image registration unit provided with a first image sub-sampled by a factor of N and a second image sub-sampled by the factor of N, said image registration unit being adapted to estimate a global sub-sample shift between both images as a best match estimation;

an offset refinement unit receiving the global sub-sample shift estimation and adapted to up-scale the sub-sample shift by the factor of N and to refine the up-scaled shift by a refinement value, preferably between -N/2 and N/2.

24. Device of claim 23, comprising

a sub-sampling unit adapted to sub-sample the first and second images by said factor of N and to supply said sub-sampled images to the image registration unit.

25. Device of claim 23 or 24, comprising

a reliability evaluation unit adapted to receive the global sub-sample shift of the image registration unit, to evaluate the reliability of the sub-sample shift and to provide said offset refinement unit with a respective reliability signal.

26. Device of any of claims 23 to 25, wherein said image registration unit comprises a first image matching unit adapted to find similarities from the first image in the second image and a second image matching unit adapted to find similarities from the second image in the first image.

27. Device of claim 26, wherein said matching units are adapted to carry out a correlation function, preferably a cross-correlation function to find similarities.

28. Device of claim 27, comprising a correlation fusion unit adapted to receive the correlation results of the matching units and to combine both results into a single correlation surface.

29. Device of claim 28, comprising a peak search unit adapted to carry out a peak search in the correlation surface and to provide a position value of the peak.

30. Device of claim 29, wherein said offset refinement unit comprises

a sub-pixel shift estimation unit adapted to estimate a sub-pixel shift on the basis of the peak value and the correlation values of two neighboring positions of the peak and a periodic form compensation unit adapted to estimate the refinement value on the basis of said sub-pixel shift.

31. Multi stage image shift estimation system comprising

at least two image shift estimation devices according to any of claims 23 to 30, and an image region separation unit adapted to provide at least one predetermined region of an image, wherein each image shift estimation device is provided with a different region of the image.

32. System of claim 31 , wherein a region of an image is square region covering at least on of center area, left top area, right top area, left bottom area and right bottom area of the image.

33. Stereoscopic camera system comprising an image shift estimation device according to any of claims 23 to 30 or a multi stage image shift estimation system according to claim 31 or 32 for vertical misalignment correction between left and right images, preferably in real-time.

34. Computer program comprising program code means for causing a processor to perform the steps of said method as claimed in any of claims 1 to 22, when said computer program is carried out on a processor.

35. Computer readable non-transitory medium having instructions stored thereon which, when carried out on a processor, cause the processor to perform the steps of the method as claimed in any of claims 1 to 23.