WO2012080961A1

WO2012080961A1 - Improved stereoscopic shooting apparatus, and method thereof

Info

Publication number: WO2012080961A1
Application number: PCT/IB2011/055663
Authority: WO
Inventors: Mario Muratori
Original assignee: Rai Radiotelevisione Italiana S.P.A.
Priority date: 2010-12-14
Filing date: 2011-12-14
Publication date: 2012-06-21
Also published as: ITTO20100996A1; EP2652955A1; IT1403163B1

Abstract

A stereoscopic shooting apparatus is described which comprises shooting means (10a, 10b) for taking a stereoscopic shot, which means are adapted to provide a pair of stereoscopic images comprising a right image and a left image, wherein said shooting apparatus comprises: a computation block (12) receiving at its input said pair of stereoscopic images from said shooting means (10a, 10b); a normalization and quantization block (14) subjecting the values of said parameter computed in said computation block (12) to normalization, and preferably also to quantization; association means (16) adapted to associate graphic information with said parameter; an overlay block (18) overlaying said graphic information on said pair of stereoscopic images and outputting said overlaid images to visualization means (22); visualization means (22) for displaying said graphic information through an image overlaid on at least one image of said pair of stereoscopic images, thus forming a composite image.

Description

IMPROVED STEREOSCOPIC SHOOTING APPARATUS, AND METHOD THEREOF

DESCRIPTION

The present invention relates to the field of television or cinematographic stereoscopic shooting methods.

More in particular, the present invention relates to an improved stereoscopic shooting apparatus, and to the method thereof, adapted to provide real-time information to an operator about the values of parameters of interest concerning a stereoscopic television or cinematographic shot.

As known, traditional stereoscopic shooting, which is based on a pair of channels, i.e. one associated with the right eye and one associated with the left eye, can be performed by adopting two distinct geometric configurations, i.e. with "converging axes" or with "parallel axes", respectively.

In principle, both of these configurations use two sensors and two corresponding optics, but differ from each other as to the relative arrangement of these two fundamental elements.

In the configuration with "parallel axes", shown in Fig. 1, the sensors 1,1 ' are coplanar and their surface lies on a plane perpendicular to the respective optical axes 3,3', which are parallel to each other, hence the configuration name.

In the configuration with "converging axes", shown in Fig. 2, the optical axes 3,3' are rotated so as to converge towards a convergence point 5, located at the convergence distance from the sensors 1,1 '.

The characteristics of said two configurations must be analyzed by taking also into account the perceptive effect obtained when displaying a stereoscopic shot.

It is appropriate to remind some fundamental concepts:

- in all visualization techniques (with the exception of the HMD (Head Mount Display) technique), the two images of the left and right channels are overlaid;

- the image dots shown as coinciding are perceived as "on the screen";

- those connected dot pairs of the images which are presented with negative disparity, i.e. when the dot for the right eye is presented on the left of the one for the left eye, are perceived as being "ahead" of the screen;

- those connected dot pairs of the images which are presented with positive disparity, i.e. when the dot for the left eye is presented on the left of the one for the right eye, are perceived as being "behind" the screen;

The best three-dimensional perception is obtained when the displayed object is perceived more or less at the screen distance.

As a practical rule, the objects perceived as being "ahead" of the screen should not be seen at a distance from the screen exceeding one third of the screen-to-audience distance. This implies that, in the case wherein the distance between the screen and the audience is 3H (i.e. three times the screen height, as recommended for HDTV), the negative disparity absolute value must not exceed half the spectator's interocular distance. This measurement will be referred to below as "one-third rule".

In order to avoid any divergence of the ocular optical axes, the positive disparity value must not exceed the interocular distance value. This error will be referred to below as "ocular divergence error".

The two above-described conditions are represented by the formula -e/2 < δ < e, wherein "e" is the interocular distance and "δ" is the disparity measured on the visualization screen. The configuration with "parallel axes" is free from trapezoidal distortion and can be implemented more easily. In the visualization stage, however, the whole shot environment is perceived as being "ahead" of the screen, i.e. in a space portion which appears too artificially limited. Moreover, the "one-third rule" is easily infringed.

On the other hand, if horizontally-pivoted optics or similar mechanisms are not used, the configuration with "converging axes" is characterized by trapezoidal distortion due to sensor rotation, which implies the existence of spurious vertical disparities, and is more difficult to implement. In the visualization stage, however, the shot environment is perceived as being both "ahead" and "behind" the screen, and therefore appears to be more natural. Actually, this configuration imitates the human vision geometry. Nevertheless, when using this type of configuration it is also easy to infringe the "one-third rule" and to incur in the ocular divergence error.

Stereoscopic shooting is done by using apparatuses based on different configurations. One of such configurations, which is very widespread, is based on the use of two distinct video cameras mounted on a suitable mechanical support. In general, this allows adjusting the "stereoscopic base" B, i.e. the distance between the optics' optical centres X_s,Xd, and the "convergence angle", i.e. the angle of rotation of the optics' optical axes, as shown in Fig. 3. One variant of this configuration employs specific apparatuses consisting of two fully separate video cameras, i.e. each fitted with its own sensor and lens, arranged within a single casing. One example of this system is the Panasonic "Twin Lens Full HD 3D camcorder" system (http://www.dv.com/article/90970). This system allows adjusting the "convergence angle", but the "stereoscopic base" remains fixed by mechanical construction.

In order to avoid any problems related to controlling two separate optics, specific apparatuses have been developed which employ two sensors and a single lens. One example of such a system is the "Sony HFR Comfort 3D" video camera (http://www.sonv.net/SonvInfo/News/Press/200910/09-117E/). This system adopts the configuration with parallel optical axes, but allows no "stereoscopic base" adjustment. In the techniques using the configuration with converging axes, in addition to making the usual colorimetric and optic calibrations, one must also adjust those parameters that characterize the shooting geometry, hereafter referred to as "stereoscopic parameters", which include the convergence angle and the stereoscopic base.

It must be pointed out that the combination^" of stereoscopic base and convergence angle determines the "convergence distance", i.e. the distance between the point of convergence of the optical axes of the video cameras' optics and the shooting system itself. These values are indicated in Fig. 3.

During a stereoscopic shooting, the operator must therefore pay much attention to some stereoscopic parameters that might turn out to be critical, since they are potential sources of perception errors in the visualization stage.

It is therefore one object of the present invention to provide an improved stereoscopic shooting apparatus, and a method thereof, adapted to provide real-time information which can be used by a stereoscopic camera operator to eliminate or correct potential sources of perception errors in the visualization stage.

It is another object of the present invention to provide an improved stereoscopic shooting apparatus, and a method thereof, adapted to provide information that can be easily interpreted by a stereoscopic camera operator.

It is a further object of the present invention to provide an improved stereoscopic shooting apparatus, and a method thereof, suitable for use with any type of stereoscopic shooting configuration.

These and other objects of the invention are achieved through the improved stereoscopic shooting apparatus, and the method thereof, as claimed in the appended claims, which are intended as an integral part of the present description.

In short, the improved stereoscopic shooting apparatus, and the method thereof, according to the invention aim at providing the value of a monitored stereoscopic parameter value to a stereoscopic camera operator, in an ergonomical and immediately comprehensible manner.

The method according to the invention comprises the following steps:

- measuring a stereoscopic parameter of interest to be monitored;

- normalizing and possibly quantizing the measured value;

- associating the stereoscopic parameter of interest with graphic information associated with the normalized and quantized value;

- displaying the graphic information by overlaying it on at least one shot image, possibly delayed in order to compensate for processing time.

In this way, the stereoscopic camera operator is given information useful to eliminate or correct any potential sources of perception errors.

The apparatus according to the invention is particularly interesting when shooting very dynamic scenes, in which quick situation changes require rapid adjustments of the shooting apparatus to keep the monitored stereoscopic parameter within the desired range.

The apparatus according to the invention is also useful for television or cinematographic stereoscopic shooting systems wherein the horizontal disparity value cannot exceed the limits imposed by the necessity of avoiding perception errors in the visualization stage, but wherein there is no ergonomical and immediately comprehensible manner of displaying said value or for displaying the values of parameters connected thereto, such as the "depth map" or the displacement vectors.

Further features of the invention will be set out in the appended claims, which are intended as an integral part of the present description.

The above objects will become more apparent from the detailed description of the improved stereoscopic shooting apparatus and method, with particular reference to the annexed figures, wherein:

- Fig. 1 shows a diagram of a stereoscopic shooting system adopting a configuration with parallel axes;

- Figs. 2 and 3 show a diagram of a stereoscopic shooting system adopting a configuration with converging axes; - Fig. 4 is a block diagram of an improved stereoscopic shooting apparatus according to the invention;

- Fig. 5 shows one possible association between graphic information and a stereoscopic parameter;

- Fig. 6 is a block diagram of a particular embodiment of the apparatus according to the invention, wherein the stereoscopic parameter is a displacement vector.

Referring now to Fig. 4, there is shown a stereoscopic shooting apparatus 100 which comprises:

- a stereoscopic shooting apparatus 10, comprising optics 10a for taking a left image and optics 10b for taking a right image, said images making up a pair of stereoscopic images, a sequence of pairs of stereoscopic images making up a stereoscopic video stream;

- a computation block 12;

- a normalization and quantization block 14;

- a graphic information association block 16;

- an overlay block 18;

- a delay block 20;

- visualization means 22.

The following will describe an improved stereoscopic shooting method according to the present invention.

The two video streams making up one stereoscopic pair, obtainable through the stereoscopic shooting apparatus 10, are inputted to a computation block 12, in which the value of a stereoscopic parameter of interest is computed.

The term "stereoscopic parameter" refers herein to one of the following parameters:

- convergence angle;

- convergence distance;

- difference between the distance of the object and the "convergence distance";

- stereoscopic base;

- a depth map, computed as known in the application field, whose values are directly connected to the distance between the point represented by a single pixel and the shooting apparatus 10;

- displacement vectors, which represent the apparent displacement of parts of the image, caused by the parallax between the video cameras of the optics 10a, 10b;

- horizontal projections of the displacement vectors, i.e. the horizontal disparities evaluated on the sensor (in pixel terms, such disparities have the same values as the disparities evaluated on the display);

- vertical projections of the displacement vectors, i.e. the vertical disparities evaluated on the sensor.

Said displacement vectors can either be defined for each pixel ("pixel based") or be evaluated by groups of pixels ("pixel block based"). The bigger the block the vectors refer to, the rougher the monitored indication. However, the optimal size of the block can be defined on the basis of a compromise between the apparatus' computation capability and visualization effectiveness.

The stereoscopic parameter of interest computed by the computation block 12 is then transferred to the normalization and quantization block 14, which carries out a normalization, and possibly also a quantization, of the value computed in the computation block 12, in order to adapt the value to the next operations.

In the graphic information association block 16, graphic information is associated with every single value of the normalized and possibly quantized parameter. The graphic information may be represented by a colour, extracted from a suitable set of colours ("palette"), or a graphic sign, e.g. a geometric figure, or symbols such as screens, grids, bars, and so on.

In a preferred embodiment of the invention, a gradient of visibly contrasting colours is determined and associated with the range of values of the parameter being monitored. Particular values of the parameter are associated with suitably defined colours, graphic signs or symbols, as will be illustrated more in detail hereafter.

The output of the association block 16 is an image containing information about the characteristics of the monitored parameter, e.g. a false-colour image. Said image is overlaid on a delayed background image coming from a delay block 20, wherein the delay is appropriately sized to allow processing by the computation block 12, the normalization and quantization block 14 and the association block 16. An overlay block 18 overlays the background image on the image comprising graphic information about the monitored parameter, and outputs said overlaid images to the visualization means 22, which displays a composite image. The graphic information may also be overlaid on one image only, whether the right one or the left one, of the pair of stereoscopic images.

Said composite image can advantageously be used by the three-dimensional camera operator in order to obtain useful information about the shooting being carried out. The following will describe one example of an improved stereoscopic shooting method according to the present invention, wherein the critical stereoscopic parameter being monitored is horizontal disparity, i.e. the horizontal projections of the displacement vectors.

In stereoscopic shooting, the most important horizontal disparity values that must be appropriately highlighted in order to avoid any perception errors are three: the null value, the minimum value, and the maximum value.

The preferred embodiment adopts a gradient of colours that varies from blue, associated with negative disparities (dots perceived as being "ahead" of the screen), to red, associated with positive disparities (dots perceived as being "behind" the screen).

Colour saturation decreases from the maximum absolute value to the null value of disparity.

Therefore, saturated blue is associated with the minimum disparity value; the saturation of blue decreases with the disparity value, and becomes null when disparity is null. As the positive disparity value increases, the colour becomes red, which gets more and more saturated up to the maximum disparity value, as shown in Fig. 5.

No colour is associated with the null disparity value, more in particular within a range around the null value whose extension can be defined at will during the quantization step, so that total transparency is displayed on the visualization means 22.

Any values below the minimum value of negative disparity are associated with green, which is easily distinguishable from blue.

Any values exceeding the maximum value of positive disparity are associated with yellowish orange, which is easily distinguishable from red.

Of course, a gradient of colours can be applied to any other stereoscopic parameter of interest.

Referring now to Fig. 6, there are shown more in detail the steps necessary for associating a colour with a displacement vector.

The method for evaluating the displacement vector described below is said to be "exhaustive" in that it compares the values of the mean square difference obtained from all the possible positions of the displacement vector. This method is the least sophisticated and employs the least complex algorithms, but it is slow and requires a lot of computational power. Other methods are known in the industry which, while still being very effective, are nonetheless faster and require less computational power. In step 50 a pair of right and left images relating to a stereoscopic frame are loaded into the memory.

In step 52 a luminance-only stripe j of the left image is extracted, possibly, in order to simplify the next computation, by adding neutral-colour blocks to the head and tail of the stripe, e.g. average-level gray (value of 128 when luminance is expressed on one byte). In step 54 a luminance-only block i is extracted from the right image.

In step 56 the parameters "vector" (two-dimensional variable indicating the displacement vector corresponding to the energy minimum found until the step being carried out; at the end of the process, it indicates the displacement vector pertaining to the image block taken into consideration) and "min_energy" (temporary variable indicating the minimum value of the mean square difference, i.e. energy, computed until the step being carried out).

In step 58 the block i is convoluted on the stripe j from a minimum (negative) limit to a maximum (positive) limit of the displacement vector: this step also computes the energy of the differences, which is then compared with the value stored in the variable "min_energy"; if it is smaller, then its value is stored into the variable "min_energy", and the corresponding vector is stored into the variable "vector".

In step 60 it is verified if the operation on the block i has been completed. If not, the flow chart returns to step 58.

If yes, in step 62 it is verified if . all stripes have been analyzed; if not, the parameter i is incremented and the flow chart goes on to step 54.

If not, in step 66 it is verified if all the values j of the image have been computed. If not, the parameter j is incremented by one unit and the flow chart goes on to step 52. If yes, in step 70 the map of the displacement vectors is scanned.

In step 72 the previously acquired values are subjected to normalization and possibly also to quantization and regularization.

In step 74 the actual association between vector values and colours is made. For example, the vectors having a negative value are associated with blue; the negative values below a lower limit are associated with yellow; the vectors having a positive value are associated with red; finally, the vectors having a positive value above an upper limit are associated with green. It should be noted that the colours are suitably defined in such a manner as to be perceptively opposite. Differentiated visualization therefore occurs when intermediate values and/or limit values are exceeded.

If the critical stereoscopic parameter being monitored is a depth map, reference should not be made to the absolute value thereof, but to the difference between the map value and the convergence distance value, taking for granted that both are normalized and therefore comparable with each other. The process for associating the depth map with false colours is similar to the one described above with reference to the example taking into account horizontal disparity as a stereoscopic parameter.

The features of the present invention, as well as the advantages thereof, are apparent from the above description.

A first advantage of the apparatus and method according to the present invention is that a stereoscopic camera operator can evaluate in real time any perception error sources which might generate a particularly annoying or unreal image when displayed on a screen.

A second advantage of the apparatus and method according to the present invention is that the operator receives in graphic form information which can be easily interpreted, so that he/she can immediately correct the shooting being carried out.

A further advantage of the apparatus and method according to the present invention is that the operator can select the stereoscopic parameter of his/her interest regardless of the shooting configuration in use.

The improved stereoscopic shooting apparatus and method described herein by way of example may be subject to many possible variations without departing from the novelty spirit of the inventive idea; it is also clear that in the practical implementation of the invention the illustrated details may have different shapes or be replaced with other technically equivalent elements.

For example, although not exclusively, the invention is mainly intended for use in the television or cinematographic stereoscopic shooting field.

For example, the visualization means may be either included in or associated with a stereoscopic video camera.

For example, the method according to the present invention, when appropriately modified, may also be implemented in combination with other types of shooting, e.g. "multiview", wherein it may be useful to be able to present the value of parameters of interest in an ergonomical manner.

For example, the visualization means may allow for two-dimensional or three-dimensional representation, and may in particular be a screen, a monitor or a viewfinder.

It can therefore be easily understood that the present invention is not limited to the above- described improved stereoscopic shooting apparatus and method, but may be subject to many modifications, improvements or replacements of equivalent parts and elements without departing from the novelty spirit of the inventive idea, as clearly specified in the following claims.

Claims

1. A stereoscopic shooting apparatus comprising shooting means (10a, 10b) for taking a stereoscopic shot, which are adapted to provide a pair of stereoscopic images comprising a right image and a left image, characterized in that it comprises:

- a computation block (12), which receives at its input said pair of stereoscopic images from said shooting means (10a, 10b) and computes values of a parameter relating to said pair of stereoscopic images;

- a normalization and quantization block (14), which subjects to normalization, and preferably also to quantization, the values of said parameter computed in said computation block (12);

- association means (16) adapted to associate graphic information with said values of said parameter;

- an overlay block (18), which overlays said graphic information on said pair of stereoscopic images and outputs said overlaid images to visualization means (22);

- visualization means (22) for displaying said graphic information by means of an image overlaid on at least one image of said pair of stereoscopic images, thus forming a composite image.

2. An apparatus according to claim 1, wherein said values of said parameter depend on the shooting geometry of said pair of stereoscopic images.

3. An apparatus according to claim 1, wherein a delay block (20) is further included which is appropriately sized to take into account the processing carried out by said computation block (12), normalization and quantization block (14) and association block (16), for the purpose of forming said composite image correctly.

4. An apparatus according to one or more of the preceding claims, wherein said graphic information comprises a false-colour image.

5. An apparatus according to claim 4, wherein said false colours comprise a gradient of colours associated with a range of values of said parameter so as to appear visibly contrasting.

6. An apparatus according to one or more of claims 1 to 3, wherein said graphic information comprises a graphic sign, in particular screens, grids and bars, or a geometric figure.

7. An apparatus according to one or more of claims 4 to 6, wherein differentiated visualization occurs when intermediate values and/or limit values of said parameter are exceeded.

8. An apparatus according to one or more of the preceding claims, wherein said parameter represents one of the following parameters: convergence angle, convergence distance, difference between the distance of an object being shot and said convergence distance; stereoscopic base; depth map; displacement vector; horizontal disparity; vertical disparity.

9. A stereoscopic shooting method, wherein a stereoscopic shot is taken through shooting means (10a, 10b) adapted to provide a pair of stereoscopic images comprising a right image and a left image, characterized in that it comprises the steps of:

- receiving at the input of a computation block (12) said pair of stereoscopic images from said shooting means (10a, 10b) in order to compute values of a parameter relating to said pair of stereoscopic images;

- normalizing, and preferably also quantizing, through a normalization and quantization block (14), the values of said parameter computed in said computation block (12);

- associating graphic information with said values of said parameter through association means (16);

- overlaying said graphic information on said pair of stereoscopic images and outputting said overlaid images to visualization means (22);

- displaying said information, through said visualization means (22), by means of an image overlaid on at least one image of said pair of stereoscopic images, thus forming a composite image.

10. A method according to claim 9, further comprising the step of delaying, through a suitably sized delay block (20), the visualization of said at least one image of said pair of stereoscopic images of said composite image so as to take into account the processing carried out by said computation block (12), normalization and quantization block (14) and association block (16).

1 1. A method according to one or more of claims 9 to 11, wherein said graphic information comprises a false-colour image.

12. A method according to the preceding claim, wherein said false colours comprise a gradient of colours associated with a range of values of said parameter so as to appear visibly contrasting.

13. A method according to one or more of claims 9 to 11, wherein said graphic information comprises a graphic sign, in particular screens, grids and bars, or a geometric figure.

14. A method according to one or more of claims 9 to 13, wherein differentiated visualization occurs when intermediate values and/or limit values of said parameter are exceeded.

15. A method according to one or more of claims 9 to 14, wherein said parameter represents one of the following parameters: convergence angle, convergence distance, difference between the distance of an object being shot and said convergence distance; stereoscopic base; depth map; displacement vector; horizontal disparity; vertical disparity.