WO2015019208A9 - Appareil et procédé pour corriger des distorsions de perspective d'images - Google Patents

Appareil et procédé pour corriger des distorsions de perspective d'images Download PDF

Info

Publication number
WO2015019208A9
WO2015019208A9 PCT/IB2014/062727 IB2014062727W WO2015019208A9 WO 2015019208 A9 WO2015019208 A9 WO 2015019208A9 IB 2014062727 W IB2014062727 W IB 2014062727W WO 2015019208 A9 WO2015019208 A9 WO 2015019208A9
Authority
WO
WIPO (PCT)
Prior art keywords
point
projection
image
center
plane
Prior art date
Application number
PCT/IB2014/062727
Other languages
English (en)
Other versions
WO2015019208A1 (fr
Inventor
Pietro Porzio Giusto
Original Assignee
Sisvel Technology S.R.L.
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sisvel Technology S.R.L. filed Critical Sisvel Technology S.R.L.
Publication of WO2015019208A1 publication Critical patent/WO2015019208A1/fr
Publication of WO2015019208A9 publication Critical patent/WO2015019208A9/fr

Links

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/80—Geometric correction
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N13/00—Stereoscopic video systems; Multi-view video systems; Details thereof
    • H04N13/10—Processing, recording or transmission of stereoscopic or multi-view image signals
    • H04N13/106—Processing image signals
    • H04N13/111—Transformation of image signals corresponding to virtual viewpoints, e.g. spatial image interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10004—Still image; Photographic image
    • G06T2207/10012—Stereo images
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10016—Video; Image sequence
    • G06T2207/10021—Stereoscopic video; Stereoscopic image sequence

Definitions

  • the present invention relates to an apparatus and a method for correcting images, so as to reduce the deformations that appear, in both bidimensional and stereoscopic vision, when the images are viewed from a point of view not corresponding to the center of projection of the perspective according to which they have been produced.
  • Linear perspective also called simply perspective
  • Fig. 1 The basic criterion of perspective construction, shown in Fig. 1, consists in projecting onto a plane 101, referred to as “projection plane” or “projection frame” or simply “frame”, the points of the three-dimensional space as viewed from a "center of projection” C.
  • the straight line extending from the center of projection in the direction towards which the viewer's sight, or the camera lens, is oriented is called “optical axis”.
  • a Cartesian reference is defined, such as the one shown in Fig.
  • the axes originate from the center of projection C, the z-axis coinciding with the optical axis, the j-axis being vertical, oriented upwards, and the -axis being horizontal, oriented from left to right for the viewer.
  • the following will generally refer to the case wherein the frame is perpendicular to the optical axis (this is the case of the perspective referred to as "vertical frame perspective"), but the man skilled in the art will understand that the method of the present invention is not limited to such a case, but is also applicable to cases wherein the frame is not perpendicular to the optical axis.
  • the z-axis is also called “depth axis", since the "depth" of a point of the three-dimensional space is defined as the distance of that given point from the y-plane.
  • the projection plane 101 is at a distance / from the center of projection C and is perpendicular to the optical axis, which intersects the projection plane (101) at the point Ic.
  • the projection Q of a point of the space d results from the intersection between the projection plane 101 and the "projective straight line", i.e. the straight line that passes through the point to be projected A and the center of projection C.
  • Photo and video cameras produce images that are theoretically compliant with linear perspective, but in fact real lenses often introduce more or less visible distortions, e.g. like the so-called “barrel” and “cushion” distortions.
  • the present invention will not deal with such kinds of distortions, which are the subjects of many studies and correction techniques (see for example European Patent EP1333498 Bl to Agilent Technologies Inc. and International patent application WO 98/57292 Al by Apple Computer).
  • the present invention will not even tackle those image deformations which are caused by errors in the positioning of the lenses with respect to the desired position and orientation, such as tapering of images of tall buildings taken from below and deformation of images of documents photographed obliquely or from point of views offset from the document axis.
  • image deformations have been amply discussed in the literature as well (see for example documents US 7,990,412 Al, US 2009/0103808 Al, BR PI0802865 A2, US 2004/0022451 Al, US 2006/0210192 Al,
  • the correction technique of the present invention processes the images as if they were perfectly compliant with linear perspective. Any distortions with respect to the linear perspective, and in particular those mentioned above, will not be taken into account and will be present in the processing results.
  • the present invention relates to deformations that appear in perspective images when said images are viewed from a point of view not corresponding to the center of projection, e.g. if the image of the frame 101 (Fig. 1) is viewed from the point V, not from the center of projection C.
  • FIG. 2 An example of such deformations is shown in Fig. 2.
  • This shows the image of a solid 202 drawn by a calculation program rigorously in accordance with the linear perspective rules.
  • the image of the solid 202 is located in the lower right corner of a frame, of which the figure only shows the lower right quadrant (also called fourth quadrant) 201 in order to enlarge the elements of interest (i.e. the solid 202 and the center Ic of the projection plane) compared to the dimensions that they would have if the entire frame were reproduced.
  • the lower right quadrant also called fourth quadrant
  • Fig. 2 is observed from a point close to the corresponding center of projection, which is located on the straight line passing through Ic, orthogonal to the figure plane, at a distance from the sheet (i.e. the image plane) equal to 25 times the cube edge, then the solid 202 will appear correctly with a cube shape, not with the deformed parallelepiped shape.
  • Fig. 3 represents the horizon plane of a perspective, i.e. the z-plane of Fig. 1, in which a square ABDE is drawn, with its front side adjacent to the projection plane.
  • the element 301 represents the projection plane 101 of Fig. 1, as viewed from the direction of the j-axis.
  • the projection plane and the image plane coincide, although in reality the image is normally reproduced on a support which is distinct from the projection plane. If the two planes are distinct, they can be transferred one over the other, with all their respective geometric elements, by means of a homothetic transformation known to those skilled in the art.
  • the linear perspective reproduces the reality well if the images are viewed from the point corresponding to the center of projection, but the images will turn out to be deformed if the viewer moves away from such point.
  • Pannini painted suggestive architectural views with wide angles of view, with no visible perspective deformations (v. Thomas K. Sharpless, Bruno Postle, and Daniel M. German, “Pannini: A New Projection for Rendering Wide Angle Perspective Images”, Computational Aesthetics in Graphics, Visualization, and Imaging (2010), The Eurographics Association 2010, http://vedutismo.net/Pannini/panini.pdf).
  • Denis Zorin, Alan H. Barr see Zorin D., Barr A. H., "Correction of geometric perceptual distortions in pictures", SIGGRAPH '95: Proceedings of the 22nd annual conference on Computer graphics and interactive techniques (1995), pp.
  • Robert Carroll et al. (Carroll R., Agrawal M., Agarwala A., "Optimizing content-preserving projections for wide-angle images", SIGGRAPH '09: ACM SIGGRAPH 2009 papers (New York, NY, USA, 2009), ACM, pp. 1-9) have proposed to minimize deformations by adapting the projection to the contents.
  • a man- machine interface is provided, through which the user can characterize the areas and elements of the images to be corrected in particular ways, such as straight lines that must remain as such, people's faces, etc. This method is however time-consuming and uncomfortable, and requires specific calibrations for various types of elements.
  • Image recognition and correction are made easier by determining the orientation of the video cameras by means of sensors (accelerometers, gyroscopes). As can be guessed, this method is very complex and is only applicable in particular circumstances. Furthermore, it does not solve the problem of deformations that arise, in general, when a perspective image is viewed from a point of view not corresponding to the center of projection.
  • the present invention provides an adequate solution to the above-described problem by disclosing a method, and the associated apparatus, for correcting the deformations that appear on images when the latter are viewed from a point not corresponding to the center of projection of the perspective.
  • the apparatus and the associated method are applicable to both the bidimensional reproduction of a single image and the reproduction of a pair of stereoscopic images.
  • Said apparatus comprises suitable means for acquiring a bidimensional image or a pair of stereoscopic images, with sufficient data to determine the coordinates of the point corresponding to the center of projection of the perspective (e.g. image center and focal length) and with the associated depth map.
  • the latter is defined as the set of depths of the points of the scene represented in the image, that is, with reference to Fig. 1, the set of ⁇ -coordinates of the points of the three-dimensional scene.
  • data may be provided from which it can then be obtained.
  • this apparatus comprises storage means and processing means (e.g. a processor executing a suitable software code) configured to correct the position of the points of the acquired images, according to a technique called "Partial Perspective Gradient" (PPG).
  • PPG Partial Perspective Gradient
  • the corrections made by this technique tend to represent, in the image plane, the position of each point as if it had been captured with the lens pointed at it.
  • Such technique numerous variants of which can be defined, is based on the calculation of a gradient of the position of the points in the image plane. By integrating the components of this gradient, the functions can be found according to which the points will be positioned in order to make said correction.
  • each pixel represents, in an approximate manner, a small area of the image; for simplicity, however, in some parts of this description a geometric point will be identified as a pixel, accepting the approximation according to which the pixel's discrete coordinates are assumed to correspond to those of the geometric point it is intended to represent.
  • said gradient is calculated by taking into account a generic point A of the three-dimensional space, corresponding, according to linear perspective, to the point Q of the image plane 401, and to the point P resulting from the intersection between the projective straight line of A and the plane 402, which will be called ⁇ (or auxiliary ⁇ -plane), and which, in the preferred embodiment of the invention, is orthogonal to the projective straight line of A.
  • An incremental displacement Aa of A corresponds, in the ⁇ -plane 402, to an incremental displacement ofP (hereafter, an incremental displacement of A is meant to be an infinitesimal virtual increment, whether positive or negative, of one or more coordinates of the point A; such increment is hypothetically applied, according to the mathematical method of infinitesimal calculation, to determine the mathematical functions that bind the position of the points in the neighbourhood of A in the three- dimensional space to the positions of the points in the neighbourhood of the projection of A on the projection planes, as will be explained below). From the components of the incremental displacement the gradient components, i.e. the partial derivatives, of the functions FX(XQ, jg, ZA) and Fy(x& ZA) are calculated, with which to represent, in the image plane 401, the correct coordinates of the image of the point A.
  • the gradient components i.e. the partial derivatives, of the functions FX(XQ, jg, ZA) and Fy(x& ZA) are calculated, with which to represent,
  • Fig. 1 geometrically illustrates the operation of linear perspective
  • Fig. 2 illustrates the perspective representation of a cube in an offset position relative to the optical axis
  • Fig. 3 qualitatively illustrates the deformations inherent in linear perspective when viewing images from a point not corresponding to the center of projection
  • Fig. 4 geometrically illustrates one embodiment of the invention
  • Fig. 5 geometrically illustrates the operation of linear perspective in the stereoscopic case
  • Fig. 6 illustrates a plan view of a part of Fig. 4
  • Fig. 7 illustrates a representation of Cartesian references
  • Fig. 8 illustrates the trend of the raised cosine function and its complement that are used in the method and apparatus according to the invention
  • Fig. 9 illustrates stereoscopic images of the cube of Fig. 2;
  • Fig. 10 illustrates a block diagram of an apparatus according to the invention
  • Fig. 11 illustrates a flow chart of a process wherein the perspective correction of the present invention is applied.
  • the present invention relates to the correction of single bidimensional images or pairs of stereoscopic images, aimed at reducing the deformations that appear in perspective images when they are viewed from a point not corresponding to the center of projection of the perspective.
  • the following will describe an apparatus, and the method it implements, with reference to a preferred embodiment and some exemplary but non- limiting variants thereof.
  • the apparatus of the present invention processes images by using a technique called "Partial Perspective Gradient” (PPG), which corrects the position of the points (pixels) of the images in such a way as to locate them as if each point of the scene represented in the image had been captured by a lens pointed at it.
  • PPG Partial Perspective Gradient
  • said technique uses the coordinates of the point the representation of which has to be corrected, and the data defining the geometry of the perspective according to which the image has been generated.
  • the coordinates of the point A taken into account are obtained from the image, i.e. from the x, y coordinates of the point Q, and from the distance of the point s from the y-plane.
  • this distance is called “depth”
  • the set of distances of the points of the three-dimensional space from the y-plane is called “depth map”.
  • the geometry of the perspective according to which the image has been generated is essentially defined by the focal length and by the frame dimensions.
  • the interoptical distance b i.e. the distance between the centers of projection from which the two stereoscopic images have been generated, is also considered in addition to the focal length and the frame dimensions.
  • the data sufficient to determine the depth of the points represented in the image also known as depth data, may comprise the depth map, and can be obtained by using various methods known to those skilled in the art, whether for an image intended for bidimensional vision or for a pair of stereoscopic images. In the case of drawings or paintings, such data are implicit in the artist's project.
  • the depth map can be obtained from the disparity map, which represents the difference between the horizontal coordinates of homologous points of the two images, as shown in Fig. 5.
  • the two centers of projection are horizontally aligned, and that the horizontal axes of the Cartesian references lie in the horizon plane, i.e. the horizontal plane that contains the centers of projection.
  • Fig. 5 shows one example wherein a point A of the three-dimensional space is projected from two distinct centers of projection CL and CR, onto two distinct projection planes, i.e. plane 501 and plane 503, whereon the coordinates are referred to the Cartesian references ICLX JL and Ic R XRy R , respectively.
  • the optical axes starting off from the centers of projection, i.e. ZL from CL and ZR from CR, are parallel to each other and are located at a distance b (interoptical distance) from each other.
  • the planes 501 and 503 are orthogonal to said optical axes and equidistant from the respective centers of projection by a distance /, whereas their horizontal axes, respectively XL and «, lie on one same straight line, which is parallel to the line joining the two centers of projection CL and CR, and intersects the optical axes at the points Ic L and Ic R , respectively.
  • the homologous points QL and QR resulting from the projection of A onto the planes X JL and ⁇ , respectively, have the same vertical coordinate, which for simplicity is not indicated in Fig. 5, and have the horizontal coordinates XQL and XQR, respectively.
  • z A "depth" of the point taken into account i.e. the z coordinate of the point A of Fig. 4 or the coordinates of the point A of Fig. 5 on the axes z L and z R ;
  • b interoptical distance i.e. the distance between the two centers of projection CL and C R ,
  • f focal length i.e. the distance between the projection center C L or C R and the respective projection plane 501 or 503;
  • disp disparity i.e.
  • the equation (2) is defined with reference to the centers of projection and the projection plane, but, as is known to the man skilled in the art, the same equation, mutatis mutandis, can also be used for relating the depth of a point represented on the two images of a stereoscopy with the disparity measured on the pair of stereoscopic images and with the value of the equivalent interoptical distance of the geometrical configuration according to which the stereoscopy has been generated.
  • the center of projection of the perspective according to which an image has been generated can be located by providing the center of the projection plane and the distance between the center of projection and the projection plane (i.e. the focal length in the case of video cameras), or by providing the dimensions of the projection plane and the angle of view, since the angle of view, the image diagonal and the distance of the center of projection from the projection plane are bound by the equation (1).
  • the apparatus claimed by the present invention comprises suitable means for acquiring, in numerical form, a bidimensional image or a pair of stereoscopic images, along with depth or disparity data and other data sufficient to determine the coordinates of the center of projection C corresponding to the center of projection of the perspective, as mentioned above.
  • the apparatus claimed by the present invention applies the method ("Partial Perspective Gradient") according to the invention, wherein said method has been developed on the basis of the principle of representing the neighbourhood of each point of an image as if it had been pointed at while shooting or drawing. This principle takes into account that, in such conditions of sight orientation, small displacements of the point in question (e.g. the point A in the annexed figures) would be perceived.
  • the plane 402 lies at the same distance from the center of projection C as the projection plane 401. This configuration should however only be considered as a non-limiting explanatory example of the preferred embodiment. As the man skilled in the art will guess, and as will be explained below, the plane 402 can be set at any distance from the center of projection and with any orientation.
  • an incremental displacement Aa can be defined, in general, from a composition of a plurality of components, wherein these components, which are preferably three (three-dimensional space), are oriented along various directions.
  • the most common decomposition is the one which is made by using the three directions corresponding to the axes of the Cartesian reference, to which reference will be made in this description. Since in the explanatory example Aa has a direction which is parallel to the x-axis, Aa is completely characterized by its component along this axis, which will be called . Likewise, in this example the incremental displacement Aq of Q is completely characterized by its component along the -axis, which will be called Ax g . Note that, at any rate, a displacement Aa parallel to the x- axis of the point s will cause a displacement Aq , also parallel to the -axis, of the point
  • this projection Ap can be defined by using the C 6y Cartesian reference.
  • This reference has its origin at C, which is coincident with the origin of the Cxyz reference, and its
  • the ⁇ -plane is orthogonal to the
  • the a-axis is defined by the intersection between the ⁇ -plane and the xz-plane (the a-axis lies in the xz-plane), whereas the ⁇ -axis, also passing through C, is orthogonal to both the a-axis and the
  • a displacement Aa parallel to the -axis of the point A implies, in the plane 402, a displacement Ap of P with components in both the direction of the a-axis and the direction of the ⁇ -axis.
  • the component in the direction of the ⁇ -axis is null, and hence only the component in the direction of the a-axis, referred to as Aa p , will remain to be treated.
  • Aa p is related with Ax g .
  • the calculation can be made by using the common rules of geometry and mathematics, which are known to the man skilled in the art. They essentially provide for changing the reference system, switching the representation of the displacement Aa from the Cxyz Cartesian reference to the C 6y Cartesian reference.
  • the formulae for changing the reference systems can be found in school books and on various Internet sites, among which, for example, the following: http://www.cns.gatech.edu/ ⁇ predrag/courses/PHYS-4421-10/Lautrup/space.pdf.
  • the expression (3 ay) indicates that the incremental displacement Aa p of the point P in the direction of the a-axis does not depend on the displacement component Avg 0 f the point Q in the direction of the y-axis, because the j-axis is orthogonal to the plane in which the a-axis lies, as aforesaid when commenting on Fig. 7 (the a-axis lies in the xz- plane).
  • the formula (4axa) provides results that are only slightly different from the integral calculation (4ax). For example, in the case of application to the cube shown in Fig. 2, in which there are points corresponding to angle of views of 90°, the calculations made with the formula (4axa) will differ by less than 1% from those made with the formula (4ax). As can be noticed, the use of the formula (4axa) does not require, unlike that of the formula (4ax), any steps of numerical integration, thus advantageously reducing the processing time and load.
  • the formulae (4..) constitute the first embodiment of the "Partial Perspective Gradient” technique of the present invention, consisting of representing, in terms approximated in the image plane, what would appear from the center of the perspective, in the neighbourhood of each point of the scene to be reproduced, if the optical axis passed through that point.
  • a certain number of variants of the above technique can be taken into consideration.
  • formulae may be used which represent dependence from the "disparity" between homologous points of stereoscopic images, coherently with the formula (2).
  • the formulae are determined by imposing that the point P is located on a segment VA, instead of the segment CA, with V distinct from C (Fig. 1, Fig. 3, Fig. 4).
  • the point V may preferably be located on the optical axis passing through the points C and Ic, or away from said optical axis.
  • a second embodiment of the idea consists of applying partly the formulae (4..) and partly the linear perspective formulae.
  • linear perspective reproduces images well within certain limits, and therefore within such limits it may be profitable to maintain the reproduction provided by linear perspective, combining the formulae (4..) with the linear perspective ones.
  • such combination may be made in such a way as to gradually switch from exclusive application of the formulae (4..) to exclusive application of the linear perspective formulae, but it may also be carried out in other manners that the man skilled in the art will be able to imagine.
  • One way to make such combination is to multiply the results of the formulae (4..) by a first factor, preferably comprised between the unitary value and the null value, and to multiply the results of the linear perspective formulae by a second factor, preferably complementary to the first factor; the results of the products thus obtained are then added up.
  • one example of a function suitable for creating the multiplicative factors is the raised cosine function represented by the curve 801, together with its complement 802.
  • the curve 801 stays at the unitary value for abscissa values between zero and a limit t s ; afterwards, in the interval from t s to t f , it gradually decreases to zero, and then it stays at the null value. Instead, its complementary function 802 has the opposite trend.
  • the abscissas of Fig. 8, and hence the limits t s to t f , can be related with the offset angle at which the point to be represented is seen from the center of projection, or with the distance of the point from the center of projection, or with other parameters or combinations of parameters allowing the man skilled in the art to meet the requirements of a specific application of the method according to the invention.
  • Said second embodiment of the invention is also liable to all variations that may be conceived for the first embodiment.
  • aF is the enlarged image of the cube 202 (cube projected from the center of projection Ic);
  • aE is the image of the same cube 202 projected from a center of projection with an abscissa equal to twice the cube side;
  • bF is the image obtained by processing the image aF with the "Partial
  • Fig. 9 The improvement is even more apparent in stereoscopic vision, which can be obtained by viewing Fig. 9 as indicated below, the images and of Fig. 9 being intended for the left eye and their homologous images "as" and being intended for the right eye.
  • the latter have been generated from a center of projection located at an interoptical distance ("6" in Fig. 5) equal to twice the cube side from the center of projection of and "b F ".
  • the figure In order to obtain the stereoscopic vision of the images of Fig. 9, the figure needs to be reproduced in such a way that the distance between the vertical dashes hanging from the upper horizontal line is about equal to, or slightly shorter than, the viewer's interpupillary distance.
  • An adequate dimension is normally obtained by reproducing the sheet containing Fig. 9 in the A4 format (21cm wide). After making sure that the straight line joining the centers of the viewer's pupils is parallel to the straight lines that delimit the figure at the top and at the bottom, it is then necessary to look fixedly at the figure, so as to obtain the merging of the right and left images.
  • Such merging can be facilitated by initially looking fixedly at the arrows running from the lower images to the upper images, or, better still, by placing a card near the viewer's forehead in a position orthogonal to said delimiting horizontal straight lines, so that the right eye will not see the left image, or at least most of it, and vice versa.
  • the image "b" (obtained from the merging of the images b F and bs) well represents the shapes of a cube
  • the image "a” does not even look like a parallelepipedon, because the dimensions of the rear face appear to be bigger than those of the front face.
  • the horizontal and vertical lines maintain their own directions; in particular, the segments that lie in the projection plane, such as the edges of the front face of the cube reproduced in Fig. 9, maintain their length and their orientation, so that the front face of the cube will appear perfectly square.
  • the result of the processing carried out by using the PPG method instead, shows a cube which is seen obliquely, coherently with the fact that it is offset from the optical axis by 21° horizontally and by -15° vertically.
  • the PPG technique can be applied to stereoscopic images with even more advantage than to monoscopic images.
  • the PPG method can be applied to each one of the images of the stereoscopic pair, with all the possible variants mentioned above, but in the stereoscopy case there exist additional variants and expedients.
  • An image processing apparatus 1 like for example a photo camera or a video camera or the like, comprises image acquisition means 1001, input/output means 1002, a central processing unit (CPU) 1003, a read-only memory 1004, a random access memory (RAM) 1005, and means for producing processed images 1006. All of these elements of the apparatus 1 are in signal communication with one another, so that data can be mutually exchanged between any two of such elements.
  • CPU central processing unit
  • RAM random access memory
  • the image acquisition means 1001 can acquire both bidimensional images and pairs of stereoscopic images. If the images are equipped with the respective depth map and with data allowing to go back to the geometry with which they have been generated (e.g. focal length, angle of view, sensor inclination, or the like), the image acquisition means 1001 will acquire such data as well. Otherwise, some data can be set through a user interface (not shown in the drawings) in signal communication with the input/output means 1002, whereas for stereoscopic images the depth map may even be produced by the apparatus 1 itself, as will be explained below.
  • the user interface also allows the user to set options and variants, along with their parameters, that he/she may prefer to use in the specific case. For example, one can choose from the following settings:
  • the point of view relative to which the processing of the method of the invention is applied may be different from the center of projection C (see Fig. 1) relative to which the image to be processed has been generated.
  • the coordinates of Q and the depth of A being known, and having calculated with them the position of A in the three- dimensional space, one can in fact determine the projection of A onto any plane and from any point of view. For example, assuming as a center of projection a point K (the point corresponding, for example, to the point of view from which the viewer is supposed to be looking at the image) different from the center of projection C, one can determine the projection of A onto a plane perpendicular to the straight line that joins A to V, or onto a different plane. Processing the image from a point of view other than the center of projection C and onto various planes allows to make particular corrections to the image and to produce useful artificial images, such as those which can be used in stereoscopy for filling blanks, as will be discussed below.
  • the central processing unit (CPU) 1003 is that part of the apparatus which executes the calculation algorithms, including complementary operations that, after the application of the PPG technique, are useful for completing the correction of the images to be returned. These operations will be discussed while commenting on Fig. 11.
  • the central processing unit 1003 may actually comprise specially developed integrated circuits, one or more microprocessors, programmable logic circuits (e.g. CPLD, FPGA), and the like. These and other implementation possibilities neither anticipate nor make obvious the teachings of the present invention.
  • the read-only memory 1004 is preferably used for permanently storing some instructions for managing the apparatus and the instructions that implement the calculation algorithms, while the random access memory (RAM) 1005 is typically used for temporarily storing the images and the intermediate processing results.
  • RAM random access memory
  • the means for producing processed images 1006 return the processed images, e.g. by transferring them from the RAM memory 1005 to the input/output means 1002, so that said means 1002 can save said processed images into a permanent memory (e.g. a hard disk or a type of Flash memory, such as Secure Digital, MMC or the like), display them on one or more screens or other display means (not shown in the annexed drawings), print them, and the like.
  • a permanent memory e.g. a hard disk or a type of Flash memory, such as Secure Digital, MMC or the like
  • display them e.g. a screen or other display means (not shown in the annexed drawings), print them, and the like.
  • the process that implements the invention may comprise the following steps:
  • - start step 1101 during which the apparatus 1 is configured for processing at least one image
  • - setting step 1102 for acquiring the settings and the data that the user intends to manually provide for processing at least said image
  • step 1 103 during which the image to be processed is acquired by the image acquisition means 1001 and is preferably transferred into the RAM memory 1005 (for simplicity, it is assumed that, downstream of step 1103, the images are in numerical form, and that, in the event that they should be available in another form, the man skilled in the art will know how to convert them into numerical form); it is understood that the image acquired at step 1103 is equipped with data defining the geometry according to which the image has been generated, and that it is preferably also equipped with the depth map, or with data allowing to create it; the subsequent steps 1104 and 1105 will consider the case wherein the depth map is not provided at this step 1103;
  • step 1 104 depth map presence verification step 1 104, during which it is determined whether the depth map is available or not;
  • step 1106 during which the apparatus 1, by using the depth map acquired at step 1103 or calculated at step 1105, determines the position in the three- dimensional space of the points corresponding to the pixels of the image acquired at step 1103 and applies thereto the position correction algorithm according to the invention;
  • the result of step 1106 is a matrix indicating the position that the pixels of the image acquired at step 1103 must take after said processing; at this stage no pixel shifting occurs yet, in order to avoid having to repeat this kind of operation after the additional processing carried out in the next steps;
  • the resizing process consists of recalculating, starting from the result of step 1106, the position that the pixels of the processed image must take after the resizing applied at this step 1107;
  • - processed image returning step 1110 during which the image is preferably stored into an area of the RAM memory 1005, or another memory, and is made available for display, printing, transfer to other apparatuses, and other operations;
  • the apparatus 1 When the apparatus 1 is in an operating condition, after the start step 1101 said apparatus 1 enters the setting step 1102, and then, simultaneously or afterwards, the image acquisition step 1103. At the end of step 1103, the apparatus 1 verifies the availability of a depth map (step 1104) for what has been acquired at step 1103; if the depth map is available, the apparatus 1 enters the processing step 1106; otherwise, if the map is not present, the apparatus enters the depth map calculation step 1105 prior to proceeding with the processing step 1106. After step 1106, the apparatus may optionally carry out the resizing step 1107. Then the apparatus 1 carries out the overlap elimination step 1108, followed by the pixel shifting and blank filling step 1109 and by the processed image returning step 1110. The process ends at the final step 1111. At step 1106 the PPG technique of the present invention is applied in one of the above- described embodiments thereof.
  • the processing means 1003 and the storage means 1004 and 1005 of the apparatus 1 are configured to correct the image represented in the plane 401, which has been generated in compliance with the linear perspective rules relative to the center of projection C and comprises at least one first point Q, wherein said first point Q is the result of the perspective projection of the second point A of a region of the three- dimensional space from the center of projection C; to do so, the processing means 1003 and the storage means 1004 and 1005 execute the method according to the invention, which comprises the following steps: a) calculating the position of the second point A in the three-dimensional space;
  • some pixels of this area may overlap the pixels of areas that have not been shifted, or that have been shifted less, or that have been shifted in different directions.
  • one may, for example, have the pixel with less depth occupy the contended position (i.e. the pixel located at the smaller distance from the xy- plane will prevent seeing the farther one).
  • stereoscopy uses two images of the same scene viewed from two different points of view.
  • the corrective shift of a point of the left image is generally different from the corrective shift of the homologous point of the right image.
  • the blanks in either one of the two processed images can be at least partially filled by appropriately processing the other image.
  • one way of filling the blanks in the left (right) image is to superimpose it on an "artificial" image obtained by processing the right (left) image, assuming as a center of projection the same center of projection of the left (right) image.
  • step 1109 It is advantageous to carry out the pixel shifting and blank filling operations during step 1109 on already resized images and after having resolved any conflicts, i.e. downstream of step 1108, because resizing may also cause overlaps and blanks.
  • the application of the PPG method as described in the present invention will improve the vision of the images of objects located in offset positions relative to the pointing direction of the capturing device. Such improvements are evident in monoscopic vision, and are even more evident in stereoscopic vision.
  • the PPG technique which can be applied in real time while shooting or afterwards on acquired images, allows to capture scenes with angles of view exceeding the currently recommended limits, leading to significant advantages for both stereoscopic and monoscopic shooting.
  • the PPG technique turns out to be very versatile and can be optimized for different types of applications and apparatuses, which may even be characterized by very different processing capabilities.
  • the PPG technique can be used for correcting deformations in video streams, by applying it to every image it is composed of. This applies to both 2D and 3D video streams; in the latter case, the technique will have to be applied to each image of the stereoscopic pairs forming the 3D stream.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Processing Or Creating Images (AREA)
  • Image Processing (AREA)
  • Analysing Materials By The Use Of Radiation (AREA)
  • Apparatus For Radiation Diagnosis (AREA)

Abstract

La présente invention concerne un appareil et un procédé qui permettent de corriger les déformations qui apparaissent, à la fois dans une vision bidimensionnelle et stéréoscopique, lorsque des images générées conformément aux règles de perspective linéaire sont visualisées d'un point de vue ne correspondant pas au centre de projection de la perspective. Les corrections sont déterminées par une technique qui tend à représenter les images comme si chaque point les constituant était capturé par la lentille pointant vers ce dernier. Dans une telle hypothèse d'orientation de lentille, le gradient de position des points à représenter est calculé, et les coordonnées des positions que les points doivent prendre dans le plan image sont obtenues par intégration des composantes d'un tel gradient. La plage d'application du procédé est très large, étant donné qu'il permet différents modes de réalisation et de nombreuses variantes.
PCT/IB2014/062727 2013-08-08 2014-06-30 Appareil et procédé pour corriger des distorsions de perspective d'images WO2015019208A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
IT000683A ITTO20130683A1 (it) 2013-08-08 2013-08-08 Apparato e metodo per la correzione delle deformazioni prospettiche delle immagini
ITTO2013A000683 2013-08-08

Publications (2)

Publication Number Publication Date
WO2015019208A1 WO2015019208A1 (fr) 2015-02-12
WO2015019208A9 true WO2015019208A9 (fr) 2015-07-16

Family

ID=49354843

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/IB2014/062727 WO2015019208A1 (fr) 2013-08-08 2014-06-30 Appareil et procédé pour corriger des distorsions de perspective d'images

Country Status (2)

Country Link
IT (1) ITTO20130683A1 (fr)
WO (1) WO2015019208A1 (fr)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
FR3034078B1 (fr) 2015-03-27 2017-03-24 Airbus Helicopters Procede et dispositif pour signaler au sol un aeronef en vol, et aeronef muni de ce dispositif
CN110246169B (zh) * 2019-05-30 2021-03-26 华中科技大学 一种基于梯度的窗口自适应立体匹配方法及系统
DE102021103323A1 (de) 2021-02-12 2022-08-18 Carl Zeiss Ag Pannini-Objektiv und abbildendes optisches Gerät

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPH07306955A (ja) * 1992-07-24 1995-11-21 Walt Disney Co:The 三次元イリュージョンを生み出す方法およびシステム
US5960108A (en) 1997-06-12 1999-09-28 Apple Computer, Inc. Method and system for creating an image-based virtual reality environment utilizing a fisheye lens
WO2000035200A1 (fr) * 1998-12-07 2000-06-15 Universal City Studios, Inc. Procede de correction d'images destine a compenser la distorsion de celles-ci en fonction du point de vue
US20030141433A1 (en) 2002-01-31 2003-07-31 Gordon Gary B. Solid state image sensor array for correcting curvilinear distortion of a camera lens system and method for fabricating the image sensor array
JP2004040395A (ja) 2002-07-02 2004-02-05 Fujitsu Ltd 画像歪み補正装置、方法及びプログラム
US7990412B2 (en) 2004-11-01 2011-08-02 Hewlett-Packard Development Company, L.P. Systems and methods for correcting image perspective
US20060210192A1 (en) 2005-03-17 2006-09-21 Symagery Microsystems Inc. Automatic perspective distortion detection and correction for document imaging
US8244062B2 (en) 2007-10-22 2012-08-14 Hewlett-Packard Development Company, L.P. Correction of distortion in captured images
BRPI0802865A2 (pt) 2008-08-14 2011-11-22 Audaces Automacao E Informatica Ind Ltda processo de geração de arquivos de imagens com método automatizado de correção de distorções óticas
US8599238B2 (en) 2009-10-16 2013-12-03 Apple Inc. Facial pose improvement with perspective distortion correction
US8687070B2 (en) 2009-12-22 2014-04-01 Apple Inc. Image capture device having tilt and/or perspective correction

Also Published As

Publication number Publication date
WO2015019208A1 (fr) 2015-02-12
ITTO20130683A1 (it) 2015-02-09

Similar Documents

Publication Publication Date Title
US10609282B2 (en) Wide-area image acquiring method and apparatus
US10460459B2 (en) Stitching frames into a panoramic frame
CN109615703B (zh) 增强现实的图像展示方法、装置及设备
US9774837B2 (en) System for performing distortion correction and calibration using pattern projection, and method using the same
US8890934B2 (en) Stereoscopic image aligning apparatus, stereoscopic image aligning method, and program of the same
JP4918689B2 (ja) メッシュマップを用いて二次元画像から立体画像を生成する立体画像生成方法及び立体画像生成装置
US6204876B1 (en) Stereoscopic computer graphics moving image generating apparatus
CN109509146B (zh) 图像拼接方法及装置、存储介质
US20070248260A1 (en) Supporting a 3D presentation
JP6810873B2 (ja) 電子ディスプレイの前方または上に投影されて見える仮想３次元画像を作製するためのシステム、方法、およびソフトウェア
WO2015048694A2 (fr) Systèmes et procédés destinés à la correction de la distorsion de la perspective utilisant la profondeur
KR20110015452A (ko) 입체 이미지의 블러 향상
CN109191506B (zh) 深度图的处理方法、系统及计算机可读存储介质
US20190266802A1 (en) Display of Visual Data with a Virtual Reality Headset
EP3189493B1 (fr) Correction de la perspective des photos numériques avec l'aide d'une carte de profondeur
WO2015019208A9 (fr) Appareil et procédé pour corriger des distorsions de perspective d'images
JP5233868B2 (ja) 画像切出装置
US20180213215A1 (en) Method and device for displaying a three-dimensional scene on display surface having an arbitrary non-planar shape
CN112970044A (zh) 根据广角图像的视差估计
US9225960B2 (en) Apparatus and method for attenuating stereoscopic sense of stereoscopic image
CN107798703B (zh) 一种用于增强现实的实时图像叠加方法以及装置
JP2011211551A (ja) 画像処理装置および画像処理方法
WO2012176526A1 (fr) Dispositif de traitement d'images stéréoscopiques, procédé de traitement d'images stéréoscopiques et programme associé
US9602708B2 (en) Rectified stereoscopic 3D panoramic picture
KR20110025083A (ko) 입체 영상 시스템에서 입체 영상 디스플레이 장치 및 방법

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 14752400

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 14752400

Country of ref document: EP

Kind code of ref document: A1