FR2873213A1 - Stereoscopic signal obtaining method for domestic application, involves forming pairs of images from sequence of monoscopic images based on preset time slot that is function of image acquisition speed, to construct stereoscopic signal - Google Patents

Abstract

The method involves obtaining a sequence of monoscopic images captured by an image acquisition apparatus according to an acquisition mode permitting capture of multiple images during a regular movement tangential to a plane of a lens of the apparatus. Pairs of images coming from the sequence are formed based on a preset time slot that is function of speed of acquisition of the images. A stereoscopic signal is constructed from the pairs. Independent claims are also included for the following: (A) a device for obtaining a stereoscopic signal from a sequence of monoscopic images (B) a computer readable medium for implementing a method of obtaining a stereoscopic signal (C) a computer program product having instructions for implementing a method of obtaining a stereoscopic signal.

Description

The present invention relates to a method for obtaining a signal

  stereoscopic from a sequence of monoscopic images.

  The present invention is more particularly applicable to a domestic use that does not require an image acquisition device with particular functions.

  Correlatively, the present invention relates to a device adapted to implement such a method.

  In order to visualize images in three dimensions, it is necessary to obtain a pair of particular images that can be viewed in stereo by specific stereo display devices. For this, the one who wants to obtain stereo viewable images seeks to associate images taken with a different angle of view corresponding to the angle of view that can be with the right eye and with the left eye. After combining these two images taken with a different angle of view, it is possible to obtain the stereoscopic effect by using for example specific glasses which have the effect of superimposing the two images and thus give the impression of relief to the image.

  In the Japanese patent applications JP20020035013 and JP20020035090, there is described a technology used in digital cameras that helps the user to take adapted shots to obtain a pair of images that will be viewable in stereo.

  This method of assisting the user allows only a pair of still images to be viewed in stereo. It does not make it possible, for example, to obtain a video sequence that can be viewed in stereo. In addition, this method does not allow the user to adjust or move a defect and does not allow a moving object to be taken since it requires a considerable adjustment time for the second shot.

  Finally, this state-of-the-art method makes it necessary to use a specific acquisition device equipped with this technology.

  The present invention aims to remedy the aforementioned drawbacks by proposing a method for obtaining a stereoscopic signal from a sequence of monoscopic images without it being necessary to use a specific acquisition device.

  The invention also proposes such a method capable of obtaining pairs of stereoscopic images automatically and adaptively.

  Finally, the invention proposes to obtain both the possibility of viewing in stereo still images or video sequences.

  For this purpose, the present invention aims a method for obtaining a stereoscopic signal from a sequence of monoscopic images. The method comprises the following steps: obtaining a sequence of monoscopic images having been captured by an image acquisition apparatus according to an acquisition mode allowing the taking of several images during a regular and substantially tangential movement; in terms of the purpose of the acquisition device; forming pairs of images from the sequence of images, each pair being formed according to a predetermined time distance; - construction of a stereoscopic signal from the pairs thus formed.

  Thus, it is possible from a sequence of images previously obtained in a homogeneous manner with respect to an object, without using a specific image acquisition apparatus, to construct one or more pairs of viewable images. in stereo automatically. The formation of stereoscopic pairs and the construction of the stereoscopic signal can be done adaptively with respect to the sequence of images.

  According to a preferred embodiment, the predetermined temporal distance is a function of the image acquisition speed of the image sequence.

  For this, preferably, this image acquisition speed is deduced by computing at least one motion vector between the images.

  Thus, it is possible to adapt the temporal distance which will determine the stereoscopic pairs according to the sequence. If this has not been taken at a constant speed, the temporal distance can be adapted accordingly and the stereoscopic signal will have a visual coherence.

  Advantageously, the step of forming a pair of images comprises the following sub-steps: selecting an image of the sequence constituting the first image of the pair; determining a group of images located temporally at a close distance from the first image of the predetermined time distance; - construction of the second image of the pair from the images of the determined group.

  According to a particular embodiment of the invention, the construction of the second image of the pair is carried out by selecting the image situated at a closest temporal distance from the predetermined one.

  Thus, the pair of images thus constructed will be viewable with a stereo effect.

  According to another particular embodiment, the construction of the second image of the pair is performed by interpolation of at least a portion of the images of the group determined.

  Thus, if there are no images in the sequence at a distance precisely equal to the predetermined temporal distance with respect to the first image of the pair, it is possible to construct this image constituting the second image of the pair for that this one forms with the first a pair stereoscopic adapted.

  According to a preferred embodiment, the method comprises a step 5 of calibration of the images of the pairs formed, so as to improve the visual correspondence between the two images.

  The calibration will for example make it possible to obtain a better coherence between the two images of the pair in the case where during the acquisition of the image sequence, the user has not exactly respected the regular movement and tangential to the plane. of the goal.

  It is thus possible to perform a geometric registration such as a vertical registration, and / or a registration of the signal such as a luminance registration.

  According to a preferred embodiment, the construction of a stereoscopic signal is effected by grouping the image pairs formed so as to obtain a sequence of stereoscopic images.

  This produces a video sequence that can be viewed in stereo.

  According to another embodiment, the construction of a stereoscopic signal is performed by selecting a pair of images from the pairs of images formed, so as to obtain a stereoscopic image. This selection is done for example according to a criterion specific to the signal such as the variance of the histogram or the mathematical correlation between the images of the pair.

  Thus, a representative image of the image sequence will be selected to be viewable in stereo.

  The present invention also relates to a device for implementing the method according to the invention. This device comprises: means for obtaining a sequence of monoscopic images having been captured by an image acquisition apparatus according to an acquisition mode allowing the taking of several images during a regular movement and substantially tangential to the lens plane of the acquisition device; means for forming pairs of images originating from the image sequence, each pair being formed as a function of a predetermined temporal distance; means for constructing a stereoscopic signal from the pairs thus formed.

  The device has the same advantages as the method it implements.

  The present invention also provides a computer-readable information storage means or a microprocessor retaining instructions of a computer program for implementing a method for obtaining a stereoscopic signal as above.

  The present invention also aims at a means for storing removable information, partially or totally, readable by a computer or a microprocessor retaining instructions of a computer program, allowing the implementation of a method of obtaining a signal stereoscopic as above.

  The present invention also relates to a computer program product that can be loaded into a programmable apparatus, comprising instruction sequences for implementing a method for obtaining a stereoscopic signal as above, when the program is loaded. and executed by the programmable device.

  Other features and advantages of the invention will become apparent in the description below.

  In the accompanying drawings, given by way of nonlimiting examples: FIG. 1 illustrates a device embodying the invention; - Figure 2 schematically shows the positioning of a camera for acquiring a sequence of images to which one can apply a treatment according to the invention; FIG. 3 is a block diagram showing the processing steps according to a preferred embodiment; FIG. 4 is a block diagram describing the step of forming stereoscopic pairs according to one embodiment; FIG. 5 diagrammatically illustrates the temporal distance used in the embodiment of FIG. 4; FIG. 6 is a block diagram describing a mode of implementation of the calibration according to the invention; FIG. 7 illustrates the calibration according to the invention on a pair of stereoscopic images; - Figure 8 schematically shows a device adapted to implement the invention.

  First of all, with reference to FIG. 1, a device 2 for obtaining a stereoscopic signal from a sequence of monoscopic images in accordance with the invention will be described, the sequence being acquired by a device of FIG. acquisition of images. The final stereoscopic signal is viewable by a suitable device.

  The acquisition device comprises means 1 for acquiring a sequence of monoscopic images, for example using a video mode camera or a digital video camera. Alternatively, one can use a digital camera in burst mode, that is to say, shooting at a fixed rate of 2 or 3 images per second. The set of images acquired sequentially in this mode can also be considered a digital video. The acquisition of the signal must be done by moving the device. As illustrated in FIG. 2 on the example of an APN digital camera, the capture apparatus has three approximate axes of symmetry: a horizontal axis 10, a vertical axis 11 and a depth axis 12. The axes 10 and 11 define the shot objective plan. The camera APN describes a trajectory 13 in space. It is important that when shooting the speed of movement of the camera does not vary too much. The acquisition of the signal must be done by moving the apparatus tangentially to the plane defined by the axes 10 and 11, as shown in the diagram of Figure 2. For example, it is possible to perform a signal acquisition under in the form of a traveling shot (translation of the camera in the plane defined by the axes 10 and 11), or by aiming at the subject whose signal is to be acquired while rotating around concentrically.

  Returning to FIG. 1, the device implementing the method according to the invention comprises processing means 2 making it possible to obtain a stereoscopic signal, in the form of a still image or video, from the video sequence previously recorded. According to a preferred embodiment of the invention, the processing is performed by software, executed on a personal computer or embedded in a printing device or image acquisition for example. The user must either transfer the video to a computer equipped with this software, or connect the camera to a printer with this software.

  The processing device 2 comprises means 21 for forming pairs of images originating from the sequence, comparable to images respectively viewed by each eye of the user, optional means 22 for calibrating the images obtained and means 23 for constructing a final stereoscopic signal.

  The signal thus obtained is viewable by the user using a suitable device provided with stereoscopic image display means 3 such as a stereoscope, 3D glasses or polarized light display.

  We will now describe in detail the method of processing a video sequence to obtain a stereoscopic signal with reference to FIG. 3. At the input of the algorithm there is a video V, acquired at the preliminary step El0 according to a method respecting the rules described above with reference to Figure 2. The first step before processing E30 is the loading of the video on a device provided with the software implementing the invention. Such a device will be described in detail with reference to FIG. 8. In the preferred embodiment, the software is installed on the user's personal computer. In this case, the loading of the video is carried out in the usual manner as to display a set of digital photos on a personal computer. In an alternative implementation mode, the software is embedded in an apparatus intended to display the final result of the invention: a printer or a video projector adapted to project stereoscopic sequences for example.

  The treatment begins at step E31 of forming intermediate stereoscopic pairs. For a pair of images to be a stereoscopic pair, each of the images in the pair must match what one of the user's eyes sees. In order to extract such pairs of images from a simple video sequence, we will go through the sequence a first time to extract a signal corresponding to an eye, for example the left eye, then we will go through the initial sequence a second time with a time shift to generate the second signal, corresponding to the right eye in this example. The technical problem to be solved is to automatically determine the time shift that corresponds to a shooting displacement substantially equal to the distance between the eyes of the user. An algorithm giving a solution to this problem according to a preferred embodiment will be described with reference to FIG. 4.

  Step E31 is followed by a calibration step E32 of the intermediate stereoscopic pairs obtained. Indeed, it is sometimes necessary to correct a number of shooting defects, such as slight vertical movements of the camera, in order to obtain a stereoscopic sequence with better visual rendering. The implementation of this step will be detailed with reference to FIG.

  After calibration, a stereoscopic signal ready for viewing is produced in step E33. Depending on the number of images of the initial image sequence and the wishes of the user, it is possible to produce either a stereoscopic video signal or a stereoscopic still image signal.

  If the user wishes to obtain a video stereoscopic signal, in a preferred embodiment it is proposed to assemble the left view and the right view of the stereoscopic video side by side, and to save the whole in a conventional format such as the MPEG-2 standard to facilitate storage and transport of the resulting video.

  Alternatively, if the user wishes to obtain a pair of still images, in the preferred embodiment is implemented in step E33 the selection of a preferred stereo pair in the set of available pairs. For example, the choice of this pair can be left to the user interactively. Alternatively, in the preferred embodiment of the invention, it is proposed to automatically select the best stereoscopic pair according to a predetermined criterion, for example the pair whose histogram has the greatest variance, or the pair which has the greatest correlation between images. The selected still image pair can be stored in a standardized digital format such as JPEG, or printed on paper.

  With reference to FIG. 4, we will describe in detail the formation of intermediate stereoscopic pairs according to a preferred embodiment of the invention. Note that in this example it is assumed, without loss of generality, that the shooting was done from left to right, so we first select the left view of the stereoscopic pair and then determine its corresponding right view. A reverse method can of course be considered.

  At the initialization step, E41, the first image for the left view is selected, it then constitutes the first image of the pair, denoted IG here. For example, we can take the first image of the video sequence.

  In the next step E42 is determined a temporal distance or time shift d which is related to the movement of shooting between the two eyes. This temporal distance d is a function of the acquisition speed of the images of the sequence, assuming that this acquisition speed is greater than the moving speed of the filmed objects. This speed can be almost constant during the duration of the shooting, or slightly variable in time. It is therefore possible to determine the temporal distance of one single time for the entire sequence, and then to associate it with all the processed images in the event that the acquisition speed is constant. If one has knowledge on the acquisition speed, one can determine d = d0 constant a priori, in which case the step E42 is limited to reading this distance d0.

  However in practice it can be assumed that if the video is acquired by a user hands-free, the video acquisition speed is not exactly constant. In the preferred embodiment, the acquisition speed is determined punctually on subsets of images of the image sequence, in order to obtain an adaptive time distance value d over time. Preferably, the estimation of the acquisition speed of the video is done by estimating the motion vector between successive images of the sequence, using known techniques such as block matching. It is assumed here that the movement is regular on a group of successive images and that the motion of the objects filmed is weak, that is to say that the displacement of the objects of the scene is mainly due to the movement of the camera. shooting.

  Thus, the time distance d is adaptively determined. The value determined for d can be applied to a set N of successive images of the left view if it is found during motion estimation that it is regular on a group of N images.

  We then go to step E43 where is determined an image search time window, located around the image at a distance d from the image of the current selected left view, IGt. For example, we take the group of images formed images before and after the image at a distance approximately equal to d of IGt. The group of images thus formed, all located at a distance close to the temporal distance d of the first image, is illustrated in FIG.

  In step E44 a subset of images is selected within the window F. In the preferred embodiment, this subset is reduced to an image, selected so that the viewpoint from which it has been taken either distant from the point of view of the current left-hand image IGt by a distance substantially equal to the distance between the two eyes. To do this, the correlation between the current left image IGt and each of the images of the window F is calculated and the one for which the correlation value obtained is closest to a predetermined correlation value is selected. According to an alternative embodiment, the variance of the difference between the two images is compared to a certain threshold. In the case where all the images of the window F result in correlation values with the image IGt very close, we select all of these images.

  In step E45 the right view IDt corresponding to the left view IGt is finally constructed. If only one image has been retained in step E44, this becomes the second image of the pair, IDt. If several images have been retained, the selected images are interpolated in order to obtain this second image of the IDt pair. This interpolation consists, in the preferred embodiment, of generating an image in which each pixel has the average value of the pixels of the same spatial position in each of the images selected.

  Other alternative embodiments can be envisaged. For example, if the estimated time distance d does not have an integer value, it is possible to interpolate between the images of the window F so as to obtain an estimated representation of the image taken at a distance precisely equal to d of the IGt image. This image will then be chosen as the second image of the stereoscopic pair, IDt.

  Once the stereoscopic pair obtained, it is sent to the calibration module, detailed below in FIG.

  The algorithm continues with the test in step E47 to check whether the group of images belonging to the time window F contains the last image of the sequence. If this is the case, the treatment ends. Otherwise, the next image of the sequence is selected as the current image of the left view, IGt, and steps E42 to E47 are performed again.

  The calibration of a stereoscopic pair of images will now be described with reference to FIGS. 6 and 7. Indeed, as explained above, two views are available corresponding to a left-hand view IG and a right-hand view ID, represented in FIG. Figure 7. The correspondence of these images may be imperfect because they are extracted from a sequence of images that may have been taken "hands-free", without specific positioning mechanism of the camera.

  Several corrections are possible in this calibration step. In the preferred embodiment, two types of corrections are described, a geometric correction concerning the vertical offset and a signal correction by compensation of the luminance. These corrections can be implemented independently.

  The steps of the calibration algorithm are described with reference to FIG.

  First, in step E61 a vertical offset is determined between the first image IG and the second image ID of a previously built pair. For this, consider a predetermined search interval, for example -10 to +10 pixels. For each value p pixels of this interval, the vertical shift of one of the images, for example of the image ID, of pixels, is performed and the correlation between the shifted image and the original image of the other is calculated. view. The vertical offset selected is the one that maximizes the correlation value.

  In step E62, the offset determined in the step prior to the concerned view, for example in the right-hand view, is then applied. In the preferred embodiment, the missing pixels will be replaced by black pixels. We then obtain a new right view, ID ', illustrated in FIG.

  In the next step E63, a luminance compensation is performed on the pair of images IG and ID '. Luminance compensation is a technique known to those skilled in the art, also called histogram equalization. It consists of calculating the histogram of the image IG, then modifying the luminance levels of the image ID 'so that its histogram is as close as possible to that of the image IG in the sense of a mathematical distance. An ID image *, also shown in FIG. 7, is thus obtained.

  The stereoscopic pair finally obtained (OG, ID *) is the final stereoscopic pair after calibration.

  We will now describe, with reference to Figure 8, a device diagram adapted to implement the method according to the invention.

  Such an apparatus is for example a microcomputer 800 connected to different peripherals, for example a digital camera 801 connected to a graphics card. The apparatus may also be connected via a specific port to an image acquisition apparatus such as a digital camera for receiving a data stream to be processed according to the invention, in particular a sequence of digital images.

  The apparatus could also be a printer or other device capable of implementing the invention.

  The device 800 comprises a communication interface 818 connected to the communication network 80 capable of transmitting digital data processed by the device to possibly send them to a remote machine for viewing / printing. The device 800 also comprises a storage means 812 such as for example a hard disk. It also includes a disk drive 814 816. This disk 816 can be a diskette, a CD-ROM or a DVD-ROM, for example. The disk 816, like the disk 812, may contain processed data according to the invention, such as for example an initial digital image sequence, as well as the program or programs implementing the invention which, once read by the device 800, will be stored in the hard disk 812. According to one variant, the program Progr allowing the device to implement the invention may be stored in ROM 804 (called ROM in the drawing). In the second variant, the program may be received to be stored in the same manner as previously described via the communication network 80.

  This same device has a screen 808 making it possible in particular to display the data to be processed and to serve as an interface with the user who can thus parameterize certain processing modes, using the keyboard 810 or any other means of pointing, such as for example a mouse, an optical pencil or a touch screen.

  The central processing unit 803 (called the CPU in the drawing) executes the instructions relating to the implementation of the invention, instructions stored in the read-only memory 804 or in the other storage elements. When powering up, the processing programs stored in a non-volatile memory, for example the ROM 804, are transferred into the RAM RAM 806 which will then contain the executable code of the invention as well as registers for storing the variables. necessary for the implementation of the invention.

  More generally, an information storage means, readable by a computer or by a microprocessor, integrated or not into the device, possibly removable, stores a program implementing the method according to the invention. The communication bus 802 allows communication between the different

  elements included in the microcomputer 800 or connected to it. The representation of the bus 802 is not limiting and in particular the central unit 803 is able to communicate instructions to any element of the microcomputer 800 directly or through another element of the microcomputer 800.

Claims (32)

  1. A method for obtaining a stereoscopic signal from a sequence of monoscopic images, characterized in that it comprises the following steps: - obtaining (El0) a sequence of monoscopic images having been captured by a image acquisition apparatus according to a mode of acquisition allowing the taking of several images during a regular movement and substantially tangential to the plane of the lens of the acquisition apparatus; - forming (E31) image pairs from the image sequence, each pair being formed according to a predetermined time distance; - construction (E33) of a stereoscopic signal from the pairs thus formed.   2. Method for obtaining a stereoscopic signal according to claim 1, characterized in that the predetermined temporal distance is a function of the acquisition speed of the images of the image sequence.   3. Method for obtaining a stereoscopic signal according to claim 2, characterized in that the acquisition speed of the images is deduced by calculating at least one motion vector between the images.   4. A method for obtaining a stereoscopic signal according to one of claims 1 to 3, characterized in that the step of forming a pair of images comprises the following substeps: - selection (E41) d an image of the sequence constituting the first image of the pair; determining (E43) a group of images temporally located at a close distance from the first image of the predetermined time distance; - construction (E44, E45) of the second image of the pair from the images of the determined group.   5. A method for obtaining a stereoscopic signal according to claim 4, characterized in that the construction of the second image of the pair is performed by selecting the image located at a time distance closest to that predetermined.   6. Method for obtaining a stereoscopic signal according to claim 4, characterized in that the construction of the second image of the pair is performed by interpolation of at least a part of the images of the determined group (E45).   7. Method for obtaining a stereoscopic signal according to one of claims 1 to 6, characterized in that it comprises a calibration step (E32) of the images of the pairs formed, so as to improve the visual correspondence between the two images.   8. A method for obtaining a stereoscopic signal according to claim 7, characterized in that the calibration step is performed by a geometric registration (E61).   9. A method for obtaining a stereoscopic signal according to claim 7 or 8, characterized in that the calibration step is performed by a registration of the signal (E63).   10. A method for obtaining a stereoscopic signal according to claim 8, characterized in that the geometric registration is a vertical registration.   11. A method for obtaining a stereoscopic signal according to claim 9, characterized in that the registration of the signal is a luminance registration.   12. Method for obtaining a stereoscopic signal according to one of claims 1 to 11, characterized in that the construction of a stereoscopic signal is performed by grouping (E33) the image pairs formed so as to obtain a sequence of stereoscopic images.   13. Method for obtaining a stereoscopic signal according to one of claims 1 to 11, characterized in that the construction of a stereoscopic signal is carried out by selecting (E33) a pair of images from the pairs of images formed, so as to obtain a stereoscopic image.   14. Method for obtaining a stereoscopic signal according to claim 13, characterized in that the selection of a pair is carried out according to a criterion specific to the signal, such as the variance of the histogram or the mathematical correlation between the images of the pair.   15. Device for obtaining a stereoscopic signal from a sequence of monoscopic images characterized in that it comprises: - means (1) for obtaining a sequence of monoscopic images having been captured by an image acquisition apparatus according to an acquisition mode allowing the taking of several images during a regular movement and substantially tangential to the objective plane of the acquisition apparatus; means (21) for forming pairs of images from the sequence of images, each pair being formed as a function of a predetermined temporal distance; means (23) for constructing a stereoscopic signal from the pairs thus formed.   16. Device according to claim 15, characterized in that the predetermined time distance is a function of the acquisition speed of the images of the image sequence.   17. Device according to claim 16, characterized in that it comprises means for calculating at least one motion vector between the images so as to deduce the acquisition speed of the images.   18. Device according to one of claims 15 to 17, characterized in that the means for forming a pair of images comprise: means for selecting an image of the sequence constituting the first image of the pair; means for determining a group of images situated temporally at a close distance with respect to the first image of the predetermined temporal distance; means for constructing the second image of the pair from the images of the determined group.   19. Device according to claim 18, characterized in that the means for constructing the second image of the pair comprise means for selecting the image located at a time distance closest to that predetermined.   20. Device according to claim 18, characterized in that the means for constructing the second image of the pair comprise means for interpolating at least part of the images of the determined group.   21. Device according to one of claims 15 to 20, characterized in that it comprises means for calibrating the images of the pairs formed, so as to improve the visual correspondence between the two images.   22. Device according to claim 21, characterized in that the calibration means comprise geometric registration means.   23. Device according to claim 21 or 22, characterized in that the calibration means comprise means for resetting the signal.   24. Device according to claim 22, characterized in that the geometric registration means are vertical registration means.   25. Device according to claim 23, characterized in that the signal resetting means are luminance registration means.   26. Device according to one of claims 15 to 25, characterized in that the means for constructing a stereoscopic signal comprise means for grouping the image pairs formed so as to obtain a sequence of stereoscopic images.   27. Device according to one of claims 15 to 25, characterized in that the means for constructing a stereoscopic signal comprise means for selecting a pair of images from the pairs of images formed, so as to obtain a stereoscopic image.   28. Device according to claim 27, characterized in that the means for selecting a pair comprise means for calculating a criterion specific to the signal such as the variance of the histogram or the mathematical correlation between the images of the pair.   29. An image acquisition device, characterized in that it comprises a device according to one of claims 15 to 28.   30. A computer-readable information storage medium or a microprocessor holding instructions of a computer program, characterized in that it allows the implementation of a method for obtaining a stereoscopic signal according to the present invention. any of claims 1 to 14.   31. A partially or fully removable information storage medium readable by a computer or a microprocessor retaining instructions of a computer program, characterized in that it allows the implementation of a method of obtaining a stereoscopic signal according to any one of claims 1 to 14.   32. Computer program product that can be loaded into a programmable device, characterized in that it comprises sequences of instructions for implementing a method for obtaining a stereoscopic signal according to any one of claims 1 to 14, when the program is loaded and executed by the programmable device. 25