WO2002063568A1

WO2002063568A1 - Method and system for generating virtual and co-ordinated movements by sequencing viewpoints

Info

Publication number: WO2002063568A1
Application number: PCT/FR2001/000363
Authority: WO
Inventors: Robin Larroque
Original assignee: Robin Larroque
Priority date: 2001-02-08
Filing date: 2001-02-08
Publication date: 2002-08-15

Abstract

Processing digital images essentially consists in mixing several sequences of virtual movements of different objects, by recalculating the position of each digitised image of a sequence on the co-ordinates of a compatible original sequence, obtained for example from the same shooting conditions. The acquisition of an object (8) or of a human subject (7) is carried out from several viewpoints at an instantaneous or quasi-instantaneous moment and the display of successive shots sequenced on a common screen (5) of the terminal (6) is capable of tracing the steps of a visually regular movement, thereby transforming a situation shot an instantaneous moment into a sequence developed in time, thus specifically justifying the designation virtual movement. Said method and its system can be used for motion representation of a person (9) virtually trying on spectacles, clothes, accessories or any other element modifying his appearance.

Description

METHOD AND SYSTEM FOR CREATING VIRTUAL AND COORDINATED MOVEMENTS BY LINKING VIEWPOINTS

The problem which the present invention proposes to solve is that of creating a virtual movement around a real object in coordination with a virtual movement around another object, with the possibility of mixing the views in such a way specifies, this being able to be used in particular for the representation in movement of a person trying in a virtual way glasses, clothes, accessories or any other element modifying his appearance.

Concerning a particular application such as visual aids for the sale of glasses, a first known device consists in taking a digital photograph of the user without glasses. The image of the face is stored in the memory of a microcomputer and is displayed on a screen. A pair of glasses is chosen from a library of objects and is superimposed on the image before the eyes of the digitized person. Such a device makes it possible to change the frames on a face without the user actually doing it, therefore virtually. But thus conceived, this type of device remains static. The problem is to give back movement to the virtual fitting so as to approach the natural conditions of a real fitting. Another method (n ° FR 96 13054) consists in placing on the face of the user a marker like a special pair of glasses, for example very colored so that it distinguishes well from the color of the skin, so as to an image capture system can electronically recognize user movements in real time. It is then necessary to recalculate the identified area to replace it with a pair of glasses from a library of objects according to the user's choice. This process is complex, particularly to replace a pair of glasses smaller than that which serves as a marker. Indeed, in such a case, pixels of the skin of the face located under the glasses serving as a marker will have to be recalculated in software to allow surrounding the glasses of smaller format tested virtually. These numerous software calculations are necessarily cumbersome and seem ill-suited to the current constraints of data transfer over a network, in particular over the Internet.

The present invention, on the contrary, makes it possible to obtain a representation in movement by resorting to a minimum of calculations, and by allowing very precise assemblies, such as that of a pair of glasses rotating at the same time as the face of a user. .

To this end, according to a brief presentation, the process of creation around various objects of virtual and coordinated movements includes at least the following steps: a) the digitization of a real subject from several shooting sources b) the processing of digitized images c) the display of recomposed images and is distinguished in that the processing of digitized images essentially consists of mixing several sequences virtual movement of different objects, by authorizing the repositioning of each digitized image of a sequence on the coordinates of a compatible original sequence, resulting for example from the same shooting conditions The acquisition in a) of an object or a human subject is made from several points of view in an instantaneous or almost instantaneous time and the display of the successive views on the same screen is able to mark the stages of a visually regular movement, thus transforming a grasped situation in instantaneous time in a sequence developed over time, thus specifically justifying the name used for virtual movement.

The points of view are oriented along a series of axes close to each other determined by the shape of the movement sought during the display in step c).

The shooting stage can be reproduced on another object or on a human subject. We analyze, to possibly correct it, the subject's position by an automatic visual recognition system or even by a physical survey, the stages being essentially the following:

- from landmarks defined for the digitization of the object from the first view, we note, for an image from the same angle, the position of the new object relative to the landmarks.

- if necessary, the coordinates of the second view compared are shifted, in particular along the vertical axis, the horizontal axis, the axis of rotation until its correspondence with the points which served to locate the exact position of the first view from the same angle - depending on the constraints specific to the configuration of points of view chosen, we usefully reproduce the comparison step on the view from another angle, significantly distant from the first, and we first correct the display of this new view according to the same criteria as previously, then we specify the position of the view previously described on the depth axis by modifying the scale of the image if the readings reveal a significant difference, this being calculated according to the proportionality formulas applicable to the optical situation.

- the resizing corrections are declined on all the views provided by the other angles. The digitized object having a shape which may have to be hidden depending on the movement and the shape of the other object or the subject with which we want to mix it, a processing step on the appearance of the first object can consist in partially hide and add for example fade effects by playing on the alpha channel.

The display of the different synchronized and calibrated views of the objects to be assembled is displayed on the same screen. These views can come from an immediate or older record and stored in a suitable database. They can come from an analog or digital source, or have been created in synthesis. The display technique uses any known technology to improve the visual appearance, in particular for smoothing outlines.

This process is accompanied by a system comprising at least:

- a device for digital acquisition and storage of images - a device for correcting images

- an image display device and is distinguished in that the capture and processing device is able to precisely register digitized images of different objects and display them in a mixed and synchronous manner.

The accompanying drawings illustrate the invention:

FIG. 1 schematically represents the functional system of the invention and the software steps

FIG. 2 schematically represents the device of the invention around a human subject.

FIG. 3 schematically represents the device around a spectacle frame

Figure 4 gives an example of another possible configuration of the camera. Figure 5 shows the spectacle frames on a user and their rotation effect.

Figure 6 shows a fading of some parts of the frame

Figure 7 shows an example of two possible correction readings

A first step of digital image acquisition and storage is done by means of a device from several recording sources, having their own memory or being connected to a remote memory, configured around the subject according to the movement that one wishes to create. These can be digital cameras.

This step is applied to each object that we want to animate. It can include specific digital processing steps adapted to the visual constraints of the subsequent representation of several objects mixed on the same image, such as for example fade effects on different parts of an object.

The second step consists in correcting the position and the scale of the images of an object that we want to mix with the images of another object. The images of a library of objects having been recorded according to an ideal place in relation to the sources and a subject, particularly if it is a living subject, which may not occupy exactly the ideal place provided by As part of each source, a step of analysis, comparison of the data and correction consists in digitally cropping the object on each of the images of the movement. This corrective step uses any useful technique, automatic correction by visual recognition or, in a lighter way, correction by physical reading of a deviation indicated by the user himself.

A step of visualization on screen synchronously displays the different views of the different objects. It uses any image processing software necessary to optimize the visual rendering as well as to improve the transitions between each image to soften the effect of movement. It can notably be a calculation of intermediate views by interpolation or crossfades. This step can take place locally or via a wide area network, the information displayed on the user's computer being able to come from remote databases. The transition effect between each view depends on the desired aesthetic result as a function of the number of cameras placed, taking into account the amplitude of the movement created and the nature of the digitized objects.

The system according to the invention comprises a system of shots composed of devices which can be digital cameras connected to a remote control system triggering the shots instantaneously or almost instantaneously.

The relative position of the cameras with respect to the subject is controlled particularly with regard to the overall rigidity of the installation thanks to the presence of stable supports, if necessary adjustable, capable of preventing the risks of displacement during use. untimely. However, the system includes a software module allowing to digitally crop the images of each source, this in order to allow the installer to perfect his adjustments and to carry out more comfortably the framing corrections necessary for the harmony of the movement sought. This device is shown diagrammatically in FIG. 1 by the presence of the module M2. This M2 module intervenes on the parameters, in particular two-dimensional, in order to refine the relative positioning of the object on the screen for each of the sources. These possible corrections are in practice part of the initial installation phase and remain memorized for subsequent uses by the appropriate instructions.

It should be noted that the device does not impose any particular movement a pήori. We can choose to arrange the sources so as to create a rotating movement, but we can choose any other movement such as that of moving towards the object, which can be useful in the case where we would like to facilitate the reading of a written mention on an object or underline a detail.

Provision is made, depending on the modes of application, for a possible step of aligning the sources of images which may consist in placing the subject on a seat adjustable in height, or conversely in moving the sources themselves, or even in multiplying these sources. according to several levels, planning to use those of the level most suited to the size of the subject.

The image storage device can be developed in two stages. A provisional storage is carried out for the shooting of each device in Ml module, then is transferred to a database p) The shooting device (1) is repeated in (3) to the scanning of another object with which we want to be able to then mix the views. If the movement to be created is open and leaves a large amount of choice, we must however repeat the configuration of the image acquisition around a new object in a coordinated manner in order to ensure the compatibility of the movement of the two objects on the same screen. Note that refinement of the orientation of the shooting optics can be favored by the use of a target placed at a comparable location in the two configurations, this in order to facilitate the setting of frames in x and y axis. The images of (3) can be stored in the database (4) of another computer unit, according to the specific needs of the planned network development. The scanning qualities are adapted to each situation.

The M3 module is a device for cropping the images of a subject that does not occupy exactly the ideal place for which the digitization of other objects could have been planned. The corrections take into account all the degrees of freedom of the subject. We measure, for example from two angles, the differences with one or more determined reference points, the differences between the points on the images of the reference object and the corresponding points on the images of the second object. The search for these offsets can be the subject of several reading modes.

In a first option, command buttons arranged near the display screen allow the user to indicate himself the position of an agreed reference point on the screen, for example the position of his irises, of his neck or even his shoulders. These indications can be repeated from another angle of view in order to determine the three-dimensional position of the subject in relation to all of the scanning sources. The deviations noted are processed logarithmically to reflect the values on each image of each device. Each image is moved laterally, horizontally and possibly in rotation so as to be superimposed on the ideal position. If the difference results in an increase in the size of the subject on the screen, the overall size of the image in question is also corrected in order to give the animation of the subject a regular evolution. In a second option, this module integrates any other known technology for establishing the precise statement of the position of the subject, such as automatic visual recognition, if the latter is adapted to the nature of the objects as well as to the constraints of implementation cost.

The corrective values are memorized with the images in question so as to avoid new interventions during subsequent uses. An M5 module uses database interrogation and information routing software techniques. It includes a step of processing the images to perfect, if necessary, the quality of their visual rendering according to known techniques, then to control the display thereof in a mixed and synchronized manner. A display screen displays the animation. It can be tactile depending on the choices made.

The method and the system according to the invention can be advantageously applied in the field of fitting spectacle frames. They allow you to virtually try on spectacle frames to more easily compare the aesthetic effect on the representation of your face while retaining a rotating movement approaching that naturally practiced in front of a mirror. In a format compatible with the Internet, it allows remote virtual fitting over a network. Plate 2 shows a concrete installation. The number of devices is indicative. It can in particular be increased as indicated in (11) to increase the fluidity of the recreated movement, depending on the type of aesthetic rendering sought and the cost requirements to be respected. A series of digital cameras (1) is installed rigidly according to a series of determined angles noted oc, β and y. The arrangement adopted here consists in making the optical axes converge towards the theoretical axis of rotation of the user's neck and head, keeping a focal length and a distance from the film plane constant.

In this mode of use, a device for leveling the subject can consist of a height-adjustable seat.

In a corollary digitization step (1,8), various spectacle frames are seized under the same conditions as (3,7) by occupying the place that they would have occupied on the nose of the user (7), for example the median of the shooting screen, a line thus conceived as corresponding to the level of the interpupillary line of the user in the frame of the image. The centering of the framing of the image in the horizontal direction between (3.7) and (1.8) obeys the same imperatives of homogeneity. It is recalled that one of the precise framing techniques of those skilled in the art consists in placing a small mirror as a target at the point where one wishes to direct the optical axis so as to immobilize the source of the image when it encounters its own reflection. A spherical target is also useful when planning to compare the framing, for example by superimposing on the same monitor, views of the sphere provided by each point of view. As indicated in the system, a module makes it possible to digitally play the setting of each source when it is displayed and thus facilitate the finalization of the settings.

Figure 7 shows how the subject can be off-center with respect to the theoretical axis for which the glasses have been digitized. A corrective step includes, depending on the options, a physical reading of the difference communicated by the user to the computer unit or automatic processing using a known method of shape recognition capable in particular of recognizing the position of the pupils or of the iris. In the first case, electronic buttons can allow the user to move small circles on the screen to superimpose them on his irises and then to validate the operation. The image is enlarged as necessary to facilitate this task. It thus notes a difference (δ) which allows a first registration of the images obtained by this point of view.

This reading is repeated for another view so as to locate the place of the subject in three dimensions, in particular in depth and in the positioning of verticality. A measurement (λ) is this time taken from an angle significantly distant from the first, here (β + γ). The measurement (λ) also makes it possible to retrospectively correct the position of the subject from the previous angle of view along the axis of the depth and therefore to modify, if necessary, its relative size on the screen, according to a known principle of proportionality. Trigonometric calculations complete the corrections for each position of the other views by declining the readings according to the angles formed by the arrangement of the other sources.

In this embodiment, the user (7) can then choose the frames he wishes to try virtually by pressing control buttons on an interactive terminal (6) or even directly on the screen if it is with a touch function.

The images of the glasses have previously undergone a special treatment by a software module of known technique, here that of a gradient mask (10) which blurs the parts of the glasses supposed to be hidden from view by the hair or else by the stop of the nose for certain angles of views.

Figure 5 shows the final rendering on a rotational movement around the image of the face (9) of the subject (7). Software processing of the transitions between each view of a movement can provide crossfade effects or any other desired effect such as an interpolation or morphing calculation. Finally, a known image processing device can usefully supplement the device by making it possible to improve the visual rendering of the contours of the spectacle frame by a smoothing process. Instructions control the mixed and synchronized display of the processed views, as well as their sizing and positioning.

Claims

claims

1. Method for creating virtual and coordinated movements of various objects comprising at least the following steps: a) digitization of a real subject from several sources of photography b) processing of digitized images c) display recomposed images characterized in that the processing of the scanned images essentially consists of mixing several sequences of virtual movement of different objects by allowing each scanned image of a sequence to be repositioned on the coordinates of a compatible original sequence originating for example the same shooting conditions

2. Method according to claim 1, characterized in that the acquisition in a) of an object or a human subject is made from several points of view in an instant or almost instantaneous time the arrangement of which is capable of marking the stages visually regular movement when displayed on a screen, thus transforming a situation captured in instantaneous time into a sequence developed over time, thus specifically justifying the name used for virtual movement.

3. Method according to any one of the preceding claims, characterized in that the points of view are oriented along a series of axes close to each other determined by the shape of the movement sought during the display in step c)

4. Method according to claim 2 characterized in that the step of taking pictures is reproducible on another object or on a human subject and that there is analyzed, to possibly correct it, the position of the subject by an automatic system of visual recognition or a physical survey, the steps being essentially the following:

- if necessary, the coordinates of the second view compared are shifted, in particular according to the height, width, axis of rotation, until it obtains its correspondence with the reference points of the previous view

- as a function of the constraints specific to the configuration of the points of view chosen, the comparison step is usefully reproduced on the view from another angle, significantly distant from the first, and the display of this display is first corrected new view according to the same criteria as previously, then the positioning and sizing of the view previously described is specified if the readings reveal a significant difference, this according to the proportionality formulas applicable to the optical situation, taking account for example of the length focal length, the distance of the subject from the film plane, the size of the image and the size of the subject in the image.

- the repositioning and resizing corrections are declined on all the views provided by the other angles

5. Method according to claim 1 and 2 characterized in that the digitized object having a shape which may have to be hidden depending on the movement and the shape of the other object or the subject with which it is to be mixed, a step processing on the appearance of the first object can consist in partially masking it and in adding for example fade effects (10) by playing on the alpha layer.

6. Method according to claim 1 characterized in that the display of the different synchronized and calibrated views of the objects to be assembled takes place on the same screen, these views being from an immediate or older recording and stored in a database adequate, they can come from analog or digital sources or have, for some of them, been created in synthesis, the treatment technique using any known technology to improve the visual appearance, in particular to calculate the smoothing of contours, and to meet the conditions of transfer via networks.

7. Optical and digital image processing system, comprising at least: - a device for digital acquisition and storage of images

- an image correction device

- An image display device characterized in that the capture and processing device is able to precisely readjust the digitized images of different objects and to display them in a mixed and synchronized manner.

8. System according to claim 7, characterized in that it comprises a device for several sources of shooting (1, 3) which can be digital cameras arranged in an oriented manner, adjustable by any suitable system, the devices being connected to a remote control system triggering the shots instantaneously or almost instantaneously and connected to a computer unit which adequately collects the data and allows them to be displayed on a screen (5).

9. System according to any one of the preceding claims, characterized in that the device comprises memory and software allowing the comparison of the parameters of the images of each source, the application of corrective calculations, the aesthetic treatments of the images (10), the storage of the information then the transfer of the data to a display screen (5,9) in a local network and / or a remote network, in particular of the Internet type.

10. The system as claimed in claim 9, in which the processing device is capable of reading the position of the subject as desired by an automatic visual recognition system or by a physical reading on the screen, the steps to be carried out being essentially the following:

- from common reference points of correspondence between the position of the subject on the different views, comparison of the new shot with that of the object previously scanned from the same angle.

- if necessary, offset the coordinates of the second view compared according to the height, width (δ), the axis of rotation until its correspondence with the benchmarks of the previous view - repetition of the comparison step on the view from another angle, significantly distant from the first according to the constraints specific to the configuration of the chosen viewpoints, first correcting (λ) the display of this new view according to the same criteria as before, then by possibly declining, if the readings reveal a significant difference, these corrections on the previously described view, depending in particular on the perspective of the optical situation, its focal length parameters, the distance of the subject from the film plane, the size of the image and the size of the subject in the image.

- declination of repositioning and resizing corrections on all the views provided by the other angles, in particular by the use of trigonometric calculations.

11. Use in any field where a representation is likely to be modified by mixed and coordinated visual elements, in particular in that of the virtual fitting of glasses frames, clothing, accessories and that of the presentation of consumer products .