WO2006134037A1

WO2006134037A1 - Apparatus for displaying virtual shoes on the feet of a user and the corresponding method

Info

Publication number: WO2006134037A1
Application number: PCT/EP2006/062903
Authority: WO
Inventors: Stefano Mottura; Marco Sacco; Luca Greci
Original assignee: Consiglio Nazionale Delle Ricerche
Priority date: 2005-06-13
Filing date: 2006-06-05
Publication date: 2006-12-21
Also published as: ITMI20051093A1

Abstract

An apparatus is disclosed for displaying virtual shoes on the feet of a user comprising a pair of special footwear items (1) suitable for being worn by a user, a camera (2) suitable for photographing the image of the user wearing said footwear items (1), computer means (3) comprising a database containing a catalogue of virtual shoes and software (MMAPP) capable of processing the image coming from the camera (2) replacing said pair of special footwear items (1) with a pair of virtual shoes chosen by the user from said catalogue. The computer means (3) is suitable for replacing in real time a further image (4) of the user wearing the chosen virtual shoes and the apparatus comprises a display (5) for displaying said further image (4).

Description

"Apparatus for displaying virtual shoes on the feet of a user and the corresponding method"

DESCRIPTION

* * * * * The present invention relates to an apparatus for displaying virtual shoes on the feet of a user and the corresponding method.

Purchasing a new pair of shoes, or footwear in general, is a process that requires the customer to take a series of well-defined steps, some of which are vitally important, to persuade him to purchase the product. A person enters a shop, looks at the models in the window or on the shelves, chooses the one that he likes most and asks to try it on. After the aesthetic choice, trying on the footwear is the most important moment for the decision. In fact the customer, by generally looking at himself in a mirror or looking directly at his feet, and feeling how the shoes fit, understands whether they satisfy him aesthetically, perhaps also taking account of the clothes that he is wearing at that moment (many people go to shoe shops wearing the clothes that they will use with the shoes that they would like to purchase) and whether they are sufficiently comfortable. During this decision-making process some of the following problems may arise: the exact size for the model chosen does not exist and so the customer resorts to a different size that will certainly not be comfortable to wear, the selected size does not fit exactly as the customer expected, he does not actually like the shoe once he has bought it, it does not suit the person and does not match the clothes that he is wearing. Owing to these and other reasons the customer has to make do with another model, repeating the decision-making process, or forgoing the purchase.

Currently, many companies and shoe shops are focussing on the supply of customized shoes, both via the Web, where the customer customizes the model exclusively from the aesthetic point of view, and within specialized companies that are able to provide not only aesthetic customization but also dimensional- functional customization.

In the context of product customization, the customer can choose a basic model of shoe and customize it according to a series of available versions, using a digital catalogue. As at this point the product has just been described, it will obviously not be available for the test, so an absolutely fundamental step is missing for following up with the sale.

In view of the prior art, the object of the present invention is to provide an apparatus for displaying virtual shoes on the feet of a customer that overcomes the aforementioned drawback. According to the present invention, this object is achieved by means of an apparatus for displaying virtual shoes on the feet of a user characterised in that it comprises a pair of special footwear items suitable for being worn by the user, a camera suitable for filming the image of the user wearing said shoes, computer means comprising a database containing a catalogue of virtual shoes and software capable of processing the image coming from the camera replacing said pair of special footwear items with a pair of virtual shoes chosen by the user from said catalogue, said computer means being suitable for providing in real time a further image of the user wearing said chosen virtual shoes, a display for displaying said further image. Still according to the present invention, it is possible to devise a procedure for displaying virtual shoes on the feet of a user as defined in claim 11.

Owing to the present invention it is possible to provide an apparatus for displaying virtual shoes on the feet of a user that enables purchasing of comfortable shoes that are not present in the shop, avoiding the purchase of shoes that are not suitable for the feet of the user.

The features and advantages of the present invention will be clear from the following detailed disclosure of a practical embodiment thereof, illustrated merely by way of non-limitative example in the attached drawings, in which: figure 1 is a diagram of the apparatus for displaying virtual shoes on the feet of a user according to the present invention; figure 2 is a block diagram of part of the apparatus in figure 1; figure 3 is a block diagram of the software and of the libraries used by the computer of the apparatus in figure 1; figure 4 is a flow diagram of the operation of the apparatus in figure 1.

With reference to figure 1 the apparatus for displaying virtual shoes on the feet of a user according to the present invention is shown in a schematic manner. The apparatus comprises a pair of footwear items 1 suitable for being worn by a user, a camera 2 suitable for photographing the image of said footwear items 1 worn by the user, a computer 3 suitable for processing the image coming from the camera 2 by replacing said pair of footwear items 1 with a pair of virtual shoes chosen from a catalogue and providing a virtual image 4 of the user wearing said shoes chosen from the catalogue, a display 5 for displaying said virtual image 4. The shoe catalogue is stored in the memory of the computer 3 in the form of a database and the computer 3 comprises suitable software, i.e. an application program, MMAPP, for processing the image in the disclosed manner. Each footwear item 1 is provided with a marker 10 which is the instrument through which the MMAPP program recognises the orientation and position of the foot of the user to orientate and position the virtual shoes chosen by the user that appears on the display in the virtual image.

The footwear items 1 constitute the instrument through which the apparatus according to the invention enables the feet of the user to be recognised and possible overlapping between the feet and the other elements of the image (trousers, hands of the user, etc.) to be recognised in order to show as realistically as possible the process of reflection in a mirror, where obviously all the objects that are not in the foreground are not reflected but hidden. The footwear items 1 have an identical structure and composition both for the right and the left foot but are obviously made in such a way as - A -

to be able to be worn by the two feet correctly. The footwear items 1 must preferably have the following features of both material and shape: the external surface thereof must be of the same material and the same colour and this colour (in the shooting area of the camera 2) must not be present on other objects or people, the external surface thereof must not have a specular light component, it must wrap well, i.e. in an even manner and as closely as possible the user's feet, it must be as light as possible and it is anyway preferable that they have a weight that is not greater than 100 grams.

The markers 10 are identical in structure and composition both for the right foot and for the left foot, but have different graphic elements so that the left foot can be different from the right foot. The markers 10 must not be of a material with specular components and have to be rigid so as not to bend under the weight of the markers 10. The markers 10 have to have moderate dimensions, of 6-8 cm on a side and have to have a preset geometrical shape, known to the MMAPP software to be recognised, preferably a triangular or rectangular shape. The display is an electronic appliance for displaying images, so it is an active device.

The display 5 can consist of a monitor or of a helmet for virtual reality that will be worn by the user. If the display 5 consists of a monitor, it acts as a usual mirror, into which the user looks to try on the virtual shoes and see whether he likes them and how they suit him. The monitor is preferably placed on the ground, with the plane thereof perpendicular or oblique with respect to the floor plane and has to be able to receive and correctly display the digital video signal from the computer 3, which is normally 24-bit colour and with at least 640x480 pixel digital resolution.

The camera 2 acts as the reflecting capacity of the monitor 5 used as a mirror. It must therefore be positioned in an appropriate manner with respect to the position of the monitor 5 so that the user has as far as possible the natural visual and correct sensation of looking into a real mirror when he looks at himself in the monitor. The camera 2 takes the frame in real time for which it has been set and in real time sends the computer 3 the photograms taken. It is preferably a digital camera as in the case of use of an analogue camera it would be necessary to introduce a system for electronic digitalisation of the images in real time. The photograms must arrive from the camera 2 to the computer 3 in real time at a frequency of at least 15 Hz. The camera must preferably film digitally with a resolution of at least 640*480 pixels and with 24 colour bits, it must have optics such as to minimise geometrical distortions (for example barrel distortion and pin-cushion distortion) and must have a good range of colours so that the user sees himself mirrored in the screen 5 with the correct colours and so that the MMAPP software operates with good performance.

The apparatus according to the invention comprises lighting means 6 for lighting the feet of the user wearing the footwear items 1. Said means 6 emits even white light so as not to create marked shadows on the clothes and footwear items 1 worn by the user. The lighting means 6 is made such that the electric operating currents do not interfere with the filming frequency of the digital camera 2. Preferably, the lighting means 6, together with the environmental conditions of the place of use of the apparatus according to the invention, must not cause the user discomfort through excessive heat or light or dimensions of any assembly or weight-bearing structures.

The computer 3 is the unit that, in relation to what the camera 2 films, supplies at the outlet on the display 5 the image of the user with the virtual shoes instead of the footwear 1. The computer 3 comprises a memory and a data-processing unit and the application MMAPP software is installed and run thereupon. The computer 3 is configured in such a way as to be able to perform said functions in real time.

The MMAPP software produces the following result: from the video flow coming in real time from the camera 2 on the display 5 the same video flow is displayed but, instead of the image of the user with the footwear items 1, the image of the user with the virtual shoes is seen. The MMAPP software enables the virtual shoe to be displayed instead of the footwear 1 using the known technique of chromakey in real time recorded on the colour of the footwear item 1. The MMAPP software enables the virtual shoe to be displayed in a consistently correct position in relation to the rest of the body of the user thanks to the tracing system of the position of the foot in real time performed by the markers 10. The virtual shoe is a three-dimensional model that is suitably prepared and is substituted in real time for the footwear item 1 in the photograms that arrive from the camera 2. The MMAPP software del computer 3 comprises the following functions, as shown in the block diagram in figure 2. The MMAPP software comprises an initialisation function 100; in fact, before entering the continuous real-time processing cycle, an image is taken and stored, i.e. a single photogram of the frame of the camera 2 without any user or other moving objects; said photogram is indicated as the FLOOR. This image will be used to remove any parts of the footwear items 1 that are still present after the entire processing phase has been performed but not yet sent for display on the screen.

The MMAPP software further comprises a function 101 for recognising the marker 10; the current photogram 22 from the camera 2 reaches the computer 3 in the form of an image. The image is examined in real time to identify whether the figure of the marker 10 is present therein, the geometrical specifics thereof being known a priori.

The MMAPP software also comprises a function 111, associated with the function 101, for calculating the position of the marker 10 with respect to the camera 2, if the marker 10 is found; said calculation is made on the basis of the marker 10.

The MMAPP software comprises the function 102 associated with the function 111, for calculating the position of the virtual shoe. The three- dimensional model of the virtual shoe, i.e. the virtual shoe seen by the user on the display 5, is positioned in function of the position of the marker 10. The three-dimensional model of the shoe is positioned in the virtual three- dimensional space of the new reference system, indicated with a rendering system in the corresponding point in real space (filmed in real time by the camera), on the footwear items 1 worn by the user. As the marker 10 is outside the user's feet (who wears his own shoes with the footwear 1 above them) it is necessary to know the offset in the six degrees of liberty between the local reference system of the three-dimensional model of the shoe and the position of the marker 10. The position given to the three-dimensional model of the shoe is the sum of the local three-dimensional reference system and of the position of the marker 10.

The MMAPP software further comprises the extraction function of the chromakey 103, i.e. the extraction of the colour of the footwear 1; for processing this function 103 it is necessary to have the photogram 22. For each pixel of coordinates xim, yim of the image in a two-dimensional system it is checked whether the colour of the pixel is the chromakey colour within a certain tolerance, i.e. whether the pixel of the image belongs to that of the footwear items 1. If this occurs, i.e. if the colour of the pixel is the chromakey colour, the location of said pixel is marked as PIXEL CHROMAKEY.

The MMAPP software also comprises the function 104 for generating a photogram or rendering function on the chromakey of the virtual shoe; said function 104 is associated with the functions 102 and 103 and for the processing of said function it is necessary to have the image 22. The three-dimensional model of the shoe is rendered after being positioned. The rendering is carried out in such a way as to replace only the pixels that have a PIXEL CHROMAKEY marking with the pixels of the image of the virtual shoe coming from the digital catalogue, the PIXEL CHROMAKEY marking being thus removed. The MMAPP software lastly comprises the function 105 for removing the chromakey; said function 105 is associated with the functions 100 and 104. The locations of coordinates xfb, yfb marked PIXEL CHROMAKEY, the pixels of which are not replaced during the phase of rendering the three- dimensional model of the virtual shoe, still have the marking. The pixels of these coordinates are replaced by the corresponding pixels of the floor image contained in the FLOOR image.

In the same function 105 there is also the removal of the image of the marker 10, as this appears in the photogram inasmuch as it is filmed by the camera 2 to obtain a better result to be displayed on screen. For this purpose, the position of the marker 10 being known, it is calculated which pixels of the photogram are occupied by the image of the marker. These pixels, and only these, are replaced using a similar procedure to that of removing the chromakey in excess, with the pixels of the image of the pavement contained in the FLOOR image. At this point the image that is thereby obtained is sent to the display 5 for displaying the virtual shoes on the feet of the user.

The MMAPP software is an application programme that is based on four libraries, as shown in figure 3: a library 20 for the interface with the camera 2 for managing the images coming from the camera, a library 30 for recognising the markers 10, a library 40 for enabling loading and management of the three-dimensional models of the virtual shoes, a library 50 for the three-dimensional graphics in real time.

Figure 4 shows the operating flow diagram of the apparatus in figure 1. From the camera 2 the images are sent to the computer 3 for the start of a new processing cycle 50 for displaying virtual shoes on the feet of a user.

The photogram 51 coming from the camera 2 is sent to the computer 3; at the arrival of the photogram 51 there is an enquiry 52 as to whether the FLOOR image has already been stored or not in the computer 3. If the response is affirmative (YES) the processing cycle is started up for displaying virtual shoes on the feet of a user, if the response is negative (NO) the FLOOR (block 53) image is stored that will be used in the next cycle, said image is displayed on the display (block 54) and a new cycle is started.

With the affirmative reply (YES) the processing cycle starts with the recognition function 101 of the marker 10 (block 55) and with the extraction function of the chromakey 103 (block 56). There is another enquiry (block

57) as to whether the marker 10 has been found or not; if the reply is negative (NO) the photogram is displayed on the screen 5 whereas if the reply is affirmative (YES) the calculation iunction 111 of the marker position 10 (block 58) and the function 102 for calculating the position of the virtual shoe (block 59) start up.

Subsequently, the function of rendering the virtual shoe 104 (block 60) starts up, replacing the extracted chromakey with the three-dimensional model of the virtual shoe chosen by the user (block 61). Said model comes from a previously loaded digital catalogue (block 62).

Subsequently, the function 105 is started up to remove the surplus chromakey and remove the image of the marker 10 (block 63) by means of the image coming from the block 60 and the FLOOR image.

Lastly, there is the display of the image thus obtained on the display 5 (block 54) for displaying the virtual shoes on the user's feet.

Claims

1. Apparatus for displaying virtual shoes on the feet of a user, characterised in that it includes a pair of special footwear items (1) suitable for being worn by the user, a camera (2) suitable for photographing the image of the user wearing said shoes (1), computer means (3) comprising a database containing a catalogue of virtual shoes and software (MMAPP) capable of processing the image coming from the camera (2) replacing said pair of special footwear items (1) with a pair of virtual shoes chosen by the user from said catalogue, said computer means (3) being suitable for providing in real time a further image (4) of the user wearing said chosen virtual shoes, a display (5) for displaying said further image (4).

2. Apparatus according to claim 1, characterised in that it includes recognition means (10) connected to said special footwear items (1), said computer means (3) comprising a database on the features of said recognition means (10), said software (MMAPP) recognising the orientation and position of the feet of the user by means of said recognition means (10).

3. Apparatus according to claim 2, characterised in that said recognition means (10) is made of the same material and has a predefined geometrical shape.

4. Apparatus according to claim 3, characterised in that said recognition means (10) are two, one for each of the two special footwear items (1), and each comprises graphic differentiation elements.

5. Apparatus according to claim 1, characterised in that it comprises means (6) suitable for illuminating the feet of the user.

6. Apparatus according to claim 1, characterised in that said special footwear items (1) are each made of the same material and are the same colour.

7. Apparatus according to claim 1, characterised in that said screen (5) is a helmet for virtual reality.

8. Apparatus according to claim 1, characterised in that said screen (5) is a monitor.

9. Apparatus according to claim 1, characterised in that said camera (2) is a digital camera.

10. Apparatus according to claim 2, characterised in that said application software (MMAPP) comprises the following functions: storing the initial image (FLOOR) of the frame of the place where a user will be present, said image being supplied by a camera (2), recognition means (10) in an image of the user wearing special footwear items (1) provided with said recognition means (10), the calculation of the position of said recognition means (10) and the extraction from the pixels of the image of the pixels that belong to the pixels of the image of the special footwear items (1), the calculation of the position of the recognition means (10) and positioning of a pair of virtual shoes chosen by the user from a catalogue of shoes contained in a database, the replacement of the pixels belonging to the image of the special footwear items (1) with the pixels of the image of the virtual shoes chosen by the user, removal of the pixels belonging to the image of the special footwear items that are in excess and the replacement thereof with the corresponding pixels of the initial image, removal of the pixels belonging to the image of the recognition means (10) and the replacement thereof with the corresponding pixels of the initial image.

11. Procedure for displaying virtual shoes on the feet of a user comprising the wearing of a pair of special footwear items (1) by a user, filming of the image of the user wearing said footwear items (1) by a camera (2), computer processing in real time of the image coming from the camera (2) replacing said pair of special footwear items (1) with a pair of virtual shoes chosen by the user from a catalogue of shoes contained in a database, the production in real time of a further image (4) of the user wearing said chosen virtual shoes and the display of said further image (4) on a display (5).

12. Procedure according to claim 10, characterised in that before wearing of a pair of special footwear items (1) by the user, the initial image of the frame of the place where the user will be is filmed and stored.

13. Procedure according to claim 10, characterised in that said special footwear items (1) are connected to recognition means (10) and that said computer processing of the image comprises: recognition of said recognition means, calculation of the position of said recognition means (10) and extraction from the pixels of the image of the pixels that belong to the pixels of the image of the special footwear items (1), positioning of the virtual shoes chosen by the user, the replacement of the pixels belonging to the image of the special footwear items (1) with the pixels of the image of the virtual shoes chosen by the user, removal of the pixels belonging to the image of the special shoes that are in excess and the replacement thereof with the corresponding pixels of the initial image, removal of the pixels belonging to the image of the recognition means (10) and the replacement thereof with the corresponding pixels of the initial image.

14. Computer comprising hardware devices and application software (MMAPP) comprising the following functions: storing the initial image (FLOOR) of the frame of the place where a user will be present, said image being provided by a camera (2), recognition of recognition means (10) in an image of the user wearing special footwear items (1) provided with said recognition means (10), calculation of the position of said recognition means (10) and extraction from the pixels of the image of the pixels that belong to the pixels of the image of the special footwear items (1), positioning of a pair of virtual shoes chosen by the user from a catalogue of shoes contained in a database, the replacement of the pixels belonging to the image of the special footwear items (1) with the pixels of the image of the virtual shoes chosen by the user, removal of the pixels belonging to the image of the special shoes that are in excess and the replacement thereof with the corresponding pixels of the initial image, removal of the pixels belonging to the image of the recognition means (10) and the replacement thereof with the corresponding pixels of the initial image.