BG112622A - Interactive system of computer assisted contact-free trying on of clothes and accessories by adapting the augmented reality toolbox and using depth camera system - Google Patents

Abstract

The main application of the proposed interactive computer assisted contact-free trying on of clothing and accessories by adapting the augmented reality toolbox and using a depth camera system is fundamental, but not limited to fashion, commerce, advertising and entertainment. The proposed system allows advertising of targeted products, including by way of customized advertising that allows the prospective customer to see what they would look like in the advertised item offered, as well as in a combination of one or more advertised and / or available items, as based on artificial intelligence the system offers the customer the best size, clothing model and colour corresponding to the specifics of their figure and desire. The proposed system also allows the purchase of the item liked by the customer , as well as its payment and delivery, including without intervention of an employee of the retailer and regardless of the time of selection and purchase, as well as the manufacturer or supplier.

Description

A declaration of license readiness has been submitted according to Art. 30 of the ZPRPM

Description of the invention

Name of the invention:

Interactive system for computer-assisted non-contact trying on of clothes and accessories by adapting the augmented reality toolkit and using a depth camera system

Field of technique:

Multimedia interactive device

Prior art:

Attempts to create computer-based systems to assist customers in purchasing clothing and accessories marked their development with the popularization of computer technology, and especially after the advent of color CRT monitors and graphical operating systems. Through the process of helping the customer choose, merchants ensure increased sales volumes and reduced returns, and customers receive a higher degree of satisfaction with their purchase.

The main approaches for the implementation of computer-assisted customer selection when buying clothes and accessories, and in particular in the visualization part of the selected product and how it would fit the specific customer, are mainly aimed at:

1. Visualization of a photo of a garment (with or without a mannequin) (Sources: http://www2.hm.com/be bg/index.html, http://www.explosi.bg/)

2. A preview of a photo of a garment superimposed as a still position and combined with a video stream from a camera or a photo of the face, where the customer "fits" on the photo to see how it will look (Source: https://www.youtube.com /watch?v=5TZmQPdhpak, http://zugara.com/virtualdressing-room-technology/virtual-dressing-room-kiosk, https://www.youtube.com/watch?v=L9EwtGUTNzO )

3. Uploading a user photo where the system morphs the body parts to replace the head and visible parts of an existing mannequin photo

4. Visualization of ZD model of garment with option to view/rotate by the client (Source: https://www.magictoolbox.com/magic360/)

5. Building a type of computer-generated avatar that "dresses" in computer-generated clothes - entirely in a computer environment (Source: https://www.youtube.com/watch?v=ZDJsZla9SYQ, https://www.youtube. com/watch?y=mWQ InW-Zvs)

6. Building a customized (based on body type) computer-generated avatar that is "dressed" in computer-generated clothes - entirely in a computer environment (source: http://www.trimirror.com/en/about/)

7. Building a customized (with actual measured or entered customer data) computer-generated avatar that is "dressed" with computer-generated clothes - entirely in a computer environment (Source: https://www.youtube.com/watch?v= pMladTwgHoI)

8. Generating and overlaying a computer-generated garment on a video stream from a customer image camera based on a silhouette of the customer's body (in a controlled environment) (Source: https://www.youtube.com/watch?v ^:: =BD4TCfECxsU&feature =youtu.be)

9. Generating and overlaying a computer-generated garment on a camera video stream with a customer image based on a virtual skeleton of the customer obtained from depth cameras (in a controlled environment) (Source: https://www.youtube.com/watch?v= ljbvnklT4vQ)

10. Generating and overlaying a computer-generated garment on a video stream from a client image camera based on a silhouette of the client's body (in an uncontrolled environment) (Source: https://www.youtube.com/watch?y=u a- (iFzOFgO)

11. Generating and overlaying a computer-generated garment on a camera video stream with a customer image based on a virtual skeleton of the customer obtained from depth cameras (in an uncontrolled environment) (source: https://www.youtube.com/watch?v= gz5ynrfJSDM, https://www.youtube.com/watch7vHB2jutoT4Eo, https://\v\v\v.voutiibe.com/\vatch?v=h3vP3hdwf61)

And although there are a number of commercial solutions based on the mentioned approaches, at the moment none of them solves the problem of the realism of the depicted garment, so that the customer has the feeling that he sees himself with a real garment on him.

The main problems of the existing solutions are related to the fact that the customer cannot be seen from all sides (solutions related to approach 1, 2 and 3), the simulation leaves a feeling of a computer game rather than a real trying on clothes (solutions, related to approach 4, 5, 6 and 7), or the presence of a significant discrepancy between the customer's movement and the computer-generated garment (decisions related to approach 8, 9, 10 and 11), as well as the lack of realism of the behavior of the garment, according to the specifics of the specific figure of the client (decisions related to approach 1, 2, 3, 4, 5, 8, 9, 10 and 11).

Another key aspect of the problem is related to the consideration of the real physical data of the client, with a view to the correct visualization of the computer-generated garment or accessory, incl. and to determine the appropriate size garment or accessory. The main approaches to measuring the physical dimensions of the client are related to the use of:

1. Physical means for assisted or independent contact measurement

2. Means for calculating estimated sizes based on a two-dimensional photographic image (photo)

3. Means for non-contact measurement of body dimensions by creating a ZD digital model

Despite advances in non-contact measurement technology, the most accurate remain the so-called contact measurements, since alternative methods are characterized by a certain level of mathematical approximation, extrapolation and modeling, as well as by the presence of measurement errors due to the specific technology used.

As for measurement by BP modeling, the main approaches are related to the use of:

1. Photogrammetric methods based on high-quality photographic images

2. Create a ZD depth map by using one or more depth sensors (RGB-D type, ToF or other applicable technology)

In order to achieve high accuracy of the created ZD model, the indicated approaches require the use of multiple sensors or cameras, making a series of multiple photos processed with appropriate algorithms to create relatively accurate ZD models. This approach usually requires a certain period of time and does not allow the creation and tracking of a person's BP model in real time.

Technical essence of the invention:

The presented interactive system for computer-assisted non-contact try-on of clothes and accessories by adapting the augmented reality toolkit and using a depth camera system solves a number of technical problems that have long faced those working in the field and helps to satisfy a long-standing need caused by previous approaches and solutions related to the construction of computer-assisted systems for contactless trying on of clothes and accessories.

Although in its complexity there is no analogous solution to the proposed interactive system for computer-assisted non-contact trying on of clothes and accessories by adapting the instrumentation of augmented reality and using a system of depth cameras, the closest solutions and components related to individual aspects of the system are as follows:

1. Regarding the approach to visualizing the selected product and how it would fit the specific customer, as part of the systems for implementing computer-aided customer selection in the purchase of clothing and accessories, the closest solutions are related to the generation and overlay of a computer-generated garment on a video stream from a camera with an image of the customer based on a virtual skeleton of the customer obtained from depth cameras (in an uncontrolled environment). The main problem of the existing solutions is that they mainly use stand-alone depth sensors (eg MS Kinect or similar) which are created and optimized for the purposes of computer game consoles (eg MS Xbox) and are therefore characterized by a specific range of action (0.80 pt. - 4.00 pt. Source:

https://blogs.msdn.microsoft.com/kinectforwindows/2012/0 l/20/near-mode-what-it-isand-isnt/), limited angle of capture of objects (43° vertical and 57° horizontal - source: https://msdn.microsoft.com/en-us/library/iil31033.aspx), which reflects in extremely high values of error in the measurement along the Z coordinate with increasing distance (source: Maimone, A., Fuchs, F. Encumbrance-free tele-presence system with real-time 3D capture and display using commodity depth cameras.

https://www.researchgate.net/publication/221221196 Encumbrance-free telepresence system with real-time 3D capture and display using commodity depth cameras fig. 5). It should be noted that although in practice there are also solutions that use depth cameras located above the floor (on the ceiling), these systems are again characterized by filming in a single plane, and their use is aimed at security activity in closed spaces (source: Gasparrini, S., Cippitelli, E., Spinsante, S., Gambi, E. A Depth-Based Fall Detection System Using a Kinect® Sensor.

https://www.ncbi.nlm.nih.gov/pmc/articles/PMC3958279/)·

In this regard, the proposed interactive system for computer-assisted contactless trying on clothes and accessories by adapting the augmented reality toolkit and using a depth camera system introduces a new approach combining a depth camera system, at least one of which is located above the floor (on the ceiling), which allows filming a minimum of two planes, one of which must be a top view and serves as a compensator for the Z coordinate of the frontal depth camera. Adopting such an approach allows to apply a specific compensatory approach, which allows, through a system of specific parameters, to compensate for the allowed errors in the measurement of depth cameras in one of the dimensions (mainly Z), which in turn allows the construction of an accurate ZD model and virtual skeleton of the captured subject.

2. In terms of non-contact body measurement, the closest solution to the proposed interactive system for computer-assisted non-contact trying on clothes and accessories by adapting the augmented reality toolkit and using a depth camera system is the use of non-contact measurement tools of the dimensions of the body by creating a HR digital model through HR scanning methods. Typically, this approach is characterized by the use of a large number of optical (source: http://www.aniwaa.com/3d-body-scanning/) and depth cameras (source: https://www.artec3d.com/3d-scanner /shapifybooth), laser measurements (source: https://www.youtube.com/watch?v ^:: = AzBlzxU grg) or using a combination of sensors and a rotating platform (source: http://www.styku.com /). In addition, the proposed approach requires time (min. 10 seconds) to create the model and perform the measurements (source: https://www.artec3d.com/3d-scanner/shapifybooth)·

In this regard, the proposed interactive system for computer-assisted non-contact trying on clothes and accessories by adapting the instrumentation of augmented reality and using a system of depth cameras introduces a new approach, building on the approach taken by Microsoft to create a virtual skeleton (source: https:/ /msdn.microsoft.com/enus/1 ibrary/jj 131025 .aspx) as by using a system of depth cameras, with at least one of them located above the floor (on the ceiling), errors in depth measurement and subsequent generation of t .name "virtual skeleton". Unlike similar non-contact measurement solutions, the construction of a virtual skeleton using a system of depth cameras, at least one of which is located above the floor (on the ceiling), allows its generation and tracking in real time (up to 30 frames in second - source: https://i.stack.imgur.com/OQiI4.png https://stackoverilow.com/questions/12978415/all-frames-from-kinect-at-30fps), which in turn allows for of anthropometric measurement of established anthropometric ratios between key points in the skeletal structure and key points of the physical body, taking into account the contour measurements of the depth cameras along the X and Y axes.

The technical essence of the proposed interactive system for computer-assisted contactless trying on of clothes and accessories by adapting the augmented reality toolkit and using a depth camera system is concluded in a visualization of a computer-generated virtual garment superimposed on an input high-resolution video stream, as the generated layer of digital information is created by using an adapted toolkit of augmented reality and based on a customized (with non-contact measured data of the client) computer-generated invisible avatar, which is "dressed" in computer-generated clothes, taking into account the specificity of matter and its behavior, as the fitting of the computer generated layers, incl. and augmented reality, is based on a unified virtual skeleton obtained from multiple depth cameras, with at least one of them located above the floor (the virtual skeleton also serves as an augmented reality marker).

The interactive system is composed of at least (1) a camera for high-quality video recording of the space in front of the interactive installation at the level of human vision, (2) a device for visualizing the captured space in front of the interactive installation, in dimensions close to the real dimensions of the objects when preserved of the real ratio between their constituent parts, (3) a system of at least 2 depth cameras, with (3.1) at least one of them located near the camera for high-quality video recording of the space in front of the interactive installation and pointing in the same direction of capture like it, and (3.2) at least one of them is positioned to allow capture of a target area from a top view, (4) a computer system for processing the incoming streams of information from the cameras, rendering the ZD model of a selected or selected by the client's items, their merging into a single video stream for viewing on the connected viewing device, such as rendering is performed in real time or with no-more than 0.033 second delay. The system of at least two depth cameras, at least one of which is located near the camera for high-quality video recording of the space in front of the interactive installation and is directed in the same direction of recording as it, and at least one of them is located so that to allow capturing a target area with a top view, by applying a specially developed algorithm for the purpose, allowing obtaining the real position of key points of the human anatomy by eliminating measurement errors with depth cameras, and allowing non-contact measurement of various visually visible parts of human anatomy, including to determine the dimensional data of the client. These key points of the human anatomy are also used as specialized markers for coordinating and visualizing computer-generated objects, including but not limited to augmented, virtual, mixed, and other types of reality created using local, distributed or remote computing power, regardless of the type of computing device (CPU, GPU, or other applicable), the technology used (microprocessor, nanotechnology, or other applicable), and the architecture of the computing device itself (RISC, CISC, or other applicable).

At the same time, the offered interactive system for computer-assisted contactless trying on of clothes and accessories by adapting the augmented reality toolkit and using a depth camera system, allows combining multi-layered video streams and computer-generated images, incl. and transparent ones, allowing the visualization of the realistic behavior of selected clothes and accessories on the real figure of the specific user, which is also realized by creating a customized transparent computer avatar, by adapting a basic avatar built on statistical anthropometric information, by non-contact reading of the form and key dimensions of the real human body, incl. and key dimensional data, by using the interactive installation component presented in claim 2.

Examples of implementation of the invention:

The proposed interactive system for computer-assisted contactless trying on of clothes and accessories by adapting the augmented reality toolkit and using a depth camera system is a complex component system that includes at least (1) camera for high-quality video recording of the space in front of the interactive installation at the level of human vision, (2) a device for visualizing the captured space in front of the interactive installation, in dimensions close to the real dimensions of the objects while preserving the real ratio between their constituent parts, (3) a system of at least 2 depth cameras, such as ( 3.1) at least one of them is located near the camera for high-quality video recording of the space in front of the interactive installation and is directed in the same direction of filming as it, and (3.2) at least one of them is located so as to allow filming of a top-view target area, (4) a computer system for processing the incoming streams of information from the cameras, rendering the PD model of a customer-selected item or items, combining them into a single video stream for visualization on the connected visualization device, the visualization being performed in real time or with a delay of no more than 0.033 seconds .

Application (use) of the invention:

The proposed interactive system for computer-assisted non-contact trying on of clothes and accessories by adapting the instrumentation of augmented reality and using a system of depth cameras can be built mainly, but not only, in the form of a free-standing installation, and its individual components can to be built into both the main body of the installation and the room where it is located. The use of standard components of general purpose and/or specialized application, which are produced by more than one manufacturer and are available for free retail sale, enable the seamless construction and operation in an uncontrolled environment of the proposed interactive computer-assisted non-contact clothing fitting system and accessories by adapting the augmented reality toolkit and using a depth camera system.

The main application of the proposed interactive system for computer-assisted contactless trying on of clothes and accessories by adapting the augmented reality toolkit and using a depth camera system is mainly, but not exclusively, in the fields of fashion, commerce, advertising and entertainment. The proposed system can be deployed as part of a store network or in the form of a free-standing installation outside a physical store, and its main purpose is to advertise certain target products, incl. and through the method of customized advertising, allowing the potential customer to see how he would look in the respective advertised item, as well as in a combination of one or more advertised and/or available items, the visualization being carried out through a photorealistic image in real time. Based on the available artificial intelligence, the system offers the customer the best size garment that meets the specifics of his figure, and also allows a visualization of how he would look when choosing a different size from the offered items. The proposed system also allows the purchase of an item liked by the user, as well as its payment and delivery, incl. and without the intervention of an employee from the commercial outlet and regardless of the time of making the selection and purchase.

Claims (5)

Patent claims Name of the invention; Interactive system for computer-assisted contactless trying on of clothes and accessories by adapting the augmented reality toolkit and using a depth camera system Patent claims; 1. An interactive system composed of at least (1) a camera for high-quality video recording of the space in front of the interactive installation at the level of human vision, (2) a device for visualizing the captured space in front of the interactive installation, in dimensions close to the real dimensions of the objects while preserving the real ratio between their constituent parts, (3) a system of at least 2 depth cameras, with (3.1) at least one of them located near the camera for high-quality video recording of the space in front of the interactive installation and directed in the same shooting direction like it, and (3.2) at least one of them is positioned to allow top-view imaging of a target area, (4) a computer system for processing the incoming streams of information from the cameras, rendering the 3D model of a selected or customer-selected items, combining them into a single video stream for viewing on the connected viewing device, such as o rendering is done in real time or is no-more than 0.033 second delay 2. A system of at least two depth cameras, at least one of which is located close to the camera for high-quality video recording of the space in front of the interactive installation and is directed in the same direction of recording as it, and at least one of them is located so , that allows capturing a target area with a view from above, which, by applying a specially developed algorithm for the purpose, allows obtaining the real position of key points of the human anatomy by eliminating the measurement errors of the depth cameras, and allowing non-contact measurement of various visual visible parts of the human anatomy, including to determine the dimensional data of the client 3. Using key points of the human anatomy as specialized markers to coordinate and visualize computer-generated objects, including but not limited to augmented, virtual, mixed and other types of reality created using devices with local, distributed or remote computing power 4. Combining multi-layered video streams and computer-generated images, incl. and transparent ones, allowing the visualization of the realistic behavior of selected clothes and accessories on the real figure of the specific user. 5. Creation of a customized computer avatar, by adapting a basic avatar built on statistical anthropometric information, by non-contact reading of the shape and key dimensions of the real human body, incl. and key dimensional data, by using the interactive installation component presented in claim 2