WO2012158801A2

WO2012158801A2 - Augmented reality visualization system and method for cosmetic surgery

Info

Publication number: WO2012158801A2
Application number: PCT/US2012/038142
Authority: WO
Inventors: Kevin Roberts
Original assignee: Kevin Roberts, Inc.
Priority date: 2011-05-16
Filing date: 2012-05-16
Publication date: 2012-11-22

Abstract

Disclosed is an augmented reality visualization system and method for cosmetic surgery. A processor and computer-readable medium are configured to provide a user with the ability to virtually wear and manipulate alternative breast sizes and shapes so as to help visualize their optimal result as if looking in a mirror. With direct-to-consumer web and mobile platforms, as well as a more sophisticated embodiment for surgeons, the invention can help potential patients to more effectively decide on the elective surgery, direct a captive audience to the best suited surgeon, allow the surgeon to convert more patients based on examples and assist in achieving better results without as much guess work and uncertainty on behalf of the patient and surgeon. Using the disclosed system and methods, a doctor or patient can customize a breast enhancement, giving the doctor/patient a near to exact model of what the end result should be.

Description

AUGMENTED REALITY VISUALIZATION SYSTEM AND METHOD FOR COSMETIC

SURGERY

[0001] This application claims priority to U.S. Provisional Patent Application No. 61/486,674 filed May 16, 201 1, the entire disclosure of which is incorporated herein by reference. This application includes material which is subject to copyright protection. The copyright owner has no objection to the facsimile reproduction by anyone of the patent disclosure, as it appears in the Patent and Trademark Office files or records, but otherwise reserves all copyright rights whatsoever.

FIELD

[0002] The present invention relates in general to the field of augmented reality, and in particular to a novel visualization system and method that uses augmented reality to allow a surgical candidate to preview the effects of cosmetic surgery.

BACKGROUND

[0003] There currently exists some 3D visualization software, i.e. 3D breast augmentation simulation (www.loveyourlook.com and www.crisalix.com which features e-Stetix, a web based before/after visualization) for breast enhancement, none of which allow the user to visualize results in real time as if looking in the mirror. 3D Imaging Technology and software are known and available, including:

- Axis Three (www.axisthree.com)

- Vectra 3D Breast Sculptor Imaging Systems by Canfield Scientific - 3Dmd (www.3dmd.com)

- ModiFace, inc ( eShap )

- INAModel (www.lookingyourbest.com)

[0004] 3D Modeling Software is also known, including:

- www.ioveyourlook.com

- wvwv.crisaiix.com

- AMVIR

(www.auqmentationmammairevirtuelle.fr/software enq.php)

- www.natrelie.com (http://www.natrelle.com/simulator.aspx)

[0005] None of these solutions allow the end user to realize enhancements in augmented reality. SUMMARY

[0006] The invention in one embodiment provides a user with the ability to "wear/sample" and manipulate alternative breast size and shape in order to help visualize their optimal result as if looking in a mirror. With direct-to-consumer web and mobile platforms, as well as a more sophisticated embodiment for surgeons, this invention can help potential patients to more effectively decide on the elective surgery, direct a captive audience to the best suited surgeon, allow the surgeon to convert more patients based on examples and assist in achieving better results without as much guess work and uncertainty on behalf of the patient and surgeon. Using the disclosed system and methods, a doctor or patient can customize a breast enhancement, giving the doctor/patient a near to exact model of what the end result should be. This can be an invaluable sales tool in the hands of physicians wishing to convert more potential patients into clients.

DETAILED DESCRIPTION

[0007] The present invention is described below with reference to illustrations of a visualization system and method that uses augmented reality to allow a surgical candidate to preview the effects of cosmetic surgery. It is understood that the system and methods taught herein may be implemented by means of analog or digital hardware and computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, ASIC, or other programmable data processing apparatus, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, implements the functions/acts described. In some alternate implementations, the functions/acts may occur out of the order noted in these illustrations. For example, two steps shown in succession may in fact be executed substantially concurrently or the steps may sometimes be executed in the reverse order, depending upon the functionality/acts involved.

[0008] Three different platforms for this invention are described: an office based, non-web dependent version; an end-user (i.e. patient) web based version; and an end-user, web based mobile (hand-held, tablet, etc) version.

[0009] The form of a web or mobile-based application marketed directly to consumers would increase the confidence for women considering such a procedure and would thus increase the likelihood of them contacting a physician. The mobile app version can not only be viewed as a simple and logical entry point into the market for considering this type of procedure, but also provide a highly viral novelty component to the product.

OFFICE BASED

[0010] The office-based version can be located in the plastic surgeon's office. It would likely be the most robust of all three platforms. It can be licensed/customized to match the brand of that particular medical practice or individual surgeon.

[0011] The office-based platform can allow for the highest level of detail, precision and realism. With the assistance of the staff in creating a suitable model, it can present the most likely outcome for the patient following surgery. This platform may include more features than the web or mobile-based version to ensure the most accurate representation of the final result. The price point and subsequent support and training necessary would represent the premium offering of this software package. The desktop application can utilize the Microsoft Kinect (IR/RBG) camera.

WEB BASED

[0012] The web-based version typically requires only a web cam and an internet connection on a home/personal computer. In this respect, the web-based version may not be as robust as the in- office version, but can offer enough capability for the viewer to accurately sample varied looks in sizing and shape. In turn, the user may be enticed to locate a doctor that has the full office version. In general, this version can provide the functions of a virtual fitting room such as that provided by ImmediaC. This version could be licensed and skinned as part of the office-based package and but also can be used for its own marketing purposes.

[0013] This version can be used more as a tool to entice patients to look to the next step in visualizing their procedure. The data that the patient inputs into the website can be saved as a user profile and then imported and further enhance by a surgeon using the office-based platform. In addition, it can include viral components such as the ability take and share screen shot/photos and email them, thus capturing additional user data for marketing purposes.

[0014] The website application can utilize a high resolution web camera. The full feature set can be utilized with the web platform version of the application. The web platform can be self contained and can be run on a local system or internet connected computer.

MOBILE BASED

[0015] The mobile version can be configured and marketed in a variety of ways, with several targeted audiences. In order to operate effectively on a mobile, or tablet device, it may be necessary for it to have far fewer capabilities. Due to this lack of detail, it can represent as much of a novelty concept as would an actual usable profile for future augmentation. However, it can be used as a viral tool to steer potential patients to the more robust web version and thus the office-based platform as well.

[0016] This application can be configured so as to be usable on most major devices such an iPhone/ iPad and android based products.

[0017] The mobile the application can utilize a limited set of functionality. This is due to nature or the device's hardware limitation and usability of the device. For single-user operation, the system can provide two modes of operation depending on the user's specific mobile device. For devices with a front-facing camera, the user may hold the phone at arms length while capturing the torso in the field of view. If a user does not have a front-facing camera, then they will need to stand in front of a mirror while holding their device at arms length capturing their torso with the rear-facing camera. They can then see the device display in the mirror.

IMPLEMENTATION

[0018] Thus, the invention in various embodiments can provide an augmented reality breast enhancement visualization application for mobile, web, and desktop platforms. The application can use upper body recognition utilizing the body's geometry, skin color, and user-marked points to create an appropriate map to render breasts using augmented reality technology.

[0019] The system can use 3D animated mesh breast models. It can utilize high-polygon-count breast models for web and desktop/office implementations. Mobile platforms may require reduced complexity due to their limited GPU capabilities. For desktop/office applications, the model lighting will be customizable to match the intensity and direction of both diffuse and pointsource lighting in the doctor's office making for the best possible look and user experience.

[0020] The invention can utilize a physics engine in which the mass distribution and elasticity of the breasts are modeled to provide realistic movement when the user shakes side-to-side or jumps up and down. Algorithms originally intended for use in gaming engines can be used to simulate the realistic movement and appearance for fat and muscle tissue to enhance the realistic movement of the breasts.

[0021] The invention can be provided with computer vision functionality in which a variety of computer vision algorithms are used to identify and track the torso, breast outline, location of nipples and areola, and color match augmented breasts. These computer vision technologies include contour detection (via Canny edge detection and adaptive threshold), back projection of huesaturation-value histograms (identification by color), watershed algorithm (to assist other CV algorithms to segment out person from background and to help separate breasts from torso), and optical flow (for tracking and works with parallel with tracking by detection).

[0022] The invention may utilize a web app in which Adobe Flash is used as the software platform. This allows the user to use this application directly from the website without having to install a standalone software package. This can make it compatible with all desktop platforms that support Flash (i.e. Windows, Mac OS X, Linux).

[0023] Various versions of the mobile app can be provided. In an iOS (iPhone 3GS, iPhone 4, iPad 2) version, each of the newest Apple mobile devices can be supported natively in Objective - C and C++ for the highest possible performance. In an Android 2.1+ version, Android devices with cameras and Android 2.1 or newer can be supported (currently this includes over 94% of devices). Both Java and C++ can be used to maximize both performance and development efficiency.

[0024] The Microsoft Kinect camera can be utilized to provide the best possible experience and realism in the doctor's office. The Kinect camera uses infrared structured light to provide an instant 3D depth map. This allows the user an expanded range of movement without loss of tracking.

DEVELOPMENT TASKS

[0025] Development tasks for implementation of the invention will now be described.

[0026] Graphical assets: Graphics for the website, mobile application and 3D models need to be developed, including:

- 3D Meshes

- Several 3D breast meshes for common breast shapes

- Several nipple/areola shapes & sizes

- User interface graphics for mobile, web, and desktop/office.

- Entry/Exit/Help screens

- Shape/size/color selector

- Website

- Design and Layout

- Slicing

[0027] Upper Body Recognition: The application can utilize shape and contour matching to create a 'best guess' map of the body shape. The map can be visually overlaid on the users view port. This allows the user to confirm or manipulate the map to correct anything askew. The following functionality is preferably provided:

-Compute the best-fit line of bilateral symmetry in image.

-Attempt facial feature recognition if face is visible to identify location of eyes and mouth. These feature locations can be extrapolated to possible breast location to assist alignment and identification algorithms.

-Identify torso and shoulder curves to further constrain possible breast and nipple locations.

-Identify contours in upper torso (including breasts)

-Several algorithms will have independent estimates of breast locations.

Use Bayesian analysis to combine probabilities to best utilize all available information.

-Optionally have the user sweep their hands under and around sides of breast to top of shoulders to assist alignment if automatic detection algorithms fail or have a low confidence result.

[0028] Key Point Matching: After the body recognition has created a map the user will have the option of moving any points that may not be properly aligned. The user should be able to move individual point as well as an entire line.

[0029] Skin color chooser: After the key point map has been created and approved by the user, the application shall then display to the user a color chooser for skin and nipple colors. Possibly a still shot is taken of the breast and zoomed for color selection.

[0030] Implant/Cup Size: When color has been selected the users will then be given an implant size (CC/Cup) chooser. The user will be shown a preview of the sizes on their view port. This can also be a zoomed shot.

[0031] AR Rendering/Physics: After the user selects color and size they are shown the rendering of their AR breasts. So long as the user stays in alignment of the camera the breast will be rendered and as the user moves physics applied for as realistic of a preview as possible.

[0032] Email Screenshot: The user will either be able to take a frame from their preview or possibly an event may be remotely triggered to take a screenshot that may then to emailed from the device/website.

[0033] FIG. 1 shows a view of mobile embodiments of the disclosed system. [0034] While the invention has been particularly shown and described with reference to a preferred embodiment thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.

Claims

CLAIM

1. An augmented reality visualization system for cosmetic surgery, comprising:

A processor and computer-readable medium configured to provide a user with the ability to virtually wear and manipulate alternative breast sizes and shapes so as to help visualize their optimal result as if looking in a mirror.