WO2015044866A1

WO2015044866A1 - Virtual display and measurement system for spectacles on a real scene

Info

Publication number: WO2015044866A1
Application number: PCT/IB2014/064773
Authority: WO
Inventors: César SILVA
Original assignee: Silva César
Priority date: 2013-09-26
Filing date: 2014-09-23
Publication date: 2015-04-02
Also published as: WO2015044866A4; PT107198A

Abstract

The present invention relates to the technical domain of optics and ophthalmology. The present invention relates to a system and a method that enables a user not only to display virtual glasses on his own face, but also to simulate the effect of the lenses in said glasses on the scene being observed. To do so, a diverging stereo system comprising two chambers is proposed along with a method to calibrate the system, to reconstruct points on the face of the observer, enabling exact measurements, and to superimpose virtual objects on the real scene.

Description

VIRTUAL GLASS MEASURING AND VIEWING SYSTEM REAL SCENERY

Framework

The present invention is in the field of optics and optometry, and relates to the measurement and virtual viewing of eyeglasses using a camera electronics as support for eyewear e-commerce applications and the simulation of ophthalmic and optical lenses. sun protection lenses.

The present invention allows a user to respond to two problems when you want to choose remotely (for example via Internet) a particular model of glasses: (1) what the wearer looks like that particular pair of glasses, what does it look like when the glasses they are placed over your face; and (2) how glasses affect the vision of the what is the effect of the lens on the objects surrounding the user. The present invention proposes a divergent stereo system that captures simultaneously the observer and the observed scene. It also proposes a method that calibrates the system, rebuilds and measures points on the viewer's face, and allows the visualization of overlapping virtual objects over the actual scenario.

State of art

The Internet is a network that connects computers, businesses and people from around the world, providing a unique infrastructure for various aspects, such as logistics, communications, retail, transactions, payments, organizations, in the forms of work, etc. Specifically, e-commerce has growth in all business areas, including less conventional areas such as services or furniture retailing, clothing, shoes and glasses.

With the increasing processing capacity of computers and with the significant increase in bandwidth in the advanced solutions have emerged both in (through high resolution visual means) or in the experimentation of products (through processes that mix reality with virtual objects, generically referred to as Augmented Reality). However, online commerce products that are most dependent on consumer experimentation (such as clothing, footwear or glasses) is still residual compared to retail in physical store. This is partly due to the still limited means that the Internet offers to make the product experience satisfactory for part of the user.

The main solutions that currently improve the user experience in presenting the product online are, in order of complexity:

1 - Photographic presentation of the products (in format real or virtual) with high resolution and possibility of interaction (rotation and zoom);

2 - Product configurators, where the user you can change or configure online product parts, dimensions, materials or textures;

3 - Augmented reality by photography, where the user can try the virtual product over a photograph that can provided by the retailer or the user himself - in this case, usually the virtual product is placed / adjusted over the photograph with the help of user manual;

4 - Augmented reality by video, where the user you can try the virtual product in real time using a camera aimed at actual scene (which may be your living room, for furniture, or your face, for selling case) - in this case the overlay of the virtual product over the Reality is done automatically.

In the retail of glasses, many solutions have emerged in that uses video augmented reality, where glasses are put on virtual images on the user's face. However, no known solution in the Current state of the art has a global approach to the associated product and service. More specifically, no known solution integrates in the same system the user visualization with virtual glasses with the simulation of the effect of also allowing the measurement of optometric distances from the user. The present invention proposes to improve trading tools featuring an innovative global solution that allows you to simulate both the glasses on the observer want the scene observed. For this it uses a device with diverging cameras (front and rear). In the area of the invention, there are no known solutions that integrate and simultaneously use two divergent chambers for the same purpose or purpose, even though they are commercially available on common devices such as tablets or smartphones.

Summary of the Invention

The present invention is a system of measurement and visualization of virtual glasses on a real scenario involving the observer and the scene observed. It is proposed not only to simulate the general appearance of glasses on the face simulate the observed scene seen through the lens virtual.

The proposed system comprises a vision system divergent stereo (two-chamber) - which means that the fields of view are not intersect. Divergent stereo vision systems can be found in many different currently available devices such as tablets or smartphones whose cameras are in front of and behind the device.

In the present invention, the stereo vision system divergent captures the observer's face and the scene observed by the observer. Presenting these two views on the screen, the system offers a simultaneous perception of the observer and the scene observed.

This perception is enriched by the overlap of objects or virtual effects. In the case of the observer, glasses overlap virtual images over the face. In the case of the observed scene, one can overlap objects image effects to simulate the lens of glasses (ophthalmic or sun). For this overlap to be meticulously correct, it is necessary to know the metric relations between the elements of the vision system. To estimate the exact metric ratios of the cameras and the observer, it is necessary to convert the divergent stereo system into a convergent stereo (where the fields of vision of the two cameras are intersect). To do this a mirror is placed in front of one of the cameras, which allows you to deflect the projection rays that hit this camera. In the setting adopted convergent stereo vision system (1) it is possible to calibrate the (know the internal measurements of the chambers and the geometric relationship between them), (2) reconstruct three-dimensionally relevant points on the face of the such as pupil centers, thus allowing the calculation of the interpupillary distance, or (3) three-dimensionally reconstruct the contours of the spectacle lenses if the user has real glasses on his face.

Description of the Figures

FIG 1 shows the main steps that make up the method associated with the present invention.

FIG 2 shows divergent stereo system inserted in a real scenario composed of the observer and the observed scene. Also shows the movement made by the device to capture a sequence of images video.

FIG 3 shows the system in its stereo configuration convergent, including a mirror.

FIG 4 shows in geometric terms a shape simplified how the system works in its converged stereo configuration.

Figure 5 shows in geometrical terms how the calibration using a calibration device and its own points. device.

Figure 6 shows the images resulting from the capture of two cameras in the divergent stereo vision configuration, showing the observer and the observed scene, with overlapping virtual objects.

FIG 7 shows in geometric terms the measurements optometric parameters obtainable in the present invention.

Detailed Description

The present invention consists of a stereo system divergent method and method for simultaneously capturing an observer and the observed scene with the following features:

A) From the observer side: - obtain a video of the observer in motion; - get a video overlaying virtual glasses on the observer's face; - obtain precise measures from the observer, in particular the interpupillary distance;

B) From the observed scene side: - get a video of the observed scene (consistent with the movement of the observer); - get a video the observed scene in which virtual objects overlap; - simulate what the observer “sees” with virtual sun protection glasses (sunglasses); - simulate what the viewer “sees” with virtual correction glasses ophthalmic).

In the configuration proposed in this invention (Claims 1 and 2), the system consists of: a unit of processing; two camcorders - one front (201) (501), defined as the which points to the observer, and another rear (202) (502), defined as the one points to the observed scene - placed in opposite directions; a screen that displays the images captured by the cameras; a rotation estimation module system (using internal device sensors such as gyro, compass and accelerometer as claimed in Claim 4); and yet one calibration device which combined with a flat mirror (503) allows calibrate the system.

Calibration device and mirror are elements and interact with the system as follows:

- the calibration device, the realization of (Claim 3) is a planar grid (507) formed of black squares on a white background contain a set of dots previously measured and which are introduced into the calibration process, which is essential to know the geometric relationship between the two chambers;

- Mirror allows to convert divergent system in a conventional converged stereo system, making it possible not only calibrate the two chambers involved (since, with the help of the mirror, the chambers can capture the calibration device at the same time) as well as measure other points in space they capture simultaneously, such as points on the face of the observer.

The features described in the system are realized by a method comprising the following steps (Claim 5):

a) Calibration step (100):

To calibrate the stereo system converts the system into a converged stereo system by introducing a mirror (Claim 6). In FIG. 5 is a possible configuration of cameras C1 (501) and C2 (502), mirror S (503) and calibration device (507). For For purposes of explanation, an M point 505 known from the calibration device. The projection m1 of point M on camera image C1 and the projection m2 of the same point M in camera C2 are defined by the expressions: m1 = P1.M and m2 = P2.M where P1 and P2 are projection matrices of C1 and C2, and contain all the geometric information that defines the chambers, so knowing these matrices is considered the calibrated system.

The estimation of P1 and P2 corresponds to the resolution of a inverse problem when knowing m1, m2 and m. The solution uses tools known in the field of algebra and regression. For the estimation process converge and be robust, not just an M point (505) is used but a collection points taken from the surface of the calibration In this configuration are the corners of the black squares. To further increase For this collection of points, various calibration devices are used in different positions and inclinations.

You can also include points from your own device 500 as shown in FIG. 5 wherein the Q point 506 and the respective projection q (510) is an example. In this case, as the points of the captured by camera C2 only, contribute to the estimation of matrix P2 (Claim 7), using the same expression as above, where M is replaced by Q.

b) A point estimation step (101) on the observer:

Keeping the convergent stereo configuration of FIG. 3, it is possible to reconstruct and measure points of the observer's body (Claim 8).

Figure 4 shows geometrically how to reconstruct a three-dimensional point R (405) knowing the corresponding coordinates r1 and r2 projecting that point on images taken by cameras C1 and C2. Knowing r1, r2 and the relative position between the mirror S (403) and the optical centers of the cameras C1 and C2, you can draw the two projection lines starting from the optical centers and pass through r1 and r2 (where the straight line of C2 reflects in mirror S). By triangulation (intersection of two lines referred to), the point R is reconstructed in its three-dimensional coordinates.

For the method, it is important to reconstruct several points characteristic of the observer. Most relevant are the pupil centers (Claim 9).

However, other points may be rebuilt. relevant in the area of optometry, namely the contours of the lens (Claim 16), in case the observer wears real glasses over his face.

c) a step (102) of movement of the device of the system around the observer, where a front camera video image sequence and a video image sequence Rear camera:

In this step, the system is placed in its configuration divergent stereo (no mirror) as shown in FIG. 2. In this configuration, the observer (204) stands motionless looking at the observed scene (Claim 10) and proceeds to a movement (203) of the camera system whose The trajectory is preferably horizontal and rotary between -60 and 60 degrees, as set forth in Claim 11.

Fig.6 shows an example of images represented on the system screen. For the front camera image (602), the face (600) makes an apparent rotation movement between -60 and 60 degrees (604). For the rear camera image 603, the observed scene 601 has a certain relative motion in image 605 as shown in FIG 6.

Note that the videos are not intended to be in perfect conformity with the real world, because the observer is not truly following the actual scene in coherence with the movement of the cameras. In fact the observer is static relative to the observed scene throughout the entire video. However, the coherence between the apparent movement of the observer and the movement of the scene observed is accurate (in terms of speed giving the illusion that the observer follows with the gaze (and the head) to scene observed.

d) a step (103) of estimation of the rotation of device of said system, performed by the rotation estimation module said system, which in the preferred configuration uses accelerometer, gyroscope and compass. You can also use the image information to estimate the rotation from the follow-up of characteristic points of the images captured either by the front camera or the rear camera.

e) a step (104) in which virtual glasses overlap on the observer's face, taking into account the estimated device rotation in d) and points on the observer, estimated in b):

In FIG. 6 shows an image 602 with glasses virtual resources (606). Assuming that virtual glasses are known and their known three-dimensional structure, the glasses must be rotated movement (604) of the device at each moment of the video. Additionally, for each moment, the virtual glasses are scaled and positioned so that the projected lenses in the image are aligned or centered with respect to the center of the left (608) and right (607) pupils (Claim 12). To align the virtual glasses is necessary to detect in the image, for each instant of the video, the center of the pupils. This can be achieved using techniques of known image processing for eye and pupil detection.

The element overlay process applies (609) also for the observed scene (601), as presented in Claim 13, as the rotation of the video is known or can be fixed and follow visual elements in the observed scene. At the limit you can replace the scene observed by a virtual scenario generated entirely by computer or generated by from a photograph (by a technique known in computer graphics by skybox). This way you can simulate different scenarios for the same observer for the same trajectory.

f) An image display step (105) of the observer (602) and the observed scene (603):

The visualization of the two images obtained from the two simultaneously allows you to try different video virtual glasses and different scenarios, as explained in the previous step.

So for a particular pair of glasses with lenses sunscreen, with a particular transmittance function of the spectrum of the visible light F, you can try this pair of glasses on the viewer's face and simultaneously apply function F to image 603 to simulate the visual effect of the lens (Claim 14).

Similarly, for a particular pair of glasses with correction lenses (known as ophthalmic lenses) with a particular refracting function (considering the lens as a refractive medium with a known form), you can apply this function to a real or virtual scenario of simulate the visual effect of the lens (Claim 15). The result of application of the refraction function on objects with known location is an image in which objects (far or near the viewer) less or more in focus, depending on the function set for the lens.

On the other hand, having an accurate or approximate idea of the refractor function of the observer's own ocular system, one can apply this function to the observed scenario, simulating the observer's view. Per Finally, the observed scenario (real and virtual) can be simulated by combining the eyepiece with the ophthalmic lens.

In step b) of the presented Method, we estimate the pupil centers. In the same step, it is also proposed a process of estimating the lens contour if the viewer puts real glasses on the face. THE estimation of pupils and lens contours results in having some relevant measures in the area of optometry, as described below. then.

FIG. 7 shows the supporting geometry for obtaining the most relevant measures.

Some measurements are made on the front view of the face. The front view is the image taken when the viewer is looking facing the camera (formally the optical axis of the front camera is parallel to the optical axis of the observer). The measurements taken in the frontal view are: - interpupillary distance (700), which is the distance between the centers of the pupils (Claim 17); - the right height (701) consisting of the height between the center of the right pupil and the lower dimension (or horizontal line bottom) of the right lens contour (701); - the left height (702) that consists of the height between the center of the right pupil and the lower bottom horizontal line) of the right lens contour (702) (Claim 18); - the main measurements of lens contours such as the width of the lens (705), the minimum distance between the lenses (703) and the maximum height (704) (Claim 19);

Are there any measures that are not performed at front of the face because they require depth information. For To perform these measures, it is necessary to estimate the plan that best approximates the lens contour (Claim 20). This estimation is a problem of minimization solved by a known linear regression algorithm, such as minimization of mean square error. A plan is therefore generated for each wherein the plane 707 of FIG. 7 is an example of a plane for the lens right (706). The orientation of this plane (707) is a necessary angular measure the planning of ophthalmic lenses (namely progressive). Using the plan (707) It is still possible to obtain another relevant three-dimensional measure, vertex or the shortest distance (708) between the center of the pupil and the plane (707) (Claim 21).

The use of the system and method presented in The present invention is intended for any of the following applications (Claim 22): a.Media application for viewing lenses and virtual frames on the observer's face, b.Selection of lenses and frames best suited to adjust to the viewer, c.Simulation of the effect of corrective lenses and d. Obtaining optometric measurements of the observer's face, e.Construction or selection of bespoke lenses and frames.

Claims

Divergent stereo system for simultaneous capture of an observer and the observed scene characterized by :

a) a processing unit;

b) a front video camera (201) (501) and a video camera rear (202) (502), directed in opposite directions, rigidly coupled via a device (200) (500), and connected to said processing;

c) a device rotation estimation module (200) (500) connected to said processing unit;

d) a screen connected to said processing unit for presentation of images taken by said cameras;

e) a flat mirror (503) which, placed in front of said rear camera (502), allows for converged stereo vision configuration, wherein the field of view of said rear camera (502) intersects partially the field of view of said front camera (501);

(f) a calibration device which, in said configuration stereo vision system converging with said mirror (503) is placed on the same field of view of either said front camera or said camera rear.
System according to Claim 1, characterized in that said processing unit processes the images captured by said cameras and displays the processing result on said screen.
System according to Claim 1, characterized in that said calibration device has visual marks on its surface, the preferred embodiment of which is a planar grid (507) formed by black squares on a white background whose corners are identified and measured (505).
System according to Claim 1, characterized in that said device rotation estimation module comprises, but is not limited to, an accelerometer, a gyroscope and a compass.
Computer-implemented method for visualizing an observer with virtual glasses along with the observed scene applied to the system according to Claim 1, characterized in that it comprises the following steps:

a) a step (100) of calibration of said front cameras and rear of said system so as to know the geometric relationship between said chambers,

b) a step (101) of estimating points on the observer,

c) a step (102) of movement of the device of said system around the observer, which simultaneously captures a sequence front camera video frame and a camera video frame sequence rear,

d) a step (103) of estimating the rotation of the device of the estimated system by the rotation estimation module of said system,

e) a step (104) in which virtual glasses overlap over the observer's face, taking into account the device rotation estimated at (d) and observer points estimated in b),

f) a step (105) showing on the screen of said rear camera video and at the same time the video resulting from the step e).
Method according to Claim 5, characterized in that , in step a), the calibration may be performed for the stereo vision configuration converging with the mirror of said system, the calibration being performed by estimating the projection matrices P1 and P2. from the ratios m1 = P1.M and m2 = P2.M where each point M (505) is measured over the calibration device, where projections m1 (508) and m2 (509) are detected on camera images front and rear, corresponding to the projections of point M in the respective cameras, and where P1 and P2 are the projection matrices of camera C1 (501) and C2 (502) respectively.
Method according to Claim 5, characterized in that , in calibration step a), additionally known Q-points (506) may be used on the surface of the device for the purpose of calibrating the rear chamber (502) and the mirror of said system 503, knowing that the rear chamber geometry, defined by the projection matrix P2, can be estimated using the ratio q2 = P2.Q based on the Q points and their projections q2 (510) detected in the image.
Method according to Claim 5 and 6, characterized in that , in step b) of Claim 5, the three-dimensional estimation of an R point (405) on the observer is performed maintaining the calibration configuration of step a) so as to that the front C1 (401) and rear C2 (402) chambers of said system simultaneously capture point R (405), and the three-dimensional estimation of R is made by triangulating the projection lines from each chamber, based on the knowledge of the matrices P1 and P2 and the projection of the point R in said chambers.
Method according to Claim 5, characterized in that , in step b), two of the points on the observer's body are the centers of the two observer pupils.
Method according to Claim 5, characterized in that , in step c), the capture of said video is taken to a mirrorless configuration, where the viewer (204) is motionless in front of the observed scene (205), and wherein the observer moves the device 200 along a trajectory around its immobile face.
Method according to Claim 10, characterized in that a possible embodiment of the movement path of the device (200) is a rotary horizontal path (203), with the preferred variation of the rotation angle being between -60 degrees and 60 degrees. degrees.
Method according to Claim 5 and 9, characterized in that in step e) of Claim 5, the overlay is performed for each image of the video sequence captured in step c) of Claim 5, and consists of the overlap of three-dimensional virtual glasses ( 606) on the observer's face (600), whose lenses are centered and scaled according to the positions given by the projections of the pupil centers (607) and (608) on the front camera image, and wherein the rotation of said glasses (604) is estimated in step d) of Claim 5.
Method according to Claim 5, characterized in that , in step f) of viewing the observed scene (603), it can additionally overlay three-dimensional virtual elements (609) or pre-recorded scenarios over the observed scene.
Method according to Claim 5, characterized in that , in step f) of viewing the observed scene, the transmittance function characteristic of a given sunscreen lens can be additionally applied to the image (603).
Method according to Claim 5, characterized in that , in step f) of viewing the observed scene, different levels of focus on objects according to their distance, near or far, can be applied additionally on the observed scene (603), taking into account the refractive characteristics of a given ophthalmic lens and the eye characteristics of the observer.
Method according to Claim 5, characterized in that , in step b), the observer's points include the three-dimensional contours (706) of the frames and lenses, if the observer has placed real glasses over the face.
Method according to Claim 9, characterized in that , in step b) of Claim 5, it estimates the interpupillary distance (700), given by the distance between the centers of the pupils.
Method according to Claim 9 and 16, characterized by estimating the right and left height, respectively given by the distance (701) between the lower dimension of the right lens contour and the center of the right pupil, and by the distance (702) between the point lower left lens contour and the center of the left pupil.
Method according to Claim 16, characterized by estimating the width of the glasses (705) and the maximum height of the right and left lens (704), such as the minimum distance between the lenses (703).
Method according to Claim 16, characterized in that by linear regression estimating both the three-dimensional plane (707) that best approximates the contours of the right lens (706) and the three-dimensional plane that best approximates the contours of the left lens.
Method according to Claims 9 and 20, characterized in that , for each observer's eye, it estimates the vertex distance (708) consisting of the shortest distance from the center of the pupil to the plane (707).
Use of the system and method according to the preceding Claims characterized in that it is intended, but not limited to, any of the following applications:

a) Multimedia application for viewing lenses and virtual frames on the observer's face,

b) Selection of lenses and frames that best fit the observer,

c) Simulation of the effect of corrective lenses and sun protection,

(d) obtaining optometric measurements of the observer's face;

e) Custom construction or selection of lenses and frames.