RU2793157C2 - Devices, systems, and methods for capture and display of external appearance - Google Patents

Abstract

FIELD: image processing.

SUBSTANCE: invention relates to technologies of reception and display of images. A method is proposed for operation of a system containing a monitor, a camera, and a processor in such a way that an image imitating a reflection in a mirror is displayed on the monitor. The method is implemented by performing in any order following stages: reception of a digital image from the camera, determination of a distance from a user to the camera, application to the digital image of a transformation algorithm to combine a camera observation point with a theoretical mirror observation point. Next, an image size is changed to weaken changes caused by changes in the distance from the object to the camera. Eye pixels are transformed or replaced so that to locate irises and pupils in the center of eyes looking forward. The image is displayed on the monitor after the image size is changed, and eye pixels are transformed or replaced.

EFFECT: provision of natural image orientation for an image displayed on a monitor.

19 cl, 21 dwg

Description

This application claims priority over US 13/843,001, filed March 15, 2013, and provisional US 61/738,957, filed December 18, 2012. The entire content of these applications is incorporated into this application by reference.

Technical field

The invention relates in general to systems for obtaining and displaying images, and more specifically to monitors and interactive displays, for example, in commerce and/or in the service sector, in medicine, in everyday life, for videoconferencing, for games, etc. Special implementations relate to the implementation of a flat display that performs the function of a mirror. Another special implementation relates to the implementation of a flat display that provides a video image of a person looking directly into the eyes during a video conference.

State of the art

Customers may purchase consumer goods such as clothing, such as shirts, trousers, jackets, and other garments, as well as shoes, glasses, and/or other items such as cosmetics, furniture, and the like. The purchase of goods is usually carried out in commercial establishments, for example, in retail stores. Before making a purchase decision, the customer may want to try them on, for example, garments, and / or imagine himself against the background of other products, such as furniture, and each time he needs to see how it looks, in a mirror, which can be installed, for example, in the fitting room of a retail store. For example, a customer may try on the first item, such as a suit, and look in the mirror to see how they look in that suit. The customer can then try on a second item, such as a different suit. The customer may then need to memorize how he looks in the first case, in order to mentally compare how he looks in the first and second suits, and evaluate which item suits him best (fits better).

Unfortunately, because the customer may try on multiple items, and/or because there may be a long time between trying on the second item and the first item, or because the items may be tried on in different stores, the customer may have little memory of what they looked like during the first fitting, and therefore you may need to go back and try on the first piece of clothing again. Naturally, this can annoy customers, and sales efficiency is reduced.

A traditional mirror (i.e., a reflective surface) is the familiar and most reliable means for a person to view their real appearance in real time. Various alternative means have been proposed, which are combinations of a television camera and a screen that can replace a conventional mirror. However, these technologies have not become commonplace and are not yet accepted as a reliable means of depicting a person, which works in exactly the same way as a conventional mirror. This is due to the fact that the image formed by the camera is very different from the image in the mirror.

When a person looks at himself in a mirror, he sees his reflection, which is at a distance from him that is twice the distance from the mirror. This is illustrated in figure 5A, in which the user, standing at a distance D1 from the mirror, sees himself at a distance of 2 x D1. Similarly, as shown in Figure 5B, a user standing at a distance D2 from the mirror will see himself at a distance of 2 x D2. In addition, the field of view (FOV) of the user changes as the distance changes, such as when the user moves closer to the mirror. The field of view is limited by the angle (β) of specular reflection from the mirror between the user's eye and the edge of the visible image on all sides of the mirror (four sides for a rectangular or square mirror). In Figure 5B, the bottom line of the vertical field of view is shown at a double angle (β) formed by lines connecting the user's eyes to the bottom edge of the mirror and reflected onto the user's shoes. Accordingly, as shown in Figure 5B, as the user approaches the mirror, the field of view increases (FOV1 < FOV2) and the user continues to see the same size reflection, so that the user actually sees himself at about the same size, but closer. This is a significant difference from the TV camera, that is, when the user comes closer to the TV camera, the size of the user appears larger. This is mainly because the camera's field of view is fixed and is determined primarily by the dimensions of the camera's lens or focal length.

Other effects associated with reflection in a mirror should also be noted. For example, when a user approaches a mirror, the reflection of their eyes will always travel from the mirror along the same virtual line. In the case of a camera, depending on its height, as the user gets closer to the camera, their eyes may appear to be at different heights. Another difference from the camera is that when the user looks in the mirror, their image is inverted (for example, if you raise your right hand, then the image in the mirror raises your left hand). However, a mirror does not swap left and right sides any more than it swaps top and bottom. The mirror reverses forward-backward (that is, what is in front of the mirror appears to be behind it), and we define left-right directions relative to the forward-backward directions. In addition, since the image in the mirror is virtual, the mirror may be smaller than the human body, and the user will still see his full reflection in it. This is because specular reflection (in Figure 5A, angle β of incidence is equal to angle β of reflection) can increase the effective field of view as the user approaches the mirror. In addition, although the mirror is a two-dimensional object, the user sees his own three-dimensional image in it.

At least some of the above reasons have so far prevented the creation of a system that would completely replace the mirror. A system that mimics a mirror could have a wide variety of applications in retail and other areas, opening up the possibility of combining real, virtual sensations, such as in social networks and other mobile technologies.

The essence of the invention

Some exemplary embodiments of the invention include devices, systems, and/or methods that enable comparison in appearance.

In accordance with some illustrative embodiments of the invention, a system that provides the ability to compare appearance, may contain at least one interactive station for obtaining and displaying images. The station may include, for example: a "mirror-display" device that can selectively operate in mirror and/or display mode; an image acquisition device for capturing one or more images of external views in a field of view in front of the mirror-display device; and/or an image control unit for setting the operating mode of the "mirror-display" device at the user's command. The "mirror-display" device may be in the form of a flat-panel TV, and when operating in mirror mode, the TV shows the converted images of the video stream received from the camera in real time, and in the display mode it shows the converted images of the video stream received earlier and retrieved from the storage device .

In accordance with some exemplary embodiments of the invention, the image control unit may include an input device for receiving user commands.

In accordance with some exemplary embodiments of the invention, the image control unit may include a data storage device for storing information of one or more images, which may correspond to one or more appearances.

In accordance with some exemplary embodiments of the invention, the mirror-display device may be divided into at least a first and a second portion, on which images may be simultaneously displayed. The first part can selectively work both in mirror mode and in display mode. The second part can work, for example, in mirror mode.

In accordance with some exemplary embodiments of the invention, an imaging device may be capable of capturing 3D appearance images.

In accordance with some exemplary embodiments of the invention, a mirror-display device may be capable of displaying images of appearances in predetermined sequences.

In accordance with some illustrative embodiments of the invention, the image control unit may allow the user to access images of the appearances allowed to this user, for example, based on identification data obtained from the user.

In accordance with some exemplary embodiments of the invention, said at least one interactive image acquisition and display system may comprise two or more interactive image acquisition and display stations capable of exchanging information over a network. For example, two or more stations may exchange information representing appearance images with each other.

In accordance with some exemplary embodiments of the invention, the image control unit may control the mirror-display device to display, for example, in display mode, one or more images corresponding to appearances. One or more images may contain, for example, one or more mirror images of appearances. Mirror images of appearances are obtained by converting images or video stream received from a camera to form images and video stream which, when displayed on a monitor, correspond to the reflection of the appearance in a mirror.

In accordance with some illustrative embodiments of the invention, a method for enabling appearance comparison may include: using the operation of a mirror-display device capable of selectively operating in a mirror mode or a display mode in a mirror mode; capturing an image corresponding to the appearance during the first fitting in front of the "mirror-display" device; recording the image of the first fitting for storage; setting the operation of the "mirror-display" device in the display mode; and/or reading the image of the first fitting and displaying it on the device "mirror-display".

In other embodiments, methods and apparatuses are provided that use cameras and a flat panel display to form a lifelike mirror image of an appearance.

Brief description of the drawings

[0021] The object of the present invention is defined and clearly stated in the final part of the application. However, the invention as regards organization and mode of operation, together with its features and merits, will be best understood from the following detailed description, together with the accompanying drawings, which show:

Figure 1 is a schematic view of an interactive system that provides the ability to compare the appearance, in accordance with some illustrative embodiments of the invention.

Figures 2A and 2B are schematic illustrations of two successive stages of appearance comparison using an interactive system in accordance with some exemplary embodiments of the invention.

Figures 3A, 3B, and 3C are schematic illustrations of three successive steps in appearance comparison using an interactive system in accordance with some illustrative embodiments of the invention.

Figure 4 is a flowchart of a method for enabling one or more user appearances to be compared, in accordance with some exemplary embodiments of the invention.

Figures 5A and 5B are diagrams of the reflection of the user in the mirror.

Figure 6 is a view of a variant with a single camera or multiple cameras located vertically at the level of the user's eyes to ensure a low level of image distortion.

Figure 7 is a schematic view illustrating the user approaching or moving away from the mirror using a camera mounted at the top of the screen and pointing horizontally.

Figure 8 is a schematic view of a system in which the camera is mounted above the monitor screen and tilted down.

Figure 9 is a block diagram of an embodiment of the system of the present invention in which transformation is performed to obtain an image simulating a reflection in a mirror.

Figures 10A and 10B are schematic views illustrating a calibration process in accordance with embodiments of the invention.

Figure 11 is a block diagram of the method according to one of the embodiments of the invention.

Figure 12 is a block diagram illustrating the modules and processes used to perform image calibration and transformation according to one embodiment of the invention.

Figure 13 is a diagram of another option, in which the calibration and the formation of the conversion algorithm are carried out on site after the installation of the system.

Figure 14 is a flowchart of a method for extracting data from an image captured by a camera.

Figure 15 is a block diagram of a variant in which the connection ("stitching") of images obtained from n cameras is carried out.

Figure 16 is a diagram of an embodiment that provides an accurate image of the eyes to simulate the user looking directly into his own eyes (as in a mirror).

It should be borne in mind that for the sake of simplicity and clarity of illustration of the invention, the elements in the figures may not be reproduced exactly or not to scale. For example, the dimensions of some elements may be disproportionately enlarged relative to other elements for clarity or to show multiple physical components included in one element. In addition, where appropriate, the same reference numerals may be used throughout the various figures to indicate corresponding or similar elements. It should be understood that the accompanying figures represent examples of the present invention and should not be construed as limiting the scope of the invention.

Detailed description of the invention

In the following description, various aspects of the present invention will be discussed. For the purposes of a detailed description, specific configurations and their details are given in order to provide a comprehensive understanding of the present invention. However, the person skilled in the art will be clear that the present invention can be implemented without some of these specific details. In addition, certain features of the invention, based on principles and implementations known in the art, may be omitted or simplified so as not to unnecessarily clutter the description.

Some illustrative embodiments of the invention may include an interactive system that allows the user to compare one or more of its appearances, for example, comparing different appearances, as will be described below.

The term "appearance of the user", as used in this description, refers to the appearance of the buyer, who tries on some thing. Such a thing can be, for example, a piece of clothing, shoes, accessories, glasses, ties, etc., a product, such as furniture, located near the buyer, as well as other products, goods or products, such as, for example, cosmetics, hats , hairstyle etc. . Likewise, some embodiments of the invention may be used for technical purposes, such as in fashion design where different versions of a pattern need to be viewed, and other embodiments may be used by users in their homes to create albums containing pictures of family members.

In accordance with some illustrative embodiments of the invention, the system may include an image acquisition device capable of capturing images of the user's appearance, and a mirror-display device capable of selectively operating in mirror mode or display mode. When operating in mirror mode, the "mirror-display" device may allow the user to evaluate and/or view its appearance during the trying on of the product in real time. As will be described in detail below, this is done by displaying a real-time video stream on a flat screen display, and before displaying the video stream on the display screen, it is converted to simulate an image in a mirror. When operating in display mode, the mirror-display device may allow the user to evaluate and/or view one or more of its appearances, for example, previously acquired by the imaging device during a previous try-on, for example, as will be described in detail below.

Figure 1 is a block diagram of an interactive system 100 in accordance with some exemplary embodiments of the invention.

In accordance with these embodiments, system 100 may include an interactive image acquisition and display station 110, which may include an image control unit 120, an image acquisition device 130 (e.g., a photo or video camera), and a "mirror-display" device 140 ( e.g. flat screen display). The image control unit 120 may include a controller 121, a network interface 122, a storage device 123, and an input device 124. All of these devices will be described in detail below.

Aspects of the present invention are described herein in the context of exemplary embodiments of an imaging device, such as an imaging device 130, a mirror-display device, such as a mirror-display device 140, and/or an image control unit, such as an image control unit 120. , which are separate components of an appearance comparison system, such as system 100. However, those skilled in the art will appreciate that the invention is not limited to such an embodiment, and that, in accordance with other embodiments of the invention, the system may be any suitable configuration, combination and/or organizing an image acquisition device, a mirror-display device, and/or an image control unit. For example, the system may comprise a single module including a mirror-display device, an image acquisition device, and/or an image control unit. For example, the image acquisition device and/or the image control unit may be implemented as an integral part of the mirror-display device.

In accordance with some exemplary embodiments of the invention, the mirror-display device 140 may be configured and/or may include components and mechanisms that allow the device 140 to operate selectively in two modes. In the first mode of operation, the mirror-display device 140 can operate as a mirror. In the second mode of operation, the mirror-display device 140 can operate as a display. When the mirror-display device 140 operates in mirror mode, the user 111 of the system 110 can evaluate and/or view their appearance when trying on an item for the first time, in which case the device 140 operates in real time. The image acquisition device 130 can capture an image of the user's appearance during the first fitting process. The resulting image may be recorded for storage in the data storage device 123, for example, as will be described below. The user 111 can then stand in front of the mirror display device 140 while trying on the second thing, and the image acquisition device 130 can capture the second image of the user's appearance during the second trying on. The user may be able to view the fitting of the second item in real time using the mirror-display device 140 set to the mirror mode of operation. After switching the device 140 to the "display" mode of operation, one or more previously recorded images can be displayed on the device 140. Using the ability to selectively set the operation of the device 140 in the "mirror" or "display" mode, the user 111 can compare simultaneously or sequentially, images of his appearance during the first and second fitting, as described in detail below.

In some illustrative embodiments of the invention, such as those shown in Figures 1, 2A, 2B, 3A, and/or 3B, controller 121 may control block 140 to also display mirrored appearances in the display mode of operation. However, those skilled in the art will appreciate that the invention is not limited to this possibility, and that in other embodiments, the controller may control the device 140 to display other appearance related images as well, such as a rotated image, an inverted image, a substantially unchanged image, such as a frontal image, a toned image, and the like, for example, as will be described below.

Device 140 may include any suitable configuration and/or mechanism to selectively set operation in the first or second modes. For example, in one embodiment, device 140 may be an array of liquid crystal elements that can change their optical characteristics, such as reflectivity, refractive index, and the like, for example, depending on the voltage applied to the liquid crystals. For example, when a first voltage is applied, the optical characteristics of the liquid crystals may change such that the mirror-display device 140 can operate as a mirror, and when a second voltage is applied, the optical characteristics of the liquid crystals may change such that the mirror-display device 140 can work as a liquid crystal display.

Alternatively, the mirror-display device 140 may comprise a liquid crystal display embedded in a translucent or one-way mirror. Accordingly, if the liquid crystal display is turned off, the mirror-display device 140 can passively reflect sufficient light falling on it so that the user sees his own reflection in it with sufficient quality and sufficient brightness. If the liquid crystal display is turned on, then the images displayed on it can be seen by the user 111, since their brightness can be significantly higher than the brightness of the light reflected from the surface of the "mirror-display" device. In other embodiments, the system is implemented on a stand-alone computer without a separate display, so that the buyer of the system can choose the appropriate display, or the system can be used without a display, but images can be displayed on users' mobile devices or tablets so that they can see themselves on their mobile devices. devices and remotely control the recording of information.

In addition, in some cases, the camera may be located away from the display, for example, in luxury stores, the display screen may be embedded in the wall, and the camera with appropriate lenses may be located away from the display.

[0051] In accordance with some illustrative embodiments of the invention, the device 140 "mirror-display" can be implemented using a liquid crystal HD ready-TV in a mirror, such as, for example, model 32PM8822/10 from Royal Philips Electronics (see description on the site http://www.research.philips.com/newscenter/archive/2003/mirrortv.html-). Such a device may comprise, for example, a polymer based organic light emitting display (OLED). The mirror-display device 140 may be any other suitable device that implements any suitable image display technology. For example, device 140 may use displays such as: Nanotube Electron Emission Display (NED); plasma panel (PDP); cathode ray tube (CRT) display; digital light processing (DLP) display; surface electron emitter display (SED); tablet display; flat SED display; display on organic electronic elements; electronic paper; a three-dimensional display such as a holographic display; thin film resistor display (TFT); optical TFT display, LED matrix display; An LCD display with charge-coupled device capabilities such that the mirror-display device 140 can perform the functions of the image acquisition device 130; paintable LCD; high definition television display (HDTV); projection screen display (rear projector), etc.

In accordance with some exemplary embodiments of the invention, the imaging device 130 may be adapted to capture images of one or more external views in front of the mirror-display device 140. The field of view in front of the device 140 "mirror-display" may include, for example, a field, zone, area, area and/or area in front of the device 140 "mirror-display". For example, the field of view may include at least a portion of the field, zone, region, area, and/or area that is reflected in the device 140 when it is in mirror mode.

The imaging device 130 may be, or may include, a CCD camera, a video camera, a still camera, and/or a 3D camera, such as a stereoscopic camera, or the like. A stereoscopic camera may be adapted, for example, to capture a three-dimensional image of the user's appearance. A stereoscopic camera may, for example, include two lenses, the distance between which corresponds to the distance between the eyes of a person. Accordingly, the stereo camera may be capable of simulating human binocular vision, whereby it may provide a three-dimensional image.

In accordance with some exemplary embodiments of the invention, station 110 may be a stand-alone unit that may be located in an appearance comparison area located in a desired location, such as an office, home, or store, such as a ready-to-wear store.

In accordance with some illustrative embodiments of the invention, station 110 may be connected, for example, through a network interface 122, to a network, such as network 150, as a result of which communication is provided between station 110 and one or more other stations connected to network 150, such as station 160 and/or station 170.

In accordance with some exemplary embodiments of the invention, station 110 may include a network interface 122 that may be adapted to interface with network 150 to exchange information with other stations on network 150. Such information may include, for example, data corresponding to images of users received at various stations of system 100, such as stations 160 and/or 170, as well as identifying user information to provide secure access to the system, as will be described below. Network 150 may be, for example: a local area network (LAN), a wide area network (WAN), a wide area communications network such as the Internet, a wireless communications network such as a wireless LAN (WLAN), a Bluetooth network, a virtual private network (VPN) , a cellular communications network, for example, a 3rd Generation Partnership Project (3GPP) network such as, for example, a frequency division duplex (FDD) network, a global system for mobile communications (GSM) network, a WCDMA (Wideband Multiple Access) cellular communications network code division), etc.

[0057] In accordance with some exemplary embodiments of the invention, one or more stations 160, 170 may be portable devices. Non-limiting examples of such portable devices include a mobile phone, laptop, laptop, netbook, tablet, personal communication system device, personal digital assistant, wireless communication device, personal digital assistant with built-in wireless communication device, cellular phone, smart card, memory card, block memory, etc. In some embodiments of the invention, one or more stations 160, 170 may be stationary devices such as, for example, a desktop computer, television, server, and the like.

In accordance with some exemplary embodiments of the invention, system 100 may also include a control center 190 that may be connected to stations 110, 160, and/or 170, such as over network 150. Control center 190 may receive and store data that may represent, for example, images of the appearance of users received from one or more stations 110, 160 and/or 170.

In accordance with some exemplary embodiments of the invention, stations 110, 160 and/or 170 may be located in different locations, for example, in different stores of the same retail chain. Stations 110, 160, and/or 170 may also be located in different locations within the same building, such as on different floors, in different sections on the same floor, and the like. Such places can be, for example, ready-made clothing stores, shoe stores, retail outlets, showrooms, exhibitions, shopping centers, optics stores, cosmetics stores, sports clubs, health centers, fitness centers, railway stations, restaurants, hotels, residential buildings and so on. One or more stations 110, 160, 170 may also be used as interactive bulletin boards. For example, the image acquisition device 130 may capture images that can be displayed on a bulletin board (not shown). The system 100 may allow the user 111 to select an image to be displayed on the bulletin board from among a plurality of images, such as from various previous fitting results.

In accordance with some exemplary embodiments of the invention, images of a user's appearance may be viewed in different locations. For example, the imaging device 130 may capture an image of the first fitting. The image may then be transmitted from network interface 122 over network 150 using messages 151 and 152 to station 160, for example. Accordingly, user 111 will be able to see an image of the user's first appearance at station 160. Therefore, user 111 may be able to view, for example, the user's first fitting in a first retail store, such as a store associated with station 110, and can compare the appearance during the first fitting with the appearance during the second fitting, which may occur in a second store of the same or an affiliate network, for example, at a store associated with station 160 and/or at other times, such as hours, days, or weeks after the first fitting.

In accordance with some illustrative embodiments of the invention, the image acquisition device 130 may acquire images of the user's appearance during the first fitting and transmit the image through the network interface 122 over the network 150 to the control center 190, where these images can be stored for later reading. Accordingly, the user 111 can access the first fitting images at any station, such as station 160 connected to control center 190 via network 150.

In accordance with some exemplary embodiments of the invention, storage device 123 may be, for example, a hard disk drive, floppy disk drive, CD drive, CD-ROM drive, or other suitable removable or permanent storage device.

In accordance with some illustrative embodiments of the invention, the controller 121 may be or may include, for example, a central processing unit, a digital signal processor, a microprocessor, a controller, a chip, a microchip, an integrated circuit, or any other suitable general purpose or special purpose processor or controller, for example, such devices as are known in the art.

The input device 124 may be a keyboard, a remote control, a pointing device such as, for example, a laser pointer, a mouse, a touch pad (touchpad), a touch screen, which may be embedded, for example, in the mirror-display device 140 , or may be implemented in the form of another suitable device, for example, separate from device 140, a biometric input device, such as a fingerprint scanner and/or a face scanning camera, and/or any other suitable pointing or input device. Input device 124 may be adapted to receive information identifying a user, for example, to provide access, such as secure access, for user 111 to system 100, as described below.

In accordance with some exemplary embodiments of the invention, the user 111 may enter commands using the data input device 124 to control the operation of the image acquisition device 130. The input device 124 may include, for example, an interface that allows the user 111 of the system 100 to set the operating parameters of the image acquisition device 130. The controller 121 may receive input from the user 111 in the form of messages 131 and control the operation of the image acquisition device 130 accordingly. The user commands may include, for example, commands related to setting times for image acquisition, positioning of the image acquisition device 130, for example, in accordance with an auto-tracking algorithm for tracking the position of the user 111, and/or image acquisition characteristics such as focus , camera position, shooting angle, dynamic range, etc. The user commands may also contain commands for setting operating modes of the image pickup device 130, such as, for example, a video mode, a photo mode, and the like. In accordance with some embodiments of the invention, the imaging device 130 may include an audio recording device, such as, for example, a microphone, and/or an audio output device, such as, for example, a speaker. Accordingly, the image pickup device can receive audio signals, such as the speech of the user 111, which can be recorded and stored, for example, in the data storage device 123, and output audio information through the audio output device. The audio output device can reproduce any other kind of audio information such as radio programs, CD recordings, and the like.

In accordance with some illustrative embodiments of the invention, the controller 121 may, for example, set the mode of operation of the mirror-display device 140 in accordance with commands received from the user 111. For example, if the mirror-display device 140 is operating in mirror mode, the user system 100 may enter a switch command via input device 124, for example, by pressing a specific button on device 124 to switch device 140 to display mode. Controller 121 may receive input from input device 124 and may instruct device 140 to switch to display mode, such as using messages 141.

In accordance with some illustrative embodiments of the invention, the image acquisition device 130 may be installed in various positions, for example, on top, below, or on one side of the mirror-display device 140 to obtain images of the appearance of the user 111 related to trying on any a piece of clothing, or images of the user against the background of various products, for example, against the background of some furniture. In some embodiments, the image acquisition device 130 may capture an image of the user's appearance as it appears on the mirror-display device 140, ie, a mirror image of the user. In other embodiments, image acquisition device 130 may capture an image of the appearance and controller 121 may generate a mirror image corresponding to the appearance captured by device 130. or an algorithm for rotating, inverting, and/or mirroring the appearance captured by the image acquisition device 130, resulting in image information representing a rotated, inverted, and/or flipped image generated from an image acquired by the image acquisition device 130. According to these embodiments, the controller 121 may control the operation of the mirror-display device 140 to display a rotated, inverted, and/or flipped image in a display mode. In other embodiments, the controller 121 may control the operation of the mirror-display device 140 to display in display mode an image corresponding to the image captured by the device 130, for example, no rotation, no inversion, and/or no mirroring. In some embodiments, the imaging device 130 may not be visible to the user 111 and may be positioned behind and/or incorporated into the mirror display device 140, the device 140 being, for example, an LCD display with instrument capabilities charge-coupled, which can both display and capture images. For example, in one of the illustrative embodiments of the invention, the device 140 may include a screen or surface containing liquid crystals to provide information display in mirror/display modes, for example, as described, as well as the ability to receive images, that is, the device 140 can perform the functions devices 130 and 140 described above.

In some illustrative embodiments of the invention, one or more stations 110, 160 and/or 170 may not contain an image acquisition device 130, and/or one or more stations 110, 160 and/or 170 may not contain a mirror-display device 140. For example, the first station of system 100 may only contain an image acquisition device 130 and may not contain a mirror-display device 140. The user 111 may use the first station to capture images of the user's appearance during the first fitting, for example, without being able to view the resulting images. Later, the user 111 can view the images obtained during the first fitting at another station of the system 100, which contains the device 140 "mirror-display".

In accordance with some exemplary embodiments of the invention, the image acquisition device 130 may be positioned to capture an image and/or sequence of images, video, or the like of the area in front of the mirror-display device 140. Instead or in addition to this, the image acquisition device 130 may be positioned so that it captures the image reflected from the mirror display device 140 . For example, the image acquisition device 130 may be capable of capturing an image of the user 111 standing in front of the mirror display device 140 . The user 111, while in front of the device 140 "mirror-display", can evaluate their appearance during the first fitting of clothes. In accordance with the command entered by the user 111 through the input device 124, the image acquisition device 130 can capture images of the appearance of the user, who may try on a piece of clothing, for example. It should be noted that the fitting may also include the view of the user 111 along with various objects that may be near it, such as furniture, room furnishings, and the like. Accordingly, the image acquisition device 130 can capture the appearance of the user 111, for example, during the first fitting, second fitting, etc., and can transmit the corresponding acquired images to the storage device 123 using messages 131 and 30. The user can then call recorded images of, for example, the first fitting, the second fitting and other subsequent fittings for comparison, as described below with reference to figures 2A, 2B and 2C.

In accordance with some illustrative embodiments of the invention, the data storage device 123 may be adapted to receive data representing images acquired by the device 130 and record to store an image of the appearance, more specifically, the appearance of a user, such as trying on garments. The user's appearance images for given fittings may be retrieved from the storage device 123, for example, by the controller 121, and displayed on the display 140. The user 111 may compare the displayed images, for example, as will be described below.

In accordance with some illustrative embodiments of the invention, the data storage device 123 may contain data, which is programs that require user identification data, such as a user ID, password, login time, biometric data, etc., to provide a secure access to station 110, as will be described below. For example, the controller 121 may control the mirror display device 140 to display images corresponding to the identity of the user 111, for example, in accordance with the identification data provided by it. For example, the user 111 may provide the input device 124 with their identification data, which may include biometric data such as a face image, palm print, fingerprint, retinal image, voice, and the like. The data entered to identify the user may include other suitable features such as a credit card, personal identification number, password, smart card, customer card, club card, and the like. The controller 121 checks, in accordance with any suitable method and/or algorithm, whether the data entered by the input device 124 matches the information identifying the user, which may be stored, for example, in the data storage device 123 or in the control center 190. As software for confirming the input biometric data, for example, "Active ID FaceVision technology" from Geometric Inc. can be used. If the controller 121 determines that the data entered by the user 111 matches the recorded identification data, then the controller 121 may allow the user 111 to access data representing, for example, images of the previous appearances of the user 111.

In accordance with some illustrative embodiments of the invention, the storage device 123 may contain data representing, for example, software algorithms that provide additional system capabilities, such as the rendering of virtual effects on the device 140 "mirror-display". For example, the controller may be capable of displaying on the mirror display device 140 images of the user's appearance along with virtual objects such as clothing, hats, hairstyles, furniture, and the like, for example. In addition, the controller 121 may be configured to display on the device 140 a "mirror display" of the appearance of users having different body shapes to simulate a decrease or increase in the user's weight, or the like. For example, the user 111 may choose to first try on a specific piece of clothing, such as a suit from a certain collection, and the controller 121 may "put on" the suit on the image of the user 111 displayed on the device 140 "mirror-display". The controller 121 may record images of the first fitting in the storage device 123 and may perform the same step for the second fitting. Accordingly, the user 111 of the system 100 will be able to compare their images in the first and second fittings.

In accordance with some illustrative embodiments of the invention, the controller 121 may provide, for example, the ability to view images and/or video files, repeat playback of images and/or video files, and the functionality may be predetermined by the system 100 or may be determined during operation, in in accordance with the commands of the user 111 entered by him using the input device 124. For example, the controller 121 may be able to recall one or more previous user appearance images and display them on the mirror display device 140 in a different order. For example, the images obtained in previous fittings may be displayed substantially continuously forward, backward, or selected randomly and in any order. In addition, images of previous fittings can be displayed simultaneously on the device 140 "mirror-display", as will be described below. The controller 121 may also delete previously recorded user appearance images, limit the amount of data that can be stored in the storage device 123, and other similar functions, and may further control the size, shape, color, and so on. images displayed on the device 140 "mirror-display".

In accordance with some illustrative embodiments of the invention, the user may use a portable storage device 180 data, which can record one or more received images. Such a device may be any suitable portable storage device such as a smart card, flash drive, or the like. The user 111 can read the images represented by the messages 50 of his appearance during the first fitting from the storage device 123, for example via the drive interface 125 or via any other suitable connection. Later, the user 111 can download their first try-on images to another device, such as another station in system 100, such as station 170.

In some exemplary embodiments of the invention, station 110 may contain more than one mirror-display device, or may contain a mirror-display device 140, which can be divided into two parts, as described below with reference to figures 2A and 2B.

In accordance with some illustrative embodiments of the invention, the controller 121 may record for storage, for example, in the data storage device 123, parameters characterizing the user 111. For example, the system 100 may contain a scale connected, for example, to the data storage device 123 through the controller 121. The controller 121 may be able to record the weight of the user 111, for example, during trying on things. Accordingly, the user 111 can later call up his parameters, including his weight.

Figures 2A and 2B schematically illustrate the steps for comparing appearance using an interactive system in accordance with some exemplary embodiments of the invention.

In accordance with some exemplary embodiments of the invention, the mirror-display device 140 can be divided into two parts, where one part 192 can work as a mirror, and the other part 191 can work selectively as a mirror and as a display. As shown in Figure 2A, the user 11 may stand in front of the mirror 192 during the first fitting, the images of which may be captured by the imaging device 130 and stored in the storage device 123. As shown in figure 2B, the user 111 can simultaneously see in the part 191 the appearance image obtained during the first fitting, and/or any other images of himself, for example, images recorded in the data storage device 123 and/or received over the network 150 (figure 1), next to its normal reflection in the mirror 192 during the second fitting, and compare their appearance during the first and second fittings.

Figures 3A, 3B and 3C schematically illustrate three successive stages of appearance comparison using an interactive system in accordance with some exemplary embodiments of the invention.

As shown in Figure 3A, the user of the system 100 can see their reflection during their first fitting in the mirror-display device 140 operating in mirror mode. The controller 121 may receive, for example, from the input device 124, user input, which may include a request to use the image acquisition device 130 to capture images of the user's appearance during the first fitting. As a result, the image acquisition device 130 can capture images of the user's appearance during the first fitting, and these images can be recorded in the storage device 123 .

As shown in Figure 3B, the user 111 can see their reflection during a second fitting in a mirror-display device 140 that can operate in a mirror mode. Then, when the user wants to see their image during the previous fitting, for example, for comparison, the controller 121 may receive a user command entered through the input device 124 that sets the display of the images obtained during the first fitting. At this point, as shown in Figure 3C, the controller 121 may send messages 141 to the display device 140 to enter the display mode. The controller 121 may also instruct the device 140 to display the images taken during the first fitting. Thus, by switching the operating modes of the mirror-display device 140, the user 111 can compare his images during the first and second fitting and/or any other images previously recorded in the data storage device 123.

Figure 4 is a flowchart of a method for comparing one or more images of a user's appearance, in accordance with some exemplary embodiments of the invention. One or more of the steps of the method illustrated in Figure 4 may be performed by one or more elements of system 100 (Figure 1), although the invention is not limited to this possibility.

At step 410 of the method, for example, setting the operating mode of the "mirror-display" device can be carried out. For example, the user 111 (FIG. 1) may first set the mirror-display device 140 to operate in mirror mode. In other embodiments, display 140 may be configured to default to mirror mode each time a new user connects to system 100 (FIG. 1).

At method step 420, the user may, for example, try on clothes in front of the mirror-display device. For example, the user may stand in front of the "mirror-display" device and assess their appearance during the first try-on, for example, clothes, shoes and/or any other items.

At 430, an image of the user's appearance during the first fitting may be captured. For example, the user 111 may instruct the device 120 to capture the image acquisition device 130 of the user's appearance during the first fitting.

At method step 440, the user may also, for example, change the appearance in front of the mirror-display device. For example, the user 111 can change the details of the environment, such as furniture, or change clothing accessories, or change the posture in front of the mirror-display device 140, which can operate in a mirror mode, and evaluate their appearance in a different posture.

At step 450 of the method, switching of the operating modes of the "mirror-display" device can also be carried out. For example, the user 111 (FIG. 1) can switch the mirror-display device 140 between mirror and display modes. Accordingly, the user 111 may be able to compare their appearance during the first try-on and/or any other appearance, for example, recorded in the storage device 123 and/or received over the network 150 (FIG. 1), with their appearance during the second fitting.

In accordance with some illustrative embodiments of the invention, the user 111 may specify and/or the station 110 may automatically save parameters for each fitting, for example, characteristics of stores (name, address, name of the seller) and goods (price, time and/or date of fitting) and so on. The user 111 can, for example, store recorded images of his appearance on a removable or portable storage device, as already indicated with reference to figure 1, and later can view these images, and it is possible to sign each image, for example, indicating the name of a store, etc. .P. In addition, the user 111 may set and/or the controller 121 may generate and/or store in the storage device 123 memos or reminders, such as discounts, seasonal sales, and the like.

Below we will consider the option of using an image taken by a camera, for example, a digital photo or video camera, and converting the image in such a way that when it is displayed on the display screen, it will be similar to a mirror image, that is, the image that the user would see if if he stood in front of an ordinary mirror.

Figure 6 is a schematic view to help better understand the options described below, showing a digital display screen 640, a camera 630, and an image processor 645 that can create a realistic mirror image. Ideally, the camera would be able to move in a parallel plane behind the screen according to the position of the user's eyes to create a real sense of gaze. However, placing the camera behind the screen is impractical as it will block the camera's field of view. Theoretically, this can be overcome by using a translucent screen or a plurality of very small holes, but such solutions unnecessarily increase the cost and complexity of the system. The simplest solution is to place the camera above the display screen and transform the resulting image to simulate a mirror image. Below are examples of the corresponding transformations.

To compensate for changes in the distance between the user and the screen, it is necessary to use an adaptively adjustable field of view so that the user can see his image at the same size as in the mirror. In accordance with one of the options, this problem is solved by using the zoom (zoom) of the camera, digital and/or optical. Traditional cameras have a fixed field of view or an optically variable field of view. In order to provide an adaptively adjustable, that is, continuously variable, field of view, the system must change the resolution or control the zoom or focal length of the camera in real time in accordance with the tracked position of the user.

In addition, the image needs to be mirrored vertically to allow left-to-right direction changes, as occurs in mirroring. This image transformation can be performed relatively simply by changing the pixel addresses of the digital image accordingly.

As already indicated with reference to figure 1, the mirror may be smaller than the user, but still show him in full growth. This can be achieved together with an appropriate choice of field of view and an appropriate choice of screen size. The idea is to project an image onto a screen, with the field of view set so that the user sees the reflection at twice the distance from the mirror, so that he is seen at full height, even if the digital screen is smaller than the user's height. This is illustrated in Figure 6 for user positions A, B, and C, which schematically shows the user captured by the camera at different distances from the mirror, yet rendered the same size by changing the camera's field of view. In some embodiments, the resolution of the image also changes according to the user's distance from the mirror. For example, for smaller distances, one embodiment may use multiple cameras, and the image processor performs the connection ("stitching") of their images to reduce distortion.

To acquire 3D images, it will be necessary to use two cameras at a distance corresponding to the distance between the user's eyes, or one camera with two effective virtual points of view. In order to obtain a realistic 3D impression, it is necessary to use an adaptively controlled closed loop system in which 3D imaging is performed based on distance. When the user looks at his reflection in the mirror, he sees a three-dimensional image, although when he approaches or moves away from the mirror, the angle between his eyes and the reflection changes, resulting in a change in the depth of the three-dimensional image.

As already mentioned with reference to figure 6, placing the camera behind the display at the eye level of the user leads to a higher cost or complexity of the system. Therefore, in the following embodiments, a practical method for realizing the above-described ideal system with a fixed arrangement of camera(s) around the perimeter of a display screen is described. The main objective is to compensate for image distortion in an adaptively controlled manner and in real time, in order to give the user the experience that would be experienced in an ideal system. Figures 7 and 8 show examples of problems that arise when using a practical configuration when the camera is located above the screen of the digital display, that is, not at the level of the user's eyes.

Figure 7 is a schematic view illustrating the user's approach to or away from the mirror when using a camera mounted on top of the screen and pointing horizontally. If the system is calibrated for the average distance to the screen, then when the user is farther from the screen (position C in figure 7), the image will be smaller and taller, and when the user is closer to the screen (position A in figure 7), the image will be larger, and the camera's field of view will crop the user's image. In addition, when the user approaches the screen, distortions become noticeable, that is, the sensations will be different from the usual sensations of the user looking at himself in the mirror.

To capture the user in full growth at any distance in one of the options, the camera is located on top of the screen and tilted down to provide maximum dynamic range of user movement in front of the screen. As shown in figure 8, since the camera is tilted downward, the image distortion will be much larger and much more noticeable. The closer the user is to the camera, the more distorted their image will be. Distortions are manifested in the fact that the user seems to be smaller, and the upper body increases in size. In this configuration, the user's image becomes smaller as the user moves away from the screen. On the other hand, the effective/utilized field of view of the camera will cover a larger area in front of the screen and will allow the user to be much closer to the screen and still see themselves in their entirety.

To ensure the formation of a plausible (expanded) mirror image obtained from a camera located in front of the user and shifted by some distance relative to the screen, a method is used to convert the captured image before displaying it on the screen. The method may provide the converted image in real time or with a delay, depending on the mode of operation of the screen.

This method takes as input an image captured by a camera and transforms this image to correct the camera's viewpoint and field of view so that the viewpoint matches the viewpoint in the case of a conventional mirror. That is, when the user approaches or moves away from the mirror, the viewpoint that will be reflected in the mirror will be different from the viewpoint captured by the camera. Embodiments of the proposed method include modules for adaptive adjustment of the viewpoint and field of view in accordance with the tracked position of the user.

It should be noted that the solution of the virtual mirror problem only by inverting the image relative to the vertical, as is done in known methods, is not enough to form a mirror image close to the real one. As the user gets closer to the camera/screen, the size of the image increases and vice versa, in contrast to the reflection in a normal mirror where the image dimensions change little with little change in the field of view as the distance from the user to the mirror changes. In addition, when the user approaches the camera, image distortion occurs, which must be corrected in the improved method. By adaptively adjusting the viewpoint and field of view before displaying the image on the screen, an image can be obtained that simulates the reflection from a mirror located at the location of the screen. To obtain an image that more accurately reproduces the specular reflection, the method may also include the following additional features.

In accordance with one of the variants, the method also performs a dynamic adaptive connection of images obtained from several cameras. In accordance with this embodiment, the system contains several cameras, for example, located in different places and/or having different characteristics, so that it is possible to increase image resolution, image fidelity, increase the field of view of one camera, reduce distortion for different users and different postures of users. , and the formation of an improved model of the user's body to compensate for camera distortions. For example, since image resolution decreases when the user moves away from the screen (there will be fewer pixels in the user's image), it is advisable to use optical zoom and change the focal length of the camera. The problem is that this comes at the cost of shrinking the camera's field of view. Image fusion or dynamic zoom can be used simultaneously to improve resolution while maintaining the field of view.

As already mentioned in the previous paragraph, to obtain an image close to the real reflection in the mirror, the system can also use an adaptively adjustable optical zoom. The adaptively adjustable optical zoom increases the quality/resolution of the image/video according to the continuously monitored distance from the user to the screen. It also reduces asymmetric distortion that occurs when the camera is mounted on one side of the screen.

To provide the necessary image conversion accuracy, the system can be calibrated using discrete pointers, and the method can interpolate and extrapolate to provide accurate conversion at various user positions from the screen. According to one embodiment, the projection distortions of the images may be calculated according to the tracked position of the user and the location of the camera. According to another embodiment, the amount of image distortion is measured in front of the screen, and this amount is used instead of directly calculating the projection.

The proposed method is optimized to provide the lowest possible delay by using pre-calculations to transform images depending on the distance to the screen, so that for the measured distance, the image transformation algorithm will already be prepared in advance. In accordance with another option, direct calculation without calibration is used by generating a transformation to compensate for projection distortions in accordance with the calculated/measured user's viewpoint.

The following is a general description of the modules that work together to convert a captured image to a mirror image. The video imaging module performs: capturing video images, improving their quality, optimizing camera parameters and controlling the camera to obtain the best video image quality under the limitations of the hardware used. The geometric measurement module measures or determines any combination of parameters such as the distance from the screen to the user, the height of the user, the position of the head and eyes of the user, and also evaluates the three-dimensional image of the user's body, etc. The camera control module sets the camera operation parameters to provide the required image/video quality and maximum resolution. In the case of multiple cameras, the field of view will be optimized for maximum resolution according to the user's distance from each camera.

The geometric transformation module selects the video frame by video frame along with the corresponding information about the user's geometric position and orientation, transforms the raw images according to their exact position, and fills in the unfilled pixels, if any. For example, a geometric transformation is performed according to the position of the eyes so as to obtain an image in which the user is looking into a mirror. That is, the method provides an appropriate geometric transformation of the input video images to obtain images in which the user makes eye contact. This conversion is also useful in other situations. For example, when videoconferencing using computers, because the camera is located above the computer monitor, the user's gaze will always be directed downward in the resulting image, since the user is looking at the monitor screen, and not at the camera lens. Using the above geometric transformation in such a situation allows you to get an image of the user in which he is looking at the camera, although in reality he is looking at the monitor screen. This allows you to provide closer visual contact of the participants in the video conference. For video conferencing, geometric correction is performed similar to the case with a mirror, although inverting with respect to the vertical (left-right) is not required here.

The image conversion module may be implemented according to various embodiments. In one embodiment, scaling is used, in which to obtain the size of the image that the user should see in the mirror, the dimensions of the captured image are changed in proportion to the distance from the user. According to another embodiment, image projection and scaling are used, and in accordance with the distance to the user, the displacement in space and the location of the eyes for different distances and displacements, the projection error between the user's eyes and the location of the camera is calculated or measured. Additional conversion may be performed to correct optical distortion.

In yet another embodiment, a superposition method is used, which has been found to provide fairly accurate results. According to this method, the transformation is calculated in advance according to the image alignment technique of the image of the user standing in front of the camera and looking into the lens. The reference image may be generated from one of several cameras. For example, the reference camera is placed at a height and distance that minimizes most optical distortion, such as at eye level and 2 to 3 meters away from the user. Image alignment provides the best conversion results in terms of the position of the eyes and the whole body of the user, based on multiple pointers. In one example, a plurality of pointers, such as white dots, on a reference object, such as a person or a mannequin, will form a two-dimensional mesh on his body. For example, on the legs, on the chest, on the shoulders, on the eyes, etc. ring stickers can be installed. To improve accuracy, this calibration can be repeated for different locations and user positions in front of the screen to account for, for example, different lateral offsets, different heights, different distances, and so on. For each location, the best transformation algorithm is created, taking into account various distortions (projection, barrel, fisheye, any combination of them, etc.), which can be corrected using various registration techniques (for example, projection, affine transformation, transformation similarity, polynomial transformation, their combinations, etc.).

The above options can be implemented together with the following additional improvement. For example, a combination of images from multiple cameras can be used to improve quality and resolution, to minimize distortion created by a single camera, or to expand the field of view. The image connection element is a complete connection according to the geometric position of the user. Each camera uses its own corrective transformation, since the offset relative to the virtual location of the eye will be different. Multiple cameras can also be used to form a 3D image to enhance the user experience.

A 3D IR image can also be formed. Such an image can be used to form an accurate virtual model of the user, which can then be used by the application programming interface for virtual fittings and augmented reality generation. Using the capabilities of augmented reality technology, you can change the background, introduce objects, perform virtual fittings on various items of clothing, etc. It can also be used to analyze the behavior of the human body and link it to augmented reality applications, virtual fitting applications, gaming applications, video conferencing applications, etc.

Since the system records images (photos or videos), various analyzes of these images can be performed. For example, video analysis can be used to collect information about a shopper or a store, such as how often a user smiles, whether they are satisfied, approximate age, gender, ethnic group, and the like. A shopper behavior analysis system can be fed a variety of data and can be used to actively integrate e-commerce into everyday life, or it can be used as a component of any other application. For example, such a system may analyze lists of products tried on by users and link them to store inventory, 3D printing and e-commerce site, e-shop stylist, social network, and the like.

Figure 9 is a block diagram of an embodiment of the system of the present invention in which transformation is performed to obtain an image simulating a reflection in a mirror. The various modules of the system shown in a block diagram in Figure 9 may be programmed on a general purpose computer, implemented on DSPs, CPUs, GPUs, digital signal processors or dedicated camera ICs, digital signal processors or display special ICs, standalone computing device, on a user-programmable gate array, on a specialized IC, and the technique of parallel computing using "cloud" technologies, etc. can also be used for this.

Block 930 represents one or more 1:n cameras from which an image stream is transmitted to the image capturing module 932. Block 930 may contain photo and/or video cameras, an IR camera, devices for obtaining two-dimensional and/or three-dimensional images. Block 930 may also include an electro-acoustic or electronic range finder.

An image capturing module 932 receives the image stream from block 930 and may perform filtering to improve their quality. In addition, images can be cropped or resized to optimize them. If multiple cameras are used, the module 932 performs the joining ("stitching") of the images to improve their quality or to expand the effective field of view.

The trigger module 960 is a parallel process that can receive input information directly from the output of the image capture module 932, although it can also receive images after the transform module 934 according to the position of the eyes. The input information can be optimized in size, bandwidth and speed to implement the desired functionality. Below are examples of elements that may be contained in the module 960 triggering events. Determining that the user is in front of the camera 930, for example, based on the disappearance of a part of the background, changes in a predetermined area, pattern recognition, and the like. Measuring the distance to the user, for example, by correlation between stereo cameras, 3D IR camera, etc. In another embodiment, the distance to the user is determined using measurements from one of the cameras based on some geometric assumptions, for example, the user is standing in front of a mirror on the floor, so that the user's distance, height or theoretical point of view in the mirror can be determined from the location of his shoes and from special lateral displacement relative to the mirror.

A face recognition module can be used to facilitate user identification. During the operation of the system after registration, it must store information for each user, and if the user is recognized by the system, it can call up his data, make updates to them, offer products, etc. In the case of face recognition, the user does not need identification to enter the system, as a result, he saves time and simplifies the work with the system. Face recognition can be a trigger event for automatic appearance recording. In some embodiments, the length of the recording of each session is predetermined and may begin immediately after successful face recognition. In other embodiments, triggering may use a remote control technique such as, for example, a feature found in mobile applications or dedicated tablet/mobile or other remote control devices.

Other elements that may be contained in the triggering event module 960 include video analysis programs (for generating effective feedback, for example, to estimate age, gender, mood, other significant characteristics, etc.), try-on recognition (so that the system can easily link try-on items to the store, improve the stocking and e-commerce system by generating recommendations and speeding up the assessment and planning of the range) and gesture recognition (for seamless user control without the need for an input device).

The conversion module 934 according to the position of the eyes converts the image before displaying it on the screen 940. The module 934 receives images from the image capture module 932, and in addition, it can receive the calculated distance, or the distance and height, or the actual point user observations. The transformation module 934 calculates the necessary transformation according to the position of the eyes. Such a transformation can be calculated directly from the difference in projection angles between the camera and the user's theoretical point of view, for example, to obtain the required image scale and position. Alternatively, an exact match method may be used in which a factory calibration process may be used to obtain a very accurate conversion between the camera and the computed viewpoint at discrete user positions in front of the mirror. Optionally, base images and transformations can also be used, which are generated from one or more parameters, such as distance, height, pose, measurements made at the factory, or in real time when the user is standing in front of the screen.

According to the distance to the user, or a combination of distance and eye viewpoint, the eye position transform module 934 interpolates the transform for any position of the user in front of the camera. Since the distance of different parts of the user's body from the camera is not the same, projection distortion occurs, in which closer parts, such as the head, are displayed with increased resolution (more pixels) than other parts of the body, such as legs. Accordingly, the image of the user will be with shortened legs and with an elongated head, that is, closer parts of the body will be seen enlarged, and more distant parts will appear reduced. This transformation is non-linear, that is, each pixel represents a different length and width (dx, dy), so pixel filling or exclusion must be scaled along the coordinate axes. Any interpolation (linear, cubic, polynomial, etc.) can be used as a filling method, or a more complex filling using projections of different cameras can be used. For example, cameras at the top and bottom of the screen can complement and allow better detection of fill pixels or correction of eye orientation. An additional option is to use special lenses/prisms that correct most of the projection distortion and will improve resolution. Some resizing issues that affect quality can also be compensated for optically with a mechanical zoom.

The system may additionally include a virtual fitting/augmented reality module 936. The input information enters the module 936 from the module 934 of the transformation in accordance with the position of the eyes (module EyesMatch) or from the module 938 recording. The conversion unit 934 performs image conversion according to the position of the eyes, which will be described later. The eye position conversion unit 934 can be used in a variety of applications other than mirroring, such as video cameras, video conferencing, and the like. Module 936 may receive additional input from a host that is in active communication with a stock database, 3D printing, e-commerce database, and the like. Using the input data, module 936 can render digital images to be merged with the transformed image. For example, module 936 can be used to change the color of the item being tried on according to the available assortment and the user's choice. It should be appreciated that the playback of module 936 is different from a normal virtual fitting, and that module 936 performs playback on an article of clothing that is displayed worn by the user. Thus, reproducing an item in a different color does not change the physical appearance of the item worn by the user, but only the color. Accordingly, the user can feel the real thing on himself, see how it really looks and whether it suits him, see its real folds, etc. The 936 module can also be used to add accessories to the user's real image or add virtual fitting capability to the system. Likewise, module 936 can be used to complement the background, to change or create a different environment to match the item being tried on, such as a picture of a beach for a swimsuit try-on, a nightclub setting for an evening dress, and the like.

Module 938 recording video or photo images receives them directly from the module 932 capture images, from the module 934 transformation in accordance with the position of the eyes or from the module 936 virtual fitting and augmented reality. In addition, it receives control messages from the control module 962, which indicates when to start/stop recording, what information to store for each user, and what video or photo images to perform subsequent transformations/additions, virtual fittings, quality enhancements, etc. .P. The method, which can also be used for recording without loading system resources, can also include encoding / decoding capabilities on video cards.

In addition to the need to have a local copy of the video/photo images on the "mirror" (station), the images must be automatically duplicated on the "cloud" and can be automatically encoded to any desired size.

The recording process can also change the frame rate, compression, format, remove parts of the image, and change video effects and color effects.

In addition to recording, the system can additionally provide a stream of video information in real time from the "mirror" memory. For example, the system is capable of transmitting a video stream on the network in real time from any "mirror", that is, it will be possible to view this video on any device, including the transfer of a video stream from one "mirror" to another.

The control module 962 controls the operation of all other modules. The control module 962 configures all hardware and software components of the system, including cameras 930, screen 940, digital signal processor, and others. -applications 952, smartphone applications 954, and the like. The information to be recorded in module 938 can be delivered to the user immediately using radio signals, IR signals or by wire, or can be transmitted with the user's permission to a "cloud" application such as Facebook, Google or others, or to a server company that can be associated with the e-commerce application.

The factory calibration module 964 is used when the transformation relies on real image transformations. The geometry of the camera-to-screen configuration may be determined/measured at the factory, or may be measured at the installation site, and the offsets may be factory calibrated. In addition, the correction can correspond to the height or angle of the installed mirror relative to the floor. A possible calibration process is described below. At the first stage, reference information is created that determines how the image should look in the mirror at different discrete distances, at certain orientations, at different points of reflection of the eyes (that is, theoretical observation points). This reference information can be obtained in different ways:

1. As shown in Figure 10A, the camera 14 is at the intended location of the mirror at or near the height of the user's eyes. The user or dummy with pointers (eg, with white marks 10 glued on) is positioned at a distance from the camera that eliminates most of the camera's distortion when the user looks into its lens. The image can be scaled and the image can be cropped so that it is equal to or slightly smaller than the virtual image in the mirror. That is, the generated reference image must correspond to the actual size of the user, which can be seen if a mirror is placed in place of the camera, or slightly smaller so that it fits into the screen size (depending on the screen size).

2. In Figure 10B, the user or pointer dummy is positioned in front of a conventional mirror 12. The camera is positioned near the user at eye level and captures the reflection in the mirror. The reference image can also be obtained if several cameras are used, which in this case are installed vertically to eliminate distortion. After that, the images taken by several cameras are combined to obtain one homogeneous image with high resolution and low distortion. Google glasses can also be used in this calibration option. The user, being in front of a mirror, uses this camera to take a picture of himself. Since the field of view is fixed, the only thing that needs to be done is to resize the image to get the correct size.

After obtaining the reference image, another image is taken by moving the camera to its regular place where it will be during operation, for example, on top of the screen, and this second image will be indicated here as a real image. Then the reference and real images are combined by aligning the pointers on the real and reference images. Based on the results of the combination, a transformation is obtained that can best correct image distortions and extract the body orientation. In addition, offsets or additional corrections can be introduced into the transformation according to the actual installation geometry in the field. According to the distance of the user from the screen and the viewpoint of the eyes, the transformation can be interpolated at each position of the user in front of the camera.

Figure 11 is a flowchart illustrating a method according to one embodiment of the invention, which can be used in the systems discussed in the present description. The method begins at 1100 with a factory calibration procedure, which may be performed using the following steps, for example. Select an object to be calibrated, such as a mannequin, a user, a shield, etc. Then pointers are installed on the object, for example, stickers shown in figure 10A. After that, the camera is installed in its regular place, for example, on top of the screen, tilted down, as shown in figure 8, and set up to receive images at the maximum range. They take photographs of the object in different positions (different distances, heights, lateral displacements, etc.). On the other hand, for simplification, pictures taken at different distances along the central virtual line in front of the screen are quite enough. One or more reference images are then obtained using, for example, one of the methods illustrated in Figures 10A and 10B. In one embodiment, the camera is mounted at the user's eye level at the location of the screen. Then one or more snapshots of the same object are obtained using the same attached pointers. If only one shot is taken, it should be taken at an appropriate distance that minimizes camera distortion. If multiple reference shots are taken, they should be taken at different distances to capture the small changes that occur as the user approaches the mirror. However, it should be noted that if several objects are used and several reference images are obtained, then it is advisable to obtain them at the same distances and in the same positions in order to form more accurate transformations. The dimensions of the reference images can be changed to obtain the correct size on the screen, that is, the size of the image that would be seen in a mirror placed at the location of the screen.

After the images of the object and the reference images are obtained, the images of the object with distortions obtained using the tilted camera are combined (combined) with the reference images using pointers to obtain images of the object as it will look in the mirror installed at the location of the camera . As a result of this operation, a transformation operator will be obtained, for example, a set of pointers, each of which has two coordinates: one in the input image, and one in the reference image. Various transformations can be tested on multiple object images and reference images. For example, various distortion functions can be applied to obtain the best image matching results. The main functions are: projection, resizing, translation along the XY axes (shift) and reflection (replacement of the left and right parts). The best combination of transformation functions is chosen, and a transformation algorithm is created for each calibration point. This conversion is a factory calibration that will convert the user's real-time images into images that mimic a reflection in a mirror. It should be noted that the mapping transformation may include different transformations for each pixel or for each part of the resulting image to provide the best representation of the specular reflection.

As can be appreciated, step 1100 may be performed at the factory prior to shipment of the system. On the other hand, this operation can also be performed locally in the system installation, especially when the camera and the screen are separate elements, so that different arrangements are possible relative to each other. In such cases, it may be useful to ship the system with a calibration object, such as a manikin (3D object) or a shield with calibration marks (2D object), so that the user can perform an on-site calibration operation using a standard object. In this case, the system will be pre-programmed to perform the calibration operation and the formation of the conversion algorithm at the installation site of the system.

After the system is installed at its place of normal operation, step 1105 is performed, in which real photo or video images of the user are obtained and entered into the system, for example, using an image capture unit. At the same time, the position of the user is measured so that his point of observation can be determined. These measurements can be made using IR measurement technology or with sensors such as the Kinect® available from Microsoft®. At 1115, a scaling factor is additionally (if necessary) applied to match the screen size to the user's height.

At step 1120, an appropriate transformation algorithm is selected or the transformation obtained at the factory as a result of standard calibration is interpolated to obtain an accurate transformation scheme for a particular user's position in front of the screen. Then, the conversion algorithm is applied by the conversion module according to the position of the eyes to the video image received in real time. In the present description, the term "real" also refers to the video stream that is captured by the image capture unit and may be buffered before being displayed on the monitor screen for display. Also, if it is necessary in connection with the transformation, at step 1125, the necessary pixels are filled using an interpolation technique. At step 1130, the image from the buffer is transmitted for display on the screen. It should be understood that due to the high speed processors available on the market, steps 1105 - 1130 can be performed in real time, so that the user will not experience any delay in displaying images on the screen.

The eye position image conversion module can be used to provide a more natural environment that is more in line with daily life. For example, when making a video call using a PC, such as using Skype®, because the camera is usually located above the monitor, the caller often appears to be looking away from the camera because they are actually looking at the monitor screen. This makes the conversation seem unnatural because the speakers do not look at each other. In such an application, an eye-position conversion module can be used so that the speaker appears to be looking directly at the camera in the image displayed on the screen, when in fact he is looking at the screen. This conversion can be performed either on the caller's computer or on the called party's computer. For example, a user using Skype® may have a conversion module installed on their computer according to the position of the user's eyes, so that whenever the video camera is activated for a video conference, the EyesMatch module intercepts the video stream coming from the video camera and applies the transformation to it. before transferring this video stream to another user through Skype. When another user receives the video stream, it has already been converted so that the other user sees the image in which the first user is looking into the camera lens. The same can be done on any standard video conferencing system such as WebEx, Polycom, etc.

Figure 12 is a block diagram illustrating the modules and processes used to perform image calibration and transformation according to one embodiment of the invention. Block 1230 represents an imaging module, which may include one or more still or video cameras, an IR camera or distance measurement sensor, a 3D imaging module, and the like. Module 1232 manages settings to optimize camera performance, such as obtaining an appropriate field of view, capturing the desired area in front of the screen, setting the focal length and/or resolution, and so on. Block 1260 is a trigger event module that can be used to detect the user's presence in front of the screen and start the image capture and conversion process. The rest of the time, the system can be in standby mode, in which only the background is displayed on the screen until the user is detected.

Bracket 1360 encompasses the steps in the calibration operation that will be described here for a system configured for on-site calibration, with camera 1230 positioned above display screen 1240 as indicated in Figure 1. Calibration 1260 may be performed at the factory prior to system shipment or on-site after its installation. At step 1261, an object with pointers is set up, and at step 1262, one set of reference images is obtained (for example, using a camera installed at the same level with the user's eyes and directed horizontally), as well as a set of input images (for example, using a camera installed in its regular place, above the screen, and tilted down). At 1263, the pointers on each input image are compared with the corresponding pointers on the respective reference images. Thus, transformation vectors are generated. Depending on the quality of the visible mirror image, step 1264 additionally fine-tunes the vectors corresponding to the areas around the user's eyes.

Using a comparison of the pointers on the input images and the reference images, at step 1265, the transformation parameters that give the best image registration can be determined and set. Below are the options that can be used for the conversion. A very effective parameter is the slope transformation. Tilt conversion corrects image distortion caused by tilting the camera down. Tilt transformation modifies the input image taken by the camera tilted down, transforming it so that it appears that the camera is "looking" horizontally (the video system can also correct for azimuth error). The other transformation option is the height transformation. The Spatial Height Transformation linearly changes the offset given by a camera mounted outside the screen, eg above the screen, and this transform produces an image that would be captured by a camera mounted at the user's eye level on the screen. A horizontal shift can also be applied to correct misalignment between the camera and the center of the screen. Another important transformation for mirror image reproduction is scaling. As already mentioned, in a reflection in a conventional mirror, the user sees very small changes in the size of his body as he approaches the mirror. Conversely, in an image captured by a camera, the dimensions change significantly as the distance from the camera changes. Scale transformation reduces the effect of resizing with distance from the screen so that the user will see their image on the screen at a nearly constant size regardless of distance from the screen. Depending on the position and field of view of the camera, it may also be necessary to shift the image along the X-Y axes.

The practical implementation of the invention has shown that the implementation of tilt, height adjustment and zoom provides a fairly convincing image, which only needs to be reflected relative to the vertical (change the right and left sides), to effectively simulate the reflection in the mirror. Further image enhancements are associated with transformations to correct barrel and fisheye distortion. It is also possible to further enhance the depth of perception of the displayed image by adding lights and shadows, or artificially changing the intensity and/or hue of image pixels in a controlled manner, or by adjusting lighting elements such as LEDs located on the sides of the screen or user. In addition, a three-dimensional image effect can be created either by using special glasses or by using glasses-free three-dimensional imaging technology. In the second case, different sets of images are projected into each eye. Since the distance to the user and the position of the user's eyes are measured or determined in the embodiments described above, it facilitates the task of presenting three-dimensional images to the user by projecting different sets of images into different eyes.

At 1266, a transformation algorithm is generated and stored for storage. This conversion algorithm will be used at step 1234 to convert the video stream coming from the real-time imaging module 1230 using the estimated distance to the user. Depending on the action of the transformation algorithm, it may be necessary to use fill pixels. For example, when the camera is positioned at the top of the screen and tilted down, tilt and height transformations must be applied, resulting in an image that can be enhanced by the use of fill pixels. Filling the image with pixels can be performed at step 1237, and the resulting image at step 1240 is displayed on the screen. Applying transformation and pixel filling to the incoming video stream can easily be done in real time using appropriate processing power.

Figure 13 shows a diagram of another option, in which the calibration and compilation of the conversion algorithm are carried out on site after the installation of the system. In the example shown in figure 13, the camera 14 is located above the video screen 12 and tilted downward. The camera 14 and the screen 12 are connected to the controller 18, which controls their operation. The user uses an input device 16 that can communicate with the controller using a wired or wireless communication link. The user stands in a suitable location, for example 2 to 3 meters from the screen 12 and starts the calibration process, for example by entering the command "calibration" from the device 16 remote control. During the calibration process, the video stream arrives in real time from the camera 14 to the controller 18, which reflects the image about the central vertical axis (changes the left and right sides of the image with each other) before displaying it on the screen 12. Using the input device, the user can control different conversion functions. For example, the input device may include buttons for specifying tilt, elevation, and scale corrections, as shown schematically in Figure 13. Commands entered by the user are passed to controller 18, which then applies real-time transformations to the incoming video stream so that the image on the screen in front of the user changes in real time. The user can change the intensity of each transformation function until he gets a satisfactory image on the screen, after which he can press the "Enter" button, indicating the completion of the calibration. The controller 18 then records the corresponding calibration parameters and uses them for all subsequent real-time video streams.

Another feature of the circuit shown in Figure 13 is distance determination. Since the position and tilt of the camera 14 relative to the screen 12 is known (for example, due to a fixed bracket that secures the camera to the top of the screen), the image captured by the camera 14 can be used for triangulation and distance calculation. For example, when a user appears in a frame of the video stream transmitted by camera 14, the user's shoe tips can be triangulated to calculate their distance from the screen 12. This operation can be performed every fixed number n of frames, so that the user's distance from the screen can be continuously updated to track his movements. The background can be updated when the user is not in front of the mirror (adaptive background generation).

Another feature of the scheme shown in figure 13 is the use of lighting to create depth and/or atmosphere. In particular, multiple light sources 17, such as groups of LEDs, can be placed at various locations around screen 12 and controlled by controller 18. Light sources 17 can be controlled to increase image depth, for example by adding shadows and brightly lit areas. The light sources 17 can be of different colors to improve the available illumination and create a suitable overall color "temperature" of the displayed image. The light sources can also be adjusted according to the user's distance to form a uniform image and eliminate artifacts (interference) that can be created by store lighting. Instead or in addition to this, changes in color and lightness can be made by the controller directly in the digital image received from the camera 14. For example, color conversion can be used to improve the specular nature of the image, for example, parameters that affect the following characteristics of the image can be changed in a controlled manner. : gloss, clarity, sharpness, haze, metallic luster, etc. Controller 18 may also add virtual highlights and/or shadows to the image to add depth. In addition, an anti-reflective coating can be applied to the front of the screen to eliminate or reduce reflections that are common with flat panel displays. The group of LEDs may be connected to a photosensitive light sensor and/or a color temperature sensor to automatically maintain the light characteristics of the group of LEDs within predetermined limits.

In accordance with other options, the acquisition of three-dimensional images is carried out using two cameras located at a distance D from each other. Distance D is defined as the average distance between a person's eyes, usually referred to as the interpupillary distance, which for adults is approximately 54 - 68 mm. The transformation can be obtained as a result of combining the image obtained from the input camera at a distance D with the image obtained from two reference (base) cameras, which are located in space like the user's eyes. The base image for alignment can be obtained using a Kinect® device or any 3D IR camera. Google glasses can also be used for calibration, with the user wearing these glasses having to film themselves in front of a regular mirror. Since the field of view is fixed, the only thing that needs to be done is to resize the image to get the correct size. An image conversion module, such as controller 18, may also resize the image to reduce or enlarge it to fit the user in the image on the screen, or to focus on some part of it.

As can be understood, while some of the above options relate to generating a transformation algorithm by image alignment or other empirical methods, other options may directly calculate the differences between the camera and the user's theoretical viewpoint based on the analysis. In this case, the image alignment operation is not needed. In this case, an analytical transformation algorithm is formed that provides distortion correction.

Another feature of the scheme shown in figure 13 is the use of the Internet or "cloud" to provide services related to the transformation and representation of the mirror image. For example, in accordance with one of the options, the transformation can actually be carried out in the "cloud" if it has a fast enough communication channel. In this case, the video stream from the cameras 14 can be transmitted to the server 181, which performs the conversion and sends the video stream back to the video controller 18 for display on the monitor screen 11. Alternatively, images from camera 14 or converted images from camera 14 can be stored in the cloud for storage and distributed from it to devices such as a smartphone 183, tablet 187, and other monitor screen 189 such as a flat-panel TV. This can be done in real time when the user is trying on the clothes, so that other people in other places can see them and give their opinion on how the clothes being tried on fit the user. In addition, after the user leaves the store, he will have access to all the images taken using a smartphone, tablet, PC, etc. It should be noted that, in order to enhance the convenience of the user interface, as shown in the monitor 189 of the circuit shown in Figure 13, thumbnails of various examples can be displayed along with the current image, and the user can select them for viewing.

Figure 14 is a flowchart of a method for extracting data from an image taken by camera 1430. The triggering event module 1450 detects the presence of a user and activates the data extraction process. The camera optimization module 1432 can be used to control the camera in order to obtain the best quality images. In this embodiment, when the camera 1430 is positioned above the screen and directed downward, it can see the user's shoes at a distance of, for example, 1 m to 4 m from the screen. However, since the camera is pointing downwards and the shoes are further away from the camera than the user's head, the distortion will be maximum. When using the above conversion algorithm, distortion correction will be provided at all distances.

Additionally, according to this option, at step 1462, the user's image is separated from the background. In step 1464, the center of mass is calculated by multiplying the user's binary image by the matrix of indices only, and then finding the average value of the pixel index. The bottommost pixel of the body, below the center of mass (J, k) is determined by opening a window around k (center of mass) and finding the lowest active index, which can be assumed to be the edge of the shoes. In step 1468, the height of the user is calculated by determining the top of the head according to the camera resolution in terms of distance, field of view, and camera angle.

The figure 15 shows a block diagram of an option in which the connection ("stitching") of images obtained from n cameras. Image stitching is especially useful for improving resolution and field of view as the user approaches the screen. In the variant, the scheme of which is shown in figure 15, the images of the video streams 1530, 1532 coming from n cameras are connected. Trigger event module 1560 can be used to start the connection process when the user's presence is determined in front of the screen. A distance measurement may also be provided, either in the form of a single value representing the average distance to the user, or multiple values, such as the distance for each camera. Instead, or in addition, multiple values may be provided representing different distances to different parts of the user's body.

The optimization of each camera can be performed independently (blocks 1533, 1536) and the video streams coming from each camera will go through different transformation algorithms (blocks 1560, 1562) because the location and orientation of the cameras relative to the viewpoint (blocks 1564, 1566) of the user are different . Block 1570 determine the connection scheme determines the best place to connect images. If the system uses two cameras, one above and one below the screen, the connection pattern definition block will indicate the index of the best cut. Since the distance is measured continuously, the connection line can be kept almost fixed as the user approaches the mirror. For best performance, the connection (block 1572) should be optimized according to the user's distance from the cameras. To remove lines that may occur at the intersections of different images from different cameras, smoothing is necessary (block 1574). Images from different cameras will be slightly different due to differences in camera quality and lighting, as well as different distortion correction for each camera. To reduce the backlight effect, interlacing rows of different images can be used. At the output of block 1576, a connected ("stitched") image is transmitted.

The embodiments described above use a conversion algorithm to correct the images taken by the cameras and produce an image that mimics the reflection of the user in a mirror. In the embodiment described below, a block diagram of which is shown in Figure 16, the quality of representation of the user's eyes is improved. The idea behind this option is to replace the eye area, or just the eyes, with altered pixels that will give the user the complete impression that the user is looking directly at themselves in the mirror. As can be understood, the user will look directly at himself displayed on the screen, however, since the camera is not at eye level and is not directed horizontally, the user's eyes captured by the camera will not look straight ahead. The variant whose block diagram is shown in Figure 16 corrects this problem.

The main body of the components in figure 16 is similar to the components of the block diagram shown in figure 12 and has the same last two digits of the reference numbers (different first digits 16). These components will no longer be described here. The focus is on block 1680, in which the eye position is corrected. Since the scale has already been corrected in the previous stages, at this stage it is necessary to correct and/or transform the internal elements of the eyes, such as the pupil, iris, etc., in order to obtain an appropriate image.

First, module 1680 detects eyes in the image and determines that the user is indeed looking at the screen. Information for this stage and for adjustments can be obtained from archived images, estimates of average eye proportions, or measurements when the user is looking directly into the camera lens, or when the user is standing far enough from the screen that the distortion will be relatively low after the conversion algorithm has been run. If it is determined that the user is looking at the screen, then the pixels corresponding to the eye zone are either transformed or replaced with pixels that place the iris and pupil in the center of the eye, i.e., the user will look forward. In addition, the albuginea of the eye can be enlarged to simulate looking forward, especially when the camera is located at the top of the screen and points downward.

Another feature that can be implemented in any of the above options is the update rate of the transform parameters. That is, sometimes the user may want to see a still image when standing still in front of a screen that simulates a mirror. If the conversion algorithm is changed while the user is stationary, this may cause discomfort to the user. On the other hand, when the user moves faster forward towards or away from the screen, then the transformation parameters can be updated faster without any problems. Accordingly, several zones are installed with different modes for updating parameters. According to one embodiment, zones of different regimes are set according to distance only. For example, when the user moves a distance greater than a given amount, the parameters are updated. Accordingly, in one embodiment, if the user moves less than 25 cm, the transformation parameters are updated less frequently, for example, every x seconds, and if the user moves more than 25 cm, then the transformation parameters are updated at a second, higher frequency. In another embodiment, the zones are set according to the user's movement speed. For example, if it is determined that the user is moving at a speed less than x cm/sec, then the first update rate of the transform parameters is used, and if the user is moving faster, then the update of the transform parameters is performed at a second, higher frequency.

Thus, as can be understood from the above, although a real-time update of the video processing system can be used to display user images, the conversion of images for display on a monitor screen does not need to be changed in real time, and its update will depend on the user's behavior. For example, if the user is essentially stationary or moving very slowly, the transformation parameters may be updated in "semi-real" time. Then, as the user starts moving faster, the video processing system can be updated more frequently.

It should be understood that the above embodiments may be implemented using a digital camera (eg, video camera) and a monitor, with images from the camera being fed to a processor. The processor applies a conversion algorithm to the images and displays them on the monitor in either display mode or mirror mode. From the above, it should be understood that the mirror mode can be implemented by displaying on the monitor a video stream received in real time, the images of which are processed in accordance with the conversion algorithm. That is, in the mirror mode, the images that are displayed on the monitor are real-time images taken by users. In the display mode, the displayed images are the images previously acquired and retrieved from the storage device. These recorded images may be raw, that is, as received from the camera, or already processed images. In any case, previously recorded images that are shown on the display are converted images. Therefore, if the recorded images are raw images, the processor applies a conversion algorithm to them before displaying them on the monitor.

Whether the system is running in display mode or mirror mode, the problem of time can be considered: in mirror mode, the image that is displayed on the monitor is a converted image of what the camera sees at the current time (or a little earlier), and in In display mode, the displayed image is a converted image of what this or another camera has seen before and is different from what the camera is currently seeing. This problem also applies to perception: in mirror mode, because the monitor displays a transformed image of what the camera is currently seeing, the user looking at the monitor has the feeling that he is looking into a mirror, and in display mode, the user, looking at the monitor, realizes that he is seeing a video recording of what happened earlier.

It should be understood that the system may be implemented separately and independently of the monitor screen. For example, in some places (for example, in a fitting room) only a monitor screen may be installed, without any camera. The monitor is configured to communicate with the system so that recorded images can be read and displayed on the monitor screen. The user can interact with the monitor to view previously acquired images, for example, to compare them with the item being tried on. In another example, all received images can be uploaded to the "cloud", so that the user can view these images on the screen of a PC or mobile device, for example, using an application on a tablet.

As can be understood from the above description, various embodiments of the invention are an image conversion apparatus, which includes: an input port for receiving digital images from a camera; an output port for transmitting converted images for display on a monitor or for recording in a storage device; and a transformation module that receives images from the input port and applies a transformation to them, which includes: at least a reflection of the image about a vertical axis, as a result of which the right and left sides of the image are reversed; applying a transformation algorithm to an image to change it so that it appears as a reflection in a mirror; resizing the image to reduce the changes caused by changes in the object's distance from the camera. In other embodiments, a program is provided which, when executed on a computer, causes the computer to transform a digital image received from a camera such that the transformed image resembles a reflection in a mirror, the transformation comprising: at least reflecting the image about a vertical axis, resulting in the right and the left side of the image are swapped; applying a transformation algorithm to an image to change it so that it appears as a reflection in a mirror; resizing the image to reduce the changes caused by changes in the object's distance from the camera. The program can be run on any universal computer such as a server, PC, tablet, smartphone, etc.

In other embodiments, a system is provided that allows a user to see their own image on a digital screen, the image displayed on the screen simulating a reflection in a mirror. The system contains: a digital camera that generates a stream of user images; a controller having an input port for receiving images from the camera and applying a transformation to the images to generate transformed images reproducing the reflection of the user in the mirror; wherein the controller has an output port for transmitting a stream of converted images for display on a monitor; and a storage device for storing the converted images. The controller is also connected to the Internet to upload a stream of converted images to the "cloud". The system also includes client programs that provide the ability to download converted images from the "cloud" for viewing.

In various embodiments, the controller applies a transformation to the image stream, which includes: flipping the image about a vertical axis, as a result of which the right and left sides of the image are reversed; applying a transformation algorithm to an image to change it so that it appears as a reflection in a mirror; and resizing the image to reduce changes caused by changing the distance of the object from the camera. The conversion algorithm essentially changes the address of each pixel of the original image. The conversion algorithm provides an image that would be obtained from a camera mounted at the level of the user's eyes and directed horizontally. The transformation algorithm includes at least a tilt transformation and a height transformation, wherein the tilt transformation changes the image in such a way that it is equivalent to tilting the camera, and the height transformation changes the image in such a way that it is equivalent to changing the camera height.

The transformation algorithm may include changing the angle of inclination, the angle of azimuth, scale, spatial shift (ie, linear change in height or horizontal shift), and the like. The final transformation is a matrix multiplication of the individual transformation to correct for the individual distortion.

In other embodiments, a system is provided for enhancing video conferencing or video calling facilities. The system contains a conversion processor for converting a stream of images received from a video camera installed on the perimeter of the monitor screen, so that in these images the user looks directly at the screen (which is what it really is), and not at the camera lens. The conversion subsystem receives a stream of images and converts them in such a way that the user looks directly into the camera lens on the converted images. The transformation algorithm includes any combination of corrections for tilt, azimuth, scale, spatial shift (i.e., ramp or horizontal shift), and the like. The transformation may also involve replacing the eye area, or just the eyes, with modified pixels that will give the user the complete impression that the user is looking directly into the camera lens.

In various embodiments, the method also includes calculating the distance to a user who appears in the image by tracking the distance between their eyes or head size and scaling the image accordingly. For example, according to one embodiment, the system records the expected distance between the user's eyes, i.e. interpupillary distance, at an average distance from the camera. 95% of the male population in the US has an interpupillary distance of 70 mm, while in women this distance is 65 mm. When a user's presence in front of the camera is determined, the system may first determine the user's gender, or simply jump to a distance measurement program that uses an average interpupillary distance of 68mm. For distance measurement, the system identifies the pupils in the image and rescales to maintain the expected interpupillary distance. The system processes the incoming video stream by scaling the images to keep the interpupillary distance constant, equal to the expected value. Thus, when the user moves away from the cameras or approaches it, the size of the user's figure on the image displayed on the monitor screen remains approximately the same, as it should be for reflection in the mirror. As can be understood, standard sizes such as ear spacing and the like can be used in the system, however, eye spacing is easier to use because eyes can be recognized very easily in the system. However, if the user is wearing sunglasses, the system will need to refer to other measurements of other parts of the body. This switching is performed dynamically, that is, if the system finds eyes, it uses the interpupillary distance, if not, then the transition to the measurement of other parts of the body follows.

Effect filters such as highlight effect filter, reflective texture effect filter, metallic color effect filter, etc. can also be used. Similarly, parameters such as camera shutter speed, sensor gain, white balance, and any combination of these can be adjusted to change the resulting image. In either case, these parameters are adjusted according to the dynamic region of interest, so that changes applied to the parameters apply only to the selected region of the image, and not to the entire image. For example, the parameters can be changed and updated according to the distance to the user and are limited to a selected area, which is a window that encloses the user. For example, the user's image can be separated from the background, and the settings will only apply to pixels in the user's image.

In other embodiments, an improved real-time effect is realized by recording video using a high-definition camera at a first frame rate, and playing back at a higher frame rate for smoother screen movements. In addition, the background of the received video stream can be replaced with an artificial background recorded in the system. Additional processing may be used to add or change the color or textures of an accessory or other element in an image. In addition, for videoconferencing and other applications, the conversion can be performed without reflecting the image about the vertical axis.

Although specific features of the invention have been discussed and described in the present specification, numerous modifications, substitutions, alterations, and equivalents will be apparent to those skilled in the art. Therefore, it is intended that such modifications and changes, without departing from the spirit of the invention, be covered by the appended claims.

Claims (36)

1. A method for implementing the operation of a system containing a monitor, a camera and a processor, so that the monitor displays an image imitating a reflection in a mirror by performing in any order the steps, including: obtaining a digital image from the camera; determining the distance from the user to the camera; applying a transformation algorithm to the digital image to align the viewpoint of the camera with the theoretical viewpoint of the mirror; resizing the image to reduce the changes caused by changes in the distance from the object to the camera; locating the user's eyes in the digital image; transforming or replacing the pixels of the eyes, thereby positioning the irises and pupils in the center of the forward-facing eyes; displaying the image on the monitor after resizing the image and converting or replacing the eye pixels. 2. The method according to claim. 1, in which the transformation or replacement of the pixels of the eyes additionally provides for an increase in the albuginea of the eyes. 3. The method of claim. 1, further providing for the reflection of the image about the vertical axis so that the right and left sides of the image are reversed. 4. The method of claim. 1, in which determining the distance from the user to the camera involves tracking the distance between the eyes of the user. 5. The method of claim 1, wherein determining the distance from the user to the camera involves identifying the user's shoes and solving a triangulation problem to determine the distance to the shoes. 6. The method of claim. 1, further comprising using an adaptively adjustable field of view to compensate for changes in distances from the user, so that the user sees an image of the same size. 7. The method of claim 1, further comprising the following steps: obtaining a reference image; combining the digital image with the reference image; use of data obtained as a result of combination to form a transformation algorithm. 8. The method of claim 1, further comprising applying an offset transform to move the image within the display area of the monitor. 9. The method according to 1, additionally providing for changing the backlight intensity of individual pixels or groups of monitor pixels in such a way as to improve the quality of reproduction of dark and light areas of the image. 10. The method of claim 1, further comprising applying at least one of a barrel distortion correction transform and a fisheye distortion correction transform. 11. The method of claim 1, further comprising performing spatial height adjustment to linearly change the offset in the image to match the height of the user's eyes with the image at screen height. 12. The method of claim 1, further comprising a pixel filling step, in which extrapolation is used to fill in empty pixels resulting from image resizing and eye pixel transformation or replacement. 13. The method of claim 1, further comprising the following steps: obtaining a second digital image from the second camera; connecting the image and the second image; display the linked image on the monitor. 14. The method of claim 13, wherein the fields of view of the camera and the second camera overlap, the method further comprising an overlap determination step of determining the best location of the overlapping portions of the fields of view for performing the joining step. 15. The method of claim 14, further comprising a smoothing operation to correct interference resulting from the connection step. 16. The method of claim 1, further comprising the following steps: reflection of the image about the vertical axis so that the right and left sides of the image are reversed; And resizing the image to reduce the changes caused by changes in the distance from the subject to the camera. 17. The method according to claim 1, wherein said steps are performed on a sequence of images of a video stream coming from a camera in real time. 18. The method of claim 17, further comprising: determining the distance of movement of the user present in the images of the video stream; And changing the frequency with which said steps are performed on the image sequence according to the moving distance. 19. The method of claim 17, wherein the live video stream is stored in a digital storage device and the system is selectively set to mirror mode or display mode, wherein the mirror mode displays converted images of what the camera sees on the monitor. at the current time, and in display mode, the monitor displays the converted images read from the digital storage device that the camera has seen before.

Images