WO2021102417A1 - Système de miroir intelligent pour la collecte et la visualisation d'informations d'apparence physique - Google Patents

Système de miroir intelligent pour la collecte et la visualisation d'informations d'apparence physique Download PDF

Info

Publication number
WO2021102417A1
WO2021102417A1 PCT/US2020/061822 US2020061822W WO2021102417A1 WO 2021102417 A1 WO2021102417 A1 WO 2021102417A1 US 2020061822 W US2020061822 W US 2020061822W WO 2021102417 A1 WO2021102417 A1 WO 2021102417A1
Authority
WO
WIPO (PCT)
Prior art keywords
user
mirror
collecting
visualizing
technology
Prior art date
Application number
PCT/US2020/061822
Other languages
English (en)
Inventor
Haesung LEE
Sanghyeop LEE
Original Assignee
Lee Haesung
Lee Sanghyeop
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Lee Haesung, Lee Sanghyeop filed Critical Lee Haesung
Publication of WO2021102417A1 publication Critical patent/WO2021102417A1/fr

Links

Classifications

    • A—HUMAN NECESSITIES
    • A47—FURNITURE; DOMESTIC ARTICLES OR APPLIANCES; COFFEE MILLS; SPICE MILLS; SUCTION CLEANERS IN GENERAL
    • A47G—HOUSEHOLD OR TABLE EQUIPMENT
    • A47G1/00—Mirrors; Picture frames or the like, e.g. provided with heating, lighting or ventilating means
    • A47G1/02—Mirrors used as equipment
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N23/00—Cameras or camera modules comprising electronic image sensors; Control thereof
    • H04N23/57—Mechanical or electrical details of cameras or camera modules specially adapted for being embedded in other devices
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/18—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast
    • H04N7/181—Closed-circuit television [CCTV] systems, i.e. systems in which the video signal is not broadcast for receiving images from a plurality of remote sources

Definitions

  • the present invention relates generally to an imaging and display system. More specifically, the present invention relates to a smart mirror system and/or device for collecting and visualizing a subject’s physical appearance information using sensors, cameras, and artificial intelligence technologies.
  • the conventional mirror i.e., reflective surface
  • the conventional mirror is the most common and reliable tool for individuals to use in reviewing their appearance in real-time, with many people looking in mirrors daily to check their appearance.
  • the data that users obtain from traditional mirrors immediately disappear after walking away from the mirror.
  • the present invention aims to solve the aforementioned problems and/or improve on conventional devices by integrating sensors and other related devices into a set of mirrors that form a substantially improved image collecting and display device.
  • the present invention offers a smart mirror system that effectively and efficiently collects and visualizes appearance information and data of a user with artificial intelligence (AI) technologies, including, but not limited to, facial recognition, machine learning, deep learning.
  • AI artificial intelligence
  • the smart mirror system acquires the user’s appearance data using various sensors and multiple cameras while the user looks in a mirror of the smart mirror system.
  • the cameras can capture three-dimensional (3D) image/video data of the user in a 360-degree range.
  • the sensors can measure distances of all visible features of the user’s appearance including outfit, pose, etc., measure body temperature, color, etc. of the user. Additionally, the sensors can measure environmental temperature, relative humidity, light and color, etc.
  • the smart mirror system allows the user to access the collected appearance information even when away from the mirror.
  • the user’s appearance data that are effortlessly collected enables predictive analyses of future conditions based on the acquired appearance data and detects changes over time.
  • the smart mirror system allows the user to obtain many categories of appearance information, including, but not limited to, current appearance, daily outfit, changes in body shape/condition, and medical condition, per the desired use of the acquired data. Additionally, the mirror system acquires video data together with sensor data to present videos including, but not limited to, dressing, clothing fittings, shaving, hair-dressing, face makeup, poses, exercises, etc., to the user.
  • the mirror system can provide simulated images and/or videos in the form of a three-dimensional (3D) avatar to the user using AI technologies, including, but not limited to, deep learning, machine learning, facial recognition, etc.
  • the 3D avatar may include, but is not limited to, a recommended outfit, pose, user’s body measurements and/or simulations, etc.
  • the mirror system may present videos including, but not limited to, an instructor’s exercise video, a standard/recommended pose, etc., to the user along with the current video of the user.
  • the mirror system may provide multiple mirrors to catch a 3D image/video of the user.
  • the smart mirror system can be made in one piece, or a plurality of mirrors connected in a form of a foldable and movable system that further provides physical flexibility to the user.
  • a computing device of the smart mirror system may include a memory and allow the user appearance data to be saved on the system.
  • the computing device provides secure data transfer to any extraneous storage medium, including, but not limited to, a portable disk drive, a flash drive, etc.
  • the computing device allows the mirror system to communicate to an extraneous device, including, but not limited to, a server, a computer, a cloud device, a mobile device, a personal computing (PC) device, etc., through a wireless network, including, but not limited to, Internet.
  • the present invention offers the user a versatile, flexible, user-friendly, and effective smart mirror system.
  • FIG. 1 is a front view of a smart mirror system of the present invention.
  • FIG. 2 is a perspective rear view of one embodiment of the present invention, wherein a computing device attached to the back face of the smart mirror system.
  • FIG. 3 is a perspective rear view of the smart mirror system of the present invention in a folded state.
  • FIG. 4 is a top view of the smart mirror system of the present invention.
  • FIG. 5 is an illustration showing one embodiment of the smart mirror system of the present invention, wherein a display is attached to the back face of the mirror.
  • FIG. 6 is an illustration showing another embodiment of the smart mirror system of the present invention, wherein the display is attached to the mirror surface of the mirror.
  • FIG. 7 is a system diagram of a computing device of the smart mirror system of the present invention.
  • FIG. 8 is a flowchart diagram of the computing device of the present invention, wherein the computing device communicates with a server and a personal computing (PC) device through an extraneous network.
  • PC personal computing
  • FIG. 9 is an illustration showing one embodiment of the smart mirror system of the present invention in use, wherein acquired sensor data and a three-dimensional (3D) avatar of the user is displayed on the display.
  • FIG. 10 is an illustration showing another embodiment of the smart mirror system of the present invention in use, wherein acquired sensor data, outfit recommendations, and a simulated three-dimensional (3D) avatar of the user are displayed on the display.
  • FIG. 11 is an illustration showing another embodiment of the smart mirror system of the present invention in use, wherein acquired sensor data, user’s pose, and an instructor’s pose are displayed on the display.
  • the present invention comprises a smart mirror system that effectively and efficiently acquires and visualizes appearance information of a user with artificial intelligence (AI) technologies through the use of multiple cameras and various sensors.
  • the smart mirror system of the present invention comprises a mirror assembly 10, a display 50, and a computing device 70. More specifically, the mirror assembly 10 comprises a mirror 11, a plurality of cameras 12, and a plurality of distance detectors 13.
  • the mirror 11 comprises a mirror surface 32 and a back face 33. Both the plurality of cameras 12 and the plurality of distance detectors 13 are distributed on the mirror surface 32 of the mirror 11. Additionally, both the plurality of cameras 12 and the plurality of distance detectors 13 are electronically connected to the computing device 70 through a wired or wireless connection.
  • the display 50 is attached to the mirror 11 of the mirror assembly 10 and is electronically connected to the computer device 70 as well.
  • the computing device 70 is mounted to the back face 33 of the mirror 11 of the mirror assembly 10. Additionally, the computing device 70 is adapted to acquire a plurality of appearance data of the user through the plurality of cameras 12 and the plurality of distance detectors 13. Further, the computing device 70 is adapted to display appearance information on the display 50 using the AI technologies, including, but not limited to, machine learning, deep learning, facial recognition, etc.
  • the mirror 11 of the smart mirror assembly 10 may be a foldable mirror.
  • a foldable mirror may be one continuous piece of mirror, or comprises a plurality of mirrors 31.
  • the mirror 11 can include a special mirror that allows light through it to display information/data directly on the mirror 11 and the rest of the mirror surface 32 can reflect an object as does a normal mirror.
  • the mirror assembly 10 comprises the plurality of cameras 12 and the plurality of distance detectors 13, a plurality of sensors 14, a light 15, a speaker 16, a plurality of handles 17, and a plurality of wheels 18.
  • the plurality of cameras 12 may include, but is not limited to, a video camera, a Charge Coupled Device (CCD) camera, a camera and/or camera set-up enabling the capturing of 3D-images, an Intel ® Realsense D400 series camera, a stereoscopic camera, a camera integrated with infrared (IR) sensor, a camera comprising image sensors, including, but not limited to, CCD, Complementary Metal Oxide Semiconductor (CMOS), X3 ® PRO 10MTM CMOS image sensor from Foveon ® , etc.
  • CCD Charge Coupled Device
  • a camera and/or camera set-up enabling the capturing of 3D-images
  • an Intel ® Realsense D400 series camera a stereoscopic camera
  • IR infrared
  • a camera comprising image sensors including, but not limited to, CCD, Complementary Metal Oxide Semiconductor (CMOS), X3 ® PRO 10MTM CMOS image sensor from Foveon
  • the plurality of distance detectors 13 may include, but is not limited to, an IR sensor, a light detection and ranging (LiDAR) sensor, a short frequency infrared sensor, touchless LiDAR sensor, a LiDAR camera, an Intel ® Realsense L515, etc.
  • the plurality of sensors 14 is distributed across the mirror surface 32 of the mirror 11 and electronically connected to the computing device 70.
  • the plurality of sensors 14 may include, but is not limited to, a temperature sensor that measures room temperature and/or the user’s body temperature, a relative humidity sensor that measures room relative humidity, a proximity sensor, a touchless proximity sensor, a motion sensor, a color sensor, a touch sensor, a light sensor, a brightness sensor, a light dependent resistors (LDR) sensor, a microphone/sound sensor, IR sensor, etc.
  • the IR sensor identifies the user’s height, shape, and distance. Additionally, the IR sensor allows the user’s image to be distinguished from the background.
  • the color sensor is used to identify color, providing advanced pixel data for the captured image and helping to identify the user from the background.
  • the light sensor can measure the illuminance and automatically adjust the brightness so that the subject can be properly illuminated with appropriate brightness.
  • the plurality of sensors 14 provides the sensor data to the computing device 70 for real-time analyses of the user’s appearance, body conditions, environmental conditions, etc., and to display actual measured data and analysis results to the mirror system of the present invention.
  • the light 15 is mounted to the mirror surface 32 of the mirror 11 and electronically connected to the computing device 70.
  • the light 15 may include, but is not limited to, a light-emitting diode (LED), an organic light-emitting diode (OLED) light, etc., which can be attached to any place on the mirror 11, including the edges.
  • the lighting can be of different colors to improve the existing lighting.
  • the lighting may set the mood by changing the level and color, and a spotlight can allow users to see themselves brightly reflected in the mirror 11.
  • the speaker 16 is mounted to the mirror surface 32 of the mirror 11 and electronically connected to the computing device 70 as well.
  • a plurality of handles 17 is mounted to the back face 33 of the mirror 11 to provide flexible handling of the smart mirror system in situations, including, but not limited to, folding, extending, rotating, etc.
  • the plurality of wheels is mounted to the bottom of the mirror system 10 and offers easy moving, rotating, etc., of the smart mirror system.
  • the smart mirror system of the present invention comprises the display 50.
  • the display 50 may comprise a plurality of displays.
  • the computing device 70 may be adapted to project a specific display of the plurality of displays to each of the plurality of mirrors 31 of the mirror 11.
  • the display 50 comprises a display panel 51, which is attached to the back face 33 of the mirror 11 to display the user appearance information and data on the mirror surface 32.
  • the display 50 comprises a transparent display 52, which is positioned to the mirror surface 32 of the mirror 11.
  • the display 50 can show current time, date, temperature, humidity, etc.
  • the user may choose to check the user’s appearance data via the display 50, as can be seen in FIG. 9 to FIG. 11.
  • the smart mirror system can be configured to use machine leaming/deep learning algorithms to provide suggestions based on predictive analyses of the user’s future conditions and detected changes over time.
  • the user can access the information/data on a personal computing (PC) device, including, but not limited to, a mobile phone, a smart phone, a tablet, a smart watch, etc., or a website.
  • the user can use natural languages to operate different types of applications powered by the computing device 70.
  • the computing device 70 of the smart mirror system comprises a central processing unit (CPU) 71, an AI module 73, a graphics processing unit (GPU) 72, a deep learning accelerator 74, and a vision accelerator 75, a memory unit 76.
  • the AI module 73 is electronically connected to the CPU 71, which is adapted to process the plurality of appearance data through the AI module 73 before displaying the appearance information of the user on the display 50.
  • the deep learning accelerator 74 is electronically connected to the AI module 73.
  • the GPU 72 is electronically connected to the CPU 71, while the vision accelerator 75 is electronically connected to the GPU 72.
  • the CPU 71 is adapted to acquire a plurality of 3D data from the plurality of cameras 12 of the mirror assembly 10 through the GPU 72 and the vision accelerator 75, wherein each of the plurality of cameras 12 is a 3D camera. With the 3D data of the user’s appearance, the CPU 71 is adapted to process the plurality of 3D data through the AI module 73 and the deep learning accelerator 74. Subsequently, the CPU 71 creates a 3D avatar of the user and displays the 3D avatar of the user to the display 50. As can be seen in FIG. 9, in an alternative embodiment, the smart mirror system may use 3D video data acquired by the CPU 71 of the computing device 70 through the plurality of cameras 12 of the mirror assembly 10.
  • the resulting 3D avatar is a 3D avatar video presentation of the user.
  • the 3D avatar and/or 3D avatar video of the user may be displayed in a 360-degree rotating view on the display 50.
  • the CPU 71 may be adapted to provide a plurality of appearance analysis results using the plurality of 3D data and the plurality appearance data of the user through the AI module 73, wherein the plurality of appearance analysis results includes a change of body condition of the user. The CPU 71 then displays the plurality of appearance analysis results to the display 50. As can be seen in FIG.
  • the plurality of appearance analysis results includes an outfit video of the user
  • the plurality of appearance analysis results includes an outfit video of the 3D avatar with an outfit recommended by the AI module 73 of the computing device 70.
  • the AI technologies including, but not limited to, deep learning, machine learning, facial recognition, etc., are implemented through the AI module 73 of the computing device 70 to conduct the learning and analysis of the appearance input data and to create the outfit video for the user. As can be seen in FIG.
  • the AI module 73 may conduct learning and analysis of a body pose of the user and provides pose image/video of the user as the plurality of appearance analysis results, which is displayed on the display 50. Further, the CPU 71 may simultaneously display a body pose of any instructor to the user to guide the user in any form of task, including, but not limited to, exercises, fitness lessons, physical training sessions, yoga lessons, or any other suitable teaching and learning. In another embodiment of the present invention, the AI module 73 may conduct learning and analysis of exercise videos of the user, thus the CPU 71 can display the actual video of the user with a simultaneous showing of an instructor’s video to provide efficient and effective learning to the user. As can be seen in FIG.
  • the computing device 70 of the smart mirror system comprises the memory unit 76, which is electronically connected to the CPU 71 of the computing device 70.
  • the CPU 71 is adapted to securely save the plurality of appearance data of the user to the memory unit 76.
  • the memory unit 76 may comprise a storage device, including, but not limited to hard disk drive (HDD), solid-state drive (SSD), or any other storage device.
  • the storage device may include an extraneous storage device, which is electronically connected to the memory unit 76.
  • the connection may be made through a wired connection, including, but not limited to, a universal serial bus (USB), or a protected/secure wireless connection, including, but not limited to, Bluetooth ® .
  • the CPU 71 is adapted to securely save the plurality of appearance data of the user to the extraneous electronic storage device through the memory unit 76.
  • the storage device is used not only to enhance the storage capacity of the memory unit, but also to provide data security, especially when collecting the user’s body information and data.
  • the storage device and the memory unit 76 can safely and securely keep the personal image/video and other data within the smart mirror system and utilize the data for machine leaming/deep learning of the AI module 73.
  • the smart mirror system of the present invention comprises a server 77, a database 78, an extraneous communication network 79, and a personal computing (PC) device 90.
  • the database 78 is stored on the server 77.
  • the server 77 may include, but is not limited to, a server computer, a laptop computer, a desktop computer, a cloud computing device, and any other suitable computing device.
  • the server 77 is managed by an administrator, including, but not limited to, a system administrator, a computer professional, a software engineer, a software specialist, an information technology (IT) engineer, and any other suitable person.
  • the PC 90 device is electronically connected to the server 77 through the extraneous communication network 79, and the server 77 is electronically connected to the computing device 70 through the extraneous communication network 79 as well.
  • the extraneous communication network 79 may include a wireless network, including, but not limited to, Internet, the fifth generation (5G) network, etc.
  • the server 77 may be adapted to communicate with the computing device 70 to securely transfer all information, data, and analysis results and make safe storage to an external medium per the user’s request and/or by the administrator.
  • the database 78 is used to store all the data captured by the smart mirror system and the computing device 70 can be configured to access the database78 to transfer any and all necessary data through the extraneous communication network 79.

Landscapes

  • Engineering & Computer Science (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Mirrors, Picture Frames, Photograph Stands, And Related Fastening Devices (AREA)

Abstract

Un système de miroir intelligent collecte et visualise des données d'apparence d'un utilisateur avec des technologies d'intelligence artificielle (IA). Le système utilise divers capteurs et de multiples caméras pour acquérir des données d'apparence en temps réel pendant que l'utilisateur se regarde dans un miroir du système. Grâce à l'utilisation d'IA, le système analyse les données et crée un avatar tridimensionnel (3D) de l'utilisateur, puis projette l'avatar 3D conjointement avec des données de mesure sur une unité d'affichage sur la surface de miroir d'un miroir pliable. De plus, le système fournit des images/vidéos de poses d'exercice de l'utilisateur avec un affichage simultané de la pose d'un instructeur pour guider l'apprentissage de l'utilisateur. En outre, le système peut fournir des suggestions de tenue par analyse de la tenue actuelle de l'utilisateur et l'affichage d'avatars 3D simulés avec des tenues suggérés pour la comparaison. Toutes les données et tous les résultats sont sauvegardés de manière sécurisée dans le système. Ainsi, le système propose un miroir intelligent polyvalent et innovant.
PCT/US2020/061822 2019-11-22 2020-11-23 Système de miroir intelligent pour la collecte et la visualisation d'informations d'apparence physique WO2021102417A1 (fr)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
US201962939055P 2019-11-22 2019-11-22
US62/939,055 2019-11-22

Publications (1)

Publication Number Publication Date
WO2021102417A1 true WO2021102417A1 (fr) 2021-05-27

Family

ID=75980028

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/US2020/061822 WO2021102417A1 (fr) 2019-11-22 2020-11-23 Système de miroir intelligent pour la collecte et la visualisation d'informations d'apparence physique

Country Status (1)

Country Link
WO (1) WO2021102417A1 (fr)

Cited By (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117575636A (zh) * 2023-12-19 2024-02-20 东莞莱姆森科技建材有限公司 一种基于视频处理的智能镜控制方法及系统

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040078301A1 (en) * 2002-06-10 2004-04-22 Martin Illsley Interactive trying-on cubicle
US20130229482A1 (en) * 2005-03-01 2013-09-05 Nissi Vilcovsky Devices, systems and methods of capturing and displaying appearances
US20170287060A1 (en) * 2016-03-31 2017-10-05 Samsung Electronics Co., Ltd. Method for composing image and electronic device thereof
US20180247356A1 (en) * 2015-02-26 2018-08-30 Finity Technology Limited Computer implemented platform for the creation of a virtual product

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20040078301A1 (en) * 2002-06-10 2004-04-22 Martin Illsley Interactive trying-on cubicle
US20130229482A1 (en) * 2005-03-01 2013-09-05 Nissi Vilcovsky Devices, systems and methods of capturing and displaying appearances
US20180247356A1 (en) * 2015-02-26 2018-08-30 Finity Technology Limited Computer implemented platform for the creation of a virtual product
US20170287060A1 (en) * 2016-03-31 2017-10-05 Samsung Electronics Co., Ltd. Method for composing image and electronic device thereof

Cited By (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN117575636A (zh) * 2023-12-19 2024-02-20 东莞莱姆森科技建材有限公司 一种基于视频处理的智能镜控制方法及系统
CN117575636B (zh) * 2023-12-19 2024-05-24 东莞莱姆森科技建材有限公司 一种基于视频处理的智能镜控制方法及系统

Similar Documents

Publication Publication Date Title
US11341711B2 (en) System and method for rendering dynamic three-dimensional appearing imagery on a two-dimensional user interface
Cai et al. RGB-D datasets using microsoft kinect or similar sensors: a survey
US10789699B2 (en) Capturing color information from a physical environment
US11736756B2 (en) Producing realistic body movement using body images
US9495008B2 (en) Detecting a primary user of a device
WO2020010979A1 (fr) Procédé et appareil d'apprentissage de modèle permettant la reconnaissance de points clés d'une main et procédé et appareil de reconnaissance de points clés d'une main
US9245180B1 (en) Body modeling and garment fitting using an electronic device
CN110363867B (zh) 虚拟装扮系统、方法、设备及介质
JP2019510297A (ja) ユーザの真実の人体モデルへの仮想的な試着
US20210319585A1 (en) Method and system for gaze estimation
EP3394709B1 (fr) Miroir augmenté
EP3394712A1 (fr) Miroir augmenté
CN110377149A (zh) 场景分类
US20190045270A1 (en) Intelligent Chatting on Digital Communication Network
WO2017108703A1 (fr) Miroir augmenté
US20230065288A1 (en) Electronic Devices with Body Composition Analysis Circuitry
US20170148225A1 (en) Virtual dressing system and virtual dressing method
WO2021102417A1 (fr) Système de miroir intelligent pour la collecte et la visualisation d'informations d'apparence physique
US11599739B2 (en) Image suggestion apparatus, image suggestion method, and image suggestion program
WO2017108702A1 (fr) Miroir augmenté
JP7428874B2 (ja) 体型を判別するための演算デバイス、プロセッサー実施方法、および体型判別システム
Bhowmik Interactive and immersive devices with perceptual computing technologies
Huang et al. Implementation Simple Fitting System Using Image Recognition for Portable Device
TWI541732B (zh) 活動基礎成本制的支援方法與系統
Barapatre et al. Sixth Sense Technology using gesture control

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20890368

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20890368

Country of ref document: EP

Kind code of ref document: A1