KR20220028529A - A smart mirror apparatus and method, and a system thereof - Google Patents

Abstract

The present invention relates to a smart mirror device and method, and a system thereof. To this end, the smart mirror device can receive an identifier for a user entering inside a home from an electronic device located outside the home, and allow an image of the user and the information for the entry of the user to be displayed through the transparent display based on the received identifier. Therefore, the present invention can provide a person in the home with a convenience of more easily identifying various information on the user.

Description

A SMART MIRROR APPARATUS AND METHOD, AND A SYSTEM THEREOF

The present invention relates to a smart mirror apparatus and method, and a system thereof.

In general, a smart mirror device is a device that can recognize a user's movement or voice, and uses the mirror as a display. Such a smart mirror device may provide useful functions to the user based on the user's command.

However, the conventional smart mirror device only provides limited functions based on a limited command input from a user.

Accordingly, there is a need for a smart mirror device that not only functions according to various user commands, but also interworks with other electronic devices. In addition, as the smart mirror device is installed inside the home, the necessity of providing occupants with various information based on the user entering the home through connection with the electronic device installed outside the front door is also raised. In addition, the necessity of controlling various electronic devices in the home through the smart mirror device based on the user entering the home is also raised.

The conventional smart mirror device only provides limited functions according to a user's limited command, and there is a problem in that it cannot provide various functions.

Accordingly, it is an object of the present invention to provide various functions to a user through a smart mirror device.

It is also an object of the present invention to provide a smart mirror device with an identifier for an occupant from an electronic device that obtains biometric information for the occupant, and to provide information on such a situation to a occupant.

It is also an object of the present invention to provide a smart mirror device that receives an identifier for a visitor from an electronic device outside a home and provides various information about a user based on the identifier.

Another object of the present invention is to provide a smart mirror device that receives an identifier for a person entering or exits a home from an electronic device outside the home and controls the operation of at least one electronic device inside the home based on the identifier.

Another object of the present invention is to obtain an electronic device for obtaining biometric information about a user outside a home and transmit an identifier for the user to a smart mirror device inside the home, and at least one corresponding to the user based on the identifier. It is to provide a system for controlling the operation of home appliances.

The objects of the present invention are not limited to the above-mentioned objects, and other objects and advantages of the present invention that are not mentioned may be understood by the following description, and will be more clearly understood by the embodiments of the present invention. Further, it will be readily apparent that the objects and advantages of the present invention may be realized by the means and combinations thereof indicated in the claims.

In order to achieve this object, the present invention can provide various information about a user entering a home through the smart mirror device based on communication between the electronic device and the smart mirror device.

In addition, the present invention may receive an identifier for an authenticated user from the electronic device, and control the operation of at least one home appliance device corresponding to the received identifier.

In addition, the present invention provides communication between the electronic device and the smart mirror device, so that operations of the electronic device and the door opening and closing device can be controlled through the smart mirror device.

Also, according to the present invention, the operation mode of the smart mirror device may be operated in at least one of a mirror mode, a smart mirror mode, and a display mode.

In addition, the present invention may provide a smart home control system through a door opening and closing device, an electronic device, and a smart mirror device.

To this end, the smart mirror device of the present invention may include a display unit including a mirror and a transparent display, a communication unit, and a processor electrically connected to the display unit, and the communication unit. In addition, the processor receives an identifier for a user entering the home from the electronic device located outside the home through the communication unit, and receives the user's image and information on the user's entry, based on the received identifier. It can be displayed through the transparent display.

In addition, the method of the smart mirror device of the present invention includes the process of receiving an identifier for a user entering the home from an electronic device located outside the home through a communication unit, and based on the received identifier, the user's image and the It may include a process of displaying information about the user's entry through the transparent display of the display unit.

In addition, the system of the present invention may include an electronic device and a smart mirror device. The electronic device acquires a first image of the user through a first camera based on recognition of the user, performs authentication of the user based on the acquired first image, and performs authentication of the user based on the authentication of the user. , an identifier for the user may be transmitted to the smart mirror device.

Then, the smart mirror device acquires information on at least one home appliance located inside the home, and among the acquired information on the at least one home appliance device, at least one home appliance corresponding to the identifier received from the electronic device The device may be identified, and based on a preset condition of the identified at least one home appliance, an operation of the identified at least one home appliance may be controlled.

The electronic device according to the present invention acquires a first image of a user through at least one camera, performs authentication of the user based on the acquired first image and a pre-stored second image, and provides information on the authenticated user. By providing the identifier to the smart mirror device inside the home, it is possible to provide the occupants with the convenience of more easily identifying various types of information about the user.

In addition, the smart mirror device according to the present invention outputs information on the user's image and entry based on the identifier received from the electronic device, so that a occupant can acquire information about the user.

In addition, the smart mirror device according to the present invention displays weather information, standby information, and power consumption information of at least one home appliance inside the home through a transparent display, so that the user can more easily see the situation outside the home and the situation inside the home. It can provide perceptible convenience.

In addition, the smart mirror device according to the present invention displays location information for each occupant through a transparent display, so that other occupants can more easily identify the current location of each occupant.

In addition, the smart mirror device according to the present invention can provide the convenience of transmitting and receiving various signals with the portable device by setting a communication channel with the user's portable device based on the user's selection displayed on the transparent display.

In addition, the smart mirror device according to the present invention calculates the distance to the user based on the user's approach, and controls the display mode based on the calculated distance, so that the user can control the smart mirror device more conveniently. there is.

In addition, the smart mirror device according to the present invention identifies the corresponding at least one home appliance based on the reception of the identifier for the user from the electronic device, and sets the operation of the identified at least one home appliance in a mode in which the user presets the operation. By operating as , it is possible to eliminate the inconvenience of the user directly operating the home appliance.

In addition to the above-described effects, the specific effects of the present invention will be described together while describing specific details for carrying out the invention below.

1 is an exemplary diagram illustrating a home control system according to an embodiment of the present invention.
2 is a block diagram of an electronic device and a door opening and closing device according to an embodiment of the present invention.
3 is a block diagram of an electronic device and a smart mirror device according to an embodiment of the present invention.
4 is a flowchart illustrating a process of authenticating a user in an electronic device according to an embodiment of the present invention.
5 is a flowchart illustrating a process of authenticating a user in an electronic device according to another embodiment of the present invention.
6 is a flowchart illustrating a process in which the smart mirror device controls the operation of the home appliance according to an embodiment of the present invention.
7A is an exemplary diagram illustrating a display area of a display unit of a smart mirror device according to an embodiment of the present invention.
7B is an exemplary diagram illustrating a standby screen of a display unit of a smart mirror device according to an embodiment of the present invention.
8A is an exemplary diagram illustrating a case in which a customer is located at the door 130 according to an embodiment of the present invention. 8B is an exemplary diagram illustrating a screen of a smart mirror device when a guest visits according to an embodiment of the present invention.
9A is an exemplary diagram illustrating a standby screen of a smart mirror device according to an embodiment of the present invention. 9B is an exemplary view showing a situation outside the home through the smart mirror device according to an embodiment of the present invention.

The above-described objects, features and advantages will be described below in detail with reference to the accompanying drawings, and accordingly, those of ordinary skill in the art to which the present invention pertains will be able to easily implement the technical idea of the present invention. In describing the present invention, if it is determined that a detailed description of a known technology related to the present invention may unnecessarily obscure the gist of the present invention, the detailed description will be omitted. Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. In the drawings, the same reference numerals are used to indicate the same or similar components.

Although the first, second, etc. are used to describe various elements, these elements are not limited by these terms, of course. These terms are only used to distinguish one component from other components, and unless otherwise stated, it goes without saying that the first component may be the second component.

Also, when it is described that a component is "connected", "coupled" or "connected" to another component, the components may be directly connected or connected to each other, but other components are "interposed" between each component. It is to be understood that “or, each component may be “connected,” “coupled,” or “connected,” through another component.

Throughout the specification, unless otherwise stated, each element may be singular or plural.

As used herein, the singular expression includes the plural expression unless the context clearly dictates otherwise. In the present application, terms such as "consisting of" or "comprising" should not be construed as necessarily including all of the various components or various steps described in the specification, some of which components or some steps are It should be construed that it may not include, or may further include additional components or steps.

Throughout the specification, when “A and/or B” is used, it means A, B or A and B, unless otherwise stated, and when “C to D” is used, it means that there is no specific opposite description. Unless otherwise specified, it means that it is greater than or equal to C and less than or equal to D.

Hereinafter, an electronic device, a door opening and closing device, a smart mirror device and method, and a system for performing user authentication according to some embodiments of the present invention will be described.

1 is an exemplary diagram illustrating a home control system according to an embodiment of the present invention.

Referring to FIG. 1 , a home control system 100 according to an embodiment of the present invention may include an electronic device 110 , a door opening/closing device 120 , a door 130 , and a smart mirror device 140 . there is. The door opening and closing device 120 may be disposed on the door 130 , and the electronic device 110 may be disposed on a wall outside the groove. The door opening and closing device 120 and the electronic device 110 may transmit/receive signals through wired or wireless. The smart mirror device 140 may be disposed on a wall inside the home. For example, the smart mirror device 140 may be disposed in at least one or more places such as a front room, a living room, a room, and a kitchen inside the home.

According to an embodiment, the smart mirror device 140 includes at least one home appliance (eg, a heating/cooling device 151 , an oxygen generator 152 , a sound output device 153 , and a lighting device 154 ) existing inside the home. ), the air purifier 155 , the humidifier 156 , the dehumidifier 157 , and the TV 158 ) and may be connected by wire or wirelessly. The home appliance may include various devices that can be used while being disposed inside the home.

According to an embodiment, the smart mirror device 140 may receive information including power consumption, a location, and an identifier from each home appliance device. The information may be different depending on the type of home appliance.

According to an embodiment, the electronic device 110 recognizes the user, and based on the user's recognition, the electronic device 110 is located on one side (eg, upper side, lower side, left side, or right side) of the front side of the electronic device 110 . The disposed first camera may be activated. In addition, the electronic device 110 acquires a first image of the user through the first camera, and based on the acquired first image and a second image pre-stored in the electronic device 110 , authentication can be performed.

In addition, the electronic device 110 may transmit a signal for controlling the operation of the door opening and closing device 120 mounted on the door 130 to the door opening and closing device 120 based on the user's authentication. Also, the electronic device 110 may transmit an identifier for the user to the at least one smart mirror device 140 disposed inside the home based on the user's authentication. The identifier is an image previously stored in the user's image (eg, the electronic device 110 (eg, the storage unit 213 ) or a preview image acquired by the electronic device 110 (eg, the camera unit 216 ). ), a user name (eg, nickname) preset by the user, and various information such as the user's entry/exit time (eg, outing time, return time), and the like.

According to an embodiment, the door opening/closing device 120 may receive a signal for controlling the door opening/closing device 120 from the electronic device 110 disposed adjacent to the door 130 . In addition, the door opening and closing device 120 may unlock the door opening and closing device 120 based on the received control signal.

According to an embodiment, the door 130 may include a door (eg, a front door) through which a user can freely enter and exit. Opening or closing of the door 130 may be controlled through a lock of the door opening/closing device 120 .

The configuration of the home control system 100 shown in FIG. 1 is according to an embodiment, and the components of the home control system 100 are not limited to the embodiment shown in FIG. 1 , and as necessary, some components may be added, changed or deleted.

2 is a block diagram of an electronic device and a door opening and closing device according to an embodiment of the present invention.

Referring to FIG. 2 , the electronic device 110 according to an embodiment of the present invention includes a communication unit 210 , an input unit 211 , a display unit 212a , a speaker 212b , a storage unit 213 , and a processor 214 . , a camera unit 216 , and a sensor unit 219 .

The configuration of the electronic device 110 illustrated in FIG. 2 is according to an embodiment, and the components of the electronic device 110 are not limited to the embodiment illustrated in FIG. 2 , and some components may be added as necessary. , may be changed or deleted.

According to an embodiment, the communication unit 210 includes at least one component (eg, the input unit 211 , the display unit 212a , the speaker 212b , the storage unit 213 , and the processor included in the electronic device 110 ). 214 , the camera unit 216 , and the sensor unit 219) and at least one circuit capable of transmitting and receiving at least one signal or information through wired communication or wireless communication.

According to an embodiment, the communication unit 210 includes components included in the door opening and closing device 120 (eg, a speaker 222 , a battery 223 , a motor 224 , a locking unit 225 , and a processor 226 ). ), UVC sterilization unit 231, light emitting unit 232, and sensor unit 233), and components included in the smart mirror device 140 (eg, communication unit 310, input unit 311, display unit ( 312), the storage unit 313, the processor 314, the camera unit 316, the speaker 317, and the sensor unit 318) can transmit and receive at least one signal or information through wired communication or wireless communication. It may include at least one circuit with

According to an embodiment, it is obvious that the communication unit 210 may receive or transmit signals or data from various external devices.

According to an embodiment, the input unit 211 may include an interface capable of transmitting/receiving data to/from an external device (eg, a Universal Serial Bus (USB), an external hard drive (not shown), etc.). The input unit 211 may provide various types of information input by the user to the processor 214 . To this end, the input unit 211 may include a physical manipulation member such as a switch or a button, or may include an electrical manipulation member such as a touch key, a touch pad, and a touch screen. Alternatively, the input unit 211 may further include a microphone capable of receiving a user's voice signal.

According to an embodiment, the display unit 212a may display various types of information (eg, multimedia data or text data). The display unit 212a may display a result processed by the processor 214 , in the process of being processed, or to be processed. The display unit 212a may visually provide various information on the operating state of the door opening/closing device 120 through the electronic device 110 and may include a control circuit for displaying such various information.

According to an embodiment, the display unit 212a may include a touch circuitry for sensing a touch or a pressure sensor capable of measuring the intensity of a pressure applied to the touch.

According to an embodiment, the speaker 212b may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. The speaker 212b may output sound through a sound output device (eg, earphone or headphone). The speaker 212b may output a sound signal to the outside of the electronic device 110 . The speaker 212b may output information provided to a user entering and exiting the door by voice.

According to an embodiment, the speaker 212b may be disposed as a separate component in the electronic device 110 or may be included in an output unit (not shown) together with the display unit 212a.

According to an embodiment, the storage unit 213 may include a volatile memory or a non-volatile memory. For example, the storage unit 213 may store information, data, and programs necessary for the operation of the electronic device 110 or the door opening/closing device 120 . Accordingly, the processor 214 may perform a control operation to be described later with reference to information stored in the storage unit 213 . The storage unit 213 may store various platforms. The storage unit 213 is, for example, a flash memory type, a hard disk type, a multimedia card micro type, or a card type memory (eg, SD or XD memory, etc.), RAM (RAM), ROM (EEPROM, etc.), and may include at least one type of storage medium among USB memory.

According to an embodiment, the storage unit 213 includes at least one component of the electronic device 110 (eg, the communication unit 210 , the input unit 211 , the display unit 212a , the speaker 212b , and the storage unit). Various data (eg, software, application, acquired information, measured information, control signal, etc.) acquired or used by the 213 , the processor 214 , the camera unit 216 , and the sensor unit 219 ) , and commands related thereto. For example, the storage unit 213 may store a control signal or data received from the door opening and closing device 120 .

According to an embodiment, the storage unit 213 may store information about a pre-registered user (eg, an entrant or a family member). The storage unit 213 includes an image (eg, a face image, etc.) of a pre-registered user (eg, a visitor or a family member), a biometric image (eg, an image including vein information), and each user (eg, each family members) information (eg name, date of birth, gender, age, etc.). The storage unit 213 may store various information (eg, authentication success, authentication failure, sterilization of door opening/closing device, home address, password for opening a door, etc.) provided to a user entering and exiting the door.

According to an embodiment, the camera unit 216 may include a first camera 217a , a second camera 217b , and a lighting unit 218 . At least one camera of the camera unit 216 may be disposed at a position where it is easy to obtain an image of a user (or a body part of the user) entering and exiting the door. The camera unit 216 may include a first camera 217a (eg, a Red Green Blue (RGB) camera) capable of acquiring an image of a user even in a situation where the illumination by the lighting unit 218 is high. Also, the camera unit 216 may include at least one second camera 217b (eg, an RGB camera, an IR (infrared) camera, etc.).

According to an embodiment, at least one of the first camera and the second camera may acquire an image of the user (or a body part of the user) and transmit the acquired image to the processor 214 . The second camera 217b may include at least one IR camera capable of acquiring an image of a user even in low brightness (eg, low illuminance). The illumination unit 218 may include an element (eg, LED, IR pattern illumination) that emits illumination necessary to acquire an image through at least one of the first camera 217a and the second camera 217b. there is.

According to an embodiment, the camera unit 216 may obtain an image of the user through at least two IR cameras, respectively, and extract depth information of the user through the obtained two images. . In addition, the camera unit 216 may acquire a stereoscopic image (eg, a 3D image) of the user based on the extracted depth information. In addition, the processor 214 may perform an authentication procedure for the user through the obtained 3D image. The camera unit 216 may be disposed on one side (eg, upper side, lower side, left side, or right side) of the electronic device 110 . In this way, the electronic device 110 extracts depth information through at least two IR cameras of the camera unit 216 and performs authentication, thereby preventing illegal access through photos.

According to an embodiment, the camera unit 216 may acquire a face image of a user (or an occupant) through at least one camera, and extract depth information from the acquired face image. In addition, the processor 214 may generate an ID (eg, face ID) based on the face image (including depth information) acquired by the camera unit 216 . As such, the processor 214 may determine that the face images of various users acquired by the camera unit 216 are different from each other based on the depth information.

According to an embodiment, the sensor unit 219 may include at least one sensor capable of detecting a distance from a user entering and leaving a home, a user's movement, or the brightness of a place where a door is installed. For example, the sensor unit 219 may include a distance measuring sensor 219a, a motion detecting sensor 219b, and an illuminance sensor 219c.

According to an embodiment, the distance measuring sensor 219a may be disposed at a location (eg, at least one side of up, down, left, and right of the electronic device 110 ) capable of measuring the distance to the user. In addition, the motion detection sensor 219b may be disposed at a location capable of detecting the user's motion (eg, at least one side of up, down, left, and right of the electronic device 110 ). In addition, the illuminance sensor 219c may be disposed at a position capable of detecting the brightness of the outside of the door (eg, at least one side of up, down, left, and right of the electronic device 110 ).

According to an embodiment, the electronic device 110 may include a power supply unit (not shown). The power unit may be connected to an external power source (not shown) by wire or wirelessly, and may receive power from the external power source through such a wired or wireless connection. The power supply unit (not shown) adjusts a voltage to a voltage capable of operating the electronic device 110 under the control of the processor 126 , and adjusts the adjusted voltage to each of the voltages included in the electronic device 110 . Can be supplied as components.

According to an embodiment, the processor 214 runs software to at least one component connected to the processor 214 (eg, the communication unit 210, the input unit 211, the display unit 212a, the speaker 212b). , the storage unit 213 , the camera unit 216 , and the sensor unit 219) may be controlled based on wired communication or wireless communication. In addition, the processor 214 may perform various data processing and operations based on the wired communication or the wireless communication.

According to an embodiment, the processor 214 includes a communication unit 210 , an input unit 211 , a display unit 212a , a speaker 212b , a storage unit 213 , a camera unit 216 , and a sensor unit 219 . A command or data received from the storage unit 213 may be loaded and processed, and the processed data may be stored in the storage unit 213 . Alternatively, the processor 214 may display the processed data through the display unit 212a or output it through the speaker 212b.

According to an embodiment, when at least one user (eg, each family member) performs authentication (eg, authentication using a QR code) using a portable device (eg, a smart phone), the processor 214 is configured to An identifier (eg, a phone number) of the portable device may be stored, and the portable device may be determined as an authenticated device. In addition, the processor 214 may selectively (eg, a user's request) establish a communication channel through the portable device (eg, a smart phone) of at least one authenticated user (eg, a family member) and the communication unit 210 . there is. In addition, the processor 214 may receive a command from a remotely located portable device through such a communication channel and perform a function corresponding to the received command.

According to an embodiment, the processor 214 activates the first camera 217a of the camera unit 216 based on the user's recognition, and uses the recognized first camera to display the user's first image. can be obtained. The processor 214 may measure a distance between a user (eg, an occupant) adjacent to the door 130 and the electronic device 110 through the distance measuring sensor 219a. When the measured distance is within a predetermined distance (eg, 2 m), the processor 214 may determine that the user is going to enter and exit the door 130 . When it is determined that the user is going to enter and exit the door 130 , the processor 214 may execute an operation of the home control system 100 (eg, the door opening/closing device 120 ).

According to an embodiment, the processor 214 may authenticate the user based on the acquired first image and the second image pre-stored in the storage unit 213 . The processor 214 obtains a first image (eg, a first face image) of a user (eg, Gildong Hong) through the first camera 217a (eg, an RGB camera), and the acquired first image and a second image (eg, a second face image) of the user (eg, Gil-dong Hong) pre-stored in the storage unit 213 may be compared to authenticate the user (eg, Gil-dong Hong). The processor 214 extracts the feature points of the first image (eg, the length of the eyebrows compared to the face ratio, the distance between the eyebrows, the mouth size, the nose size, the eye size, etc.), and the extracted feature points and the feature points of the second image (eg, eyebrow length compared to face ratio, distance between eyebrows, mouth size, nose size, eye size, etc.) may be compared to perform authentication (eg, primary authentication) for the user (eg, Gil-dong Hong).

According to an embodiment, when the authentication for the user is successful, the processor 214 may display a message about the success of the authentication through the display unit 212a or output it as a voice through the speaker 212b. .

According to an embodiment, when authentication (eg, primary authentication) for the user is unsuccessful, the processor 214 displays a message about the failure of the authentication through the display unit 212a or the speaker 212b ) to output audio. The processor 214, for example, when the authentication fails, a guide message (eg, please stand in front of the camera) so that the user is positioned in the first camera 217a through the display unit 212a, or, Alternatively, the guide message may be output as voice through the speaker 212b. The processor 214 may perform the authentication procedure a predetermined number of times (eg, three times). When the processor 214 drives the timer 215 to obtain information about the user (eg, a face image, or a biometric information image) within a predetermined time period (eg, 10 seconds) or authentication is not performed , a message indicating that authentication cannot be performed may be displayed through the display unit 212a. Alternatively, the processor 214 may output the message as voice through the speaker 212b.

According to an embodiment, the processor 214 transmits, to the communication unit, a signal for controlling the operation of the door opening and closing device 120 mounted on the door 130 based on the user's authentication (eg, primary authentication). It can be transmitted to the door opening and closing device 120 through 210 . The processor 214 transmits a signal for unlocking (or setting) the door opening and closing device 120 to the communication unit 210 based on the success of authentication (eg, primary authentication) of the user (eg, Gil-dong Hong). ) through the door opening and closing device 120 (eg, the communication unit 221).

According to an embodiment, when authentication (eg, primary authentication) for the user (eg, Gil-dong Hong) is successful through the first image and the second image, the processor 214 performs the camera unit 216 of the second camera 217b may be activated. In addition, the processor 214 displays a message informing the movement of a body part (eg, palm) of the user (eg, Gil-dong Hong) to the position of the activated second camera 217b through the display unit 212a or , or may be output as voice through the speaker 212b. The second camera 217b may include two IR cameras.

According to an embodiment, while outputting the message, the processor 214 may display a guide line for guiding an area in which the body part is to be located through the display unit 212a. The processor 214 is positioned at an optimal position for photographing the body part so as to obtain an image of the user's (eg Hong Gil-dong) body part (eg, palm) through the second camera 217b. A guide line may be displayed through the display unit 212a to be placed.

According to an embodiment, the processor 214 acquires a preview image of the body part through the first camera 217a while the guide line is being displayed through the display unit 212a, and the display unit (212a) may be indicated. The processor 214 may acquire a preview image of the body part in real time, and display the preview image acquired in real time on the display unit 212a.

According to an embodiment, the processor 214 may display the preview image acquired in real time and the guide line together on the display unit 212a.

According to an embodiment, the processor 214 may calculate the interval between the displayed guide line and the displayed preview image in real time. Further, based on the interval calculated in real time, the processor 214 may sound a louder warning sound through the speaker 212b as the distance increases, and may sound a small warning sound as the distance decreases.

According to an embodiment, the processor 214 may display visual information according to whether the preview image matches the displayed guide line through the display unit 212a. The visual information may include information (eg, color, arrow, gradation effect, etc.) that allows the user to move a body part to a position corresponding to the guide line. The user may move the body part so that an image of the body part (eg, palm) can be easily obtained through the second camera 217b through the visual information. The second camera 217b may include at least two IR cameras.

According to an embodiment, the processor 214 may acquire information about veins in the body part (eg, palm) through the second camera 217b. The processor 214 acquires (eg, at least two) images of the user's body part through the second camera 217b (eg, at least two IR cameras), and stores the two acquired images. Through this, depth information about the user may be extracted. In addition, the processor 214 may acquire a stereoscopic image (eg, a 3D image) of the user based on the extracted depth information. In addition, the processor 214 may obtain information (eg, thickness, direction, etc. of veins) on the veins in the body part (eg, the palm) through the second camera 217b.

According to an embodiment, the processor 214 is configured to authenticate the user (eg, secondary authentication) can be performed. The processor 214 acquires a third image (eg, a first palm image) of a user (eg, Gildong Hong) through the second camera 217b (eg, at least two IR cameras), and the acquisition The third image (eg, Hong Gil-dong) previously stored in the storage unit 213 is compared with a fourth image (eg, second palm image) for the user (eg, Hong Gil-dong) to authenticate the user (eg, Hong Gil-dong) (eg: secondary authentication).

According to an embodiment, the processor 214 extracts feature points (eg, vein vessel thickness, blood vessel direction, shape, degree of bending, etc.) of the third image, and extracts the extracted feature points and the feature points of the fourth image ( For example, authentication (eg, secondary authentication) for the user (eg, Gil-dong Hong) may be performed by comparing venous vessel thickness, vessel direction, shape, degree of bending, etc.).

According to an embodiment, the processor 214 transmits a signal for controlling the operation of the door opening and closing device 120 mounted on the door 130 to the communication unit based on the user's authentication (eg, secondary authentication). It can be transmitted to the door opening and closing device 120 through 210 . The processor 214 transmits a signal for unlocking (or setting) the door opening and closing device 120 to the communication unit 210 based on the success of authentication (eg, secondary authentication) of the user (eg, Hong Gil-dong). ) through the door opening and closing device 120 (eg, the communication unit 221).

According to an embodiment, the processor 214 may identify whether a signal for closing the door 130 is received from the door opening and closing device 120 through the communication unit 210 . When the user enters and exits through the door 130 , the door 130 is opened and closed. The sensor unit 233 of the door opening/closing device 120 may detect the opening/closing of the door, and the door opening/closing device 120 may transmit a signal for opening/closing the door to the electronic device 110 through the communication unit 221 . . Upon receiving the signal for opening and closing the door, the electronic device 110 may detect the opening and closing of the door 130 .

According to an embodiment, the processor 214 transmits a signal for controlling the sterilization of the door opening and closing device 120 through the communication unit 210 based on the reception of a signal for closing the door 130 . It can be transmitted to the door opening and closing device 120 . The processor 214 may identify that the door 130 has been opened or closed based on the signal received from the door opening and closing device 120 . When it is identified that the door 130 has been opened and closed, the processor 214 sends a control signal for sterilizing the door opening and closing device 120 (eg, the handle 230 ) to the door opening and closing device 120 . can be transmitted Alternatively, when the door 130 is opened and closed, the door opening and closing device 120 (eg, the processor 226 ) may sterilize the handle 230 through the UVC sterilization unit 231 by itself. .

According to an embodiment, the processor 214 may identify the brightness around at least one of the electronic device 110 and the door opening/closing device 120 through the illuminance sensor 219c. The processor 214 may select and operate at least one of the first camera 217a and the second camera 217b based on the identified ambient brightness. For example, the processor 214 may operate the first camera 217a when the brightness is equal to or greater than a reference value, and operate the second camera 217b when the brightness is less than the reference value. Alternatively, the processor 214 may acquire an image of the user by operating the first camera 217a and the second camera 217b regardless of brightness.

According to an embodiment, the processor 214 is configured to provide at least one operating state of the electronic device 110 , at least one operating state of the door opening and closing device 120 , and at least one related to opening and closing of the door 130 . Some information may be displayed on the display unit 212a. The processor 214 may display various information regarding entry and exit through the smart door, such as user detection, camera operation state, image acquisition, authentication, and door opening/closing control, on the display unit 212a.

Also, referring to FIG. 2 , the door opening and closing device 120 according to an embodiment of the present invention may include a body part 220 and a handle part 230 . The body 220 may include a communication unit 221 , a speaker 222 , a battery 223 , a motor 224 , a locking unit 225 , and a processor 226 . The handle unit 230 may include a UVC sterilization unit 231 , a light emitting unit 232 , and a sensor unit 233 .

The configuration included in the body 220 and the handle 230 of the door opening and closing device 120 shown in FIG. 2 is according to an embodiment, and the body 220 and the handle of the door opening and closing device 120 are Components of 230 are not limited to the embodiment shown in FIG. 2 , and some components may be added, changed, or deleted as necessary.

According to an embodiment, the communication unit 221 includes at least one component (eg, a speaker 222 , a battery 223 , a motor 224 , and a locking unit 225 ) included in the door opening and closing device 120 . , the processor 226, the UVC sterilization unit 231, the light emitting unit 232, and the sensor unit 233) and at least one circuit capable of transmitting and receiving at least one signal or information through wired communication or wireless communication may include

According to an embodiment, the communication unit 221 includes components included in the electronic device 110 (eg, the input unit 211 , the display unit 212a , the speaker 212b , the storage unit 213 , and the processor 214 ). , the camera unit 216 , and the sensor unit 219) and at least one circuit capable of transmitting and receiving at least one signal or information through wired communication or wireless communication.

According to an embodiment, it is obvious that the communication unit 221 may receive or transmit signals or data from various external devices.

According to an embodiment, the speaker 222 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. The speaker 222 may output sound through a sound output device (eg, earphone or headphone). The speaker 222 may output a sound signal to the outside of the door opening and closing device 120 . The speaker 222 may output information provided to a user entering and exiting the door by voice.

According to an embodiment, the speaker 222 may be included in the door opening/closing device 120 or may be disposed as a separate component.

According to an embodiment, the battery 223 may supply power to at least one component of the door opening/closing device 120 . The battery 223 may include, for example, a non-rechargeable primary battery and a rechargeable secondary battery. One side of the battery 223 may include a USB terminal (not shown) capable of receiving power from an external power source or supplying power to the door opening/closing device 120 .

According to an embodiment, the motor 224 may control the movement of the handle part 230 so that the handle part 230 is spaced apart from the body part 220 . For example, when a signal based on the success of user authentication is received from the electronic device 110 through the communication unit 221 , the processor 226 controls the handle unit 230 through the motor 224 to set the watch. It is possible to control the operation of the handle part 230 so as to be separated from the body part 220 while rotating in the direction or counterclockwise direction. The handle part 230 may be separated from the body part 220 so that the user can easily grip it under the control of the motor 224 .

According to an embodiment, the lock 225 prevents the door 130 from being opened by protruding a member (not shown) for preventing the door 130 from being opened toward the wall in close contact with the door 130 . can be prevented

According to an embodiment, the UVC sterilization unit 231 may sterilize (or disinfect) the door opening and closing device 120 by emitting ultraviolet rays having a short wavelength (eg, about 100 nm to 280 nm). Alternatively, the UVC sterilization unit 231 may sterilize (or disinfect) the handle unit 230 of the door opening and closing device 120 . The UVC sterilizer 231 may be disposed on one side of the door opening and closing device 120 . Alternatively, the UVC sterilization unit 231 may be configured to sterilize the portion where the user grips the handle portion 230 of the door opening and closing device 120 , so that the body portion 220 and the handle portion 230 come into contact with each other. It can be placed in a part.

According to an embodiment, the light emitting unit 232 may include at least one light emitting device (eg, a light emitting diode (LED)) that emits light in various colors. The light emitting unit 232 may emit light under the control of the processor 226 to visually indicate the operating state of the body 220 and/or the handle 230 . The light emitting part 232 may emit light in different colors according to different operations of the body part 220 and/or the handle part 230 .

According to one embodiment, when the handle part 230 is spaced apart from the body part 220 so that the user can easily grip the handle part 230 (eg, the handle part 230 is the body part ( 220) at a predetermined angle (eg, when spaced apart from 30 ^o ), the LED disposed on one side (eg, border) of the handle 230 has a predetermined color (eg, white) under the control of the processor 226 . can be emitted as For example, when the handle unit 230 is being disinfected, an LED disposed on one side (eg, an edge) of the handle unit 230 may emit light in a certain color (eg, purple).

According to an embodiment, the sensor unit 233 may include at least one sensor capable of detecting a distance from a user entering and leaving a home, a user's movement, or the brightness of a place where a door is installed. For example, the sensor unit 219 may include a distance measuring sensor 219a, a motion detecting sensor 219b, and an illuminance sensor 219c. Alternatively, the sensor unit 219 may include a proximity sensor 233a.

According to an embodiment, the sensor unit 233 (eg, proximity sensor 233a) includes the body unit 220 and the handle unit for the safety of the user's hand who wants to grip the handle unit 230 . It may be disposed in a spaced gap between the 230 .

For example, when an object (eg, a user's hand) is detected before the handle part 230 is completely separated from the body part 220 , the sensor part 233 (eg, a proximity sensor 233a) )) may detect the object and provide a signal based on the detection of the object to the processor 226 .

For example, when an object (eg, a user's hand) is detected before the handle part 230 is completely coupled to the body part 220 in a state in which the handle part 230 is spaced apart from the body part 220 , the The sensor unit 233 (eg, the proximity sensor 233a) may detect the object and provide a signal based on the detection of the object to the processor 226 . For example, when a signal based on the detection of the object is received, the processor 226 may temporarily stop the operation of the door opening/closing device 120 .

According to an embodiment, the door opening and closing device 120 may further include a power supply unit (not shown). The power supply unit (not shown) adjusts a voltage to a voltage at which the door opening and closing device 120 can be operated under the control of the processor 126 , and applies the adjusted voltage to the voltage included in the door opening and closing device 120 . It can be supplied as individual components.

According to an embodiment, the power supply unit 319 may include an interface (not shown) for a wired connection or a wireless connection with an external power source. The interface (not shown) may include a high-definition multimedia interface (HDMI), a universal serial bus (USB), an optical interface, or a D-subminiature (D-sub). The interface (not shown) may include, for example, a mobile high-definition link (MHL) interface, a secure digital (SD) card/multi-media card (MMC) interface, or an infrared data association (IrDA) standard interface. there is.

According to an embodiment, the processor 226 drives at least one component (eg, a communication unit 221 , a speaker 222 , a battery 223 , a motor 224 ) connected to the processor 226 by driving software. , the lock unit 225 , the UVC sterilization unit 231 , the light emitting unit 232 , and the sensor unit 233) may be controlled based on wired communication or wireless communication. In addition, the processor 226 may perform various data processing and operations based on the wired communication or the wireless communication.

According to an embodiment, the processor 226 includes a communication unit 221 , a speaker 222 , a battery 223 , a motor 224 , a locking unit 225 , a UVC sterilization unit 231 , and a light emitting unit 232 . , and may process commands or data received from the sensor unit 233 , and transmit the processed data to the electronic device 110 . Alternatively, the processor 226 may transmit the processed data to the electronic device 110 or output it through the speaker 222 .

According to an embodiment, the processor 226 may receive a control signal for controlling the door opening and closing device 120 from the electronic device 110 disposed adjacent to the door 130 through the communication unit 221 . there is. The control signal may include various signals for controlling an operation (eg, door opening or door closing) of the door opening and closing device 120 . Alternatively, the control signal may include various signals for controlling an operation (eg, opening or closing of the handle) of the handle 230 of the door opening and closing device 120 .

According to an embodiment, the processor 226 may unlock (or lock) the door opening and closing device 120 based on the received control signal. The processor 226 obtains a signal transmitted through the communication unit 210 of the electronic device 110 through the communication unit 221 , and locks or locks the door opening and closing device 120 based on the obtained signal. Opening can be controlled. The communication unit 210 of the electronic device 110 and the communication unit 221 of the door opening and closing device 120 may be connected by wire or wirelessly.

According to an embodiment, the processor 226 may control the motor 224 so that the handle 230 rotates and protrudes from the body 220 based on the received control signal.

According to an embodiment, the processor 226 obtains a signal for closing or opening the door 130 through the sensor unit 233 (eg, an opening/closing detection sensor (not shown)), and the door 130 Closed or opened of can be identified. When the closing of the door 130 is sensed, the processor 226 may activate the locking unit 225 to lock the door opening/closing device 120 . In addition, when the door opening and closing device 120 is switched to the locked state through the locking unit 225 , the processor 226 generates a light emitting unit 232 (eg, at least one Ultraviolet C Light Emitting Diode (UVC LED)). Through this, the door opening and closing device 120 (eg, the handle part 230) can be sterilized. The handle unit 230 includes at least one UVC LED, and the at least one UVC LED is positioned at an appropriate location for disinfecting (eg, sterilizing) the area where the user's hand gripping the handle unit 230 is located. can be placed.

According to an embodiment, the processor 226 may control the at least one light emitting device so that the at least one light emitting device emits light in different colors based on the opening and closing of the door 130 . The processor 226 may emit light in different colors from at least one light emitting device according to the opening or closing of the door 130 .

According to one embodiment, the processor 226 drives the timer 227 to grip the handle 230 when the user does not grip the handle 230 within a predetermined time (eg, 10 seconds). A message indicating not to do may be displayed through the speaker 222 . Alternatively, the processor 226 may emit light through the light emitting unit 232 (eg, an LED) indicating that the handle 230 is not gripped.

3 is a block diagram of an electronic device and a smart mirror device according to an embodiment of the present invention.

The electronic device 110 illustrated in FIG. 3 is substantially the same as the electronic device 110 illustrated in FIG. 2 . At least one function or operation performed by the electronic device 110 illustrated in FIG. 2 may be performed by the electronic device 110 illustrated in FIG. 3 .

Referring to FIG. 3 , the smart mirror device 140 according to an embodiment of the present invention includes a communication unit 310 , an input unit 311 , a display unit 312 , a storage unit 313 , a processor 314 , and a camera unit ( 316 ), a speaker 317 , a sensor unit 318 , and a power supply unit 319 may be included.

The configuration of the smart mirror device 140 shown in FIG. 3 is according to an embodiment, and the components of the smart mirror device 140 are not limited to the embodiment shown in FIG. 3 , and as necessary, some components may be added, changed or deleted.

According to an embodiment, the communication unit 310 includes at least one component included in the smart mirror device 140 (eg, the input unit 311 , the display unit 312 , the storage unit 313 , the processor 314 , The camera unit 316, the speaker 317, the sensor unit 318, and the power supply unit 319) may include at least one circuit capable of transmitting and receiving at least one signal or information through wired or wireless communication. there is.

According to an embodiment, the communication unit 310 may transmit and receive at least one signal or information through wired communication or wireless communication with a component included in the electronic device 110 or a component of the door opening and closing device 120 . It may include at least one circuit with

According to an embodiment, it is obvious that the communication unit 310 may receive or transmit signals or data from various external devices.

According to an embodiment, the input unit 311 may include an interface capable of transmitting/receiving data to/from an external device (eg, a Universal Serial Bus (USB), an external hard drive (not shown), etc.). The input unit 311 may provide various types of information input by the user to the processor 314 . To this end, the input unit 311 may include a physical manipulation member such as a switch or a button, or may include an electrical manipulation member such as a touch key, a touch pad, and a touch screen. Alternatively, the input unit 311 may further include a microphone capable of receiving a user's voice signal.

According to an embodiment, the display unit 312 may display various types of information (eg, multimedia data or text data, etc.). The display unit 312 may display a result processed, in process, or to be processed by the processor 314 . The display unit 312 may visually provide various information about the operating state of the door opening and closing device 120 or at least one home appliance through the smart mirror device 140, and a control circuit for displaying such various information. may include

According to an embodiment, the display unit 312 may include a touch circuitry for sensing a touch or a pressure sensor capable of measuring the intensity of a pressure applied to the touch.

According to an embodiment, the display unit 312 may operate in a first mode (eg, mirror mode), a second mode (eg, smart mirror mode), and a third mode (eg, display mode). The first mode is a mode for completely reflecting an object like a general mirror, the second mode is a mode for displaying at least some information based on the first mode, and the third mode is the processor 314 . In this mode, only at least part of information by As such, the display unit 312 may include various panels (eg, a polarization layer, a conductive layer, an insulating layer, a display panel, a touch sensing panel, etc.) to support these various modes.

According to an embodiment, the display unit 312 may operate in various modes by adjusting transmittance and reflectance. For example, the display unit 312 reflects the image of an object (such as a user) located in front of the smart mirror device 140 under the control of the processor 314 to display a mirror image (eg, the first 1 mode state), various information generated by the processor 314 (eg, letters, numbers, images, etc.) and various information received from the electronic device 120 (eg, preview image, images, characters, numbers, etc.) may be operated in a mode (eg, a second mode) for displaying .

According to an embodiment, the display unit 312 may include a mirror 312a and a transparent display unit 312b. The transparent display unit 312b may be disposed on the front surface of the mirror 312a to display a mirror image of a user on the front surface of the transparent display unit 312b through the mirror 312a (first mode). And, as the transmittance and reflectance of the transparent display 312b are adjusted by the processor 314, the display unit 312 displays a mirror image through the mirror 312a while displaying various information through the transparent display unit 312b. can be displayed

According to an embodiment, the transparent display unit 312b may include an optical member that selectively transmits and reflects light to improve light efficiency. The transparent display unit 312b may have a structure in which a plurality of films having the same X-axis refractive index and different Y-axis refractive indexes are alternately stacked. The transparent display unit 312b may transmit light as a transmission axis along an X axis having the same refractive index, and may reflect light as a reflection axis through a Y axis having a different refractive index. For example, the transparent display unit 312b may transmit P waves among light components and continuously reflect S waves.

According to an embodiment, the transparent display unit 312b may be applied from a small portable display to a large display.

According to an embodiment, the display area 312 may be divided into a first area and a second area, so that the display area is displayed in different areas of the transparent display 312b according to the importance and/or type of content. can be implemented. According to an embodiment to which the present invention is applied, the display area of the transparent display 312b may be divided into various shapes.

According to an embodiment, the storage unit 313 may include a volatile memory or a non-volatile memory. For example, the storage unit 313 may store information, data, programs, etc. necessary for the operation of the smart mirror device 140 or the electronic device 110 . Accordingly, the processor 314 may perform a control operation to be described later with reference to information stored in the storage 313 . The storage unit 313 may store various platforms. The storage unit 313 is, for example, a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (eg, SD or XD memory, etc.), RAM (RAM), ROM (EEPROM, etc.), and may include at least one type of storage medium among USB memory.

According to an embodiment, the storage unit 313 includes at least one component of the smart mirror device 140 (eg, the communication unit 310 , the input unit 311 , the display unit 312 , the storage unit 313 , Various data (eg, software, applications, acquired information, measured information, control signals, etc.) acquired or used by the processor 314 , the camera unit 316 , the speaker 317 , and the sensor unit 318 ) ), and instructions related thereto. For example, the storage unit 313 may store a control signal or data received from the electronic device 110 .

According to an embodiment, the storage unit 313 may store information about a pre-registered user (eg, an entrant or a family member). The storage unit 313 includes an image (eg, a face image, etc.) of a pre-registered user (eg, a visitor or a family member), a biometric information image (eg, an image including vein information), and each user (eg, each family members) information (eg name, date of birth, gender, age, etc.). The storage unit 313 stores various information provided to the user in the home (eg, authentication success, authentication failure, preview image obtained from the electronic device 110, home address, password for opening a door, etc.) can

According to an embodiment, the camera unit 316 may include at least one camera. The camera unit 316 may acquire an image of an object or a user in the home. The camera unit 316 may include an intelligent image camera (or smart image intelligence camera) in which an artificial intelligence chip (eg, DQ1) is embedded. Through such an intelligent video camera, the face of the occupant can be recognized and the presence of the occupant can be detected.

According to an embodiment, the image intelligence camera is a camera to which a deep learning-based algorithm that can be operated on an artificial intelligence chip (eg, DQ1) is applied, user registration and recognition, user access detection, occupancy recognition, user's ear recognition, and self-monitoring photography We can provide personalized services such as functions. Such an intelligent image camera may be included in the camera unit 316 of the smart mirror device 140 . In addition, the processor 314 transmits information (eg, images) collected from at least one occupant in the home through the image intelligence camera through the communication unit 210 to a server (not shown) (eg, ThinQ artificial intelligence server). ), the server (not shown) may generate information for the convenience of occupants, and then transmit it back to the smart mirror device 140 . Through this, the smart mirror device 140 can provide a more convenient service to occupants.

According to an embodiment, the camera unit 316 is a camera capable of acquiring an image of the user even in a situation in which illumination is high due to lighting (eg, a red green blue (RGB) camera), and an image in a low illumination situation. It may include an IR (infrared) camera capable of acquiring . The camera unit 316 may include a device (eg, LED, IR pattern illumination) that emits light necessary to acquire an image.

According to an embodiment, the speaker 317 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. The speaker 317 may output sound through a sound output device (eg, earphone or headphone). The speaker 317 may output an acoustic signal to the outside of the smart mirror device 140 . The speaker 317 may output information provided to a user existing inside the home by voice.

According to an embodiment, the speaker 317 may be disposed as a separate component in the smart mirror device 140 or may be included in an output unit (not shown) together with the display unit 312 .

According to one embodiment, the sensor unit 318 is at least capable of detecting the distance from the user approaching the smart mirror device 140, the user's movement, or the brightness of the place where the smart mirror device 140 is installed. It may include one sensor. For example, the sensor unit 318 may include a distance measuring sensor (not shown), a motion detection sensor (not shown), and an illuminance sensor (not shown).

According to an embodiment, the power supply unit 319 may be connected to an external power source (not shown) by wire or wirelessly, and may receive power from the external power source through such a wired or wireless connection. The power supply unit 319 adjusts a voltage to a voltage at which the smart mirror device 140 can be operated under the control of the processor 126 , and adjusts the adjusted voltage to each of the voltages included in the smart mirror device 140 . It can be supplied as a component of

According to an embodiment, the power supply unit 319 may include an interface (not shown) for a wired connection or a wireless connection with an external power source. The interface (not shown) may include a high-definition multimedia interface (HDMI), a universal serial bus (USB), an optical interface, or a D-subminiature (D-sub). The interface (not shown) may include, for example, a mobile high-definition link (MHL) interface, a secure digital (SD) card/multi-media card (MMC) interface, or an infrared data association (IrDA) standard interface. there is.

According to an embodiment, the processor 314 drives software to operate at least one component (eg, a communication unit 310 , an input unit 311 , a display unit 312 , and a storage unit 313 connected to the processor 314 ). ), the camera unit 316 , the speaker 317 , the sensor unit 318 , and the power supply unit 319) may be controlled based on wired communication or wireless communication. In addition, the processor 314 may perform various data processing and operations based on the wired communication or the wireless communication.

According to an embodiment, the processor 314 includes a communication unit 310 , an input unit 311 , a display unit 312 , a storage unit 313 , a camera unit 316 , a speaker 317 , a sensor unit 318 , And a command or data received from the power supply unit 319, etc. may be loaded into the storage unit 313 for processing, and the processed data may be stored in the storage unit 313 . Alternatively, the processor 314 may display the processed data through the display unit 212 or output it through the speaker 317 .

According to an embodiment, an artificial intelligence chip (not shown) (eg, DQ1) may be embedded in the smart mirror device 140 or at least one home appliance device. Alternatively, the algorithm for the artificial intelligence chip may be implemented by the processor 314 . The artificial intelligence chip is a processor that mimics a human brain neural network, and can support a deep learning algorithm that analyzes, recognizes, infers, and judges various data by itself. Through this, the smart mirror device 140 (eg, the processor 314) of the present invention can recognize and distinguish a space, a location, an object, a user, etc. through the camera 316, and the characteristics of the user's voice or ambient noise can recognize In addition, at least one home appliance may also have such an artificial intelligence chip embedded therein. Through this, the smart mirror device 140 (eg, the processor 314 ) may provide an AloT (Aritificial Intelligence Internet of Things) platform while interworking with home appliances in the home through an artificial intelligence chip.

According to an embodiment, the processor 314 receives an identifier for a user entering the home from the electronic device 110 located outside the home through the communication unit 310, and based on the received identifier, the The user's image and information on the user's entry may be displayed through the transparent display 312b of the communication unit 310 . The identifier is an image previously stored in the user's image (eg, the electronic device 110 (eg, the storage unit 213 ) or a preview image acquired by the electronic device 110 (eg, the camera unit 216 ). ), a user name (eg, nickname) preset by the user, and various information such as the user's entry/exit time (eg, outing time, return time), and the like.

According to an embodiment, the processor 314 is configured to receive information about the user from the storage 313 (eg, an image pre-stored in the storage 313 , an image previously set by the user, based on the reception of the identifier). Various information such as a user name (eg, nickname) and the user's entry/exit time (eg, outing time, return time) may be acquired.

According to an embodiment, when an identifier for a user (eg, father) entering the home is received from the electronic device 110 located outside the home, the processor 314 is configured to display at least one home appliance located inside the home. information (eg, home appliance device name, identifier, location, etc.) can be obtained through the AI chip. In addition, the processor 314 is configured to include at least one home appliance (eg, an air conditioning unit 151 , a TV 158 , a lighting device ( ) corresponding to the received identifier among the obtained information on the at least one home appliance device 154)) can be identified. And, the processor 314 is based on the preset condition of the identified at least one home appliance (eg, the air conditioning unit 151, the TV 158, the lighting unit 154), the identified at least one The operation of home appliances can be controlled through an artificial intelligence chip.

According to an embodiment, the processor 314 determines whether the identifier of the user entering the home is included in the identifier stored in the storage 313, and stores the identifier of the user entering the home. If it is not included in the identifier stored in the unit 313 , an image of a user entering the home may be displayed as a pop-up on the display unit 312 .

According to an embodiment, the identifier is based on at least one of the success of the first authentication by the recognition of the user's face by the electronic device 110 and the success of the second authentication by the user's vein information. Based on it, it may be received from the electronic device 110 .

According to an embodiment, the processor 314 receives weather information, standby information, and power consumption information of at least one home appliance in the home through the communication unit 310, and receives the received weather information, standby At least a portion of information and power consumption information may be displayed through the display unit 312 (eg, the transparent display 312b). For example, the processor 314 receives weather information (eg, temperature, humidity, probability of rain, probability of snowing, etc.) from an external device (eg, a weather server, a standby server, etc.) through the communication unit 310; and atmospheric information (eg, fine dust concentration, yellow sand concentration, air pollution degree, etc.).

According to an embodiment, the processor 314 includes at least one home appliance (eg, a heating/cooling device 151 , an oxygen generator 152 , a sound output device 153 , a lighting device 154 , The power consumed in real time (or by period) from the air purifier 155 , the humidifier 156 , the dehumidifier 157 , and the TV 158 , the identifier of each household appliance, and the location of each household appliance (eg, 1 floor, second floor, master bedroom, study room, kitchen, living room, etc.) may receive information. For example, the processor 314 may monitor the electricity usage of each home appliance through at least one home appliance and various wireless communication protocols (eg, Zigbee, WiFi, Bluetooth, WLAN, etc.). In addition, the processor 314 may minimize energy consumption by controlling each home appliance through an artificial intelligence chip.

According to an embodiment, the processor 314 displays on the display unit 310 (eg, the transparent display 312b) at least a portion of weather information, standby information, and power consumption information of each home appliance received from the outside. can do.

According to an embodiment, the processor 314 may store the identifier for at least one user transmitted from the electronic device 110 in the storage unit 313 . The at least one user may include a family member living a residential life at home. In addition, the processor 314 displays information on at least one user located in the home (eg, currently present in the home) among users corresponding to the stored identifier on the display unit 312 (eg, a transparent display). (312b)).

According to an embodiment, the processor 314 may display information on each of a plurality of users currently present in the home through the display unit 312 (eg, the transparent display 312b). The information about the user may include at least one of an image of the user, an identifier of the user, and a location of the user.

According to an embodiment, the processor 314 is configured to provide information (eg, an image) about at least one user (eg, a father) among a plurality of users displayed on the display unit 312 (eg, the transparent display 312b). ) is selected. The processor 314 identifies a location of a portable device (not shown) (eg, a smart phone) of the corresponding user (eg, father) based on the identified selection, and determines the location of the identified portable device (not shown). A location (eg, a bedroom) of the user (eg, father) may be determined based on the location.

According to an embodiment, when information (eg, an image) about a user (eg, father) is selected while information on at least one home appliance located inside the home is stored in the storage unit 313 , the processor 314 may identify at least one home appliance device corresponding to the user (eg, father) from among the stored information on the at least one home appliance device. In addition, the processor 314 controls the operation of the identified at least one home appliance based on a preset condition (eg, may be different depending on the type of each home appliance) of the identified at least one home appliance. can do.

According to an embodiment, the processor 314 communicates for transmitting and receiving signals or data to and from a portable device (eg, a smart phone) of the user (eg, father) based on a selection of the user (eg, father) A channel may be set through the communication unit 310 . In addition, the processor 314 may transmit/receive signals or data to and from the portable device (not shown) through the communication unit 310 based on the set communication channel.

According to an embodiment, an input requesting authentication for a user (eg, a guest) not included in the identifier (eg, family member's identifier) stored in the storage unit 313 is displayed on the display unit 312 (eg, transparent) When received through the display 312b), the processor 314 generates an image (eg, a face image) of a user (eg, a guest) requesting the authentication through the camera unit 316 based on the received input. can be obtained.

According to an embodiment, the processor 314 obtains (or receives an input) an identifier (eg, a nickname) for the user requesting the authentication through the display unit 312 (eg, the transparent display 312b). , performs registration (or authentication) for a user requesting the authentication based on the obtained image and identifier, and displays the obtained image and identifier through the display unit 312 (eg, transparent display 312b) can be displayed In this way, the processor 314 may not only register a new user, but also cancel or delete it.

According to an embodiment, the processor 314 may detect a user's approach through the sensor unit 318 and calculate a distance between the user and the smart mirror device based on the sensed user's approach. For example, if the distance between the smart mirror device 140 and the user is within a predetermined distance (eg, 50 cm), the processor 314 may determine that the user is approaching to operate the smart mirror device 140 . can be judged as The processor 314 may determine an operation mode (eg, any one of a first mode, a second mode, a third mode, and a combination mode thereof) of the display unit 312 according to whether the user approaches.

According to an embodiment, the processor 314 may control the operation of the smart mirror device 140 (eg, the display unit 312 ) based on the calculated distance. For example, when the calculated distance is within a predetermined distance (eg, 50 cm), the processor 314 operates the display unit 312 in a mirror mode, and the calculated distance is within a predetermined distance (eg, 50 cm). ), the display unit 312 may be operated in a smart mirror mode or a display mode.

According to an embodiment, when the calculated distance is greater than a predetermined distance (eg, 50 cm), the processor 314 displays weather information and standby information while the display unit operates in a mirror mode through the mirror. The display unit may be controlled to operate in a standby mode. Alternatively, when the calculated distance is less than the predetermined distance (eg, 50 cm), the processor 314 displays at least a portion of the information by the processor 314 on the display unit 312 on the transparent display 312b. The display unit 312 may be controlled to operate in a smart mirror mode displayed through .

4 is a flowchart illustrating a process of authenticating a user in an electronic device according to an embodiment of the present invention.

Hereinafter, a process of authenticating a user in an electronic device according to an embodiment of the present invention will be described in detail with reference to FIG. 4 .

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may detect an approaching user ( S410 ). The electronic device 110 (eg, the processor 214 ) may detect whether the user approaches the door 130 . The electronic device 110 (eg, the processor 214 ) is a user (eg, a user who approaches the door 130 ) through a sensor unit 219 (eg, a distance measuring sensor 219a or a motion detection sensor 219b ). e.g., entrants) can be detected.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may acquire the first image of the user ( S412 ). The electronic device 110 (eg, the processor 214 ) is configured to receive the electronic device 110 through at least one camera 217a and 217b of the camera unit 216 disposed on one side (eg, front) of the electronic device 110 . A first image of a body part (eg, face) of a user (eg, Hong Gil-dong) who approaches the door 130 may be acquired.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) compares the acquired first image with a pre-stored second image to authenticate the user ( S414 ). The electronic device 110 (eg, the processor 214 ) acquires a first image (eg, a first face image) of the user (eg, Gildong Hong) through a first camera 217a (eg, an RGB camera) and comparing the obtained first image with a second image (eg, a second face image) of the user (eg, Gil-dong Hong) pre-stored in the storage unit 213 to authenticate the user (eg, Gil-dong Hong) can be performed.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) uses the second camera 217b (eg, at least one IR camera) to display a first image (eg, Hong Gil-dong) of the user (eg, Gil-dong Hong). : acquires a first face image), and compares the obtained first image with a second image (eg, a second face image) of the user (eg, Gil-dong Hong) pre-stored in the storage unit 213 , and the user (eg Hong Gil-dong) can be authenticated.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may identify whether authentication is successful ( S416 ). The electronic device 110 (eg, the processor 214 ) extracts feature points (eg, the length of the eyebrows compared to the face ratio, the distance between the eyebrows, the mouth size, the nose size, the eye size, etc.) from the first image, and the extraction Authenticate the user (eg, Gil-dong Hong) by comparing the characteristic points of the second image with the characteristic points of the second image (eg, eyebrow length to face ratio, distance between eyebrows, mouth size, nose size, eye size, etc.), and You can decide whether authentication is successful or not. The electronic device 110 (eg, the processor 214 ) may determine that two characteristic points are the same when a difference value between each characteristic point is within a predetermined range (eg, 5 mm). The predetermined range (eg, 5 mm) may be variably adjusted.

According to an embodiment, the electronic device 110 (eg, the processor 214) may output an authentication failure message (S418). The electronic device 110 (eg, the processor 214 ) determines that the first image and the second image are different from each other when the difference value between each feature point is out of a predetermined range (eg, 5 mm) and authentication fails. can be recognized as The electronic device 110 (eg, the processor 214 ) displays a message about the failure of the authentication through the display unit 212a when authentication (eg, primary authentication) for the user is not successful, or It is possible to output audio through the speaker 212b.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may transmit an identifier for the authenticated user to the smart mirror device ( S420 ). When it is determined that authentication (eg, primary authentication) is successful in the process ( S416 ), the electronic device 110 (eg, the processor 214 ) transmits an identifier for the authenticated user to the smart mirror device 140 (eg, : Can be transmitted to the communication unit 310). The identifier may include various information such as the user's image, name, and access time. Alternatively, if the authentication (eg, primary authentication) is not performed, the electronic device 110 (eg, the processor 214 ) displays the preview image of the user obtained from the camera unit 216 on the smart mirror. It may transmit to the device 140 (eg, the communication unit 310).

According to an embodiment, when the electronic device 110 (eg, the processor 214) succeeds in user authentication (eg, primary authentication), the door opening and closing device 120 provides a signal for unlocking the door. may be transmitted to the door opening and closing device 120 (eg, the communication unit 221 ). 5 is a flowchart illustrating a process of authenticating a user in an electronic device according to another embodiment of the present invention.

Hereinafter, a process of authenticating a user in an electronic device according to another embodiment of the present invention will be described in detail with reference to FIG. 5 .

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may detect a user approaching the door ( S510 ). The electronic device 110 (eg, the processor 214 ) may detect whether the user approaches the door 130 . The electronic device 110 (eg, the processor 214 ) accesses the door 130 through at least one of a sensor unit 219 (eg, a distance measuring sensor 219a and a motion detection sensor 219b). It can detect a user (eg, a person entering and leaving).

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may acquire the first image of the user through the first camera 217a ( S512 ). The electronic device 110 (eg, the processor 214 ) is configured to use at least one camera (eg, an RGB camera, at least one IR camera) of the camera unit 216 disposed in front of the electronic device 110 . An image of a body part (eg, face) of a user (eg, Gil-dong Hong) who approaches the door 130 may be acquired.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may compare the acquired first image with a pre-stored second image to perform first authentication for the user ( S514). The electronic device 110 (eg, the processor 214 ) acquires a first image (eg, a first face image) of the user (eg, Gildong Hong) through a first camera 217a (eg, an RGB camera) and comparing the obtained first image with a second image (eg, a second face image) of the user (eg, Gil-dong Hong) pre-stored in the storage unit 213 to authenticate the user (eg, Gil-dong Hong) can be performed.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) transmits a first image (eg, Hong Gil-dong) of the user (eg, Hong Gil-dong) through the second camera 217b (eg, at least one IR camera). Example: acquiring a first face image), comparing the acquired first image with a second image (eg, a second face image) of the user (eg, Gil-dong Hong) pre-stored in the storage unit 213, Based on the comparison result, authentication (eg, primary authentication) for the user (eg, Gil-dong Hong) may be determined.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may identify whether the first authentication is successful ( S516 ). The electronic device 110 (eg, the processor 214 ) extracts feature points (eg, the length of the eyebrows compared to the face ratio, the distance between the eyebrows, the mouth size, the nose size, the eye size, etc.) of the first image, and the extraction The calculated feature point and the feature point of the second image (eg, the length of the eyebrows compared to the face ratio, the distance between the eyebrows, the mouth size, the nose size, the eye size, etc.) may be compared. In addition, the electronic device 110 (eg, the processor 214 ) determines that the two characteristic points are the same when the difference value between each characteristic point is within a predetermined range (eg, 5 mm), and the user (eg, Hong Gil-dong) ) can be determined to be successful. The predetermined range (eg, 5 mm) may be variably adjusted.

According to an embodiment, the electronic device 110 (eg, the processor 214) may output an authentication failure message (S518). The electronic device 110 (eg, the processor 214 ) determines that the first image and the second image are different when the difference value between each feature point is out of a predetermined range (eg, 5 mm), and authentication is executed can be judged to have been The electronic device 110 (eg, the processor 214 ) displays a message about the failure of the authentication (eg, the primary authentication) when the authentication (eg, the primary authentication) for the user is unsuccessful, the display unit 212a ), or the content of the message may be output as voice through the speaker 212b.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may activate the second camera 217b and display a guide message ( S520 ). The electronic device 110 (eg, the processor 214 ) compares the characteristic points of the first image with the characteristic points of the second image, and when it is determined that the authentication (eg, primary authentication) is successful, the user ( At least one IR camera 217b may be activated for additional authentication (eg, secondary authentication) for eg Hong Gil-dong. In addition, the electronic device 110 (eg, the processor 214 ) provides a guide message (eg, a guide message) for allowing a user (eg, Hong Gil-dong) to place a body part (eg, a palm) on the second camera 217b. : Please spread your palm in front of the camera) may be displayed through the display unit 212a, or the guide message may be outputted by voice through the speaker 212b.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may acquire a third image ( S522 ). The electronic device 110 (eg, the processor 214 ) may acquire a third image of a body part (eg, palm) of the user (eg, Gildong Hong). The obtained third image may include vein information (eg, thickness and direction of venous blood vessels) distributed on the palm of the user (eg, Gil-dong Hong). For example, if the vein information is not included in the third image, the electronic device 110 (eg, the processor 214) re-outputs the guide message (eg, please spread your palm in front of the camera). can In addition, when the vein information is not included in the third image, the electronic device 110 (eg, the processor 214 ) uses a body part (eg, palm) of the user (eg, Gildong Hong) as a second camera. (217b) can be retaken.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may compare the acquired third image with a pre-stored fourth image to perform second authentication for the user ( S524). The electronic device 110 (eg, the processor 214 ) uses a second camera 217b (eg, at least one IR camera) for a third image (eg, a first palm image) of the user (eg, Gildong Hong). ), and compares the obtained third image with a fourth image (eg, second palm image) for the user (eg, Hong Gil-dong) pre-stored in the storage unit 213 to compare the user (eg, Hong Gil-dong) authentication can be performed.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may identify whether the second authentication is successful ( S526 ). The electronic device 110 (eg, the processor 214 ) extracts a feature point (eg, a venous vessel length, a distance between venous vessels, a thickness of a venous vessel, etc.) of the third image, and extracts the extracted feature point and the second 4 You can compare the feature points of the image (eg, the length of veins, the distance between veins, the thickness of veins, etc.). In addition, the electronic device 110 (eg, the processor 214 ) determines that the two characteristic points are the same when the difference value between the respective characteristic points is within a predetermined range (eg, 2 mm), and the user (eg, Hong Gil-dong) ) for authentication (eg, secondary authentication) may be determined to be successful.

According to an embodiment, the electronic device 110 (eg, the processor 214 ) may transmit an identifier for the authenticated user to the smart mirror device ( S528 ). When it is determined that the authentication (eg, secondary authentication) is successful in the process S526, the electronic device 110 (eg, the processor 214) generates an identifier for the authenticated user to the smart mirror device 140 ( For example, it can be transmitted to the communication unit 310). The identifier may include various information such as the user's image, name, and access time. Alternatively, if the authentication is not performed, the electronic device 110 (eg, the processor 214 ) displays the preview image of the user obtained from the camera unit 216 to the smart mirror device 140 (eg: It can be transmitted to the communication unit 310).

According to an embodiment, when the electronic device 110 (eg, the processor 214) succeeds in user authentication (eg, secondary authentication), the door opening and closing device 120 provides a signal for unlocking the door. may be transmitted to the door opening and closing device 120 (eg, the communication unit 221 ).

6 is a flowchart illustrating a process in which the smart mirror device controls the operation of the home appliance according to an embodiment of the present invention.

Hereinafter, with reference to FIG. 6 , the process of controlling the operation of the home appliance by the smart mirror device according to an embodiment of the present invention will be described in detail as follows.

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) may store an identifier for at least one home appliance in the home ( S610 ). The smart mirror device 140 includes at least one home appliance (eg, an air conditioner 151 , an oxygen generator 152 , a sound output device 153 , a lighting device 154 , and an air purifier 155 ) existing in the home. ), the humidifier 156 , the dehumidifier 157 , and the TV 158 ). The information may include power consumption, location, and identifier for each home appliance. The smart mirror device 140 may store information received from each home appliance device in the storage unit 313 .

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) may identify whether an identifier is received from the electronic device 110 ( S612 ). The smart mirror device 140 (eg, the processor 314 ) may receive an identifier for the user from the electronic device 110 based on the user's authentication. The identifier is an image of the user (eg, an image pre-stored in the electronic device 110 (eg, the storage unit 213 ), or a preview acquired by the electronic device 110 (eg, the camera unit 216 )). image), a user name (eg, nickname) preset by the user, and various information such as the user's entry/exit time (eg, outing time and return time).

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) provides information about the user (eg, an image pre-stored in the storage 313 , a preset by the user based on the identifier). Various information, such as a user name (eg, nickname), and the user's entry/exit time (eg, outing time, return time), etc. may be acquired from the storage unit 313 .

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) may display an image for the user and information on the user's entry ( S614 ). The smart mirror device 140 (eg, the processor 314 ) displays an image of the user and information about the user's entry through the identifier received from the electronic device 110 based on the user's authentication on the display unit 312 . (eg, the transparent display 312b).

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) may display the display unit 312 in a first mode (eg, mirror mode), in a second mode (eg, smart mirror mode), and The display unit 312 may be controlled to operate in a third mode (eg, a display mode).

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) displays the display unit 312 through the mirror 312a when the user's distance is greater than a predetermined distance (eg, 50 cm). While operating in a mirror mode, it may be operated in a standby mode displaying weather information and standby information.

According to an embodiment, the display unit 312 displays, under the control of the processor 314, a mirror mode that completely reflects an object, and at least some information by the processor 314 based on the mirror mode. It may be operated in any one of a smart mirror mode and a display mode in which only at least a portion of information by the processor 314 is displayed.

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) displays the display unit 312 on the processor 314 when the user's distance is less than a predetermined distance (eg, 50 cm). It may operate in a mode (eg, a smart mirror mode or a display mode) in which at least some of the information by the display is displayed through the transparent display 312b.

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) displays the display unit 312 on the processor 314 when the user's distance is greater than a predetermined distance (eg, 50 cm). It may operate in a mode (eg, a mirror mode or a smart mirror mode) in which at least some of the information by the display is displayed through the transparent display 312b.

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) receives the display unit 312 from the electronic device 110 based on the user's authentication while the display unit 312 is operating in the mirror mode. When the identifier is received through the communication unit 310 , the mirror mode of the display unit 312 may be switched to a smart mirror mode or a display mode. In addition, the smart mirror device 140 (eg, the processor 314 ) may display an image of the user and information about the user's entry through either the switched smart mirror mode or the display mode.

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) receives weather information (eg, temperature, humidity) from an external device (eg, a weather server, a standby server, etc.) through the communication unit 310 . , rain probability, snow probability, etc.), and atmospheric information (eg, fine dust concentration, yellow sand concentration, etc.). In addition, the smart mirror device 140 (eg, the processor 314 ) includes at least one home appliance (eg, the air conditioner 151 , the oxygen generator 152 , the sound output device 153 ) existing in the home. The power consumed in real time (or by period) from the lighting device 154 , the air purifier 155 , the humidifier 156 , the dehumidifier 157 , and the TV 158 , the identifier of each household appliance, and each household appliance information including the location (eg, the first floor, the second floor, the master bedroom, the study room, the kitchen, the living room, etc.) may be received.

According to an embodiment, the smart mirror device 140 (eg, the processor 314) displays at least some of the received weather information, standby information, and power consumption information on the display unit 312 (eg, mirror mode, Smart Mirror Mode or Display Mode).

According to an embodiment, the smart mirror device 140 (eg, the processor 314) may identify at least one home appliance device based on the received identifier (S616). The identifier may be included in information transmitted by each home appliance. The smart mirror device 140 (eg, the processor 314 ) may identify at least one home appliance device assigned to each user.

For example, when there are three family members (eg, father, mother, and son), at least one home appliance device may be set for each member as shown in [Table 1] below.

identifier household appliance name identifier location father Air conditioning (151) air conditioner 1 Livingroom TV(158) living room tv Livingroom lighting device (154) light 1 Inner room mother Air Purifier(155) air cleaner Livingroom Oxygen Generator (152) oxygen generator Kitchen lighting device (154) light 2 Kitchen son Air conditioning (151) air conditioner 2 son's room sound output device (153) radio son's room lighting device (154) light 3 son's room

As shown in [Table 1], the father is a heating and cooling unit 151 (eg, air conditioner 1), and a TV 158 (eg, living room TV) located in the living room, and a lighting device 154 located in the master bedroom (eg, TV) : Lighting 1) can be set. In addition, the mother can set the air purifier 155 (eg, air purifier) located in the living room, the oxygen generator 152 (eg, oxygen generator) located in the kitchen, and the lighting device 154 (eg, light 2). there is. In addition, the son may set the air conditioning unit 151 (eg, air conditioner 2), the sound output device 153 (eg, radio), and the lighting device 154 (eg, light 3) located in the son's room. In addition to the present invention, various home appliance devices can be set.

According to an embodiment, the smart mirror device 140 (eg, the processor 314 ) may identify at least one home appliance device allocated to each user using, for example, [Table 1]. For example, if the identifier (eg, father) is the identifier (eg, father) in the process S612, the smart mirror device 140 (eg, the processor 314) is at least one home appliance device corresponding to the identifier (eg, father). (eg, the air conditioner 1 located in the living room, the living room TV, and the lighting lamp 1 located in the master room) can be identified.

According to an embodiment, the smart mirror device 140 (eg, the processor 314) may control the operation of the identified at least one home appliance device based on the identifier (S618).

According to an embodiment, when at least one home appliance device (eg, air conditioner 1 located in the living room, living room TV, and lighting lamp 1 located in the master room) is identified based on the identifier (eg, father) in the process (S614) , the smart mirror device 140 (eg, the processor 314) may transmit a signal (eg, POWER ON/OFF) for controlling the operation of each home appliance to the corresponding home appliance.

In this way, when a user who has successfully authenticated through the electronic device 110 enters through the door 120 , the smart mirror device 140 (eg, the processor 314 ) performs at least one home appliance preset for the user. You can control the operation of the device.

7A is an exemplary diagram illustrating a display area of a display unit of a smart mirror device according to an embodiment of the present invention. 7B is an exemplary diagram illustrating a standby screen of a display unit of a smart mirror device according to an embodiment of the present invention.

Referring to FIG. 7A , the screen 710 of the smart mirror device according to an embodiment of the present invention includes a first area 711 displaying weather information, a second area 712 displaying date information, authentication status, A third area 713 for displaying greetings, various icons, a fourth area 714 for displaying icons for the media player, a fifth area 715 for displaying various information about the smart mirror device, A sixth area 716 for receiving a command for setting, a seventh area 717 for receiving a command to execute a mirror function, an eighth area 718 for displaying information on overall cleanliness, power of home appliances It may be configured to include a ninth area 719a for displaying information on occupants, a tenth area 719b for displaying various information for informing occupants, and an eleventh area 719c for displaying occupant information.

According to an embodiment, the first region 711 includes temperature (eg, current temperature, lowest temperature/highest temperature of the day), weather (eg, sunny, rain, cloud, lightning, snow, hail, strong wind, cloudy, yellow sand). , showers, etc.), fine dust (eg, good, normal, bad, very bad, etc.), and humidity levels may be displayed.

According to an embodiment, in the second area 712 , information about the current time, the current day of the week, and the current date may be displayed.

According to an embodiment, the third area 713 is a display area of the smart mirror device, and notifies a common message before authentication, a mode change notification, and start/end of a quick menu function operation. In addition, the third area 713 may display messages for each time period, each situation, greetings, suggestions, and notification (emergency), and various icons (eg, waiting, speaking, listening, etc.). For example, the third area 713 may contain various messages depending on the situation (eg, hello, nice to meet you, go out mode, rain is forecast, take an umbrella, snow forecast, icy road safety) Please note, user authentication failed, etc.) may be displayed.

According to an embodiment, an icon for executing media (eg, music, news) may be displayed in the fourth area 714 . In addition, various information about the smart mirror device may be displayed in the fifth area 715 .

According to an embodiment, the sixth area 716 is an area in which the smart mirror device 140 can be set. For example, the sixth area 716 is an area in which various settings such as in-home situation, energy situation, security situation, family call, wake-up mode, sleep mode, and outing mode can be input. And, the seventh area 717 is an area for receiving a command to execute the mirror function. For example, when the user inputs a command on the seventh area 717 to operate the smart mirror device 140 with the mirror function, the processor 314 sets the smart mirror device 140 with the mirror function. can make it work

According to an embodiment, in the eighth area 718, a degree of cleanliness (eg, good, normal, bad, very bad, etc.) may be displayed.

According to an embodiment, in the ninth area 719a, the accumulated energy status may be displayed for each time period or date, and an energy mode, power consumption, and power generation amount (kWh) may be displayed. For example, in the ninth area 719a, messages indicating various conditions for the interior of the home (eg, the air purifier has been activated, the living room TV is turned on, the electric curtain is closed, the ventilation is executed, etc.) are displayed. can be

According to an embodiment, in the tenth area 719b, various information (eg, running home appliance, excess energy consumption, switching to an energy saving mode, etc.) may be displayed on the standby screen. Alternatively, in the tenth area 719b, information about the person entering or leaving may be displayed. In addition, in the eleventh area 719c, an icon, nickname, and current location (number of floors) of a occupant may be displayed, and an icon of an unauthenticated occupant, a current location, and a text message transmission icon may be displayed.

Referring to FIG. 7B , the standby screen 720 of the display unit of the smart mirror device according to an embodiment of the present invention displays weather information (eg, current temperature 15 ^o ), current humidity (23%) in the first area 721 . ), minimum/highest temperature (10 ^o /21 ^o ), fine dust concentration (PM2.5) (24 mg/m ² )), and date information (eg, Saturday, June 17th) in the second area 722 12 o'clock) may be included. In addition, in the eighth area 728 , the cleanliness (eg, air pollution level in the home) is displayed, and in the ninth area 729 , the energy mode (eg, saving) of current home appliances and the power consumed (eg, 5.0) are displayed. kWh) and power generation (3.0 kWh) can be displayed. In addition, in the tenth area 730, information on various situations inside the home is displayed, and in the eleventh area 731, an image, an identifier (father), and a location (1st floor) of an identified occupant currently present in the home. ), an icon for an unidentified occupant (eg, a guest), an identifier (nickname), and information about a location (underground) may be displayed.

8A is an exemplary diagram illustrating a case in which a customer is located at the door 130 according to an embodiment of the present invention. 8B is an exemplary diagram illustrating a screen of a smart mirror device when a guest visits according to an embodiment of the present invention.

8A and 8B , when the guest 810 is located in front of the electronic device 110 installed outside the home to visit, the electronic device 110 determines whether the guest 810 is a previously registered user. can do. Also, when it is determined that the guest 810 is not a previously registered user, the electronic device 110 may transmit a preview image 831 of the guest 810 to the smart mirror device 140 . Also, the electronic device 110 may notify the smart mirror device 140 that the current guest 810 is in front of the door 130 .

According to an embodiment, the smart mirror device 140 may display a pop-up 830 including a preview image 831 of the guest on the standby screen 820 . In addition, the smart mirror device 140 may display a message indicating that a guest has visited (eg, a guest has visited) in the tenth area. Alternatively, the smart mirror device 140 may output a notification through an output device (eg, the speaker 317 ) so that a occupant (eg, father) can know that a guest has visited.

9A is an exemplary diagram illustrating a standby screen of a smart mirror device according to an embodiment of the present invention. 9B is an exemplary view showing a situation outside the home through the smart mirror device according to an embodiment of the present invention.

9A and 9B , when the door lock/camera 911 is selected while the smart mirror device 140 is displaying the standby screen 910 according to an embodiment of the present invention, the smart mirror device 140 (eg, the processor 314 ) may activate at least one camera included in the electronic device 130 . Then, the smart mirror device 140 (eg, the processor 314) acquires an image of the outside of the home (eg, outside the front door) through the activated camera, and a pop-up ( 922 ) may be displayed on the standby screen 910 .

According to an embodiment, the idle screen 910 may display various icons 918a, 918b, 918c, and 918d for security. According to an embodiment, the standby screen 910 may display icons 912 , 913 , 914 , 915 , 916 , and 917 for at least one camera installed inside or outside the home.

Each step in each of the above-described flowcharts may be operated regardless of the illustrated order, or may be performed simultaneously. In addition, at least one component of the present invention and at least one operation performed by the at least one component may be implemented in hardware and/or software.

As described above, the present invention has been described with reference to the illustrated drawings, but the present invention is not limited by the embodiments and drawings disclosed in this specification, and various methods can be obtained by those skilled in the art within the scope of the technical spirit of the present invention. It is obvious that variations can be made. In addition, although the effects according to the configuration of the present invention have not been explicitly described and described while describing the embodiments of the present invention, it is natural that the effects predictable by the configuration should also be recognized.

110: electronic device 120: door opening and closing device
130: door 140: smart mirror device
151: air conditioning unit 152: oxygen generator
153: sound output device 154: lighting device
310: communication unit 311: input unit
312: display unit 312a: mirror
312b: transparent display 314: processor
316: camera unit 317: speaker
318: sensor unit 319: power unit

Claims (20)

In the smart mirror device,
a display unit including a mirror and a transparent display;
communication department; and
and a processor electrically connected to the display unit and the communication unit,
The processor is
receiving an identifier for a user entering the home from an electronic device located outside the home through the communication unit;
A smart mirror device configured to display the user's image and information on the user's entry through the transparent display based on the received identifier.
According to claim 1,
The processor is
Receive weather information, standby information, and power consumption information of at least one home appliance in the home through the communication unit,
A smart mirror device configured to display at least a portion of the received weather information, standby information, and power consumption information through the transparent display.
According to claim 1,
It further comprises a storage unit,
The processor is
storing the identifier for at least one user transmitted from the electronic device in the storage unit;
It is set to display information on at least one user located inside the home among users corresponding to the stored identifier through the transparent display,
Information about the user,
A smart mirror device comprising at least one of the user's image, the user's identifier, and the user's location.
4. The method of claim 3,
The processor is
Identifies whether some of the information about the user is selected through the transparent display,
based on the identified selection, identify a location of the user's portable device;
A smart mirror device configured to determine the location of the user based on the identified location of the portable device.
4. The method of claim 3,
The processor is
establishing a communication channel with the user's portable device based on the identified selection;
A smart mirror device configured to transmit/receive a signal to and from the portable device through the communication unit based on the set communication channel.
4. The method of claim 3,
It further includes a camera unit,
The processor is
Receive an input requesting authentication for a user not included in the stored identifier through the transparent display,
Obtaining an image of the user requesting the authentication through the camera unit based on the received input,
Obtaining an identifier for the user requesting the authentication through the transparent display,
Perform authentication on the user requesting the authentication based on the acquired image and identifier,
A control device configured to display the obtained image and identifier through the transparent display.
According to claim 1,
It further comprises a sensor unit,
The processor is
Detects the user's approach through the sensor unit,
Based on the sensed user's approach, calculating a distance between the user and the smart mirror device,
A control device configured to control an operation of the display unit based on the calculated distance.
8. The method of claim 7,
The processor is
When the calculated distance is greater than a predetermined distance, the display unit is operated in a mirror mode through the mirror, and controls the display unit to operate in a standby mode for displaying weather information and standby information,
When the calculated distance is less than the predetermined distance, the smart mirror device for controlling the display unit to operate in a smart mirror mode in which the display unit displays at least some information by the processor through the transparent display.

According to claim 1,
The processor is
Obtaining information on at least one home appliance located inside the home,
Identify at least one home appliance corresponding to the received identifier from among the obtained information on the at least one home appliance,
A smart mirror device configured to control an operation of the identified at least one home appliance based on a preset condition of the identified at least one home appliance.
According to claim 1,
The identifier is
The smart mirror device received from the electronic device based on the success of the first authentication based on the recognition of the user's face and the success of the second authentication based on the user's vein information.
4. The method of claim 3,
The processor is
determining whether the identifier of the user entering the home is included in the identifier stored in the storage unit;
If the identifier of the user entering the home is not included in the identifier stored in the storage unit, the smart mirror device configured to display an image of the user entering the home as a pop-up on the display unit.
In the method of the smart mirror device,
receiving, through a communication unit, an identifier for a user entering the home from an electronic device located outside the home; and
based on the received identifier, displaying the user's image and information on the user's entry through a transparent display of a display unit.
13. The method of claim 12,
receiving weather information, standby information, and power consumption information of at least one home appliance in the home through the communication unit; and
The method further comprising the step of displaying at least a portion of the received weather information, standby information, and power consumption information through the transparent display.
13. The method of claim 12,
storing an identifier for at least one user transmitted from the electronic device in a storage unit; and
The method further comprising the step of displaying information on at least one user located inside the home from among the users corresponding to the stored identifier through the transparent display,
Information about the user,
a method comprising at least one of an image of the user, an identifier of the user, and a location of the user.
15. The method of claim 14,
a process of identifying whether some of the information about the user is selected through the transparent display;
identifying a location of the user's portable device based on the identified selection; and
The method further comprising the step of determining the location of the user based on the identified location of the portable device.
15. The method of claim 14,
establishing a communication channel with the user's portable device based on the identified selection; and
The method further comprising the step of transmitting and receiving a signal to and from the portable device through the communication unit based on the set communication channel.
15. The method of claim 14,
receiving an input requesting authentication for a user not included in the stored identifier through the transparent display;
obtaining an image of a user requesting the authentication through the camera unit based on the received input;
obtaining an identifier for the user requesting the authentication through the transparent display;
performing authentication on the user requesting the authentication based on the acquired image and the identifier; and
and displaying the obtained image and the identifier through the transparent display.
13. The method of claim 12,
detecting a user approaching the smart mirror device;
calculating a distance between the user and the smart mirror device based on the sensed user's approach;
controlling the display unit so that the display unit operates in a first mode when the calculated distance is greater than a predetermined distance; and
When the calculated distance is less than the predetermined distance, the method further comprising the step of controlling the display unit so that the display unit operates in a second mode.
13. The method of claim 12,
obtaining information on at least one home appliance located inside the home;
identifying at least one home appliance device corresponding to the received identifier from among the acquired information on the at least one home appliance device; and
The method further comprising the step of controlling the operation of the identified at least one home appliance based on a preset condition of the identified at least one home appliance.
In the system,
electronic device,
Based on the user's recognition, acquiring a first image of the user through a first camera,
Perform authentication of the user based on the obtained first image,
is set to transmit an identifier for the user to the smart mirror device based on the user's authentication,
The smart mirror device,
Obtaining information about at least one home appliance located inside the home,
Identify at least one home appliance device corresponding to the identifier received from the electronic device from among the acquired information on the at least one home appliance device,
A system configured to control an operation of the identified at least one home appliance based on a preset condition of the identified at least one home appliance.

Images