WO2020230929A1

WO2020230929A1 - Robot

Info

Publication number: WO2020230929A1
Application number: PCT/KR2019/005851
Authority: WO
Inventors: 김남건
Original assignee: 엘지전자 주식회사
Priority date: 2019-05-15
Filing date: 2019-05-15
Publication date: 2020-11-19

Abstract

A robot according to an embodiment of the present invention comprises: a camera; a display for displaying a graphical user interface (GUI); an input unit for receiving a camera OFF request; and a processor that switches the camera to an OFF-state in response to the camera OFF request and controls the display to display a GUI corresponding to the OFF-state of the camera, wherein a GUI corresponding to an ON-state of the camera may be different from the GUI corresponding to the OFF-state of the camera. According to an embodiment, the robot is implemented as an artificial intelligence device, and may recognize a camera OFF request in a voice form through an artificial intelligence-based learning model.

Description

robot

The present invention relates to a robot, and more particularly, to a robot that intuitively displays the on/off state of a camera.

Robots are machines that automatically process or operate a given task by their own capabilities, and their application fields are generally classified into various fields such as industrial, medical, space, and submarine. Recently, communication robots capable of communicating or interacting with humans through voice or gestures are increasing.

Such a communication robot may include various types of robots, such as a guide robot arranged in a specific place to guide various information to a user, or a home robot provided in a home. Further, the communication robot may include an educational robot that guides or assists the learner's learning through interaction with the learner.

The communication robot can be implemented to perform interactions with users or learners using various configurations. For example, the communication robot may include a microphone that acquires sound generated around the robot or a camera that acquires an image around the robot.

Meanwhile, in the conventional case, the communication robot does not provide information on whether the camera is in an on state or an off state to a user. Accordingly, the user may have a concern about the possibility that the user's privacy may be violated by the camera mounted on the communication robot. This concern can act as a major disadvantage in terms of activating the spread of communication robots.

The problem to be solved by the present invention is to provide a robot that enables a user to conveniently switch on/off of a camera provided in the robot.

Another problem to be solved by the present invention is to provide a robot that can intuitively inform a user of an on/off state of a camera.

A robot according to an embodiment of the present invention includes a camera; A display for displaying a graphical user interface (GUI); An input unit for receiving a camera off request; And a processor that switches the camera to an off state in response to the camera off request and controls the display to display a GUI corresponding to the off state of the camera, wherein the GUI corresponding to the on state of the camera and the The GUI corresponding to the off state of the camera may be different from each other.

According to an embodiment, the input unit includes at least one microphone, and the processor receives a first voice including the camera off request through the at least one microphone, and in response to the received first voice, the You can turn the camera off.

According to an embodiment, the processor receives a second voice corresponding to a preset starting word through the at least one microphone and, in response to the received second voice, processes the camera off request for the operation mode of the robot. After switching to a possible mode, the camera may be switched to an off state in response to the first voice.

According to an embodiment, the robot further includes a memory for storing a learning model learned by a learning processor, and the processor recognizes the starting word from the received second voice through a learning model stored in the memory, , It is possible to recognize the camera off request from the received first voice.

According to an embodiment, the robot further includes a communication unit for connecting with a voice recognition server, and the processor transmits a voice corresponding to the camera off request received through the at least one microphone to the voice recognition server. When the communication unit is controlled, the speech recognition result is received from the speech recognition server through the communication unit, and the received recognition result corresponds to the camera off request, the camera may be turned off.

According to an embodiment, the memory stores a first GUI set including at least one GUI corresponding to the on state of the camera, and a second GUI set including at least one GUI corresponding to the off state of the camera. I can. When the camera is switched to an off state, the processor may load any one GUI included in the second GUI set from the memory, and display the loaded GUI through the display.

According to an embodiment, the robot may further include a speaker, and the processor may control the speaker to output a voice message corresponding to an off state of the camera.

According to an embodiment, the robot further includes a light output unit including at least one light source, and the processor turns off the at least one light source included in the light output unit as the camera is turned off. I can make it.

The processor, when switching the operation mode of the robot from a first mode in which the camera is turned off to a second mode in which the camera is turned on, turns the camera on when the camera is turned off by the camera off request. May not be converted.

The processor may control the display to receive a camera-on request through the input unit, switch the camera to an on-state in response to the received camera-on request, and display a GUI corresponding to the on-state of the camera. have.

According to an embodiment, the processor receives a first voice corresponding to a preset starting word through the at least one microphone and, in response to the received first voice, processes the camera-on request for the operation mode of the robot. It is possible to switch to a possible mode, receive a second voice corresponding to the camera-on request through the at least one microphone, and switch the camera to an on-state in response to the received second voice.

According to an embodiment, the processor may recognize the starting word from the received first voice through a learning model stored in the memory, and recognize the camera-on request from the received second voice.

When the camera is turned on, the processor loads a GUI corresponding to an emotional expression to be provided among at least one GUI included in the first GUI set from the memory, and loads the loaded GUI through the display. Can be displayed.

According to an embodiment, the processor may control the speaker to output a voice message corresponding to the on state of the camera.

According to an embodiment, the processor may control the at least one light source to output light of a set color or pattern as the camera is turned on.

According to an embodiment of the present invention, the robot provides a function of turning on/off the camera through voice input, etc., so that the user can conveniently control the camera state when desired. Accordingly, it is possible to prevent a problem of exposure to private life as a camera of a robot disposed at home or the like records an image including a user's private life regardless of the user's intention.

In addition, the robot may inform the user of the current state of the camera through a GUI change or a voice message output according to the state change of the camera. Accordingly, the user can conveniently recognize the camera state, which is difficult to recognize with the naked eye, through the GUI or voice message.

In addition, the robot can more accurately recognize the starter word or request by recognizing the starter word or request for camera on/off in the form of voice through the learning model learned by the learning processor.

1 shows an AI device including a robot according to an embodiment of the present invention.

2 shows an AI server connected to a robot according to an embodiment of the present invention.

3 shows an AI system including a robot according to an embodiment of the present invention.

4 is a perspective view of a robot according to an embodiment of the present invention.

5 is a block diagram showing a control configuration of a robot according to an embodiment of the present invention.

6 is a view for explaining operation modes of a robot and switching of the operation mode according to an embodiment of the present invention.

7 is a flowchart illustrating a control operation of a robot according to an embodiment of the present invention.

8 is an exemplary diagram of a GUI displayed on a display when the camera of the robot is on.

9 to 10 are exemplary diagrams illustrating a robot's processing operation according to a user's camera off request.

11 is an exemplary diagram of a GUI displayed on a display when the robot's camera is in an off state.

12 to 13 are exemplary diagrams illustrating a robot processing operation according to a user's camera-on request.

Hereinafter, exemplary embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings. The accompanying drawings are only for making it easier to understand the embodiments disclosed in the present specification, and the technical idea disclosed in the present specification is not limited by the accompanying drawings, and all changes and equivalents included in the spirit and scope of the present invention It should be understood to include water or substitutes.

Artificial intelligence refers to the field of researching artificial intelligence or the methodology to create it, and machine learning (Machine Learning) refers to the field of researching methodologies to define and solve various problems dealt with in the field of artificial intelligence. do. Machine learning is also defined as an algorithm that improves the performance of a task through continuous experience.

An artificial neural network (ANN) is a model used in machine learning, and may refer to an overall model with problem-solving capabilities, composed of artificial neurons (nodes) that form a network by combining synapses. The artificial neural network may be defined by a connection pattern between neurons of different layers, a learning process for updating model parameters, and an activation function for generating an output value.

The artificial neural network may include an input layer, an output layer, and optionally one or more hidden layers. Each layer includes one or more neurons, and the artificial neural network may include neurons and synapses connecting neurons. In an artificial neural network, each neuron can output a function of an activation function for input signals, weights, and biases input through synapses.

Model parameters refer to parameters determined through learning, and include weights of synaptic connections and biases of neurons. In addition, hyperparameters refer to parameters that must be set before learning in a machine learning algorithm, and include a learning rate, iteration count, mini-batch size, and initialization function.

The purpose of learning artificial neural networks can be seen as determining model parameters that minimize the loss function. The loss function can be used as an index to determine an optimal model parameter in the learning process of the artificial neural network.

Machine learning can be classified into supervised learning, unsupervised learning, and reinforcement learning according to the learning method.

Supervised learning refers to a method of training an artificial neural network when a label for training data is given, and a label indicates the correct answer (or result value) that the artificial neural network should infer when training data is input to the artificial neural network. It can mean. Unsupervised learning may refer to a method of training an artificial neural network in a state where a label for training data is not given. Reinforcement learning may mean a learning method in which an agent defined in a certain environment learns to select an action or action sequence that maximizes the cumulative reward in each state.

Among artificial neural networks, machine learning implemented as a deep neural network (DNN) including a plurality of hidden layers is sometimes referred to as deep learning (deep learning), and deep learning is a part of machine learning. Hereinafter, machine learning is used in the sense including deep learning.

The AI device 100 includes a TV, a projector, a mobile phone, a smartphone, a desktop computer, a laptop computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a navigation system, a tablet PC, a wearable device, a set-top box (STB). ), a DMB receiver, a radio, a washing machine, a refrigerator, a desktop computer, a digital signage, a robot, a vehicle, and the like.

Referring to FIG. 1, the AI device 100 includes a communication unit 110, an input unit 120, a running processor 130, a sensing unit 140, an output unit 150, a memory 170, and a processor 180. It may include.

The communication unit 110 may transmit and receive data with external devices such as other AI devices 100a to 100e or the AI server 200 using wired/wireless communication technology. For example, the communication unit 110 may transmit and receive sensor information, a user input, a learning model, and a control signal with external devices.

At this time, the communication technologies used by the communication unit 110 include Global System for Mobile communication (GSM), Code Division Multi Access (CDMA), Long Term Evolution (LTE), 5G, Wireless LAN (WLAN), and Wireless-Fidelity (Wi-Fi). ), Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), ZigBee, and Near Field Communication (NFC).

The input unit 120 may acquire various types of data.

In this case, the input unit 120 may include a camera for inputting an image signal, a microphone for receiving an audio signal, a user input unit for receiving information from a user, and the like. Here, by treating a camera or microphone as a sensor, a signal obtained from the camera or microphone may be referred to as sensing data or sensor information.

The input unit 120 may acquire training data for model training and input data to be used when acquiring an output by using the training model. The input unit 120 may obtain unprocessed input data, and in this case, the processor 180 or the running processor 130 may extract an input feature as a preprocess for the input data.

The learning processor 130 may train a model composed of an artificial neural network using the training data. Here, the learned artificial neural network may be referred to as a learning model. The learning model can be used to infer a result value for new input data other than the training data, and the inferred value can be used as a basis for a decision to perform a certain operation.

In this case, the learning processor 130 may perform AI processing together with the learning processor 240 of the AI server 200.

In this case, the learning processor 130 may include a memory integrated or implemented in the AI device 100. Alternatively, the learning processor 130 may be implemented using the memory 170, an external memory directly coupled to the AI device 100, or a memory maintained in an external device.

The sensing unit 140 may acquire at least one of internal information of the AI device 100, information about the surrounding environment of the AI device 100, and user information by using various sensors.

At this time, the sensors included in the sensing unit 140 include a proximity sensor, an illuminance sensor, an acceleration sensor, a magnetic sensor, a gyro sensor, an inertial sensor, an RGB sensor, an IR sensor, a fingerprint recognition sensor, an ultrasonic sensor, an optical sensor, a microphone, and a lidar. , Radar, etc.

The output unit 150 may generate output related to visual, auditory or tactile sense.

In this case, the output unit 150 may include a display unit that outputs visual information, a speaker that outputs auditory information, and a haptic module that outputs tactile information.

The memory 170 may store data supporting various functions of the AI device 100. For example, the memory 170 may store input data, training data, a learning model, and a learning history acquired from the input unit 120.

The processor 180 may determine at least one executable operation of the AI device 100 based on information determined or generated using a data analysis algorithm or a machine learning algorithm. Further, the processor 180 may perform the determined operation by controlling the components of the AI device 100.

To this end, the processor 180 may request, search, receive, or utilize data from the learning processor 130 or the memory 170, and perform a predicted or desirable operation among the at least one executable operation. The components of the AI device 100 can be controlled to execute.

In this case, when connection of an external device is required to perform the determined operation, the processor 180 may generate a control signal for controlling the corresponding external device and transmit the generated control signal to the corresponding external device.

The processor 180 may obtain intention information for a user input, and determine a user's requirement based on the obtained intention information.

In this case, the processor 180 uses at least one of a Speech To Text (STT) engine for converting a speech input into a character string or a Natural Language Processing (NLP) engine for obtaining intention information of a natural language. Intention information corresponding to the input can be obtained.

At this time, at least one or more of the STT engine and the NLP engine may be composed of an artificial neural network, at least partially trained according to a machine learning algorithm. In addition, at least one of the STT engine or the NLP engine is learned by the learning processor 130, learned by the learning processor 240 of the AI server 200, or learned by distributed processing thereof. Can be.

The processor 180 collects history information including user feedback on the operation content or operation of the AI device 100 and stores it in the memory 170 or the learning processor 130, or the AI server 200 Can be transferred to an external device. The collected history information can be used to update the learning model.

The processor 180 may control at least some of the components of the AI device 100 to drive an application program stored in the memory 170. Furthermore, the processor 180 may operate by combining two or more of the components included in the AI device 100 to drive the application program.

Referring to FIG. 2, the AI server 200 may refer to a device that trains an artificial neural network using a machine learning algorithm or uses the learned artificial neural network. Here, the AI server 200 may be composed of a plurality of servers to perform distributed processing, or may be defined as a 5G network. In this case, the AI server 200 may be included as a part of the AI device 100 to perform at least part of AI processing together.

The AI server 200 may include a communication unit 210, a memory 230, a learning processor 240, and a processor 260.

The communication unit 210 may transmit and receive data with an external device such as the AI device 100.

The memory 230 may include a model storage unit 231. The model storage unit 231 may store a model (or artificial neural network, 231a) being trained or trained through the learning processor 240.

The learning processor 240 may train the artificial neural network 231a using the training data. The learning model may be used while being mounted on the AI server 200 of the artificial neural network, or may be mounted on an external device such as the AI device 100 and used.

The learning model can be implemented in hardware, software, or a combination of hardware and software. When part or all of the learning model is implemented in software, one or more instructions constituting the learning model may be stored in the memory 230.

The processor 260 may infer a result value for new input data using the learning model, and generate a response or a control command based on the inferred result value.

3, the AI system 1 includes at least one of an AI server 200, a robot 100a, an autonomous vehicle 100b, an XR device 100c, a smartphone 100d, or a home appliance 100e. It is connected to the cloud network 10. Here, the robot 100a to which the AI technology is applied, the autonomous vehicle 100b, the XR device 100c, the smartphone 100d, or the home appliance 100e may be referred to as the AI devices 100a to 100e.

The cloud network 10 may constitute a part of the cloud computing infrastructure or may mean a network that exists in the cloud computing infrastructure. Here, the cloud network 10 may be configured using a 3G network, a 4G or Long Term Evolution (LTE) network, or a 5G network.

That is, the devices 100a to 100e and 200 constituting the AI system 1 may be connected to each other through the cloud network 10. In particular, the devices 100a to 100e and 200 may communicate with each other through a base station, but may communicate with each other directly without through a base station.

The AI server 200 may include a server that performs AI processing and a server that performs an operation on big data.

The AI server 200 includes at least one of a robot 100a, an autonomous vehicle 100b, an XR device 100c, a smartphone 100d, or a home appliance 100e, which are AI devices constituting the AI system 1 It is connected through the cloud network 10 and may help at least part of the AI processing of the connected AI devices 100a to 100e.

In this case, the AI server 200 may train an artificial neural network according to a machine learning algorithm in place of the AI devices 100a to 100e, and may directly store the learning model or transmit it to the AI devices 100a to 100e.

At this time, the AI server 200 receives input data from the AI devices 100a to 100e, infers a result value for the received input data using a learning model, and generates a response or control command based on the inferred result value. It can be generated and transmitted to the AI devices 100a to 100e.

Alternatively, the AI devices 100a to 100e may infer a result value of input data using a direct learning model, and generate a response or a control command based on the inferred result value.

Hereinafter, various embodiments of the AI devices 100a to 100e to which the above-described technology is applied will be described. Here, the AI devices 100a to 100e illustrated in FIG. 3 may be viewed as a specific example of the AI device 100 illustrated in FIG. 1.

The robot 100a is applied with AI technology and may be implemented as a guide robot, a transport robot, a cleaning robot, a wearable robot, an entertainment robot, a pet robot, an unmanned flying robot, and the like.

The robot 100a may include a robot control module for controlling an operation, and the robot control module may refer to a software module or a chip implementing the same as hardware.

The robot 100a acquires status information of the robot 100a by using sensor information acquired from various types of sensors, detects (recognizes) the surrounding environment and objects, generates map data, or moves paths and travels. It can decide a plan, decide a response to user interaction, or decide an action.

Here, the robot 100a may use sensor information obtained from at least one sensor from among a lidar, a radar, and a camera in order to determine a moving route and a driving plan.

The robot 100a may perform the above operations using a learning model composed of at least one artificial neural network. For example, the robot 100a may recognize a surrounding environment and an object using a learning model, and may determine an operation using the recognized surrounding environment information or object information. Here, the learning model may be directly learned by the robot 100a or learned by an external device such as the AI server 200.

At this time, the robot 100a may perform an operation by generating a result using a direct learning model, but it transmits sensor information to an external device such as the AI server 200 and performs the operation by receiving the result generated accordingly. You may.

The robot 100a determines a movement path and a driving plan using at least one of map data, object information detected from sensor information, or object information acquired from an external device, and controls the driving unit to determine the determined movement path and travel plan. Accordingly, the robot 100a can be driven.

The map data may include object identification information on various objects arranged in a space in which the robot 100a moves. For example, the map data may include object identification information on fixed objects such as walls and doors and movable objects such as flower pots and desks. In addition, the object identification information may include a name, type, distance, and location.

In addition, the robot 100a may perform an operation or run by controlling a driving unit based on a user's control/interaction. In this case, the robot 100a may acquire interaction intention information according to a user's motion or voice speech, and determine a response based on the obtained intention information to perform an operation.

Referring to FIG. 4, the robot 100a may correspond to a communication robot that performs an operation such as providing information or content to a user or inducing a specific action through communication or interaction with the user.

For example, the robot 100a may be a home robot disposed at home. The home robot may provide various information or contents to the user through interaction with the user, or perform an operation such as monitoring an event occurring in the home.

In order to perform the above-described operation, the robot 100a includes a camera 142 that acquires an image of the user or around the robot, and at least one microphone 124 that acquires a user's voice or sound around the robot (see FIG. 5). , A display 152 for outputting graphics or text, a sound output unit 154 (for example, a speaker) for outputting voice or sound, and a light output unit 156 for outputting light of a color or pattern mapped to a specific event or situation; 5) and the like.

The robot 100a includes at least one microphone hole 125a to 125c formed on the outer surface of the cover (or case) in order to smoothly acquire sound from the outside of the robot through at least one microphone 124 implemented therein. I can. Each of the microphone holes 125a to 125c is formed at a position corresponding to one of the microphones 124, and the microphone 124 may communicate with the outside through the microphone holes 125a to 125c. Meanwhile, the robot 100a may include a plurality of microphones disposed to be spaced apart from each other, and in this case, the robot 100a may detect the direction in which the sound is generated using the plurality of microphones.

The display 152 may be disposed to face one side from the robot 100a. Hereinafter, the direction in which the display 152 faces is defined as the front of the robot 100a. Meanwhile, although the sound output unit 154 is shown to be disposed under the robot 100a, the position of the sound output unit 154 may be variously changed according to embodiments.

The light output unit 156 is implemented as a light source such as an LED, and may indicate a state or event of the robot 100a through a change in color or output pattern. In FIG. 4, a first light output unit 156a disposed on both sides of the robot 100a and a second light output unit 156b disposed below the robot 100a are shown, but the light output unit 156 The number and placement position of) can be variously changed.

Although not shown, the robot 100a may further include a moving means (driving means) for moving from one position to another. For example, the moving means may include at least one wheel and a motor that rotates the wheel.

Referring to FIG. 5, the robot 100a includes a communication unit 110, an input unit 120, a running processor 130, a sensing unit 140, an output unit 150, a rotation mechanism 160, a memory 170, and And a controller 180. The configurations illustrated in FIG. 5 are examples for convenience of description, and the robot 100a may include more or less configurations than those illustrated in FIG. 5.

Meanwhile, the contents related to the AI device 100 of FIG. 1 are similarly applied to the robot 100a of the present invention, and the contents overlapping with the above-described contents in FIG. 1 are omitted.

The communication unit 110 may include communication modules for connecting the robot 100a to a server, a mobile terminal, or another robot through a network. Each of the communication modules may support any one of the communication technologies described above in FIG. 1.

For example, the robot 100a may be connected to a network through an access point such as a router. Accordingly, the robot 100a may provide various types of information acquired through the input unit 120 or the sensing unit 140 to a server or a mobile terminal through the network.

The input unit 120 may include at least one input means for obtaining various types of data. For example, the at least one input means is a physical input means such as a button or a dial, a touch input unit 122 such as a touch pad or a touch panel, and a microphone 124 that receives a user's voice or sounds around the robot 100a. And the like. The user may input various requests or commands to the robot 100a through the input unit 120.

Meanwhile, the processor 182 may transmit the user's voice data received through the microphone 124 to the server through the communication unit 110. The server may analyze the voice data to recognize a startup word, command, request, etc. in the voice data, and provide a recognition result to the robot 100a. According to an embodiment, the server may be implemented as the AI server 200 described above in FIG. 2, in which case the server is the voice data through the model (artificial neural network 231a) learned through the learning processor 240 Can recognize startup words, commands, requests, etc. The processor 182 may switch an operation mode or process a command or request based on the recognition result.

According to an embodiment, the processor 182 may directly recognize a start word, a command, a request, etc. in the voice data through a model learned by the learning processor 130 in the robot 100a. Alternatively, the processor 182 may receive data corresponding to the learned model from the server, store it in the memory 170, and recognize a start word, command, request, etc. in the voice data through the stored data.

The sensing unit 140 may include at least one sensor that senses various information around the robot 100a. For example, the sensing unit 140 may include various sensors such as a camera 142, a proximity sensor 144, an illuminance sensor 146, a touch sensor, and a gyro sensor (gyroscope).

The camera 142 may acquire an image around the robot 100a. According to an embodiment, the controller 180 may recognize the user by acquiring an image including the user's face through the camera 142 or acquire a gesture or facial expression of the user.

The proximity sensor 144 may detect that an object such as a user approaches the robot 100a. For example, when a user's approach is detected by the proximity sensor 144, the controller 180 outputs an initial screen or an initial voice through the output unit 150 to induce the user to use the robot 100a. have.

The illuminance sensor 146 may detect the brightness of a space in which the robot 100a is disposed. The controller 180 may control components to perform various operations based on a detection result of the illumination sensor 146 and/or time zone information.

The touch sensor may detect that a part of the user's body comes into contact with a predetermined area of the robot 100a. For example, the touch sensor may be disposed behind the head of the robot 100a, specifically a face area including the display 152, but this is not necessarily the case.

The gyro sensor may detect a rotation angle or tilt of the robot 100a. The processor 182 may recognize a direction in which the robot 100a is directed based on a detection result of the gyro sensor, or detect an external impact.

The output unit 150 may output various information or contents related to the operation or state of the robot 100a, various services, programs, applications, etc. executed by the robot 100a. In addition, the output unit 150 may output various messages or information for performing an interaction with a user.

The output unit 150 may include a display 152, a sound output unit 154, and a light output unit 156.

The display 152 may output the above-described various types of information or messages in a graphic form. According to an embodiment, the display 152 may be implemented in the form of a touch screen together with the touch input unit 122, and in this case, the display 152 may function not only as an output unit but also as an input unit.

The sound output unit 154 may output the various information or messages in a voice or sound format. For example, the sound output unit 154 may include a speaker.

The light output unit 156 may be implemented as a light source such as an LED. The processor 182 may indicate the state of the robot 100a through the light output unit 156. According to an embodiment, the light output unit 156 may provide various types of information to the user together with the display 152 and/or the sound output unit 154 as an auxiliary output means.

The rotation mechanism 160 may include components (such as a motor) for rotating the robot 100a about a vertical axis. The controller 180 controls the rotation mechanism 160 to rotate the robot 100a, thereby changing the direction in which the display 152 and the camera 142 of the robot 100a are directed.

According to an embodiment, the rotation mechanism 160 may further include components for tilting the robot 100a in the front-rear direction by a predetermined angle.

The memory 170 is used to perform an operation based on control data for controlling the operation of components included in the robot 100a, an input obtained through the input unit 120 or information obtained through the sensing unit 140. Various data, such as data, may be stored.

In addition, the memory 170 may store program data such as software modules or applications executed by at least one processor or controller included in the controller 180.

In addition, a graphic user interface (GUI) for expressing emotions of the robot 100a through the display 152 may be stored in the memory 170 according to an embodiment of the present invention. In particular, in the memory 170, a first GUI set 172, which is a set of GUIs corresponding to when the camera 142 is on, and a second GUI set, which is a set of GUIs, corresponding to when the camera 142 is off. The GUI set 174 may be stored. The GUIs included in each of the GUI sets 172 and 174 may be GUIs related to emotional expression of the robot 100a.

As described later, the controller 180 can intuitively provide on/off information of the camera 142 to the user by loading a GUI set corresponding to the state of the camera 142 and displaying it through the display 152. have.

In terms of hardware, the memory 170 may include various storage devices such as ROM, RAM, EPROM, flash drive, and hard drive.

The controller 180 may include at least one processor or controller that controls the operation of the robot 100a. Specifically, the controller 180 may include at least one CPU, an application processor (AP), a microcomputer (or microcomputer), an integrated circuit, an application specific integrated circuit (ASIC), and the like.

For example, the controller 180 may include a processor 182, an image signal processor (ISP) 184, a display controller 186, and the like.

The processor 182 may control the overall operation of components included in the robot 100a. The ISP 184 may generate image data by processing an image signal acquired through the camera 142. The display controller 186 may control the operation of the display 152 based on a signal or data provided from the processor 182. The display 152 may output graphics or text according to the control of the display controller 186.

According to an embodiment, the ISP 184 and/or the display controller 186 may be included in the processor 182. In this case, the processor 182 may be implemented as an integrated processor that also serves as the ISP 184 and/or the display controller 186.

Referring to FIG. 6, the robot 100a may perform various operations and provide functions through a screen off mode, a surrounding environment concentration mode, a user concentration mode, a user conversation mode, and a content providing mode.

Briefly describing each operation mode, the screen off mode is a kind of standby mode, and the camera 142 and the display 152 may be turned off. However, in order to receive a user's voice (including a starting word, a command, etc.) or to detect surrounding sounds, the microphone 124 may maintain the on state.

The ambient environment concentration mode may be a mode for recognizing surrounding acoustic and visual information using the microphone 124 and the camera 142. For example, in the surrounding environment concentration mode, the controller 180 may receive the user's voice or acquire an image of the user. In addition, the controller 180 may output information by turning on the display 152 or may output a GUI for expressing emotions of the robot 100a.

The user concentration mode may be a mode in which the user (person) is recognized by the camera 142 and focuses on the user by tracking the recognized user using the camera 142. The controller 180 continuously acquires an image through the camera 142 to recognize the user's position (face position, etc.), and controls the rotation mechanism 160 so that the front of the robot 100a faces the recognized position. can do. In addition, the controller 180 may maintain the microphone 124 in an ON state and output information through the display 152 or may output a GUI for expressing emotions of the robot 100a.

The user conversation mode may be a mode for receiving a command or request from a user, processing a received command or request, or interacting with a user based on the command or request. For example, when a starting word and a command (or a predetermined request) are recognized from the voice received through the microphone 124, the controller 180 performs an interaction based on the command or request through the display 152 or the sound output unit 154, etc. Can provide. Also, the controller 180 may track a user based on the image acquired through the camera 142. In addition, the controller 180 may output a GUI for expressing emotion of the robot 100a through the display 152.

The content providing mode may be a mode in which various types of content are provided through the display 152 or the sound output unit 154. The content providing mode is a mode in which a separate application related to the content is executed, and functions (user tracking, startup word recognition, etc.) previously provided using the camera 142 or microphone 124 are applied in the application. Can be provided.

Also, the robot 100a may perform switching between the modes according to a specific condition. Some examples related to switching between the modes are as follows, but are not limited thereto.

(1) Switching from the screen off mode to the surrounding environment concentration mode: When a sound signal (voice, sound, etc.) of a predetermined level or higher is acquired through the microphone 124

(2) Switching from the ambient environment concentration mode to the screen off mode: No sound signal above a predetermined level is obtained through the microphone 124 for a predetermined time, and an event or situation is not detected from the image acquired through the camera 142. Occation

(3) Switching from the surrounding environment concentration mode to the user concentration mode: A user (person) is detected from an image acquired through the camera 142, or a physical stimulation (touch) through the touch input unit 122, a touch sensor, a gyro sensor, etc. , Shaking, external shock, etc.) are detected

(4) Switching from the user concentration mode to the surrounding environment concentration mode: When the user (person) is no longer detected from the image acquired through the camera 142, a separate input is received through the input unit 120 for a predetermined time. If not

(5) Switching from user conversation mode to user concentration mode: When the execution of the interaction (speech, information display, etc.) according to the recognized command (or request) is finished

(6) Switching from user concentration mode to user conversation mode: When the voice acquired through the microphone 124 contains a starting word (or includes a starting word and a command)

(7) Switching from the ambient environment concentration mode to the user conversation mode: When the voice acquired through the microphone 124 contains a starting word (or includes a starting word and a command)

(8) Switching from screen off mode to user conversation mode: When the voice acquired through the microphone 124 contains a starting word (or includes a starting word and a command)

(9) Switching from user conversation mode to screen off mode: When a command or input to turn off the display 152 is received

(10) Switching from user conversation mode to content provision mode: When a separate application is executed by command or input

(11) Switching from content provision mode to user conversation mode: When a separate application that was executed is terminated

Depending on the embodiment, the content providing mode may be switched to other modes as well as the user conversation mode.

As described above, the camera 142 may be turned on in other modes other than the screen off mode among the operation modes of the robot 100a.

Meanwhile, the robot 100a of the present invention may be a communication robot disposed in a home or the like. In this case, the robot 100a may acquire an image of a space in the home while the camera 142 is on.

However, it is difficult for the user of the robot 100a to visually recognize whether the camera 142 is currently in an ON state or an OFF state. That is, the user may be concerned about the exposure of private life as an image including the user, etc. is recorded by the camera 142 regardless of the user's intention.

The robot 100a according to an embodiment of the present invention allows a user to conveniently control the on/off state of the camera 142. In addition, the robot 100a may relieve the above-described concerns such as privacy exposure by notifying the user of the current state of the camera 142 through a GUI change according to the state change of the camera 142.

Hereinafter, with reference to FIGS. 7 to 13, an embodiment related to a GUI change according to the switching of the on/off state of the camera 142 included in the robot 100a will be described.

7 is a flowchart illustrating a control operation of a robot according to an embodiment of the present invention. 8 is an exemplary diagram of a GUI displayed on a display when the camera of the robot is on. 9 to 10 are exemplary diagrams illustrating a robot's processing operation according to a user's camera off request. 11 is an exemplary diagram of a GUI displayed on a display when the robot's camera is in an off state. 12 to 13 are exemplary diagrams illustrating a robot processing operation according to a user's camera-on request.

Referring to FIG. 7, the robot 100a may receive a camera off request from a user during operation (S100).

The camera off request may be received in the form of a voice, but is not necessarily the case, and may be received in the form of a touch input through the touch input unit 122 or a touch screen, or a gesture through the camera 142.

For example, when the camera-off request is received in the form of a voice, the processor 182 receives the voice corresponding to the camera-off request together with a voice corresponding to a preset starting word, or a predetermined voice after receiving the voice corresponding to the starting word. You can receive it within hours.

Meanwhile, the camera-off request may be received in operation modes in which the camera 142 is on (ambient environment concentration mode, user concentration mode, user conversation mode, etc.), but the camera 142 is turned off according to an embodiment. It can also be received in an operation mode (screen off mode) in a state.

The processor 182 may transmit the received voice to a voice recognition server connected through a network. The processor 182 may receive a result of recognizing the voice from the voice recognition server. For example, the processor 182 may first recognize the voice corresponding to the starting word and then obtain a recognition result of the voice corresponding to the camera off request, but this is not necessarily the case.

According to an embodiment, the voice recognition server may be the AI server 200 shown in FIG. 2. In this case, each of the voice corresponding to the starting word and the voice corresponding to the camera off request may be recognized through a model (artificial neural network 231a) learned through the learning processor 240 of the voice recognition server.

According to an embodiment, the processor 182 may directly recognize the received voice without transmitting it to the voice recognition server. For example, the processor 182 may recognize a voice corresponding to the starting word and a voice corresponding to the camera off request, respectively, through a model learned by the learning processor 130 in the robot 100a. Alternatively, the processor 182 receives data corresponding to the learned model from the voice recognition server and stores it in the memory 170, and the voice corresponding to the starting word and the voice corresponding to the camera off request through the stored data You can also recognize each. Accordingly, the recognition accuracy of the voice may be improved.

The robot 100a may turn off the camera 142 in response to the received request (S110).

When the camera-off request is recognized from the received voice, the processor 182 may process the camera-off request by switching the camera 142 to the off state (or maintaining the off state).

When the camera 142 is switched to the off state according to the camera off request, the camera 142 may not be turned on until a separate camera on request is received.

For example, when the operation mode of the robot 100a is switched from the screen off mode to the surrounding environment concentration mode in a general situation, the processor 182 may turn on the camera 142.

However, when the operation mode is switched from the screen off mode to the ambient environment concentration mode after the camera 142 is switched to the off state according to the camera off request, the processor 182 does not turn on the camera 142. I can.

That is, since the camera 142 maintains the off state before the user's camera-on request is received, exposure of the user's private life can be effectively prevented.

The robot 100a may display a GUI corresponding to the off state of the camera 142 through the display 152 (S120). According to an embodiment, the robot 100a may output a voice indicating the off state of the camera 142 through the sound output unit 154 (S130).

As described above in FIG. 5, the memory 170 includes a first GUI set 172 corresponding to the on state of the camera 142 and a second GUI set 174 corresponding to the off state of the camera 142. Can be saved. Each of the first GUI set 172 and the second GUI set 174 may include at least one GUI related to emotional expression of the robot 100a, and includes a GUI included in the first GUI set 172, GUIs included in the second GUI set 174 may be different from each other.

Referring to the example of FIG. 8, when the camera 142 is in the on state, the processor 182 corresponds to an emotion expression to be output from among at least one GUI included in the first GUI set 172 from the memory 170. The GUI can be loaded. The processor 182 may provide an emotional expression of the robot 100a by displaying the loaded GUI on the display 152.

For example, at least one GUI included in the first GUI set 172 includes a first GUI 801 corresponding to a basic emotional expression (eg, calmness or expressionlessness), a second GUI corresponding to a positive emotional expression such as pleasure ( 802), and a third GUI 803 corresponding to negative emotion expression such as sadness. The processor 182 may set an emotion expression to be output based on a state or event of the robot 100a, and may display a GUI corresponding to the set emotion expression on the display 152.

Meanwhile, when the camera 142 is switched to the off state according to the camera off request, the processor 182 loads at least one GUI included in the second GUI set 174 from the memory 170 and displays the display 152. ) Can be displayed. The user can conveniently recognize that the camera 142 has been turned off by confirming that the GUI included in the second GUI set 174 is displayed on the display 152.

Referring to FIGS. 9 to 10 with respect to steps S110 to S130, when it is assumed that the operation mode of the robot 100a is the ambient environment concentration mode, the camera 142 may be in the ON state. Since the camera 142 is in the on state, the processor 182 may display the first GUI 801 included in the first GUI set 172 on the display 152.

When the user 900 wants to turn off the camera 142, the user 900 may utter a voice 902 (eg, “Turn off the camera”) including a camera off request. According to an embodiment, the user 900 may first utter a voice 901 including a start word (eg, “high LG”) in order for the robot 100a to receive and process the voice 902. .

The processor 182 switches the operation mode to the user conversation mode according to the recognition result of the voice 901 including the starting word, and turns off the camera 142 based on the recognition result of the voice 902 including the camera-off request. I can make it. As described above, the recognition results of the

voices

901 and 902 may be received from a voice recognition server connected through a network.

As the camera 142 is turned off, the processor 182 may load the GUI 1001 included in the second GUI set 174 stored in the memory 170 and display it through the display 152. For example, the GUI 1001 may be a GUI that expresses the robot 100a with its eyes closed, but is not limited thereto. Based on the GUI 1001, the user 900 can intuitively recognize that the robot 100a cannot look around, that is, that the camera 142 is off.

According to an embodiment, the processor 182 may effectively inform the user 900 that the camera 142 is off by additionally outputting a voice message 1002 indicating that the camera 142 is in an off state.

According to an embodiment, the processor 182 may additionally guide that the camera 142 is turned off by turning off the

light output units

156a and 156b.

Meanwhile, as shown in FIGS. 11A and 11B, GUIs included in the second GUI set 174 may be implemented in various forms. For example, the GUI may be implemented as a GUI 1101 representing the appearance of the robot 100a wearing sunglasses, or may be implemented as a GUI 1102 indicating that the camera 142 is turned off. The

GUIs

1001, 1101, and 1102 of FIGS. 10 to 11 may be displayed identically regardless of the type of emotion of the robot 100a, but this is not necessarily the case.

According to an embodiment, at least one GUI included in the first GUI set 172 is output in the form of a color image, and at least one GUI included in the second GUI set 174 is output in the form of a black and white image. , On-state and off-state of the camera 142 may be distinguished from each other and displayed.

That is, the user can intuitively recognize that the camera 142 has been switched to the off state because the camera off request is normally processed based on the change in the GUI displayed through the display 152.

7 will be described again.

The robot 100a may provide a service or perform an operation while the camera 142 is turned off (S140).

The processor 182 may maintain the off state of the camera 142 even when a service or operation that requires driving the camera 142 is performed.

For example, in the above-described user concentration mode, the processor 182 may use the camera 142 to track the user. However, when the camera 142 is turned off according to the camera off request, the processor 182 may provide a user concentration mode with the camera 142 turned off. Accordingly, the processor 182 may perform a user's voice reception operation through the microphone 124, but may not perform the user's tracking operation.

Thereafter, the robot 100a may receive a camera-on request from the user (S150), and turn on the camera 142 in response to the received request (S160).

Similar to the camera-off request, the camera-on request may be received in various forms such as a touch input as well as a voice form.

When the camera-on request is received in the form of a voice, the processor 182 receives the voice corresponding to the camera-on request together with a voice corresponding to a preset starting word, or within a predetermined time after receiving the voice corresponding to the starting word. can do.

The camera-on request may be received in all operation modes (excluding a content providing mode depending on the embodiment).

The processor 182 may transmit the received voice to a voice recognition server connected through a network. The processor 182 may receive a result of recognizing the voice from the voice recognition server. For example, the processor 182 may first recognize the voice corresponding to the starting word and then obtain a recognition result of the voice corresponding to the camera-on request, but this is not necessarily the case.

According to an embodiment, the voice recognition server may be the AI server 200 shown in FIG. 2. In this case, each of the voice corresponding to the starting word and the voice corresponding to the camera-on request may be recognized through a model (artificial neural network 231a) learned through the learning processor 240 of the voice recognition server.

According to an embodiment, the processor 182 may directly recognize the received voice without transmitting it to the voice recognition server. For example, the processor 182 may recognize a voice corresponding to the starting word and a voice corresponding to the camera-on request, respectively, through a model learned by the learning processor 130 in the robot 100a. Alternatively, the processor 182 receives data corresponding to the learned model from the voice recognition server and stores it in the memory 170, and the voice corresponding to the starting word and the voice corresponding to the camera-on request through the stored data You can also recognize each. Accordingly, the recognition accuracy of the voice may be improved.

When a camera-on request is recognized from the received voice, the processor 182 may process the camera-on request by switching the camera 142 to an on state.

The robot 100a may display a GUI corresponding to the on state of the camera 142 (S170).

When the camera 142 is switched to the on state, the processor 182 may provide an emotional expression of the robot 100a using at least one GUI included in the first GUI set 172.

Depending on the embodiment, the processor 182 may output a voice indicating the on-state of the camera 142 through the sound output unit 154.

Referring to FIGS. 12 to 13 in relation to steps S150 to S170, since the camera 142 is in an off state, the processor 182 displays the GUI 1001 included in the second GUI set 174 (152). Can be marked on.

When the user 1200 wants to turn on the camera 142, the user 1200 may utter a voice 1202 (eg, “Turn on the camera”) including a camera-on request. According to an embodiment, the user 1200 may first utter a voice 1201 including a starter word (eg, “high LG”) so that the robot 100a may receive and process the voice 1202. .

The processor 182 switches the operation mode to the user conversation mode according to the recognition result of the voice 1201 including the starting word, and turns on the camera 142 based on the recognition result of the voice 1202 including the camera-on request. I can make it. As described above, the recognition result of the

voices

1201 and 1202 may be received from a voice recognition server connected through a network.

As the camera 142 is turned on, the processor 182 may load the GUI 801 included in the first GUI set 172 stored in the memory 170 and display it through the display 152. For example, the GUI 801 may be a GUI that expresses the appearance of the robot 100a opening its eyes. Based on the GUI 801, the user 1200 can intuitively recognize that the robot 100a can look around, that is, that the camera 142 is on.

According to an embodiment, the processor 182 may effectively inform the user 1200 that the camera 142 is on by additionally outputting a voice message 1301 indicating that the camera 142 is on.

According to an embodiment, the processor 182 may indicate that the camera-on request has normally been processed by outputting light of a predetermined color or pattern through the

light output units

156a and 156b.

That is, the user can conveniently recognize that the camera 142 is turned on by confirming that the GUI included in the first GUI set 172 is displayed on the display 152 or by listening to a voice indicating the on state of the camera 142. I can.

The robot 100a may provide a service or perform an operation while the camera 142 is turned on (S180).

The processor 182 may perform various services or operations using the turned on camera 142.

For example, in the above-described user concentration mode, the processor 182 may continuously acquire an image using the turned on camera 142. The processor 182 may track the user by controlling the rotation mechanism 160 based on the user's position included in the acquired image.

The above description is merely illustrative of the technical idea of the present invention, and those of ordinary skill in the art to which the present invention pertains will be able to make various modifications and variations without departing from the essential characteristics of the present invention.

Accordingly, the embodiments disclosed in the present invention are not intended to limit the technical idea of the present invention, but to explain the technical idea, and the scope of the technical idea of the present invention is not limited by these embodiments.

The scope of protection of the present invention should be interpreted by the following claims, and all technical ideas within the scope equivalent thereto should be interpreted as being included in the scope of the present invention.

Claims

camera;

A display for displaying a graphic user interface (GUI);

An input unit for receiving a camera off request; And

Switching the camera to an off state in response to the camera off request,

And a processor for controlling the display to display a GUI corresponding to the off state of the camera,

The GUI corresponding to the on state of the camera and the GUI corresponding to the off state of the camera are different from each other.
The method of claim 1,

The input unit includes at least one microphone,

The processor,

A robot that receives a first voice including the camera off request through the at least one microphone, and switches the camera to an off state in response to the received first voice.
The method of claim 2,

The processor,

Receiving a second voice corresponding to a preset starting word through the at least one microphone,

In response to the received second voice, switching the operation mode of the robot to a mode capable of processing the camera off request,

A robot that switches the camera to an off state in response to the first voice received through the at least one microphone.
The method of claim 3,

Further comprising a memory for storing the learning model learned by the learning processor,

The processor,

A robot that recognizes the starting word from the received second voice through the learning model stored in the memory and recognizes the camera off request from the received first voice.
The method of claim 2,

Further comprising a communication unit for connecting with the voice recognition server,

The processor,

Controlling the communication unit to transmit a voice corresponding to the camera off request received through the at least one microphone to the voice recognition server,

Receiving the recognition result of the voice from the voice recognition server through the communication unit,

A robot that switches the camera to an off state when the received recognition result corresponds to the camera off request.
The method of claim 1,

A first GUI set including at least one GUI corresponding to the on state of the camera, and a memory storing a second GUI set including at least one GUI corresponding to the off state of the camera,

The processor,

When the camera is switched to the off state, any one GUI included in the second GUI set is loaded from the memory,

A robot that displays a loaded GUI through the display.
The method of claim 1,

Including more speakers,

The processor,

A robot that controls the speaker to output a voice message corresponding to the off state of the camera.
The method of claim 1,

Further comprising a light output unit including at least one light source,

The processor,

A robot that turns off the at least one light source included in the light output unit as the camera is switched to an off state.
The method of claim 1,

The processor,

When switching the operation mode of the robot from the first mode in which the camera is turned off to a second mode in which the camera is turned on, a robot that does not switch the camera to the on state when the camera is turned off by the camera off request .
The method of claim 1,

The processor,

Receives a camera-on request through the input unit,

In response to the received camera-on request, the camera is turned on,

A robot that controls the display to display a GUI corresponding to the on state of the camera.
The method of claim 10,

The input unit includes at least one microphone,

The processor,

Receiving a first voice corresponding to a preset starting word through the at least one microphone,

In response to the received first voice, switching the operation mode of the robot to a mode capable of processing the camera-on request,

A robot that receives a second voice corresponding to the camera-on request through the at least one microphone and switches the camera to an on-state in response to the received second voice.
The method of claim 11,

Further comprising a memory for storing the learning model learned by the learning processor,

The processor,

A robot that recognizes the starting word from the received first voice and the camera on request from the received second voice through a learning model stored in the memory.
The method of claim 10,

A first GUI set including at least one GUI corresponding to the on state of the camera, and a memory storing a second GUI set including at least one GUI corresponding to the off state of the camera,

The processor,

When the camera is switched to the on state, a GUI corresponding to an emotional expression to be provided among at least one GUI included in the first GUI set is loaded from the memory,

A robot that displays a loaded GUI through the display.
The method of claim 10,

Including more speakers,

The processor,

A robot controlling the speaker to output a voice message corresponding to the on state of the camera.
The method of claim 10,

Further comprising a light output unit including at least one light source,

The processor,

As the camera is turned on, the robot controls the at least one light source to output light of a set color or pattern.