KR102636908B1 - Method and system for interaction between social robot and user based on non-verbal recognition - Google Patents

Abstract

A method and system for interaction between a social robot and a user are provided. The social robot learns the biometric information of at least one user, assigns an intimacy value when the biometric information collected from the user requesting interaction matches the learned biometric information, and interacts with multiple users according to the assigned intimacy value. It is configured to classify intimacy classes and prioritize interactions for each user according to the classified intimacy class level, or to determine the type of interaction that can be performed according to the classified intimacy class level.

Description

Method and system for interaction between social robot and user based on non-verbal recognition {METHOD AND SYSTEM FOR INTERACTION BETWEEN SOCIAL ROBOT AND USER BASED ON NON-VERBAL RECOGNITION}

The present invention relates to a method and system for interaction between a social robot and a user.

Recently, with the development of artificial intelligence (AI) technology, research on social service robots capable of interacting with humans is being actively conducted. In particular, social robots in the home are expected to provide customized services to users.

For example, social robots are being applied to artificial intelligence speakers, home robot vacuum cleaners, etc., or are being used as IoT hubs along with Internet of Things (IoT) technology. Unlike service robots that target an unspecified number of people, social robots used in spaces such as homes and offices are characteristic in that they generally provide customized services to a few specific users.

Meanwhile, thanks to the development of image recognition technology and data processing technology, technology to recognize or classify users based on image data is gradually being commercialized. For example, Patent Document 1 discloses a technology for recognizing a user using an image acquisition device such as a camera.

U.S. Patent Publication No. 7,930,555, Image recognition, 2011.04.19

Conventional social robots were based on user voice recognition. In other words, the situation was limited to collecting the user's voice information and performing actions according to the meaning of the instructions contained in the voice information. Examples of such voice recognition-based social robots include artificial intelligence speakers or voice recognition software installed in smartphones or vehicles.

However, as users' expectations for service robots rise, the demand for user-customized services is increasing compared to before. In addition, emotional communication between social robots and users is also required. However, there are limits to the functions that social robots based on user voice recognition can provide.

Accordingly, the present invention seeks to provide a social robot based on non-verbal recognition or non-verbal instructions. As described above, among various types of social robots, social robots used at home or in the office are differentiated from social robots that provide services to an unspecified number of people, such as kiosk services, in that they are mainly used by a few users. These home/office social robots seem to be capable of applying unique technologies in that they target a small number of users.

In other words, the problem to be solved by the present invention is to provide a non-verbal motion-based social robot that improves user convenience and provides customized services to users.

Another problem that the present invention aims to solve is to provide a control method or operation method for a social robot that improves user convenience and provides customized services.

Another problem that the present invention aims to solve is to provide an interaction system for non-verbal interaction between people and robots, including such social robots.

The problems of the present invention are not limited to the technical problems mentioned above, and other technical problems not mentioned will be clearly understood by those skilled in the art from the description below.

A social robot according to an embodiment of the present invention to solve the above problem learns the biometric information of at least one user, and sets an intimacy value when the biometric information collected from the user requesting interaction matches the learned biometric information. classifies the intimacy class of multiple users according to the assigned intimacy value, prioritizes interactions for each user according to the classified intimacy class level, or determines the types of interactions that can be performed according to the classified intimacy class level. It is configured to do so.

The social robot further classifies the age classes of multiple users according to the collected biometric information, wherein the age classes include a first level age class and a second level age class with higher authority than the first level age class, A certain interaction is configured to be performed only by users belonging to the second level age class among users of the affinity class above the reference level.

The social robot further classifies the gender classes of multiple users according to the collected biometric information, where the gender classes include a first gender class and a second gender class, and in which interaction, among users of the intimacy class above the reference level, It is configured to be performed only by users belonging to the second gender class.

In addition, the social robot receives a first interaction request from a first user of a first level intimacy class, and receives the first interaction request from a second user of a second level intimacy class higher than the first level intimacy class within a predetermined interval. When receiving a different second interaction request, it may be configured to perform the second interaction first.

The social robot may be configured not to perform the first interaction when the first interaction request and the second interaction request are of conflicting types.

The interaction may include a limited type of restricted interaction and a non-general purpose interaction.

At this time, the social robot does not perform the limited interaction request when receiving it from the first user of the first level intimacy class, but receives the limited interaction request from the second user of the second level intimacy class higher than the first level intimacy class. It can be configured to perform if so.

The interaction may include a voice interaction that can be performed based on the user's voice recognition, a non-verbal interaction that can be performed based on the user's non-verbal actions, and a complex interaction that requires non-verbal actions along with voice recognition of the user.

At this time, when a complex interaction request is received from a user, the social robot may be configured to sequentially determine voice instructions and non-verbal actions to perform complex interaction.

The biometric information may be collected including the user's eye position, head direction, arm position, hand position, eye blink frequency, voice waveform, iris information, facial color, eye corner position, eyebrow position, and mouth corner position.

In addition, the eye position, head direction, arm position, and hand position are defined as at least indicator biometric information, and the eye blink frequency, voice waveform, iris information, and facial color are defined as at least user identification biometric information, The eye corner position, eyebrow position, and mouth corner position may be defined as at least user state biometric information.

The social robot may be configured to determine the frequency of robot preemptive interaction based on the state biometric information.

The frequency of eye blinking is further defined as user state biometric information, and may be configured to further determine the frequency of robot preemptive interaction based on the eye blinking frequency.

The interaction may include at least a non-verbal or complex interaction based on a user's non-verbal movement, and the indicated biometric information may be configured to be used to determine whether to perform the non-verbal or complex interaction.

A method of controlling a social robot according to an embodiment of the present invention to solve the above other problems includes the steps of learning user information by scanning the user's biometric information in a state where there is no interaction from the user; When there is a request for interaction from a user, scanning biometric information of the requesting user and comparing it with learned biometric information; assigning an intimacy value when the scanned biometric information matches the learned biometric information; and classifying intimacy classes of multiple users according to the intimacy values of the multiple registered users.

In some embodiments, the method includes classifying a type of interaction; And it may further include comparing the type of interaction requested from a user with the intimacy class of the user.

Additionally, the method may further include determining the priority of interaction for each user based on the intimacy class level.

In some embodiments, the method includes classifying an age class based on the user's biometric information; Classifying gender classes based on the user's biometric information; and determining whether to perform the interaction request according to the level of the classified age class or the level of the gender class.

Interactions requested from the user can be classified into voice interactions that can be performed based on the user's voice recognition, non-verbal interactions that can be performed based on the user's non-verbal actions, and complex interactions that require non-verbal actions along with the user's voice.

When a voice-based interaction request is received from a user, the method includes determining whether the request is classified as a voice interaction or a complex interaction; And if the request is classified as a complex interaction, it may further include scanning whether the user's non-verbal behavior is present during a predetermined interval.

The non-verbal gesture may include at least one of eye position, head direction, arm position, and hand position.

The step of scanning the user's biometric information without interaction from the user may include collecting the frequency of the user's eye blinks.

At this time, the method may further include determining the frequency of robot preemptive interaction based on the frequency of the user's eye blinks.

A user-robot interaction system according to an embodiment of the present invention for solving the above-mentioned another problem includes the above-described social robot; and one or more Internet of Things devices.

The biometric information collected by the social robot may include indicated biometric information including eye position, head direction, arm position, and hand position.

In addition, when indicative biometric information related to the Internet of Things device is received from the user, it may include operating the Internet of Things device.

A device according to an embodiment of the present invention for solving the above-described still other problem includes a memory that stores one or more instructions, and a processor that executes the instructions stored in the memory to perform the above-described method of controlling a social robot.

Specific details of other embodiments are included in the detailed description.

According to embodiments of the present invention, the intimacy value between the social robot and the user can be calculated as the interaction is repeated based on the user's biometric information, for example, physical characteristics. Through this, the intimacy class between the social robot and the user can be classified, and based on this, the priority of the interaction task can be determined, or whether the interaction task can be performed.

Therefore, in the process of collaborating with social robots, the intimacy value and intimacy class can naturally improve, and users can feel the social robot as an entity they share their daily lives with, rather than simply an electronic device, and achieve emotional communication.

In particular, when a small number of users mainly use the social robot, it is possible to prevent others with insufficient intimacy, such as visitors, from interacting with the social robot. Alternatively, even among a small number of users, it is possible to classify unsuitable tasks by classifying interactive tasks and non-interactable tasks according to the class classified for each user, thereby allowing social robots to be more integrated into daily life.

Effects according to embodiments of the present invention are not limited to the contents exemplified above, and further various effects are included in the present specification.

1 is a configuration diagram of a user-robot interaction system according to an embodiment of the present invention.
FIG. 2 is a configuration diagram of the social robot of the interaction system of FIG. 1.
3 to 20 are diagrams for explaining the operation of a social robot according to an embodiment of the present invention.

The advantages and features of the present invention and methods for achieving them will become clear by referring to the embodiments described in detail below along with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below and will be implemented in various different forms, and only the embodiments serve to ensure that the disclosure of the present invention is complete, and those skilled in the art It is provided to fully inform the person of the scope of the invention, and the present invention is only defined by the scope of the claims. That is, various changes may be made to the embodiments presented by the present invention. The embodiments described below are not intended to limit the embodiments, but should be understood to include all changes, equivalents, and substitutes therefor.

Unless otherwise defined, all terms (including technical and scientific terms) used in this specification may be used with meanings that can be commonly understood by those skilled in the art to which the present invention pertains. Additionally, terms defined in commonly used dictionaries are not interpreted ideally or excessively unless clearly specifically defined.

As used herein, 'and/or' includes each and every combination of one or more of the mentioned items. Additionally, the singular form also includes the plural form unless specifically stated in the phrase. As used herein, 'comprises' and/or 'comprising' do not exclude the presence or addition of one or more other components in addition to the mentioned components. The numerical range expressed using 'to' indicates a numerical range that includes the values written before and after it as the lower limit and upper limit, respectively. ‘About’ or ‘approximately’ means a value or numerical range within 20% of the value or numerical range stated thereafter.

Hereinafter, the present invention will be described in detail with reference to the attached drawings.

1 is a configuration diagram of a user-robot interaction system according to an embodiment of the present invention.

Referring to FIG. 1, the interaction system 1 according to this embodiment includes users 20 and a social robot 10 that interacts with the users 20 based on verbal and/or non-verbal actions or recognition. And, it may further include Internet of Things devices 30 that are directly or indirectly controlled by the social robot 10. In some embodiments, the interaction system 1 may further include a user terminal 40 for controlling or setting the social robot 10.

Users 20 may refer to registered users registered by the social robot 10. Users 20 may be comprised of a plurality of users, including at least one user. In some embodiments, the interaction system 1 may further include unregistered users. This will be described later.

The social robot 10 may refer to a component that provides services by interacting with registered users 20. 1 and the like illustrate a social robot 10 in a form that roughly imitates the shape of a human, including a face and operable arms implemented by a display device, but the present invention is not limited thereto. That is not the case. In another embodiment, the social robot 10 is implemented in the form of a non-moving electronic device such as a set-top box or speaker, a moving electronic device such as a robot vacuum cleaner, or a home appliance electronic device such as a TV or refrigerator. It can also function as a social robot. The operation of the social robot 10 will be described in detail later.

In some embodiments, social robot 10 may communicate with Internet of Things devices 30 and function as an Internet of Things hub. The Internet of Things device 30 may include various electronic devices installed in a home or office. For example, the Internet of Things device 30 includes a TV, monitor, refrigerator, washing machine, door lock, security system, access system, power system, gas system, induction, lighting, air conditioner, heating equipment, air purification system, air purifier, It may include sprinklers, elevators, etc.

The social robot 10 can communicate with the Internet of Things device 30 using short-range wireless communication. Examples of the short-range wireless communication include Wireless-Fidelity (WiFi), WiFi-Direct, Wibro, Wimax, Zigbee, Zigbee-mesh, Bluetooth, NFC, and UWB, but this Of course, the invention is not limited thereto.

Additionally, the social robot 10 can communicate with the user terminal 40. The user terminal 40 may be a user terminal 40 used by one, some, or all of the plurality of users 20 . The user terminal 40 may include electronic devices such as a smartphone, tablet PC, PC, and smart watch. The social robot 10 according to the present invention performs mutual interaction with the user without direct manipulation by the user, but a user terminal 40 may be provided for direct manipulation, settings, and control of the social robot 10.

According to the interaction system 1 according to this embodiment, users 20 and the social robot 10 may be able to interact through verbal and/or non-verbal requests, instructions, and actions. Therefore, user convenience is improved compared to the prior art, and emotional communication with the social robot 10 can be achieved. In addition, when the social robot 10 is used in a space such as a home, it can provide a function as an IoT hub for various Internet of Things devices 30 in the home.

Hereinafter, the configuration of the social robot 10 according to this embodiment will be described in more detail.

FIG. 2 is a configuration diagram of the social robot of the interaction system of FIG. 1.

Referring further to FIG. 2, the social robot 10 according to this embodiment includes a processor 100, memory 200, storage 300, communication unit 400, input unit 500, and output unit 600. And may further include a sensor unit 700. However, since these components are merely examples of the various components included in the social robot 10 by separating them functionally, in an actual physical environment, the components described below may be integrated and implemented with each other. Additionally, the term '~ part' used in this specification may be used to include a hardware configuration such as a processor or circuit and/or a software configuration executed by the hardware configuration.

The processor 100 may implement operations and/or functions related to the method according to the present invention based on instructions according to software 200s in which the method according to the present invention is implemented, which resides in the memory 200. The processor may use a known processor, but may be implemented through, for example, an Application-Specific Integrated Circuit (ASIC), or another chipset, logic circuit, and/or data processing device. Additionally, at least some of the processors 100 may be manufactured in the form of dedicated hardware chips for artificial intelligence.

Software 200s implementing the method according to the present invention may be loaded in the memory 200. The memory may use any known memory, but may be implemented through, for example, Read-Only Memory (ROM), Random Access Memory (RAM), flash memory, a memory card, a storage medium, and/or other storage devices. The memory 200 may be inside or outside the processor 100 and may be connected to the processor 100 by known means.

The storage 300 may store application programming interfaces (APIs), libraries, resource files, etc. required for execution of software 300s in which the method according to the present invention is implemented. Additionally, the storage 300 can store software 300s in which the method according to the present invention is implemented. Additionally, the storage 300 may store a database 310 for performing the method according to the present invention.

Here, the database 310 may include user registration information, biometric information of the registered user, classified age class, classified gender class, intimacy value information, classified intimacy class, type of interaction, etc., but the present invention It is not limited to this.

However, the database 310 according to this embodiment is not limited to those listed above, and more detailed contents of the database 310 can be understood from the description below to explain the operation of the social robot 10. .

The software 300s, which resides in the memory 200 or stored in the storage 300, scans the user's biometric information in the absence of user interaction to learn user information. When there is an interaction request from the user, the requested user's A step of scanning biometric information and comparing it with learned biometric information, assigning an intimacy value if the scanned biometric information matches the learned biometric information, and intimacy classes of multiple users according to the intimacy values of multiple registered users. It may be a computer program recorded on a recording medium to execute the method according to the present invention including the step of classifying.

In some embodiments, the method includes classifying the type of interaction, comparing the type of interaction requested from a user with the intimacy class of the user, and determining the priority of interaction for each user based on the intimacy class level. A step of classifying an age class based on the user's biometric information, a step of classifying a gender class based on the user's biometric information, and determining whether to perform an interaction request according to the level of the classified age class or the level of the gender class. Determining step, when a voice-based interaction request is received from the user, determining whether the request is classified as a voice interaction or a complex interaction. If the request is classified as a complex interaction, the user's non-verbal It may further include scanning for operation and determining the frequency of robot preemptive interaction based on the frequency of the user's eye blinks.

However, the method according to this embodiment is not limited to those listed above, and more detailed content of the method can be understood from the following description to explain the operation of the social robot 10.

The method according to the present invention may be implemented by hardware, firmware, software, or a combination thereof. For example, when implemented as hardware, it may be understood as being implemented by the processor 100. For another example, when implemented as software, the above-described method may be implemented as a module, such as a process or function, that performs functions to be described later. The module resides in memory 200 and can be executed by processor 100. That is, it can be understood that a hardware device can be configured to operate as one or more software to perform operations according to the present invention, and vice versa.

Additionally, in the case of implementation using firmware or software, the method according to the present invention may be implemented in the form of a module, procedure, or function that performs functions or operations described later and recorded on a recording medium readable through various computer means. Here, the recording medium may include program instructions, data files, data structures, etc., singly or in combination. Program instructions recorded on the recording medium may be those specifically designed and configured for the present invention, or may be known and usable by those skilled in the art of computer software technology. Examples of program instructions may include machine language code, such as that produced by a compiler, as well as high-level language code that can be executed by a computer using an interpreter, etc.

Meanwhile, the communication unit 400 allows the social robot 10 to transmit and receive data with other devices, such as the Internet of Things device 30, the user terminal 40, and/or a server (not shown). The communication unit 400 may include a short-range communication unit 410 and/or a mobile communication unit 430.

As described above, the short-range communication unit 410 may be provided to communicate with the Internet of Things device 30, etc. Since the communication method of the short-range communication unit 410 has been described above, redundant description will be omitted.

The mobile communication unit 430 may transmit and receive data with at least one of a base station, an external terminal, or an external server using a communication network that is a combination of one or more of a mobile communication network or a public wired communication network. Examples of mobile communication networks include Code Division Multiple Access (CDMA), Wide Band Code Division Multiple Access (WCDMA), High Speed Packet Access (HSPA), and Long Term Evolution (LTE). In addition, examples of public wired communication networks include Ethernet, digital subscriber line (xDSL), hybrid optical coaxial network (HFC), and optical subscriber network (FTTH). You can communicate with the user terminal 40 using a mobile communication network, or collect data such as current time and weather information.

The input unit 500 allows the social robot 10 to recognize the external environment or receives a signal regarding a manipulation trigger from the user 20 . The input unit 500 may include a camera 510, a microphone 530, and/or a touch panel 550.

Camera 510 is a general term for an image recognition unit. The social robot 10 can use the camera 510 to collect biometric information such as the user's face, expression, body shape, and iris information and data about the surrounding environment. Specifically, the camera 510 can obtain image frames such as still images or moving images using an image sensor. The camera 510 may be located adjacent to the display panel 610 disposed on the facial area of the social robot 10.

Microphone 530 is a general term for voice recognition units. The social robot 10 can collect the user's voice information using the microphone 530. The collected voice information can be processed into electrical voice data. Here, the user's voice information may include information including the user's instruction interaction and voice waveform information for user identification. This will be described later.

The touch panel 550 is a general term for a unit that receives input information through a touch operation by the user 20. The touch panel 550 may be implemented by being combined with or built into the display panel 610, which will be described later.

Although not shown in the drawing, the input unit 500 may further include a USB terminal for connection to a mouse, keyboard, etc. in addition to the units described above.

The output unit 600 may implement data output by the social robot 10, the current state of the social robot 10, or an alarm in a form that can be recognized by humans. The output unit 600 may include a display panel 610, a speaker 630, and/or a vibration unit 650.

The display panel 610 is a general term for an image output unit. The display panel 610 can display information that the social robot 10 wants to convey to the user in a visual form. In some embodiments, the display panel 610 may be placed on the facial area of the social robot 10, and the display panel 610 may be used to display facial expressions such as virtual eyes of the social robot 10. Thus, emotional communication between the user 20 and the social robot 10 can be improved.

Speaker 630 is a general term for an audio output unit. The speaker 630 can express information that the social robot 10 wants to convey to the user in an auditory form.

The vibration unit 650 may be implemented including a vibration motor, etc. The vibration unit 650 may provide auditory and tactile information when the social robot 10 provides an alarm to the user.

The sensor unit 700 can collect various environmental stimuli around the social robot 10. The sensor unit 700 may be implemented by being integrated with any of the above-described components, such as the camera 510 and the touch panel 550, but the present invention is not limited thereto. The sensor unit 700 may include a temperature sensor, a humidity sensor, a gas sensor, a biometric sensor, a proximity sensor, an illumination sensor, a gyro sensor, and/or a gesture sensor. The above-mentioned sensors can be used as known sensors, and the functions of the sensors can be intuitively inferred from their names, so detailed descriptions are omitted.

In an exemplary embodiment, the biometric sensor may collect and process the user's biometric information. For example, the biometric sensor may process the user's iris information collected through the camera 510, etc. Additionally, the gesture sensor can process the presence or absence of a user's motion or the type of motion collected through the camera 510 or the like.

The data bus 800 includes a processor 100, memory 200, storage 300, communication unit 400, input unit 500, output unit 600, sensor unit 700, and/or other components. It is connected to and can serve as a moving path to transmit data between each component.

The social robot 10 according to this embodiment may include a logical component for implementing the method according to the present invention. Examples of the logical components include a user registration unit, a user recognition unit, an intimacy value arithmetic unit, an intimacy class determination unit, a gender class determination unit, an age class determination unit, a communication unit 400, an input unit 500, an output unit 600, and a sensor. It may be unit 700, etc., but of course, the present invention is not limited thereto.

Hereinafter, features related to the operation method or control method of the social robot 10 according to an embodiment of the present invention, interaction between the social robot 10 and the user using this, and the resulting increased user convenience effect will be described in detail. do.

3 to 20 are diagrams for explaining the operation of a social robot according to an embodiment of the present invention.

First, referring to FIGS. 3 and 4, the social robot 10 continuously scans the biometric information of users periodically or aperiodically (S110), stores the biometric information for each user in a database, and can learn by accumulating it. There is (S210). Although the present invention is not limited to this, the social robot 10 can learn biometric information of users mainly in the absence of requests or instructions from users, that is, in the absence of user preemptive interaction.

Here, the user's biometric information learned by the social robot 10 includes the user's eye position, head direction, arm position, hand position, eye blink frequency, voice waveform, iris information, eye corner position, eyebrow position, and mouth corner position. It can be included.

More specifically, the user's biometric information learned by the social robot 10 may include eye blink frequency, voice waveform, iris information, eye corner position, eyebrow position, and mouth corner position. Such information can be used to at least identify the user or determine the user's status.

That is, biometric information such as eye position, head direction, arm position, hand position, eye blink frequency, voice waveform, iris information, eye corner position, eyebrow position, and mouth corner position collected by the social robot 10 is indicated. It may be classified and defined as at least one of biometric information, identification biometric information, and status biometric information.

Specifically, the eye position, head direction, arm position, and hand position may be defined as indicated biometric information. However, the present invention is not limited thereto. The indicated biometric information will be described later along with FIG. 15 and the like.

Additionally, the eye blink frequency, voice waveform, iris information, and facial color may be defined as at least user identification biometric information. User identification biometric information may be information used to register a user, assign an intimacy value to be described later, and determine whether the user matches a classified class.

Additionally, the eye corner position, eyebrow position, and mouth corner position may be defined as at least user state biometric information. User status biometric information may be information used to infer the user's current status. The above-mentioned eye blink frequency can also be used as user state identification information.

It is already known that the voice waveform, iris information, facial color, and facial shape using the positions of the eyes, nose, and mouth are different for each person, and that these can be used as identification marks for each person. In addition, technology has been developed to identify users using differences in voice, iris, facial color, or facial shape. In addition, the present invention proposes identifying the user using the frequency of eye blinks.

For smartphones, where user identification technology is mainly applied, the time the user spends facing the smartphone is not very long. Therefore, the need to identify users based on the frequency of eye blinks has not been proposed, or at least it has been difficult to identify users using the frequency of eye blinks. However, the social robot 10 used in home/office, such as the present invention, can track the user's location within the same space and continuously observe the user's eye blinks. Additionally, since there are differences in the frequency of eye blinks for each person, this can be used to identify the user, or at least can be used as an auxiliary method in addition to other biometric information, such as voice or facial shape, to identify the user.

If there is no matching user in the database after scanning the user (S110), for example, in the initial state of the social robot 10 or when a visitor other than a family member visits, register as a new user based on the collected biometric information. Do it (S220). In other words, the social robot 10 can register a user recognized as a user by collecting biometric information to a sufficient extent to specify a certain user, and determine the number of registered users.

In addition, the social robot 10 can cumulatively reflect the scans (S110) of users registered as existing users, that is, users whose biometric information exists in the database, in the learning results (S210).

Meanwhile, the social robot 10 may classify the intimacy class, gender class, and/or age class for each user based on a database of collected biometric information (S300).

For example, the social robot S10 may classify users' age classes based on a database of collected biometric information. Here, the age class may be a classification based on age. For example, infants and toddlers can be classified into the first level age class, children or young children can be classified into the second level class, and adults and older adults such as young adults can be classified into the third level class.

Classification of age classes can be learned by the social robot 10 based on the user's voice, appearance, and other physical characteristics. However, in other embodiments, age classes may be directly entered or modified by the user using the user terminal 40, etc. .

Additionally, the social robot 10 can classify users' gender classes based on a database of collected biometric information. Here, the gender class can be classified as male, female, or other genders. Likewise, the social robot 10 can learn the classification of the gender class based on the user's voice, appearance, and other physical characteristics, but in another embodiment, the gender class may be directly entered or modified by the user using the user terminal 40, etc. there is.

The intimacy class will be explained with Figure 5.

If there is no user interaction from users or if interaction requests are not sufficiently accumulated, the class for each user stored by the social robot 10 may not exist.

Here, learning can be accomplished by an artificial intelligence learning model based on a neural network. For example, models such as Deep Neural Network (DNN), Recurrent Neural Network (RNN), and Bidirectional Recurrent Deep Neural Network (BRDNN) may be used as data recognition models, but the present invention is not limited thereto. It is clear that the artificial intelligence learning model performs learning based on user-specific characteristics, which will be described above or later.

Next, referring further to FIGS. 5 and 6, when there is an interaction or request for interaction from a user (S120), the social robot 10 identifies the requesting user (S130), scans the requested user's biometric information again, and then scans The biometric information of the user is compared with the biometric information of the stored or learned user (S205).

In order to determine whether the user attempted to interact with the social robot 10 (S120), the user may use a trigger for interaction. For example, the name of the social robot 10, for example, the word 'Kongkong', can be used as a trigger.

The user's interaction may include everyday conversations or jokes as well as requests for operations of other IoT devices 30, instructions, or simple information queries such as checking the date or weather.

The social robot 10 determines whether there is a match by comparing the results of scanning the biometric information of the user requesting interaction with the information learned from the biometric information of the user (S205). If the scan result and the learned result match, intimacy class classification for the user can be performed (S310).

Specifically, the social robot 10 may assign an intimacy value to the user when the scan result and the learning result match (S311). On the other hand, if the scan result and the learned result do not match, search for a matching user among other registered users, such as Users 1 to 5 (e.g., S200 in FIG. 4), or register as a new user and learn. You can proceed (S225).

Additionally, the social robot 10 can classify or update the intimacy class by accumulating the intimacy value assigned to each user (S312). For example, if the intimacy value is 0, there is no intimacy class, if the intimacy value is 1 to 10, it is classified as a first level intimacy class, and if the intimacy value is 10 to 50, it is classified as a second level intimacy class. It may be, etc.

As a plurality of users repeat the above process, the intimacy value with the social robot 10 accumulates and the intimacy class may gradually increase. Additionally, users classified into an intimacy class higher than a predetermined standard can be recognized as family.

As explained with FIGS. 3 to 6, the social robot 10 according to this embodiment simply repeats learning of biometric information (S200, S210 of FIG. 4) or grants intimacy (S210 of FIG. 4) depending on the presence or absence of user interaction. Steps S205 and S311 of FIG. 6 can be performed. In other words, the user scan (S110) of FIG. 4, which is performed without preemptive user interaction, and the search or scan (S205) to match the learning results with the recognized biometric information of FIG. 6, which is performed in response to the user's interaction. ) can be distinguished from each other.

Next, with further reference to FIGS. 7 and 8, when new users that were not previously recognized, such as User 6 and User 7, visit, the social robot 10 registers the visitors because the biometric information of the users has not been sufficiently learned. You may not be recognized as a user. However, it is possible to learn biometric information for a certain period of time, register as a new user, and perform classification into age class and/or gender class with priority.

In the initial state, affinity classes may not exist for User 6 and User 7. Additionally, even if there is a user preemptive interaction from User 6, an intimacy value may not be assigned to the social robot 10 because there is no accumulated biometric information for User 6. On the other hand, if biometric information accumulates over time, such as when User 6 visits multiple times, an intimacy value may be assigned even though he is not an actual family member.

Meanwhile, user instructions, that is, user preemptive interaction, can be classified into several types depending on their nature. For example, depending on the level of the intimacy class, it can be classified into interactions that can be performed and interactions that cannot be performed. The operation of operating the Internet of Things device 30 in the home, for example, a TV, may require an intimacy class level higher than a predetermined standard.

Accordingly, despite User 6's request (S120, S130) where the intimacy class does not exist or is insufficient, the social robot 10 may provide feedback that it does not respond to the interaction or cannot perform the request. On the other hand, if User 2 whose intimacy class is level 6 requests the same interaction, the social robot 10 can respond to the interaction.

The present invention is not limited to this, but interactions other than instructions for specific tasks, such as simple weather queries or date queries, may be performed regardless of the intimacy class for each user.

As described above, interaction tasks that require an intimacy class level higher than a predetermined standard can be defined as restricted interactions, and interaction tasks that do not require an intimacy class level can be defined as general-purpose interactions. Also, in restricted interactions, the types of restricted interactions can be further subdivided, such as requiring a higher intimacy class level or being possible only with a relatively low intimacy class level. Additionally, as will be described in detail later, restricted interactions may additionally require gender class or age class information. This will be described later.

In this way, the social robot 10 according to the present invention can recognize the verbal or non-verbal characteristics of users, classify the intimacy class, and distinguish between interactions that can be performed and interactions that cannot be performed according to the intimacy class (S400). In other words, different feedback can be provided depending on the intimacy class. To this end, the social robot 10 may first determine whether the interaction request is a general-purpose interaction or a limited interaction (S400), and determine the user's intimacy class level (S410) for those defined as limited interactions. Therefore, the user can recognize the social robot 10 as a member of the family. Additionally, interactions of visitors or outsiders other than family members can be selected and manipulation of the Internet of Things device 30, etc. can be prevented.

Next, with further reference to FIGS. 9 and 10 , the social robot 10 may prioritize interactions according to the intimacy class level for each user.

For example, when user 1 requests a decrease in the TV volume (S120a, S130a) and at the same time user 2 requests an increase in the TV volume (S120b, S130b), the social robot 10 selects the user with a sufficient affinity class level. Face situations where you receive conflicting, conflicting, or conflicting instructions from others. Although not shown in the drawing, the social robot 10 may determine whether two instructions conflict or whether they can be performed simultaneously.

At this time, although not shown in the drawing, the social robot 10 determines whether each user's request is a general-purpose interaction or a limited interaction, as shown in FIG. 8, and in the case of an instruction belonging to a limited interaction, the request of User 1 and User 2 It is determined whether each interaction meets the intimacy class restriction conditions (S500). If it is not appropriate when comparing User 1's instructions and User 1's intimacy class level, it may not respond to User 1's interaction. Similarly, when comparing User 2's instructions with User 2's intimacy class level, if it is not appropriate, it may not respond to User 2's interaction.

Additionally, if both User 1 and User 2 meet the intimacy class level conditions, the intimacy level classes of User 1 and User 2 can be compared (S510). Additionally, the interaction of User 1, which has a higher intimacy class level, can be performed with priority (S520).

Alternatively, even if the users' instructions are not received simultaneously, but User 2 requests an increase in TV volume and then User 1 requests a decrease in TV volume within a predetermined interval, the social robot 10 responds to User 1's interaction. It can be done first.

For another example, when User 4 and User 5 do not conflict with each other but request different types of interactions at the same time, especially when one user's request requires a certain amount of time, the social robot 10 may request user 4 The intimacy class levels of user 5 and user 5 can be compared (S510). Additionally, the interaction of User 4, who has a higher intimacy class level, can be performed with priority (S520).

Alternatively, even if User 4 makes a request within a predetermined interval after User 5 makes a request, the social robot 10 may perform User 4's interaction with priority.

In order to perform the above operation, the social robot 10 can determine the priority for each user even within the same intimacy class level. The priority may additionally use age class information, or users who first reach a predetermined intimacy class level may be prioritized. For example, even if User 2 and User 3, both of whom have affinity class levels of 6, make conflicting requests, the social robot 10 determines whose interaction between User 2 and User 3 will be processed first based on the stored data. can be decided.

Next, with further reference to FIGS. 11 and 12 , the social robot 10 can determine whether to perform user interaction based on not only the intimacy class for each user but also the age class. Here, interaction limited by age class can be understood as a type of restricted interaction.

For example, opening the front door lock is related to security, and it may not be appropriate to perform the request simply because the intimacy class level is high. Therefore, interaction requests classified into a specific type, such as opening the front door lock, can be performed only by requests from users of an intimacy class higher than a predetermined standard as well as users of a standard age class or higher.

That is, when User 2, whose age class is level 3 (adult), requests to open the front door lock, the social robot 10 can follow the request. On the other hand, even if User 4, whose age class is level 1 (infant), requests to open the front door lock, even if User 4's intimacy class is higher than that of User 2, the social robot 10 will not comply with the request. You can.

Of course, the interaction regarding the door lock opening request as described above is not performed by User 7, who does not have a sufficient intimacy class, even if the age class is high, such as that of the visitor.

To this end, the social robot 10 checks the user's interaction request, identifies the user (S120, S130), and then determines whether the type of interaction is a general interaction or a restricted interaction (S400). If it is determined to be a restricted interaction, especially an age-restricted interaction, the intimacy class level determination (S410) and the age class level determination (S430) are sequentially performed as described above.

Next, with further reference to FIGS. 13 and 14 , the social robot 10 may determine whether to perform user interaction based on not only the intimacy class for each user but also the gender class and/or age class. Here, interaction limited by gender class and/or age class can be understood as a type of restricted interaction.

For example, it may be appropriate for a specific person in the household to order and pay for food ingredients at a supermarket. This is because food ingredients may be ordered twice due to misunderstandings between family members. Accordingly, interaction requests classified into a specific type, such as ordering food ingredients, can be performed only by requests from users of an intimacy class higher than a predetermined standard, as well as user requests higher than a specified gender class and/or standard age class.

That is, when user 1, whose gender class is female and who has a high age class level, requests an order for food ingredients, the social robot 10 can comply. On the other hand, even if it has sufficient intimacy class and age class, even if User 2, whose gender class is male, requests an order for food ingredients, the social robot 10 may not comply with the request or may provide feedback that the order is scheduled to be ordered by User 1. In addition, even if it corresponds to a sufficient intimacy class and an appropriate gender class, even if User 5, who is in a lower age class, requests an order for food ingredients, the social robot 10 may not comply with the request or may provide feedback that the order is scheduled to be ordered by User 1. .

It goes without saying that the interaction related to ordering food ingredients as described above, for example, is not performed by User 7, who does not have a sufficient intimacy class even if the age class and gender class match the visitor.

To this end, the social robot 10 checks the user's interaction request, identifies the user (S120, S130), and then determines whether the type of interaction is a general interaction or a restricted interaction (S400). If it is judged to be a restricted interaction, especially a gender-restricted or gender/age-restricted interaction, the intimacy class level is determined first (S410) as described above, and additionally, the gender class level is determined (S420) and the age class level is determined (S420). S430) is performed.

Meanwhile, the type of food ingredients, supermarket, payment method, etc. to be ordered may be pre-specified through the user terminal 40, but the present invention is not limited thereto.

Next, with further reference to FIGS. 15 and 16 , the social robot 10 may determine whether to perform user interaction based on not only voice-based instructions from the user but also non-verbal characteristics, such as movements. Here, the nonverbal characteristic may refer to the above-described indicated biometric information.

In addition, the interaction related to the above-mentioned voice-based instructions is classified as a voice interaction that can be performed only by recognizing the user's voice among limited interactions, and the interaction that can be performed only with non-verbal characteristics is classified as a non-verbal interaction that can be performed only by the user's movements among limited interactions. It can be understood that

For example, when User 3 feels hungry, he or she may perform an action other than the action targeting the social robot 10. If User 3 feels hungry, he or she may stare at the refrigerator for more than a predetermined period of time, or may stare alternately at the refrigerator and the kitchen cabinet.

In this case, although User 3 did not perform an instruction interaction with the social robot 10, the social robot 10 scans User 3 (S110) to capture a type of instruction biometric information and performs a robot preemptive interaction based on this. You can.

Specifically, if the non-verbal movement scanned from User 3 (S110) is determined to belong to a pre-registered user (S200), it is determined whether there is anything classified as indicated biometric information among the movements of User 3, and if indicated biometric information exists, Determine whether a corresponding robot preemptive interaction exists (S610). If there is no matching result as a result of the judgment, the user's scan (S110) continues, and if there is a matching one, a robot preemptive interaction is suggested.

In this way, according to the social robot 10 and the interaction system 1 according to the present invention, even if the user does not give a direct command to the social robot 10 based on voice recognition, the social robot 10 and the person can use the non-verbal interaction. The social robot 10 may perform preemptive interaction. Through this, emotional communication between people and social robots (10) can be further strengthened.

Next, with further reference to FIGS. 17 and 18 , the social robot 10 may determine whether to perform user interaction based on non-verbal characteristics, such as movements, along with voice-based instructions from the user. Here, the nonverbal characteristic may refer to the above-described indicated biometric information.

Additionally, interactions that require non-verbal characteristics along with voice instructions can be understood as complex interactions among limited interactions.

For example, although it is not intended to be a word included in words exchanged between people in everyday life, that is, it is not a vocal command for interaction with the social robot 10, the social robot 10 uses it as a command interaction. If recognized as such, problems may arise. As mentioned above, words such as 'kongkong' can be used as a trigger, but this alone is not sufficient and auxiliary restrictions may be necessary.

In other words, even when users want to turn on the TV, the social robot 10 does not respond simply with a command to turn on the TV, and the social robot 10 can respond only when the user giving the voice command gazes at the TV.

Or, for another example, when accessing YouTube using a TV, User 4 may not want others to access his account. However, this classification may be difficult to sufficiently distinguish using only the above-mentioned intimacy class, age class, and gender class. In such cases, instructions based on non-verbal movements can be used like a simple password.

In other words, even if User 2 requests to access YouTube with User 4's account by voice, the interaction may not be performed if predetermined biometric information is not collected. On the other hand, when User 4 requests access to YouTube by voice and simultaneously uses predetermined biometric information, for example, drawing a V with his hand, the social robot 10 can perform an interaction request/instruction.

To this end, when a user interaction including voice is identified (S121, S130), the social robot 10 determines whether the type of interaction belongs to voice interaction, non-verbal interaction, or complex interaction (S600).

Here, determining the type of interaction (S600) refers to determining whether the interaction is a voice interaction, a non-verbal interaction, or a complex interaction, and the type of interaction is determined whether it is a limited interaction or a general-purpose interaction as shown in FIG. 8. It can be distinguished from judging (S400).

Then, by scanning the user, it is possible to determine whether indicated biometric information is obtained and whether the indicated biometric information matches pre-specified information, etc. (S620). That is, the voice instruction (S121) may be first obtained, and then the step (S620) of determining whether the non-verbal actions match may be performed.

Next, with further reference to FIGS. 19 and 20, the social robot 10 may determine the frequency or whether to perform robot preemptive interaction based on state biometric information (S110, S700) collected from the user (S710).

While a user, for example, user 4, is watching TV, the social robot 10 can determine the frequency or whether to perform robot preemptive interaction. The type of robot preemptive interaction is not particularly limited, but may include informing the user of the current time or asking the user if there is anything he/she needs.

At this time, the social robot 10 may scan User 4 (S110) and collect biometric information, particularly state biometric information, from User 4. Here, the scan (S110) may be collected when there is no user preemptive interaction. If the average eye blink frequency of User 4 is stored as 8 to 11 times per 60 seconds, and the currently collected average eye blink frequency of User 4 is lower than the above average, the social robot 10 detects that User 4 is watching TV. It is determined that the robot is in a state of concentration (S700), and the frequency of robot preemptive interaction can be reduced, or the robot preemptive interaction can not be provided (S710).

As described above, the social robot 10 and the interaction system 1 using the same according to the present invention can use non-verbal interaction rather than simply verbal interaction between the user and the robot as in the prior art.

Therefore, users can receive customized services in a more intuitive and user-friendly manner, and furthermore, emotional communication with the social robot 10 can be improved.

Each component on the configuration diagram or block diagram of the attached drawings may be implemented with software, hardware such as FPGA (field-programmable gate array), or ASIC (application specific integrated circuit). However, each component on the configuration diagram or block diagram may be implemented in an addressable storage medium as well as software and hardware, and may be configured to execute one or more processors.

Each component on a configuration diagram or block diagram may mean a module, segment, or part of code that includes one or more executable instructions for executing a specified logical function. Therefore, the functions provided by the components on the configuration diagram or block diagram may be implemented by a plurality of more detailed components, or the multiple components on the configuration diagram or block diagram may be implemented by one integrated component. Of course.

That is, within the scope of the purpose of the present invention, each component can be operated by selectively combining one or more of them. In addition, all components may be implemented as independent hardware, and some or all of the components may be selectively combined to have a program module that performs some or all of the combined functions in one or more pieces of hardware. It may also be implemented as a computer program. The codes and code segments that make up the computer program can be easily deduced by those skilled in the art. Such a computer program can be stored in a computer-readable recording medium or storage medium and can be read and executed by a computer to implement embodiments of the present invention. Examples of the recording medium include magnetic recording media, optical recording media, and the like.

Although the description has been made above with a focus on embodiments of the present invention, this is merely an example and does not limit the present invention, and those skilled in the art will be able to understand the present invention without departing from the essential characteristics of the embodiments of the present invention. It will be apparent that various modifications and applications not exemplified above are possible.

Therefore, the scope of the present invention should be understood to include changes, equivalents, or substitutes of the technical ideas exemplified above. For example, each component specifically shown in the embodiments of the present invention can be modified and implemented. And these variations and differences in application should be construed as being included in the scope of the present invention as defined in the appended claims.

1: Human-robot interaction system
10: Social robots
20: User
30: Internet of Things devices
40: user terminal

Claims (22)

Learn the biometric information of at least one user, and assign an intimacy value if the biometric information collected from the user requesting interaction matches the learned biometric information,
It is configured to classify the intimacy classes of multiple users according to the assigned intimacy value, prioritize interactions for each user according to the classified intimacy class level, or determine the type of interaction that can be performed according to the classified intimacy class level. ,
The biometric information includes at least the frequency of the user's eye blinks,
configured to determine the frequency of robot preemptive interaction based on the frequency of eye blinking,
The robot preemptive interaction includes informing the current time, a social robot. According to paragraph 1,
Further classifying the age classes of multiple users according to the collected biometric information, wherein the age classes include a first level age class and a second level age class with higher authority than the first level age class,
A social robot configured such that a certain interaction is performed only by users belonging to a second level age class among users of an intimacy class above the reference level. According to paragraph 1,
The gender classes of multiple users are further classified according to the collected biometric information, wherein the gender classes include a first gender class and a second gender class,
A social robot configured such that an interaction is performed only by users belonging to a second gender class among users of an intimacy class equal to or higher than a reference level. According to paragraph 1,
Receive a first interaction request from a first user of a first level affinity class,
A social robot configured to preferentially perform a second interaction when receiving a second interaction request different from the first interaction from a second user of a second level intimacy class higher than the first level intimacy class within a predetermined interval. According to paragraph 4,
A social robot configured not to perform a first interaction when the first interaction request and the second interaction request are of conflicting types. According to paragraph 1,
The interactions include limited types of restricted interactions and non-general-purpose interactions,
Social configured not to perform when receiving a limited interaction request from a first user of a first level affinity class, but to perform when receiving a limited interaction request from a second user of a second level affinity class higher than the first level affinity class. robot. According to paragraph 1,
The interaction includes a voice interaction that can be performed based on the user's voice recognition, a non-verbal interaction that can be performed based on the user's non-verbal actions, and a complex interaction that requires non-verbal actions along with voice recognition of the user,
A social robot configured to perform complex interaction by sequentially determining voice instructions and non-verbal actions when a complex interaction request is received from a user. According to paragraph 1,
The biometric information is collected including the user's eye position, head direction, arm position, hand position, voice waveform, iris information, facial color, eye corner position, eyebrow position, and mouth corner position. According to clause 8,
The eye position, head direction, arm position, and hand position are defined as at least indicated biometric information,
The eye blink frequency, voice waveform, iris information and facial color are defined as at least user identification biometric information,
A social robot wherein the eye blink frequency, the eye corner position, the eyebrow position, and the mouth corner position are defined by at least user state biometric information. delete According to clause 9,
A social robot configured to determine the frequency of the robot preemptive interaction further based on the eye corner position, the eyebrow position, and the mouth corner position. According to clause 9,
The interaction includes at least a non-verbal or complex interaction based on the user's non-verbal actions,
A social robot configured to use the indicated biometric information to determine whether to perform non-verbal or complex interaction. A step of learning user information by scanning the user's biometric information in a state in which there is no interaction from the user, comprising: scanning and learning biometric information including at least the frequency of the user's eye blinks;
When there is a request for interaction from a user, scanning biometric information of the requesting user and comparing it with learned biometric information;
assigning an intimacy value when the scanned biometric information matches the learned biometric information;
Classifying intimacy classes of multiple users according to the intimacy values of the multiple registered users; and
A method of controlling a social robot, comprising determining a frequency of robot preemptive interaction based on the eye blink frequency, wherein the frequency of robot preemptive interaction includes at least informing the current time. According to clause 13,
Classifying the type of interaction; and
A method of controlling a social robot further comprising comparing the type of interaction requested from a user with the intimacy class of the user. According to clause 14,
A method of controlling a social robot further comprising determining the priority of interaction for each user based on the intimacy class level. According to clause 13,
Classifying age classes based on the user's biometric information;
Classifying gender classes based on the user's biometric information; and
A method of controlling a social robot further comprising determining whether to perform an interaction request according to the level of the classified age class or the level of the gender class. According to clause 13,
Interactions requested from the user are classified as voice interactions that can be performed based on the user's voice recognition, non-verbal interactions that can be performed based on the user's non-verbal actions, and complex interactions that require non-verbal actions along with the user's voice,
When a voice-based interaction request is received from a user,
Determining whether the request is classified as a voice interaction or a complex interaction; and
A method of controlling a social robot further comprising scanning whether a user's non-verbal behavior is present during a predetermined interval when the request is classified as a complex interaction. According to clause 17,
The non-verbal action is a method of controlling a social robot including at least one of eye position, head direction, arm position, and hand position. delete A social robot according to claim 1; and
A user-robot interaction system that includes one or more Internet of Things devices. According to clause 20,
The biometric information collected by the social robot further includes instructional biometric information including eye position, head direction, arm position, and hand position,
A user-robot interaction system comprising operating the Internet of Things device when indicative biometric information related to the Internet of Things device is received from a user. An apparatus comprising a memory for storing one or more instructions, and a processor for performing the method of claim 17 by executing the instructions stored in the memory.