KR102581000B1 - Method and ststem for providing assistant service - Google Patents

Abstract

The present invention relates to a method and system for providing assistant services. The method of providing an assistant service according to the present invention includes the steps of sensing a facial image of a target user using a camera, sensing the voice of the target user using a microphone, and receiving the facial image corresponding to the facial image from a user DB. Extracting user information of the target user, confirming the target user's authority for the target function corresponding to the voice based on the user information, and performing control related to the target function based on the confirmation result. It may include steps.

Description

Assistant service provision method and system {METHOD AND STSTEM FOR PROVIDING ASSISTANT SERVICE}

The present invention relates to a method and system for providing assistant services.

As technology develops, the utilization of electronic devices is increasing, and recently, they have been used as a means of providing assistant services to assist users.

The assistant service may help the user complete various actual or virtual tasks by outputting useful information to the user, responding to functions requested by the user, or performing related tasks.

Electronic devices that provide such assistant services are developing into a format that provides customized services to users or provides functions that users actually need by understanding their intentions.

For example, in Republic of Korea Patent Publication No. 2019-0062521 (Personalization of Virtual Assistant), the user's personal record is automatically stored, allowing faster access to the user's personal information, accurately identifying the user's needs, and responding quickly. We provide a system that can be used.

Meanwhile, the key to these assistant services is not only to accurately understand the user's needs, but also to provide appropriate functions depending on the user providing the assistant service. Research in this field is actively being conducted. .

The present invention is intended to provide a method and system for providing an assistant service that can provide an assistant service more intuitively by understanding the user's intention.

Furthermore, the present invention is intended to provide an assistant service providing method and system that can provide an assistant service without contact input to an electronic device.

Furthermore, the present invention is intended to provide a method and system for providing an assistant service that can provide functions intended by the user, taking into account the user's authority and situation.

In order to solve the problems described above, the method of providing an assistant service according to the present invention includes the steps of sensing the face image of a target user using a camera, sensing the voice of the target user using a microphone, and user DB. extracting user information of the target user corresponding to the face image from the user information, confirming the target user's authority for the target function corresponding to the voice based on the user information, and based on the confirmation result, It may include performing control related to the target function.

Furthermore, the assistant service providing system according to the present invention includes a camera that senses the face image of the target user, a microphone that senses the target user's voice, and a user DB to extract user information of the target user corresponding to the face image, , based on the user information, includes a control unit that verifies the target user's authority for the target function corresponding to the voice, and the control unit can perform control related to the target function based on the confirmation result. there is.

Furthermore, a program that is executed by one or more processes in an electronic device according to the present invention and can be stored in a computer-readable medium includes the steps of: sensing the face image of a target user using a camera; using a microphone; Sensing the target user's voice, extracting user information of the target user corresponding to the face image from a user DB, based on the user information, selecting the target user for the target function corresponding to the voice. It may include instructions for performing a step of checking authority and a step of performing control related to the target function based on the confirmation result.

As seen above, the assistant service provision method and system according to the present invention can utilize facial recognition to provide customized functions that can be provided to recognized users. Furthermore, the present invention can provide functions according to the voice uttered by the recognized user by performing not only face recognition but also voice recognition along with face recognition. Accordingly, the user can receive the desired function by simply speaking near the electronic device that provides the assistant service.

Furthermore, the assistant service provision method and system according to the present invention senses the user subject to authentication using a camera and confirms the sensed user's authority, thereby assisting in the provision of functions that exist within the user's authority. You can. In this case, in the present invention, when the function requested by the user does not exist within the user's authority, the user's convenience can be improved by recommending the provision of a similar function that exists within the user's authority.

1 is a conceptual diagram for explaining an assistant service system according to the present invention.
Figure 2 is a flowchart for explaining the assistant service method according to the present invention.
FIGS. 3, 4A, 4B, 5A, 5B, 6, 7, and 8 are conceptual diagrams for explaining the assistant service method and system according to the present invention.
Figures 9, 10, and 11 are conceptual diagrams to explain examples of use of the assistant service according to the present invention.

Hereinafter, embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings. However, identical or similar components will be assigned the same reference numbers regardless of drawing symbols, and duplicate descriptions thereof will be omitted. The suffixes “module” and “part” for components used in the following description are given or used interchangeably only for the ease of preparing the specification, and do not have distinct meanings or roles in themselves. Additionally, in describing the embodiments disclosed in this specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed descriptions will be omitted. In addition, the attached drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed in this specification is not limited by the attached drawings, and all changes included in the spirit and technical scope of the present invention are not limited. , should be understood to include equivalents or substitutes.

Terms containing ordinal numbers, such as first, second, etc., may be used to describe various components, but the components are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is said to be "connected" or "connected" to another component, it is understood that it may be directly connected to or connected to the other component, but that other components may exist in between. It should be. On the other hand, when it is mentioned that a component is “directly connected” or “directly connected” to another component, it should be understood that there are no other components in between.

Singular expressions include plural expressions unless the context clearly dictates otherwise.

In this application, terms such as “comprise” or “have” are intended to designate the presence of features, numbers, steps, operations, components, parts, or combinations thereof described in the specification, but are not intended to indicate the presence of one or more other features. It should be understood that this does not exclude in advance the possibility of the existence or addition of elements, numbers, steps, operations, components, parts, or combinations thereof.

The present invention relates to a method and system for providing an assistant service that can provide convenience functions to users. It proposes a method that can assist a user's intended function just by being located near an electronic device. do.

Hereinafter, the present invention will be described in detail with the accompanying drawings. Figure 1 is a conceptual diagram for explaining an assistant service system according to the present invention, and Figure 2 is a flow chart for explaining an assistant service method according to the present invention. Furthermore, FIGS. 3, 4A, 4B, 5A, 5B, 6, 7, and 8 are conceptual diagrams for explaining the assistant service method and system according to the present invention. And, Figures 9, 10, and 11 are conceptual diagrams for explaining examples of use of the assistant service according to the present invention.

As shown in FIG. 1, the assistant service system 100 according to the present invention may be configured to include at least one of a communication unit 110, a storage unit 120, and a control unit 130. At this time, the assistant service system 100 according to the present invention is not limited to the above-described components, and may further include components that perform the same or similar roles as the functions described in the present specification.

The assistant service system 100 according to the present invention can operate in conjunction with the electronic device 10. Here, the electronic device 10 is at least one of a smart phone, a mobile phone, a tablet PC, a kiosk, a computer, a laptop, a digital broadcasting terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). may include.

Meanwhile, the assistant service system 100 according to the present invention may be implemented as an application or software. At this time, an application or software according to the assistant service system 100 may be installed on the electronic device 10 and perform a process according to the assistant service provision method described below.

For example, the communication unit 110, storage unit 120, and control unit 130 of the assistant service system 100 according to the present invention utilizes the hardware configuration of the electronic device 10 to provide the assistant service according to the present invention. A series of functions related to can be performed. For example, the communication unit 110 configures the communication module (e.g., mobile communication module, short-range communication module, wireless Internet module, location information module, broadcast reception module, etc.) provided in the communication unit of the electronic device 10. Using this, data can be transmitted and received with at least one external server (for example, the database (DB) server 140 shown in FIG. 1 or a cloud server, etc.).

Furthermore, the storage unit 120 may be configured to store data related to the assistant service according to the present invention using a memory provided in (or inserted into) the electronic device 10 or a cloud server.

The control unit 130 may utilize at least one of the CPU and the cloud server provided in the electronic device 10 to perform control of each component and overall control related to the present invention. Furthermore, the control unit 130 may have control authority over at least one component of the electronic device 10 in order to perform the present invention. Furthermore, the control unit 130 can utilize information received or sensed from components of the electronic device 10 for the present invention. For example, the control unit 130 may perform the assistant service according to the present invention based on the image received from the camera 11 of the electronic device 10 and the voice received through the microphone 13.

Meanwhile, the control unit 130 can provide an assistance service according to the present invention based on learning and reasoning based on artificial intelligence, machine learning, or deep learning.

Furthermore, the control unit 130 may output information related to the present invention using the display unit 12 of the electronic device 10. At this time, the display unit 12 may be implemented as a touch screen capable of receiving user input.

In addition, the control unit 130 can perform the assistant service according to the present invention using other components (eg, proximity sensor, infrared sensor, input unit, etc.) provided in the electronic device 10.

Meanwhile, according to FIG. 1, the components of the assistant system 100 include a communication unit 110, a storage unit 120, and a control unit 130. However, the assistant system 100 according to the present invention is an electronic device. Of course, at least one of the components (eg, camera 11, display unit 12, microphone 13, etc.) may be included as a component.

Meanwhile, the assistant service system 100 according to the present invention may exist not only in the form of a program or application, but also in various forms capable of performing the configuration and functions described below.

Accordingly, hereinafter, the assistant service system 100 according to the present invention will be described without placing strict limitations thereon.

As shown in FIG. 1, the assistant service system 100 according to the present invention can be configured to sense a user using the camera 11 of the electronic device 10 and perform an assistant service for the sensed user. there is.

For convenience of explanation, in the present invention, the user to whom the assistant service is provided is referred to as “target user (1000).”

In the present invention, in order to perform a target function for the target user 1000, user authentication is performed on the target user 1000, thereby providing the target function within the preset authority of the target user 1000.

In the present invention, “user authentication” may mean user authentication for the target user (1000).

In the present invention, a series of controls related to the “target function” can be performed using “user authentication.”

Here, the type of “targeted function” can be very diverse, for example, a reservation function for a venue (e.g., conference room reservation, restaurant reservation, etc.), payment function, attendance confirmation function, visit confirmation at the hospital. , ordering functions (e.g., ordering drinks in a cafe, etc.), electronic device control functions (e.g., electronic devices equipped in a conference room (e.g., TV, speakers, lights, etc.), vehicle search functions in a parking lot, etc. As such, it may refer to various functions that can be provided for the convenience of the user.

Meanwhile, the electronic device control function may include a function for controlling electronic devices installed in a specific space, such as a conference room, cafe, etc. Control of such electronic devices can use Internet of Things (IoT)-based control. Electronic device control functions may include air conditioner control, projector control, etc. in addition to the control of the TV, speakers, and lighting discussed above.

This electronic device control function can be activated or deactivated through a voice command input through the microphone 13 provided in the electronic device 10.

When a voice command such as “Turn off the lights,” “Turn on the lights,” or “Turn on the air conditioner,” is received, the control unit 130 generates a control command for the electronic device corresponding to the received voice command and sends the corresponding electronic device to the corresponding electronic device. By transmitting control commands, control of the electronic device can be performed.

In the present invention, the control unit 130 acquires a face image of the target user 1000 from the image received through the camera 11, and uses user information corresponding to the face image stored in the user DB to target the target user 1000. An assistant service for the user 1000 can be performed.

At this time, the target function subject to user authentication may be specified based on various criteria.

As an example, the control unit 130 may specify a target function depending on the location where the electronic device 10 is installed. For example, as shown in FIG. 1, when the electronic device 10 is placed around a conference room, the control unit 130 may specify a location reservation (e.g., conference room reservation) function as the target function. . And, as shown in FIG. 10, when the electronic device 10 is placed around a parking lot, the control unit 130 may specify, as a target function, a function for searching the location where the target user's vehicle is parked. Meanwhile, which function to specify as the target function may be determined in advance based on the location where the electronic device 10 is placed.

As another example, the control unit 130 may specify a target function based on the voice received through the microphone 13. The control unit 130 senses the face image of the target user 1000 located around the electronic device 10 and receives voice information of the target user 1000 through the microphone 13 of the target user 1000. You can.

The control unit 130 may analyze voice (or voice information) information sensed (or received) through the microphone 13 based on a preset Natural Language Processing algorithm. And the control unit 130 can specify the target function from the received voice based on analysis. The control unit 130 may specify the target function based on keywords or commands included in the voice.

As another example, the control unit 130 may specify a target function based on a user input to the electronic device 10. At this time, the user input may be a user input input through the display unit 12 equipped with a touch screen or another user input means. The control unit 130 may specify a target function based on such user input.

As another example, the control unit 130 may specify an appropriate function as the target function based on the target user's schedule included in the target user's user information.

In addition, the target function can be specified in various ways, and in the present invention, an assistant service can be provided so that the target user can smoothly perform or receive the target function.

Meanwhile, the control unit 130 uses the face image of the target user 1000 and user information extracted from the user DB to check the authority of the target user 1000 for the target function, and Assistant services for target functions can be performed within the authority of.

Additionally, the control unit 130 may utilize user information of the target user 100 to perform the target function. For example, when the target function is an attendance check function, the control unit 130 performs either an attendance check approval process or an attendance check rejection process for the target user 1000, based on the user information of the target user 1000. can do.

As another example, when the target function is a payment function, the control unit 130 may use the user information of the target user 1000 to perform either payment approval processing or payment rejection processing for the amount subject to payment. .

Meanwhile, the user information of the target user 1000 is various information related to the target user 1000, and the control unit 130 extracts the face image of the target user 1000 from the image received through the camera 11. And, user information corresponding to (or matched with) the extracted face image can be extracted from the user DB.

Here, the user DB may also be called “user-related information,” “user personal information,” or “user identification information.”

The user DB may be included in at least one of the storage unit 120, the external database 140, and the cloud server.

The control unit 130 may search for a person corresponding to the face image of the target user 1000 from the user DB and extract user information matched to the searched person.

In the user DB, user information for each person may be stored on an individual person (or person) basis. That is, in the user DB, user information corresponding to each different person can be matched and stored.

Here, user information includes the name, date of birth, address, phone number, email address, account information, employee number, ID, face image, and biometric information (e.g., fingerprint information, iris information, etc.) of each person. ), authority information for specific functions (or specific events), reference facial image, identification information of electronic devices, schedule (schedule, schedule), work department, position, job title, position, payment information (e.g., account number, It may include at least one piece of information related to credit card information, etc.).

Here, the reference face image is a face image for a specific person and may be one or more. In the present invention, “reference face image” may also be referred to as “reference data.”

The control unit 130 compares the face image of the target user acquired through the camera 11 with the reference face image stored in the user DB, and searches for a person who matches the reference face image corresponding to the face image of the target user. can do. Additionally, the control unit 130 may specify the searched person as the target user and extract user information matched to the searched person from the user DB.

And, the control unit 130 can use the extracted user information to perform an assistant service for the target function.

As shown in FIG. 1, the electronic device 10 can be placed in any location, and the target user 1000 can perform an assistant service just by being near the electronic device 10.

Below, we will look at the assistant service providing method according to the present invention in more detail along with a flowchart.

First, as shown in FIG. 2, according to the assistant service providing method according to the present invention, a process of sensing the face image of the target user can be performed using the camera 11 (S210).

That is, the control unit 130 can use the camera 11 of the electronic device 10 (see FIG. 1) to detect the target user's face image from the image received from the camera 11.

More specifically, as shown in Figure 3 (a) and Figure 4a, the control unit 130 can sense the face of the target user 1000 located around the electronic device 10 through the camera 11. there is.

The control unit 130 acquires the face image 410 of the target user 1000 from the image received through the camera 11, and uses the user information stored in the user DB corresponding to the face image to target the target user 1000. An assistant service for the user 1000 can be performed.

When the face image of the target user 1000 is detected in the image received through the camera 11, the control unit 130 may proceed with a series of processes for the assistant service.

The timing at which the face image of the target user 1000 is sensed or detected through the camera 11 may vary greatly.

As an example, when a user input is applied to the electronic device 10, the control unit 130 may control the camera 11 so that the face of the target user 1000 is captured. Here, the user input may be input in various ways, such as input through physical contact with the electronic device 10 or input using voice.

As another example, as shown in FIG. 3, when a person is detected around the electronic device 10, the control unit 130 receives an image of the target user 1000 through the camera 10. The camera 10 can be controlled as much as possible. At this time, when the target user 1000 is located within a preset range (or distance) based on the electronic device 10, the control unit 130 controls the camera 10 to detect the face image of the target user 1000. can do. The control unit 130 uses at least one sensor (for example, a proximity sensor, an infrared sensor, or a lidar sensor) provided in the electronic device 10 to operate around the electronic device 10 or to the electronic device 10. ), it is possible to sense (or sense) that the target user is located within a preset range.

Meanwhile, the control unit 130 operates the display unit 12 so that the display unit 12 of the electronic device 10 is deactivated when the target user 1000 is not located within a preset range of the electronic device 10. can be controlled. The display unit 12 in an inactive state may be in at least one of a power-off state, a standby state, and a power saving state.

Furthermore, the control unit 130 controls the display unit 12 so that the display unit 12 of the electronic device 10 is activated when the target user 1000 is located within a preset range of the electronic device 10. can do. At this time, guide information on how to specify the target function (for example, “Please tell me the function you want”) may be displayed on the display unit 12.

As another example, the control unit 130 may determine whether the image received through the camera 11 at preset intervals or in real time includes the face image of the target user 1000. And, as a result of the determination, if the image received through the camera 11 includes the face image of the target user 1000, a series of processes for an assistant service related to the target function can be performed.

As another example, the control unit 130 may activate the camera 11 when sound (or voice) is sensed through the microphone 13 provided in the electronic device 10. That is, when a person's sound is sensed through the microphone 13, the control unit 130 may determine that a person is located around the electronic device 10 and activate the camera 11.

As another example, the control unit 130 may activate the camera 11 based on the function icon 450 provided in the electronic device 10 being selected, as shown in FIG. 4B. These function icons 450 may be matched with functions for activating the assistance service according to the present invention. The function icon 450 may be output on the display unit 12 provided in the electronic device 10. The control unit 130 can activate the assistance service according to the present invention when the function icon 450 is selected by a user or the like by touching. In this case, the camera 11 may be activated based on activation of the assistance service.

Meanwhile, the control unit 130 may detect the face image of the target user 1000 from the image received from the camera 11 using at least one face recognition algorithm. The type of face recognition algorithm used in the present invention is not limited, and algorithms that operate based on artificial intelligence or deep learning can also be used.

Meanwhile, even when the face image of the target user 100 is detected through the camera 11, the control unit 130 determines that the detected face image meets preset conditions (or preset authentication conditions) for performing the assistant service. If is not satisfied, guide information for acquiring a face image that satisfies the preset conditions may be provided.

More specifically, the control unit 130 may determine whether the facial image of the target user 1000 sensed by the camera 11 satisfies the preset condition. Here, the preset conditions include the size of the face image included in the image, the angle of the face corresponding to the face image, the position of the face image in the image, and the appearance feature elements of the face in the image (e.g., eyes, nose, etc.) , mouth) may be related to at least one of the following:

The control unit 130 may output at least one of information about the posture and location of the target user 1000 as guide information so that a face image that satisfies preset conditions can be obtained. Guide information may be related to at least one of the relative position and relative distance between the camera 11 and the target user 1000.

For example, the control unit 130 provides i) location information of the target user 1000 based on the electronic device (e.g., please come closer, please go further back, please adjust the distance to the screen), ii) Posture information of the target user 1000 (e.g., please look at the screen); iii) If the face of the target user 1000 is obscured by an object (e.g., a mask), information recommending removing it (e.g., For example, guide information such as (Please take off your mask) can be provided.

Furthermore, in the method of providing an assistant service according to the present invention, a process of sensing the target user's voice may be performed using a microphone (S220).

In the present invention, the process of sensing the face image of the target user 1000 through the camera 11 (S210) and the process of sensing the target user's voice using the microphone 13 (S220) are performed using the electronic device 10 ) It is also possible to do this simultaneously. In other words, the time when the control unit 130 receives image information through the camera 11 and the time when audio information is received through the microphone 13 on the electronic device 10 may be the same. In this case, the target user 1000 is located near the electronic device 10 and makes a speech.

Meanwhile, the sequence of the process of sensing the face image of the target user 1000 through the camera 11 (S210) and the process of sensing the target user's voice using the microphone 13 (S220) can be freely adjusted depending on the situation. It can be transformed. That is, after the process of sensing the face image of the target user 1000 through the camera 11 (S210), the process of sensing the voice of the target user using the microphone 13 (S220), or similar Conversely, after the process of sensing the target user's voice using the microphone 13 (S220), the process of sensing the face image of the target user 1000 using the camera 11 (S210) may proceed. .

Meanwhile, the process of sensing the target user's voice using the microphone 13 can be understood as a process of receiving sound information through the microphone.

The control unit 130 may analyze voice information received through the microphone 13 based on a natural language processing algorithm and determine a target function based on the analysis result.

For example, as shown in FIG. 3, the target user 1000 may utter information about the target function (eg, “Please reserve a conference room”) through voice. Based on the reception of this voice, the control unit 130 can specify the function corresponding to the voice as the target function. The control unit 130 can specify the target function by comparing the function list corresponding to each of the various keywords with the information included in the voice. This function list may exist and be stored in the storage unit 120.

Meanwhile, when the target function is specified, guidance information indicating that a specific function has been determined as the function corresponding to the voice may be output on the display unit 12. Additionally, the control unit 130 may output guidance information through an audio output unit (or speaker).

Next, in the present invention, a process of extracting user information corresponding to the detected face image from the user DB may be performed (S230).

More specifically, the control unit 130 extracts the face image of the target user 1000 from the image received through the camera 11, and selects a face image corresponding to (or matched to) the extracted face image from the user DB. User information can be extracted.

As seen above, the user DB may also be called “user-related information,” “user’s personal information,” or “user’s identification information.”

The user DB may be included in at least one of the storage unit 120, the external database 140, and the cloud server.

The control unit 130 may search for a person corresponding to the face image of the target user 1000 from the user DB and extract user information matched to the searched person.

In the user DB, user information for each person may be stored on an individual person (or person) basis. That is, in the user DB, user information corresponding to each different person can be matched and stored.

Here, user information includes the name, date of birth, address, phone number, email address, account information, employee number, ID, face image, and biometric information (e.g., fingerprint information, iris information, etc.) of each person. ), authority information for specific functions (or specific events), reference facial image, identification information of electronic devices, schedule (schedule, schedule), work department, position, job title, position, payment information (e.g., account number, It may include at least one piece of information related to credit card information, etc.).

Here, the reference face image is a face image for a specific person and may be one or more. In the present invention, “reference face image” may also be referred to as “reference data.”

The control unit 130 compares the face image of the target user acquired through the camera 11 with the reference face image stored in the user DB, and searches for a person who matches the reference face image corresponding to the face image of the target user. can do. Additionally, the control unit 130 may specify the searched person as the target user and extract user information matched to the searched person from the user DB.

The control unit 130 may extract from the user DB a specific reference face image having a similarity level that satisfies a preset condition with the detected face image of the target user 1000. In addition, a person matching the extracted specific reference face image can be specified as the target user.

While the search for the target user 1000 is in progress, the control unit 130 controls the display unit 12 so that guidance information about the current progress (for example, “I am looking for you”) is output to the electronic device 10. ) and sound output units (e.g. speakers, etc.) can be controlled.

As shown in FIG. 4A, an image captured through the camera 11 may be output to the display unit 12. At this time, the output image may include the face image 410 of the target user 1000.

As seen above, when a person corresponding to the target user 1000 is specified and user information about the specified person (for example, Lee * Lee) is output, the control unit 130 controls at least some of the user information. can be provided to the display unit 12.

Therefore, as shown in FIG. 4A, at least part of the user information (for example, name information (i*ri)) corresponding to the person corresponding to the target user 1000 may be output on the display unit 12. Therefore, the target user 1000 can refer to the information output on the display unit 12 to confirm whether the user has accurately searched for himself or herself on the electronic device 10.

As seen above, when user information about the target user 1000 is extracted, in the present invention, a process of confirming the authority of the target user 1000 for the target function can be performed using the extracted user information. (S240).

As discussed above, the target function can be specified based on the voice input through the microphone.

Additionally, the target function can be specified by various methods other than voice input through a microphone. If the target function is not specified based on voice, it is also possible to omit step S220 in the assistant service providing method according to the present invention.

Looking at an example in which the target function is specified by another method, as an example, the control unit 130 may specify the target function depending on the location where the electronic device 10 is installed. For example, as shown in FIG. 3, when the electronic device 10 is placed around a conference room, the control unit 130 may specify a location reservation (e.g., conference room reservation) function as the target function. . In this case, the control unit 130 checks the authority of the target user 1000 based on the target user 1000 being sensed by the camera 11, even if no separate user input is received from the target user 1000. The process can be performed.

And, as shown in FIG. 10, when the electronic device 10 is placed around a parking lot, the control unit 130 may specify, as a target function, a function for searching the location where the target user's vehicle is parked. Meanwhile, which function to specify as the target function may be determined in advance based on the location where the electronic device 10 is placed.

As another example, the control unit 130 may specify a target function based on a user input to the electronic device 10. At this time, the user input may be a user input input through the display unit 12 equipped with a touch screen or another user input means. The control unit 130 may specify a target function based on such user input.

As another example, the control unit 130 may specify an appropriate function as the target function based on the target user's schedule included in the target user's user information.

Meanwhile, in order to specify the target function, the control unit 130 may output a question to the target user 1000 and specify the target function based on the response to the output question.

Specification of the target function may require information detailed enough to confirm the authority of the target user. For example, if the target function is a reservation for a location, the control unit 130 may require specific reservation location (eg, conference room number, etc.) and reservation time information.

In order to specify the target function, the control unit 130 may ask the target user 1000 questions in various ways. For example, the control unit 130 may output a question to the target user 1000 through a speaker and receive an answer to the question by receiving a voice through the microphone 13. The control unit 130 may output questions and receive answers multiple times until information sufficient to specify (or determine) the target function is collected.

Additionally, the control unit 130 can collect information for specifying the target function by outputting a question to the display unit 12 and receiving an answer through the display unit 12 or the microphone 13.

Meanwhile, the control unit 130 may not output a question when the specification of the target function is completed, even if the question is not asked to the target user 1000. For example, the target user 1000 may initially provide sufficient information to the electronic device 10 to specify a specific function, such as “Please reserve conference room number 3 at 3 p.m. on Monday.”

In this way, when the target function is specified by the user's voice or another method, the control unit 130 can check the target user's authority for the target function.

Here, “target user authority” can be interpreted in various ways. For example, the target user's authority includes: i) whether the target user (1000) has the authority to receive the target function, ii) whether the target user (1000) has the authority to access the target function, iii) whether the target user ( iv) whether the target user 1000 has permission to enter a specific location, iv) whether the target user 1000 has permission to control a specific electronic device, etc., in relation to any function, device or location. This may be information about whether you have permission to access it.

Information about the rights (or rights information) of the target user 1000 may be included in the user information of the target user 1000.

For example, when the target function is a function to reserve a place as shown in FIG. 3, the control unit 130 allows the target user 1000 to reserve a specific place (e.g. For example, it can be determined whether the authority to enter or make a reservation exists for the reservation target location.

In this case, the target function can be understood as a reservation function for a specific time at a specific location.

Furthermore, in order to check information about the authority (or authority information) of the target user 1000, the control unit 130 may check information for confirmation of authority from the target user 1000. Here, the information for verifying authority may be information for authenticating the target user 1000, such as biometric information or password information of the target user 1000.

Meanwhile, as discussed above, information related to reservations, such as a specific location or reservation time, can be specified based on at least one of the voice received from the target user 1000 and the target user's schedule information.

Additionally, information related to reservations, such as a specific location or reservation time, may be specified based on at least one of the locations where the electronic device 10 is placed.

For example, the control unit 130 may determine a specific location based on the location where the electronic device 10 that senses the facial image and voice of the target user 1000 is placed. For example, the control unit 130 may specify the location (e.g., a conference room) closest to the location where the electronic device 10 is placed as a specific location. Additionally, for each electronic device 10, information about the location may be specified. Information (for example, conference room number) may be matched, and the control unit 130 may determine a specific location based on information about the location matched to the electronic device 10.

Meanwhile, as seen above, when the target user's authority for the target function is confirmed, in the present invention, a process of performing control related to the target function can be performed based on the confirmation result (S250).

More specifically, the control unit 130 may perform different controls depending on whether the target user 1000 has authority for a specific function.

The control unit 130 may perform or execute a specific function when the target user 1000 has authority for the specific function.

Furthermore, if the target user 1000 has authority for a specific function but cannot perform or execute the specific function, the control unit 130 may recommend or perform a function similar to the specific function.

That is, the control unit 130 may restrict performance of a specific function when the target user 1000 does not have authority for the specific function. In this case, the control unit 130 may recommend or perform a function similar to a specific function.

For example, when the specific function is a function to reserve a specific place, the control unit 130 determines whether the target user 1000 has reservation authority for the specific place, based on whether the reservation authority for the specific place exists. Other controls can be performed.

For example, if the target user 1000 has reservation authority for a specific place, the control unit 130 may make a reservation for the specific place, as shown in (b) of FIG. 3. In this case, the control unit 130 provides guidance information indicating that a specific function can be performed normally (for example, “I will reserve conference room number 4 right now!”) in at least one of visual and auditory information. , can be output on the electronic device 10.

As shown in FIG. 4A, information about reservation for a specific place (for example, situation information or guidance information, etc., 420) may be displayed on the display unit 12. The control unit 130 may provide a graphic user interface (GUI) 430 on the display unit 12 that allows the target user 1000 to change the reservation time or reservation location or confirm the reservation at any time. . This GUI 430 may also be named an icon or graphic object.

As another example, if the target user 1000 has reservation authority for a specific place, but cannot make a reservation for the specific place (for example, if the reservation for the specific place has already been completed), the control unit 130 case, etc.), as shown in (c) of FIG. 3, reservations can be made or recommended for places that can make reservations different from a specific place. The control unit 130 may determine whether a reservation is possible for a location (e.g., a conference room, etc.) in a space containing the target user, and may recommend other reservationable locations based on the determination result.

At this time, the control unit 130 is i) the closest place to a specific place, ii) the place closest to the current location of the target user 1000 excluding the specific place, iii) the work location of the target user 1000 excluding the specific place. Other reservable places based on various criteria, such as the place closest to the location, iv) a place that takes into account the distance of both other users (e.g., meeting attendees, etc.) and target users (1000), excluding a specific place. You can search and create the searched information as recommended information.

As another example, when the target user 1000 does not have reservation authority for a specific location, the control unit 130 makes a reservation for a location different from the specific location, as shown in (d) of FIG. 3. I can guide you. Guide to other places can be provided through the electronic device 10, and the control unit 130 can guide the target user 1000 through at least one of an auditory and visual method.

In this case, the control unit 130 provides guide information (e.g., “You do not have reservation authority for conference room 4”) informing the target user that he or she does not have reservation authority for a specific location. You can print it out along with “Can I reserve conference room number 7?” This information may be output as visual information 510 on the display unit 12, as shown in FIG. 5A. Furthermore, the display unit 12 may include GUIs 520 and 530 for terminating the performance of a specific function or providing other recommended locations. These GUIs 520 and 530 may also be named icons or graphic objects. When one of the GUIs 520 and 530 is selected, the control unit 130 may execute a function matched to the selected GUI. For example, when the GUI 520 corresponding to the end of performance of a specific function is selected by the user (or the target user 1000), the control unit 130 ends the place reservation process for the target user 1000. can do. In contrast, when the GUI 530 corresponding to the function of providing another recommended place to the user (or the target user 1000) is selected, the control unit 130 may output recommended place information. Furthermore, it goes without saying that recommended place information can be output even when the GUI 530 is not selected by the user.

As discussed above, the control unit 130 can search for places where the target user 1000 can make a reservation while having reservation authority, and generate the searched information as recommended information. The control unit 130 can search for recommended places based on various criteria and provide information about one or more recommended places to the target user 1000.

The control unit 130 is configured to: i) be the closest location to a specific location, ii) be the closest location to the current location of the target user (1000) excluding the specific location, iii) work location of the target user (1000) excluding the specific location, and Search for other reservable places based on various criteria, such as the nearest place, iv) a place that takes into account the distance of both other users (e.g., meeting attendees, etc.) and target users (1000), excluding a specific place. , the searched information can be created as recommended information.

For example, as shown in FIG. 5B, the place closest to the current location of the target user 1000 may be provided as recommended information 550 on the display unit 12. Likewise, as shown in FIG. 5B, on the display unit 12, the place closest to the work location of the target user 1000 may be provided as recommended information 550. In this case, the place provided as recommended information may be a place where the target user 1000 has the authority to make a reservation.

Meanwhile, as shown in Figure 6, a specific function is a reservation function for a location, and information about attendees (e.g., Kim * Dong, Yu * Ae) is received based on user input by the target user. In this case (see reference numeral 620 in FIG. 6), the control unit 130 selects a specific place (for example, A You can search for buildings (buildings 1, 3, and conference rooms 5) as recommended locations. And, as shown in FIG. 6, information 630 about the searched recommended place can be output on the display unit. In this case, the control unit 130 displays the recommended place location, the location related to the target user 1000 (e.g., the target user's current location or the target user's work location, etc.), and the location related to the attendee (e.g., the target user's current location, etc.) on the map image. For example, the current location of the attendee or the work location of the target user, etc.) can be displayed.

When information about the attendee is received, the control unit 130 extracts the attendee's user information from the user DB and searches for recommended place information based on the location information related to the attendee included in the extracted user information of the attendee ( or specific) can be done.

Meanwhile, in the examples above, we looked at cases where a specific function and location were specifically specified. However, the present invention is not limited to this, and information on places that can be reserved can be recommended based on authority information (eg, access rights to the place, entrance rights, etc.) that the target user 1000 has. Of course, the control unit 130 can search for a place where the target user 1000 has the authority to make a reservation and can make a reservation, based on the place recommendation method described above, and generate recommendation information for the searched place.

For example, the target user 1000 may input a voice such as “Please reserve a conference room” into the electronic device 10, and the control unit 130 may use at least one of the various methods described above based on the user's voice. Based on this, the target user 1000 can generate a place that can be reserved as recommended place information.

Meanwhile, in the present invention, in order to provide an assistant service that increases user convenience, when a reservation for a specific place is made, based on the user information of the target user 1000, reservation information for the specific place is provided to the target user. The ability to add to schedule information can be provided.

As shown in FIG. 7, when a schedule event related to the schedule of the target user 1000 occurs as a result of performing a specific function, the control unit 130 may add the schedule event to the user's schedule information.

The control unit 130 may extract account information in which the user's schedule information is stored from the user information, and additionally store a schedule corresponding to the schedule event in the account information.

In this way, when an assistant service related to adding a schedule is provided, the control unit 130 provides guidance information 710 indicating that an assistant service related to adding a schedule is provided in at least one of audible and visual ways. Can be printed.

The control unit 130 may provide a graphic user interface (GUI) 720 on the display unit 12 that allows the target user 1000 to approve or reject an assistant service related to schedule addition.

Furthermore, when a reservation for a specific place is made and information about the attendee related to the specific place is confirmed, the control unit 130 may extract the attendee's user information from the user DB. Additionally, the control unit 130 may provide an assistant service that shares reservation information for the specific location with the attendee using the attendee's user information. The control unit 130 may transmit reservation information to the attendee using contact information such as email account, phone number, etc. included in the attendee's user information. This reservation information may be transmitted in the form of e-mail, messages, etc.

Furthermore, the control unit 130 may extract account information in which the attendee's schedule information is stored from the attendee's user information and additionally store a schedule corresponding to the schedule event in the account information.

In this way, when an assistant service related to adding a schedule is provided, the control unit 130 may output guidance information indicating that an assistant service related to adding a schedule is provided in at least one of audible and visual ways. there is.

Furthermore, the control unit 130 displays a GUI (Graphic User Interface, 720) that allows the target user 1000 to approve or reject assistant services related to attendees (for example, sending reservation information or adding a schedule, etc.) to the display unit 12. ) can be provided on the table.

Meanwhile, the electronic device 10 applying the service provision method and system described above may be installed in various places. As previously shown in FIG. 3, the electronic device 10 is placed around a conference room, or as shown in (a) of FIG. 8, in a common living space (e.g., around an elevator), or at the workplace of users. It can be.

In addition, when a user (e.g., target user 1000) is recognized around the electronic device 10, the control unit 130 may provide an assistant service for the recognized target user 1000.

For example, as shown in (b) of FIG. 8, the control unit 130 is used to provide a specific function (for example, a conference room reservation function) to the target user 1000 through the electronic device 10. Guide information 810 can be output.

As another example, as shown in (c) of FIG. 8, when the target user 1000 is specified, the control unit 130 refers to the schedule information included in the user information of the target user 1000 and performs the target function. can be provided. Here, the target function may be a function that takes into account the characteristics of the location where the electronic device 10 is located. For example, when the place where the electronic device 10 is located is around an elevator, the control unit 130 selects the target user ( 1000) schedule information can be extracted, and guide information (e.g., conference room 4) is provided so that the target user 1000 can move to a location (e.g., conference room 4) corresponding to the extracted schedule information. Movement to this location (3rd floor) can be output through the electronic device 10.

Furthermore, as shown in (d) of FIG. 8, the control unit 130 refers to the schedule information included in the user information of the target user 1000 and reminds the target user 1000 of the schedule information. The information 830 can be output through the electronic device 10.

Meanwhile, the method and system for providing an assistant service according to the present invention can provide various functions in addition to the function for reserving a location discussed above.

As an example, when the electronic device 10 is placed in a space 900 such as a conference room, office, or home, the control unit 130 controls the space based on the user input input through the electronic device 10 in FIG. Electronic devices 901, 902, and 903 placed in 900 can be controlled. Even at this time, the control unit 130 recognizes the face of the target user 1000 to determine whether the target user 1000 has authority to control the electronic devices 901, 902, and 903, and uses the recognized face User information corresponding to the image can be extracted from the user DB. And, the control unit 130 checks the control authority information for the electronic devices 901, 902, and 903 from the extracted user information, and performs control of the electronic devices 901, 902, and 903 based on the confirmation result. can do.

On the other hand, control of the electronic devices 901, 902, 903, and 904, such as turning the power of the electronic devices 901, 902, and 903 on or off, or adjusting the volume, It may include various controls related to (901, 902, 903).

As another example, the electronic device 10 may be placed in a parking lot 1010, as shown in FIG. 10 . In this case, when the target user 1000 is recognized through the electronic device 10, the control unit 130 may perform control related to the vehicle 1020 within the authority of the target user 1000.

The control unit 130 recognizes the face of the target user 1000 through the electronic device 10 in order to extract information about the vehicle 1020 within the authority of the target user 1000, and generates information corresponding to the recognized face image. User information can be extracted from the user DB. Additionally, the control unit 130 may check information on the vehicle 1020 (e.g., a vehicle owned by the target user 1000, a vehicle registered for use, etc.) within the authority of the target user 1000 from the extracted user information. Additionally, the control unit 130 may perform controls related to the confirmed vehicle 1020 (for example, providing parking location information for the vehicle 1020, paying parking fees for the vehicle 1020, etc.).

Meanwhile, information about the number, parking location, parking time, etc. of the vehicle 1020 may be sensed based on a sensor provided in the parking lot 1010 when the vehicle 1020 enters the parking lot 1010. . In addition, if there is a user with user information corresponding to the vehicle number information among the information of the sensed vehicle 1020, the control unit 130 adds information related to parking of the vehicle (parking location, parking time, etc.) to the user's information. ) can be registered by matching. Therefore, when the target user 1000 requests a function related to the vehicle 1020 (for example, providing parking location information, parking fee payment, etc.) through the electronic device 10, information registered by matching the user information By referring to , appropriate information can be provided.

As another example, as shown in FIG. 11, when the electronic device 10 is placed in a cafe 1100, a restaurant, etc., the control unit 130 processes an order from the target user 1000 through the electronic device 10. You can receive information and make electronic payments.

In this case, the control unit 130 can recognize the target user 1000 and extract user information of the target user 1000 from the user DB. Additionally, electronic payment for the order of the target user 1000 may be performed using the extracted user information (e.g., payment method information (credit card information, account information, etc.)).

Meanwhile, the user information of the target user 1000 includes history information of the target user 1000, and the history information may include past order information of the target user 1000.

Based on this history information, when the target user 1000 is recognized, the control unit 130 provides a specific menu (for example, “cafe latte”) through the electronic device 10 based on past order information. Order recommendation information 1110 recommending an order may be output as shown in FIG. 11.

On the other hand, when a space such as the cafe 1100 is a space where a robot (e.g., a serving robot, etc.) is operated, the control unit 130 uses the robot to provide a menu ordered by the target user 1000 (e.g., The robot can be controlled so that drinks (drinks, food, etc.) are served to the target user (1000). In this case, the control unit 130 can sense where the target user 1000 is located in space and control the robot to move to the sensed location.

As discussed above, the assistant service provision method and system according to the present invention can provide various functions that can assist the user depending on the location of the electronic device to which the assistant service provision method and system is applied.

As seen above, the assistant service provision method and system according to the present invention can utilize facial recognition to provide customized functions that can be provided to recognized users. Furthermore, the present invention can provide functions according to the voice uttered by the recognized user by performing not only face recognition but also voice recognition along with face recognition. Accordingly, the user can receive the desired function by simply speaking near the electronic device that provides the assistant service.

Furthermore, the assistant service provision method and system according to the present invention senses the user subject to authentication using a camera and confirms the sensed user's authority, thereby assisting in the provision of functions that exist within the user's authority. You can. In this case, in the present invention, when the function requested by the user does not exist within the user's authority, the user's convenience can be improved by recommending the provision of a similar function that exists within the user's authority.

Meanwhile, computer-readable media includes all types of recording devices that store data that can be read by a computer system. Examples of computer-readable media include HDD (Hard Disk Drive), SSD (Solid State Disk), SDD (Silicon Disk Drive), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is.

Furthermore, the computer-readable medium may be a server or cloud storage that includes storage and can be accessed by electronic devices through communication. In this case, the computer can download the program according to the present invention from a server or cloud storage through wired or wireless communication.

Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, that is, a CPU (Central Processing Unit), and there is no particular limitation on its type.

Meanwhile, the above detailed description should not be construed as restrictive in all respects and should be considered illustrative. The scope of the present invention should be determined by reasonable interpretation of the appended claims, and all changes within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (13)

Sensing a face image of a target user using a camera;
Sensing the target user's voice using a microphone;
extracting user information of the target user corresponding to the face image from a user DB;
Based on the user information, confirming the target user's authority for the target function corresponding to the voice; and
Based on the verification result, performing control related to the target function,
Further comprising analyzing the voice and determining the target function based on the analysis result,
As a result of the analysis, if the target function is a reservation function for a specific location, in the step of checking the permission,
An assistant service providing method characterized by determining whether the target user has reservation authority for the specific location based on the user information. delete According to paragraph 1,
The method of providing an assistant service further comprising providing, on a display unit, at least a portion of user information of the target user and guidance information indicating that the target function has been determined as a function corresponding to the voice. delete According to paragraph 1,
The specific location above is,
A method of providing an assistant service, characterized in that the decision is made based on the location where the electronic device including the camera and the microphone is placed. According to paragraph 1,
In the step of performing control related to the target function,
An assistant service providing method, characterized in that performing different controls related to reservations for the specific place based on whether the target user has reservation authority for the specific place. According to clause 6,
In the step of performing control related to the target function,
If the target user has reservation authority for the specific location, make a reservation for the specific location,
A method of providing an assistant service, characterized in that when the target user does not have reservation authority for the specific location, guiding reservations for a location different from the specific location. In clause 7,
Locations different from the above specific location,
A method of providing an assistant service, characterized in that the location is closest to the specific location and the target user has reservation authority. In clause 7,
When a reservation for the specific place is made, providing an assistant service further comprising adding reservation information for the specific place to the target user's schedule information based on the user information of the target user. method. In clause 7,
When a reservation for the specific location is made and information about attendees related to the specific location is received,
An assistant service providing method comprising the step of extracting user information of the attendee from the user DB and sharing reservation information for the specific place with the attendee using the attendee's user information. According to paragraph 1,
As a result of the analysis, if the target function is a reservation function for a location and information about attendees is received based on user input by the target user,
An assistant service providing method further comprising searching for a specific place within the reservation authority of the target user as a recommended place based on the locations associated with the attendees and the target user, respectively. A camera that senses the face image of the target user;
A microphone that senses the target user's voice;
A control unit extracting user information of the target user corresponding to the face image from a user DB and confirming the target user's authority for the target function corresponding to the voice based on the user information,
The control unit,
Based on the confirmation result, perform control related to the target function, analyze the voice, and determine the target function based on the analysis result. If the analysis result is that the target function is a reservation function for a specific location,
To check the above permissions,
An assistant service system characterized in that it determines whether the target user has reservation authority for the specific location based on the user information. A program that is executed by one or more processes in an electronic device and can be stored in a computer-readable medium,
The above program is,
Sensing a face image of a target user using a camera;
Sensing the target user's voice using a microphone;
extracting user information of the target user corresponding to the face image from a user DB;
Based on the user information, confirming the target user's authority for the target function corresponding to the voice; and
Based on the confirmation result, it includes instructions for performing steps to perform control related to the target function,
Further comprising instructions for analyzing the voice and determining the target function based on the analysis result,
As a result of the analysis, if the target function is a reservation function for a specific location, in the step of checking the permission,
A program storable on a computer-readable medium, characterized in that it determines whether the target user has reservation authority for the specific place based on the user information.

Images