KR20220109764A - Method and ststem for providing assistant service - Google Patents

Abstract

The present invention relates to a method and system for providing an assistant service. The method for providing the assistant service according to the present invention may comprise: a step of sensing a face image of a target user using a camera; a step of sensing a speech of the target user using a microphone; a step of extracting user information of the target user corresponding to the face image from a user DB; a step of confirming an authority of the target user for a target function corresponding to the speech based on the user information; and a step of performing control related to the target function based on the confirmation result. Therefore, the present invention is capable of improving a user convenience.

Description

ASSISTANT SERVICE PROVIDED METHOD AND STSTEM FOR PROVIDING ASSISTANT SERVICE

The present invention relates to a method and system for providing an assistant service.

As technology develops, the utilization of electronic devices is increasing, and recently, it is utilized as a means of providing an assistant service to assist a user.

The assistant service may help the user to complete various real or virtual tasks by outputting useful information to the user, responding to a function requested by the user, or performing a related task.

Electronic devices that provide such an assistant service are developing into a form of providing a customized service to a user or providing a function that a user actually needs by identifying a user's intention.

For example, in Korean Patent Application Laid-Open No. 2019-0062521 (Personalization of Virtual Assistant), by automatically storing user personal records, you can access personal information about users more quickly, accurately understand user needs, and respond quickly. We provide a system that can

On the other hand, the key to such an assistant service is not only to accurately understand the user's needs, but also to provide an appropriate function according to the user who provides the assistant service, and research in this field is being actively conducted. .

An object of the present invention is to provide an assistant service providing method and system capable of providing an assistant service more intuitively by recognizing a user's intention.

Furthermore, it is another object of the present invention to provide an assistant service providing method and system capable of providing an assistant service without a touch input to an electronic device.

Furthermore, it is another object of the present invention to provide an assistant service providing method and system capable of providing a function intended by a user in consideration of the user's authority and situation.

In order to solve the above problems, the assistant service providing method according to the present invention includes the steps of sensing a face image of a target user using a camera, sensing the voice of the target user using a microphone, and a user DB extracting user information of the target user corresponding to the face image from It may include performing control related to the target function.

Furthermore, the assistant service providing system according to the present invention extracts user information of the target user corresponding to the face image from a camera for sensing the face image of the target user, a microphone for sensing the voice of the target user, and a user DB, , based on the user information, comprising a control unit for confirming the authority of the target user for the target function corresponding to the voice, the control unit, based on the verification result, can perform control related to the target function have.

Furthermore, the program, which is executed by one or more processes in the electronic device according to the present invention, and can be stored in a computer-readable medium, uses a camera to sense a face image of a target user, using a microphone, Sensing the voice of the target user, extracting user information of the target user corresponding to the face image from the user DB, based on the user information, the target user for the target function corresponding to the voice It may include instructions for performing the step of checking the authority and performing the control related to the target function based on the check result.

As described above, the method and system for providing an assistant service according to the present invention can provide a customized function that can be provided to a recognized user by utilizing face recognition. Furthermore, according to the present invention, a function according to the recognized user's voice can be provided by performing voice recognition together with face recognition as well as face recognition. Accordingly, the user can be provided with the performance of a desired function simply by uttering a speech around the electronic device that provides the assistant service.

Furthermore, the method and system for providing an assistant service according to the present invention sense a user to be authenticated using a camera and check the sensed user's authority, thereby assisting in providing a function existing within the user's authority. can In this case, in the present invention, when the function requested by the user does not exist within the user's authority, the user's convenience can be improved by recommending provision of a similar function existing within the user's authority.

1 is a conceptual diagram illustrating an assistant service system according to the present invention.
2 is a flowchart illustrating an assistant service method according to the present invention.
3, 4A, 4B, 5A, 5B, 6, 7, and 8 are conceptual diagrams for explaining an assistant service method and system according to the present invention.
9, 10, and 11 are conceptual diagrams for explaining an example of application of the assistant service according to the present invention.

Hereinafter, the embodiments disclosed in the present specification will be described in detail with reference to the accompanying drawings, but the same or similar components will be given the same reference numerals regardless of reference numerals, and redundant descriptions thereof will be omitted. The suffixes "module" and "part" for components used in the following description are given or mixed in consideration of only the ease of writing the specification, and do not have distinct meanings or roles by themselves. In addition, in describing the embodiments disclosed in the present specification, if it is determined that detailed descriptions of related known technologies may obscure the gist of the embodiments disclosed in this specification, the detailed description thereof will be omitted. In addition, the accompanying drawings are only for easy understanding of the embodiments disclosed in this specification, and the technical idea disclosed herein is not limited by the accompanying drawings, and all changes included in the spirit and scope of the present invention , should be understood to include equivalents or substitutes.

Terms including ordinal numbers such as first, second, etc. may be used to describe various elements, but the elements are not limited by the terms. The above terms are used only for the purpose of distinguishing one component from another.

When a component is referred to as being “connected” or “connected” to another component, it may be directly connected or connected to the other component, but it is understood that other components may exist in between. it should be On the other hand, when it is said that a certain element is "directly connected" or "directly connected" to another element, it should be understood that the other element does not exist in the middle.

The singular expression includes the plural expression unless the context clearly dictates otherwise.

In the present application, terms such as “comprises” or “have” are intended to designate that a feature, number, step, operation, component, part, or combination thereof described in the specification exists, but one or more other features It is to be understood that this does not preclude the possibility of the presence or addition of numbers, steps, operations, components, parts, or combinations thereof.

The present invention relates to a method and system for providing an assistant service capable of providing a convenient function to a user, and proposes a method in which a user can assist a user's intended function simply by being located around an electronic device. do.

Hereinafter, the present invention will be described in detail with reference to the accompanying drawings. 1 is a conceptual diagram illustrating an assistant service system according to the present invention, and FIG. 2 is a flowchart illustrating an assistant service method according to the present invention. Furthermore, FIGS. 3, 4A, 4B, 5A, 5B, 6, 7, and 8 are conceptual diagrams for explaining the assistant service method and system according to the present invention. 9, 10, and 11 are conceptual diagrams for explaining an example of application of the assistant service according to the present invention.

1 , the assistant service system 100 according to the present invention may be configured to include at least one of a communication unit 110 , a storage unit 120 , and a control unit 130 . In this case, the assistant service system 100 according to the present invention is not limited to the above-described components, and may further include components that perform the same or similar roles as those described in the present specification.

The assistant service system 100 according to the present invention may operate in conjunction with the electronic device 10 . Here, the electronic device 10 includes at least one of a smart phone, a mobile phone, a tablet PC, a kiosk, a computer, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), and a portable multimedia player (PMP). may include.

Meanwhile, the assistant service system 100 according to the present invention may be implemented as an application or software. In this case, the application or software according to the assistant service system 100 may be installed in the electronic device 10 and perform a process according to the assistant service providing method described below.

For example, the communication unit 110 , the storage unit 120 , and the control unit 130 of the assistant service system 100 according to the present invention utilize the hardware configuration of the electronic device 10 to provide an assistant service according to the present invention. A series of related functions can be performed. For example, the communication unit 110 configures a communication module (eg, a mobile communication module, a short-range communication module, a wireless Internet module, a location information module, a broadcast reception module, etc.) provided in the communication unit of the electronic device 10 . By using it, data may be transmitted/received to and from at least one external server (eg, the database (DB) server 140 or cloud server shown in FIG. 1 ).

Furthermore, the storage unit 120 may be configured to store data related to the assistant service according to the present invention by using a memory provided (or inserted) in the electronic device 10 or a cloud server.

The controller 130 may utilize at least one of a CPU and a cloud server provided in the electronic device 10 to perform control of each configuration and overall control related to the present invention. Furthermore, in order to carry out the present invention, the controller 130 may have control authority for at least one of the components of the electronic device 10 . Furthermore, the controller 130 may utilize information received or sensed from the components of the electronic device 10 for the present invention. For example, the controller 130 may perform the assistant service according to the present invention based on an image received from the camera 11 of the electronic device 10 and a voice received through the microphone 13 .

Meanwhile, the controller 130 may provide an assistance service according to the present invention based on artificial intelligence, machine learning, or deep learning, based on learning and inference.

Furthermore, the control unit 130 may be configured to output information related to the present invention by using the display unit 12 of the electronic device 10 . In this case, the display unit 12 may be implemented as a touch screen capable of receiving a user input.

In addition, the controller 130 may perform the assistant service according to the present invention by using other components (eg, a proximity sensor, an infrared sensor, an input unit, etc.) provided in the electronic device 10 .

Meanwhile, according to FIG. 1 , as components of the assistant system 100 , the communication unit 110 , the storage unit 120 , and the control unit 130 were included. However, the assistant system 100 according to the present invention is an electronic device. It goes without saying that at least one of the components (eg, the camera 11 , the display unit 12 , the microphone 13 , etc.) may be included as a component.

Meanwhile, the assistant service system 100 according to the present invention may exist not only in the form of a program or an application, but may also exist in various forms in which configurations and functions described below can be performed.

Accordingly, hereinafter, the assistant service system 100 according to the present invention will be described without any strict limitations thereon.

As shown in FIG. 1 , the assistant service system 100 according to the present invention senses a user using the camera 11 of the electronic device 10 and performs an assistant service for the sensed user. have.

For convenience of description, in the present invention, a user who is provided with the assistant service will be referred to as a “target user (1000)”.

In the present invention, in order to perform a target function for the target user 1000 , by performing user authentication for the target user 1000 , a target function within a preset authority of the target user 1000 may be provided.

In the present invention, “user authentication” may mean user authentication for the target user 1000 .

In the present invention, a series of controls related to a “target function” may be performed using “user authentication”.

Here, the type of “target function” may be very diverse, for example, a reservation function for a place (eg, a conference room reservation, a restaurant reservation, etc.), a payment function, an attendance confirmation function, a visit confirmation at a hospital , ordering function (eg, ordering a drink in a cafe, etc.), electronic device control function (eg, electronic device provided in the conference room (eg, TV, speaker, lighting, etc.), vehicle finding function in the parking lot It may mean various functions that may be provided for the convenience of the user, such as.

Meanwhile, the electronic device control function may include a function for controlling an electronic device provided in a specific space, such as a conference room or a cafe. Control of such an electronic device may use Internet of Things (IoT)-based control. The electronic device control function may additionally include an air conditioner control, a projector control, and the like, in addition to the control of the TV, the speaker, and the lighting described above.

This electronic device control function may be activated or deactivated through a voice command input through the microphone 13 provided in the electronic device 10 .

When a voice command such as “Turn off the lights”, “Turn on the lights”, “Turn on the air conditioner” is received, the controller 130 generates a control command for the electronic device corresponding to the received voice command, and sends the control command to the corresponding electronic device. By transmitting the control command, it is possible to control the electronic device.

In the present invention, the controller 130 obtains a face image of the target user 1000 from the image received through the camera 11, and uses user information corresponding to the face image stored in the user DB, An assistant service for the user 1000 may be performed.

In this case, the target function to be subjected to the user authentication may be specified based on various criteria.

As an example, the controller 130 may specify a target function according to a place where the electronic device 10 is installed. For example, as shown in FIG. 1 , when the electronic device 10 is disposed around a conference room, the control unit 130 may specify a place reservation (eg, conference room reservation) function as a target function. . And, as shown in FIG. 10 , when the electronic device 10 is disposed around a parking lot, the control unit 130 may specify a function of searching for a location where the target user's vehicle is parked as a target function. On the other hand, as the target function, whether or not to specify which function may exist may be determined in advance based on the location where the electronic device 10 is disposed.

As another example, the controller 130 may specify a target function based on a voice received through the microphone 13 . The control unit 130 senses the face image of the target user 1000 located around the electronic device 10 and receives voice information of the target user 1000 through the microphone 13 of the target user 1000 . can

The controller 130 may analyze voice (or voice information) information sensed (or received) through the microphone 13 based on a preset natural language processing algorithm. In addition, the controller 130 may specify a target function from the received voice based on the analysis. The controller 130 may specify a target function based on a keyword or command included in the voice.

As another example, the controller 130 may specify a target function based on a user input to the electronic device 10 . In this case, the user input may be a user input input through the display unit 12 provided with a touch screen or other user input means. The controller 130 may specify a target function based on such a user input.

As another example, the controller 130 may specify an appropriate function as the target function based on the target user's schedule included in the target user's user information.

In addition, the target function may be specified in various ways, and in the present invention, an assistant service may be provided so that the target user may smoothly perform or receive the target function.

Meanwhile, the control unit 130 uses the face image of the target user 1000 and user information extracted from the user DB to confirm the authority of the target user 1000 for the target function, and the target user 1000 You can perform an assistant service for the target function within the authority of .

Also, the controller 130 may utilize user information of the target user 100 to perform a target function. For example, when the target function is an attendance check function, the controller 130 performs either an attendance check approval process or an attendance check rejection process for the target user 1000 based on the user information of the target user 1000 . can do.

As another example, when the target function is a payment function, the control unit 130 may perform either payment approval processing or payment rejection processing on the payment target amount using user information of the target user 1000 . .

Meanwhile, the user information of the target user 1000 is various information related to the target user 1000 , and the controller 130 extracts a face image of the target user 1000 from the image received through the camera 11 . and extracting user information corresponding to (or matched with) the extracted face image from the user DB.

Here, the user DB may also be named “user-related information”, “user personal information” or “user identification information”.

The user DB may be included in at least one of the storage unit 120 , the external database 140 , and the cloud server.

The controller 130 may search for a person corresponding to the face image of the target user 1000 from the user DB, and extract user information matched with the searched person.

In the user DB, user information for each person may be stored on a per person (or person) basis. That is, user information corresponding to each different person may be matched and stored in the user DB.

Here, the user information includes a person's name, date of birth, address, phone number, e-mail address, account information, employee number, ID, face image, and biometric information (eg, fingerprint information, iris information, etc.) corresponding to each person. ), permission information for a specific function (or specific event), reference face image, identification information of electronic devices, schedule (schedule), work department, title, title, position, payment information (eg, account number, credit card information, etc.) and the like).

Here, the reference face image may be one or a plurality of face images for a specific person. In the present invention, the “reference face image” may also be referred to as “reference data”.

The controller 130 compares the target user's face image obtained through the camera 11 with the reference face image stored in the user DB, and searches for a person matching the reference face image corresponding to the target user's face image can do. Then, the controller 130 may specify the searched person as the target user, and extract user information matched with the searched person from the user DB.

In addition, the controller 130 may perform an assistant service for a target function by using the extracted user information.

As shown in FIG. 1 , the electronic device 10 may be disposed at any location, and the target user 1000 may perform an assistant service just by being located in the vicinity of the electronic device 10 .

Hereinafter, an assistant service providing method according to the present invention will be described in more detail along with a flowchart.

First, as shown in FIG. 2 , according to the method for providing an assistant service according to the present invention, a process of sensing a face image of a target user using the camera 11 may be performed ( S210 ).

That is, the controller 130 may detect the face image of the target user from the image received from the camera 11 by using the camera 11 (refer to FIG. 1 ) of the electronic device 10 .

More specifically, as shown in FIGS. 3A and 4A , the controller 130 may sense the face of the target user 1000 located in the vicinity of the electronic device 10 through the camera 11 . have.

The controller 130 obtains a face image 410 of the target user 1000 from the image received through the camera 11, and uses the user information stored in the user DB corresponding to the face image, An assistant service for the user 1000 may be performed.

When the face image of the target user 1000 is detected from the image received through the camera 11 , the controller 130 may perform a series of processes for the assistant service.

The timing at which the face image of the target user 1000 is sensed or detected through the camera 11 may be very diverse.

As an example, when a user input is applied to the electronic device 10 , the controller 130 may control the camera 11 to photograph the face of the target user 1000 . Here, the user input may be an input according to a physical contact to the electronic device 10 or an input using a voice input using various methods.

As another example, as shown in FIG. 3 , when it is sensed that a person is located around the electronic device 10 , the controller 130 receives an image of the target user 1000 through the camera 10 . The camera 10 can be controlled as much as possible. In this case, the controller 130 controls the camera 10 to detect a face image of the target user 1000 when the target user 1000 is located within a preset range (or distance) based on the electronic device 10 . can do. The control unit 130 uses at least one sensor (eg, a proximity sensor, an infrared sensor, or a lidar sensor) provided in the electronic device 10 , to the vicinity of the electronic device 10 or to the electronic device 10 . ), it is possible to sense (or sense) that the target user is located within a preset range.

On the other hand, when the target user 1000 is not located within a preset range of the electronic device 10 , the control unit 130 controls the display unit 12 to deactivate the display unit 12 of the electronic device 10 . can control The display unit 12 in the inactive state may be in at least one of a power-off state, a standby state, and a power saving state.

Furthermore, the controller 130 controls the display unit 12 to activate the display unit 12 of the electronic device 10 when the target user 1000 is located within a preset range of the electronic device 10 . can do. In this case, guide information on a method of specifying a target function (eg, “Tell me a desired function”) may be output to the display unit 12 .

As another example, the controller 130 may determine whether the face image of the target user 1000 is included in the image received through the camera 11 at a preset interval or in real time. And, as a result of the determination, when the face image of the target user 1000 is included in the image received through the camera 11 , a series of processes for the assistant service related to the target function may be performed.

As another example, when a sound (or voice) is sensed through the microphone 13 provided in the electronic device 10 , the controller 130 may activate the camera 11 . That is, when a sound of a person is sensed through the microphone 13 , the controller 130 may determine that a person is located around the electronic device 10 and activate the camera 11 .

As another example, as shown in FIG. 4B , the controller 130 may activate the camera 11 based on selection of the function icon 450 included in the electronic device 10 . A function for activating the assistance service according to the present invention may be matched to the function icon 450 . The function icon 450 may be output on the display unit 12 provided in the electronic device 10 . When the function icon 450 is selected by a user or the like by a method such as a touch, the controller 130 may activate the assistance service according to the present invention. In this case, the camera 11 may be activated based on the activation of the assistance service.

Meanwhile, the controller 130 may detect the face image of the target user 1000 from the image received from the camera 11 by using at least one face recognition algorithm. The face recognition algorithm utilized in the present invention is not limited in its type, and algorithms operating based on artificial intelligence or deep learning may also be utilized.

On the other hand, even when the face image of the target user 100 is detected through the camera 11 , the controller 130 determines that the detected face image is a preset condition (or preset authentication condition) for performing the assistant service. is not satisfied, guide information for obtaining a face image that satisfies the preset condition may be provided.

More specifically, the controller 130 may determine whether the face image of the target user 1000 sensed by the camera 11 satisfies the preset condition. Here, the preset conditions include the size of the face image included in the image, the angle of the face corresponding to the face image, the position of the face image in the image, and features of the face in the image (eg, eyes, nose). , mouth) or not).

The controller 130 may output at least one of information on the posture and location of the target user 1000 as guide information so that a face image satisfying a preset condition can be obtained. The guide information may be related to at least one of a relative position and a relative distance between the camera 11 and the target user 1000 .

For example, the control unit 130 may i) location information of the target user 1000 based on the electronic device (eg, come closer, go further, please adjust the distance from the screen), ii) Posture information of the target user 1000 (eg, look at the screen), iii) Information recommending removal of the target user 1000's face when it is covered by an object (eg, a mask) (eg, For example, please take off your mask).

Furthermore, in the assistant service providing method according to the present invention, a process of sensing the target user's voice may be performed using a microphone ( S220 ).

In the present invention, the process of sensing the face image of the target user 1000 through the camera 11 ( S210 ) and the process of sensing the voice of the target user using the microphone 13 ( S220 ) are performed by the electronic device 10 . ) at the same time is also possible. That is, on the electronic device 10 , the controller 130 may receive the image information through the camera 11 and the audio information through the microphone 13 at the same time. This case corresponds to a case where the target user 1000 speaks while being located near the electronic device 10 .

On the other hand, the process ( S210 ) of sensing the face image of the target user 1000 through the camera 11 and the process ( S220 ) of sensing the voice of the target user by using the microphone 13 are free according to the situation. can be deformed. That is, after the process (S210) of sensing the face image of the target user 1000 through the camera 11 is performed, the process of sensing the voice of the target user using the microphone 13 (S220), or Conversely, after the process ( S220 ) of sensing the target user's voice by using the microphone 13 is performed, the process ( S210 ) of sensing the face image of the target user 1000 through the camera 11 may be performed. .

Meanwhile, the process of sensing the target user's voice using the microphone 13 may be understood as a process of receiving sound information through the microphone.

The controller 130 may analyze the voice information received through the microphone 13 based on a natural language processing algorithm, and determine a target function based on the analysis result.

For example, as illustrated in FIG. 3 , the target user 1000 may utter information about the target function (eg, “reserve a conference room please”) through voice. The controller 130 may specify a function corresponding to the voice as a target function based on the reception of the voice. The controller 130 may specify the target function by comparing the function list corresponding to various keywords with information included in the voice. Such a function list may be stored in the storage unit 120 .

Meanwhile, when a target function is specified, guide information indicating that a specific function has been determined as a function corresponding to the voice may be output on the display unit 12 . Also, the controller 130 may output guide information through the sound output unit (or speaker).

Next, in the present invention, the process of extracting user information corresponding to the detected face image from the user DB may proceed (S230).

More specifically, the controller 130 extracts the face image of the target user 1000 from the image received through the camera 11, and corresponds (or matches) the extracted face image from the user DB. User information can be extracted.

As described above, the user DB may also be named “user-related information”, “user personal information” or “user identification information”.

The user DB may be included in at least one of the storage unit 120 , the external database 140 , and the cloud server.

The controller 130 may search for a person corresponding to the face image of the target user 1000 from the user DB, and extract user information matched with the searched person.

In the user DB, user information for each person may be stored on a per person (or person) basis. That is, user information corresponding to each different person may be matched and stored in the user DB.

Here, the user information includes a person's name, date of birth, address, phone number, e-mail address, account information, employee number, ID, face image, and biometric information (eg, fingerprint information, iris information, etc.) corresponding to each person. ), permission information for a specific function (or specific event), reference face image, identification information of electronic devices, schedule (schedule), work department, title, title, position, payment information (eg, account number, credit card information, etc.) and the like).

Here, the reference face image may be one or a plurality of face images for a specific person. In the present invention, the “reference face image” may also be referred to as “reference data”.

The controller 130 compares the target user's face image obtained through the camera 11 with the reference face image stored in the user DB, and searches for a person matching the reference face image corresponding to the target user's face image can do. Then, the controller 130 may specify the searched person as the target user, and extract user information matched with the searched person from the user DB.

The controller 130 may extract the detected face image of the target user 1000 and a specific reference face image having a degree of similarity satisfying a preset condition from the user DB. In addition, a person matching the extracted specific reference face image may be specified as the target user.

While the search for the target user 1000 is in progress, the control unit 130 controls the display unit 12 to output guide information (eg, “I am looking for you”) on the current progress to the electronic device 10 . ) and a sound output unit (eg, a speaker, etc.) can be controlled.

As shown in FIG. 4A , an image captured by the camera 11 may be output to the display unit 12 . In this case, the output image may include a face image 410 of the target user 1000 .

As described above, when a person corresponding to the target user 1000 is specified and user information for the specified person (eg, Lee*ri) is output, the controller 130 controls at least a portion of the user information. may be provided to the display unit 12 .

Accordingly, as shown in FIG. 4A , at least a portion of user information (eg, name information (Lee * Lee)) corresponding to the person corresponding to the target user 1000 may be output to the display unit 12 . Accordingly, the target user 1000 may check whether the electronic device 10 has correctly searched for itself with reference to the information output on the display unit 12 .

As described above, when user information on the target user 1000 is extracted, in the present invention, a process of confirming the authority of the target user 1000 for the target function may be performed using the extracted user information. (S240).

As described above, the target function may be specified based on a voice input through a microphone.

In addition, the target function may be specified by various methods other than the voice input through the microphone. When the target function is not specified based on the voice, it is also possible that step S220 is omitted in the assistant service providing method according to the present invention.

Referring to an example in which the target function is specified by another method, as an example, the control unit 130 may specify the target function according to a place where the electronic device 10 is installed. For example, as shown in FIG. 3 , when the electronic device 10 is disposed around a conference room, the control unit 130 may specify a place reservation (eg, conference room reservation) function as a target function. . In this case, even if a separate user input is not received from the target user 1000 , the control unit 130 checks the authority of the target user 1000 based on the sensing of the target user 1000 by the camera 11 . process can be performed.

And, as shown in FIG. 10 , when the electronic device 10 is disposed around a parking lot, the control unit 130 may specify a function of searching for a location where the target user's vehicle is parked as a target function. On the other hand, as the target function, whether or not to specify which function may exist may be determined in advance based on the location where the electronic device 10 is disposed.

As another example, the controller 130 may specify a target function based on a user input to the electronic device 10 . In this case, the user input may be a user input input through the display unit 12 provided with a touch screen or other user input means. The controller 130 may specify a target function based on such a user input.

As another example, the controller 130 may specify an appropriate function as the target function based on the target user's schedule included in the target user's user information.

Meanwhile, in order to specify the target function, the controller 130 may output a question to the target user 1000 and specify the target function based on a response to the output question.

The specification of the target function may require information specific enough to confirm the authority of the target user. For example, when the target function is a reservation for a place, the controller 130 may need information about a specific reservation place (eg, a conference room number, etc.) and reservation time.

In order to specify the target function, the controller 130 may ask the target user 1000 in various ways. For example, the controller 130 may output a question to the target user 1000 through a speaker and receive an answer to the question by receiving a voice through the microphone 13 . The control unit 130 may output a question a plurality of times and receive an answer until information that is enough to materialize the target function (or to the extent that it can be determined) is collected.

Also, the controller 130 may collect information for specifying the target function by outputting a question to the display unit 12 and receiving an answer through the display unit 12 or the microphone 13 .

Meanwhile, even if the control unit 130 does not ask the target user 1000 , when the specification of the target function is completed, the control unit 130 may not output the question. For example, the target user 1000 may initially provide information sufficient to specify a specific function to the electronic device 10 , such as “Reserve a conference room 3 at 3 pm on Monday”.

As such, when the target function is specified by the user's voice or another method, the controller 130 may check the authority of the target user for the target function.

Here, “the right of the target user” may be interpreted in various meanings. For example, the authority of the target user is i) whether the target user 1000 has the authority to receive the target function, ii) whether the target user 1000 has the authority to access the target function, iii) whether the target user ( Whether the target user 1000 has permission to access a specific place, iv) whether the target user 1000 has permission to control a specific electronic device, etc. It may be information about whether you have permission to access.

Such information (or authority information) on the authority of the target user 1000 may be included in the user information of the target user 1000 .

For example, when the target function is a function for reserving a place as shown in FIG. 3 , the control unit 130 determines that the target user 1000 selects a specific place (eg, based on user information of the target user 100 ). For example, it may be determined whether the authority to access or reserve the reservation target place) exists.

In this case, the target function may be understood as a reservation function for a specific time for a specific place.

Furthermore, in order to check information (or authority information) on authority of the target user 1000 , the controller 130 may check information for authority check from the target user 1000 . Here, the information for verifying authority may be information for authenticating the target user 1000 , such as biometric information and password information of the target user 1000 .

Meanwhile, as described above, the specification of information related to a reservation, such as a specific place or a reservation time, may be made based on at least one of a voice received from the target user 1000 and schedule information of the target user.

In addition, the specification of information related to a reservation, such as a specific place or a reservation time, may be made based on at least one of the places where the electronic device 10 is disposed.

For example, the controller 130 may determine a specific location based on a location in which the electronic device 10 sensing the face image and voice of the target user 1000 is disposed. For example, the control unit 130 may specify a place (eg, a conference room) closest to the location where the electronic device 10 is disposed as a specific place. Information (eg, a conference room number) may be matched, and the controller 130 may determine a specific place based on the information on the place matched with the electronic device 10 .

Meanwhile, as described above, when the target user's authority to the target function is checked, in the present invention, a process of performing control related to the target function may be performed based on the verification result (S250).

More specifically, the control unit 130 may perform different controls according to whether the target user 1000 has an authority for a specific function.

When the target user 1000 has authority for a specific function, the controller 130 may perform or execute a specific function.

Furthermore, when the target user 1000 has the authority for a specific function, but cannot or cannot perform the specific function, the control unit 130 may recommend or perform a function similar to the specific function,

That is, when the target user 1000 has no authority for a specific function, the controller 130 may restrict the execution of the specific function. In this case, the controller 130 may recommend or perform a function similar to a specific function.

For example, if the specific function is a function of reserving a specific place, the control unit 130 may control each other related to the reservation for the specific place based on whether the target user 1000 has a reservation right for the specific place. Other controls can be performed.

For example, when the reservation right for a specific place exists in the target user 1000 , the controller 130 may make a reservation for the specific place as shown in FIG. 3B . In this case, the control unit 130 provides guidance information indicating that a specific function can be normally performed (eg, “I will reserve the 4th meeting room right now!”) in at least one of visual and auditory information. , may be output on the electronic device 10 .

As shown in FIG. 4A , information about a reservation for a specific place (eg, situation information or guide information, etc. 420 ) may be output on the display unit 12 . The control unit 130 may provide a graphical user interface (GUI) 430 on the display unit 12 to allow the target user 1000 to change the reservation time or place at any time, or to confirm the reservation. . Such a GUI 430 may also be called an icon or a graphic object.

As another example, when the control unit 130 has the reservation right for a specific place to the target user 1000 , but cannot make a reservation for the specific place (eg, reservation for the specific place has already been completed) case, etc.), as shown in (c) of FIG. 3 , it is possible to make or recommend a reservation for a place that is different from a specific place. The controller 130 may determine whether a reservation is possible for a place (eg, a conference room, etc.) of a space including the target user, and based on the determination result, may recommend a place where another reservation is possible.

At this time, the control unit 130, i) a place closest to a specific place, ii) a place closest to the current location of the target user 1000 excluding a specific place, iii) a work of the target user 1000 except for a specific place The location closest to the location, iv) Except for a specific location, other reservationable locations are selected based on various criteria, such as a location that considers both the distance of other users (eg, meeting attendees, etc.) and the target user 1000 . It is possible to search and generate the retrieved information as recommended information.

As another example, when the reservation right for a specific place does not exist in the target user 1000, the control unit 130 makes a reservation for a place different from the specific place, as shown in (d) of FIG. 3 . can guide A method of guiding to another place may be performed through the electronic device 10 , and the controller 130 may guide the target user 1000 through at least one of an auditory and a visual method.

In this case, the control unit 130 provides guide information (eg, “You do not have reservation rights for the 4th meeting room”) to guide information informing the target user that there is no reservation authority for a specific place. “Can I reserve the 7th meeting room?) and print it out. Such information may be output as visual information 510 on the display unit 12 as shown in FIG. 5A . Furthermore, the display unit 12 may include GUIs 520 and 530 for terminating the execution of a specific function or providing another recommended place. These GUIs 520 and 530 may also be called icons or graphic objects. When any one of the GUIs 520 and 530 is selected, the controller 130 may execute a function matched to the selected GUI. For example, when the GUI 520 corresponding to the end of the execution of a specific function is selected by the user (or the target user 1000 ), the controller 130 terminates the place reservation process for the target user 1000 . can do. On the other hand, when the GUI 530 corresponding to the function of providing another recommended place to the user (or the target user 1000 ) is selected, the controller 130 may output the recommended place information. Furthermore, it goes without saying that the output of the recommended place information may be output even when the GUI 530 is not selected by the user.

As described above, the control unit 130 may search for a place where a reservation is possible while the target user 1000 has reservation authority, and may generate the searched information as recommendation information. The controller 130 may search for a recommended place based on various criteria, and may provide information on one or a plurality of recommended places to the target user 1000 .

The control unit 130, i) a place closest to a specific place, ii) a place closest to the current location of the target user 1000 excluding a specific place, iii) a work location of the target user 1000 excluding a specific place, and The nearest place, iv) Except for a specific place, other users (eg, meeting participants, etc.) and a place that considers the distance of the target user 1000, search for other reservationable places based on various criteria, , the searched information may be generated as recommended information.

For example, as shown in FIG. 5B , on the display unit 12 , a place closest to the current location of the target user 1000 may be provided as the recommendation information 550 . Similarly, as shown in FIG. 5B , on the display unit 12 , a place closest to the work location of the target user 1000 may be provided as the recommendation information 550 . In this case, the place provided as the recommendation information may be a place where the target user 1000 has the right to make a reservation.

On the other hand, as shown in FIG. 6 , a specific function is a reservation function for a place, and information on participants (eg, Kim * Dong, Yu * Ae) is received based on a user input by the target user. In this case (refer to reference numeral 620 in FIG. 6 ), the controller 130 controls a specific place (for example, A Meeting rooms 1, 3, 5) can be searched as recommended places. Then, as shown in FIG. 6 , information 630 on the searched recommended place may be output on the display unit. In this case, the control unit 130 on the map image, a recommended place location, a location related to the target user 1000 (eg, the current location of the target user or the working location of the target user, etc.), a location related to the participant (eg, For example, the current location of the participant or the work location of the target user) may be displayed.

When the information on the participant is received, the controller 130 extracts the user information of the participant from the user DB, and searches for recommended place information based on the location information related to the participant included in the extracted user information of the participant ( or specific).

On the other hand, in the above examples, a case in which a specific function and a place are specifically specified was examined. However, the present invention is not limited thereto, and based on authority information (eg, access authority to a place, access authority, etc.) of the target user 1000, information on a place that can be reserved may be recommended. Of course, the controller 130 may search for a place that can be reserved while the target user 1000 has the right to make a reservation based on the above-described method for recommending a place, and may generate recommendation information on the searched place.

For example, the target user 1000 may input a voice such as “reserve a meeting room” into the electronic device 10 , and the controller 130 may use at least one of the various methods described above based on the user's voice. Based on , a place that the target user 1000 can reserve may be generated as recommended place information.

Meanwhile, in the present invention, in order to provide an assistant service that enhances user convenience, when a reservation for a specific place is performed, reservation information for a specific place is provided to the target user based on user information of the target user 1000 . It is possible to provide a function to add to the schedule information of

As shown in FIG. 7 , when a schedule event related to the schedule of the target user 1000 occurs as a result of performing a specific function, the control unit 130 may add the schedule event to the schedule information of the user.

The controller 130 may extract account information in which the user's schedule information is stored from the user information, and additionally store a schedule corresponding to a schedule event in the corresponding account information.

In this way, when the assistant service related to schedule addition is provided, the controller 130 transmits guide information 710 informing that the assistant service related to schedule addition is being provided in at least one of an auditory and visual method. can be printed out.

The controller 130 may provide a graphic user interface (GUI) 720 on the display unit 12 for allowing the target user 1000 to approve or reject an assistant service related to schedule addition.

Furthermore, when a reservation for a specific place is performed and information on the attendee related to the specific place is confirmed, the controller 130 may extract user information of the attendee from the user DB. In addition, the controller 130 may provide an assistant service for sharing reservation information for the specific place to the attendee by using the user information of the attendee. The controller 130 may transmit reservation information to the attendees by using contact information such as a mail account and a phone number included in the user information of the attendees. Such reservation information may be transmitted in the form of e-mail, message, or the like.

Furthermore, the controller 130 may extract account information in which schedule information of the attendee is stored from the user information of the attendee, and additionally store a schedule corresponding to a schedule event in the corresponding account information.

As such, when an assistant service related to schedule addition is provided, the controller 130 may output guide information indicating that an assistant service related to schedule addition is being provided in at least one of an auditory and visual method. have.

Furthermore, the controller 130 may display a graphical user interface (GUI) 720 that allows the target user 1000 to approve or reject an assistant service (eg, transmission of reservation information or addition of a schedule, etc.) related to the attendee on the display unit 12 . ) can be provided.

Meanwhile, the electronic device 10 to which the service providing method and system described above is applied may be installed in various places. As previously shown in FIG. 3, the electronic device 10 is disposed around the conference room, or as shown in FIG. can be

And, when the user (eg, the target user 1000) is recognized in the vicinity of the electronic device 10, the controller 130 may provide an assistant service for the recognized target user 1000,

For example, as shown in (b) of FIG. 8 , the control unit 130 is configured to provide a specific function (eg, a conference room reservation function) to the target user 1000 through the electronic device 10 . Guide information 810 may be output.

As another example, as shown in (c) of FIG. 8 , when the target user 1000 is specified, the control unit 130 refers to schedule information included in the user information of the target user 1000 and performs a target function. can provide Here, the target function may be a function in consideration of the characteristics of the place where the electronic device 10 is located. For example, when the place where the electronic device 10 is disposed is around an elevator, the control unit 130 controls the target user ( 1000) can be extracted, and guide information (eg, conference room 4) so that the target user 1000 can move to a place (eg, conference room 4) corresponding to the extracted schedule information. This location (moving to the third floor) can be output through the electronic device 10 .

Furthermore, as shown in (d) of FIG. 8 , the control unit 130 refers to schedule information included in the user information of the target user 1000 to remind (reminder) schedule information to the target user 1000 . Information 830 for this may be output through the electronic device 10 .

Meanwhile, the method and system for providing an assistant service according to the present invention may provide various functions in addition to the above-described function for place reservation.

As an example, when the electronic device 10 is disposed in a space 900 such as a conference room, office, home, etc., the control unit 130 based on a user input input through the electronic device 10 in FIG. Electronic devices 901 , 902 , and 903 disposed in 900 may be controlled. Even at this time, the controller 130 recognizes the face of the target user 1000 and recognizes the face of the target user 1000 in order to determine whether the target user 1000 has the authority to control the electronic devices 901 , 902 , and 903 . User information corresponding to the image can be extracted from the user DB. Then, the controller 130 checks the control authority information for the electronic devices 901 , 902 , 903 from the extracted user information, and performs control of the electronic devices 901 , 902 , 903 based on the check result can do.

On the other hand, the control of the electronic devices 901 , 902 , 903 , and 904 is to turn on or off the power of the electronic devices 901 , 902 , 903 , or to adjust the volume of the electronic devices. It may include various controls associated with (901, 902, 903).

As another example, the electronic device 10 may be disposed in a parking lot 1010 as shown in FIG. 10 . In this case, when the target user 1000 is recognized through the electronic device 10 , the controller 130 may perform a control related to the vehicle 1020 within the authority of the target user 1000 .

The controller 130 recognizes the face of the target user 1000 through the electronic device 10 in order to extract information on the vehicle 1020 that is within the authority of the target user 1000 , and corresponds to the recognized face image. User information can be extracted from the user DB. In addition, the controller 130 may check information of a vehicle 1020 within the authority of the target user 1000 (eg, a vehicle owned by the target user 1000 , a vehicle registered for use, etc.) from the extracted user information. In addition, the controller 130 may perform a control related to the identified vehicle 1020 (eg, providing parking location information of the vehicle 1020 , payment of a parking fee for the vehicle 1020 , etc.).

Meanwhile, information on the number of the vehicle 1020 , the parking location, the caring time, etc. may be sensed based on a sensor provided in the parking lot 1010 while the vehicle 1020 enters the parking lot 1010 . . And, when there is a user having user information corresponding to the vehicle number information among the sensed vehicle 1020 information, the control unit 130 includes information related to parking of the vehicle (parking location, parking time, etc.) in the user's information. ) can be matched. Accordingly, when the target user 1000 requests a function (eg, providing parking location information, payment of parking fee, etc.) related to the vehicle 1020 through the electronic device 10 , information registered with the user information You can provide appropriate information by referring to .

As another example, as shown in FIG. 11 , when the electronic device 10 is disposed in a cafe 1100 , a restaurant, etc., the control unit 130 orders from the target user 1000 through the electronic device 10 . Receive information and perform electronic payment.

In this case, the controller 130 may recognize the target user 1000 and extract user information of the target user 1000 from the user DB. Then, using the extracted user information (eg, payment method information (credit card information, account information, etc.)), an electronic payment for the order of the target user 1000 may be performed.

Meanwhile, the user information of the target user 1000 may include history information of the target user 1000 , and the history information may include past order information of the target user 1000 .

When the target user 1000 is recognized based on this history information, the control unit 130 sends to a specific menu (eg, “cafe latte”) through the electronic device 10 based on past order information. As shown in FIG. 11 , order recommendation information 1110 for recommending an order for the user may be output.

On the other hand, when the space such as the cafe 1100 is a space in which a robot (eg, a serving robot, etc.) is operated, the control unit 130 uses the robot and a menu ordered by the target user 1000 (eg, The robot may be controlled so that drinks, food, etc.) are served to the target user 1000 . In this case, the controller 130 may sense where the target user 1000 is located in the space and control the robot to move to the sensed position.

As described above, the method and system for providing an assistant service according to the present invention can provide various functions capable of assisting a user according to a location in which an electronic device to which the method and system for providing an assistant service is applied is disposed.

As described above, the method and system for providing an assistant service according to the present invention can provide a customized function that can be provided to a recognized user by utilizing face recognition. Furthermore, according to the present invention, a function according to the recognized user's voice can be provided by performing voice recognition together with face recognition as well as face recognition. Accordingly, the user can be provided with the performance of a desired function simply by uttering a speech around the electronic device that provides the assistant service.

Furthermore, the method and system for providing an assistant service according to the present invention sense a user to be authenticated using a camera and check the sensed user's authority, thereby assisting in providing a function existing within the user's authority. can In this case, in the present invention, when the function requested by the user does not exist within the user's authority, the user's convenience can be improved by recommending provision of a similar function existing within the user's authority.

Meanwhile, the computer-readable medium includes all types of recording devices in which data readable by a computer system is stored. Examples of computer-readable media include Hard Disk Drive (HDD), Solid State Disk (SSD), Silicon Disk Drive (SDD), ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, etc. There is this.

Furthermore, the computer-readable medium may be a server or a cloud storage that includes a storage and that an electronic device can access through communication. In this case, the computer may download the program according to the present invention from a server or cloud storage through wired or wireless communication.

Furthermore, in the present invention, the computer described above is an electronic device equipped with a processor, that is, a CPU (Central Processing Unit, Central Processing Unit), and there is no particular limitation on the type thereof.

On the other hand, the above detailed description should not be construed as limiting in all respects, but should be considered as exemplary. The scope of the present invention should be determined by a reasonable interpretation of the appended claims, and all modifications within the equivalent scope of the present invention are included in the scope of the present invention.

Claims (13)

sensing a face image of a target user using a camera;
sensing the target user's voice by using a microphone;
extracting user information of the target user corresponding to the face image from a user DB;
confirming the authority of the target user for the target function corresponding to the voice based on the user information; and
and performing control related to the target function based on a result of the verification. According to claim 1,
and analyzing the voice based on a natural language processing algorithm, and determining the target function based on a result of the analysis. 3. The method of claim 2,
and providing, on a display unit, at least part of guide information indicating that the target function has been determined as a function corresponding to the voice and user information of the target user, on a display unit. 3. The method of claim 2,
As a result of the analysis, if the target function is a reservation function for a specific place, in the step of checking the authority,
and determining whether the target user has a reservation right for the specific place, based on the user information. 5. The method of claim 4,
The specific place is
The assistant service providing method, characterized in that it is determined based on a location in which the electronic device including the camera and the microphone is disposed. 5. The method of claim 4,
In the step of performing control related to the target function,
and performing different controls related to a reservation for the specific place, based on whether the target user has a reservation right for the specific place. 7. The method of claim 6,
In the step of performing control related to the target function,
When the reservation right for the specific place exists in the target user, the reservation is performed for the specific place,
When the reservation right for the specific place does not exist in the target user, the method for providing an assistant service, characterized in that the guide guides a reservation for a place different from the specific place. 8. The method of claim 7,
A place different from the specific place mentioned above,
The assistant service providing method, characterized in that it is a place closest to the specific place and a place where the target user has a reservation right. 8. The method of claim 7,
When the reservation for the specific location is performed, the method further comprising the step of adding reservation information for the specific location to the schedule information of the target user based on the user information of the target user. Way. 8. The method of claim 7,
When a reservation for the specific place is made, and information on attendees related to the specific place is received,
and extracting user information of the attendee from the user DB and sharing reservation information for the specific place with the attendee using the user information of the attendee. 3. The method of claim 2,
As a result of the analysis, when the target function is a reservation function for a place, and information on attendees is received based on a user input by the target user,
and searching, as a recommended place, a specific place existing within the reservation right of the target user, based on the respective locations associated with the participant and the target user. a camera for sensing a face image of a target user;
a microphone for sensing the target user's voice;
a control unit for extracting user information of the target user corresponding to the face image from a user DB, and confirming the authority of the target user for the target function corresponding to the voice based on the user information;
The control unit is
and performing control related to the target function based on a result of the check. A program that is executed by one or more processes in an electronic device and can be stored in a computer-readable medium,
The program is
sensing a face image of a target user using a camera;
sensing the target user's voice by using a microphone;
extracting user information of the target user corresponding to the face image from a user DB;
confirming the authority of the target user for the target function corresponding to the voice based on the user information; and
A program storable in a computer-readable medium, comprising instructions for performing a control related to the target function based on a result of the check.

Images