KR102511517B1 - Voice input processing method and electronic device supportingthe same - Google Patents

Abstract

An electronic device comprising a microphone, a communication circuit for communication with an external electronic device, a processor electrically connected to the microphone and the communication circuit, and a memory electrically connected to the processor, wherein the memory, when executed, The processor receives a first voice input related to performing a task through the microphone, checks a first voice code necessary for performing the task, obtains surrounding information of the electronic device, and determines the surrounding information. When a condition is satisfied, an electronic device that stores instructions for changing password authentication settings necessary for performing the task and transmitting a signal related to the password authentication setting change to the external electronic device through the communication circuit is initiated In addition to this, various embodiments identified through the specification are possible.

Description

Voice input processing method and electronic device supporting the same

Embodiments disclosed in this document are related to voice input processing technology.

An artificial intelligence system (or integrated intelligence system) is a computer system that implements human-level intelligence, and a machine learns and judges itself, and the recognition rate improves as it is used.

Artificial intelligence technology consists of machine learning (deep learning) technology using an algorithm that classifies/learns the characteristics of input data by itself, and element technologies that mimic the functions of the human brain, such as recognition and judgment, by using machine learning algorithms.

Elemental technologies include, for example, linguistic understanding technology that recognizes human language/characters, visual understanding technology that recognizes objects as human eyes, reasoning/prediction technology that logically infers and predicts information by judging information, and human experience information. It may include at least one of a knowledge expression technology for processing as knowledge data and a motion control technology for controlling the autonomous driving of a vehicle and the movement of a robot.

Among the above-mentioned element technologies, linguistic understanding is a technology for recognizing and applying/processing human language/text, and includes natural language processing, machine translation, dialogue system, question and answering, voice recognition/synthesis, and the like.

Meanwhile, an electronic device equipped with an artificial intelligence system analyzes voice data input by a user's speech and determines a task (or function) to be performed by the electronic device or an external electronic device connected to the electronic device based on the analyzed result. and may support the task to be performed in an electronic device or an external electronic device.

Also, the electronic device may receive a password required to perform a task from a user. For example, the electronic device may receive a set voice password from the authenticated user to confirm whether or not the task is performed.

However, depending on the surrounding information of the electronic device, there is a possibility of exposure of the voice code. For example, if another user other than the authenticated user exists around the electronic device and the authenticated user utters a voice code, the voice code may be exposed, which may cause a security problem.

Embodiments disclosed in this document may provide a voice input processing method and an electronic device supporting the voice input processing method for changing password authentication settings necessary for performing a task according to peripheral information of an electronic device.

In addition, the embodiments disclosed in this document may provide a voice input processing method for providing relevant information so as to remind a user of a password authentication setting change, and an electronic device supporting the same.

An electronic device according to an embodiment disclosed in this document includes a microphone, a communication circuit for communication with an external electronic device, a processor electrically connected to the microphone and the communication circuit, and a memory electrically connected to the processor, , When the memory is executed, the processor receives a first voice input related to performing a task through the microphone, checks a first voice password necessary for performing the task, and information about the electronic device's surroundings. is acquired, and when the surrounding information satisfies a specified condition, password authentication setting required for performing the task is changed, and a signal related to the password authentication setting change is transmitted to the external electronic device through the communication circuit. You can store instructions to do it.

In addition, a voice input processing method of an electronic device according to an embodiment disclosed in this document includes an operation of receiving a first voice input related to performing a task through a microphone, and checking a first voice code necessary for performing the task. an operation of acquiring information about the surroundings of the electronic device, an operation of changing password authentication settings necessary for performing the task, and an operation of obtaining information about the surroundings of the electronic device when the surrounding information satisfies a specified condition; It may include an operation of transmitting a signal related to setting change of password authentication.

According to the embodiments disclosed in this document, it is possible to reduce the possibility that a voice password is exposed by changing password authentication settings necessary for performing a task according to peripheral information of an electronic device.

In addition, according to the embodiments disclosed in this document, it is possible to provide a reminder to the user of the password authentication setting change by providing information related to the password authentication setting change.

In addition to this, various effects identified directly or indirectly through this document may be provided.

1 is a diagram showing an integrated intelligent system according to various embodiments of the present invention.
Figure 2 is a block diagram showing a user terminal of the integrated intelligent system according to an embodiment of the present invention.
3 is a diagram illustrating execution of an intelligent app of a user terminal according to an embodiment of the present invention.
4 is a diagram illustrating that a context module of an intelligent service module collects a current state according to an embodiment of the present invention.
5 is a block diagram showing a proposal module of an intelligent service module according to an embodiment of the present invention.
6 is a block diagram showing an intelligent server of an integrated intelligent system according to an embodiment of the present invention.
7 is a diagram illustrating a method of generating a path rule of a path planner module according to an embodiment of the present invention.
8 is a diagram showing that a persona module of an intelligent service module manages user information according to an embodiment of the present invention.
9 is a block diagram of an electronic device related to voice input processing according to an embodiment of the present invention.
10 is a diagram illustrating a method of operating an electronic device related to voice input processing according to an embodiment of the present invention.
11 is a diagram illustrating a method of changing password authentication settings according to whether or not a user is authenticated according to an embodiment of the present invention.
12 is a diagram illustrating a method of changing password authentication settings according to whether an input means can be changed according to an embodiment of the present invention.
13 is a diagram for explaining a method of changing a password authentication setting to a public voice password input method according to an embodiment of the present invention.
14 is a diagram for explaining a method of changing a password authentication setting to a one-time voice password input method according to an embodiment of the present invention.
15 is a diagram for explaining a method of setting a one-time voice password according to an embodiment of the present invention.
16 is a diagram for explaining a method of authenticating a password through a one-time voice password according to an embodiment of the present invention.
17 is a diagram for explaining a method of changing a password input means according to an embodiment of the present invention.
18 is a diagram for explaining a method of reminding a user when password authentication settings are not changed according to an embodiment of the present invention.
19 is a diagram for explaining a method of reminding a user of a password authentication setting change according to an embodiment of the present invention.
20 is a block diagram of an electronic device in a network environment according to various embodiments.
In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar elements.

Prior to describing an embodiment of the present invention, an integrated intelligent system to which an embodiment of the present invention can be applied will be described.

1 is a diagram showing an integrated intelligent system according to various embodiments of the present invention.

Referring to Figure 1, the integrated intelligence system 10 may include a user terminal 100, an intelligent server 200, a personalized information server 300 or a suggestion server 400.

The user terminal 100 provides services required by the user through an app (or application program) stored in the user terminal 100 (eg, an alarm app, a message app, a photo (gallery) app, etc.) can provide. For example, the user terminal 100 may execute and operate another app through an intelligent app (or voice recognition app) stored in the user terminal 100 . The user terminal 100 may receive a user input for executing and operating the other app through the intelligent app. The user input may be received through, for example, a physical button, a touch pad, a voice input, or a remote input. According to an embodiment, the user terminal 100 may correspond to various terminal devices (or electronic devices) capable of connecting to the Internet, such as a mobile phone, a smart phone, a personal digital assistant (PDA), or a notebook computer.

According to an embodiment, the user terminal 100 may receive a user's speech as a user input. The user terminal 100 may receive a user's speech and generate a command for operating an app based on the user's speech. Accordingly, the user terminal 100 may operate the app using the command.

Intelligent server 200 may receive a user voice input (voice input) from the user terminal 100 through a communication network and change it to text data (text data). In another embodiment, the intelligent server 200 may generate (or select) a path rule based on the text data. The pass rule may include information on an action (or operation) for performing a function of the app or information on parameters required to execute the action. Also, the pass rule may include the order of the operation of the app. The user terminal 100 may receive the pass rule, select an app according to the pass rule, and execute an operation included in the pass rule in the selected app.

The term "path rule" in this document generally means a sequence of states for an electronic device to perform a task requested by a user, but is not limited thereto. In other words, a pass rule may include information about a sequence of states. The task may be, for example, any action that an intelligent app can provide. The task may include creating a schedule, sending a photo to a desired partner, or providing weather information. The user terminal 100 may perform the above task by sequentially having at least one state (eg, an operating state of the user terminal 100).

According to one embodiment, the pass rule may be provided or generated by an artificial intelligent (AI) system. The artificial intelligence system may be a rule-based system, a neural network-based system (e.g., a feedforward neural network (FNN)), a recurrent neural network (RNN) ))) may be. or a combination of the foregoing or other artificially intelligent systems. According to an embodiment, the pass rule may be selected from a set of predefined pass rules or generated in real time in response to a user request. For example, the artificial intelligence system may select at least one pass rule from among a plurality of predefined pass rules or dynamically (or in real time) generate a pass rule. Also, the user terminal 100 may use a hybrid system to provide pass rules.

According to an embodiment, the user terminal 100 may execute the operation and display a screen corresponding to the state of the user terminal 100 having executed the operation on the display. For another example, the user terminal 100 may execute the operation and may not display the operation result on the display. The user terminal 100 may, for example, execute a plurality of operations and display only some results of the plurality of operations on a display. The user terminal 100 may display, for example, only the result of executing the last sequence of operations on the display. For another example, the user terminal 100 may receive a user's input and display a result of executing the operation on a display.

The personalized information server 300 may include a database in which user information is stored. For example, the personalized information server 300 may receive user information (eg, context information, app execution, etc.) from the user terminal 100 and store it in the database. The intelligent server 200 may use the user information when receiving the user information from the personalized information server 300 through a communication network and generating a pass rule for user input. According to an embodiment, the user terminal 100 may receive user information from the personalized information server 300 through a communication network and use it as information for managing a database.

The suggestion server 400 may include a database in which information about functions or applications to be introduced or provided in the terminal is stored. For example, the suggestion server 400 may receive user information of the user terminal 100 from the personalization information server 300 and may include a database in which information on functions usable by the user is stored. The user terminal 100 may receive information on the function to be provided from the suggestion server 400 through a communication network and provide the received information to the user.

Figure 2 is a block diagram showing a user terminal of the integrated intelligent system according to an embodiment of the present invention.

Referring to FIG. 2 , the user terminal 100 may include an input module 110, a display 120, a speaker 130, a memory 140, or a processor 150. The user terminal 100 may further include a housing, and components of the user terminal 100 may be seated inside the housing or located on the housing. The user terminal 100 may further include a communication circuit located inside the housing. The user terminal 100 may transmit and receive data (or information) with an external server (eg, the intelligent server 200) through the communication circuit.

According to an embodiment, the input module 110 may receive a user input from a user. For example, the input module 110 may receive a user input from a connected external device (eg, a keyboard or a headset). For another example, the input module 110 may include a touch screen (eg, a touch screen display) coupled to the display 120 . For another example, the input module 110 may include a hardware key (or physical key) located in the user terminal 100 (or a housing of the user terminal 100).

According to an embodiment, the input module 110 may include a microphone capable of receiving a user's speech as a voice signal. For example, the input module 110 may include a speech input system and receive a user's speech as a voice signal through the speech input system. The microphone may be exposed through, for example, a part (eg, a first part) of the housing.

According to an embodiment, the display 120 may display an image or video, and/or an execution screen of an application. For example, the display 120 may display a graphical user interface (GUI) of an app. According to one embodiment, the display 120 may be exposed through a part (eg, the second part) of the housing.

According to one embodiment, the speaker 130 may output a voice signal. For example, the speaker 130 may output a voice signal generated inside the user terminal 100 to the outside. According to one embodiment, the speaker 130 may be exposed through a part (eg, a third part) of the housing.

According to an embodiment, the memory 140 may store a plurality of apps (or application programs) 141 and 143 . The plurality of apps 141 and 143 may be programs for performing functions corresponding to user input, for example. According to one embodiment, the memory 140 may store the intelligent agent 145, the execution manager module 147, or the intelligent service module 149. The intelligent agent 145, the execution manager module 147, and the intelligent service module 149 are, for example, a framework (or application framework) for processing received user input (eg, user utterance). (application framework)).

According to one embodiment, the memory 140 may include a database capable of storing information necessary for recognizing a user input. For example, the memory 140 may include a log database capable of storing log information. For another example, the memory 140 may include a persona database capable of storing user information.

According to an embodiment, the memory 140 stores a plurality of apps 141 and 143, and the plurality of apps 141 and 143 may be loaded and operated. For example, a plurality of apps 141 and 143 stored in the memory 140 may be loaded and operated by the execution manager module 147 . The plurality of apps 141 and 143 may include execution service modules 141a and 143a that perform functions. In one embodiment, the plurality of apps 141 and 143 may execute a plurality of operations (eg, a sequence of states) 141b and 143b through the execution service modules 141a and 143a to perform functions. In other words, the execution service modules 141a and 143a are activated by the execution manager module 147 and may execute a plurality of operations 141b and 143b.

According to an embodiment, when the operations 141b and 143b of the apps 141 and 143 are executed, an execution state screen according to the execution of the operations 141b and 143b may be displayed on the display 120 . The execution state screen may be, for example, a screen in a state in which operations 141b and 143b have been completed. The execution state screen may be, for example, a partial landing screen in which execution of the operations 141b and 143b is stopped (eg, when parameters required for the operations 141b and 143b are not input). .

According to an embodiment, the execution service modules 141a and 143a may execute operations 141b and 143b according to a pass rule. For example, the execution service modules 141a and 143a are activated by the execution manager module 147, receive an execution request from the execution manager module 147 according to the pass rule, and operate according to the execution request (141b). , 143b), the functions of the apps 141 and 143 can be executed. The execution service modules 141a and 143a may transfer completion information to the execution manager module 147 when the execution of the operations 141b and 143b is completed.

According to an embodiment, when a plurality of operations 141b and 143b are executed in the apps 141 and 143, the plurality of operations 141b and 143b may be sequentially executed. When execution of one operation (eg, operation 1 of the first app 141 and operation 1 of the second app 143) is completed, the execution service modules 141a and 143a perform the next operation (eg, operation 1 of the first app 141). Operation 2 of ) and operation 2) of the second app 143 may be opened and completion information may be transmitted to the execution manager module 147 . Here, opening an arbitrary operation may be understood as transitioning an arbitrary operation to an executable state or preparing for execution of an arbitrary operation. In other words, if any operation is not open, that operation cannot be executed. When the completion information is received, the execution manager module 147 may transmit an execution request for the next operation (eg, operation 2 of the first app 141 and operation 2 of the second app 143) to the execution service module. . According to an embodiment, when a plurality of apps 141 and 143 are executed, the plurality of apps 141 and 143 may be sequentially executed. For example, when execution of the last operation of the first app 141 (eg, operation 3 of the first app 141) is completed and completion information is received, the execution manager module 147 executes the second app 143. An execution request of the first operation (eg, operation 1 of the second app 143) of may be transmitted to the execution service 143a.

According to an embodiment, when a plurality of operations 141b and 143b are executed in the apps 141 and 143, a result screen according to each execution of the plurality of operations 141b and 143b is displayed on the display 120. can be displayed on According to an embodiment, only some of the plurality of result screens according to the execution of the plurality of operations 141b and 143b may be displayed on the display 120 .

According to one embodiment, the memory 140 may store an intelligent app (eg, a voice recognition app) linked with the intelligent agent 145 . An app linked with the intelligent agent 145 may receive and process the user's speech as a voice signal. According to an embodiment, the app interlocked with the intelligent agent 145 is operated by a specific input (eg, an input through a hardware key, an input through a touch screen, a specific voice input) input through the input module 110. can

According to one embodiment, the intelligent agent 145, the execution manager module 147, or the intelligent service module 149 stored in the memory 140 may be executed by the processor 150. Functions of the intelligent agent 145 , the execution manager module 147 , or the intelligent service module 149 may be implemented by the processor 150 . The functions of the intelligent agent 145, the execution manager module 147, and the intelligent service module 149 can be described as the operation of the processor 150. According to an embodiment, the intelligent agent 145, the execution manager module 147, or the intelligent service module 149 stored in the memory 140 may be implemented as hardware as well as software.

According to an embodiment, the processor 150 may control overall operations of the user terminal 100 . For example, the processor 150 may control the input module 110 to receive a user input. The processor 150 may control the display 120 to display an image. The processor 150 may control the speaker 130 to output a voice signal. The processor 150 may control the memory 140 to execute a program and to load or store necessary information.

According to an embodiment, the processor 150 may execute the intelligent agent 145, the execution manager module 147, or the intelligent service module 149 stored in the memory 140. Accordingly, the processor 150 may implement functions of the intelligent agent 145, the execution manager module 147, or the intelligent service module 149.

According to an embodiment, the processor 150 may execute the intelligent agent 145 to generate a command for operating an app based on a voice signal received as a user input. According to an embodiment, the processor 150 may execute the execution manager module 147 to execute the apps 141 and 143 stored in the memory 140 according to the generated command. According to an embodiment, the processor 150 may execute the intelligent service module 149 to manage user information and process user input using the user information.

The processor 150 may execute the intelligent agent 145 to transmit the user input received through the input module 110 to the intelligent server 200 and process the user input through the intelligent server 200.

According to an embodiment, the processor 150 may pre-process the user input before executing the intelligent agent 145 to transmit the user input to the intelligent server 200 . According to an embodiment, the intelligent agent 145 may use an adaptive echo canceller (AEC) module, a noise suppression (NS) module, and end-point detection in order to pre-process the user input. detection) (EPD) module or automatic gain control (AGC) module. The adaptive echo cancellation unit may cancel an echo included in the user input. The noise suppression module may suppress background noise included in the user input. The endpoint detection module may detect an endpoint of the user's voice included in the user input, and find a portion of the user's voice using the detected endpoint. The automatic gain control module may recognize the user input and adjust the volume of the user input to be suitable for processing the recognized user input. According to one embodiment, the processor 150 may execute all of the preprocessing components for performance, but in another embodiment, the processor 150 may execute some of the preprocessing components to operate with low power.

According to one embodiment, the intelligent agent 145 may execute a wake up recognition module stored in the memory 140 to recognize a user's call. Accordingly, the processor 150 may recognize a user's wake-up command through the wake-up recognition module, and upon receiving the wake-up command, the processor 150 may execute the intelligent agent 145 for receiving a user input. . The wake-up recognition module may be implemented with a low-power processor (eg, a processor included in an audio codec). According to one embodiment, the processor 150 may execute the intelligent agent 145 when receiving a user input through a hardware key. When the intelligent agent 145 is executed, an intelligent app (eg, a voice recognition app) linked with the intelligent agent 145 may be executed.

According to one embodiment, intelligent agent 145 may include a voice recognition module for executing user input. The processor 150 may recognize a user input for executing an operation in the app through the voice recognition module. For example, the processor 150 may perform a limited user (voice) input (e.g., a shooting operation when a camera app is running) to execute an operation such as the wake-up command in the apps 141 and 143 through the voice recognition module. utterances such as "click" to be executed, etc.) can be recognized. The processor 150 assists the intelligent server 200 to recognize and quickly process a user command that can be processed in the user terminal 100 through the voice recognition module. According to one embodiment, the voice recognition module of the intelligent agent 145 for executing user input may be implemented in an app processor.

According to an embodiment, the voice recognition module of the intelligent agent 145 (including the voice recognition module of the wake-up module) may recognize a user input using a voice recognition algorithm. The algorithm used to recognize the voice may be, for example, at least one of a hidden markov model (HMM) algorithm, an artificial neural network (ANN) algorithm, and a dynamic time warping (DTW) algorithm.

According to an embodiment, the processor 150 may execute the intelligent agent 145 to convert the user's voice input into text data. For example, the processor 150 may transmit the user's voice to the intelligent server 200 through the intelligent agent 145, and receive text data corresponding to the user's voice from the intelligent server 200. Accordingly, the processor 150 may display the converted text data on the display 120 .

According to an embodiment, the processor 150 may receive a pass rule from the intelligent server 200 by executing the intelligent agent 145 . According to an embodiment, the processor 150 may transfer the pass rule to the execution manager module 147 through the intelligent agent 145 .

According to an embodiment, the processor 150 executes the intelligent agent 145 and transfers the execution result log according to the pass rule received from the intelligent server 200 to the intelligence service module 149 The delivered execution result log may be accumulated and managed in the user's preference information of the persona manager 149b.

According to an embodiment, the processor 150 executes the execution manager module 147 to receive a pass rule from the intelligent agent 145 and executes the apps 141 and 143, and the apps 141 and 143 execute the pass rule. Operations 141b and 143b included in the rule may be executed. For example, the processor 150 may transmit command information (eg, pass rule information) for executing the operations 141b and 143b to the apps 141 and 143 through the execution manager module 147. Completion information of the operations 141b and 143b may be received from the apps 141 and 143 .

According to an embodiment, the processor 150 executes the execution manager module 147 to execute operations 141b and 143b of the apps 141 and 143 between the intelligent agent 145 and the apps 141 and 143 command information (for example, pass rule information) may be transmitted. The processor 150 binds the apps 141 and 143 to be executed according to the pass rule through the execution manager module 147, and command information of the operations 141b and 143b included in the pass rule (eg: pass rule information) may be transmitted to the apps 141 and 143 . For example, the processor 150 sequentially transfers the operations 141b and 143b included in the pass rule to the apps 141 and 143 through the execution manager module 147, so that the operations of the apps 141 and 143 (141b, 143b) can be sequentially executed according to the pass rule.

According to an embodiment, the processor 150 may execute the execution manager module 147 to manage execution states of the operations 141b and 143b of the apps 141 and 143 . For example, the processor 150 may receive information about the execution state of the operations 141b and 143b from the apps 141 and 143 through the execution manager module 147 . When the execution state of the operations 141b and 143b is, for example, a partial landing (eg, when parameters required for the operations 141b and 143b are not input), the processor 150 executes Information on the stopped state may be delivered to the intelligent agent 145 through the manager module 147 . The processor 150 may request input of necessary information (eg, parameter information) from the user by using the information received through the intelligent agent 145 . For another example, when the execution state of the operations 141b and 143b is an operation state, the processor 150 may receive an utterance from the user through the intelligent agent 145 . The processor 150 may transmit information about the running apps 141 and 143 and the execution state of the apps 141 and 143 to the intelligent agent 145 through the execution manager module 147 . The processor 150 may transmit the user utterance to the intelligent server 200 through the intelligent agent 145 . The processor 150 may receive parameter information of the user's speech from the intelligent server 200 through the intelligent agent 145 . The processor 150 may transfer the received parameter information to the execution manager module 147 through the intelligent agent 145 . The execution manager module 147 may change the parameters of operations 141b and 143b into new parameters by using the received parameter information.

According to an embodiment, the processor 150 may execute the execution manager module 147 and deliver parameter information included in the pass rule to the apps 141 and 143 . When the plurality of apps 141 and 143 are sequentially executed according to the pass rule, the execution manager module 147 may transfer parameter information included in the pass rule from one app to another app.

According to an embodiment, the processor 150 may receive a plurality of pass rules by executing the execution manager module 147 . The processor 150 may receive a plurality of pass rules based on the user's speech through the execution manager module 147 . For example, the processor 150 through the execution manager module 147 specifies some apps 141 to execute some actions 141b through the user's utterance, but selects another app 143 to execute the remaining actions 143b. If not specified, the same app 141 (e.g. gallery app) that will run some actions 141b will be launched and different apps 143 (e.g. message app, telegram app) that can run the rest of the actions 143b. ) may receive a plurality of different pass rules, each of which is executed. The processor 150 may execute, for example, the same operations 141b and 143b of the plurality of pass rules (eg, consecutive identical operations 141b and 143b) through the execution manager module 147. When the same operation is executed, the processor 150 displays a status screen for selecting different apps 141 and 143 respectively included in the plurality of pass rules through the execution manager module 147 on the display 120. can be displayed

According to an embodiment, the intelligent service module 149 may include a context module 149a, a persona module 149b, or a suggestion module 149c.

The processor 150 may execute the context module 149a to collect the current state of the apps 141 and 143 from the apps 141 and 143 . For example, the processor 150 executes the context module 149a to receive context information indicating the current state of the apps 141 and 143, and the current state of the apps 141 and 143 through the received context information. can be collected.

The processor 150 may execute the persona module 149b to manage personal information of a user using the user terminal 100 . For example, the processor 150 executes the persona module 149b to collect usage information and performance results of the user terminal 100, and uses the collected usage information and performance results of the user terminal 100 to user can manage your personal information.

The processor 150 may predict the user's intention by executing the suggestion module 149c and recommend a command to the user based on the user's intention. For example, the processor 150 may recommend a command to the user according to the user's current state (eg, time, place, situation, app) by executing the suggestion module 149c.

3 is a diagram illustrating execution of an intelligent app of a user terminal according to an embodiment of the present invention.

Referring to FIG. 3 , the user terminal 100 receives a user input and executes an intelligent app (eg, a voice recognition app) interlocked with the intelligent agent 145 .

According to one embodiment, the user terminal 100 may execute an intelligent app for recognizing voice through the hardware key 112 . For example, the user terminal 100 may display a user interface (UI) 121 of an intelligent app on the display 120 when a user input is received through the hardware key 112 . The user may, for example, touch the voice recognition button 121a on the UI 121 of the intelligent app to input voice 120b while the UI 121 of the intelligent app is displayed on the display 120. . For another example, the user may input voice 120b by continuously pressing the hardware key 112 to input voice 120b.

According to an embodiment, the user terminal 100 may execute an intelligent app for recognizing voice through the microphone 111 . For example, the user terminal 100 displays the UI 121 of the intelligent app on the display 120 when a designated voice (eg, wake up!) is input 120a through the microphone 111. can do.

4 is a diagram illustrating that a context module of an intelligent service module collects a current state according to an embodiment of the present invention.

Referring to FIG. 4 , when the processor 150 receives a context request from the intelligent agent 145 (①), it requests context information indicating the current state of the apps 141 and 143 through the context module 149a (②) )can do. According to an embodiment, the processor 150 may receive (③) the context information from the apps 141 and 143 through the context module 149a and transmit it to the intelligent agent 145 (④).

According to an embodiment, the processor 150 may receive a plurality of pieces of context information from the apps 141 and 143 through the context module 149a. The context information may be information about the most recently executed apps 141 and 143, for example. For another example, the context information may be information about a current state in the apps 141 and 143 (eg, information about a corresponding photo when viewing a photo in a gallery).

According to an embodiment, the processor 150 may receive context information indicating the current state of the user terminal 100 from the device platform as well as the apps 141 and 143 through the context module 149a. there is. The context information may include general context information, user context information, or device context information.

The general context information may include general information of the user terminal 100 . The general context information may be checked through an internal algorithm by receiving data through a sensor hub of the device platform. For example, the general context information may include information about the current time and space. The information on the current time and space may include, for example, current time or information on the current location of the user terminal 100 . The current time may be checked through time on the user terminal 100, and information on the current location may be checked through a global positioning system (GPS). For another example, the general context information may include information about physical motion. The information about the physical movement may include, for example, information about walking, running, driving, and the like. The physical movement information may be checked through a motion sensor. The driving information can not only check driving through the motion sensor, but also check boarding and parking by detecting a Bluetooth connection in the vehicle. For another example, the general context information may include user activity information. The user activity information may include, for example, commuting, shopping, and travel information. The user activity information may be checked using information about a place registered in a database by a user or an app.

The user context information may include information about a user. For example, the user context information may include information about a user's emotional state. The information on the emotional state may include, for example, information about the user's happiness, sadness, anger, and the like. For another example, the user context information may include information about a user's current state. The information on the current state may include, for example, information about interests, intentions, and the like (eg, shopping).

The device context information may include information about the state of the user terminal 100 . For example, the device context information may include information about a pass rule executed by the execution manager module 147 . For another example, the device context information may include battery information. Information about the battery may be checked through, for example, charging and discharging states of the battery. For another example, the device context information may include information about connected devices and networks. Information on the connected device may be checked through, for example, a communication interface to which the device is connected.

5 is a block diagram showing a proposal module of an intelligent service module according to an embodiment of the present invention.

Referring to FIG. 5, the suggestion module 149c includes a hint providing module 149c_1, a context hint generating module 149c_2, a condition checking module 149c_3, a condition model module 149c_4, a reuse hint generating module 149c_5, or an introduction. A hint generation module 149c_6 may be included.

According to an embodiment, the processor 150 may provide a hint to the user by executing the hint providing module 149c_1. For example, the processor 150 receives the hint generated from the context hint generation module 149c_2, the reuse hint generation module 149c_5, or the introduction hint generation module 149c_6 through the hint providing module 149c_1, and provides the hint to the user. can provide.

According to an embodiment, the processor 150 may generate a hint that may be recommended according to the current state by executing the condition checking module 149c_3 or the condition model module 149c_4. The processor 150 may execute the condition checking module 149c_3 to receive information corresponding to the current state, and execute the condition model module 149c_4 to set a condition model using the received information. can For example, the processor 150 executes the condition model module 149c_4 to determine the time, location, situation, app in use, etc. at the time of providing the hint to the user, and prioritizes hints that are highly likely to be used in the corresponding condition. can be provided to the user in order of highest.

According to an embodiment, the processor 150 may generate a hint that may be recommended according to the frequency of use by executing the reuse hint generating module 149c_5. For example, the processor 150 may generate a hint based on a user's usage pattern by executing the reuse hint generating module 149c_5.

According to an embodiment, the introduction hint generation module 149c_6 may generate a hint introducing a new function or a function frequently used by other users to the user. For example, the hint introducing the new function may include an introduction (eg, an operation method) of the intelligent agent 145 .

According to another embodiment, the context hint generation module 149c_2, the condition checking module 149c_3, the condition model module 149c_4, the reuse hint generation module 149c_5, or the introduction hint generation module 149c_6 of the suggestion module 149c It may be included in the personalized information server 300. For example, the processor 150 generates a context hint generating module 149c_2, a reuse hint generating module 149c_5, or an introduction hint of the personalized user information server 300 through the hint providing module 149c_1 of the suggestion module 149c. A hint may be received from module 149c_6 and the received hint may be provided to the user.

According to an embodiment, the user terminal 100 may provide a hint according to the following series of processes. For example, upon receiving a hint providing request from the intelligent agent 145, the processor 150 may transfer the hint generating request to the context hint generating module 149c_2 through the hint providing module 149c_1. Upon receiving the hint generation request, the processor 150 may receive information corresponding to the current state from the context module 149a and the persona module 149b through the condition checking module 149c_3. The processor 150 transfers the received information through the condition checking module 149c_3 to the condition model module 149c_4, and uses the information through the condition model module 149c_4 to determine the condition among the hints provided to the user. You can prioritize hints in order of availability. The processor 150 may check the condition through the context hint generation module 149c_2 and generate a hint corresponding to the current state. The processor 150 may transfer the hint generated through the context hint generating module 149c_2 to the hint providing module 149c_1. The processor 150 may arrange the hints according to a rule specified through the hint providing module 149c_1 and deliver the hints to the intelligent agent 145 .

According to an embodiment, the processor 150 may generate a plurality of context hints through the hint providing module 149c_1 and prioritize the plurality of context hints according to a specified rule. According to an embodiment, the processor 150 may first provide a higher priority among the plurality of context hints to the user through the hint providing module 149c_1.

According to an embodiment, the user terminal 100 may suggest a hint according to the frequency of use. For example, upon receiving a hint provision request from the intelligent agent 145, the processor 150 may transmit the hint generation request to the reuse hint generation module 149c_5 through the hint provision module 149c_1. Upon receiving the hint generation request, the processor 150 may receive user information from the persona module 149b through the reuse hint generation module 149c_5. For example, the processor 150 generates a pass rule included in the user preference information of the persona module 149b, a parameter included in the pass rule, an execution frequency of the app, and the app used through the reuse hint generation module 149c_5. You can receive spatio-temporal information. The processor 150 may generate a hint corresponding to the received user information through the reuse hint generation module 149c_5. The processor 150 may transfer the hint generated through the reuse hint generating module 149c_5 to the hint providing module 149c_1. The processor 150 may sort the hints through the hint providing module 149c_1 and deliver the hints to the intelligent agent 145 .

According to an embodiment, the user terminal 100 may suggest a hint for a new function. For example, upon receiving a hint provision request from the intelligent agent 145, the processor 150 may transmit the hint generation request to the introduction hint generation module 149c_6 through the hint provision module 149c_1. The processor 150 may receive information about a function to be introduced from the suggestion server 400 by transmitting an introduction hint provision request from the suggestion server 400 through the introduction hint generation module 149c_6. The suggestion server 400 may store, for example, information on a function to be introduced, and a hint list for the function to be introduced may be updated by a service operator. The processor 150 may transfer the generated hint to the hint providing module 149c_1 through the introduction hint generation module 149c_6. The processor 150 may sort the hints through the hint providing module 149c_1 and transmit the hints to the intelligent agent 145 .

Accordingly, the processor 150 may provide the hint generated by the context hint generation module 149c_2, the reuse hint generation module 149c_5, or the introduction hint generation module 149c_6 to the user through the suggestion module 149c. For example, the processor 150 may display the generated hint through the suggestion module 149c on an app that operates the intelligent agent 145, and receive an input for selecting the hint from a user through the app. can do.

6 is a block diagram showing an intelligent server of an integrated intelligent system according to an embodiment of the present invention.

Referring to FIG. 6 , the intelligent server 200 includes an automatic speech recognition (ASR) module 210, a natural language understanding (NLU) module 220, and a path planner module. 230, a dialogue manager (DM) module 240, a natural language generator (NLG) module 250 or a text to speech (TTS) module 260. can According to one embodiment, the intelligent server 200 may include a communication circuit, a memory and a processor. The processor executes commands stored in the memory to perform automatic voice recognition module 210, natural language understanding module 220, pass planner module 230, dialogue manager module 240, natural language generation module 250, and text-to-speech conversion. The module 260 can be driven. The intelligent server 200 may transmit and receive data (or information) with an external electronic device (eg, the user terminal 100) through the communication circuit.

The natural language understanding module 220 or the path planner module 230 of the intelligent server 200 may generate a path rule.

According to an embodiment, the automatic voice recognition module 210 may convert a user input (eg, voice data) received from the user terminal 100 into text data. For example, the automatic speech recognition module 210 may include a speech recognition module. The speech recognition module may include an acoustic model and a language model. For example, the acoustic model may include vocalization-related information, and the language model may include unit phoneme information and information about a combination of unit phoneme information. The speech recognition module may convert user speech into text data using information related to speech and information on unit phoneme information. Information on the acoustic model and the language model may be stored in, for example, an automatic speech recognition database (ASR DB) 211 .

According to an embodiment, the natural language understanding module 220 may determine user intention by performing syntactic analysis or semantic analysis. The grammatical analysis may divide the user input into grammatical units (eg, words, phrases, morphemes, etc.) and determine which grammatical elements the divided units have. The semantic analysis may be performed using semantic matching, rule matching, formula matching, and the like. Accordingly, the natural language understanding module 220 may obtain a domain, an intention, or a parameter (or slot) required to express the intention.

According to one embodiment, the natural language understanding module 220 uses a matching rule divided into domains, intents, and parameters (or slots) necessary to identify the intents, and uses the user The intent and parameters of can be determined. For example, the one domain (eg, alarm) may include a plurality of intents (eg, alarm setting, alarm release, etc.), and one intent may include a plurality of parameters (eg, time, number of repetitions, alarm sound). etc.) may be included. A plurality of rules may include, for example, one or more mandatory element parameters. The matching rules may be stored in a natural language understanding database (NLU DB) 221 .

According to an embodiment, the natural language understanding module 220 determines the meaning of a word extracted from a user input using linguistic features (eg, grammatical elements) such as morphemes and phrases, and assigns the meaning of the identified word to a domain. And the user's intention may be determined by matching with the intention. For example, the natural language understanding module 220 may determine the user intent by calculating how many words extracted from the user input are included in each domain and intent. According to an embodiment, the natural language understanding module 220 may determine a parameter of the user input using a word based on determining the intention. According to an embodiment, the natural language understanding module 220 may determine the user's intention by using the natural language recognition database 221 in which linguistic features for identifying the intention of the user input are stored. According to another embodiment, the natural language understanding module 220 may determine the user's intention using a personal language model (PLM). For example, the natural language understanding module 220 may determine the user's intention using personalized information (eg, a contact list, a music list). The personalized language model may be stored in the natural language recognition database 221, for example. According to an embodiment, the automatic speech recognition module 210 as well as the natural language understanding module 220 may recognize the user's voice by referring to the personalized language model stored in the natural language recognition database 221 .

According to an embodiment, the natural language understanding module 220 may generate a pass rule based on the intent and parameters of the user input. For example, the natural language understanding module 220 may select an app to be executed based on the intention of the user input, and determine an operation to be performed in the selected app. The natural language understanding module 220 may generate a pass rule by determining a parameter corresponding to the determined operation. According to one embodiment, the pass rule generated by the natural language understanding module 220 includes information about an app to be executed, an operation to be executed in the app (eg, at least one state), and parameters required to execute the operation. can include

According to an embodiment, the natural language understanding module 220 may generate one pass rule or a plurality of pass rules based on the intent and parameters of the user input. For example, the natural language understanding module 220 receives a pass rule set corresponding to the user terminal 100 from the pass planner module 230, maps the intent and parameters of the user input to the received pass rule set, and passes rules can be determined.

According to another embodiment, the natural language understanding module 220 determines an app to be executed, an action to be executed in the app, and a parameter required to execute the action based on the intention and parameters of the user input, and determines one pass rule or a plurality of passes. Rules can be created. For example, the natural language understanding module 220 converts the app to be executed and the operation to be executed in the app into an ontology or a graph model according to the intention of the user input using information of the user terminal 100. You can create pass rules by arranging them. The generated path rule may be stored in a path rule database (PR DB) 231 through the path planner module 230, for example. The generated pass rule may be added to a pass rule set of the pass rule database 231 .

According to an embodiment, the natural language understanding module 220 may select at least one pass rule from among a plurality of generated pass rules. For example, the natural language understanding module 220 may select an optimal pass rule from among the plurality of pass rules. For another example, the natural language understanding module 220 may select a plurality of pass rules when only some actions are specified based on the user's speech. The natural language understanding module 220 may determine one pass rule among the plurality of pass rules according to a user's additional input.

According to an embodiment, the natural language understanding module 220 may transmit a pass rule to the user terminal 100 as a request for user input. For example, the natural language understanding module 220 may transmit one pass rule corresponding to the user input to the user terminal 100 . For another example, the natural language understanding module 220 may transmit a plurality of pass rules corresponding to the user input to the user terminal 100 . The plurality of pass rules may be generated by the natural language understanding module 220 when, for example, only some actions are specified based on user speech.

According to an embodiment, the pass planner module 230 may select at least one pass rule from among a plurality of pass rules.

According to an embodiment, the pass planner module 230 may transmit a pass rule set including a plurality of pass rules to the natural language understanding module 220 . A plurality of pass rules of the pass rule set may be stored in a table form in a pass rule database 231 connected to the pass planner module 230 . For example, the pass planner module 230 transmits a pass rule set corresponding to information (eg, OS information, app information) of the user terminal 100 received from the intelligent agent 145 to the natural language understanding module 220. can A table stored in the pass rule database 231 may be stored for each domain or version of the domain, for example.

According to an embodiment, the pass planner module 230 may select one pass rule or a plurality of pass rules from the pass rule set and transmit the selected pass rule to the natural language understanding module 220 . For example, the pass planner module 230 matches the user's intention and parameters to the pass rule set corresponding to the user terminal 100, selects one pass rule or a plurality of pass rules, and uses the natural language understanding module 220 can be forwarded to

According to an embodiment, the pass planner module 230 may generate one pass rule or a plurality of pass rules using user intention and parameters. For example, the pass planner module 230 may generate one pass rule or a plurality of pass rules by determining an app to be executed and an operation to be executed in the app based on user intention and parameters. According to an embodiment, the pass planner module 230 may store the generated pass rule in the pass rule database 231 .

According to an embodiment, the pass planner module 230 may store the pass rule generated by the natural language understanding module 220 in the pass rule database 231 . The generated pass rule may be added to a pass rule set stored in the pass rule database 231 .

According to an embodiment, a table stored in the pass rule database 231 may include a plurality of pass rules or a plurality of pass rule sets. The plurality of pass rules or the plurality of pass rule sets may reflect the type, version, type, or characteristic of a device that performs each pass rule.

According to an embodiment, the dialog manager module 240 may determine whether the user's intention identified by the natural language understanding module 220 is clear. For example, the dialog manager module 240 may determine whether the user's intention is clear based on whether the parameter information is sufficient. The dialogue manager module 240 may determine whether the parameters identified by the natural language understanding module 220 are sufficient to perform the task. According to an embodiment, the dialog manager module 240 may perform feedback requesting necessary information from the user when the user's intention is not clear. For example, the dialog manager module 240 may perform feedback requesting information on parameters for determining the user's intention.

According to one embodiment, the dialog manager module 240 may include a content provider module. When the content providing module can perform an operation based on the intention and parameter identified by the natural language understanding module 220, a result of performing a task corresponding to a user input may be generated. According to an embodiment, the dialog manager module 240 may transmit the result generated by the content providing module to the user terminal 100 in response to a user input.

According to one embodiment, the natural language generation module 250 may change designated information into a text form. The information changed to the text form may be in the form of natural language speech. The designated information may be, for example, information for additional input, information for guiding completion of an operation corresponding to the user input, or information for guiding additional user input (eg, feedback information for user input). The information changed to the text form may be transmitted to the user terminal 100 and displayed on the display 120 or may be transmitted to the text-to-speech conversion module 260 and changed to a voice form.

According to an embodiment, the text-to-speech conversion module 260 may change text-type information into voice-type information. The text-to-speech conversion module 260 may receive text-type information from the natural language generation module 250, convert the text-type information into voice-type information, and transmit the information to the user terminal 100. The user terminal 100 may output the voice-type information through the speaker 130 .

According to an embodiment, the natural language understanding module 220, the path planner module 230, and the dialog manager module 240 may be implemented as one module. For example, the natural language understanding module 220, the path planner module 230, and the dialog manager module 240 are implemented as one module to determine the user's intention and parameters, and the determined user's intention and parameters correspond to the You can generate responses (e.g. pass rules). Accordingly, the generated response may be transmitted to the user terminal 100 .

7 is a diagram illustrating a method of generating a path rule of a path planner module according to an embodiment of the present invention.

Referring to FIG. 7 , according to an embodiment, the natural language understanding module 220 may classify the app's function into any one operation (eg, state A to state F) and store it in the pass rule database 231. For example, the natural language understanding module 220 may use a plurality of pass rules (eg, A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3-D-E-F) may be stored in the pass rule database 231.

According to an embodiment, the pass rule database 231 of the pass planner module 230 may store pass rule sets for performing functions of an app. The pass rule set may include a plurality of pass rules including a plurality of actions (eg, a sequence of states). In the plurality of pass rules, operations executed according to parameters input to each of the plurality of operations may be sequentially arranged. According to an embodiment, the plurality of pass rules may be configured in the form of an ontology or a graph model and stored in the pass rule database 231 .

According to an embodiment, the natural language understanding module 220 may use the plurality of pass rules (eg, A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3-D-F, Among A-B1-C3-D-E-F), an optimal pass rule (eg, A-B1-C3-D-F) can be selected.

According to an embodiment, the natural language understanding module 220 may deliver a plurality of pass rules to the user terminal 100 when there is no pass rule that perfectly matches the user input. For example, the natural language understanding module 220 may select a pass rule (eg, A-B1) partially corresponding to the user input. The natural language understanding module 220 generates one or more pass rules (e.g. A-B1-C1, A-B1-C2, A-B1-C3) including pass rules (e.g. A-B1) partially corresponding to user input. -D-F, A-B1-C3-D-E-F) may be selected and transmitted to the user terminal 100.

According to an embodiment, the natural language understanding module 220 may select one of a plurality of pass rules based on the additional input of the user terminal 100 and deliver the selected one pass rule to the user terminal 100. . For example, the natural language understanding module 220 generates a plurality of pass rules (eg, A-B1-C1, A-B1) according to a user input additionally input from the user terminal 100 (eg, an input to select C3). -C2, A-B1-C3-D-F, A-B1-C3-D-E-F), one pass rule (eg, A-B1-C3-D-F) may be selected and transmitted to the user terminal 100.

According to another embodiment, the natural language understanding module 220 determines the user's intention corresponding to a user input (eg, an input to select C3) additionally input to the user terminal 100 through the natural language understanding module 220 and A parameter may be determined, and the determined user intention or parameter may be transmitted to the user terminal 100 . Based on the transmitted intent or the parameter, the user terminal 100 sets a plurality of pass rules (eg, A-B1-C1, A-B1-C2, A-B1-C3-D-F, A-B1-C3- D-E-F), one pass rule (eg, A-B1-C3-D-F) can be selected.

Accordingly, the user terminal 100 may complete the operation of the apps 141 and 143 according to the selected one pass rule.

According to an embodiment, the natural language understanding module 220 may generate a pass rule partially corresponding to the received user input when a user input lacking information is received by the intelligent server 200 . For example, natural language understanding module 220 may transmit the partially matched pass rule to intelligent agent 145 . The processor 150 may execute the intelligent agent 145 to receive the pass rule, and transmit the partially corresponding pass rule to the execution manager module 147 . The processor 150 may execute the first app 141 according to the pass rule through the execution manager module 147 . The processor 150 may transmit information about insufficient parameters to the intelligent agent 145 while executing the first app 141 through the execution manager module 147 . The processor 150 may request an additional input from the user through the intelligent agent 145 using the information on the insufficient parameter. When an additional input is received by the user through the intelligent agent 145, the processor 150 may transmit the user input to the intelligent server 200 for processing. The natural language understanding module 220 may create an added pass rule based on the intention and parameter information of the additionally input user input, and transmit it to the intelligent agent 145 . The processor 150 may transmit the pass rule to the execution manager module 147 through the intelligent agent 145 and execute the second app 143 according to the pass rule through the execution manager module 147 .

According to an embodiment, the natural language understanding module 220 may transmit a user information request to the personalized information server 300 when a user input missing some information is received by the intelligent server 200 . The personalized information server 300 may transmit information of a user inputting a user input stored in the persona database to the natural language understanding module 220 . The natural language understanding module 220 may select a pass rule corresponding to a user input in which some motions are omitted using the user information. Accordingly, even if a user input missing some information is received by the intelligent server 200, the natural language understanding module 220 receives an additional input by requesting the missing information or uses the user information to obtain a pass rule corresponding to the user input. can determine

Table 1 attached below may indicate an exemplary form of a pass rule related to a task requested by a user according to an embodiment.

Path ruleID State Parameter Gallery_101 PictureView(25) NULL SearchView(26) NULL SearchViewResult(27) Location,time SearchEmptySelectedView(28) NULL SearchSelectedView(29) ContentType,selectall CrossShare(30) anaphora

Referring to Table 1, a pass rule generated or selected by the intelligent server 200 according to a user's speech (eg, "Share a photo") has at least one state (25, 26, 27, 28, 29 or 30) may be included. For example, the at least one state (eg, any operating state of the terminal) includes a photo application execution (PicturesView) state 25, a photo search function execution (SearchView) state 26, and a search result display screen output (SearchViewResult ) state (27), output the search result display screen with no photo selected (SearchEmptySelectedView) state (28), output the search result display screen with at least one photo selected (SearchSelectedView) state (29), or output the shared application selection screen It may correspond to at least one of the (CrossShare) states 30.

In an embodiment, the parameter information of the pass rule may correspond to at least one state. For example, the at least one photo may be included in the selected search result display screen output state 29 .

As a result of executing the pass rule including the sequence of states 25, 26, 27, 28, and 29, a task requested by the user (eg, "Share the photo") may be performed.

8 is a diagram showing that a persona module of an intelligent service module manages user information according to an embodiment of the present invention.

Referring to FIG. 8 , the processor 150 may receive information of the user terminal 100 from the apps 141 and 143, the execution manager module 147, or the context module 149a through the persona module 149b. . The processor 150 may store information as a result of executing the actions 141b and 143b of the app through the apps 141 and 143 and the execution manager module 147 in the action log database. The processor 150 may store information about the current state of the user terminal 100 in a context database through the context module 149a. The processor 150 may receive the stored information from the action log database or the context database through the persona module 149b. Data stored in the action log database and the context database may be analyzed by, for example, an analysis engine and transmitted to the persona module 149b.

According to an embodiment, the processor 150 transmits information received from the apps 141 and 143, the execution manager module 147, or the context module 149a through the persona module 149b to the suggestion module 149c. can For example, the processor 150 may transmit data stored in the action log database or the context database to the suggestion module 149c through the persona module 149b.

According to an embodiment, the processor 150 transmits information received from the apps 141 and 143, the execution manager module 147, or the context module 149a to the personalized information server 300 through the persona module 149b. can do. For example, the processor 150 may periodically transmit data accumulated and stored in the action log database or the context database to the personalized information server 300 through the persona module 149b.

According to an embodiment, the processor 150 may transfer data stored in the action log database or the context database to the suggestion module 149c through the persona module 149b. User information generated through the persona module 149b may be stored in the persona database. The persona module 149b may periodically transmit user information stored in the persona database to the personalized information server 300 . According to an embodiment, information transmitted to the personalized information server 300 through the persona module 149b may be stored in a persona database. The personalized information server 300 may infer user information necessary for generating a pass rule of the intelligent server 200 using information stored in the persona database.

According to an embodiment, user information inferred using information transmitted through the persona module 149b may include profile information or preference information. The profile information or preference information may be inferred through the user's account and accumulated information.

The profile information may include user's personal information. For example, the profile information may include user demographic information. The demographic information may include, for example, the user's gender and age. For another example, the profile information may include life event information. The life event information may be inferred by comparing log information with a life event model, and reinforced by analyzing a behavior pattern, for example. For another example, the profile information may include interest information. The interest information may include, for example, shopping items of interest and field of interest (eg, sports, politics, etc.). As another example, the profile information may include activity area information. The activity area information may include, for example, information about a home, a place of work, and the like. The information on the activity area may include information on the location of the place as well as information about the area recorded in priority based on the cumulative stay time and the number of visits. For another example, the profile information may include activity time information. The activity time information may include, for example, information about wake-up time, commute time, sleep time, and the like. The information about the commuting time may be inferred using the activity area information (eg, information about home and workplace). The information on the sleep time may be inferred through unused time of the user terminal 100 .

The preference information may include user preference information. For example, the preference information may include information about app preference. The app preference may be inferred through, for example, app usage records (eg, usage records by time and location). The app preference may be used to determine an app to be executed according to the user's current state (eg, time, place). For another example, the preference information may include contact preference information. The contact preference may be inferred by analyzing, for example, contact frequency (eg, frequency of contact by time and location) of the contact. The contact preference may be used to determine a contact to be contacted according to the user's current status (eg, contact for duplicate names). For another example, the preference information may include setting information. The setting information may be inferred by analyzing, for example, information on the setting frequency of a specific setting value (eg, the frequency of setting a setting value by time or place). The setting information may be used to set a specific setting value according to the user's current state (eg, time, place, situation). As another example, the preference information may include place preference. The place preference may be inferred through, for example, a visit record of a specific place (eg, a visit record by time). The place preference may be used to determine the place to be visited according to the user's current state (eg, time). For another example, the preference information may include command preference. The command preference may be inferred through, for example, frequency of command use (eg, frequency of use by time or location). The command preference may be used to determine a command pattern to be used according to a user's current status (eg, time and place). In particular, the command preference may include information on a menu most frequently selected by the user in the current state of an app being executed by analyzing log information.

9 is a block diagram of an electronic device related to voice input processing according to an embodiment of the present invention. The electronic device 900 illustrated in FIG. 9 may include the same or similar configuration as the user terminal 100 in the above-described drawings.

According to an embodiment, when the electronic device 900 receives a user's voice input related to the execution of a task (or function) through the microphone 910, the electronic device 900 may check a voice password required to perform the task. In addition, the electronic device 900 may acquire surrounding information of the electronic device 900 and determine whether the surrounding information satisfies a specified condition. For example, the electronic device 900 determines whether a plurality of users exist around the electronic device 900, and among the plurality of users, a user who does not have permission to use the electronic device 900 (or an unauthenticated user) ) can be determined. When it is determined that there is a user who does not have permission to use the electronic device 900, the electronic device 900 may change password authentication settings necessary for performing the task. For example, the electronic device 900 may change the password authentication setting to a one-time voice password input method. As another example, the electronic device 900 may change the password authentication setting from a voice password input method to a password authentication method through another input means. The password authentication method through the other input means is, for example, a pin number input method through a key input device 993, a biometric password input method through a biometric sensor 994, or a pin number through a touch screen display 995. Alternatively, a pattern input method may be included.

According to an embodiment, the electronic device 900 may remind the user of a password authentication setting change required to perform the task. For example, the electronic device 900 may output information related to password authentication setting change through the display 995 . As another example, the electronic device 900 may output information related to password authentication setting change through the speaker 996 . As another example, the electronic device 900 may transmit information related to a password authentication setting change to an external electronic device connected through the communication circuit 930 . In this case, the external electronic device may output the information through a display and/or a speaker included in the external electronic device.

Referring to FIG. 9 , the electronic device 900 for performing the above functions may include a microphone 910, a communication circuit 930, a memory 950, and a processor 970. However, the configuration of the electronic device 900 is not limited thereto. According to various embodiments, the electronic device 900 may further include at least one other component in addition to the aforementioned components or may omit at least one of the aforementioned components. For example, the electronic device 900 may include a camera 991, a motion sensor 992, a key input device 993, a biosensor 994, a display 995, or a speaker 996.

The microphone 910 may receive the user's speech as a voice input. For example, the microphone 910 may convert vibration energy generated by a user's speech into an electrical signal and transmit the converted electrical signal to the processor 970 .

The communication circuit 930 may support establishing a communication channel between the electronic device 900 and an external electronic device and performing wired or wireless communication through the established communication channel.

The memory 950 may store commands or data related to at least one other element of the electronic device 900 . According to one embodiment, the memory 950 may store software and/or programs. For example, the memory 950 may store applications (eg, intelligent apps) related to artificial intelligence technology. The intelligent app obtains, for example, commands related to a function of receiving and processing a user's utterance as a voice input, commands related to a function of checking a voice password required to perform a task, and surrounding information of the electronic device 900, Commands related to a function of analyzing acquired surrounding information, commands related to a function of changing password authentication settings necessary for performing a task, commands related to a function of reminding a change in password authentication settings, or password authentication settings It may include commands related to a function of transmitting change-related information to an external electronic device. Also, instructions included in software and/or programs stored in the memory 950 may be executed by the processor 970 .

The processor 970 may execute calculations or data processing related to control and/or communication of at least one other component of the electronic device 900 . The processor 970 may control a plurality of hardware or software components connected to the processor 970 by driving, for example, an operating system or an application program, and may perform various data processing or calculations. The processor 970 may include one or more of a central processing unit, an application processor, or a communication processor. According to an embodiment, the processor 970 may be implemented as a system on chip (SoC).

According to an embodiment, when a voice input is received through the microphone 910, the processor 970 may convert voice data corresponding to the voice input into text data. In some embodiments, the processor 970 may transmit voice data to an external electronic device (eg, the intelligent server 200) through the communication circuit 930. In this case, the processor 970 may receive text data generated by converting the voice data from the external electronic device into a text format through the communication circuit 930 . Also, the processor 970 may analyze the text data and determine a task (or function) to be performed based on the analyzed result.

According to an embodiment, the processor 970 may check a password required to perform the task. For example, the processor 970 may check whether a password set to perform the task exists. The password may include, for example, a voice password set by an authenticated user before performing the task, a PIN number, a pattern, or a biometric password.

According to an embodiment, the processor 970 may acquire (or collect) surrounding information of the electronic device 900 when a password necessary for performing a task is set as a voice password. For example, the processor 970 may obtain surrounding information of the electronic device 900 through the microphone 910 . For example, the processor 970 may analyze a voice input received through the microphone 910 and determine whether a plurality of users exist around the electronic device 900 based on the analyzed information. As another example, the processor 970 may obtain surrounding information of the electronic device 900 through the camera 991 . For example, the processor 970 may analyze an image captured by the camera 991 and determine whether a plurality of users exist around the electronic device 900 based on the analyzed information. As another example, the processor 970 may obtain surrounding information of the electronic device 900 through the motion sensor 992 . For example, the processor 970 may determine whether a plurality of users exist around the electronic device 900 based on sensing information sensed by the motion sensor 992 .

According to an embodiment, when a plurality of users exist in the vicinity of the electronic device 900, the processor 970 does not have permission to use the electronic device 900 among the plurality of users (or is not authenticated). user) can be determined. For example, the processor 970 compares and analyzes the voice data of the authenticated user stored in the memory 950 and the voice data corresponding to the voice input received through the microphone 910 (eg, voiceprint recognition), It is possible to determine whether a user exists or not. As another example, the processor 970 may compare and analyze an image of an authenticated user stored in the memory 950 and an image taken by the camera 991 to determine whether an unauthenticated user exists.

According to an embodiment, when a plurality of users exist around the electronic device 900 and all of the plurality of users are authenticated users, the processor 970 configures password authentication settings necessary for performing a task using public voice. You can change the password input method. The public voice password may be, for example, a voice password set so that users (eg, family members) who have permission to use the electronic device 900 may use it.

According to an embodiment, the processor 970 configures password authentication required to perform a task when a plurality of users exist around the electronic device 900 and an unauthenticated user among the plurality of users exists. can be changed to the input method of the one-time voice password. The one-time voice password is used to prevent exposure of a voice password set by an authenticated user, and may be discarded once used. In some embodiments, the processor 970 may set the one-time voice password to be used a specified number of times or for a specified period instead of being discarded once used. For example, the processor 970 may configure the one-time voice password to be continuously used while an unauthenticated user is present.

According to an embodiment, the processor 970 may store a question leading to input of the one-time voice password and the one-time voice password (an answer to the question) in the memory 950 based on a user's input. For example, when an authenticated user sets the questions and answers to the questions in advance, the set information may be stored in the memory 950 . In this case, the processor 970 provides the question to the user before performing the task, and when the user utters an answer to the question, the processor 970 compares the answer with the answer stored in the memory 950 to provide the disposable voice. It can be determined whether the password has been entered correctly.

According to an embodiment, the processor 970 may set the one-time voice password using information related to the user of the electronic device 900 . For example, the processor 970 may process the user's personal information (eg, the user's name, age, gender, or region of residence), or the user's activity information (eg, schedule, alarm time, health information, content consumption information, or Traffic usage information, etc.) may be used to set the one-time voice password. In this case, the processor 970 may limit usable information according to privacy sensitivity. For example, the processor 970 may provide an interface allowing a user to directly select information used when setting a one-time voice password in order to prevent information that the user does not want from being exposed. In addition, the processor 970 uses information related to the user, even if the one-time voice password is set, when the user indicates that he or she does not want to answer a question leading to input of the one-time voice password (eg, the user clicks "next"). ), the one-time voice password may be reset using other information.

According to an embodiment, the processor 970 sets the password authentication required for performing the task when password authentication is possible through another input means instead of inputting a voice password through the microphone 910. You can change the authentication method. For example, if the electronic device 900 includes the key input device 993, the processor 970 may change the password authentication setting required to perform the task to a pin number input method through the key input device 993. there is. As another example, if the electronic device 900 includes the biometric sensor 994, the processor 970 may change the setting of password authentication required to perform the task to a biometric password input method through the biometric sensor 994. The biometric password input method may include, for example, fingerprint recognition through a fingerprint sensor, face recognition through a camera 991, iris recognition through an iris sensor, and the like. As another example, if the electronic device 900 includes the touch screen display 995, the processor 970 may change to a pin number input or pattern input method through the touch screen display 995.

According to an embodiment, when password authentication is possible through an external electronic device connected to the electronic device 900 through the communication circuit 930, the processor 970 sets the password authentication necessary for performing the task to the external electronic device. It can be changed to the password authentication method through . For example, when the external electronic device includes a key input device, a biometric sensor, or a touch screen display, the processor 970 performs password authentication through a key input device, bio sensor, or touch screen display included in the external electronic device. The external electronic device may be controlled to perform

According to an embodiment, even when password authentication is possible through other input means, in a situation where the user cannot use the other input means, the processor 970 configures password authentication settings necessary for performing the task to be previously set. It is possible to change from a voice password input method to another voice password input method, such as a public voice password or a one-time voice password. For example, the processor 970 may determine that the other input means cannot be used while the user is driving, and accordingly, set the password authentication required to perform the task by using a public voice password or a one-time voice password input method. can be changed As another example, the processor 970 may determine that the other input means cannot be used even when the user is cooking, and accordingly, password authentication required for performing the task is set as an input method of a public voice password or a one-time voice password. can be changed In this regard, the processor 970 may obtain location information of the user through GPS or sound information through the microphone 910 in order to determine whether the user is currently driving or cooking. .

According to an embodiment, the processor 970 may adjust a security level related to the execution of the task according to surrounding information of the electronic device 900 . For example, the processor 970 may differently set information accessible through a task, a level or level of information of a service provided by the task, and a degree of exposure of a service provided by the task according to peripheral information of the electronic device 900 . For example, when a plurality of users exist around the electronic device 900 and all of the plurality of users are users (eg, family members) who have permission to use the electronic device 900, the processor 970 The security level related to the execution of the task can be lowered. For example, the processor 970 may set all information to be accessible through a task, increase the level of a service provided by a task, and set all services provided by a task to be exposed. As another example, when a plurality of users exist around the electronic device 900 and there is a user who does not have permission to use the electronic device 900 among the plurality of users, the processor 970 performs a task and You can increase the relevant security level. For example, the processor 970 may limit information accessible through a task, lower the level of a service provided by a task, and set only some of the services provided by a task to be exposed. As another example, if the information accessible through the task is information with high privacy sensitivity, the processor 970 may restrict access to the information according to a user's setting.

According to an embodiment, the processor 970 may provide a reminder to the user of a password authentication setting change required to perform the task. For example, the processor 970 may output, on the display 995, information related to a change in password authentication settings. As another example, the processor 970 may output information related to changing password authentication settings through the speaker 996 . As another example, the processor 970 may transmit information related to changing password authentication settings to a connected external electronic device through the communication circuit 930 . In this case, the external electronic device may output the information through a display and/or a speaker included in the external electronic device.

The camera 991 may capture still images and moving images. According to one embodiment, the camera 991 may include one or more lenses, image sensors, image signal processors, or flashes.

The motion sensor 992 may recognize the movement and position of an object. According to an embodiment, the motion sensor 992 may detect the direction of the object, detect the movement of the object, and measure the moving speed of the object.

The key input device 993 may receive commands or data from the user. The key input device 993 may include, for example, a physical button, an optical key, or a keypad.

The biometric sensor 994 may recognize the user's body. For example, the biometric sensor 994 may include a fingerprint sensor, an iris sensor, a face recognition sensor, a heart rate sensor, a voice recognition sensor, or a blood vessel recognition sensor.

The fingerprint sensor may recognize a user's fingerprint in a line scan or area scan method. For example, the fingerprint sensor may acquire fingerprint information when a user swipes or touches a finger to a sensing area of the fingerprint sensor. The fingerprint sensor may use an optical, capacitive, or ultrasonic method. The optical fingerprint sensor irradiates light to a finger and collects a fingerprint image using reflected light. It is a method of collecting an image, and the ultrasonic fingerprint sensor emits ultrasonic waves to a finger and collects a fingerprint image using the reflected ultrasonic waves.

The iris sensor may recognize the user's iris using an iris camera (eg, an infrared camera) or the like. For example, the iris sensor may irradiate infrared light to the user's eyes and analyze the reflected infrared light to recognize the user's iris. For example, the iris sensor may detect the iris by finding a boundary between the iris and the pupil having a severe change in color or contrast around the pupil having the deepest color in the user's eye. In addition, the iris sensor may separate the iris region by detecting a boundary with the sclera surrounding the outside of the iris.

The face recognition sensor may recognize a user's face by analyzing a face image captured by the camera 991 . For example, the face recognition sensor may extract a face region from the image and determine whether the user's face matches. For example, the face recognition sensor may extract a face region using a difference in brightness or color or feature point extraction to distinguish a face from a background in the image. In addition, the face recognition sensor may recognize the user's face by comparing extracted data (eg, pixel values) of the face region with pre-stored face data of the user.

The heart rate sensor may measure the heart rate by radiating light to a finger using an LED and measuring an amount of reflected light according to an amount of blood flowing through the blood vessels of the finger. The heart rate sensor may calculate a heart rate by converting a change in reflected light amount into an electrical signal and analyzing a pattern in which the electrical signal is changed.

The voice recognition sensor may recognize a user's voice by analyzing a voice acquired through a microphone 910 or the like. For example, the voice recognition sensor may recognize the user's voice by removing noise components from the voice and extracting user-specific characteristics such as pronunciation and intonation.

The blood vessel recognition sensor may recognize a user's blood vessel by radiating light to a finger or the like using an LED and measuring reflected or transmitted light to determine a location or shape of a blood vessel (eg, vein).

The display 995 may display various contents (eg, text, image, video, icon, symbol, etc.) to the user. According to one embodiment, the display 995 may include a touch screen. For example, the display 995 may receive a touch, gesture, proximity, or hovering input using an electronic pen or a user's body part (eg, a finger).

The speaker 996 may output a voice signal. For example, the speaker 996 may output a voice signal generated inside the electronic device 900 to the outside.

As described above, according to various embodiments, an electronic device (eg, the electronic device 900) includes a microphone (eg, the microphone 910) and a communication circuit for communication with an external electronic device (eg, the communication circuit 930). )), a processor (eg, processor 970) electrically connected to the microphone and the communication circuit, and a memory (eg, memory 950) electrically connected to the processor, wherein the memory, when executed , The processor receives a first voice input related to performing a task through the microphone, checks a first voice code required for performing the task, obtains surrounding information of the electronic device, and When a specified condition is satisfied, instructions for changing password authentication settings necessary for performing the task and transmitting a signal related to the password authentication setting change to the external electronic device through the communication circuit may be stored.

According to various embodiments of the present disclosure, the memory may, when executed, cause the processor to receive at least one of the first voice input and the second voice input received through the microphone as at least part of an operation of obtaining the surrounding information. You can store instructions that cause analysis.

According to various embodiments of the present disclosure, the electronic device further includes a camera (eg, the camera 991), and the memory stores the camera as at least part of an operation in which the processor acquires the surrounding information when executed. It is possible to store instructions for analyzing the acquired image through the.

According to various embodiments, the electronic device further includes a motion sensor (eg, the motion sensor 992), and the memory, when executed, as at least part of an operation of obtaining the surrounding information by the processor, Instructions for analyzing sensing information sensed by the motion sensor may be stored.

According to various embodiments of the present disclosure, the memory may further store instructions that, when executed, cause the processor to determine that the specified condition is satisfied when the surrounding information includes information that a plurality of users exist.

According to various embodiments of the present disclosure, when the memory is executed, the processor includes information indicating that a plurality of users exist in the surrounding information and a user who does not have permission to use the electronic device among the plurality of users exists. If it is, instructions for determining that the specified condition is satisfied may be further stored.

According to various embodiments of the present disclosure, as at least part of an operation in which the processor changes the password authentication setting during execution, the memory sets the password authentication setting to the electronic device in the first voice password input method. Instructions for changing the input method of the second voice password that can be used by a plurality of users having permission to use may be stored.

According to various embodiments of the present disclosure, the memory, when executed, as at least part of an operation in which the processor changes the password authentication setting, sets the password authentication the number of times specified in the input method of the first voice password or Instructions for changing the input method of the second voice password that can be used for a specified period of time may be stored.

According to various embodiments of the present disclosure, the memory may further store instructions that, when executed, cause the processor to set the second voice password using information related to the user of the electronic device.

According to various embodiments, the information related to the user may include personal information of the user and activity information of the user.

According to various embodiments of the present disclosure, the electronic device further includes a key input device (eg, the key input device 993), and the memory, when executed, controls the operation of changing the password authentication setting by the processor. At least in part, instructions for changing the setting of the password authentication from an input method of the first voice password to a pin code input method through the key input device may be stored.

According to various embodiments of the present disclosure, the electronic device further includes a biometric sensor (eg, the biometric sensor 994), and the memory, when executed, performs at least a part of an operation of changing the password authentication setting by the processor. As a result, instructions for changing the setting of the password authentication from the first voice password input method to the biometric password input method through the biometric sensor may be stored.

According to various embodiments of the present disclosure, the electronic device further includes a touch screen display (eg, the display 995), and the memory stores at least part of an operation of changing the password authentication setting by the processor when executed. As a result, it is possible to store instructions for changing the password authentication setting from the first voice password input method to a pin code or pattern input method through the touch screen display.

According to various embodiments of the present disclosure, when the memory is executed, the external electronic device connected through the communication circuit may include a key input device, a biometric sensor, and a touch screen display, and if the external electronic device includes at least one of the key input device, the biometric sensor, and the touch screen display, setting the password authentication is determined by the first voice Instructions for changing a password input method to a password authentication method through the external electronic device may be stored.

According to various embodiments, the electronic device further includes at least one of a speaker (eg, the speaker 996) and a display (eg, the display 995), and the memory, when executed, the processor, Instructions for outputting information related to password authentication setting change through at least one of the speaker and the display may be further stored.

10 is a diagram illustrating a method of operating an electronic device related to voice input processing according to an embodiment of the present invention.

Referring to FIG. 10 , a processor (eg, processor 970) of an electronic device (eg, electronic device 900) may receive a voice input through a microphone (eg, microphone 910) in operation 1010. . In this case, the processor may convert voice data corresponding to the received voice input into text data. Alternatively, the processor transmits the voice data to an external electronic device through a communication circuit (eg, the communication circuit 930), converts the voice data into a text format from the external electronic device, and transmits text data generated by the communication to the communication circuit. can be received through the circuit. Also, the processor may analyze the text data and determine a task (or function) to be performed based on the analyzed result.

In operation 1020, the processor may check the voice encryption required to perform the task. For example, the processor may check whether there is a voice password set to perform the task.

In operation 1030, the processor may obtain surrounding information of the electronic device. For example, the processor may obtain information about the surroundings of the electronic device by analyzing a voice input received through the microphone. As another example, the processor may obtain surrounding information of the electronic device by analyzing an image captured by a camera (eg, the camera 991). As another example, the processor may obtain surrounding information of the electronic device based on sensing information sensed by a motion sensor (eg, the motion sensor 992).

In operation 1040, the processor may determine whether surrounding information of the electronic device satisfies a specified condition. For example, the processor may determine whether a plurality of users exist around the electronic device. As another example, the processor may determine whether a plurality of users exist around the electronic device, and determine whether a user (or an unauthenticated user) who does not have permission to use the electronic device exists among the plurality of users. there is. According to an embodiment, the processor may determine that the specified condition is satisfied when it is determined that a plurality of users exist around the electronic device. According to another embodiment, the processor determines that the specified condition is satisfied when it is determined that a plurality of users exist around the electronic device and there is a user who does not have permission to use the electronic device among the plurality of users. can do.

When the surrounding information satisfies a specified condition, in operation 1050, the processor may change password authentication settings necessary for performing the task. For example, when a plurality of users exist around the electronic device and all of the plurality of users are authenticated users, the processor may change the password authentication setting required to perform the task to a common voice password input method. there is. As another example, when a plurality of users exist around the electronic device and an unauthenticated user among the plurality of users exists, the processor configures the password authentication setting necessary for performing the task as a one-time voice password input method. can be changed to

According to an embodiment, the processor changes the password authentication setting required for task execution to a password authentication method through another input means when password authentication is possible through another input means instead of inputting a voice password through the microphone. can For example, when the electronic device includes a key input device (eg, the key input device 993), the processor changes the password authentication setting required for task execution to a pin number input method through the key input device. can As another example, when the electronic device includes a biosensor (eg, the biosensor 994), the processor may change the password authentication setting required to perform the task to a biometric password input method through the biosensor. As another example, when the electronic device includes a touch screen display (eg, the display 995), the processor sets password authentication required to perform a task through a pin number input or pattern input method through the touch screen display. can be changed to

According to an embodiment, when password authentication is possible through an external electronic device connected to the electronic device through a communication circuit (eg, the communication circuit 930), the processor sets the password authentication necessary for performing the task to the external electronic device. It can be changed to a password authentication method through an electronic device. For example, when the external electronic device includes a key input device, a biometric sensor, or a touch screen display, the processor performs password authentication through a key input device, a bio sensor, or a touch screen display included in the external electronic device. The external electronic device may be controlled to

According to an embodiment, the processor may change password authentication settings in a manner that is less likely to expose a voice password set by a user. For example, the processor may preferentially select an encryption authentication method through another input means rather than a voice authentication method.

In operation 1060, the processor may transmit a signal related to a password authentication setting change necessary for performing a task to another component of the electronic device or an external electronic device connected through the communication circuit. For example, the processor may transmit a signal related to a password authentication setting change to the display so that the display outputs information related to a password authentication setting change. As another example, the processor may transmit a signal related to the password authentication setting change to a speaker (eg, the speaker 996) so that the speaker outputs information related to the password authentication setting change. As another example, the processor transmits a signal related to password authentication setting change to an external electronic device connected through the communication circuit, so that the external electronic device displays the information through a display and/or a speaker included in the external electronic device. can be output.

11 is a diagram illustrating a method of changing password authentication settings according to whether or not a user is authenticated according to an embodiment of the present invention.

Referring to FIG. 11 , in operation 1110, a processor (eg, processor 970) of an electronic device (eg, electronic device 900) may determine whether a plurality of users exist around the electronic device. For example, the processor may analyze a voice input received through a microphone (eg, the microphone 910) and determine whether a plurality of users exist around the electronic device based on the analyzed information. As another example, the processor may analyze an image captured by a camera (eg, the camera 991) and determine whether a plurality of users exist around the electronic device based on the analyzed information.

When it is determined that a plurality of users exist around the electronic device, in operation 1130, the processor determines whether an unauthenticated user (or a user who does not have permission to use the electronic device) exists among the plurality of users. can do. For example, the processor compares and analyzes voice data of the authenticated user previously stored in a memory (eg, the memory 950) and voice data corresponding to a voice input received through the microphone (eg, voiceprint recognition) to perform authentication. It is possible to determine whether or not there is a user who is not registered. As another example, the processor may compare and analyze an image of an authenticated user pre-stored in the memory and an image captured by the camera to determine whether an unauthorized user exists.

When there is an unauthenticated user among the plurality of users, in operation 1150, the processor may change the setting of password authentication necessary for performing the task to an input method of a one-time voice password.

If there is no unauthenticated user among the plurality of users (when all of the plurality of users are authenticated users), in operation 1170, the processor configures password authentication settings necessary for performing the task of the public voice password. You can change the input method.

12 is a diagram illustrating a method of changing password authentication settings according to whether an input means can be changed according to an embodiment of the present invention.

Referring to FIG. 12 , in operation 1210, a processor (eg, processor 970) of an electronic device (eg, electronic device 900) replaces inputting a voice password through a microphone (eg, microphone 910) to another It may be determined whether password authentication is possible through the input means. For example, the processor includes a key input device (eg, key input device 993), a biosensor (eg, biosensor 994), or a touch screen display (eg, display 995) in the electronic device. can determine whether As another example, the processor may determine whether an external electronic device connected through a communication circuit (eg, the communication circuit 930) includes a key input device, a biometric sensor, or a touch screen display.

If it is determined that password authentication through the other input means is possible, in operation 1230, the processor may change the password authentication setting necessary for performing the task to a password authentication method through the other input means. For example, when the electronic device includes a key input device, the processor may change a password authentication setting required to perform a task to a pin number input method through the key input device. As another example, when the electronic device includes a biometric sensor, the processor may change a password authentication setting required to perform a task to a biometric password input method through the biometric sensor. As another example, when the electronic device includes a touch screen display, the processor may change a password authentication setting necessary for performing a task to a pin number input or pattern input method through the touch screen display. Alternatively, when password authentication is possible through an external electronic device connected to the electronic device through the communication circuit, the processor may change a password authentication setting necessary for performing a task to a password authentication method through the external electronic device. For example, when the external electronic device includes a key input device, a biometric sensor, or a touch screen display, the processor performs password authentication through a key input device, a bio sensor, or a touch screen display included in the external electronic device. The external electronic device may be controlled to

If it is determined that password authentication through the other input means is not possible, in operation 1250, the processor may change the password authentication setting required for performing the task from a previously set voice password input method to another voice password input method. there is. For example, as described with reference to FIG. 11 , the processor may change the setting of password authentication necessary for performing the task to an input method of a public voice password or a one-time voice password.

According to an embodiment, even when password authentication is possible through the other input means, if the user cannot use the other input means, as in operation 1170, the processor performs the password authentication necessary for performing the task. Settings may be changed from a previously set voice password input method to another voice password input method, such as a public voice password or a one-time voice password. For example, the processor may determine that the other input means cannot be used while the user is driving, and accordingly, the password authentication setting required for performing the task may be changed to a public voice password or a one-time voice password input method. there is. As another example, the processor may determine that the other input means cannot be used even when the user is cooking, and accordingly, the password authentication setting required to perform the task may be changed to a public voice password or a one-time voice password input method. there is.

As described above, according to various embodiments, a method for processing a voice input of an electronic device includes an operation of receiving a first voice input related to performing a task through a microphone, and an operation of checking a first voice password necessary for performing the task. , Obtaining information about the surroundings of the electronic device, changing password authentication settings necessary for performing the task when the surrounding information satisfies a specified condition, and authenticating the password with an external electronic device through a communication circuit. It may include an operation of transmitting a signal related to a setting change of .

According to various embodiments of the present disclosure, the voice input processing method may perform the specified condition when information indicating that there are a plurality of users and among the plurality of users includes information that a user who does not have permission to use the electronic device exists in the surrounding information. An operation of determining that is satisfied may be further included.

According to various embodiments of the present disclosure, the operation of changing the password authentication setting may include a second voice password that can be used by a plurality of users having permission to use the electronic device in the input method of the first voice password. It may include an operation of changing to a password input method.

According to various embodiments, the operation of changing the password authentication setting may change the password authentication setting from the first voice password input method to a second voice password input method that can be used a specified number of times or for a specified period of time. action may be included.

According to various embodiments of the present disclosure, changing the password authentication setting may include determining whether the external electronic device connected through the communication circuit includes at least one of a key input device, a biometric sensor, and a touch screen display; and When the external electronic device includes at least one of the key input device, the biometric sensor, and the touch screen display, password authentication through the external electronic device is set in the first voice password input method. It may include an operation to change in a way.

13 is a diagram for explaining a method of changing a password authentication setting to a public voice password input method according to an embodiment of the present invention.

Referring to FIG. 13 , the electronic device 1301 (eg, the electronic device 900) inputs a voice input (eg, “Bixby, play my music) of the first user 1310 through a microphone (eg, the microphone 910). Do it") can be received. In this case, the electronic device 1301 converts voice data corresponding to the received voice input into text data, or sends the electronic device 1301 to a first external electronic device (not shown) through a communication circuit (eg, the communication circuit 930). Voice data may be transmitted, and text data generated by converting the voice data into a text format from the first external electronic device may be received through the communication circuit.

According to an embodiment, the electronic device 1301 may analyze the text data and determine a task (or function) to be performed based on the analyzed result. For example, the electronic device 1301 determines whether the text data includes a phrase mapped to a specific task (eg, “music”, “play”), and determines whether the text data includes a phrase mapped to the specific task. , it may be determined that the first user 1310 wants the specific task (eg, execution of a music playing application) to be performed.

According to an embodiment, when a task to be performed is determined, the electronic device 1301 may check a password required to perform the task. For example, the electronic device 1301 may determine whether a password set to perform the task exists. In addition, the electronic device 1301 may check whether the password is set as a voice password.

When a voice password required to perform the task is set, the electronic device 1301 may acquire (or collect) surrounding information of the electronic device 1301 . For example, the electronic device 1301 may obtain information about the surroundings of the electronic device 1301 by analyzing a voice input received through the microphone. As another example, the electronic device 1301 may obtain surrounding information of the electronic device 1301 by analyzing an image captured through a camera (eg, the camera 991). As another example, the electronic device 1301 may obtain surrounding information of the electronic device 1301 based on sensing information detected by a motion sensor (eg, the motion sensor 992).

According to an embodiment, the electronic device 1301 may determine whether a plurality of users exist around the electronic device 1301. For example, the electronic device 1301 analyzes the obtained surrounding information of the electronic device 1301 and allows users other than the first user 1310 (eg, the second user 1320, third It may be determined whether the user 1330 or the fourth user 1340 exists. Also, when a plurality of users exist around the electronic device 1301, the electronic device 1301 may determine whether a user does not have permission to use the electronic device 1301 among the plurality of users.

When all of the plurality of users are users who have permission to use the electronic device 1301, the electronic device 1301 makes the password authentication setting necessary for performing the task common to the users who have permission to use the electronic device 1301. It can be changed to an input method of a public voice password that can be used. In addition, the electronic device 1301 may request users to input a public voice code required to perform the task. For example, the electronic device 1301 may output a voice requesting input of the public voice password (eg, "Yes, tell me the common voice password") through a speaker (eg, the speaker 996).

When one of the plurality of users utters the common voice password (eg, “1234”), the electronic device 1301 may perform the task (eg, execution of a music playing application). According to an embodiment, after the task is performed, the electronic device 1301 may restore password authentication settings required for the task to be performed. For example, the electronic device 1301 may restore the setting of password authentication required to perform the task from a common voice password input method to a preset voice password input method.

According to an embodiment, the electronic device 1301 may transmit a signal related to a change in settings of password authentication required to perform the task to the second external electronic device 1303 through the communication circuit. For example, the electronic device 1301 may transmit to the second external electronic device 1303 a signal corresponding to information notifying that a password authentication setting necessary for performing the task has been changed. The second external electronic device 1303 may be, for example, a user's personal terminal (eg, a smart phone) having permission to use the electronic device 1301 . Also, the second external electronic device 1303 may be a device capable of changing a configuration related to use of the electronic device 1301 (eg, password setting).

The second external electronic device 1303 receiving the signal related to the change in password authentication settings necessary for performing the task informs the user of the second external electronic device 1303 that the password authentication settings necessary for performing the task have been changed. Reminders can be made. For example, the second external electronic device 1303 provides an object notifying through a display and/or a speaker that the setting of password authentication necessary for performing the task has been changed (eg, "The public voice password has been used to play music.") can output In some embodiments, the second external electronic device 1303 displays and/or displays an object for confirming whether to change the public voice password set in the electronic device 1301 (eg, "Do you want to change the public voice password?") Alternatively, it can be output through a speaker.

According to an embodiment, the electronic device 1301 is required to perform the above task. Changes in settings for password authentication are performed on a display (e.g., display 995) and/or a speaker (e.g., speaker ( 996)) may remind the user.

14 is a diagram for explaining a method of changing a password authentication setting to a one-time voice password input method according to an embodiment of the present invention.

Referring to FIG. 14 , the electronic device 1401 (eg, the electronic device 900) inputs a voice input (eg, “Bixby, the picture taken yesterday) of the first user 1310 through a microphone (eg, the microphone 910). Show me on TV") can be received. In this case, the electronic device 1401 converts voice data corresponding to the received voice input into text data, or sends the electronic device 1401 to a first external electronic device (not shown) through a communication circuit (eg, the communication circuit 930). Voice data may be transmitted, and text data generated by converting the voice data into a text format from the first external electronic device may be received through the communication circuit.

According to an embodiment, the electronic device 1401 may analyze the text data and determine a task (or function) to be performed based on the analyzed result. For example, the electronic device 1401 determines whether the text data includes a phrase mapped to a specific task (eg, “photo”, “on TV”, “show me”), and the phrase mapped to the specific task. is included, it may be determined that the first user 1410 wants to perform the specific task (eg, displaying an execution screen of a gallery application on a TV screen).

According to an embodiment, when a task to be performed is determined, the electronic device 1401 may check a password required to perform the task. For example, the electronic device 1401 may determine whether a password set to perform the task exists. Also, the electronic device 1401 may check whether the password is set as a voice password.

When a voice password required to perform the task is set, the electronic device 1401 may acquire (or collect) surrounding information of the electronic device 1401 . According to an embodiment, the electronic device 1401 may determine whether a plurality of users exist around the electronic device 1401 . For example, the electronic device 1401 analyzes the acquired information about the surroundings of the electronic device 1401, and other users other than the first user 1410 (eg, the second user 1420 or the third It is possible to determine whether a user 1430) exists. In addition, when a plurality of users exist around the electronic device 1401, the electronic device 1401 may determine whether a user does not have permission to use the electronic device 1401 among the plurality of users.

If there is a user (eg, the second user 1420 or the third user 1430) who does not have the right to use the electronic device 1401 among the plurality of users, the electronic device 1401 is required to perform the task. You can change the required password authentication setting to a one-time voice password input method that can be used a specified number of times or for a specified period of time. Also, the electronic device 1401 may request the first user 1410 to input a one-time voice password required to perform the task. For example, the electronic device 1401 may output a voice requesting input of the one-time voice password (eg, "Yes, tell me an alarm time this morning") through a speaker (eg, the speaker 996).

When the first user 1410 utters the one-time voice password (eg, “7:00”), the electronic device 1401 may perform the task (eg, displaying an execution screen of a gallery application on a TV screen). According to an embodiment, after performing the task, the electronic device 1401 may restore password authentication settings necessary for performing the task. For example, the electronic device 1401 may restore the setting of password authentication required to perform the task from a one-time voice password input method to a preset voice password input method. Also, the electronic device 1401 may discard the used one-time voice password after performing the task.

According to an embodiment, the electronic device 1401 may transmit a signal related to a change in password authentication settings necessary for performing the task to the second external electronic device 1403 through the communication circuit. For example, the electronic device 1401 may transmit to the second external electronic device 1403 a signal corresponding to information notifying that a password authentication setting necessary for performing the task has been changed. The second external electronic device 1403 may be, for example, a personal terminal (eg, a smart phone) of a user having authority to use the electronic device 1401 (eg, the first user 1410). Also, the second external electronic device 1403 may be a device capable of changing a configuration (eg, password setting) related to the use of the electronic device 1401 .

The second external electronic device 1403 receiving the signal related to the password authentication setting change necessary to perform the task is transmitted to the user (eg, the first user 1410) of the second external electronic device 1403 to perform the task. You can be reminded that the password authentication settings required for execution have been changed. For example, the second external electronic device 1403 provides an object (e.g., "The one-time voice password used for viewing the gallery has been discarded. ") can be printed. In some embodiments, the second external electronic device 1403 displays and displays an object (eg, “Do you want to reset the one-time voice password?”) for confirming whether you want to set the one-time voice password set in the electronic device 1401. / Or it can be output through a speaker.

According to an embodiment, the electronic device 1401 is required to perform the above task. Changes in settings for password authentication are performed on a display (e.g., display 995) and/or a speaker (e.g., speaker ( 996)) may remind the user.

15 is a diagram for explaining a method of setting a one-time voice password according to an embodiment of the present invention.

Referring to FIG. 15 , an electronic device 1500 (eg, an electronic device 900, 1303, or 1403) may set a one-time voice password required to perform a task by using information related to a user of the electronic device 1500. . For example, the electronic device 1500 may provide the user's personal information (eg, the user's name, age, gender, or region of residence), or the user's activity information (eg, schedule, alarm time, health information, content consumption information, Alternatively, traffic use information, etc.) may be used to set the one-time voice password.

According to an embodiment, the electronic device 1500 may display a list 1531 of user personal information or user activity information usable for setting the one-time voice password through the display 1510 . In addition, the electronic device 1500 is an object that allows the user to select the user's personal information or the user's activity information to be used for setting the one-time voice password, together with the user's personal information or user's activity information list 1531. 1533 (eg, a toggle button object) may be output to the display 1510 . In this case, when the user selects the object 1533, the electronic device 1500 may set the one-time voice password using personal information or activity information of the user corresponding to the selected object 1533.

According to an embodiment, the user's personal information or the user's activity information list 1531 may be stored and managed in a memory (eg, the memory 950). The user's personal information or user's activity information list 1531 may include, for example, a question list 1553 prompting input of a one-time voice password according to the type of information 1551 . For example, when a one-time voice password is set in relation to a user's schedule, a question leading to input of the one-time voice password may be set as a question asking one of the user's schedules, and the electronic device 1500 may set the one-time voice password as a question. The input of the one-time voice password may be confirmed by determining whether an answer to the question corresponds to information set as a schedule.

According to an embodiment, the user's personal information or user's activity information list 1531 may include information 1555 about privacy sensitivity set according to the type of information 1551 . For example, the electronic device 1500 is configured to allow a user to directly select personal information or activity information of a user used when setting a one-time voice password in order to prevent personal information or activity information that is not desired by the user from being exposed. interfaces can be provided.

16 is a diagram for explaining a method of authenticating a password through a one-time voice password according to an embodiment of the present invention.

According to an embodiment, the electronic device 1601 (eg, the electronic device 900) uses information related to the user of the electronic device 1610 (eg, the user's personal information or the user's activity information) to provide a one-time voice password. can be set. However, the electronic device 1601 uses information related to the user, even if the one-time voice password is set, when the user indicates that he or she does not want to answer a question leading to input of the one-time voice password (eg, when the user says “Next”). "), the one-time voice password may be reset using other information.

Referring to FIG. 16 , the electronic device 1601 may receive a first user 1610's voice input (eg, “Bixby, show the picture taken yesterday on TV”) through a microphone (eg, the microphone 910). there is. In addition, the electronic device 1601 may determine a task (or function) to be performed using text data obtained by converting voice data corresponding to the received voice input into a text format.

According to an embodiment, when a task to be performed is determined, the electronic device 1601 may check a password required to perform the task. For example, the electronic device 1601 may determine whether a password set to perform the task exists. In addition, the electronic device 1601 may check whether the password is set as a voice password.

When a voice password required to perform the task is set, the electronic device 1601 may acquire (or collect) surrounding information of the electronic device 1601 . According to an embodiment, the electronic device 1601 may determine whether a plurality of users exist around the electronic device 1601. For example, the electronic device 1601 analyzes the acquired information about the surroundings of the electronic device 1601, and other users (eg, the second user 1620 or the third user 1610) in the vicinity of the electronic device 1601 It is possible to determine whether a user 1630 exists. Also, when a plurality of users exist around the electronic device 1601, the electronic device 1601 may determine whether a user does not have permission to use the electronic device 1601 among the plurality of users.

If there is a user (eg, the second user 1620 or the third user 1630) who does not have the right to use the electronic device 1601 among the plurality of users, the electronic device 1601 is required to perform the task. You can change the required password authentication setting to a one-time voice password input method that can be used a specified number of times or for a specified period of time. In addition, the electronic device 1601 may request the first user 1610 to input a one-time voice password required to perform the task. For example, the electronic device 1601 may output a voice requesting input of the one-time voice password (eg, "Yes, what was the most recently watched TV program?") through a speaker (eg, the speaker 996). .

When the first user 1610 indicates that he or she does not want to answer the question leading to input of the one-time voice password (eg, when the user utters “next”), the electronic device 1601 transmits the one-time voice password can be reset using other information. Also, the electronic device 1601 may request the first user 1610 to input the reset one-time voice password. For example, the electronic device 1601 may output a voice requesting input of the reset one-time voice password (eg, "Yes, tell me the title and artist name of one of the songs in the playlist.") through the speaker. there is.

When the first user 1610 utters the reset one-time voice password (eg, “Adele, someone like you”), the electronic device 1601 performs the task (eg, displaying the execution screen of the gallery application on the TV screen). can be done According to an embodiment, after performing the task, the electronic device 1601 may restore password authentication settings necessary for performing the task. For example, the electronic device 1601 may restore the setting of password authentication required to perform the task from a one-time voice password input method to a preset voice password input method. Also, the electronic device 1601 may discard the used one-time voice password after performing the task.

According to an embodiment, the electronic device 1601 may transmit a signal related to changing password authentication settings necessary for performing the task to the external electronic device 1603 through the communication circuit. For example, the electronic device 1601 may transmit to the external electronic device 1603 a signal corresponding to information notifying that a password authentication setting necessary for performing the task has been changed. In this case, the external electronic device 1603 receiving the signal related to the setting change of the password authentication required to perform the task requires the user (eg, the first user 1610) of the external electronic device 1603 to perform the task. You can be reminded that the password authentication settings required for According to some embodiments, the electronic device 1601 is required to perform the above task. Changes to settings of password authentication are performed on a display (eg, display 995) and/or a speaker (eg, speaker ( 996)) may remind the user.

17 is a diagram for explaining a method of changing a password input means according to an embodiment of the present invention.

According to an embodiment, the electronic device 1701 (eg, the electronic device 900) may perform a task when password authentication is possible through another input means instead of inputting a voice password through a microphone (eg, the microphone 910). It is possible to change the setting of password authentication required to perform the password authentication method through other input means. For example, when the electronic device 1701 includes a key input device (eg, the key input device 993), password authentication settings necessary for performing a task may be changed to a pin number input method through the key input device. there is. As another example, when the electronic device 1701 includes a biometric sensor (eg, the biometric sensor 994), password authentication settings necessary for performing a task may be changed to a biometric password input method through the biometric sensor. As another example, when the electronic device 1701 includes a touch screen display (eg, the display 995), a PIN number input or pattern input method may be changed through the touch screen display.

According to an embodiment, the electronic device 1701 performs a task when password authentication is possible through an external electronic device 1703 connected to the electronic device 1701 through a communication circuit (eg, the communication circuit 930). A required password authentication setting may be changed to a password authentication method through the external electronic device 1703 . For example, the electronic device 1701 may include a key input device, a bio sensor, or a touch screen included in the external electronic device 1703 when the external electronic device 1703 includes a key input device, a bio sensor, or a touch screen display. The external electronic device 1703 may be controlled to perform password authentication through the display.

Referring to FIG. 17 , the electronic device 1701 may receive a first user 1710's voice input (eg, "Bixby, show the photo taken yesterday on TV") through the microphone. In addition, the electronic device 1701 may determine a task (or function) to be performed using text data obtained by converting voice data corresponding to the received voice input into a text format.

According to an embodiment, when a task to be performed is determined, the electronic device 1701 may check a password required to perform the task. For example, the electronic device 1701 may determine whether a password set to perform the task exists. Also, the electronic device 1701 may check whether the password is set as a voice password.

When a voice password required to perform the task is set, the electronic device 1701 may obtain (or collect) surrounding information of the electronic device 1701 . According to an embodiment, the electronic device 1701 may determine whether a plurality of users exist around the electronic device 1701. For example, the electronic device 1701 analyzes the acquired information about the surroundings of the electronic device 1701, and other users other than the first user 1710 (eg, the second user 1720 or the third It is possible to determine whether a user 1730 exists.

When the plurality of users exist around the electronic device 1701, the electronic device 1701 may determine whether password authentication is possible through another input means instead of inputting a voice password through the microphone. For example, the electronic device 1701 may check whether the electronic device 1701 includes the key input device, the biometric sensor, or the touch screen display. As another example, the electronic device 1701 may check whether the external electronic device 1703 connected through the communication circuit includes the key input device, the biometric sensor, or the touch screen display.

If it is determined that password authentication through the other input means is possible, the electronic device 1701 can change the password authentication setting necessary for performing the task to a password authentication method through the other input means. Also, the electronic device 1701 may request password authentication from the first user 1710 through the other input means. For example, the electronic device 1701 may output a voice requesting password authentication through the other input means (eg, "Yes, please perform fingerprint authentication using a smartphone") through a speaker (eg, the speaker 996). there is.

When the first user 1710 performs password authentication through the other input means, the electronic device 1701 can perform the task (eg, displaying an execution screen of a gallery application on a TV screen). For example, when the first user 1710 performs fingerprint authentication through the fingerprint sensor of the external electronic device 1703, the electronic device 1710 can perform the task. According to an embodiment, after performing the task, the electronic device 1701 may restore password authentication settings necessary for performing the task. For example, the electronic device 1701 may restore the setting of password authentication necessary for performing the task to a preset voice password input method from the password authentication method through the other input means.

18 is a diagram for explaining a method of reminding when password authentication settings are not changed according to an embodiment of the present invention.

According to an embodiment, the electronic device 1801 (eg, the electronic device 900) may not change password authentication settings necessary to perform the task according to the selection of the first user 1810. Alternatively, when the first user 1810 does not want to use the changed password authentication method, the electronic device 1801 may restore the password authentication setting even if the password authentication setting necessary for performing the task is changed.

Referring to FIG. 18 , the electronic device 1801 may receive a first user 1810's voice input (eg, "Bixby, show the picture taken yesterday on TV") through a microphone (eg, the microphone 910). there is. In addition, the electronic device 1801 may determine a task (or function) to be performed using text data obtained by converting voice data corresponding to the received voice input into a text format.

According to an embodiment, when a task to be performed is determined, the electronic device 1801 may check a password required to perform the task. For example, the electronic device 1801 may determine whether a password set to perform the task exists. Also, the electronic device 1801 may check whether the password is set as a voice password.

When a voice password required to perform the task is set, the electronic device 1801 may acquire (or collect) surrounding information of the electronic device 1801 . According to an embodiment, the electronic device 1801 may determine whether a plurality of users exist around the electronic device 1801. For example, the electronic device 1801 analyzes the acquired information about the surroundings of the electronic device 1801, and other users (eg, the second user 1820 or the third user 1810) are located around the electronic device 1801. It is possible to determine whether a user 1830) exists.

When the plurality of users exist around the electronic device 1801, the electronic device 1801 may determine whether password authentication is possible through another input means instead of inputting a voice password through the microphone. If it is determined that password authentication through the other input means is possible, the electronic device 1801 can change the password authentication setting necessary for performing the task to a password authentication method through the other input means. Also, the electronic device 1801 may request password authentication from the first user 1810 through the other input means. For example, the electronic device 1801 may output a voice requesting password authentication through the other input means (eg, "Yes, please perform fingerprint authentication with a smartphone") through a speaker (eg, the speaker 996). there is.

When the first user 1810 expresses an intention not to perform password authentication through the other input means (eg, when the user utters "No, I'll just do it by voice"), the electronic device 1801 performs password authentication. settings can be reversed. For example, the electronic device 1801 may restore the setting of password authentication necessary for performing the task to a preset voice password input method from the password authentication method through the other input means. Also, the electronic device 1801 may request the first user 1810 to input a preset voice password. For example, the electronic device 1801 may output a voice requesting input of a preset voice password (eg, "Yes, tell me the voice password") through the speaker.

When the first user 1810 utters the preset voice password (eg, “3355”), the electronic device 1801 may perform the task (eg, displaying an execution screen of a gallery application on a TV screen). .

According to an embodiment, the electronic device 1801 may transmit a signal related to a change in password authentication settings necessary for performing the task to the external electronic device 1803 through a communication circuit (eg, the communication circuit 930). . For example, the electronic device 1801 is an external electronic device 1803, and the voice password required to perform the task is a user other than the first user 1810 (eg, the second user 1820 or the third user 1830). ) may transmit a signal corresponding to information indicating that it has been exposed. In this case, the external electronic device 1803 may provide a reminder that the voice password required to perform the task has been exposed to a user other than the first user 1810 (eg, the second user 1820 or the third user 1830). can For example, the external electronic device 1803 may output exposure information of the voice password required to perform the task (eg, "The voice password was used for viewing the gallery") through a display and/or a speaker. According to an embodiment, the external electronic device 1803 may output information (eg, "Do you want to change the voice password?") through a display and/or a speaker to check whether the exposed voice password has been changed. According to an embodiment, the electronic device 1801 may output information related to changing settings of password authentication necessary for performing the task through a display and/or a speaker included in the electronic device 1801.

19 is a diagram for explaining a method of reminding a password authentication setting change according to an embodiment of the present invention.

Referring to FIG. 19 , an electronic device 1900 (eg, an electronic device 900 , 1303 , 1403 , 1603 , 1703 , or 1803 ) may remind a user of a password authentication setting change required to perform a task. According to an embodiment, the electronic device 1900 outputs information related to password authentication setting change through at least one of a display 1910 (eg, the display 995) and a speaker (eg, the speaker 996). can For example, the electronic device 1900 may output, on the display 1910 , information 1931 notifying the use of a voice password (or a common voice password or a one-time voice password). In some embodiments, the electronic device 1900 changes (or sets) the voice password (or public voice password or one-time voice password) to information 1931 indicating the use of the voice password (or public voice password or one-time voice password). You can also include information to confirm whether or not.

According to an embodiment, the electronic device 1900 displays information 1931 indicating the use of the voice password (or public voice password or one-time voice password) and usage history of the voice password (or public voice password or one-time voice password). An object 1933 (eg, a button object) that allows to check may be output to the display 1910 . Accordingly, when the user selects and inputs the object 1933 (1950), the electronic device 1900 outputs usage history information 1970 of the voice password (or public voice password or one-time voice password) on the display 1910 can do. Usage history information 1970 of the voice password (or public voice password or one-time voice password) includes, for example, time information 1971 when the voice password (or public voice password or one-time voice password) was used, device information used ( 1973), used function (or task) information (1975), other user information (1977) who was present at the time of use, and the like.

20 is a block diagram of an electronic device 2001 within a network environment 2000, according to various embodiments. Referring to FIG. 20 , in a network environment 2000, an electronic device 2001 communicates with an electronic device 2002 through a first network 2098 (eg, a short-distance wireless communication network) or through a second network 2099. It is possible to communicate with the electronic device 2004 or the server 2008 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 2001 may communicate with the electronic device 2004 through the server 2008. According to an embodiment, the electronic device 2001 includes a processor 2020, a memory 2030, an input device 2050, an audio output device 2055, a display device 2060, an audio module 2070, a sensor module ( 2076), interface 2077, haptic module 2079, camera module 2080, power management module 2088, battery 2089, communication module 2090, subscriber identification module 2096, or antenna module 2097 ) may be included. In some embodiments, in the electronic device 2001, at least one of these components (eg, the display device 2060 or the camera module 2080) may be omitted or one or more other components may be added. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, the sensor module 2076 (eg, a fingerprint sensor, an iris sensor, or an illumination sensor) may be implemented while being embedded in the display device 2060 (eg, a display).

The processor 2020, for example, executes software (eg, the program 2040) to cause at least one other component (eg, hardware or software component) of the electronic device 2001 connected to the processor 2020. It can control and perform various data processing or calculations. According to one embodiment, as at least part of data processing or operation, the processor 2020 transfers commands or data received from other components (eg, sensor module 2076 or communication module 2090) to volatile memory 2032. , process commands or data stored in the volatile memory 2032, and store resultant data in the non-volatile memory 2034. According to one embodiment, the processor 2020 may include a main processor 2021 (eg, a central processing unit or an application processor), and a secondary processor 2023 (eg, a graphic processing unit, an image signal processor) that may operate independently or together therewith. , sensor hub processor, or communication processor). Additionally or alternatively, the auxiliary processor 2023 may use less power than the main processor 2021, or may be set to be specialized for a designated function. The auxiliary processor 2023 may be implemented separately from or as part of the main processor 2021 .

The auxiliary processor 2023 may, for example, take place of the main processor 2021 while the main processor 2021 is in an inactive (eg, sleep) state, or when the main processor 2021 is active (eg, running an application). ) state, together with the main processor 2021, at least one of the components of the electronic device 2001 (eg, the display device 2060, the sensor module 2076, or the communication module 2090) It is possible to control at least some of the related functions or states. According to one embodiment, the auxiliary processor 2023 (eg, image signal processor or communication processor) may be implemented as part of other functionally related components (eg, camera module 2080 or communication module 2090). there is.

The memory 2030 may store various data used by at least one component (eg, the processor 2020 or the sensor module 2076) of the electronic device 2001. The data may include, for example, input data or output data for software (eg, the program 2040) and commands related thereto. The memory 2030 may include a volatile memory 2032 or a non-volatile memory 2034 .

The program 2040 may be stored as software in the memory 2030, and may include, for example, an operating system 2042, middleware 2044, or an application 2046.

The input device 2050 may receive a command or data to be used for a component (eg, the processor 2020) of the electronic device 2001 from an outside of the electronic device 2001 (eg, a user). The input device 2050 may include, for example, a microphone, mouse, or keyboard.

The sound output device 2055 may output sound signals to the outside of the electronic device 2001 . The audio output device 2055 may include, for example, a speaker or receiver. The speaker can be used for general purposes, such as multimedia playback or recording playback, and the receiver can be used to receive an incoming call. According to one embodiment, the receiver may be implemented separately from the speaker or as part of it.

The display device 2060 can visually provide information to the outside of the electronic device 2001 (eg, a user). The display device 2060 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display device 2060 may include a touch circuitry set to detect a touch or a sensor circuit (eg, a pressure sensor) set to measure the intensity of force generated by the touch. there is.

The audio module 2070 may convert sound into an electrical signal or vice versa. According to an embodiment, the audio module 2070 acquires sound through the input device 2050, the audio output device 2055, or an external electronic device connected directly or wirelessly to the electronic device 2001 (eg: Sound may be output through the electronic device 2002 (eg, a speaker or a headphone).

The sensor module 2076 detects an operating state (eg, power or temperature) of the electronic device 2001 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to one embodiment, the sensor module 2076 may include, for example, a gesture sensor, a gyro sensor, an air pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a bio sensor, It may include a temperature sensor, humidity sensor, or light sensor.

The interface 2077 may support one or more designated protocols that may be used to directly or wirelessly connect the electronic device 2001 to an external electronic device (eg, the electronic device 2002). According to one embodiment, the interface 2077 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 2078 may include a connector through which the electronic device 2001 may be physically connected to an external electronic device (eg, the electronic device 2002). According to one embodiment, the connection terminal 2078 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 2079 may convert electrical signals into mechanical stimuli (eg, vibration or motion) or electrical stimuli that a user can perceive through tactile or kinesthetic senses. According to one embodiment, the haptic module 2079 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 2080 may capture still images and moving images. According to one embodiment, the camera module 2080 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 2088 may manage power supplied to the electronic device 2001 . According to one embodiment, the power management module 388 may be implemented as at least part of a power management integrated circuit (PMIC), for example.

The battery 2089 may supply power to at least one component of the electronic device 2001 . According to an embodiment, the battery 2089 may include, for example, a non-rechargeable primary battery, a rechargeable secondary battery, or a fuel cell.

The communication module 2090 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 2001 and an external electronic device (eg, the electronic device 2002, the electronic device 2004, or the server 2008). Establishment and communication through the established communication channel may be supported. The communication module 2090 may include one or more communication processors that operate independently of the processor 2020 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 2090 is a wireless communication module 2092 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 2094 (eg, : a local area network (LAN) communication module or a power line communication module). Among these communication modules, a corresponding communication module is a first network 2098 (eg, a short-range communication network such as Bluetooth, WiFi direct, or infrared data association (IrDA)) or a second network 2099 (eg, a cellular network, the Internet, or It may communicate with an external electronic device via a computer network (eg, a telecommunications network such as a LAN or WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or implemented as a plurality of separate components (eg, multiple chips). The wireless communication module 2092 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 2096 within a communication network such as the first network 2098 or the second network 2099. The electronic device 2001 may be identified and authenticated.

The antenna module 2097 may transmit or receive signals or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 2097 may include one or more antennas, from which at least one suitable for a communication scheme used in a communication network such as the first network 2098 or the second network 2099 is used. of antennas may be selected by the communication module 2090, for example. A signal or power may be transmitted or received between the communication module 2090 and an external electronic device through the selected at least one antenna.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and signal (eg, : commands or data) can be exchanged with each other.

According to an embodiment, commands or data may be transmitted or received between the electronic device 2001 and the external electronic device 2004 through the server 2008 connected to the second network 2099. Each of the electronic devices 2002 and 2004 may be the same as or different from the electronic device 2001 . According to an embodiment, all or part of operations executed in the electronic device 2001 may be executed in one or more external devices among the external electronic devices 2002, 2004, or 2008. For example, when the electronic device 2001 needs to perform a certain function or service automatically or in response to a request from a user or another device, the electronic device 2001 instead of executing the function or service by itself. Alternatively or additionally, one or more external electronic devices may be requested to perform the function or at least part of the service. One or more external electronic devices receiving the request may execute at least a part of the requested function or service or an additional function or service related to the request, and deliver the execution result to the electronic device 2001 . The electronic device 2001 may provide the result as at least part of a response to the request as it is or after additional processing. To this end, for example, cloud computing, distributed computing, or client-server computing technology may be used.

Electronic devices according to various embodiments disclosed in this document may be devices of various types. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance. An electronic device according to an embodiment of this document is not limited to the aforementioned devices.

Various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, and should be understood to include various modifications, equivalents, or substitutes of the embodiments. In connection with the description of the drawings, like reference numbers may be used for like or related elements. The singular form of a noun corresponding to an item may include one item or a plurality of items, unless the relevant context clearly dictates otherwise. In this document, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A Each of the phrases such as “at least one of , B, or C” may include all possible combinations of items listed together in the corresponding one of the phrases. Terms such as "first", "second", or "first" or "secondary" may simply be used to distinguish a given component from other corresponding components, and may be used to refer to a given component in another aspect (eg, importance or order) is not limited. A (e.g., first) component is said to be "coupled" or "connected" to another (e.g., second) component, with or without the terms "functionally" or "communicatively." When mentioned, it means that the certain component may be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term "module" used in this document may include a unit implemented by hardware, software, or firmware, and may be used interchangeably with terms such as logic, logic block, component, or circuit, for example. A module may be an integrally constructed component or a minimal unit of components or a portion thereof that performs one or more functions. For example, according to one embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of this document describe one or more commands stored in a storage medium (eg, internal memory 2036 or external memory 2038) readable by a machine (eg, electronic device 2001). It may be implemented as software (eg, the program 2040) including them. For example, a processor (eg, the processor 2020) of a device (eg, the electronic device 2001) may call at least one command among one or more instructions stored from a storage medium and execute it. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-temporary' only means that the storage medium is a tangible device and does not contain signals (e.g., electromagnetic waves), and this term refers to the case where data is stored semi-permanently in the storage medium. It does not discriminate when it is temporarily stored.

According to one embodiment, the method according to various embodiments disclosed in this document may be included and provided in a computer program product. Computer program products may be traded between sellers and buyers as commodities. A computer program product is distributed in the form of a device-readable storage medium (eg compact disc read only memory (CD-ROM)), or through an application store (eg Play Store ^TM ) or on two user devices ( It can be distributed (eg downloaded or uploaded) online, directly between smartphones. In the case of online distribution, at least part of the computer program product may be temporarily stored or temporarily created in a device-readable storage medium such as a manufacturer's server, an application store server, or a relay server's memory.

According to various embodiments, each component (eg, module or program) of the components described above may include a singular object or a plurality of entities. According to various embodiments, one or more components or operations among the aforementioned components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg modules or programs) may be integrated into a single component. In this case, the integrated component may perform one or more functions of each of the plurality of components identically or similarly to those performed by a corresponding component of the plurality of components prior to the integration. . According to various embodiments, operations performed by modules, programs, or other components are executed sequentially, in parallel, iteratively, or heuristically, or one or more of the operations are executed in a different order, omitted, or , or one or more other operations may be added.

Claims (20)

In electronic devices,
microphone;
a communication circuit for communication with an external electronic device;
a processor electrically connected to the microphone and the communication circuit; and
A memory electrically connected to the processor;
The memory, when executed, the processor,
Receiving a first voice input related to the performance of a task through the microphone;
confirming a first voice encryption required for performing the task;
Acquire surrounding information of the electronic device;
When the surrounding information includes information that a plurality of users exist, changing the password authentication setting required to perform the task from the first voice password input method to a common voice password input method;
When information that a plurality of users exist in the surrounding information and information that a user who does not have permission to use the electronic device exists among the plurality of users, the password authentication setting is set in the first voice password input method. Change to the input method of one-time voice password,
An electronic device that stores instructions for transmitting a signal related to a setting change of the password authentication to the external electronic device through the communication circuit. The method of claim 1,
The memory, when executed, the processor,
As at least part of the operation of obtaining the surrounding information,
An electronic device storing instructions for analyzing at least one of the first voice input and the second voice input received through the microphone. The method of claim 1,
Including more cameras,
The memory, when executed, the processor,
As at least part of the operation of obtaining the surrounding information,
An electronic device that stores instructions for analyzing an image acquired through the camera. The method of claim 1,
Including more motion sensors,
The memory, when executed, the processor,
As at least part of the operation of obtaining the surrounding information,
An electronic device that stores instructions for analyzing sensing information sensed by the motion sensor. delete delete delete The method of claim 1,
The memory, when executed, the processor,
As at least part of the operation of changing the setting of the password authentication,
An electronic device that stores instructions for changing the password authentication setting from the first voice password input method to a second voice password input method that can be used a specified number of times or for a specified period of time. The method of claim 8,
The memory, when executed, the processor,
The electronic device further stores instructions for setting the second voice password using information related to a user of the electronic device. ◈Claim 10 was abandoned when the registration fee was paid.◈ The method of claim 9,
Information related to the user,
An electronic device including the user's personal information and the user's activity information. ◈Claim 11 was abandoned when the registration fee was paid.◈ The method of claim 1,
further comprising a key input device;
The memory, when executed, the processor,
As at least part of the operation of changing the setting of the password authentication,
An electronic device storing instructions for changing the password authentication setting from the first voice password input method to a pin code input method through the key input device. The method of claim 1,
Further comprising a biosensor,
The memory, when executed, the processor,
As at least part of the operation of changing the setting of the password authentication,
An electronic device storing instructions for changing the setting of the password authentication from the first voice password input method to the biometric password input method through the biometric sensor. The method of claim 1,
Further comprising a touch screen display,
The memory, when executed, the processor,
As at least part of the operation of changing the setting of the password authentication,
An electronic device that stores instructions for changing the password authentication setting from the first voice password input method to a pin code or pattern input method through the touch screen display. The method of claim 1,
The memory, when executed, the processor,
As at least part of the operation of changing the setting of the password authentication,
determining whether the external electronic device connected through the communication circuit includes at least one of a key input device, a biometric sensor, and a touch screen display;
When the external electronic device includes at least one of the key input device, the biometric sensor, and the touch screen display, password authentication through the external electronic device is set in the first voice password input method. An electronic device that stores instructions that cause it to change in a way. ◈Claim 15 was abandoned when the registration fee was paid.◈ The method of claim 1,
Further comprising at least one of a speaker and a display,
The memory, when executed, the processor,
The electronic device further stores instructions for outputting information related to setting change of the password authentication through at least one of the speaker and the display. In the voice input processing method of the electronic device,
receiving a first voice input related to performing the task through a microphone;
checking a first voice encryption required for performing the task;
obtaining peripheral information of the electronic device;
changing a password authentication setting required to perform the task from an input method of the first voice code to an input method of a common voice code when the surrounding information includes information that a plurality of users exist;
When information that a plurality of users exist in the surrounding information and information that a user who does not have permission to use the electronic device exists among the plurality of users, the password authentication setting is set in the first voice password input method. changing to a one-time voice password input method; and
and transmitting a signal related to the password authentication setting change to an external electronic device through a communication circuit. delete delete delete ◈Claim 20 was abandoned when the registration fee was paid.◈ The method of claim 16
The operation of changing the setting of the password authentication,
determining whether the external electronic device connected through the communication circuit includes at least one of a key input device, a biometric sensor, and a touch screen display; and
When the external electronic device includes at least one of the key input device, the biometric sensor, and the touch screen display, password authentication through the external electronic device is set in the first voice password input method. A voice input processing method including an operation of changing the method.

Images