KR20210046475A - Foldable electronic device and method for driving speech recognition funtion in the same - Google Patents

Abstract

According to various embodiments, provided is a foldable electronic device, which includes: a first housing formed of a first surface and a third surface opposite to the first surface; a second housing formed of a second surface extending from the first surface and a fourth surface opposite to the second surface; a first display formed on the first surface and the second surface; a second display disposed on at least a part of the third surface; a memory; and a processor, wherein the memory can store instructions which, when executed, cause the processor to execute an intelligent application in response to a folding event of the first housing and the second housing while displaying an execution screen of at least one application on at least one of the first display and the second display, analyze an application state and context of an application screen on the basis of execution information of an application displayed on at least one of the first display and the second display, and output at least one of a voice recognition guidance message and a voice mode item including a context item designated in relation to the displayed application. According to the present invention, user convenience can be increased by assisting the user to more easily utilize a function optimized for the activity of the user.

Description

Electronic device and its voice recognition function driving method {FOLDABLE ELECTRONIC DEVICE AND METHOD FOR DRIVING SPEECH RECOGNITION FUNTION IN THE SAME}

Various embodiments of the present invention relate to an electronic device and a method of driving a voice recognition function thereof.

The display may be included in not only large electronic devices but also small and portable electronic devices. Electronic devices are evolving into structures for expanding displays or improving display utilization. Electronic devices include a deformable flexible display, and foldable electronic devices that operate in a foldable or unfoldable manner are being released.

Recently, a voice recognition function of an electronic device has been applied, and various functions of an electronic device are implemented by executing the voice recognition function through various input methods.

When a designated input event is detected, the electronic device may activate a voice recognition function (or application program) and wait for a user's voice input. However, the initial operation at the time when the voice recognition function is activated is uniformly executed by the default operation regardless of the function being executed in the electronic device. Accordingly, there is a problem in that the user has to utter a step-by-step voice command in order to execute a desired function.

According to various embodiments of the present disclosure, in a foldable electronic device, a first housing formed of a first surface and a third surface opposite to the first surface, a second surface extending from the first surface, and the second surface, A second housing formed with an opposite fourth surface, a first display formed on the first and second surfaces, a second display disposed on at least a portion of the third surface, a memory and a processor, The memory, upon execution, is intelligent in response to a folding event of the first housing and the second housing while the processor displays the execution screen of at least one app on at least one of the first display or the second display. A voice recognition guide including a specified context item in relation to the displayed app by running the app and analyzing the app status and context of the app screen based on the running information of the app displayed on at least one of the first display and the second display Instructions for outputting at least one of a message and a voice mode item may be stored.

According to various embodiments of the present disclosure, a method of driving a voice recognition function of an electronic device includes displaying an execution screen of at least one app on at least one of the first display or the second display; An operation of executing an intelligent app in response to a folding event of an electronic device including a housing, and based on the intelligent app, an app state and an app state based on execution information of the app displayed on at least one of the first display and the second display. An operation of analyzing a context of an app screen and an operation of outputting at least one of a voice recognition guide message including a context item related to the displayed app and a voice mode item.

According to various embodiments, by analyzing the context of an app screen based on a function running on an electronic device or a screen displayed on a display, and providing a customized voice recognition service in relation to the executed app, User convenience can be increased by assisting the optimized function to be more easily uttered.

1 is a block diagram of an electronic device in a network environment according to various embodiments of the present disclosure.
2A to 2C are diagrams illustrating a structure of an electronic device according to various embodiments of the present disclosure.
3 is a block diagram of an electronic device according to various embodiments of the present disclosure.
4A and 4B are diagrams illustrating a method of operating voice recognition of a foldable electronic device according to various embodiments.
5 illustrates an embodiment of driving a voice recognition function of a foldable electronic device according to various embodiments of the present disclosure.
6A and 6B illustrate examples of folding events instructing execution of an intelligent app by an electronic device according to various embodiments of the present disclosure.
7A and 7B illustrate an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.
8A and 8B illustrate an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.
9A and 9B illustrate an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.
10A and 10B illustrate an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.
11A and 11B illustrate an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.
12 is a diagram illustrating a method of operating voice recognition of a foldable electronic device according to various embodiments of the present disclosure.
13A and 13B illustrate an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.
14 illustrates a method of operating voice recognition of a foldable electronic device according to various embodiments of the present disclosure.
15 illustrates an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.

1 is a block diagram of an electronic device 101 in a network environment 100 according to various embodiments.

Referring to FIG. 1, in a network environment 100, the electronic device 101 communicates with the electronic device 102 through a first network 198 (for example, a short-range wireless communication network), or a second network 199 It is possible to communicate with the electronic device 104 or the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108. According to an embodiment, the electronic device 101 includes a processor 120, a memory 130, an input device 150, an audio output device 155, a display device 160, an audio module 170, and a sensor module ( 176, interface 177, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196, or antenna module 197 ) Can be included. In some embodiments, at least one of these components (eg, the display device 160 or the camera module 180) may be omitted or one or more other components may be added to the electronic device 101. In some embodiments, some of these components may be implemented as a single integrated circuit. For example, the sensor module 176 (eg, a fingerprint sensor, an iris sensor, or an illuminance sensor) may be implemented while being embedded in the display device 160 (eg, a display).

The processor 120, for example, executes software (eg, a program 140) to implement at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120. It can be controlled and can perform various data processing or operations. According to an embodiment, as at least part of data processing or operation, the processor 120 may convert commands or data received from other components (for example, the sensor module 176 or the communication module 190) into a volatile memory. ) 132, process commands or data stored in the volatile memory 132, and store result data in a non-volatile memory 134. According to an embodiment, the processor 120 is a main processor 121 (for example, a central processing unit (CPU) or an application processor (AP)), and a coprocessor that can be operated independently or together with the main processor 121 123 (eg, a graphic processing unit (GPU), an image signal processor (ISP), a sensor hub processor, or a communication processor (CP)). I can. Additionally or alternatively, the coprocessor 123 may be set to use lower power than the main processor 121 or to be specialized for a designated function. The secondary processor 123 may be implemented separately from the main processor 121 or as a part thereof.

The coprocessor 123 is, for example, on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or the main processor 121 While in the active (eg, application execution) state, together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display device 160, the sensor module 176) ), or at least some of the functions or states related to the communication module 190). According to an embodiment, the coprocessor 123 (eg, an image signal processor or a communication processor) may be implemented as a part of other functionally related components (eg, the camera module 180 or the communication module 190). have.

The memory 130 may store various types of data used by at least one component of the electronic device 101 (eg, the processor 120 or the sensor module 176 ). The data may include, for example, software (eg, the program 140) and input data or output data for commands related thereto. The memory 130 may include a volatile memory 132 or a nonvolatile memory 134.

The program 140 may be stored as software in the memory 130, and may include, for example, an operating system (OS) 142, a middleware 144, or an application 146. have.

The input device 150 may receive a command or data to be used for a component of the electronic device 101 (eg, the processor 120) from outside (eg, a user) of the electronic device 101. The input device 150 may include, for example, a microphone, a mouse, a keyboard, or a digital pen (eg, a stylus pen).

The sound output device 155 may output an sound signal to the outside of the electronic device 101. The sound output device 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback, and the receiver can be used to receive incoming calls. According to an embodiment, the receiver may be implemented separately from or as a part of the speaker.

The display device 160 may visually provide information to the outside of the electronic device 101 (eg, a user). The display device 160 may include, for example, a display, a hologram device, or a projector and a control circuit for controlling the device. According to an embodiment, the display device 160 includes a touch circuitry set to sense a touch, or a sensor circuit set to measure the strength of a force generated by the touch (for example, a pressure sensor). It may include.

The audio module 170 may convert sound into an electrical signal, or conversely, may convert an electrical signal into sound. According to an embodiment, the audio module 170 acquires sound through the input device 150, the sound output device 155, or an external electronic device (for example, an external electronic device directly or wirelessly connected to the electronic device 101). Sound can be output through the electronic device 102) (for example, a speaker or headphones).

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101, or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the detected state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometer sensor, a magnetic sensor, and an acceleration sensor. ), grip sensor, proximity sensor, color sensor (e.g. RGB (red, green, blue) sensor), IR (infrared) sensor, biometric sensor, temperature It may include a temperature sensor, a humidity sensor, or an illumination sensor.

The interface 177 may support one or more designated protocols that may be used to connect directly or wirelessly to an external electronic device (eg, the electronic device 102) of the electronic device 101. According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, a secure digital (SD) card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that a user can perceive through tactile or motor sensations. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture a still image and a video. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101. According to an embodiment, the power management module 188 may be implemented as at least a part of, for example, a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101. According to an embodiment, the battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 includes a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It is possible to support establishment and communication through the established communication channel. The communication module 190 operates independently of the processor 120 (eg, an application processor) and may include one or more communication processors supporting direct (eg, wired) communication or wireless communication. According to an embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range wireless communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg : A local area network (LAN) communication module, or a power line communication module) may be included. Among these communication modules, a corresponding communication module is a first network 198 (for example, a short-range communication network such as Bluetooth, Wi-Fi direct or IrDA (infrared data association)) or a second network 199 (for example, a cellular network, the Internet). Or, it is possible to communicate with an external electronic device through a computer network (for example, a telecommunication network such as a LAN or a wide area network (WAN)). These various types of communication modules may be integrated into one component (eg, a single chip), or may be implemented as a plurality of separate components (eg, multiple chips).

The wireless communication module 192 uses the subscriber information (eg, international mobile subscriber identity (IMSI)) stored in the subscriber identification module 196 to communicate with the first network 198 or the second network 199. The electronic device 101 can be identified and authenticated within the same communication network.

The antenna module 197 may transmit a signal or power to the outside (eg, an external electronic device) or receive from the outside. According to an embodiment, the antenna module 197 may include one antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas. In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is, for example, provided by the communication module 190 from the plurality of antennas. Can be chosen. The signal or power may be transmitted or received between the communication module 190 and an external electronic device through the at least one selected antenna. According to some embodiments, other components (eg, RFIC) other than the radiator may be additionally formed as part of the antenna module 197.

At least some of the components are connected to each other through a communication method (e.g., bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI))) between peripheral devices, and signal (E.g. commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199. Each of the electronic devices 102 and 104 may be a device of the same or different type as the electronic device 101.

According to an embodiment, all or part of the operations executed by the electronic device 101 may be executed by one or more of the external electronic devices 102, 104, or 108. For example, when the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or another device, the electronic device 101 In addition or in addition, a request may be made to the one or more external electronic devices 102 and 104 to perform the function or at least part of the service. The one or more external electronic devices 102 and 104 that have received the request execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit the execution result to the electronic device 101. I can deliver. The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this, for example, cloud computing, distributed computing, or client-server computing technology may be used.

The electronic device 101 according to various embodiments disclosed in this document may be a device of various types. The electronic device 101 may include, for example, a portable communication device (eg, a smartphone), a portable multimedia device, a portable medical device, a camera, a wearable device, a home appliance device, or a foldable electronic device. . The electronic device 101 according to the exemplary embodiment of the present document is not limited to the above-described devices.

2A to 2C are diagrams illustrating a structure of an electronic device according to various embodiments of the present disclosure.

2A to 2C, an electronic device (eg, the electronic device 101 of FIG. 1) according to various embodiments includes a flexible display that can be folded or unfolded, or has a hinge-based hinge structure. It may be implemented as a foldable electronic device that can fold or unfold the two displays.

According to an embodiment, the electronic device 101 includes a first housing 210 and a second surface 221 and a fourth surface 223 including a first surface 211 and a third surface 213. It may include a second housing 220 including ). When the first surface 211 of the first housing 210 and the second surface 221 of the second housing 220 are in the front direction 2001 of the electronic device 101, the first housing 210 The third surface 213 of) and the fourth surface 223 of the second housing 220 may represent a rear direction 2003 of the electronic device 101.

The first housing 210 and the second housing 220 are disposed on both sides around a folding axis (eg, A axis), and may have a shape that is generally symmetric with respect to the folding axis. For example, as shown in FIG. 2A, when the electronic device 101 is in an open state, in the front direction 2001), the first housing 210 is to the left of the electronic device 101 with respect to the folding axis. 2 The housing 220 may be the right side of the electronic device 101. The first housing 210 and the second housing 220 may be designed to be folded with respect to each other. For example, a hinge structure 260 is formed between the first housing 210 and the second housing 220 so that the front direction 2001 of the electronic device 101 may be folded.

A method in which the electronic device 101 is folded in the front direction 2001 may be understood as an in-folding method. Conversely, a method in which the electronic device 101 is folded in the rear direction 2003 may be understood as an out-folding method. Hereinafter, a description will be made based on the electronic device 101 implemented in an in-folding method, but the description is not limited thereto, and the same applies to the electronic device 101 implemented in an out-folding method. Can be applied.

The angle or distance between the first housing 210 and the second housing 220 may vary depending on whether the electronic device 101 is in an open (or open) state, a closed (or closed) state, or an intermediate state. have.

2A illustrates a front direction 2001 and a rear direction 2003 of the electronic device 101 in an open state.

The open state may mean an open state, a flat (or flat) state, or an unfolding state, unfolded state (eg, FIG. 2A ). The open state is a state in which the first housing 210 and the second housing 220 are arranged side by side, and may mean a state in which the electronic device 101 is fully unfolded. The open state may mean a state in which an angle between the first housing 210 and the second housing 220 is equal to or greater than a set angle. For example, the set angle may be 180 degrees, but may be changed according to the intention of the designer. When the electronic device 101 is in an open state, the first housing 210 and the second housing 220 may form a planar structure.

The closed state may mean a closed state or a folded state (eg, FIG. 2B). The closed state is a state in which the first housing 210 and the second housing 220 are disposed to face each other, and may mean a state in which the electronic device 101 is completely folded. The closed state may mean a state in which an angle between the first housing 210 and the second housing 220 is equal to or less than a set angle (eg, 0 degrees to 5 degrees). In the closed state, the first surface 211 of the first housing 210 and the second surface 221 of the second housing 220 may face each other.

The intermediate state is a state in which the first housing 210 and the second housing 220 are disposed at a predetermined angle, and the electronic device 101 may not be the open state or the closed state. The intermediate state means a state in which the first surface 211 of the first housing 210 and the second surface 221 of the second housing 220 form a range of a certain angle (eg, 6 degrees to 179 degrees). I can.

The electronic device 101 may form a first display 230 (eg, a main display) (eg, the display device 160 of FIG. 1) on the first surface 211 and the second surface 221. The first display 230 may be entirely formed on the first surface 211 of the first housing 210 and the second surface of the second housing 220.

The first display 230 may refer to a flexible display in which at least a partial area may be transformed into a flat or curved surface. The first display 230 may be folded left or right based on a folding axis (eg, A axis). According to an embodiment, the first display 230 may include a first display area corresponding to the first surface 211 or a second display area corresponding to the second surface 221. According to another embodiment, when a plurality of applications (hereinafter, apps) are executed, the first display 230 may include a two-part area, a three-part area, or a four-part area according to the number of executed apps.

The electronic device 101 may arrange the first camera 214 on the second surface 221. In the drawing, the first camera 214 is shown as one, but the first camera 214 may be formed in plural. Although the drawing shows that the first camera 214 is disposed on the second surface 221, the first camera 214 may be formed on the first surface 211.

The electronic device 101 forms a second display 240 (eg, a sub-display, a cover display) (eg, the display device 160 of FIG. 1) in a part of the direction opposite to the direction in which the first display 230 is disposed. can do. The second display 240 may be formed on at least a part of the third surface 213 of the electronic device 101. The electronic device 101 may include a plurality of second cameras (eg, 215, 217, 219, 225) on the rear surface 250 of the electronic device 210. The plurality of second cameras (eg, 215, 217, 219, 225) may have the same performance (eg, angle of view, resolution) or different. The electronic device 101 may further include a sensor area 241 on the fourth surface 223. Similar to the sensor module 176 of FIG. 1, an infrared sensor, a fingerprint sensor, or an illuminance sensor may be disposed in the sensor area 241.

According to an embodiment, when the electronic device 101 is open (eg, FIG. 2A ), the first display 230 is turned on (or activated), and the second display 240 is turned off ( Or deactivated). For example, when a user input (eg, touch, button selection) is not detected for a certain period of time (eg, 5 seconds, 10 seconds, 1 minute) while the first display 230 is turned on, the electronic device 101 May turn off the first display 230. For another example, when a user input (eg, touch, button selection) is detected in the second display 240 while the second display 240 is turned off, the electronic device 101 Can be turned on. For another example, when the second display 240 is turned on, the first display 230 may be turned off. Even if the second display 240 is turned on, the electronic device 101 maintains the first display 230 in the on state for a certain period of time, and the user input is not displayed on the first display 230 even after a certain period of time has elapsed. If not detected, the first display 230 may be turned off.

In the electronic device (eg, the electronic device 101 of FIG. 1) according to an embodiment, a hinge structure 260 is formed around a folding axis (eg, A-axis), so that the electronic device 101 may be in a closed state. have. 2B may show the first direction 2005 and the second direction 2007 when the electronic device 101 is closed. According to exemplary embodiments, when the electronic device 101 is closed, the first display 230 may be turned off and the second display 240 may be turned on. When a user input is not detected for a certain period of time while the second display 240 is turned on, the electronic device 101 may turn off the second display 240. When the electronic device 101 is in a closed state and the second display 240 is turned off, the electronic device 101 may turn on the second display 240 when an input signal set in the electronic device 101 is detected. have. After the electronic device 101 is in a closed state and the second display 240 is turned off, when a user input is detected on the second display 240, the electronic device 101 turns on the second display 240. I can make it.

3 is a block diagram of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 3, an electronic device (eg, the electronic device 101 of FIG. 1) according to various embodiments includes a first display 310 (eg, the first display 230 of FIG. 2A) and a second display ( 320) (e.g., the second display 240 of FIG. 2A), the processor 330 (e.g., the processor 120 of FIG. 1), the sensor module 340) (e.g., the sensor module 176 of FIG. 1) , A communication module 350 (eg, the communication module 190 of FIG. 1), and a memory 360 (eg, the memory 130 of FIG. 1).

According to an embodiment, the first display 310 and the second display 320 are under the control of the processor 330, an execution screen of an app (or an application program) or a graphic user interface. ) (GUI) can be displayed. For example, the first display 310 may be a flexible display made of a flexible material. The first display 310 terminates the operation when the electronic device 101 is folded around a folding axis (eg, A axis in FIG. 2A) (eg, in a closed state), and unfolds around the folding axis (eg. : Open state) can be operated. Conversely, when the electronic device 101 is folded around a folding axis (eg, closed), operates, and unfolds around a folding axis (eg, opened), the second display 320 terminates the operation. can do.

According to an embodiment, the sensor module 340 is applied to changes in shape (eg, folding and unfolding) of the first housing (eg, 210 in FIG. 2) and the second housing (eg, 220 in FIG. 2 ). At least one of a folding angle according to a folding angle, a folding type, a mounting direction, and a user's gaze may be detected. The sensor module 340 may measure the folding angle of the electronic device 101. The folding angle may mean an angle formed by the electronic device 101 between the first housing 210 and the second housing 220 with respect to the folding axis. According to an embodiment, the sensor module 340 may be configured on one side of the electronic device 101 (eg, a folding shaft, an end of the first or second housing, and a bottom of the first or second display (eg, under the panel). )), or bezel).

According to an embodiment, the communication module 350 may be configured to transmit and receive data by being connected to an external device.

According to an embodiment, the memory 360 may include an intelligent app and a plurality of apps. The intelligent app may constitute a framework (or solution program) for processing voice input. Each of the plurality of apps may include a plurality of operations for performing a specified function. According to an embodiment, a plurality of apps may be executed by the processor 330 to sequentially execute at least some of the plurality of operations.

According to an embodiment, the processor 330 may control the overall operation of the electronic device 101. The processor 330 may execute a program stored in the memory 360 to perform a designated function. The processor 330 may perform an operation for processing a voice input based on the intelligent app.

According to an embodiment, the processor 330 is based on a shape change (eg, folding and unfolding) of a first housing (eg, 210 in FIG. 2) and a second housing (eg, 220 in FIG. 2). This allows you to run intelligent apps to process voice input. For example, the processor 330 may execute an intelligent app when the folding angles of the first housing (eg, 210 of FIG. 2) and the second housing (eg, 220 of FIG. 2) satisfy a specified condition. As another example, the specified condition for running the intelligent app is that the folding angle of the first housing (e.g., 210 in Fig. 2) and the second housing (e.g., 220 in Fig. 2) are maintained for a specified time within a specified range. May contain conditions. The specified conditions can be changed according to the intention of the designer.

According to an embodiment, the processor 330 may execute an intelligent app for processing a voice input while at least one app is being executed and displayed on the first display 310 or the second display 320. For example, the processor 330 responds to a change in shape or a folding event of the first housing (eg, 210 in FIG. 2) and the second housing (eg, 220 in FIG. 2) while executing the schedule app. So you can run intelligent apps.

According to an embodiment, the processor 330 may analyze a state of an app and a context of an app screen based on execution information (or app data) of an app being executed on any one display. The processor 330 analyzes which app the user is currently running and which page the app is displaying, understands the intention related to the user's activity, and designates a meaningful context item related to the running app. I can. The processor 330 may execute the intelligent app in a context-based voice recognition mode related to the running app. The execution information of the app may include at least one of the type of the app, page number, execution screen information, execution function information, and metadata related to the app. According to an embodiment, the processor 330 may transmit execution information of an app being displayed on any one display to an intelligent app through an API.

The processor 330 may check execution information of the app being displayed on the first display 310 or the second display 320 based on the intelligent app, and analyze the state of the app. For example, the processor 330 may identify an app through app identification information (app_ID), check a task (or function) related to the app, and check which page is being displayed. The processor 330 may check data included in the currently displayed app page, and check whether the currently displayed app page includes user input data.

The processor 330 may analyze the context of the app screen and designate a meaningful context item. The context item can be set with intelligent parameters.

The processor 330 may generate a voice recognition guide message including a meaningful context item. The processor 330 may output at least one of a voice recognition guide message including a context item and a voice mode item through at least one of the first display 310, the second display 320, and a speaker.

According to an embodiment, the processor 330 acquires sensor information from the sensor module 340, the folding type of the first housing 310 and the second housing 320, the mounting direction of the electronic device 101, and the user At least one of the gaze information of can be checked. The processor 330 may determine a display to output at least one of the voice recognition guide message and the voice mode object based on sensor information.

According to an embodiment, the processor 330 divides the first display 310 (or the second display 320) based on execution of a plurality of apps, and controls to display respective app execution screens in the divided regions. can do. When the area of the first display 310 is divided, the processor 330 may set a divided position for designating a context item among divided areas based on execution of an intelligent app. The split position can be changed according to the designer's intention or user setting. The processor 330 may display a user interface for requesting an app selection on the first display 310 when a divided location for designating a context item among the divided areas is not set. The processor 330 may designate a context item by analyzing a state of an app running in a divided area selected by a user input and a context of an app screen.

According to an embodiment, when the intelligent app is executed, the processor 330 may activate a microphone and receive a user's utterance requesting a voice recognition service through the activated microphone. The processor 330 may configure a task (or operation) to process a user utterance based on an intelligent app and perform a function related to a context item in response to the user utterance. The intelligent app may configure a plan for performing a task corresponding to a user utterance or voice input under the control of the processor 330. The processor 330 performs the configured task and outputs information notifying that the task is being performed on the first display 310 or the second display 320 or through a speaker.

4A and 4B illustrate a method of executing a voice recognition function of a foldable electronic device according to various embodiments.

Referring to FIGS. 4A and 4B, according to an embodiment, a processor (eg, 330 of FIG. 3) of the electronic device 101 may execute at least one app in operation 410.

According to an embodiment, the processor 330 may execute at least one app installed or stored in the electronic device 101 in response to a user request or a preset event. The at least one app may include at least one of a contact app, a schedule app, a gallery app, a map app, a navigation app, a music app, and an internet browser app, but is not limited thereto.

According to an embodiment, the processor 330 is an execution screen or graphic user of an app (or application program) executed on at least one of the first display 310 and the second display 320. Display a graphic user interface (GUI).

For example, the processor 330 may display an app execution screen on the first display 310 while the electronic device 101 is open (for example, 2b ). Alternatively, the processor 330 may display an app execution screen on the second display 320 when the electronic device 101 is closed (eg, FIG. 2B ).

For another example, when a plurality of apps are executed while the electronic device 101 is open (for example, 2b), the processor 330 divides the display area of the first display 310 and executes the app in each of the divided areas. The screen can be displayed.

420 In operation, the processor 330 may detect a folding event instructing the execution of an intelligent app while displaying the execution screen of at least one app on at least one of the first display or the second display.

According to an embodiment, the folding event may be an event in which the folding angle of the first housing (eg, 210 in FIG. 2) and the second housing (eg, 220 in FIG. 2) is transformed from the first state to a designated second state. have. For example, the first shape may be at least one of an open state as shown in FIG. 3A or a state in which the displays are closed as illustrated in FIG. 3D. The second state may be an intermediate state that is folded with a certain angle. According to another example, the folding event may be an event in which a set predetermined time elapses after the folding angle is changed from the first state to the second state.

In operation 430, the processor 330 may execute the intelligent app in response to the folding event. According to an embodiment, the intelligent app analyzes user intention by processing the user's utterance, such as, for example, BIXBYTM, or goog l e assistantTM, and a function corresponding to the user intention, or It may include a speech recognition application capable of executing a service.

In operation 440, the processor 330 may analyze the state of the app displayed on the display (eg, the first display 310 or the second display 320) based on the execution of the intelligent app in response to the detection of the folding event. .

According to an embodiment, the processor 330 may analyze a state of an app based on execution information of an app running in a foreground or a top layer. The execution information of the app may include at least one of app identification information, execution screen (eg, page) information, execution function information, and metadata related to the app, but is not limited thereto, and includes all information related to the execution screen. can do.

According to an embodiment, the processor 330 executes an app Based on the information, it is possible to analyze the status of the app related to the user's activity by checking the type of the app being displayed on the display, the function being executed, the page number and/or the data included in the execution screen. For example, if the processor 330 is displaying the contact screen of a specific person on the display, the processor 330 identifies the contact app and confirms that the specific person page is being displayed in the contact app. You can analyze the state of the app.

In operation 450, the processor 330 may analyze the context of the app screen and designate a meaningful context item in relation to the app displayed on the display. The context item may correspond to a parameter of an intelligent app, and may mean content that is viewed, read, or created by a user. For example, the processor 330 checks the specific person information from the screen on which the specific person page of the contact app is displayed, analyzes the context of the app screen for the association between the specific person and the call function, and converts the specific person information as a context item. Can be specified. This is only an example, and the context item may be changed in some cases.

According to an embodiment, when the processor 330 cannot obtain the execution information of the app displayed on the display, the processor 330 captures the execution screen of the app displayed on the display, and data included in the data field of the screen form structure (for example, the name , Email, address, etc.), and analyzes the status of the app and the context of the app screen to designate a context item. According to another embodiment, the processor 330 may capture an app execution screen and obtain metadata (eg, location information) mapped to the screen information. The processor 330 may designate a context item by analyzing the state of the app and the context of the app screen based on data acquired from the captured execution screen. According to an embodiment, the processor 330 may transmit application execution information to an intelligent app through an application programming interface (API). For example, operations 440 to 460 may be operations executed based on an intelligent app, but are not limited thereto, and after the processor 330 analyzes the context of the app screen, the context item information is provided to the intelligent app. You may.

In operation 460, the processor 330 may output at least one of a voice recognition guide message including a context item related to the app and a voice recognition mode item through at least one of a display and a speaker. For example, the processor 330 analyzes the intention of the user to contact "Green" from the contact screen corresponding to "Green", and "Green" (eg: A voice recognition guide message including a context item) can be output.

According to an embodiment, the processor 330 may display a voice recognition user interface including at least one of a voice recognition guide message and a voice mode item on the display. For example, the processor 330 may switch to and display a voice recognition user interface, or may output a voice recognition user interface in the form of a pop-up on an app execution screen.

According to another embodiment, the processor 330 may output a guide voice corresponding to a voice recognition guide message including a context item through a speaker.

The user can recognize that the intelligent app has been executed through the display or speaker, and can more directly utter a voice command to execute a function related to a context item.

According to another embodiment, the processor 330 may perform operation 420 of FIG. and proceed to operation A as shown in FIG. 4B. After detecting the folding event in operation 420, the processor 330 may proceed to operation 435 to check the folding type of the electronic device 101. The processor 435 determines whether the electronic device 101 is in the in-folding state or the out-folding state by measuring the folding angles of the first housing 210 and the second housing 210 based on at least one sensor information. can do.

In operation 445, the processor 330 may check the mounting direction of the electronic device 101 and gaze information of the user based on the sensor information. For example, the processor 330 may determine whether the electronic device 101 is in a landscape mode or a portrait mode. In addition, the processor 330 may determine the display position viewed by the user based on the sensor information.

In operation 455, the processor 330 may determine a display to output at least one of a voice recognition guide message and a voice mode item based on at least one of a folding type, a mounting direction, and a user's location. Thereafter, the processor 330 may proceed to B and output at least one of a voice recognition guide message including a context item and a voice mode item to the determined display in operation 460.

5 illustrates an embodiment of driving a voice recognition function of a foldable electronic device according to various embodiments of the present disclosure.

Referring to FIG. 5, according to various embodiments, the electronic device 101 may execute a context-based intelligent app in which a user intention is reflected in relation to an executed app in response to a designated folding event while the app is being executed. .

According to an embodiment, as shown in 5001, the user makes an interaction for executing a contact app on the first display 530 in a state where the first housing 510 and the second housing 520 are open. You can do it. The processor of the electronic device 101 (eg, the processor 330 of FIG. 3) may display the app execution screen 535 on the first display 530 in response to a user interaction.

Alternatively, as shown in FIG. 2, the processor 330 displays a contact app execution screen on the second display 540 in response to a user interaction while the first housing 510 and the second housing 520 are closed. 545). For example, the app execution screens 535 and 545 may be screens related to a specific counterpart "Green" among contacts.

The processor 330 may receive a folding event instructing the execution of an intelligent app while a screen related to a specific counterpart "Green" among contacts is displayed.

In response to the folding event, the processor 330 displays a voice recognition user interface 550 including a context item (eg, Green) related to the app execution screens 535 and 545 on the second display 540 as shown in 5003. Can be printed. For example, the processor 330 analyzes the user's intention for the current execution screen based on the app execution information, designates "Green" as a context item, and a voice recognition guide message 551 including "Green" ( For example, "what do you want to do with Green?") may be displayed on the second display 540. Although the example of FIG. 5 has been described as displaying the voice recognition user interface 550 on the second display 540, the processor 330 displays the user interface including the voice mode item according to the setting or surrounding environment. The voice recognition guide message 551 displayed at 540 and including “Green” may output a speaker.

According to an embodiment, the processor 330 may generate a voice recognition guide message based on a default message form for the voice recognition guide message. For example, the default message form is "what do you want to do with blank?" In this case, the processor 330 may include a context item in a blank position. The default message form can be changed according to the type of app.

6A and 6B illustrate examples of a folding event instructing the execution of an intelligent app by an electronic device according to various embodiments of the present disclosure.

6A and 6B, according to an embodiment, the electronic device 101 is shown in FIG. 6001 when the first housing (eg, 210 in FIG. 2) and the second housing (eg, 220 in FIG. 2) are open. As shown in, it is possible to execute a context-based intelligent app related to the executed app in response to the folding event changed to various transformation types (A-1 and A-2, B-1, C-1 and C-2). have.

For example, A-1 and A-2 may be in a state in which a folding angle between the first housing and the second housing is less than or equal to a set angle (eg, in-folding state) in an open state. The electronic device 101 may display the voice recognition user interface on the second display (eg, 240 of FIG. 2) in response to the folding event changed to A-1 or A-2 in the open state. Hereinafter, an example in which the electronic device 101 displays a voice recognition user interface in response to a folding event will be described, but the electronic device 101 may output a voice recognition guide message including a context item in response to the folding event. .

For another example, the B-1 may be in a state in which the folding angle is vertical in the open state. The electronic device 101 may display a voice recognition user interface on the first display (eg, 230 of FIG. 2) in response to the folding event changed to B-1.

For another example, C-1 and C-2 may be in a state in which a folding angle between the first housing and the second housing is greater than or equal to a set angle (eg, an out-folding state) in an open state. The electronic device 101 may display the voice recognition user interface on any one of the first displays in response to a condition changed to C-1 or C-2 in the open state.

According to an embodiment, the electronic device 101 has various deformation types (D-1 and D-2, E-1, F-1 and F-) as shown in FIG. 6002 when the first housing and the second housing are closed. In response to the folding event changed to 2), a context-based intelligent app related to the executed app can be executed.

D-1 and D-2 may be in a state in which the folding angle of the first housing and the second housing is less than or equal to a set angle (eg, in-folding state) in the closed state, and the electronic device 101 is D-1 or In response to the condition changed to D-2, the voice recognition user interface may be displayed on the second display.

E-1 may be in a state in which the folding angle is vertical in the closed state, and the electronic device 101 may display the voice recognition user interface on the first display in response to the condition changed to E-1.

F-1 and F-2 may be in a state in which the folding angle of the first housing and the second housing is greater than or equal to a set angle (eg, out-folding state) in the closed state, and the electronic device 101 In response to the condition changed to F-2, the voice recognition user interface may be displayed on any one display area of the first display.

7A and 7B illustrate an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.

Referring to FIGS. 7A and 7B, according to an embodiment, a processor of the electronic device 101 (eg, the processor 330 of FIG. 3) may execute a contact app. The processor 330 displays the execution screen 735 of the contact app on the first display 730 as shown in 7001 with the first housing 710 and the second housing 720 open, or the execution screen 735 shown in 7002. As shown, in the closed state, the execution screen 745 of the contact app may be displayed on the second display 740. In the execution screens 735 and 745 of the app, display data may vary according to the size of the first display 730 or the second display 740.

The user may interact with the electronic device 101 (eg, a touch screen or an input device) to move to a green contact screen among stored contacts. The user may adjust the folding angle of the electronic device 101 in the form shown in FIG. 6A or 6B while the green contact screen is displayed.

The processor 330 may execute (or activate) an intelligent app in response to a folding event. The processor 330 may designate a context item 775, for example, "green" from the app execution screens 735 and 745 by analyzing the status of the app and the context of the app screen based on execution information of the running app. .

As illustrated in 7003, the processor 330 may output a voice recognition user interface 750 including green on a display (eg, at least one of the first display 730 or the second display 740 ).

The voice recognition user interface 750 may include a voice recognition guide message 751 (for example, “what do you want to do with Green?”) and a voice mode item 753, but in some cases, the voice recognition guide message 751 may be output as an audio signal through a speaker.

By intuitively checking the context item 775, the user can more directly utter a voice command related to the context item. For example, the user may utter a voice input to perform a function of an electronic device related to “green”.

According to an embodiment, as shown in 7004-1, when the processor 330 receives an utterance (eg, text her as I miss you) 760 from a user to transmit text to “green”, the user A text message (eg, I miss you) may be transmitted to “green” by processing the voice signal of.

The processor 330 may output notification information related to an operation for performing a task to the voice recognition user interface 750. For example, the processor 330 may output a voice input item 761 corresponding to a user utterance on the display before performing the task. Alternatively, the processor 330 may output information 762 guiding that a task is being performed through a speaker as an audio signal. The user may check whether the input speech is correctly recognized by the electronic device 101 according to the output of the notification information.

According to another embodiment, as shown in 7004-2, when the processor 330 receives an utterance (eg, call her) 770 from the user to connect the phone, the processor 330 may perform a phone connection to “green”. have. The processor 330 may output notification information about a phone connection task or a phone connection screen 771 on the display, and/or output voice information 772 indicating that a phone connection is in progress through a speaker.

According to another embodiment, as shown in 7004-3, when the processor 330 receives an utterance (eg, video call her) 780 from a user to connect a video call, the processor 330 connects a video call to “green”. You can do it. The processor 330 may output notification information 781 for the video call connection task on the display, and may output voice information 782 indicating that the video call connection is in progress through a speaker.

8A and 8B illustrate an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.

Referring to FIGS. 8A and 8B, according to an embodiment, a processor of the electronic device 101 (eg, the processor 330 of FIG. 3) may execute a gallery app. The processor 330 displays the execution screen 835 of the gallery app on the first display 830 in an open state, as shown in 8001 in response to the folding event of the first housing 810 and the second housing 820. Can be displayed. Alternatively, as shown in 8002, the processor 330 may display the execution screen 845 of the gallery app on the second display 840 in a closed state. The display of information on the execution screens 835 and 845 of the app may vary according to the size of the first display 830 or the second display 840.

The user may adjust the folding angle of the electronic device 101 in the form shown in FIG. 6A or 6B while the gallery app execution screen is displayed.

The processor 330 analyzes the state of the app and the context of the app screen based on the execution information of the gallery app based on the intelligent app executed in response to the folding event, and the context item 875, for example, "trip to Spain" can be specified. As shown in 8003, the processor 330 outputs a voice recognition user interface 850 including "trip to Spain" to a display (eg, at least one of the first display 830 or the second display 840) can do. The voice recognition user interface 850 related to the gallery app may include a voice recognition guide message 851 (eg, what do you want to do with this trip to Spain story?) and a voice mode item 853.

According to another embodiment, the processor 330 may propose a function that can be executed in conjunction with the gallery app through context analysis of the app screen, and guide information 852 about the proposed function (eg, share with your family). ?) may be included in the voice recognition user interface 850 and output.

The user can utter a voice to perform a function related to "trip to Spain".

According to an embodiment, as shown in 8004-1, when the processor 330 receives an utterance of viewing a slide show (eg, start slide show 860) from a user, the execution screen 861 for the slide show Alternatively, the processor 330 may output information 862 guiding that a task is being performed through a speaker as a voice.

According to another embodiment, as shown in 8004-2, when the processor 330 receives an utterance (eg, share with green) 870 from a user to share with a specific person, the processor 330 You can share (e.g. transfer). The processor 330 may output notification information 871 about the sharing task on a display or output voice information 872 indicating that a phone connection is being made through a speaker.

According to another embodiment, as shown in 8004-3, the processor 330 receives an answer speech (eg, yes) 880 for a suggested function from a user, or a family member as shown in 8004-3. If you receive an utterance (e.g., share with my family) (890) shared with my family, you can share (e.g., send) through the image displayed by the family. The processor 330 may output notification information 881,891 about the sharing task on a display or output voice information 882,892 indicating that sharing with a family member is being performed.

9A and 9B illustrate examples of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.

9A and 9B, according to an embodiment, the processor of the electronic device 101 (for example, the processor 330 of FIG. 3) is in a folded state of the first housing 910 and the second housing 920. Depending on, as shown in 9001, the execution screen 935 of the map app is displayed on the first display 930 in an open state, or as shown in 9002, the map app is displayed on the second display 940 in the closed state. The execution screen 945 may be displayed. The processor 330 may recognize that the map screen of California is displayed based on the app execution information.

The user may adjust the folding angle of the electronic device 101 in the form shown in FIG. 6A or 6B while the map app execution screen is displayed.

The processor 330 may execute (or activate) an intelligent app in response to a folding event. The processor 330 may analyze the status of the app and the context of the app screen based on the execution information of the gallery app, and designate a context item 975, for example, "California" from the execution screen of the app. The processor 330 displays a voice recognition user interface 950 including a voice recognition guide message 951 (eg, what do you want to do in California?) and a voice mode item 953 (eg, a first display). 930 or at least one of the second display 940).

The user can utter a voice to perform functions related to "California".

For example, as shown in 9004-1, the processor 330 performs a task according to the user's utterance when receiving a cafe recommendation utterance (eg, recommend a good cafe) 960 from the user. Therefore, it is possible to provide information on recommended cafes among cafes located around California. As shown in 9004-2, the processor 330 may output recommended cafe information 961 on a display or may output information 962 guiding a recommended cafe through a speaker as a voice.

According to another embodiment, as shown in 9004-2, when the processor 330 receives an utterance (eg, order 1 Americano at nearest Starbucks) 970 from a user to order a cafe drink, as shown in 9004-2, the processor 330 You can perform the task of searching and ordering drinks through the Starbucks Market. The processor 330 may output notification information 971 for an order task on a display or voice information 972 indicating that a phone connection is being made through a speaker.

10A and 10B illustrate examples of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.

10A and 10B, according to an embodiment, the processor of the electronic device 101 (for example, the processor 330 of FIG. 3) is in a folded state of the first housing 1010 and the second housing 1020. Depending on, as shown in 10001, the execution screen 1035 of the Internet browser is displayed on the first display 1030 in the open state, or as shown in 10002, the map app is displayed on the second display 1040 in the closed state. The execution screen 1045 may be displayed. The processor 330 may recognize that a "start game" has been searched for in an internet browser based on the app execution information.

The processor 330 displays a voice recognition user interface 1050 including a context item 1075, for example, a "start game", as shown at 10003 in response to the folding event shown in FIG. 6A or 6B. (Eg, at least one of the first display 1030 or the second display 1040). The voice recognition user interface 1050 may include a voice recognition guide message 1051 (eg, what do you want to do in start game?) and a voice mode item 1053. In addition, the processor 330 may propose a function that can be executed in relation to the start game (eg, movie reservation) through context analysis, and guide information 1052 on the proposed function (eg, RESERVE A FASTEST TICKET? ) May be included in the voice recognition user interface 1050 and output.

The user can utter various voices to perform functions related to the "start game".

According to an embodiment, as shown in 10004-1, the processor 330 receives an utterance (for example, RESERVE A FASTEST TICKET) or an utterance for approval of reservation (for example, YES) 1060 from a user, as shown in 10004-1. , A movie movie reservation task may be performed according to the user's speech. As shown in 10004-2, the processor 330 outputs information 1061 of the start game movie reservation at the earliest time to the voice recognition user interface 1050 or information 1062 guiding the movie reservation through a speaker. Can be output by voice.

According to another embodiment, as shown in 10004-2, when the processor 330 receives an utterance to add a reminder (eg, order 1 Americano at nearest Starbucks) 1070 from the user, the context for the app execution screen Through the analysis, a task of generating a reminder to check the scheduled release date of the smart game movie and to be notified on the scheduled release date may be performed. The processor 330 may output the notification information 1071 regarding the creation of the reminder to the voice recognition user interface 1050 or output the voice information 1072 guiding the generation of the reminder through a speaker.

11A and 11B illustrate examples of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.

11A and 11B, according to an embodiment, the processor of the electronic device 101 (eg, the processor 330 of FIG. 3) is in a folded state of the first housing 1110 and the second housing 1120. Depending on, as shown in 11001, the execution screen 1135 of the music app is displayed on the first display 1130 in the open state, or the music app is displayed on the second display 1140 in the closed state, as shown in 11002. The execution screen 1145 may be displayed.

When the user adjusts the folding angle of the electronic device 101 in the form shown in FIG. 6A or 6B, the processor 330 may execute (or activate) the voice recognition app according to a change in the folding state.

The processor 330 includes a context item 1175 designated through context analysis of the execution screen of the music app, for example, a voice recognition guide message 1151 including “LIPA” (eg, what do you want to do with Lipa? ) And the voice mode item 1153 may be output to a display (eg, at least one of the first display 1130 or the second display 1140) as shown in 11003. .

The user can utter various voices to perform functions related to “Lipa”.

According to an embodiment, as shown in 11004-1, the processor 330 plays back a sound source related to Lipa according to the user's utterance when receiving an utterance (eg, Play) 1160 called playback from the user. Can perform tasks. As shown in 1004-2, the processor 330 may output sound source reproduction information 1161 to a display or may output information 1162 guiding that a sound source is being reproduced through a speaker as a voice.

According to another embodiment, as shown in 11004-2, when the processor 330 receives an utterance (for example, show her latest album) 1170 from the user, as shown in 11004-2, the context for the app screen Through the analysis, you can perform the task of retrieving the last album for Lipa and providing an album list. The processor 330 may output the album information 1171 to a display or output voice information 1172 guiding the provision of an album through a speaker.

According to another embodiment, as shown in 11004-2, when the processor 330 receives an utterance (for example, tell me about her stories) 1180 from the user, "tell me about her stories", it is related to Lipa. It is possible to perform a task of searching for information and providing the searched Lipa information. The processor 330 may play a radio broadcast describing Lipa and output the radio execution information 1181 to a display, or may output audio information 1182 guiding radio play through a speaker.

12A and 12B illustrate an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.

12A and 12B, a processor of an electronic device according to an embodiment (eg, the processor 330 of FIG. 3) executes an intelligent speech recognition app in response to a folding event instructing the execution of the intelligent app in operation 1210. I can. Hereinafter, operations 1220 to 1270 may be operations based on an intelligent app.

In operation 1220, the processor 330 may determine whether a screen displayed on a display (eg, at least one of a first display or a second display) is an idle screen. In operation 1230, when the idle screen is displayed, the processor 330 may operate in a basic voice recognition mode based on the intelligent app. For example, the basic voice guidance mode may be a mode that provides a default voice recognition guide message.

In operation 1240, the processor 330 may receive a voice input while the intelligent voice recognition app is operating in a basic guide mode. In operation 1250, the processor 330 may determine whether the sound generation point exists within a preset distance based on the voice input. For example, the processor 330 may recognize the user's location based on the intensity of the received voice signal or sensor information (eg, an image sensor), and check a sound generation point corresponding to the user's location.

In operation 1260, when the sound generation point exists within a preset distance, the processor 330 may adjust the sound of the voice guidance for performing a task corresponding to the voice input to be reduced. In operation 1265, the processor 330 may adjust the sound of the voice guidance for performing a task corresponding to the voice input to increase when the sound generation point is out of a preset distance. For example, the electronic device may adjust the output intensity of the voice sound providing guidance information according to whether the user is in a close distance.

Meanwhile, in operation 1225, if the screen displayed on the display is not the standby screen, the processor 330 may determine that the app execution screen is displayed on the display. In operation 1270, when the app execution screen is displayed on the display, the processor 330 may operate in a context-based speech recognition mode related to the running app. For example, the processor 330 analyzes the app state based on the execution information of the app displayed on the display, analyzes the context of the app screen to designate a context item, and allows the context item to be used as a parameter of an intelligent app. Can be controlled. Hereinafter, a specific context-based speech recognition mode has been described in FIGS. 4A and 4B, and thus will be omitted. Thereafter, the processor 330 may proceed to operation 1240.

13A and 13B illustrate an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.

13A and 13B, a processor (eg, 330 in FIG. 3) of an electronic device according to various embodiments of the present disclosure is in a state in which the first housing 1310 and the second housing 1320 are open, as shown in 13001. In, the standby screen 1335 may be displayed on the first display 1330, or as shown in 13002, the standby screen 1345 may be displayed on the second display 1340 in a closed state.

In response to the folding event, the processor 330 displays a user interface 1350 corresponding to the basic voice recognition mode (for example, at least one of the first display 1330 or the second display 1340), as shown in 13003. ) Can be printed. The user interface 1350 corresponding to the basic voice recognition mode may include a default voice recognition guide message 1351 and a voice mode item 1352, but in some cases, the voice recognition guide message 1351 may be output through a speaker. May be.

The user may utter a voice input 1360 (eg, how is the weather today?) requesting a voice recognition service of the electronic device. The processor 330 may output text 1361 corresponding to the voice input to the user interface 1350. The processor 330 may perform a weather search task corresponding to a user's voice input and output weather information 1370 to the display. In addition, the processor 330 may output a voice 1371 guiding information related to weather through a speaker.

14 illustrates a method of operating voice recognition of a foldable electronic device according to various embodiments of the present disclosure.

Referring to FIG. 14, a processor (eg, 330 of FIG. 3) of an electronic device according to various embodiments may execute a plurality of apps in operation 1410. The processor 330 may display each app execution screen by dividing the display area of the display according to the execution of a plurality of apps.

In operation 1420, the processor 330 may detect a folding event instructing the execution of an intelligent recognition app in a state in which a plurality of apps are executed. In operation 1430, the processor 330 may determine whether a location set to designate a context item exists. In operation 1440, if the set location exists, the processor 330 may check execution information of the app running at the set location in response to the folding event. The processor 330 may designate a context item by analyzing an app state and a context of an app screen based on execution information of an app running at a set location.

In operation 1450, when there is no location set to designate a context item, the processor 330 may output a guide requesting a user selection for context analysis of the app screen on the display.

In operation 1455, the processor 330 may receive a user selection. You can check the execution information of the app running in the location selected in operation 1460. The processor may designate a context item by analyzing the app status and the context of the app screen based on the execution information of the app running in the selected location.

In operation 1470, the processor 330 may operate in a context-based speech recognition mode. The context-based speech recognition mode may include operations executed in FIGS. 4A and 4B.

15 illustrates an embodiment of driving a voice recognition function of an electronic device according to various embodiments of the present disclosure.

Referring to FIG. 15, a processor (eg, 330 in FIG. 3) of an electronic device according to various embodiments of the present disclosure includes a first housing 1510 and a second housing 1520 in an open state as illustrated in 15001. A plurality of application execution screens may be displayed on the display 1530. For example, when three apps are executed, the processor 330 may divide into a first area 1535, a second area 1356, and a third area 1537 to output an execution screen of each app. .

According to an embodiment, when a location for designating a context item is set as the first area 1535 while a plurality of app execution screens are being executed, the processor 330 responds to the folding event to the first area 1535. ), you can designate a context item by analyzing the app status and context of the app screen based on the running information of the app running in ).

According to another embodiment, when a designated location for a context item does not exist in a state in which a plurality of app execution screens are executed, the processor 330 responds to the folding event for context analysis as shown in 15002. Guide information 1547 for requesting user selection may be displayed on the display. For example, the processor 330 may output a display item that separates the first area, the second area and the third area, but is not limited to the screen shown in 15002, and a user who requests user selection in various forms Interface can be displayed.

For example, when the user selects the second area, the processor 330 analyzes the app status and the context of the app screen based on the execution information of the app running in the second area to provide a context item (eg, Lipa). And display a voice recognition user interface including lipa on the screen as shown in 15003. The voice recognition user interface may include a voice recognition guide message including lipa and a voice mode item. For example, the voice recognition user interface may be output as a full screen of the first display in response to a folding event, but is not limited thereto.

Various embodiments of the present document include one or more stored in a storage medium (eg, internal memory 136 or external memory 138) that can be read by a machine (eg, electronic device 101). It may be implemented as software (eg, program 140) including instructions. For example, the processor (eg, the processor 120) of the device (eg, the electronic device 101) may call and execute at least one command among one or more commands stored from a storage medium. This enables the device to be operated to perform at least one function according to the at least one command invoked. The one or more instructions may include code generated by a compiler or code that can be executed by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here,'non-transient' only means that the storage medium is a tangible device and does not contain a signal (e.g., electromagnetic waves). It does not distinguish between temporary storage cases.

According to an embodiment, a method according to various embodiments disclosed in the present document may be provided by being included in a computer program product. Computer program products can be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a device-readable storage medium (e.g., CD-ROM, compact disc read only memory), or through an application store (e.g., Play Store ^TM ) or two user devices (e.g. : It can be distributed (e.g., downloaded or uploaded) directly between smartphones) or online. In the case of online distribution, at least a part of the computer program product may be temporarily stored or temporarily generated in a storage medium that can be read by a device such as a server of a manufacturer, a server of an application store, or a memory of a relay server.

According to various embodiments, each component (eg, module or program) of the above-described components may include a singular number or a plurality of entities. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of constituent elements (eg, a module or program) may be integrated into one constituent element. In this case, the integrated component may perform one or more functions of each component of the plurality of components in the same or similar to that performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are sequentially, parallel, repeatedly, or heuristically executed, or one or more of the operations are executed in a different order, or , Omitted, or one or more other operations may be added.

101: electronic device
210: first housing
220: second housing
230,310: first display
240: 320: second display
330: processor
340: sensor module
350: communication module

Claims (20)

In the foldable electronic device,
A first housing formed with a first surface and a third surface opposite to the first surface;
A second housing formed with a second surface extending from the first surface and a fourth surface opposite to the second surface;
A first display formed on the first and second surfaces;
A second display disposed on at least a portion of the third surface;
Memory; And
Including a processor,
The memory, when executed, the processor,
While displaying the execution screen of at least one app on at least one of the first display or the second display, the intelligent app is executed in response to a folding event of the first housing and the second housing, and the first display and the second display Analyzing the app status and the context of the app screen based on the execution information of the app displayed on at least one of the displays to output at least one of a voice recognition guide message and a voice mode item including a specified context item in relation to the displayed app An electronic device that stores instructions. The method of claim 1,
The instructions, the processor,
The folding in which the execution screen of the at least one app executed on the first display is displayed while the first and second housings are open, and the folding angles of the first and second housings are deformed to a specified angle. An electronic device configured to designate a context item related to an app displayed on the first display in response to an event. The method of claim 1,
The instructions, the processor,
The folding event in which an execution screen of at least one app executed on the second display is displayed while the first and second housings are closed, and the folding angles of the first and second housings are deformed to a specified angle. In response to, an electronic device configured to designate a context item related to an app displayed on the second display. The method of claim 1,
The instructions, the processor,
In response to the folding event, an electronic device configured to transmit execution information of an app running in a foreground or a top layer of the first display or the second display to the intelligent app. The method of claim 1,
The application execution information includes at least one of identification information of the app, execution screen or page information, execution function information, metadata related to the app, and user input data related to the app. The method of claim 5,
The instructions, the processor,
Based on the intelligent app, the state of the app being displayed on the first display or the second display is analyzed based on the execution information of the app, the context of the app screen is analyzed to designate the context item, and the context item The electronic device further comprises instructions for generating a voice recognition guide message including the context item by setting as a parameter of an intelligent app. The method of claim 6,
The instructions, the processor,
When the execution information of the app cannot be obtained based on the intelligent app, the execution screen of the app being displayed on the first display or the second display is captured, and at least one text related to the execution screen of the captured app Or instructions for obtaining metadata to designate the context item. The method of claim 1,
The instructions, the processor,
An electronic device configured to display a voice recognition user interface including at least one of a voice recognition guide message including the context item and a voice mode item on a display on which an execution screen of the at least one app is displayed, based on the intelligent app . The method of claim 1,
Including more speakers,
The instructions, the processor,
Displaying a user interface including the voice mode item on the display of the execution screen of the at least one app based on the intelligent app, and outputting a voice recognition guide message including the context item to the speaker Electronic device. The method of claim 1,
Further comprising at least one sensor,
The instructions, the processor,
At least one of the voice recognition guide message and the voice mode object based on the folding type of the first housing and the second housing, the mounting direction of the electronic device, and gaze information of the user based on sensor information obtained from the at least one sensor. The electronic device further comprising instructions for determining a display to output one. The method of claim 1,
The instructions, the processor,
Based on the execution of a plurality of apps, when a split location to designate the context item when executing the intelligent app is set while the first display is split and each app execution screen is displayed in split regions, at the set split location The electronic device further comprising instructions for designating the context item by analyzing the status of the app and the context of the app screen based on execution information of the displayed app. The method of claim 11,
The instructions, the processor,
When a split location to designate the context item is not set when executing the intelligent app, a user interface for requesting an app selection for any one is displayed on the first display, and among the split areas based on a user selection input. The electronic device further comprising instructions for designating the context item from an execution screen of the selected app. The method of claim 11,
The instructions, the processor,
Further instructions for outputting a default voice recognition guide message when an intelligent app is executed in response to a folding event of the first housing and the second housing while the standby screen is displayed on the first display or the second display. Electronic device comprising. The method of claim 11,
Include more microphones,
The instructions, the processor,
The electronic device further comprises instructions for activating the microphone when the intelligent app is executed, and performing a function related to the context item based on a user utterance acquired through the activated microphone. The method of claim 14,
The instructions, the processor,
The electronic device further comprises instructions for adjusting a sound indicating that a function related to the context item is being performed based on a distance between the user's speech point acquired through the activated microphone and the electronic device. In a method of driving a voice recognition function of an electronic device,
Displaying an execution screen of at least one app on at least one of the first display and the second display;
Executing an intelligent app in response to a folding event of an electronic device including the first housing and the second housing;
Analyzing an app status and a context of an app screen based on the intelligent app, based on execution information of the app displayed on at least one of the first display and the second display; And
And outputting at least one of a voice recognition guide message and a voice mode item including a context item related to the displayed app. The method of claim 16,
The operation of analyzing the app state and the context of the app screen,
Designating the context item from an execution screen of an app being displayed on the first display or the second display;
Setting the context item as a parameter of an intelligent app; And
The method further comprising generating a voice recognition guidance message including the context item. The method of claim 16,
The operation of analyzing the app state and the context of the app screen,
Capturing an execution screen of the app being displayed on the first display or the second display when the execution information of the app cannot be obtained based on the intelligent app;
An operation of designating the context item by analyzing an app status and a context of the app screen based on at least one text or metadata related to the captured app execution screen, and
The method further comprising generating a voice recognition guidance message including the context item. The method of claim 16,
The operation of analyzing the app state and the context of the app screen,
Based on the execution of a plurality of apps, when a split location to designate the context item when executing the intelligent app is set while the first display is split and each app execution screen is displayed in split regions, at the set split location The method further comprising an operation of selecting the execution information of the displayed app, analyzing the app state and the context of the app screen, and designating the context item. The method of claim 16,
The operation of analyzing the app state and the context of the app screen
Based on the execution of a plurality of apps, while the first display is divided to display respective app execution screens in divided areas, when the intelligent app is executed, requesting an app selection when a divided location to designate the context item is not set. Displaying a user interface;
Receiving a user selection input; And
The method further comprising an operation of designating the context item by analyzing an app state and a context of an app screen based on execution information of an app selected from among the divided regions in response to the user selection input.

Images