WO2022197029A1

WO2022197029A1 - Electronic device and operation method of electronic device

Info

Publication number: WO2022197029A1
Application number: PCT/KR2022/003520
Authority: WO
Inventors: 공진아; 김도형; 노재연; 임용준; 진연주
Original assignee: 삼성전자 주식회사
Priority date: 2021-03-17
Filing date: 2022-03-14
Publication date: 2022-09-22
Also published as: KR20220129840A

Abstract

Disclosed is an electronic device comprising: a microphone; a communication circuit; a plurality of cameras; a sensor; a display; memory; and a processor operatively coupled to the microphone, the communication circuit, the plurality of cameras, the sensor, the display, and the memory. The memory can store one or more instructions which, upon execution, cause the processor to: receive voice data corresponding to an utterance of a user of the electronic device through the microphone; identify, in the voice data, a first keyword including place information; determine an interval of time associated with the time when the first keyword was uttered; acquire driving information, including speed information and/or direction information corresponding to the interval of time, by using the sensor; and obtain a first image corresponding to the place information, among images acquired using at least one of the plurality of cameras, the first image being obtained on the basis of the interval of time and the driving information. Various other embodiments identified through the specification are possible.

Description

Electronic device and method of operation of electronic device

Embodiments disclosed in this document relate to an electronic device and a method of operating the electronic device.

Infotainment is a compound word of information, meaning information, and entertainment, meaning entertainment. It may refer to a service that provides information related to driving of a vehicle, as well as entertainment elements for users. For example, a vehicle to which an infotainment system is applied can provide a service that not only guides a route through navigation, but also shares information on restaurants in the area in which it is being operated or incorporates IT technology to reserve a restaurant.

An electronic device including an infotainment system or an electronic device communicating with the infotainment system is being developed.

There may be a need for a convenient way for a user of an electronic device (eg, a driver or occupant of a vehicle) to store a location discovered while the vehicle is being driven. The conventional electronic device can support the function of capturing a camera while the vehicle is driving, but there is a problem in that the user cannot pass a place intended to be photographed by the user or cannot capture an image intended by the user due to the continuous movement of the vehicle.

In addition, needs to discover and visit new places other than previously registered landmarks are increasing.

Various embodiments disclosed in this document provide an electronic device for photographing a place discovered by a user of the electronic device while driving a vehicle and storing information related to the corresponding place, and an operating method thereof.

An electronic device according to an embodiment disclosed in this document includes a microphone, a communication circuit, a plurality of cameras, a sensor, a display, a memory, the microphone, the communication circuit, the plurality of cameras, the sensor, the display, and the memory. a processor operatively coupled, wherein the memory, when executed, causes the processor to receive voice data corresponding to a utterance of a user of the electronic device through the microphone, and to include location information from the voice data. Identifies a first keyword, determines a time section associated with the time when the first keyword is uttered, and uses the sensor to obtain driving information including at least one of speed information or direction information corresponding to the time section, , one or more instructions for acquiring a first image corresponding to the place information based on the time section and the driving information among images acquired using at least one of the plurality of cameras.

In addition, the method of operating an electronic device according to an embodiment disclosed in this document receives voice data corresponding to a utterance of a user of the electronic device through a microphone of the electronic device, and includes location information from the voice data driving including at least one of speed information or direction information corresponding to the time section by identifying a first keyword to The information may be acquired, and a first image corresponding to the place information may be acquired based on the time section and the driving information among images acquired using at least one of a plurality of cameras of the electronic device.

According to various embodiments disclosed in this document, a place discovered by a user while looking at the external environment in a driving vehicle can be stored (or registered) in an in-vehicle infotainment (IVI) system based on the user's utterance and utilized. An electronic device and an operating method thereof can be provided.

Also, according to various embodiments disclosed in this document, even if a vehicle passes a place that a user wants to store, an electronic device and an operating method thereof can be provided for identifying an excessive place and acquiring an image of the place.

In addition, various effects directly or indirectly identified through this document may be provided.

1 is a block diagram of an electronic device in a network environment, according to various embodiments of the present disclosure;

2 is a block diagram illustrating an integrated intelligence system according to an embodiment.

3 is a diagram illustrating a form in which relation information between a concept and an action is stored in a database, according to an exemplary embodiment.

4 is a diagram illustrating a user terminal displaying a screen for processing a voice input received through an intelligent app, according to an embodiment.

5 is a conceptual diagram illustrating an operation of an electronic device according to an exemplary embodiment.

6 is a block diagram of an electronic device according to an exemplary embodiment.

7 is a flowchart illustrating an operation of an electronic device according to an exemplary embodiment.

8 is a flowchart illustrating an operation of an electronic device according to an exemplary embodiment.

9 is a flowchart illustrating an operation of an electronic device according to an exemplary embodiment.

10 is a conceptual diagram illustrating an operation of an electronic device according to an exemplary embodiment.

11 is a diagram illustrating a user interface provided by an electronic device through a display according to an exemplary embodiment.

In connection with the description of the drawings, the same or similar reference numerals may be used for the same or similar components.

1 is a block diagram of an electronic device 101 in a network environment 100, according to various embodiments. Referring to FIG. 1 , in a network environment 100 , an electronic device 101 communicates with an electronic device 102 through a first network 198 (eg, a short-range wireless communication network) or a second network 199 . It may communicate with at least one of the electronic device 104 and the server 108 through (eg, a long-distance wireless communication network). According to an embodiment, the electronic device 101 may communicate with the electronic device 104 through the server 108 . According to an embodiment, the electronic device 101 includes a processor 120 , a memory 130 , an input module 150 , a sound output module 155 , a display module 160 , an audio module 170 , and a sensor module ( 176), interface 177, connection terminal 178, haptic module 179, camera module 180, power management module 188, battery 189, communication module 190, subscriber identification module 196 , or an antenna module 197 . In some embodiments, at least one of these components (eg, the connection terminal 178 ) may be omitted or one or more other components may be added to the electronic device 101 . In some embodiments, some of these components (eg, sensor module 176 , camera module 180 , or antenna module 197 ) are integrated into one component (eg, display module 160 ). can be

The processor 120, for example, executes software (eg, a program 140) to execute at least one other component (eg, a hardware or software component) of the electronic device 101 connected to the processor 120. It can control and perform various data processing or operations. According to one embodiment, as at least part of data processing or operation, the processor 120 converts commands or data received from other components (eg, the sensor module 176 or the communication module 190 ) to the volatile memory 132 . may be stored in , process commands or data stored in the volatile memory 132 , and store the result data in the non-volatile memory 134 . According to an embodiment, the processor 120 is the main processor 121 (eg, a central processing unit or an application processor) or a secondary processor 123 (eg, a graphic processing unit, a neural network processing unit (eg, a graphic processing unit, a neural network processing unit) a neural processing unit (NPU), an image signal processor, a sensor hub processor, or a communication processor). For example, when the electronic device 101 includes the main processor 121 and the sub-processor 123 , the sub-processor 123 uses less power than the main processor 121 or is set to be specialized for a specified function. can The auxiliary processor 123 may be implemented separately from or as a part of the main processor 121 .

The secondary processor 123 may, for example, act on behalf of the main processor 121 while the main processor 121 is in an inactive (eg, sleep) state, or when the main processor 121 is active (eg, executing an application). ), together with the main processor 121, at least one of the components of the electronic device 101 (eg, the display module 160, the sensor module 176, or the communication module 190) It is possible to control at least some of the related functions or states. According to an embodiment, the coprocessor 123 (eg, an image signal processor or a communication processor) may be implemented as part of another functionally related component (eg, the camera module 180 or the communication module 190 ). have. According to an embodiment, the auxiliary processor 123 (eg, a neural network processing device) may include a hardware structure specialized for processing an artificial intelligence model. Artificial intelligence models can be created through machine learning. Such learning may be performed, for example, in the electronic device 101 itself on which the artificial intelligence model is performed, or may be performed through a separate server (eg, the server 108). The learning algorithm may include, for example, supervised learning, unsupervised learning, semi-supervised learning, or reinforcement learning, but in the above example not limited The artificial intelligence model may include a plurality of artificial neural network layers. Artificial neural networks include deep neural networks (DNNs), convolutional neural networks (CNNs), recurrent neural networks (RNNs), restricted boltzmann machines (RBMs), deep belief networks (DBNs), bidirectional recurrent deep neural networks (BRDNNs), It may be one of deep Q-networks or a combination of two or more of the above, but is not limited to the above example. The artificial intelligence model may include, in addition to, or alternatively, a software structure in addition to the hardware structure.

The memory 130 may store various data used by at least one component (eg, the processor 120 or the sensor module 176 ) of the electronic device 101 . The data may include, for example, input data or output data for software (eg, the program 140 ) and instructions related thereto. The memory 130 may include a volatile memory 132 or a non-volatile memory 134 .

The program 140 may be stored as software in the memory 130 , and may include, for example, an operating system 142 , middleware 144 , or an application 146 .

The input module 150 may receive a command or data to be used by a component (eg, the processor 120 ) of the electronic device 101 from the outside (eg, a user) of the electronic device 101 . The input module 150 may include, for example, a microphone, a mouse, a keyboard, a key (eg, a button), or a digital pen (eg, a stylus pen).

The sound output module 155 may output a sound signal to the outside of the electronic device 101 . The sound output module 155 may include, for example, a speaker or a receiver. The speaker can be used for general purposes such as multimedia playback or recording playback. The receiver can be used to receive incoming calls. According to one embodiment, the receiver may be implemented separately from or as part of the speaker.

The display module 160 may visually provide information to the outside (eg, a user) of the electronic device 101 . The display module 160 may include, for example, a control circuit for controlling a display, a hologram device, or a projector and a corresponding device. According to an embodiment, the display module 160 may include a touch sensor configured to sense a touch or a pressure sensor configured to measure the intensity of a force generated by the touch.

The audio module 170 may convert a sound into an electric signal or, conversely, convert an electric signal into a sound. According to an embodiment, the audio module 170 acquires a sound through the input module 150 , or an external electronic device (eg, a sound output module 155 ) connected directly or wirelessly with the electronic device 101 . The electronic device 102) (eg, a speaker or headphones) may output a sound.

The sensor module 176 detects an operating state (eg, power or temperature) of the electronic device 101 or an external environmental state (eg, a user state), and generates an electrical signal or data value corresponding to the sensed state. can do. According to an embodiment, the sensor module 176 may include, for example, a gesture sensor, a gyro sensor, a barometric pressure sensor, a magnetic sensor, an acceleration sensor, a grip sensor, a proximity sensor, a color sensor, an IR (infrared) sensor, a biometric sensor, It may include a temperature sensor, a humidity sensor, or an illuminance sensor.

The interface 177 may support one or more specified protocols that may be used by the electronic device 101 to directly or wirelessly connect with an external electronic device (eg, the electronic device 102 ). According to an embodiment, the interface 177 may include, for example, a high definition multimedia interface (HDMI), a universal serial bus (USB) interface, an SD card interface, or an audio interface.

The connection terminal 178 may include a connector through which the electronic device 101 can be physically connected to an external electronic device (eg, the electronic device 102 ). According to an embodiment, the connection terminal 178 may include, for example, an HDMI connector, a USB connector, an SD card connector, or an audio connector (eg, a headphone connector).

The haptic module 179 may convert an electrical signal into a mechanical stimulus (eg, vibration or movement) or an electrical stimulus that the user can perceive through tactile or kinesthetic sense. According to an embodiment, the haptic module 179 may include, for example, a motor, a piezoelectric element, or an electrical stimulation device.

The camera module 180 may capture still images and moving images. According to an embodiment, the camera module 180 may include one or more lenses, image sensors, image signal processors, or flashes.

The power management module 188 may manage power supplied to the electronic device 101 . According to an embodiment, the power management module 188 may be implemented as, for example, at least a part of a power management integrated circuit (PMIC).

The battery 189 may supply power to at least one component of the electronic device 101 . According to one embodiment, battery 189 may include, for example, a non-rechargeable primary cell, a rechargeable secondary cell, or a fuel cell.

The communication module 190 is a direct (eg, wired) communication channel or a wireless communication channel between the electronic device 101 and an external electronic device (eg, the electronic device 102, the electronic device 104, or the server 108). It can support establishment and communication performance through the established communication channel. The communication module 190 may include one or more communication processors that operate independently of the processor 120 (eg, an application processor) and support direct (eg, wired) communication or wireless communication. According to one embodiment, the communication module 190 is a wireless communication module 192 (eg, a cellular communication module, a short-range communication module, or a global navigation satellite system (GNSS) communication module) or a wired communication module 194 (eg, : It may include a local area network (LAN) communication module, or a power line communication module). A corresponding communication module among these communication modules is a first network 198 (eg, a short-range communication network such as Bluetooth, wireless fidelity (WiFi) direct, or infrared data association (IrDA)) or a second network 199 (eg, legacy It may communicate with the external electronic device 104 through a cellular network, a 5G network, a next-generation communication network, the Internet, or a computer network (eg, a telecommunication network such as a LAN or a WAN). These various types of communication modules may be integrated into one component (eg, a single chip) or may be implemented as a plurality of components (eg, multiple chips) separate from each other. The wireless communication module 192 uses subscriber information (eg, International Mobile Subscriber Identifier (IMSI)) stored in the subscriber identification module 196 within a communication network such as the first network 198 or the second network 199 . The electronic device 101 may be identified or authenticated.

The wireless communication module 192 may support a 5G network after a 4G network and a next-generation communication technology, for example, a new radio access technology (NR). NR access technology includes high-speed transmission of high-capacity data (eMBB (enhanced mobile broadband)), minimization of terminal power and access to multiple terminals (mMTC (massive machine type communications)), or high reliability and low latency (URLLC (ultra-reliable and low-latency) -latency communications)). The wireless communication module 192 may support a high frequency band (eg, mmWave band) to achieve a high data rate, for example. The wireless communication module 192 uses various techniques for securing performance in a high-frequency band, for example, beamforming, massive multiple-input and multiple-output (MIMO), all-dimensional multiplexing. It may support technologies such as full dimensional MIMO (FD-MIMO), an array antenna, analog beam-forming, or a large scale antenna. The wireless communication module 192 may support various requirements defined in the electronic device 101 , an external electronic device (eg, the electronic device 104 ), or a network system (eg, the second network 199 ). According to an embodiment, the wireless communication module 192 may include a peak data rate (eg, 20 Gbps or more) for realizing eMBB, loss coverage (eg, 164 dB or less) for realizing mMTC, or U-plane latency for realizing URLLC ( Example: Downlink (DL) and uplink (UL) each 0.5 ms or less, or round trip 1 ms or less) can be supported.

The antenna module 197 may transmit or receive a signal or power to the outside (eg, an external electronic device). According to an embodiment, the antenna module 197 may include an antenna including a conductor formed on a substrate (eg, a PCB) or a radiator formed of a conductive pattern. According to an embodiment, the antenna module 197 may include a plurality of antennas (eg, an array antenna). In this case, at least one antenna suitable for a communication method used in a communication network such as the first network 198 or the second network 199 is connected from the plurality of antennas by, for example, the communication module 190 . can be selected. A signal or power may be transmitted or received between the communication module 190 and an external electronic device through the selected at least one antenna. According to some embodiments, other components (eg, a radio frequency integrated circuit (RFIC)) other than the radiator may be additionally formed as a part of the antenna module 197 .

According to various embodiments, the antenna module 197 may form a mmWave antenna module. According to one embodiment, the mmWave antenna module comprises a printed circuit board, an RFIC disposed on or adjacent to a first side (eg, bottom side) of the printed circuit board and capable of supporting a designated high frequency band (eg, mmWave band); and a plurality of antennas (eg, an array antenna) disposed on or adjacent to a second side (eg, top or side) of the printed circuit board and capable of transmitting or receiving signals of the designated high frequency band. can do.

At least some of the components are connected to each other through a communication method between peripheral devices (eg, a bus, general purpose input and output (GPIO), serial peripheral interface (SPI), or mobile industry processor interface (MIPI)) and a signal ( e.g. commands or data) can be exchanged with each other.

According to an embodiment, the command or data may be transmitted or received between the electronic device 101 and the external electronic device 104 through the server 108 connected to the second network 199 . Each of the external

electronic devices

102 or 104 may be the same as or different from the electronic device 101 . According to an embodiment, all or a part of operations executed in the electronic device 101 may be executed in one or more external

electronic devices

102 , 104 , or 108 . For example, when the electronic device 101 needs to perform a function or service automatically or in response to a request from a user or other device, the electronic device 101 may perform the function or service itself instead of executing the function or service itself. Alternatively or additionally, one or more external electronic devices may be requested to perform at least a part of the function or the service. One or more external electronic devices that have received the request may execute at least a part of the requested function or service, or an additional function or service related to the request, and transmit a result of the execution to the electronic device 101 . The electronic device 101 may process the result as it is or additionally and provide it as at least a part of a response to the request. For this purpose, for example, cloud computing, distributed computing, mobile edge computing (MEC), or client-server computing technology may be used. The electronic device 101 may provide an ultra-low latency service using, for example, distributed computing or mobile edge computing. In another embodiment, the external electronic device 104 may include an Internet of things (IoT) device. The server 108 may be an intelligent server using machine learning and/or neural networks. According to an embodiment, the external electronic device 104 or the server 108 may be included in the second network 199 . The electronic device 101 may be applied to an intelligent service (eg, smart home, smart city, smart car, or health care) based on 5G communication technology and IoT-related technology.

Referring to FIG. 2 , the integrated intelligent system according to an embodiment may include a user terminal 201 , an intelligent server 300 , and a service server 400 .

The user terminal 201 (eg, the electronic device 101 of FIG. 1 ) according to an embodiment may be a terminal device (or electronic device) connectable to the Internet, for example, a mobile phone, a smart phone, or a personal digital assistant (PDA). digital assistant), notebook computer, television (television), white goods, wearable device, head mounted device (HMD), or smart speaker. As another example, the user terminal 201 may be an electronic device that may be mounted in a vehicle, such as an in-vehicle infotainment (IVI) system.

According to the illustrated embodiment, the user terminal 201 may include a communication interface 290 , a microphone 270 , a speaker 255 , a display 260 , a memory 230 , and/or a processor 220 . have. The components listed above may be operatively or electrically connected to each other.

The communication interface 290 (eg, the communication module 190 of FIG. 1 ) may be configured to be connected to an external device to transmit/receive data. The microphone 270 (eg, the audio module 170 of FIG. 1 ) may receive a sound (eg, a user's utterance) and convert it into an electrical signal. The speaker 255 (eg, the sound output module 155 of FIG. 1 ) may output an electrical signal as a sound (eg, voice). The display 260 (eg, the display module 160 of FIG. 1 ) may be configured to display an image or video. The display 260 according to an embodiment may also display a graphic user interface (GUI) of an executed app (or an application program).

The memory 230 (eg, the memory 130 of FIG. 1 ) according to an embodiment may store the client module 231 , a software development kit (SDK) 233 , and a plurality of applications. The client module 231 and the SDK 233 may constitute a framework (or a solution program) for performing general functions. In addition, the client module 231 or the SDK 233 may configure a framework for processing a voice input.

The plurality of applications (eg, 355a and 355b) may be programs for performing a specified function. According to an embodiment, the plurality of applications may include a first app 235a and/or a second app 235b. According to an embodiment, each of the plurality of applications may include a plurality of operations for performing a specified function. For example, the applications may include an alarm app, a message app, and/or a schedule app. According to an embodiment, the plurality of applications may be executed by the processor 220 to sequentially execute at least some of the plurality of operations.

The processor 220 according to an embodiment may control the overall operation of the user terminal 201 . For example, the processor 220 may be electrically connected to the communication interface 290 , the microphone 270 , the speaker 255 , and the display 260 to perform a specified operation. For example, the processor 220 may include at least one processor.

The processor 220 according to an embodiment may also execute a program stored in the memory 230 to perform a designated function. For example, the processor 220 may execute at least one of the client module 231 and the SDK 233 to perform the following operations for processing a voice input. The processor 220 may control operations of a plurality of applications through, for example, the SDK 233 . The following operations described as operations of the client module 231 or the SDK 233 may be operations performed by the execution of the processor 220 .

The client module 231 according to an embodiment may receive a voice input. For example, the client module 231 may receive a voice signal corresponding to the user's utterance sensed through the microphone 270 . The client module 231 may transmit a received voice input (eg, a voice signal) to the intelligent server 300 . The client module 231 may transmit the state information of the user terminal 201 together with the received voice input to the intelligent server 300 . The state information may be, for example, execution state information of an app.

The client module 231 according to an embodiment may receive a result corresponding to the received voice input from the intelligent server 300 . For example, when the intelligent server 300 can calculate a result corresponding to the received voice input, the client module 231 may receive a result corresponding to the received voice input. The client module 231 may display the received result on the display 260 .

The client module 231 according to an embodiment may receive a plan corresponding to the received voice input. The client module 231 may display a result of executing a plurality of operations of the app according to the plan on the display 260 . The client module 231 may, for example, sequentially display execution results of a plurality of operations on the display. As another example, the user terminal 201 may display only a partial result of executing a plurality of operations (eg, a result of the last operation) on the display.

According to an embodiment, the client module 231 may receive a request for obtaining information necessary for calculating a result corresponding to a voice input from the intelligent server 300 . According to an embodiment, the client module 231 may transmit the necessary information to the intelligent server 300 in response to the request.

The client module 231 according to an embodiment may transmit result information of executing a plurality of operations according to the plan to the intelligent server 300 . The intelligent server 300 may confirm that the received voice input has been correctly processed using the result information.

The client module 231 according to an embodiment may include a voice recognition module. According to an embodiment, the client module 231 may recognize a voice input performing a limited function through the voice recognition module. For example, the client module 231 may execute an intelligent app for processing the voice input by performing an organic operation in response to a specified voice input (eg, wake up!).

The intelligent server 300 according to an embodiment receives information related to a user's voice input from the user terminal 201 through the network 299 (eg, the first network 198 and/or the second network 199 of FIG. 1 ). can receive According to an embodiment, the intelligent server 300 may change data related to the received voice input into text data. According to an embodiment, the intelligent server 300 may generate at least one plan for performing a task corresponding to the user's voice input based on the text data.

According to one embodiment, the plan may be generated by an artificial intelligent (AI) system. The artificial intelligence system may be a rule-based system, a neural network-based system (eg, a feedforward neural network (FNN)), and/or a recurrent neural network network(RNN))). Alternatively, it may be a combination of the above or other artificial intelligence systems. According to an embodiment, the plan may be selected from a set of predefined plans or may be generated in real time in response to a user request. For example, the artificial intelligence system may select at least one plan from among a plurality of predefined plans.

The intelligent server 300 of an embodiment may transmit a result according to the generated plan to the user terminal 201 or transmit the generated plan to the user terminal 201 . According to an embodiment, the user terminal 201 may display a result according to the plan on the display. According to an embodiment, the user terminal 201 may display the result of executing the operation according to the plan on the display.

Intelligent server 300 of an embodiment is a front end (front end) 310, natural language platform (natural language platform) 320, capsule database (capsule database) 330, execution engine (execution engine) 340, It may include an end user interface 350 , a management platform 360 , a big data platform 370 , or an analytics platform 380 .

The front end 310 according to an embodiment may receive a voice input received by the user terminal 201 from the user terminal 201 . The front end 310 may transmit a response corresponding to the voice input to the user terminal 201 .

According to an embodiment, the natural language platform 320 includes an automatic speech recognition module (ASR module) 321 , a natural language understanding module (NLU module) 323 , and a planner module ( planner module 325 , a natural language generator module (NLG module) 327 , and/or a text to speech module (TTS module) 329 .

The automatic voice recognition module 321 according to an embodiment may convert a voice input received from the user terminal 201 into text data. The natural language understanding module 323 according to an embodiment may determine the user's intention by using text data of the voice input. For example, the natural language understanding module 323 may determine the user's intention by performing syntactic analysis and/or semantic analysis. The natural language understanding module 323 according to an embodiment recognizes the meaning of a word extracted from a voice input using a linguistic feature (eg, a grammatical element) of a morpheme or phrase, and matches the meaning of the identified word to the intention of the user. You can decide your intentions.

The planner module 325 according to an embodiment may generate a plan using the intent and parameters determined by the natural language understanding module 323 . According to an embodiment, the planner module 325 may determine a plurality of domains required to perform a task based on the determined intention. The planner module 325 may determine a plurality of operations included in each of the plurality of domains determined based on the intention. According to an embodiment, the planner module 325 may determine a parameter required to execute the determined plurality of operations or a result value output by the execution of the plurality of operations. The parameter and the result value may be defined as a concept of a specified format (or class). Accordingly, the plan may include a plurality of actions and/or a plurality of concepts determined by the user's intention. The planner module 325 may determine the relationship between the plurality of operations and the plurality of concepts in stages (or hierarchically). For example, the planner module 325 may determine the execution order of the plurality of operations determined based on the user's intention based on the plurality of concepts. In other words, the planner module 325 may determine the execution order of the plurality of operations based on parameters required for execution of the plurality of operations and results output by the execution of the plurality of operations. Accordingly, the planner module 325 may generate a plan including related information (eg, ontology) between a plurality of operations and a plurality of concepts. The planner module 325 may generate a plan using information stored in the capsule database 330 in which a set of relationships between concepts and operations is stored.

The natural language generation module 327 according to an embodiment may change the specified information into a text form. The information changed to the text form may be in the form of natural language utterance. The text-to-speech conversion module 329 according to an embodiment may change information in a text format into information in a voice format.

According to an embodiment, some or all of the functions of the natural language platform 320 may be implemented in the user terminal 201 . For example, the user terminal 201 may include an automatic speech recognition module and/or a natural language understanding module. After the user terminal 201 recognizes the user's voice command, text information corresponding to the recognized voice command may be transmitted to the intelligent server 300 . For example, the user terminal 201 may include a text-to-speech module. The user terminal 201 may receive text information from the intelligent server 300 and output the received text information as voice.

The capsule database 330 may store information on relationships between a plurality of concepts and operations corresponding to a plurality of domains. A capsule according to an embodiment may include a plurality of action objects (or action information) and/or concept objects (or concept information) included in the plan. According to an embodiment, the capsule database 330 may store a plurality of capsules in the form of a concept action network (CAN). According to an embodiment, the plurality of capsules may be stored in a function registry included in the capsule database 330 .

The capsule database 330 may include a strategy registry in which strategy information necessary for determining a plan corresponding to a voice input is stored. The strategy information may include reference information for determining one plan when there are a plurality of plans corresponding to the voice input. According to an embodiment, the capsule database 330 may include a follow up registry in which information on a subsequent operation for suggesting a subsequent operation to the user in a specified situation is stored. The subsequent operation may include, for example, a subsequent utterance. According to an embodiment, the capsule database 330 may include a layout registry that stores layout information of information output through the user terminal 201 . According to an embodiment, the capsule database 330 may include a vocabulary registry in which vocabulary information included in the capsule information is stored. According to an embodiment, the capsule database 330 may include a dialog registry (dialog registry) in which dialog (or interaction) information with the user is stored. The capsule database 330 may update a stored object through a developer tool. The developer tool may include, for example, a function editor for updating an action object or a concept object. The developer tool may include a vocabulary editor for updating the vocabulary. The developer tool may include a strategy editor for creating and registering strategies for determining plans. The developer tool may include a dialog editor that creates a conversation with the user. The developer tool can include a follow up editor that can edit subsequent utterances that activate follow-up goals and provide hints. The subsequent goal may be determined based on a currently set goal, a user's preference, or an environmental condition. In an embodiment, the capsule database 330 may also be implemented in the user terminal 201 .

The execution engine 340 according to an embodiment may calculate a result using the generated plan. The end user interface 350 may transmit the calculated result to the user terminal 201 . Accordingly, the user terminal 201 may receive the result and provide the received result to the user. The management platform 360 according to an embodiment may manage information used in the intelligent server 300 . The big data platform 370 according to an embodiment may collect user data. The analysis platform 380 of an embodiment may manage the quality of service (QoS) of the intelligent server 300 . For example, the analysis platform 380 may manage the components and processing speed (or efficiency) of the intelligent server 300 .

The service server 400 according to an embodiment may provide a specified service (eg, food order or hotel reservation) to the user terminal 201 . According to an embodiment, the service server 400 may be a server operated by a third party. The service server 400 according to an embodiment may provide information for generating a plan corresponding to the received voice input to the intelligent server 300 . The provided information may be stored in the capsule database 330 . Also, the service server 400 may provide result information according to the plan to the intelligent server 300 . The service server 400 may communicate with the intelligent server 300 and/or the user terminal 201 via the network 299 . The service server 400 may communicate with the intelligent server 300 through a separate connection. Although the service server 400 is illustrated as one server in FIG. 2 , embodiments of the present document are not limited thereto. At least one of the

respective services

401 , 402 , and 403 of the service server 400 may be implemented as a separate server.

In the integrated intelligent system described above, the user terminal 201 may provide various intelligent services to the user in response to a user input. The user input may include, for example, an input through a physical button, a touch input, or a voice input.

In an embodiment, the user terminal 201 may provide a voice recognition service through an intelligent app (or a voice recognition app) stored therein. In this case, for example, the user terminal 201 may recognize a user utterance or a voice input received through the microphone, and provide a service corresponding to the recognized voice input to the user. .

In an embodiment, the user terminal 201 may perform a designated operation alone or together with the intelligent server and/or service server based on the received voice input. For example, the user terminal 201 may execute an app corresponding to the received voice input and perform a specified operation through the executed app.

In one embodiment, when the user terminal 201 provides a service together with the intelligent server 300 and/or the service server, the user terminal detects a user's utterance using the microphone 270, A signal (or voice data) corresponding to the sensed user's utterance may be generated. The user terminal may transmit the voice data to the intelligent server 300 using the communication interface 290 .

The intelligent server 300 according to an embodiment is a plan for performing a task corresponding to the voice input as a response to the voice input received from the user terminal 201, or performing an operation according to the plan. results can be generated. The plan may include, for example, a plurality of actions for performing a task corresponding to a user's voice input and/or a plurality of concepts related to the plurality of actions. The concept may define parameters input to the execution of the plurality of operations or result values output by the execution of the plurality of operations. The plan may include association information between a plurality of actions and/or a plurality of concepts.

The user terminal 201 according to an embodiment may receive the response using the communication interface 290 . The user terminal 201 outputs a voice signal generated inside the user terminal 201 using the speaker 255 to the outside, or an image generated inside the user terminal 201 using the display 260 to the outside. can be output as

3 is a diagram illustrating a form in which relation information between a concept and an operation is stored in a database, according to an embodiment.

The capsule database (eg, the capsule database 330 ) of the intelligent server 300 may store the capsule in the form of a concept action network (CAN). The capsule database may store an operation for processing a task corresponding to a user's voice input and parameters necessary for the operation in the form of a concept action network (CAN).

The capsule database may store a plurality of capsules (capsule A 331 and capsule B 334 ) corresponding to each of a plurality of domains (eg, applications). According to an embodiment, one capsule (eg, capsule A 331 ) may correspond to one domain (eg, location (geo), application). In addition, one capsule includes at least one service provider's capsule (eg, CP 1 (332), CP 2 (333), CP3 (335), and/or CP4 (336) for performing a function for a domain related to the capsule). )) can be matched. According to an embodiment, one capsule may include at least one operation 330a and at least one concept 330b for performing a specified function.

The natural language platform 320 may generate a plan for performing a task corresponding to the received voice input using the capsule stored in the capsule database 330 . For example, the planner module 325 of the natural language platform may generate a plan using a capsule stored in a capsule database. For example, using

operations

331a, 332a and

concepts

331b, 332b of capsule A 3310 and operations 334a and concept 334b of capsule B 334 to create plan 337 can do.

4 is a diagram illustrating a screen in which a user terminal processes a voice input received through an intelligent app according to an embodiment.

The user terminal 201 may execute an intelligent app to process a user input through the intelligent server 300 .

According to an embodiment, on the first screen 410 , when the user terminal 201 recognizes a specified voice input (eg, wake up!) or receives an input through a hardware key (eg, a dedicated hardware key), the voice You can run intelligent apps to process input. The user terminal 201 may, for example, run the intelligent app in a state in which the schedule app is running. According to an embodiment, the user terminal 201 may display an object (eg, an icon) 411 corresponding to an intelligent app on the display 260 . According to an embodiment, the user terminal 201 may receive a voice input by the user's utterance. For example, the user terminal 201 may receive a voice input saying "Tell me about this week's schedule!" According to an embodiment, the user terminal 201 may display a user interface (UI) 213 (eg, an input window) of an intelligent app in which text data of the received voice input is displayed on the display.

According to an embodiment, on the second screen 415 , the user terminal 201 may display a result corresponding to the received voice input on the display. For example, the user terminal 201 may receive a plan corresponding to the received user input and display 'this week's schedule' on the display according to the plan.

The electronic device according to various embodiments disclosed in this document may have various types of devices. The electronic device may include, for example, a portable communication device (eg, a smart phone), a computer device, a portable multimedia device, a portable medical device, a camera, a wearable device, or a home appliance device. The electronic device according to the embodiment of the present document is not limited to the above-described devices.

The various embodiments of this document and terms used therein are not intended to limit the technical features described in this document to specific embodiments, but it should be understood to include various modifications, equivalents, or substitutions of the embodiments. In connection with the description of the drawings, like reference numerals may be used for similar or related components. The singular form of the noun corresponding to the item may include one or more of the item, unless the relevant context clearly dictates otherwise. As used herein, "A or B", "at least one of A and B", "at least one of A or B", "A, B or C", "at least one of A, B and C", and "A , B, or C," each of which may include any one of the items listed together in the corresponding one of the phrases, or all possible combinations thereof. Terms such as "first", "second", or "first" or "second" may simply be used to distinguish an element from other elements in question, and may refer elements to other aspects (e.g., importance or order) is not limited. It is said that one (eg, first) component is “coupled” or “connected” to another (eg, second) component, with or without the terms “functionally” or “communicatively”. When referenced, it means that one component can be connected to the other component directly (eg by wire), wirelessly, or through a third component.

The term “module” used in various embodiments of this document may include a unit implemented in hardware, software, or firmware, and is interchangeable with terms such as, for example, logic, logic block, component, or circuit. can be used as A module may be an integrally formed part or a minimum unit or a part of the part that performs one or more functions. For example, according to an embodiment, the module may be implemented in the form of an application-specific integrated circuit (ASIC).

Various embodiments of the present document include one or more instructions stored in a storage medium (eg, internal memory 136 or external memory 138) readable by a machine (eg, electronic device 101). may be implemented as software (eg, the program 140) including For example, the processor (eg, the processor 120 ) of the device (eg, the electronic device 101 ) may call at least one of the one or more instructions stored from the storage medium and execute it. This makes it possible for the device to be operated to perform at least one function according to the called at least one command. The one or more instructions may include code generated by a compiler or code executable by an interpreter. The device-readable storage medium may be provided in the form of a non-transitory storage medium. Here, 'non-transitory' only means that the storage medium is a tangible device and does not contain a signal (eg, electromagnetic wave), and this term is used in cases where data is semi-permanently stored in the storage medium and It does not distinguish between temporary storage cases.

According to one embodiment, the method according to various embodiments disclosed in this document may be provided as included in a computer program product. Computer program products may be traded between sellers and buyers as commodities. The computer program product is distributed in the form of a machine-readable storage medium (eg compact disc read only memory (CD-ROM)), or via an application store (eg Play Store ^™ ) or on two user devices ( It can be distributed (eg downloaded or uploaded) directly or online between smartphones (eg: smartphones). In the case of online distribution, at least a portion of the computer program product may be temporarily stored or temporarily generated in a machine-readable storage medium such as a memory of a server of a manufacturer, a server of an application store, or a memory of a relay server.

According to various embodiments, each component (eg, a module or a program) of the above-described components may include a singular or a plurality of entities, and some of the plurality of entities may be separately disposed in other components. have. According to various embodiments, one or more components or operations among the above-described corresponding components may be omitted, or one or more other components or operations may be added. Alternatively or additionally, a plurality of components (eg, a module or a program) may be integrated into one component. In this case, the integrated component may perform one or more functions of each component of the plurality of components identically or similarly to those performed by the corresponding component among the plurality of components prior to the integration. . According to various embodiments, operations performed by a module, program, or other component are executed sequentially, in parallel, repeatedly, or heuristically, or one or more of the operations are executed in a different order, or omitted. , or one or more other operations may be added.

Hereinafter, an operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 5 .

5 is a conceptual diagram 500 for explaining an operation of an electronic device according to an exemplary embodiment. Operations of the electronic device (eg, the electronic device 101 of FIG. 1 , or the user terminal 201 of FIG. 2 ) described below are performed by the processor of the electronic device (eg, the processor 120 of FIG. 1 or the user terminal 201 of FIG. 2 ). processor 220).

Referring to FIG. 5 , the electronic device may receive voice data corresponding to a user's utterance through a voice input device (eg, a microphone) (eg, the input module 150 of FIG. 1 ). For example, the electronic device may receive a user's utterance "that valley looks good". According to an embodiment, a keyword 510 including place information may be included in the user's utterance. For example, the keyword 510 of "valley" may be included in the user's utterance "that valley looks good".

According to an embodiment, the electronic device may identify the keyword 510 from voice data corresponding to the user's utterance received through the voice input device. According to an embodiment, the electronic device may identify the keyword 510 from voice data based on a voice recognition technology. According to an embodiment, the identification of the keyword 510 by the electronic device may mean identification of place information included in the keyword 510 . According to an embodiment, the operation of the electronic device to identify the keyword 510 may correspond to the operation of the automatic speech recognition module 321 and the natural language understanding module 323 described above with reference to FIG. 2 .

According to an embodiment, the electronic device may determine the main utterance time 520 based on information on when the user's utterance is received. According to an embodiment, the electronic device includes a first time point 522 that is a start time of the utterance, a second time point 523 that is a time point at which the utterance of the keyword 510 is terminated, Alternatively, at least one of a third time point 524 that is an end time of the utterance may be identified. According to an embodiment, the electronic device may identify a fourth time point 521 that is a time point earlier than the first time point 522 by a specified time. The designated time may be, for example, n seconds (sec). According to an embodiment, the electronic device may determine the time from the fourth time point 521 to the second time point 523 as the main speech time 520 .

According to an embodiment, the electronic device excludes the time from the second time point 523 to the third time point 524 from the main speech time 520 to analyze information corresponding to the main speech time 520 . load can be reduced.

According to an embodiment, the electronic device may determine the interest interval 530 corresponding to the main utterance time 520 . According to an embodiment, the section of interest 530 may correspond to a section in which the vehicle 501 in which the electronic device is mounted moves during the main speech time 520 . According to an embodiment, the region of interest 530 may correspond to a distance and geographic range that the vehicle 501 in which the electronic device is mounted moves during the main utterance time 520 . For example, the electronic device may calculate the distance by multiplying the main ignition time 520 by the average driving speed of the vehicle 501 in which the electronic device is mounted during the main ignition time 520 .

For example, the electronic device determines that the vehicle 501 sets the main ignition time 520 based on driving information and global positioning system (GPS) information of the vehicle 501 in which the electronic device is mounted during the main ignition time 520 . It is possible to determine the geographic range moved during the period. According to an embodiment, the driving information may include at least one of speed information and direction information of the vehicle 501 . For example, the electronic device may acquire the driving information using on-board diagnostics (OBD). The OBD may be included in the electronic device or may be provided separately from the electronic device and may be mounted on the vehicle 501 on which the electronic device is mounted. According to an embodiment, the OBD may include various types of sensors. According to an embodiment, the electronic device may acquire GPS information using a GPS sensor. The GPS sensor may be included in the electronic device or provided separately from the electronic device to be mounted together in the vehicle 501 on which the electronic device is mounted.

According to an embodiment, the section of interest 530 may correspond to a section of an image (eg, a still image or a moving picture) obtained by the electronic device using a camera. The camera may be included in the electronic device or may be provided separately from the electronic device and mounted together in the vehicle 501 on which the electronic device is mounted. For example, the camera may include a camera set to photograph an external environment of the vehicle 501 and a camera set to photograph an internal environment of the vehicle 501 . According to an embodiment of the present disclosure, while the vehicle of interest 501 moves through the region of interest 530 based on an image corresponding to the region of interest 530 among images acquired using a camera set to capture the internal environment, the electronic device It is possible to identify the gaze direction of the driver or occupant of the vehicle. According to an embodiment, the electronic device may designate a camera set to photograph an external environment corresponding to the identified driver's or passenger's gaze direction. For example, the electronic device may designate the front camera as the gaze direction of the driver or passenger is identified as the front of the vehicle among cameras that photograph the front, side, and rear of the vehicle, respectively.

According to an embodiment, the electronic device may extract an image corresponding to place information included in the keyword 510 from an image corresponding to the ROI 530 among images acquired using a specified camera. For example, the electronic device may extract a valley image from an image captured by the front camera while the vehicle 501 moves the ROI 530 . Hereinafter, a geographic location corresponding to the place information included in the keyword 510 within the interest section 530 may be referred to as a place of interest.

According to various embodiments, the electronic device is transmitted from an external storage device (eg, a cloud server) (eg, the server 108 of FIG. 1 ) connected to the electronic device through a communication circuit (eg, the communication module 190 of FIG. 1 ). An image of a place of interest can be acquired. For example, the electronic device may acquire a clearer image from the external storage device as it is determined that the quality of the extracted image is not clear.

According to an embodiment, the electronic device may obtain information related to a place of interest from an external storage device. According to an embodiment, the external storage device may store information related to a place. For example, the external storage device may store at least one of an address, a place name, a phone number, business hours, an image, a visitor review, or place preference information. According to an embodiment, the electronic device may extract information related to a place of interest from information stored in an external storage device. According to an embodiment, the electronic device may receive information related to the extracted POI from an external storage device.

According to an embodiment, the electronic device may generate metadata of an image of the POI based on the received information related to the POI. According to an embodiment, the metadata includes not only information related to the place of interest received by the electronic device from the external storage device, but also utterance information of a user (eg, a driver or a passenger), keyword information related to the place of interest, and an image of the place of interest. It may include at least one of location and direction information of the vehicle 501 at the point in time when , and map information of the ROI 530 . According to an embodiment, the electronic device may store an image of a place of interest and metadata of the image. For example, the electronic device may store the generated metadata in association with an image of a place of interest.

According to an embodiment, the electronic device may output an image of a place of interest through an image output device (eg, a display) (eg, the display module 160 of FIG. 1 ). According to an embodiment, the electronic device may provide metadata of an image in response to receiving a user input for selecting an image of a place of interest. For example, the electronic device displays at least one of address information of the POI, keyword information related to the POI, utterance information of a user related to the POI, or review information of a visitor who has visited the POI on a display, or through a speaker. It can be output by voice.

According to an embodiment, the electronic device may provide a navigation service for moving to the POI in response to receiving a user's input for selecting an image of the POI.

Hereinafter, a configuration and operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 6 .

6 is a block diagram 600 of an electronic device according to an embodiment.

Referring to FIG. 6 , the electronic device 601 (eg, the electronic device 101 of FIG. 1 , or the user terminal 201 of FIG. 2 ) includes a microphone 610 (eg, the input module 150 of FIG. 1 ); Or the microphone 270 of FIG. 2), the communication circuit 620 (eg, the communication module 190 of FIG. 1, or the communication interface 290 of FIG. 2), the first camera 630 (eg, the communication module 190 of FIG. 1) Camera module 180), second camera 640 (eg, camera module 180 of FIG. 1 ), sensor 650 (eg, sensor module 176 of FIG. 1 ), display 660 (eg: Display module 160 of FIG. 1 , or display 260 of FIG. 2 ), memory 670 (eg, memory 130 of FIG. 1 , or memory 230 of FIG. 2 ), and processor 680 ( For example, the processor 120 of FIG. 1 or the processor 220 of FIG. 2 ) may be included. According to one embodiment, the processor 680 includes a microphone 610 , a communication circuit 620 , a first camera 630 , a second camera 640 , a sensor 650 , a display 660 , and a memory 670 . ) can be operatively connected to The processor 680 may perform operations described below by executing one or more instructions stored in the memory 670 .

According to an embodiment, the microphone 610 may receive a user's utterance and convert it into voice data. According to an embodiment, the processor 680 may receive voice data corresponding to the user's utterance through the microphone 610 .

According to an embodiment, the communication circuit 620 may be configured to be connected to the external storage device 602 to transmit/receive data. According to an embodiment, the external storage device 602 may include a cloud server. According to an embodiment, the external storage device 602 may store map data. According to an embodiment, the external storage device 602 may store information associated with a place. For example, the information associated with the place stored in the external storage device 602 may include at least one of an address, a place name, a phone number, business hours, images, visitor reviews, or place preference information of the place registered on the map. can According to various embodiments, the external storage device for storing map data and the external storage device for storing information related to a location are different external storage devices, and the communication circuit 620 may communicate with all of the different external storage devices. .

According to an embodiment, the electronic device 601 may include an IVI system mounted on a vehicle. According to an embodiment, the first camera 630 may be a camera set to photograph the internal environment of the vehicle. According to an embodiment, the first camera 630 may include at least one camera. According to an embodiment, the first camera 630 may be set to recognize at least one of an eye movement, a gesture, and a head movement of a user of the electronic device 601 (eg, a driver or an occupant of a vehicle). According to an embodiment, the second camera 640 may be a camera set to photograph the external environment of the vehicle. According to an embodiment, the second camera 640 may include at least one camera. For example, the second camera 640 may include at least one camera set to photograph an external environment corresponding to at least one direction of the front, side, and rear of the vehicle.

According to an embodiment, the electronic device 601 may include a driver monitoring solutions (DMS) image storage 690 and a driving image storage 695 . According to an embodiment, the processor 680 may store the image of the driver or passenger of the vehicle captured by the first camera 630 in the DMS image storage 690 . According to an embodiment, the processor 680 may store an image of the external environment of the vehicle captured by the second camera 640 in the driving image storage 695 . According to an embodiment, the processor 680 may store an image captured by the first camera 630 or the second camera 640 in each storage in real time.

According to an embodiment, the sensor 650 may measure (or sense) the state of the vehicle including the speed or direction of the vehicle. For example, the sensor 650 may include at least one of various sensors such as a temperature sensor, a pressure sensor, an acceleration sensor, a proximity sensor, or a gyro sensor. According to an embodiment, the processor 680 may obtain driving information of the vehicle including at least one of speed information and direction information of the vehicle using the sensor 650 . According to an embodiment, the driving information may mean on-board diagnostics (OBD) information.

According to an embodiment, the sensor 650 may include a global positioning system (GPS) sensor. According to an embodiment, the processor 680 may obtain location information of the vehicle using a GPS sensor. According to an embodiment, the location information may include real-time location information.

According to an embodiment, the display 660 may visually provide information to the user of the electronic device 601 . According to an embodiment, the processor 680 may display the user's POI list on the display 660 . According to an embodiment, the processor 680 may obtain an image of the POI intended by the user's utterance, and may provide a list of POIs including the acquired image of the POI through the display 660 . According to an embodiment, the processor 680 may provide information related to the selected POI in response to receiving a user's input for selecting one of the images of the POI.

According to an embodiment, the processor 680 may receive voice data corresponding to the user's utterance through the microphone 610 . According to an embodiment, the processor 680 may identify a first keyword (eg, the keyword 510 of FIG. 5 ) including place information from the voice data. According to an embodiment, the first keyword may be a place keyword. According to an embodiment, with respect to a method for the processor 680 to identify the first keyword, the method for identifying the keyword 510 of the electronic device described above with reference to FIG. 5 may be equally applied.

According to an embodiment, the processor 680 may determine a time interval associated with the time the first keyword was uttered. According to an embodiment, a point in time (eg, the keyword 510 of FIG. 5 ) that is advanced by a specified time from the start time (eg, the first time point 522 of FIG. 5 ) of an utterance including the first keyword (eg, the keyword 510 of FIG. 5 ) A time interval (eg, FIG. 5 ) from the fourth time point 521 of FIG. 5 to the point at which the utterance of the first keyword is terminated (eg, the second time point 523 of FIG. 5 ) It can be determined as the main ignition time 520) of .

According to an embodiment, the processor 680 may obtain driving information of a vehicle (eg, the vehicle 501 of FIG. 5 ) by using the sensor 650 . According to an embodiment, the vehicle may mean a vehicle on which the electronic device 601 is mounted. According to an embodiment, the driving information may include at least one of speed information and direction information of the vehicle. For example, the processor 680 may calculate the average driving speed and the traveling direction of the vehicle during the time section based on the sensing information of the sensor 650 .

According to an embodiment, the processor 680 may determine interest section information corresponding to the time section. The ROI information to be described below may correspond to the ROI 530 described above with reference to FIG. 5 . According to an embodiment, the interest section information may include information about an image corresponding to a time section in an image acquired using the first camera 630 or the second camera 640 . According to an embodiment, the image acquired using the first camera 630 or the second camera 640 may be a real-time image. For example, an image obtained by using the first camera 630 or the second camera 640 may include an image corresponding to each viewpoint included in the time section. According to an embodiment, the processor 680 may determine the interest section information corresponding to the time section based on the time section. According to an embodiment, the processor 680 may determine section information of an image corresponding to a time section in an image acquired using the first camera 630 or the second camera 640 . According to an embodiment, the processor 680 may determine section information of an image corresponding to a time section in the image stored in the DMS image storage 690 or the driving image storage 695 .

According to an embodiment, the interest section information may include information on a distance and a geographic range that the vehicle on which the electronic device 601 is mounted moves during a time section. According to an embodiment, the processor 680 may determine the interest section information corresponding to the time section based on the time section and the driving information. According to an embodiment, the processor 680 may determine information on the distance and geographic range the vehicle moves during the time period based on at least one of speed information and direction information of the vehicle during the time period. For example, the processor 680 may calculate the distance traveled by the vehicle during the time period by multiplying the average driving speed of the vehicle during the time period and the time corresponding to the time period.

As another example, the processor 680 may determine a geographic range in which the vehicle moves during the time period based on the traveling speed and the traveling direction of the vehicle during the time period. According to an exemplary embodiment, the geographic range to which the vehicle moves may vary depending on the case where the vehicle only goes straight and when the vehicle turns left after going straight. As another example, the processor 680 may determine a geographic range in which the vehicle moved during the time interval based on GPS information of the vehicle during the time interval. As described above, the processor 680 may acquire GPS information of the vehicle using the GPS sensor.

According to an embodiment, the processor 680 may obtain the user's eye tracking information by using the first camera 630 . According to an embodiment, the processor 680 may identify and track the gaze direction of the user based on at least one of the user's eye movement, gesture, and head movement recognized using the first camera 630 . According to an embodiment, the processor 680 may determine whether a user's specified motion event is detected from an image corresponding to a time section among images acquired using the first camera 630 . According to an embodiment, the designated movement event may include a case in which the user shows an eye movement, a gesture, or a head movement that is not related to driving of a vehicle. For example, the specified movement event may include a case in which the user looks at a side mirror or a window regardless of vehicle operation, or gazes at a specific point for a specified period of time or more, or views a specified number of times or more regardless of vehicle operation. According to an embodiment, the processor 680 may identify the gaze direction of the user at the point in time when the specified movement event is detected.

According to an embodiment, the processor 680 may designate at least one of the second cameras 640 based on the user's eye tracking information obtained using the first camera 630 . According to an embodiment, the second camera 640 may include at least one camera set to capture an image corresponding to at least one direction of the external environment of the vehicle. For example, the second camera 640 may include at least one camera set to capture an external environment image corresponding to at least one direction of the front, side, and rear of the vehicle. According to an embodiment, the processor 680 may designate at least one camera corresponding to the user's gaze direction identified by using the first camera 630 among the second cameras 640 . For example, when the user's gaze direction identified using the first camera 630 is the front of the vehicle, the processor 680 may designate a camera set to photograph the front of the vehicle among the second cameras 640 . .

According to an embodiment, the processor 680 may acquire a first image corresponding to place information included in the first keyword among images acquired using at least one designated second camera. According to an embodiment, the processor 680 may acquire the first image based on the interest section information and the first keyword. According to an embodiment, the processor 680 may acquire a first image in a section of an image corresponding to a time section among images acquired using at least one designated second camera. According to an embodiment, the first image may be an image corresponding to place information included in the first keyword.

According to an embodiment, the processor 680 selects a first image in a section of an image corresponding to a section of interest (eg, section of interest 530 in FIG. 5 ) among images acquired using at least one designated second camera. can be obtained According to an embodiment, the processor 680 acquires a first image from an image acquired using at least one designated second camera while the vehicle moves in the ROI (eg, the ROI 530 in FIG. 5 ). can do. The ROI may correspond to a distance and geographic range that the vehicle moves during the time period.

According to an embodiment, the processor 680 selects the first image corresponding to the place information included in the first keyword based on the interest section information and the first keyword among the images acquired using at least one designated second camera. can be obtained According to an embodiment, the processor 680 is configured to perform a first keyword in a section of an image corresponding to a time section (eg, the main speech time 220 of FIG. 5 ) among images acquired using at least one designated second camera. A first image (eg, valley image) may be acquired in a section of an image corresponding to place information (eg, valley) included in (eg, keyword 510 of FIG. 5 ). According to an embodiment, the processor 680 is configured with a first keyword ( Example: A first image (eg, valley image) corresponding to place information (eg, valley) included in the keyword 510 of FIG. 5 ) may be acquired.

According to an embodiment, the processor 680 uses data stored in the external storage device 602 that communicates with the electronic device 601 through the communication circuit 620 to perform a first operation based on the interest section information and the first keyword. A place of interest corresponding to the place information included in the keyword may be determined. According to an embodiment, the POI is a geographic region corresponding to place information (eg, valley) included in a first keyword (eg, keyword 510) within an ROI (eg, ROI 530 in FIG. 5). It can mean location. According to an embodiment, the processor 680 may receive information related to the place of interest from the external storage device 602 through the communication circuit 620 . The information related to the place of interest may include, for example, at least one of an address, a place name, a phone number, business hours, an image, a visitor review, a related keyword, and place preference information of the place of interest.

According to an embodiment, the processor 680 may generate metadata of the first image based on information related to the place of interest received from the external storage device 602 . According to an embodiment, the metadata of the first image may include information related to the place of interest received by the processor 680 from the external storage device 602 . According to an embodiment, the metadata of the first image may include at least one of information about a user's utterance for obtaining the first image, or information on the location and direction of the vehicle at the time the first image is captured. According to an embodiment, the processor 680 may store the generated metadata of the first image in association with the first image.

According to an embodiment, the processor 680 may display the first image through the display 660 . According to an embodiment, the processor 680 may display metadata of the first image on the display 660 in response to receiving an input of a user who selects the first image. According to an embodiment, the processor 680 may provide a service using metadata of the first image in response to receiving an input of a user who selects the first image. For example, the processor 680 may display on the display 660 a map indicating a route to a place of interest corresponding to the first image by using address information among metadata of the first image. According to an embodiment, the electronic device 601 provides a navigation service for guiding a route to a place of interest without receiving an input for inputting an address of a place of interest corresponding to an image in response to receiving an input of a user for selecting an image. can provide

In the above-described embodiment, a method in which the electronic device 601 provides information related to a place of interest through the display 660 is exemplified, but the present invention is not limited thereto. A device (eg, a speaker) may also provide information related to a place of interest.

According to an embodiment, the electronic device 601 may store (or register) a place found by the user while looking at the external environment in the driving vehicle in the IVI system based on the user's utterance and utilize it. According to an embodiment, the electronic device 601 determines a section of interest based on a speech keyword and a speech time, and analyzes a driving image corresponding to the section of interest, so that even if the vehicle passes the place the user wants to store, the place is too much. can be identified and an image of the corresponding place can be obtained.

According to an embodiment, the electronic device 601 designates a time section determined to be the main speech time (eg, the major speech time 220 of FIG. 5 ) among the total speech time, and analyzes only the image corresponding to the designated time section. By doing so, the size of the image data to be analyzed can be reduced.

According to an embodiment, the electronic device 601 may reduce the size of image data to be analyzed by designating a camera corresponding to the user's gaze direction among the second cameras 640 and analyzing only images captured by the designated camera. have.

According to an embodiment, the electronic device 601 enables the user to intuitively recognize the POI by providing an image-based POI list. According to an embodiment, the image-based POI list may be referred to as a gallery-type POI list.

Hereinafter, an operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 7 .

7 is a flowchart 700 illustrating an operation of an electronic device according to an exemplary embodiment. Operations of the electronic device described below may be performed by the electronic device 601 of FIG. 6 or the processor 680 of the electronic device 601 .

In operation 701, the electronic device may receive voice data corresponding to the user's utterance. According to an embodiment, the electronic device may receive voice data corresponding to the utterance of the user of the electronic device through a microphone (eg, the microphone 610 of FIG. 6 ).

In operation 703, the electronic device may identify the first keyword from the voice data. According to an embodiment, the first keyword may include place information. According to an embodiment, the first keyword (eg, the keyword 510 of FIG. 5 ) may be a place keyword (eg, “valley”). According to an embodiment, the method described above with reference to FIG. 5 may be equally applied to a method for the electronic device to identify the first keyword from voice data.

In operation 705, the electronic device may determine a time interval associated with the time when the first keyword was uttered. According to an embodiment, the electronic device is a time point (eg, a time point (eg, keyword 510 of FIG. 5 ) that is earlier than a start time (eg, a first time point 522 of FIG. 5 ) of an utterance including the first keyword (eg, keyword 510 of FIG. 5 )) For example: a time interval (eg, the main utterance time 520 of FIG. 5 ) from the fourth time point 521 of FIG. 5 to the point at which the first keyword is uttered (eg, the second time point 523 of FIG. )) can be determined. According to an embodiment, the designated time may be N seconds.

In operation 707, the electronic device may obtain driving information corresponding to the time section. According to an embodiment, the electronic device may be mounted (or applied) to a vehicle. According to an embodiment, the electronic device may obtain driving information of the vehicle using a sensor (eg, the sensor 650 of FIG. 6 ). According to an embodiment, the sensor 650 may measure (or sense) the state of the vehicle including the speed or direction of the vehicle. For example, the sensor 650 may include at least one of various sensors such as a temperature sensor, a pressure sensor, an acceleration sensor, a proximity sensor, or a gyro sensor. According to an embodiment, the driving information may mean on-board diagnostics (OBD) information. According to an embodiment, the electronic device may obtain driving information including at least one of speed information and direction information corresponding to a time section by using a sensor. According to an embodiment, the electronic device may acquire driving information corresponding to a time section based on driving information of a vehicle acquired in real time using a sensor. For example, the electronic device may acquire average driving speed information and traveling direction information of the vehicle during a time section by using the sensor.

In operation 709, the electronic device may acquire a first image based on a time section and operation information among images acquired using at least one of a plurality of cameras. According to an embodiment, the first image may correspond to place information included in the first keyword. According to an embodiment, the electronic device may determine interest section information corresponding to the time section. According to an embodiment, the electronic device may determine the interest section information corresponding to the time section based on the time section and the driving information. According to an embodiment, the interest section information is in a time section in an image acquired using a camera (eg, the first camera 630 of FIG. 6 or a second camera (eg, the second camera 640 of FIG. 6 )). According to an embodiment, the electronic device corresponds to a time section in an image acquired using a camera (eg, the first camera 630 or the second camera 640 ). It is possible to determine the section information of the video to be used.

According to an embodiment, the ROI information may include information on a distance and a geographic range that a vehicle on which the electronic device is mounted moves during a time period. According to an embodiment, the electronic device may determine information on the distance and geographic range the vehicle moves during the time period based on the driving information during the time period. For example, the electronic device may calculate the distance traveled by the vehicle during the time period by multiplying the average driving speed of the vehicle during the time period and a time corresponding to the time period. As another example, the electronic device may determine the geographic range in which the vehicle moves during the time period based on the traveling speed and the traveling direction of the vehicle during the time period.

According to an embodiment, the electronic device may designate at least one of a plurality of cameras. According to an embodiment, the plurality of cameras may include a first camera (eg, the first camera 630 of FIG. 6 ) and a plurality of second cameras (eg, the second camera 640 of FIG. 6 ). . According to an embodiment, the electronic device may designate at least one camera among the plurality of second cameras based on the user's eye tracking information acquired using the first camera. According to an embodiment, the first camera may be set to recognize at least one of an eye movement, a gesture, and a head movement of the user. According to an embodiment, the electronic device may acquire the user's eye tracking information based on at least one of an eye movement, a gesture, and a head movement of the user recognized using the first camera. For example, the electronic device may identify and track the user's gaze direction using the first camera.

According to an embodiment, the electronic device may detect a user's specified movement event from the image of the first camera corresponding to the ROI information. According to an embodiment, the image of the first camera corresponding to the ROI information may mean an image corresponding to a time period among images acquired using the first camera. According to an embodiment, the user's designated movement event may include a case in which the user shows an eye movement, a gesture, or a head movement that is not related to driving of a vehicle. For example, the specified movement event may include a case in which the user looks at a side mirror or a window regardless of vehicle operation, or gazes at a specific point for a specified period of time or more, or views a specified number of times or more regardless of vehicle operation. According to an embodiment, the electronic device may identify the gaze direction of the user at the point in time when the specified motion event is detected.

According to an embodiment, the electronic device detects a user's designated motion event from the image of the first camera corresponding to the ROI information, and selects a camera corresponding to the user's gaze direction at the point of occurrence of the designated motion event from among the second cameras. You can choose. According to an embodiment, the second camera may be set to photograph an external environment of a vehicle in which the electronic device is mounted. According to an embodiment, the plurality of second cameras may include at least one camera set to capture an image corresponding to at least one direction (eg, front, side, or rear) of the external environment of the vehicle. According to an embodiment, the electronic device may designate at least one camera corresponding to the user's gaze direction identified by using the first camera among the plurality of second cameras.

According to an embodiment, the electronic device may acquire a first image among images acquired by using at least one designated second camera. According to an embodiment, the electronic device may acquire the first image corresponding to the place information based on the interest section information and the first keyword among the images acquired using at least one designated second camera. According to an embodiment, the electronic device acquires the first image in a section of an image corresponding to a time section (eg, the main speech time 520 of FIG. 5 ) among images acquired using at least one designated second camera. can do. According to an embodiment, the electronic device acquires the first image in a section of an image corresponding to a section of interest (eg, section of interest 530 in FIG. 5 ) among images acquired using at least one designated second camera. can According to an embodiment, the ROI may correspond to a distance and geographic range that a vehicle on which the electronic device is mounted moves during a time period.

According to an embodiment, the electronic device may acquire a first image corresponding to place information included in a first keyword among images corresponding to an ROI among images acquired using at least one designated second camera. For example, when the place information included in the first keyword is 'valley', a valley image may be acquired from an image corresponding to an interest section among images acquired using at least one designated second camera.

Hereinafter, an operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 8 .

8 is a flowchart 800 illustrating an operation of an electronic device according to an exemplary embodiment. Operations of the electronic device described below may be performed by the electronic device 601 of FIG. 6 or the processor 680 of the electronic device 601 .

In operation 801 , the electronic device may activate a microphone (eg, the microphone 610 of FIG. 6 ). According to an embodiment, the electronic device may activate a microphone to receive the user's utterance. According to an embodiment, the electronic device may receive voice data corresponding to the user's utterance through the activated microphone.

In operation 803, the electronic device may analyze voice data. According to an embodiment, the electronic device may identify a keyword including specified information from voice data corresponding to the user's utterance received through the microphone.

In operation 805, the electronic device may determine whether a place keyword is recognized. According to an embodiment, the place keyword may be a keyword including place information. According to an embodiment, the place keyword may correspond to the keyword 510 of FIG. 5 . According to an embodiment, the electronic device may recognize (or identify) a keyword including place information from voice data. According to an embodiment, upon determining that the place keyword is recognized from the voice data (operation 805 - YES), the electronic device performs operation 807, and upon determining that the place keyword is not recognized from the voice data (operation 805 - NO) ), operation 803 may be performed again.

In operation 807, the electronic device may determine a main utterance time. According to an embodiment, the main utterance time may correspond to the main utterance time 520 of FIG. 5 . According to an embodiment, the method of determining the main utterance time by the electronic device may be the same as the method of determining the time period by the electronic device described above with reference to FIGS. 6 and 7 . For example, the electronic device may determine, as the main utterance time, from a point in time earlier by a specified time from the start point of the speech including the place keyword to the point in time when the speech of the place keyword is ended.

In operation 809 , the electronic device may obtain information on the average speed and traveling direction of the vehicle during the main ignition time. According to an embodiment, the electronic device may be mounted in a vehicle. According to an embodiment, the electronic device may obtain OBD information including at least one of speed information and direction information of a vehicle on which the electronic device is mounted by using a sensor (eg, the sensor 650 of FIG. 6 ). For example, the electronic device may obtain information on the average speed and traveling direction of the vehicle during the main ignition time based on OBD information for the main ignition time among OBD information for the entire driving time of the vehicle.

In operation 811, the electronic device may determine an ROI. According to an embodiment, the ROI may correspond to the ROI 530 of FIG. 5 . According to an embodiment, the electronic device may determine the interest section based on the main ignition time and OBD information of the vehicle during the main ignition time. For example, the electronic device may calculate the distance traveled by the vehicle during the main ignition time by multiplying the main ignition time by the average speed of the vehicle during the main ignition time. According to an embodiment, the electronic device may calculate a geographic range in which the vehicle moves during the main ignition time based on the average speed and the traveling direction of the vehicle during the main ignition time. According to an embodiment, the section of interest may mean a section in which the vehicle moves during the main talk time, and may correspond to a distance and geographic range that the vehicle moves during the main talk time.

In operation 813, the electronic device may detect a user's specified motion event from the DMS image corresponding to the ROI. According to an embodiment, the DMS image is acquired by the electronic device using a first camera (eg, the first camera 630 of FIG. 6 ) and stored in a database (eg, the DMS image storage 690 of FIG. 6 ). It could be a video. According to an embodiment, the DMS image may be an image captured by a driver or occupant of a vehicle on which the electronic device is mounted.

According to an embodiment, the electronic device may recognize at least one of an eye movement, a gesture, and a head movement of the user based on the DMS image. The electronic device may detect a user's specified movement event based on the recognized user's movement information. According to an embodiment, the user's designated movement event may include the user's movement regardless of the operation of the vehicle. For example, in the DMS image, the electronic device recognizes the movement of the user looking at the side mirror or the window regardless of the vehicle's operation, gazing at a specific point for a specified time or more, or viewing the user's view more than a specified number of times regardless of the vehicle's operation. It may be determined that a designated motion event has occurred.

According to an embodiment, as the electronic device detects the user's specified movement event, the electronic device may identify the user's gaze direction at the point in time when the specified movement event occurs. For example, the electronic device may identify the direction in which the user looked regardless of the driving of the vehicle.

According to an embodiment, the electronic device performs operation 815 upon detecting the user's specified movement event (operation 813 - YES), and does not detect the user's specified movement event (operation 813 - NO), 817 can be performed.

In operation 815, the electronic device may designate a camera corresponding to the user's gaze direction. According to an embodiment, the electronic device may designate at least one camera corresponding to the user's gaze direction among a plurality of second cameras (eg, the second camera 640 of FIG. 6 ). According to an embodiment, the plurality of second cameras may include at least one camera set to capture an image corresponding to at least one direction of an external environment of a vehicle on which the electronic device is mounted. According to an embodiment, the electronic device may designate a camera corresponding to the user's gaze direction among the at least one second camera.

In operation 817, the electronic device may designate all cameras. According to an embodiment, the electronic device may designate all of the plurality of second cameras (eg, the second camera 640 of FIG. 6 ).

In operation 819, the electronic device may acquire an image corresponding to the place of speech from the driving image corresponding to the ROI. According to an embodiment, the driving image may be an image obtained by the electronic device using the second camera and stored in a database (eg, the driving image storage 695 of FIG. 6 ). According to an embodiment, the driving image may be an image obtained by photographing an external environment while driving of a vehicle in which the electronic device is mounted. According to an embodiment, the driving image corresponding to the section of interest may mean an external environment image obtained by the electronic device using the second camera while the vehicle is moving during the main ignition time. Here, the second camera may be a camera designated when the electronic device performs operation 815 or operation 817 .

According to an embodiment, the place of speech may mean place information included in the place keyword. According to an embodiment, the electronic device may acquire an image corresponding to place information included in a place keyword from an image corresponding to a section of interest among images acquired using a designated second camera. For example, when the place of utterance is 'valley', the electronic device may acquire an image of the valley from an image corresponding to a section of interest among images acquired using a designated second camera.

In operation 821, the electronic device may determine whether the acquired image is clear. According to an embodiment, when determining that the image is clear (operation 821 - YES), the electronic device performs operation 825, and after determining that the image is not clear (operation 821 - NO), after performing operation 823 Operation 825 may be performed.

In operation 823, the electronic device may acquire an image from an external storage device (eg, the external storage device 602 of FIG. 6 ) (eg, a cloud server). According to an embodiment, the external storage device may store map data and information related to at least one place registered in the map. For example, the external storage device may store at least one of an address, a place name, a phone number, business hours, images, visitor reviews, and place preference information of a place registered on a map. According to an embodiment, the electronic device may receive an image corresponding to a utterance place within an ROI from an external storage device through a communication circuit (eg, the communication circuit 620 of FIG. 6 ).

According to an embodiment, the electronic device may determine the geographic location of the utterance within the ROI by using data stored in the external storage device. According to an embodiment, the electronic device uses location information (eg, GPS information) of the vehicle at a point in time at which an image of a place of speech in the section of interest acquired by using the designated second camera is captured to determine the location of speech in the section of interest. geographic location can be determined. The location information of the vehicle may be obtained using a GPS sensor (eg, the sensor 650 of FIG. 6 ) of the electronic device. According to an embodiment, the geographic location of the utterance place within the ROI may be referred to as a POI.

According to an embodiment, the electronic device may receive an image of a place of interest stored in an external storage device through a communication circuit.

In operation 825, the electronic device may correct the image. According to an embodiment, the electronic device may correct an image of the POI acquired using the second camera or an image of the POI received from an external storage device. According to an embodiment, the image of the place of interest may be cropped to have a relatively narrow angle of view.

In operation 827, the electronic device may generate metadata. According to an embodiment, the electronic device may generate metadata of an image of a place of interest. According to an embodiment, the electronic device may receive information related to a place of interest from an external storage device through a communication circuit. According to an embodiment, the information associated with the place of interest may include at least one of an address, a place name, a phone number, business hours, an image, a visitor review, or place preference information of the place of interest. According to an embodiment, the electronic device may generate metadata of an image of the POI based on information related to the POI received from the external storage device. According to an embodiment, the metadata includes not only information related to the place of interest received by the electronic device from the external storage device, but also utterance information of a user (eg, a driver or a passenger), keyword information related to the place of interest, and an image of the place of interest. It may include at least one of location and direction information of the vehicle at the point in time of photographing , or map information of a section of interest.

In operation 829, the electronic device may store the image in association with the metadata. According to an embodiment, the electronic device may store an image of a place of interest in association with its metadata. According to an embodiment, the electronic device may store an image of a place of interest and its metadata in association with the IVI system. According to various embodiments, the IVI system may be included in the electronic device or may be another electronic device mounted separately from the electronic device in a vehicle on which the electronic device is mounted. According to an embodiment, when the IVI system is separated from the electronic device, the electronic device may transmit an image of the POI and its metadata to the IVI system through a communication circuit.

Hereinafter, an operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 9 .

9 is a flowchart 900 illustrating an operation of an electronic device according to an exemplary embodiment. Operations of the electronic device described below may be performed by the electronic device 601 of FIG. 6 or the processor 680 of the electronic device 601 . According to an embodiment, since at least some of the operations of the electronic device illustrated in FIG. 9 may correspond to the operations of the electronic device illustrated in FIG. 7 , the duplicated description will be simplified or omitted.

In operation 901, the electronic device may receive voice data corresponding to the user's utterance. Operation 901 may correspond to operation 701 of FIG. 7 .

In operation 903, the electronic device may identify the first keyword and the second keyword from the voice data. According to an embodiment, the first keyword may be a keyword including place information. According to an embodiment, the second keyword may be a keyword including destination information associated with place information included in the first keyword. At least a portion of operation 903 may correspond to operation 703 of FIG. 7 . For the first keyword, the contents described above with reference to FIG. 7 may be equally applied. The first keyword and the second keyword differ only in information included therein, and according to an embodiment, the method of the electronic device identifying the first keyword from the voice data and the method of identifying the second keyword may be the same.

In operation 905, the electronic device may determine a time interval associated with the time when the first keyword was uttered. Operation 905 may correspond to 705 of FIG. 7 .

In operation 907, the electronic device may determine interest section information corresponding to the time section. Operation 907 may correspond to at least a part of operation 707 and operation 709 of FIG. 7 .

In operation 909 , the electronic device performs a plurality of second cameras (eg, the second camera of FIG. 6 ) based on the user's eye tracking information obtained using the first camera (eg, the first camera 630 of FIG. 6 ). At least one of (640)) may be specified. Operation 909 may correspond to at least a part of operation 709 of FIG. 7 .

In operation 911, the electronic device may acquire a first image among images acquired using at least one designated second camera. Operation 911 may correspond to at least a part of operation 709 of FIG. 7 .

In operation 913, the electronic device may determine ROI information corresponding to the time interval. According to an embodiment, the electronic device may determine the region of interest information corresponding to the time period based on the time period and the user's eye tracking information. According to an embodiment, the ROI information may include information on a geographic region corresponding to the user's gaze direction during a time period determined by the electronic device. According to an embodiment, the electronic device may determine, as the region of interest, a geographic region including a specified range based on a gaze direction of a user identified during a time interval. According to an embodiment, the electronic device may identify a region corresponding to the ROI on the map based on the ROI information and GPS information of the vehicle during the time period. According to an embodiment, the electronic device may obtain GPS information of the vehicle using a GPS sensor (eg, the sensor 650 of FIG. 6 ).

In operation 915, the electronic device may determine the POI corresponding to the target information by using data stored in the external storage device. According to an embodiment, the electronic device may communicate with the external storage device using a communication circuit (eg, the communication circuit 620 of FIG. 6 ). The electronic device may use data stored in an external storage device connected through a communication circuit. According to an embodiment, the electronic device may determine the POI corresponding to the target information based on the ROI information and the second keyword using data stored in the external storage device. According to an embodiment, the electronic device may identify a region corresponding to the ROI on the map by using map data stored in an external storage device. According to an embodiment, the electronic device may determine, as a place of interest, a place corresponding to the target information among places registered in the identified area on the map. According to an embodiment, the determined POI may be plural.

In operation 917, the electronic device may receive information related to a place of interest from an external storage device. According to an embodiment, the external storage device may store information related to a place of interest registered on a map. For example, the information related to the place of interest may include at least one of an address, a place name, a phone number, business hours, an image, a visitor review, a related keyword, or place preference information of the place of interest. According to an embodiment, the electronic device may receive information related to a place of interest from an external storage device through a communication circuit. According to an embodiment, the information related to the place of interest may include information related to the target information included in the second keyword. According to an embodiment, the electronic device may receive information related to target information among information related to a place of interest from an external storage device. For example, when the target information is 'camping', the electronic device may receive from an external storage device a review related to 'camping' among visitor reviews of a place of interest or an image related to 'camping' among images of a place of interest.

In operation 919, the electronic device may generate metadata of the first image based on information related to the place of interest. According to an embodiment, the metadata of the first image may include information related to a place of interest received by the electronic device from an external storage device. According to an embodiment, the metadata of the first image may include information related to target information among information related to a place of interest. According to an embodiment, the metadata of the first image may include at least one of information about a user's utterance for acquiring the first image, or information on the location and direction of the vehicle at the time the first image is captured. For example, the metadata of the first image may include first keyword information and second keyword information. According to an embodiment, the electronic device may store the generated metadata of the first image in association with the first image.

According to the above-described embodiment, the case in which the electronic device extracts the first image from the image obtained by using the second camera is taken as an example, but the present invention is not limited thereto, and the electronic device may The image may be stored in association with other information related to the POI.

According to the above-described embodiment, the electronic device may determine the place of interest as the place of interest by identifying the second keyword including the destination information, and store the information on the place of interest related to the purpose together with the image of the place of interest. have. As the electronic device receives a user input for selecting an image of the place of interest, when the electronic device stores (or registers) the place of interest, information related to the purpose associated with the place included in the utterance received from the user is transferred to the place of interest. It may be provided to the user as related information.

Hereinafter, an operation of an electronic device according to an exemplary embodiment will be described with reference to FIG. 10 .

10 is a conceptual diagram 1000 for explaining an operation of an electronic device according to an exemplary embodiment. Operations of the electronic device described below may be performed by the electronic device 601 of FIG. 6 or the processor 680 of the electronic device 601 . The operations of the electronic device described below may correspond to the operations illustrated in FIG. 9 .

Hereinafter, the electronic device is mounted on the vehicle 1001 , and a user of the electronic device may be a driver or a passenger of the vehicle 1001 . Hereinafter, it is assumed that the electronic device receives a voice input from the user while the vehicle 1001 is driving.

According to an embodiment, the electronic device receives voice data corresponding to the user's utterance through a microphone (eg, the microphone 610 of FIG. 6 ), and includes a first keyword 1011 including place information from the voice data, and A second keyword 1012 including destination information associated with place information may be identified. For example, the first keyword 1011 may be 'valley', and the second keyword 1012 may be 'camping'.

According to an embodiment, the electronic device may determine a time interval associated with the first keyword 1011 . According to an embodiment, the method for the electronic device to determine the time period based on the first keyword 1011 is to determine the main utterance time 520 or the time period by the electronic device described above with reference to FIGS. 5 to 8 . method may be the same. According to an embodiment, the electronic device may determine interest section information corresponding to the time section. According to an embodiment, the method for the electronic device to determine the ROI information corresponding to the time section is the same as the method for the electronic device to determine the ROI 530 or the ROI information with reference to FIGS. 5 to 8 . can do.

According to an embodiment, the electronic device may acquire an image corresponding to place information (eg, a valley) included in the first keyword 1011 based on the interest section information and the first keyword 1011 . According to an embodiment, the electronic device is a second camera set to capture an image corresponding to at least one direction of the external environment of the vehicle 1001 on which the electronic device is mounted (eg, the second camera 640 of FIG. 6 ). Among them, a second camera corresponding to the user's gaze direction may be designated based on the user's gaze tracking information during the time period. The electronic device may acquire an image corresponding to place information (eg, valley) included in the first keyword 1011 from an image corresponding to a time section among images acquired using the designated second camera.

According to an embodiment, the electronic device may determine the region of interest 1020 corresponding to the gaze direction of the user during the time interval. According to an embodiment, the electronic device may determine, as the region of interest 1020 , a geographic region including a specified range based on the gaze direction of the user identified during the time interval.

According to an embodiment, the electronic device uses data stored in an external storage device (eg, the external storage device 602 of FIG. 6 ) connected through a communication circuit (eg, the communication circuit 620 of FIG. 6 ) to provide target information It is possible to determine a place of interest corresponding to the . According to an embodiment, the electronic device may determine the POI corresponding to the target information based on the ROI information and the second keyword 1012 using data stored in the external storage device. According to an embodiment, the electronic device may identify the region B corresponding to the region of interest 1020 on the map by using map data stored in an external storage device. According to an embodiment, the electronic device may determine, as a place of interest, a place corresponding to purpose information (eg, camping) included in the second keyword 1012 among places registered in the identified area B on the map. For example, the determined POI may include a first POI B-1, a second POI B-2, and a third POI B-3.

According to an embodiment, the electronic device may receive information related to a place of interest. For example, the electronic device may display addresses, place names, phone numbers, business hours, and images of each of the first place of interest (B-1), the second place of interest (B-2), and the third place of interest (B-3). , at least one of visitor reviews, related keywords, and place preference information may be received from an external storage device. According to an embodiment, the electronic device stores the image corresponding to the place information (eg, valley) included in the first keyword 1011 obtained as described above and the second keyword 1012 received from the external storage device. Information related to a place of interest corresponding to included purpose information (eg, camping) may be linked and stored.

For example, the electronic device may display the user interface 1030 including the valley image on a display (eg, the display 660 of FIG. 6 ). According to an embodiment, in response to receiving an input for selecting a valley image, the electronic device may provide information related to a good place to camp in a specific valley corresponding to the selected valley image.

Hereinafter, a user interface provided by an electronic device according to an exemplary embodiment will be described with reference to FIG. 11 .

11 is a diagram 1100 illustrating a user interface provided by an electronic device through a display according to an exemplary embodiment. Operations of the electronic device described below may be performed by the electronic device 601 of FIG. 6 or the processor 680 of the electronic device 601 .

Referring to FIG. 11 , the electronic device has a first user interface ( 1110) may be provided through a display (eg, the display 660 of FIG. 6 ). According to an embodiment, the electronic device may categorize and display the stored images by keywords associated with places.

According to an embodiment, in response to receiving an input for selecting the first image 1115 , the electronic device may display the second user interface 1120 on the display. According to an embodiment, the electronic device may provide metadata of the first image 1115 through the second user interface 1120 . According to an embodiment, the second user interface 1120 may include information related to a place of interest corresponding to the first image 1115 . For example, the second user interface 1120 may provide navigation information for guiding a route to the POI corresponding to the first image 1115 , keyword information and review information related to the POI corresponding to the first image 1115 . , and utterance information of the user for obtaining the first image 1115 .

According to an embodiment, the electronic device may share an image of the place of interest and information related to the place of interest with another electronic device (eg, an electronic device mounted in another vehicle) connected to the electronic device through a communication circuit.

An electronic device according to an embodiment disclosed in this document (eg, the electronic device 101 of FIG. 1 , the user terminal 201 of FIG. 2 , the user terminal 401 of FIG. 4 , or the electronic device 101 of FIG. 6 ) )), a microphone (eg, the input module 150 of FIG. 1 , the microphone 270 of FIG. 2 , or the microphone 610 of FIG. 6 ), a communication circuit (eg, the communication module 190 of FIG. 1 , FIG. 2, the communication interface 290, or the communication circuit 620 of FIG. 6), a plurality of cameras (eg, the camera module 180 of FIG. 1, or the first camera 630 of FIG. 6 and the second of FIG. 6) camera 640 ), a sensor (eg, the sensor module 176 of FIG. 1 , or the sensor 650 of FIG. 6 ), a display (eg, the display module 160 of FIG. 1 , the display 260 of FIG. 2 ); or display 660 of FIG. 6 ), a memory (eg, memory 130 of FIG. 1 , memory 230 of FIG. 2 , or memory 670 of FIG. 6 ), the microphone, the communication circuitry, the plurality of a processor (eg, processor 120 of FIG. 1 , processor 220 of FIG. 2 , or processor 680 of FIG. 6 ) operatively coupled to the camera, the sensor, the display, and the memory; When executing, the processor receives voice data corresponding to the utterance of the user of the electronic device through the microphone, identifies a first keyword including place information from the voice data, and the first keyword Determining a time section associated with the time the false utterance was made, using the sensor to obtain driving information including at least one of speed information or direction information corresponding to the time section, and using at least one of the plurality of cameras One or more instructions for obtaining a first image corresponding to the place information based on the time section and the driving information among the acquired images may be stored.

According to an embodiment disclosed in this document, the instructions include, by the processor, from a point in time earlier by a specified time from a start point of the speech including the first keyword to a point in time when the speech of the first keyword is finished. It can be decided by time interval.

According to an embodiment disclosed in this document, the plurality of cameras include a first camera (eg, the first camera 630 of FIG. 6 ) and a plurality of second cameras (eg, the second camera 640 of FIG. 6 ) ), wherein the instructions include, by the processor, the user's gaze tracking acquired by using the first camera to determine the interest section information corresponding to the time section based on the time section and the driving information Designate at least one of the plurality of second cameras based on information, and acquire the first image based on the ROI information and the first keyword among images acquired using the designated at least one second camera can make it

According to an embodiment disclosed in this document, the interest section information may include information about an image corresponding to the time section in an image acquired using the first camera or the second camera.

According to an embodiment disclosed in this document, the interest section information may include information on a distance and geographic range that a vehicle (eg, vehicle 501 ) on which the electronic device is mounted moves during the time section. have.

According to an embodiment disclosed in this document, the first camera is set to recognize at least one of an eye movement, a gesture, and a head movement of the user, and the second camera is a vehicle in which the electronic device is mounted. It can be set to photograph the external environment of

According to an embodiment disclosed in this document, the instructions may be generated from among the plurality of second cameras as the processor detects the user's specified movement event from the image of the first camera corresponding to the POI information. It is possible to select a camera corresponding to the gaze direction of the user at the time of occurrence of the specified motion event.

According to an embodiment disclosed in this document, the instructions include, by the processor, an external storage device that communicates with the electronic device through the communication circuit (eg, the electronic device 102 of FIG. 1 , the electronic device of FIG. 1 ) 104, the server 108 of FIG. 1, the service server 400 of FIG. 2, or the external storage device 602 of FIG. 6) based on the information stored in the POI information and the first keyword Determining a place of interest that is a geographic location corresponding to the place information, receiving information related to the place of interest from the external storage device through the communication circuit, and meta of the first image based on the information related to the place of interest You can create data (metadata).

According to an embodiment disclosed in this document, the instructions include, when the processor displays the first image on the display and receives an input for selecting the first image, metadata of the first image may be displayed on the display.

According to an embodiment disclosed in this document, the instructions include, by the processor, identifying a second keyword including purpose information associated with the place information from the voice data, based on the time interval and the eye tracking information to determine the region of interest information corresponding to the time period, and use data stored in an external storage device that communicates with the electronic device through the communication circuit to obtain the target information based on the region of interest information and the second keyword. determine a corresponding POI, receive information related to the POI from the external storage device through the communication circuit, and generate metadata of the first image based on the information associated with the POI; The information related to the place of interest may include information related to the target information.

An electronic device according to an embodiment disclosed in this document (eg, the electronic device 101 of FIG. 1 , the user terminal 201 of FIG. 2 , the user terminal 401 of FIG. 4 , or the electronic device 101 of FIG. 6 ) )), a user's utterance of the electronic device through a microphone of the electronic device (eg, the input module 150 of FIG. 1 , the microphone 270 of FIG. 2 , or the microphone 610 of FIG. 6 ) Receive voice data corresponding to , identify a first keyword including place information from the voice data, determine a time interval associated with a time when the first keyword is uttered, and determine a sensor (eg, FIG. Using the sensor module 176 of 1 or the sensor 650 of FIG. 6) obtains driving information including at least one of speed information or direction information corresponding to the time section, and a plurality of cameras of the electronic device (eg, the time section and the driving information among images acquired using at least one of the camera module 180 of FIG. 1 , or the first camera 630 of FIG. 6 and the second camera 640 of FIG. 6 ) Based on the , the first image corresponding to the place information may be acquired.

According to an embodiment disclosed in this document, it is possible to determine as the time interval from a point in time preceding the start point of the speech including the first keyword by a specified time until the point in time at which the speech of the first keyword ends.

According to an embodiment disclosed in this document, the plurality of cameras include a first camera (eg, the first camera 630 of FIG. 6 ) and a plurality of second cameras (eg, the second camera 640 of FIG. 6 ) ), wherein the method determines the interest section information corresponding to the time section based on the time section and the driving information, and based on the user's eye tracking information obtained using the first camera, At least one of the plurality of second cameras may be designated, and the first image may be acquired based on the ROI information and the first keyword among images acquired using the designated at least one second camera.

According to an embodiment disclosed in this document, the ROI information includes information on a distance and a geographic range that a vehicle on which the electronic device is mounted (eg, the vehicle 501 of FIG. 5 ) moved during the time period. may include

According to an embodiment disclosed in this document, as the specified motion event of the user is detected from the image of the first camera corresponding to the POI information, the specified motion event occurs from among the plurality of second cameras. A camera corresponding to the user's gaze direction may be selected.

According to an embodiment disclosed in this document, through a communication circuit of the electronic device (eg, the communication module 190 of FIG. 1 , the communication interface 290 of FIG. 2 , or the communication circuit 620 of FIG. 6 ) An external storage device communicating with the electronic device (eg, the electronic device 102 of FIG. 1 , the electronic device 104 of FIG. 1 , the server 108 of FIG. 1 , the service server 400 of FIG. 2 , or FIG. 6 ) A place of interest that is a geographic location corresponding to the place information is determined based on the POI information and the first keyword using data stored in the external storage device 602) of the external storage device through the communication circuit. may receive information related to the POI from the POI, and generate metadata of the first image based on the information related to the POI.

According to an embodiment disclosed in this document, the first image is displayed on the display of the electronic device (eg, the display module 160 of FIG. 1 , the display 260 of FIG. 2 , or the display 660 of FIG. 6 ) and, upon receiving an input for selecting the first image, metadata of the first image may be displayed on the display.

According to an embodiment disclosed in this document, a second keyword including target information associated with the place information is identified from the voice data, and an interest corresponding to the time period based on the time period and the eye tracking information region information is determined, and a place of interest corresponding to the target information is selected based on the region of interest information and the second keyword using data stored in an external storage device that communicates with the electronic device through a communication circuit of the electronic device determine, receive information related to the place of interest from the external storage device through the communication circuit, generate metadata of the first image based on the information related to the place of interest, and the information related to the place of interest is It may include information related to the target information.

Claims

In an electronic device,

MIC;

communication circuit;

a plurality of cameras;

sensor;

display;

Memory;

a processor operatively coupled to the microphone, the communication circuitry, the plurality of cameras, the sensors, the display and the memory;

The memory, when executed, causes the processor to:

receiving voice data corresponding to the user's utterance of the electronic device through the microphone;

identify a first keyword including place information from the voice data;

determining a time interval associated with the time the first keyword was uttered,

Obtaining driving information including at least one of speed information and direction information corresponding to the time section by using the sensor,

An electronic device that stores one or more instructions for acquiring a first image corresponding to the place information based on the time section and the driving information among images acquired using at least one of the plurality of cameras .
The method according to claim 1,

The instructions, the processor,

and determining, as the time interval, from a point in time earlier by a specified time from a start point of the speech including the first keyword to a point in time at which the speech of the first keyword ends.
The method according to claim 1,

The plurality of cameras includes a first camera and a plurality of second cameras,

The instructions, the processor,

Determine the interest section information corresponding to the time section based on the time section and the driving information,

Designating at least one of the plurality of second cameras based on the user's eye tracking information acquired using the first camera,

and acquiring the first image based on the ROI information and the first keyword among images acquired using the designated at least one second camera.
4. The method of claim 3,

The interest section information includes information on an image corresponding to the time section in an image acquired using the first camera or the second camera.
4. The method of claim 3,

The information on the section of interest includes information on a distance and a geographic range that the vehicle on which the electronic device is mounted moves during the time section.
4. The method of claim 3,

The first camera is set to recognize at least one of the user's eye movement, gesture, and head movement,

The second camera is set to photograph an external environment of a vehicle on which the electronic device is mounted.
4. The method of claim 3,

The instructions, the processor,

Upon detecting the user's designated motion event from the image of the first camera corresponding to the interest section information, a camera corresponding to the user's gaze direction at the time of the designated motion event occurrence from among the plurality of second cameras is selected an electronic device to make it happen.
4. The method of claim 3,

The instructions, the processor,

determining a place of interest that is a geographic location corresponding to the place information based on the POI information and the first keyword using data stored in an external storage device that communicates with the electronic device through the communication circuit;

receiving information associated with the place of interest from the external storage device through the communication circuit;

and generating metadata of the first image based on information related to the place of interest.
9. The method of claim 8,

The instructions, the processor,

Display the first image on the display,

and display metadata of the first image on the display in response to receiving an input for selecting the first image.
4. The method of claim 3,

The instructions, the processor,

identify a second keyword including purpose information associated with the place information from the voice data;

determining region of interest information corresponding to the time interval based on the time interval and the eye tracking information;

determining a place of interest corresponding to the target information based on the region of interest information and the second keyword using data stored in an external storage device communicating with the electronic device through the communication circuit;

receiving information associated with the place of interest from the external storage device through the communication circuit;

to generate metadata of the first image based on information related to the place of interest;

The electronic device, wherein the information associated with the place of interest includes information associated with the target information.
A method of operating an electronic device, comprising:

receiving voice data corresponding to the utterance of the user of the electronic device through the microphone of the electronic device;

identify a first keyword including place information from the voice data;

determining a time interval associated with the time the first keyword was uttered,

obtaining driving information including at least one of speed information and direction information corresponding to the time section by using a sensor of the electronic device;

A method of acquiring a first image corresponding to the place information based on the time section and the driving information among images acquired using at least one of a plurality of cameras of the electronic device.
12. The method of claim 11,

and determining, as the time interval, from a point in time earlier by a specified time from a start point of the speech including the first keyword to a point in time at which the speech of the first keyword ends.
12. The method of claim 11,

The plurality of cameras includes a first camera and a plurality of second cameras,

The method is

Determine the interest section information corresponding to the time section based on the time section and the driving information,

Designating at least one of the plurality of second cameras based on the user's eye tracking information acquired using the first camera,

The method of acquiring the first image based on the ROI information and the first keyword among images acquired using the designated at least one second camera.
14. The method of claim 13,

determining a place of interest, which is a geographic location corresponding to the place information, based on the interest section information and the first keyword, using data stored in an external storage device that communicates with the electronic device through the communication circuit of the electronic device;

receiving information associated with the place of interest from the external storage device through the communication circuit;

A method of generating metadata of the first image based on information associated with the place of interest.
15. The method of claim 14,

displaying the first image on a display of the electronic device;

upon receiving an input for selecting the first image, displaying metadata of the first image on the display.