KR102012774B1 - Mobil terminal and Operating Method for the Same - Google Patents

Abstract

A method of operating a mobile terminal according to an embodiment of the present invention includes displaying a predetermined image on a display unit, entering a voice command input mode, receiving a voice input signal, a received voice input signal, and a predetermined predetermined display. Recognizing a voice command based on the image and executing an operation corresponding to the recognized voice command. Accordingly, the voice command of the user can be recognized more quickly and accurately, and the user's convenience can be improved.

Description

Mobile terminal and its operation method {Mobil terminal and Operating Method for the Same}

The present invention relates to a portable terminal and a method of operating the same, and more particularly, to a portable terminal and a method of operating the same, which can execute a predetermined operation more quickly and conveniently while preventing a malfunction.

A portable terminal is a portable device that is portable and has one or more functions for making voice and video calls, inputting and outputting information, and storing data. As the functions of the mobile terminals are diversified, they are equipped with complex functions such as taking pictures or videos, playing music files or video files, playing games, receiving broadcasts, wireless Internet, and the like. It is implemented in the form.

In the mobile terminal implemented in the form of such a multimedia device, new attempts have been applied in various ways in terms of hardware and software to implement complex functions. For example, a user interface environment for a user to search for or select a function easily and conveniently.

On the other hand, according to the diversification of functions and services of the mobile terminal, research on a voice recognition technology that can recognize and manipulate a user voice is increasing.

Accordingly, an object of the present invention is to provide a portable terminal with improved user convenience and an operation method thereof.

Another object of the present invention is to provide a portable terminal which can be operated accurately and conveniently through a voice recognition technology and a method of operating the same.

A method of operating a mobile terminal according to an embodiment of the present invention includes displaying a predetermined image on a display unit, entering a voice command input mode, receiving a voice input signal, a received voice input signal, and a predetermined predetermined display. Recognizing a voice command based on the image and executing an operation corresponding to the recognized voice command.

According to an embodiment of the present invention, a portable terminal may include a display unit for displaying a predetermined image, a microphone (MIC) for receiving a voice input signal in a voice command input mode, and the received voice input signal and the displayed predetermined image. And a controller configured to recognize a voice command and to control an operation corresponding to the recognized voice command.

According to the embodiment of the present invention, the user's voice command can be recognized more quickly and accurately, thereby improving user convenience.

1 is a block diagram of a portable terminal according to an embodiment of the present invention;
2 is a front perspective view of a portable terminal according to an embodiment of the present invention;
3 is a rear perspective view of a portable terminal according to an embodiment of the present invention;
4 is a flowchart showing an embodiment of a method of operating a mobile terminal according to an embodiment of the present invention;
5A to 8C are views referred to for describing various embodiments of a method of operating a mobile terminal according to the present invention.

Hereinafter, with reference to the drawings will be described the present invention in more detail.

The portable terminal described in the present specification includes a mobile phone, a smart phone, a notebook computer, a digital broadcasting terminal, a personal digital assistant (PDA), a portable multimedia player (PMP), a camera, a navigation computer, a tablet computer ( tablet computer, e-book terminal, and the like. In addition, the suffixes "module" and "unit" for the components used in the following description are merely given in consideration of ease of preparation of the present specification, and do not give particular meanings or roles by themselves. Therefore, the "module" and "unit" may be used interchangeably.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention. Referring to FIG. 1, the portable terminal according to an embodiment of the present invention will be described in terms of components according to functions.

Referring to FIG. 1, the mobile terminal 100 includes a wireless communication unit 110, an A / V input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, and a memory. 160, an interface unit 170, a controller 180, and a power supply unit 190 may be included. Such components may be configured by combining two or more components into one component, or by dividing one or more components into two or more components as necessary when implemented in an actual application.

The wireless communication unit 110 may include a broadcast receiving module 111, a mobile communication module 113, a wireless internet module 115, a short range communication module 117, and a GPS module 119.

The broadcast receiving module 111 receives at least one of a broadcast signal and broadcast related information from an external broadcast management server through a broadcast channel. In this case, the broadcast channel may include a satellite channel, a terrestrial channel, and the like. The broadcast management server may mean a server for generating and transmitting at least one of a broadcast signal and broadcast related information, or a server receiving at least one of the pre-generated broadcast signal and broadcast related information and transmitting the same to a terminal.

The broadcast signal may include not only a TV broadcast signal, a radio broadcast signal, and a data broadcast signal, but also a broadcast signal having a data broadcast signal combined with a TV broadcast signal or a radio broadcast signal. The broadcast related information may mean information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast related information may also be provided through a mobile communication network, and in this case, may be received by the mobile communication module 113. Broadcast related information may exist in various forms.

The broadcast receiving module 111 receives broadcast signals using various broadcast systems, and in particular, digital multimedia broadcasting-terrestrial (DMB-T), digital multimedia broadcasting-satellite (DMB-S), and media forward link only (MediaFLO). ), Digital broadcast signals may be received using digital broadcasting systems such as DVB-H (Digital Video Broadcast-Handheld) and ISDB-T (Integrated Services Digital Broadcast-Terrestrial). In addition, the broadcast receiving module 111 may be configured to be suitable for all broadcast systems providing broadcast signals as well as such digital broadcast systems. The broadcast signal and / or broadcast related information received through the broadcast receiving module 111 may be stored in the memory 160.

The mobile communication module 113 transmits and receives a radio signal with at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to voice call signal, video call signal, or text / multimedia message transmission and reception.

The wireless internet module 115 refers to a module for wireless internet access. The wireless internet module 115 may be embedded or external to the portable terminal 100. Wireless Internet technologies may include Wireless LAN (Wi-Fi), Wireless Broadband (Wibro), World Interoperability for Microwave Access (Wimax), High Speed Downlink Packet Access (HSDPA), and the like.

The short range communication module 117 refers to a module for short range communication. As a short range communication technology, Bluetooth, Radio Frequency Identification (RFID), Infrared Data Association (IrDA), Ultra Wideband (UWB), ZigBee, Near Field Communication (NFC), and the like may be used.

The GPS (Global Position System) module 119 receives position information from a plurality of GPS satellites.

The A / V input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone 123. The camera 121 processes image frames such as still images or moving images obtained by the image sensor in the video call mode or the photographing mode. The processed image frame may be displayed on the display unit 151.

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. Two or more cameras 121 may be provided according to the configuration aspect of the terminal.

The microphone 123 receives an external sound signal by a microphone in a call mode, a recording mode, a voice recognition mode, or the like, and processes the external sound signal into electrical voice data. The processed voice data may be converted into a form transmittable to the mobile communication base station through the mobile communication module 113 and output in the call mode. The microphone 123 may use various noise removing algorithms for removing noise generated in the process of receiving an external sound signal.

The user input unit 130 generates key input data input by the user for controlling the operation of the terminal. The user input unit 130 may be configured of a key pad, a dome switch, a touch pad (constant voltage / capacitance), etc. that may receive a command or information by a user's pressing or touch manipulation. In addition, the user input unit 130 may be configured as a jog wheel or a jog method or a joystick that rotates a key, a finger mouse, or the like. In particular, when the touch pad forms a mutual layer structure with the display unit 151 to be described later, this may be referred to as a touch screen.

The sensing unit 140 detects a current state of the mobile terminal 100 such as an open / closed state of the mobile terminal 100, a position of the mobile terminal 100, presence or absence of a user contact, and controls the operation of the mobile terminal 100. Generate a sensing signal. For example, when the portable terminal 100 is in the form of a slide phone, whether the slide phone is opened or closed may be sensed. In addition, it may be responsible for sensing functions related to whether the power supply unit 190 is supplied with power, whether the interface unit 170 is coupled to an external device, and the like.

The sensing unit 140 may include a proximity sensor 141, a pressure sensor 143, a motion sensor 145, and the like. The proximity sensor 141 may detect the presence or absence of an object approaching the portable terminal 100 or an object present in the vicinity of the portable terminal 100 without mechanical contact. The proximity sensor 141 may detect a proximity object by using a change in an alternating magnetic field or a change in a static magnetic field, or by using a change rate of capacitance. In addition, it is possible to detect which side of the mobile terminal 100 the user is holding. Two or more proximity sensors 141 may be provided according to the configuration aspect.

The pressure sensor 143 may detect whether pressure is applied to the portable terminal 100, the magnitude of the pressure, and the like. The pressure sensor 143 may be installed at a portion of the portable terminal 100 that requires the detection of pressure according to the use environment.

If the pressure sensor 143 is installed in the display unit 151, a touch input through the display unit 151 and a greater pressure than the touch input are applied according to a signal output from the pressure sensor 143. The pressure touch input can be identified. In addition, according to the signal output from the pressure sensor 143, it can also know the magnitude of the pressure applied to the display unit 151 when the pressure touch input.

If the pressure sensor 143 is disposed outside the portable terminal 100, the pressure may be detected to detect which side of the portable terminal 100 is held by the user.

The motion sensor 145 detects the position or movement of the portable terminal 100 using an acceleration sensor, a gyro sensor, or the like. The acceleration sensor that can be used for the motion sensor 145 is an element that converts an acceleration signal in one direction into an electrical signal, and is widely used with the development of micro-electromechanical systems (MEMS) technology.

Accelerometers measure the acceleration of a large value embedded in an airbag system of a car and used to detect a collision, and to measure the acceleration of a minute value used as an input means such as a game by recognizing a minute motion of a human hand. There are many different types. Accelerometers are usually configured by mounting two or three axes in one package. Depending on the environment, only one axis may be needed. Therefore, if for some reason it is necessary to use the acceleration sensor in the X-axis or Y-axis direction instead of the Z-axis direction, the acceleration sensor may be mounted on the main substrate using a separate engraving substrate.

In addition, the gyro sensor is a sensor for measuring the angular velocity, and may sense a direction returned to the reference direction.

Meanwhile, the sensing unit 140 may include sensors for user authentication. For example, when the user authentication is performed through the user's biometric information, a sensor for recognizing body temperature, fingerprint, iris, face, etc. may be provided and required according to a user authentication method set in the mobile terminal 100. A sensor may be provided.

The output unit 150 is for outputting an audio signal, a video signal, or an alarm signal. The output unit 150 may include a display unit 151, an audio output module 153, an alarm unit 155, and a haptic module 157.

The display unit 151 displays and outputs information processed by the mobile terminal 100. For example, when the mobile terminal 100 is in a call mode, the mobile terminal 100 displays a user interface (UI) or a graphic user interface (GUI) related to the call. When the mobile terminal 100 is in a video call mode or a photographing mode, the mobile terminal 100 may display a captured or received image, respectively, or simultaneously, and display a UI and a GUI.

Meanwhile, as described above, when the display unit 151 and the touch pad form a mutual layer structure and constitute a touch screen, the display unit 151 is an input device capable of inputting information by a user's touch in addition to the output device. May also be used.

If the display unit 151 is configured as a touch screen, the display unit 151 may include a touch screen panel and a touch screen panel controller. In this case, the touch screen panel is a transparent panel attached to the outside and may be connected to the internal bus of the portable terminal 100. The touch screen panel keeps an eye on the contact result and sends corresponding signals to the touch screen panel controller when there is a touch input. The touch screen panel controller processes the signals and then transmits corresponding data to the controller 180 so that the controller 180 can determine whether there is a touch input and which area of the touch screen has been touched.

The display unit 151 may be composed of an electronic paper (e-Paper). Electronic paper (e-Paper) is a kind of reflective display that has excellent visual characteristics with high resolution, wide viewing angle and bright white background like conventional paper and ink. The e-paper can be implemented on any substrate such as plastic, metal, paper, etc., and the image is maintained even after the power is cut off, and there is no back light power, so the battery life of the mobile terminal 100 is long. Can be maintained. As the electronic paper, hemispherical twist balls filled with electrostatics may be used, or electrophoresis and microcapsules may be used.

In addition, the display unit 151 may include a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, and a three-dimensional display. It may include at least one of (3D display). In addition, two or more display units 151 may exist according to the implementation form of the mobile terminal 100. For example, the external display unit (not shown) and the internal display unit (not shown) may be simultaneously provided in the portable terminal 100.

The sound output module 153 outputs audio data received from the wireless communication unit 110 or stored in the memory 160 in a call signal reception, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, and the like. In addition, the sound output module 153 outputs a sound signal related to a function performed in the portable terminal 100, for example, a call signal reception sound and a message reception sound. The sound output module 153 may include a speaker, a buzzer, and the like.

The alarm unit 155 outputs a signal for notifying occurrence of an event of the portable terminal 100. Examples of events occurring in the mobile terminal 100 include call signal reception, message reception, key signal input, and the like. The alarm unit 155 outputs a signal for notifying occurrence of an event in a form other than an audio signal or a video signal. For example, the signal may be output in the form of vibration. The alarm unit 155 may output a signal to inform the call signal when the call signal or the message is received. In addition, when the key signal is input, the alarm unit 155 may output the signal as a feedback to the key signal input. The user may recognize the occurrence of an event through the signal output from the alarm unit 155. The signal for notifying the event occurrence in the mobile terminal 100 may also be output through the display unit 151 or the sound output module 153.

The haptic module 157 generates various haptic effects that a user can feel. A representative example of the haptic effect generated by the haptic module 157 is a vibration effect. When the haptic module 157 generates vibration by the tactile effect, the intensity and pattern of the vibration generated by the haptic module 157 may be converted, and may be output by combining different vibrations or sequentially.

In addition to the vibration, the haptic module 157 may be operated by a pin array that vertically moves with respect to the contact skin surface, by the stimulation through the injection force or the suction force of the air through the injection or inlet, or by the stimulus that strikes the skin surface. Various tactile effects can be generated, such as effects of stimulation through contact of electrodes, effects of stimulation using electrostatic force, and effects of reproducing cold / warm feeling using elements capable of endothermic or exothermic heat. The haptic module 157 may not only deliver the haptic effect through direct contact, but also may realize the haptic effect through the muscle sensation of a user's finger or arm. Two or more haptic modules 157 may be provided according to a configuration aspect of the mobile terminal 100.

The memory 160 may store a program for processing and controlling the controller 180 and may provide a function for temporarily storing input or output data (for example, a phone book, a message, a still image, a video, etc.). It can also be done.

The memory 160 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), RAM The storage medium may include at least one type of storage medium. In addition, the mobile terminal 100 may operate a web storage that performs a storage function of the memory 150 on the Internet.

The interface unit 170 serves as an interface with all external devices connected to the mobile terminal 100. Examples of external devices connected to the mobile terminal 100 include wired / wireless headsets, external chargers, wired / wireless data ports, memory cards, subscriber identification module (SIM) cards, and user identity module (UIM) cards. Card sockets, audio input / output (I / O) terminals, video input / output (I / O) terminals, and earphones. The interface unit 170 may receive data from such an external device or receive power and transmit the data to each component in the mobile terminal 100, and may transmit data in the mobile terminal 100 to an external device. .

When the portable terminal 100 is connected to an external cradle, the interface unit 170 may be a passage through which power from the connected cradle is supplied to the portable terminal 100, or various command signals input from the cradle by the user may be carried. It may be a passage that is delivered to the terminal 100.

The controller 180 typically controls the operation of each unit to control the overall operation of the mobile terminal 100. For example, perform related control and processing for voice calls, data communications, video calls, and the like. In addition, the controller 180 may include a multimedia playback module 181 for multimedia playback. The multimedia playback module 181 may be configured in hardware in the controller 180 or may be configured in software separately from the controller 180.

In addition, the power supply unit 190 receives an external power source and an internal power source under the control of the controller 180 to supply power for operation of each component.

The portable terminal 100 having such a configuration may be configured to be operable in a communication system capable of transmitting data through a frame or packet, including a wired / wireless communication system and a satellite based communication system. have.

2 is a perspective view of a portable terminal according to an embodiment of the present invention, and FIG. 3 is a rear perspective view of the portable terminal illustrated in FIG. 2. Hereinafter, referring to FIGS. 2 and 3, a portable terminal related to the present invention will be described in terms of components according to appearance. In addition, hereinafter, for convenience of description, a bar type portable terminal having a front touch screen among various types of portable terminals such as a folder type, a bar type, a swing type, a slider type, and the like will be described as an example. However, the present invention is not limited to the bar type portable terminal, but can be applied to all types of portable terminals including the aforementioned type.

Referring to FIG. 2, the case forming the exterior of the portable terminal 100 is formed by the front case 100-1 and the rear case 100-2. Various electronic components are embedded in the space formed by the front case 100-1 and the rear case 100-2.

In the main body, specifically, the front case 100-1, the display unit 151, the first audio output module 153a, the first camera 121a, and the first to third user input units 130a, 130b, and 130c are provided. Can be arranged. The fourth user input unit 130d, the fifth user input unit 130e, and the microphone 123 may be disposed on the side surface of the rear case 100-2.

The display unit 151 may be configured such that the touch pad is overlapped in a layer structure so that the display unit 151 operates as a touch screen to input information by a user's touch.

The first sound output module 153a may be implemented in the form of a receiver or a speaker. The first camera 121a may be implemented in a form suitable for capturing an image or a video of a user or the like. The microphone 123 may be implemented in a form suitable for receiving a user's voice or other sound.

The first to fifth user input units 130a, 130b, 130c, 130d, and 130e and the sixth and seventh user input units 130f and 130g to be described later may be collectively referred to as the user input unit 130, and the user may feel tactile. Any manner may be employed as long as it is a tactile manner.

For example, the user input unit 130 may be implemented as a dome switch or a touch pad capable of receiving a command or information by a user's pressing or touch manipulation, or a manipulation method such as a wheel or jog or a joystick that rotates a key. Or the like. In functional terms, the first to third user input units 130a, 130b, and 130c are for inputting commands such as start, end, scroll, and the like, and the fourth user input unit 130d is for inputting a selection of an operation mode or the like. It is for. In addition, the fifth user input unit 130e may operate as a hot-key for activating a special function in the mobile terminal 100.

 Referring to FIG. 3, a second camera 121b may be additionally mounted on the rear of the rear case 100-2, and the sixth and seventh user inputs 130f, 130g) and the interface unit 170 may be disposed.

The second camera 121b may have a photographing direction substantially opposite to that of the first camera 121a, and may have different pixels from the first camera 121a. A flash (not shown) and a mirror (not shown) may be further disposed adjacent to the second camera 121b. In addition, another camera may be further provided adjacent to the second camera 121b to be used for capturing a 3D stereoscopic image.

The flash shines toward the subject when the subject is photographed with the second camera 121b. The mirror allows the user to see his / her own face or the like when photographing (self-photographing) the user using the second camera 121b.

A second sound output module (not shown) may be further disposed on the rear case 100-2. The second sound output module may implement a stereo function together with the first sound output module 153a and may be used for a call in the speakerphone mode.

The interface unit 170 may be used as a passage through which data is exchanged with an external device. In addition, an antenna (not shown) for receiving a broadcast signal may be disposed in one region of the front case 100-1 and the rear case 100-2. The antenna may be installed to be pulled out of the rear case 100-2.

The power supply unit 190 for supplying power to the portable terminal 100 may be mounted on the rear case 100-2 side. The power supply unit 190 is, for example, a rechargeable battery, and may be detachably coupled to the rear case 100-2 for charging.

Meanwhile, in the present embodiment, the second camera 121b and the like have been described as being disposed in the rear case 100-2, but the present disclosure is not necessarily limited thereto.

In addition, even if the second camera 121b is not separately provided, the first camera 121a may be rotatably formed to be able to photograph up to the photographing direction of the second camera 121b.

4 is a flowchart illustrating an embodiment of a method of operating a mobile terminal according to an embodiment of the present invention.

Referring to FIG. 4, the mobile terminal 100 according to an embodiment of the present invention first displays a predetermined image on the display unit 151. (S410)

The predetermined image may be an execution screen of a predetermined function or application using the mobile terminal 100, or may be an image of a specific content such as a picture or a video.

Alternatively, the image may be an image acquired through the camera 121, that is, a picture or a video taken by the camera 121, or a preview image in the photographing mode.

Thereafter, the portable terminal 100 enters the voice command input mode (S420).

The controller 180 may control to enter the voice command input mode according to a user's command, or control to enter the voice command input mode automatically when the voice command input is possible.

For example, when the user long-presses the touch screen, the user may be set to enter the voice command input mode.

In this case, the operation method of the portable terminal according to the embodiment of the present invention further includes the step of receiving a long-press (long-press) input, the voice command input mode according to the long-press input You can enter.

Accordingly, the user may enter the voice recognition mode according to the embodiment of the present invention through a set input such as a long-press input at any time during the use of the portable terminal 100.

 On the other hand, according to the setting, when the user long-press, the voice recognition mode is executed, and if there is no speech for a predetermined time, for example, about 10 seconds, the voice recognition mode may be automatically terminated. Alternatively, the voice recognition mode may be released after executing the corresponding command.

On the other hand, the controller 180 displays a window for displaying a message or a recognition result in a predetermined area of the display unit 151 as the voice command input mode is entered, and an icon indicating the voice command input mode. The display state of the icon representing the voice command input mode may be changed as the voice recognition process proceeds.

Meanwhile, a user interface (UI) and detailed operations of the voice command input mode will be described later in detail with reference to FIGS. 5A to 8C.

Thereafter, in the voice command input mode, the mobile terminal 100 may receive a voice input signal through the microphone 123.

The controller 180 may recognize a voice command based on the received voice input signal and the displayed predetermined image (S440), and control to execute an operation corresponding to the recognized voice command (S450).

That is, the present invention may not only analyze the voice input signal to recognize the voice but also additionally use the content currently being used, in particular, the image being displayed, thereby increasing the voice recognition speed and increasing the accuracy of the result.

For example, the amount of database search required for speech recognition can be reduced to a related category based on the attributes and contents of the displayed image, and focusing on the vocabulary and context related to the related category can be performed more faithfully. Do.

In the past, it has relied on natural language search engines with vast databases for speech recognition. Therefore, the speed of speech recognition is slow and the reliability of speech recognition results is deteriorated. As a result, users often have a hard time telling how to express their intentions.

In addition, it was more difficult to perform complex tasks such as "Email A's B photos."

Unlike the conventional method, the present invention combines touch and voice to obtain a more accurate result value.

In addition, by combining the image of the content being used during the use of the mobile terminal 100 and the action (action), such as the user's touch input, the user can ask the correct intention and derive the desired result.

Thus, by allowing the user to filter exactly what he wants through an action that reflects the intention of the user. Increase the speed and accuracy of speech recognition.

In addition, even after a speech recognition result is obtained, it is possible to continuously perform speech recognition according to a use scene, and thus it is easy to perform a complex task.

On the other hand, when entering the voice recognition mode by the long-press input-in the step of recognizing the voice command (S440), the controller 180, the long-press input object of the object included in the image And recognize the voice command based on the received voice input signal.

That is, the speed and accuracy of speech recognition may be further improved by focusing on the vocabulary and context related to a specific object touched by a user more accurately than the attributes, types, and contents of a predetermined image.

In addition, the method of operating a mobile terminal according to an embodiment of the present invention further includes extracting a keyword based on the predetermined image, and in step S440 of recognizing the voice command, the controller. The controller 180 may control to recognize the voice command based on the keyword and the received voice input signal.

In this case, the controller 180 extracts a feature vector based on the received voice input signal, and extracts the feature vector from among a voice database including an acoustic model and a language model. The data corresponding to the received voice input signal may be determined by comparing the data of the category related to the keyword.

Meanwhile, in the present specification, a description will be given of a case in which the controller 180 performs speech recognition, but the controller 180 does not have a speech recognition engine and is separately provided in the portable terminal 100. It is also possible to perform a speech recognition process by controlling the speech engine.

Alternatively, the controller 180 may transmit and receive data with an externally connected voice recognition engine or a voice recognition service provider to control voice recognition.

As an example of the speech recognition process, the speech recognition engine provided inside or outside the controller 180 may perform preprocessing such as noise processing on the first speech input signal.

In the preprocessing process, such as end-point detection, a processing method of detecting only a speech section from an input signal and a method of distinguishing a speech section from a silent section by extracting statistical features having a relatively simple calculation process from the input signal may be used. Can be.

A commonly used method is a method of determining a speech section and a silent section by obtaining an energy value (or log energy value) in every section of an input signal and comparing the threshold with a predetermined threshold value.

Also, the preprocessing process may include a noise processing process for removing noise.

Meanwhile, the preprocessing process may further include necessary additional noise processing and signal processing.

Thereafter, the speech recognition engine may extract a feature vector (parameter) effective for recognition from the input speech input signal. Here, a method based on LPC (Linear Predictive Coefficients) and a Mel Frequency Cepstral Coefficients (MFCC) extraction method may be used.

Thereafter, the speech recognition engine may recognize a pattern of a speech signal and the extracted feature parameter, and an acoustic model may be used for pattern recognition.

This is a method of modeling and comparing the signal characteristics of speech, and a direct comparison method of setting a recognition object as a feature vector model and comparing it with a feature vector of an input signal may be used. In the direct comparison method, a unit of a word or phoneme to be recognized as a feature vector model can be set, and how similar the input speech can be.

Alternatively, a statistical method may be used that statistically processes the feature vector of the object to be recognized. This statistical method is composed of units of state sequence (State Sequence) and can use the relation between the state columns, and compare the DTW (Dynamic Time Warping) method using temporal arrangement, probability value, average, variance HMM (Hidden Markov Model) method.

In more detail, DTW (Dynamic Time Warping) is a method of obtaining a distance between a reference voice signal and an input voice signal by using dynamic programming, and is mainly used to construct a speaker-dependent isolated word recognition system and has a high recognition rate.

HMM (Hidden Markov Model) expresses the transition of negative state from one state to the next state as a transition probability.The actual voice signal is formed by constructing a model representing them from the training data using the temporal statistical characteristics of the voice signal. A probability model with high similarity to and is adopted as a recognition result.

Thereafter, the speech recognition engine may determine data corresponding to the received speech input signal and output a recognition result.

On the other hand, the data determination process may include a language processing operation using a language model.

The language model is generally a search for probability values for all possible word sequences, a grammar-based approach that considers only those strings that are grammatically correct for a given situation during a possible combination of words, from a database of spoken speech in a given situation. For example, a statistical based method of statistically estimating probability values of possible word strings may be used.

Meanwhile, data used for speech recognition, such as the acoustic model and the language model, may be pre-stored in the storage 160 and updated by a user or a manufacturer.

According to an exemplary embodiment of the present invention, a search can be performed based on a sound model and a language model of a category related to a predetermined image or object, thereby improving speed and accuracy of speech recognition.

Meanwhile, the portable terminal 100 according to an embodiment of the present invention may be connected to a network with an external electronic device through the wireless internet module 115 or the like.

In this case, the step (S440) of recognizing the voice command may include extracting a feature vector based on the received voice input signal, and transmitting the feature vector and the keyword to an electronic device connected through a network. And receiving a voice recognition result from the electronic device.

That is, the extracted feature vector and the keyword may be transmitted to an external speech recognition engine or a speech recognition server, and the result value may be received.

Natural language processing (NLP) is a technology that understands and analyzes human language using speech-to-text (STT) results. More operations may be required, such as searching. Therefore, it may be more effective to use an external server for such natural language processing.

For example, the voice recognition engine of the mobile terminal 100 may transmit the voice signal to the STT server when the received voice input signal or the extracted feature vector is needed, and receive a text recognition result from the STT server. .

In addition, the mobile terminal 100 may transmit the first speech recognition result processed or received internally to a natural language processing server (NLU Server), and may receive data for natural language processing corresponding to a user's intention.

5A to 8C are views referred to for describing various embodiments of a method of operating a mobile terminal according to the present invention. Hereinafter, although a long-press input is set as a command for entering the voice recognition mode, the present invention is not limited thereto.

5A through 5E illustrate a case in which a predetermined image is an album picture view screen or an execution screen of a photo related application.

Referring to the drawings, in the album screen 610 including a plurality of pictures, the user may enter the voice recognition mode. When the long press input is set as the voice recognition mode entry command, the long press input 520 may enter the voice recognition mode with the long press input 520 for the predetermined image 510 or the specific object 511.

When the user enters the voice recognition mode, a window 530 displaying a message or a recognition result on a predetermined area of the display unit 151, for example, the lower end, and icons 532, 534, and 535 indicating the voice command input mode. Can be displayed.

On the other hand, unlike FIG. 5B, the icon 532 indicating the voice command input mode may be displayed so as not to overlap with the window 530.

 The window 530 may display a help message 531 in a voice recognition standby state, and output a recognized result value 533 when a voice command is input.

Meanwhile, as illustrated in FIGS. 5B to 5D, icons 532, 534, and 535 indicating the voice command input mode may be displayed differently according to the progress of the voice recognition process.

The controller 180 may perform an operation corresponding to the voice command recognition result. For example, as shown in the e-mail transmission screen 540 illustrated in FIG. 5E, the user may send a long-pressed picture by e-mail to the counterpart.

5A to 5E, when the user inputs a command to send the A picture 511 to B by mail (c), the user does not input a voice command for the A picture. The controller 180 may determine that there is a voice command related to a command related to the A picture 511 through a touch on the A picture 511, and further combines the A picture 511 with a voice input of the user. Accurate intent results can be obtained. In addition, since the amount of searching and calculations required for speech recognition can be reduced, the speed can be increased.

6A to 6C illustrate a case where a predetermined image is an application list screen. The application list screen may be a home screen.

Even in an application list screen including an application area 610 including a plurality of applications and a launcher area in which a fixed application is disposed, the user may long press a desired application icon 621 to perform a voice recognition mode. Once you enter and talk to what you want, you can automatically connect to and run the relevant application.

6A to 6C, when the user long presses the call icon 621, a window 630 including a message 531 and an icon 623 at the bottom and recognizing and dictating the user's language may be used. ) May be displayed.

When a user inputs a name or ID of a counterpart who wants to make a call, the controller 180 recognizes this and connects the call with the input counterpart 640.

7A to 7D illustrate a case where a continuous voice command is input when a predetermined image is an application list screen.

Referring to the drawings, in an application screen including a plurality of applications, a user may long press a desired application icon 710 to enter a voice recognition mode, and thus a window 720 is displayed.

 When the user inputs a voice, the window 720 displays the voice recognition result, and if the user wants to speak, the user can search for information or execute a function in the application.

In addition, while the search information 731, 732, 733 is displayed, the user presses a long press on any one of the search information 731, 732, 733 again in the application, When the command 740 is input, the result may be output 750.

8A to 8C illustrate a case in which a predetermined image is an image captured by a camera or a preview image.

Referring to the drawings, the camera 121 is executed to photograph a subject. In a state in which the captured image or the preview image 810 is displayed on the display unit 151, the user may long press 811 the subject 812 to be examined.

Accordingly, the controller 180 may enter the voice recognition mode to receive a user's voice command and display the recognition result in the window 820.

In addition, the controller 180 may perform a corresponding operation according to the voice recognition result and output the result 830.

According to the embodiment of the present invention, the user's voice command can be recognized more quickly and accurately, thereby improving user convenience.

On the other hand, the present invention can be implemented as a processor-readable code in a processor-readable recording medium provided in the portable terminal. The processor-readable recording medium includes all kinds of recording devices that store data that can be read by the processor. Examples of the processor-readable recording medium include ROM, RAM, CD-ROM, magnetic tape, floppy disk, optical data storage device, and the like, and also include a carrier wave such as transmission through the Internet. The processor-readable recording medium can also be distributed over network coupled computer systems so that the processor-readable code is stored and executed in a distributed fashion.

In addition, although the preferred embodiment of the present invention has been shown and described above, the present invention is not limited to the above-described specific embodiment, the technical field to which the invention belongs without departing from the spirit of the invention claimed in the claims. Of course, various modifications can be made by those skilled in the art, and these modifications should not be individually understood from the technical spirit or the prospect of the present invention.

110: wireless communication unit 120: A / V input unit
130: user input unit 140: sensing unit
150: output unit 151: display unit
160: memory 170: interface unit
180: control unit

Claims (18)

Displaying a predetermined image on a display unit;
Entering a voice command input mode;
Receiving a voice input signal;
Recognizing a voice command based on the received voice input signal and the displayed predetermined image; And,
Executing an operation corresponding to the recognized voice command;
Extracting a keyword based on the predetermined image;
The recognizing of the voice command may include recognizing the voice command based on the keyword and the received voice input signal. The method of claim 1,
In the step of entering the voice command input mode, a window for displaying a message or a result of recognition is displayed on a predetermined area of the display unit, and an icon indicating the voice command input mode is displayed. The method of claim 2,
And a display state of the icon indicating the voice command input mode is changed as the voice recognition process proceeds. The method of claim 1,
Receiving a long-press input; further comprising:
And entering the voice command input mode according to the long-press input. The method of claim 4, wherein
The recognizing of the voice command may include recognizing the voice command based on the long-press input object and the received voice input signal among objects included in the image. delete The method of claim 1,
Recognizing the voice command,
Extracting a feature vector based on the received voice input signal; And,
Comparing the feature vector with data of a category related to the keyword in a speech database including an acoustic model and a language model, and determining data corresponding to the received speech input signal; Method of operation of a portable terminal comprising a. The method of claim 1,
Recognizing the voice command,
Extracting a feature vector based on the received voice input signal; And,
Transmitting the feature vector and the keyword to an electronic device connected through a network;
And receiving a result of speech recognition from the electronic device. The method of claim 1,
Acquiring an image through a camera; further comprising:
And the predetermined image is an image obtained through the camera. A display unit displaying a predetermined image;
A microphone (MIC) for receiving a voice input signal in a voice command input mode; And,
And a controller configured to recognize a voice command based on the received voice input signal and the displayed predetermined image, and to control an operation corresponding to the recognized voice command.
The controller extracts a keyword based on the predetermined image, and recognizes the voice command based on the keyword and the received voice input signal. The method of claim 10,
The control unit controls to display a window for displaying a message or a recognition result in a predetermined area of the display unit, and an icon indicating the voice command input mode in the voice command input mode. . The method of claim 11,
And the controller controls the display state of the icon representing the voice command input mode to change as the voice recognition process proceeds. The method of claim 10,
The control unit, when receiving a long-press (long-press) input, characterized in that the control to enter the voice command input mode, characterized in that the portable terminal. The method of claim 13,
The controller may be configured to recognize the voice command based on the long-press input object and the received voice input signal among the objects included in the image. delete The method of claim 10,
The controller extracts a feature vector based on the received voice input signal, and includes the feature vector data of a category related to the keyword in a voice database including an acoustic model and a language model. And the data corresponding to the received voice input signal is determined. The method of claim 10,
And a wireless internet module for connecting to an external electronic device through a network.
The controller extracts a feature vector based on the received voice input signal, transmits the feature vector and the keyword to the electronic device, and receives a voice recognition result from the electronic device. Mobile terminal. The method of claim 10,
And a camera for acquiring the predetermined image.

Images