KR20150025611A - Mobile terminal and operation method thereof - Google Patents

Abstract

The present invention relates to a method for controlling operation of a mobile terminal for the visually impaired, comprising the steps of: displaying a preview image input through a camera on a display unit; executing pattern recognition on subjects in the preview image; providing a voice service on the preview image; and photographing the preview image in response to a user′s photographing command.

Description

[0001] MOBILE TERMINAL AND OPERATION METHOD THEREOF [0002]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a mobile terminal for a visually impaired person and a method of controlling an operation thereof, and more particularly, to a mobile terminal for taking an image based on voice guidance and a method of controlling the operation thereof.

A mobile terminal is a portable device having one or more functions capable of carrying out voice and video communication, capable of inputting and outputting information, and storing data, while being portable. As these functions are diversified, the mobile terminal has complicated functions such as photographing and photographing, reproduction of music files and video files, games, broadcast reception, wireless Internet, and message transmission and reception. .

In order to implement complex functions, mobile terminals implemented in the form of multimedia devices are being applied variously in terms of hardware and software. For example, there is a user interface environment in which a user can easily and conveniently search for or select a function.

In addition, for example, there is a user interface environment for providing a voice recognition function for a visually impaired person and / or a voice service function (i.e., an accessibility function). Here, the voice recognition function is a function of controlling a terminal used by a person in daily life without using a mouse or a keyboard, by using a voice which is the most friendly and convenient communication tool possessed by a person, It is a function that supports to receive information service. The voice service function refers to a voice service function that reads a portion of a mobile terminal by touching the touch screen using a finger or the like instead of looking at the touch screen. Examples of such voice service functions include talkback of Android and voice over of iOS.

However, the conventional voice service function provides a basic voice service only for the basic operation of the mobile terminal and the operation until the specific application is executed. Accordingly, there is an urgent need for a concrete method for supporting a voice service so that a visually impaired user using the mobile terminal can easily perform various operations of the application when executing a specific application.

The present invention provides a mobile terminal and a method of controlling the operation of the mobile terminal that can easily photograph a desired subject based on a voice service provided in an image photographing mode.

In addition, the present invention provides a mobile terminal and a method of controlling the same, which can easily recognize contents of an image displayed on a screen based on a voice service provided in an image view mode.

The method includes displaying a preview image input through a camera on a display unit; Performing shape recognition on the objects of the preview image; Providing a voice service for the preview image; And photographing the preview image corresponding to a user's photographing command.

According to an embodiment of the present invention, a mobile terminal provides a voice service related to a preview image in real time in an image photographing mode, thereby allowing a visually impaired person to easily photograph a desired subject.

Also, according to another embodiment of the present invention, the mobile terminal provides the voice service related to the image displayed on the screen in the image view mode in real time, thereby allowing the visually impaired to easily recognize the contents of the image.

Meanwhile, various other effects will be directly or implicitly disclosed in the detailed description according to the embodiment of the present invention to be described later.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention;
FIG. 2 is a perspective view of a mobile terminal according to an embodiment of the present invention; FIG.
FIG. 3 is a rear perspective view of the mobile terminal shown in FIG. 2; FIG.
4 is a flowchart illustrating an operation of a mobile terminal according to a first embodiment of the present invention;
5 is a diagram illustrating an operation of a mobile terminal for setting a voice service function for a visually impaired person;
6 and 7 are diagrams illustrating operations of a mobile terminal that executes a camera application according to a user's touch input or voice command;
8 is a diagram illustrating an operation of a mobile terminal providing a voice service related to a preview image;
9 is a view for explaining an operation of a mobile terminal providing a voice service for a touched subject when a touch is input to a preview image;
10 is a diagram illustrating an operation of a mobile terminal providing a voice service such that a subject to be photographed is located at the center of a preview image;
11 is a diagram illustrating an operation of a mobile terminal that performs image capturing based on a voice command of a user;
12 is a diagram illustrating an operation of a mobile terminal providing a voice service for sharing a photographed image with a third party;
13 is a flowchart illustrating an operation of a mobile terminal according to a second embodiment of the present invention;
14 is a diagram illustrating an operation of a mobile terminal that executes a gallery application according to a touch input of a user;
15 is a view for explaining an operation of a mobile terminal providing a voice service relating to an image displayed on a screen;
16 is a view for explaining an operation of a mobile terminal providing voice service regarding a touched object when a touch is inputted to an image;

Hereinafter, the present invention will be described in detail with reference to the drawings.

Examples of the mobile terminal described in the present specification include a mobile phone, a smart phone, a notebook computer, a digital broadcasting terminal, a PDA (Personal Digital Assistants), a PMP (Portable Multimedia Player), a camera, tablet computers, e-book terminals, and the like. In addition, suffixes "module" and " part "for the components used in the following description are given merely for convenience of description, and do not give special significance or role in themselves. Accordingly, the terms "module" and "part" may be used interchangeably.

1 is a block diagram of a mobile terminal according to an embodiment of the present invention. Referring to FIG. 1, a mobile terminal according to an exemplary embodiment of the present invention will be described in terms of functional components.

1, a mobile terminal 100 includes a wireless communication unit 110, an audio / video (A / V) input unit 120, a user input unit 130, a sensing unit 140, an output unit 150, A controller 160, an interface 170, a controller 180, and a power supply 190. When such components are implemented in practical applications, two or more components may be combined into one component, or one component may be divided into two or more components as necessary.

The wireless communication unit 110 may include a broadcast receiving module 111, a mobile communication module 113, a wireless Internet module 115, a short distance communication module 117, and a GPS module 119.

The broadcast receiving module 111 receives at least one of a broadcast signal and broadcast related information from an external broadcast management server through a broadcast channel. At this time, the broadcast channel may include a satellite channel, a terrestrial channel, and the like. The broadcast management server may refer to a server for generating and transmitting at least one of a broadcast signal and broadcast related information and a server for receiving at least one of the generated broadcast signal and broadcast related information and transmitting the broadcast signal to the terminal.

The broadcast signal may include a TV broadcast signal, a radio broadcast signal, a data broadcast signal, and a broadcast signal in which a data broadcast signal is combined with a TV broadcast signal or a radio broadcast signal. The broadcast-related information may mean information related to a broadcast channel, a broadcast program, or a broadcast service provider. The broadcast-related information can also be provided through a mobile communication network, in which case it can be received by the mobile communication module 113. Broadcast-related information can exist in various forms.

The broadcast receiving module 111 receives broadcast signals using various broadcasting systems. In particular, the broadcast receiving module 111 may be a Digital Multimedia Broadcasting-Terrestrial (DMB-T), a Digital Multimedia Broadcasting-Satellite (DMB-S) ), Digital Video Broadcast-Handheld (DVB-H), Integrated Services Digital Broadcast-Terrestrial (ISDB-T), and the like. In addition, the broadcast receiving module 111 may be configured to be suitable for all broadcasting systems that provide broadcasting signals, as well as the digital broadcasting system. The broadcast signal and / or broadcast related information received through the broadcast receiving module 111 may be stored in the memory 160.

The mobile communication module 113 transmits and receives a radio signal to at least one of a base station, an external terminal, and a server on a mobile communication network. Here, the wireless signal may include various types of data according to a voice call signal, a video call signal, or a text / multimedia message transmission / reception.

The wireless Internet module 115 is a module for wireless Internet access, and the wireless Internet module 115 can be built in or externally attached to the mobile terminal 100. WLAN (Wi-Fi), Wibro (Wireless broadband), Wimax (World Interoperability for Microwave Access), HSDPA (High Speed Downlink Packet Access) and the like can be used as wireless Internet technologies.

The short-range communication module 117 refers to a module for short-range communication. Bluetooth, radio frequency identification (RFID), infrared data association (IrDA), ultra wideband (UWB), ZigBee, and Near Field Communication (NFC) may be used as the short distance communication technology.

A GPS (Global Position System) module 119 receives position information from a plurality of GPS satellites.

An A / V (Audio / Video) input unit 120 is for inputting an audio signal or a video signal, and may include a camera 121 and a microphone (not shown). The camera 121 processes image frames such as still images or moving images obtained by the image sensor in the video communication mode or the photographing mode. Then, the processed image frame can be displayed on the display unit 151. [

The image frame processed by the camera 121 may be stored in the memory 160 or transmitted to the outside through the wireless communication unit 110. [ The camera 121 may be equipped with two or more cameras according to the configuration of the terminal.

A microphone (not shown) receives an external sound signal by a microphone in a communication mode, a recording mode, a voice recognition mode, and the like and processes it as electrical voice data. The processed voice data can be converted into a form that can be transmitted to the mobile communication base station through the mobile communication module 113 and output when the voice data is in the call mode. The microphone (not shown) may be used with various noise reduction algorithms to remove noise generated in receiving an external sound signal.

The user input unit 130 generates key input data that the user inputs to control the operation of the terminal. The user input unit 130 may include a key pad, a dome switch, and a touch pad (static / static) capable of receiving commands or information by a user's pressing or touching operation. The user input unit 130 may be a jog wheel for rotating the key, a jog type or a joystick, or a finger mouse. Particularly, when the touch pad has a mutual layer structure with the display unit 151 described later, it can be called a touch screen.

The sensing unit 140 senses the current state of the mobile terminal 100 such as the open / close state of the mobile terminal 100, the position of the mobile terminal 100, Thereby generating a sensing signal. For example, when the mobile terminal 100 is in the form of a slide phone, it is possible to sense whether the slide phone is opened or closed. In addition, a sensing function related to whether or not the power supply unit 190 is powered on, whether the interface unit 170 is coupled to an external device, and the like can be handled.

The sensing unit 140 may include a proximity sensor 141, a pressure sensor 143, a motion sensor 145, and the like. The proximity sensor 141 can detect an object approaching the mobile terminal 100 or the presence or absence of an object in the vicinity of the mobile terminal 100 without mechanical contact. The proximity sensor 141 can detect a nearby object by using a change in the alternating magnetic field or a change in the static magnetic field, or a rate of change in capacitance. The proximity sensor 141 may be equipped with two or more sensors according to the configuration.

The pressure sensor 143 can detect whether or not pressure is applied to the mobile terminal 100, the magnitude of the pressure, and the like. The pressure sensor 143 may be installed at a portion where the pressure of the mobile terminal 100 is required according to the use environment. When the pressure sensor 143 is installed on the display unit 151, the touch input through the display unit 151 and the pressure applied by the touch input The pressure touch input can be identified. Also, the magnitude of the pressure applied to the display unit 151 at the time of the pressure touch input can be determined according to the signal output from the pressure sensor 143. [

The motion sensor 145 senses the position or movement of the mobile terminal 100 using an acceleration sensor, a gyro sensor, or the like. An acceleration sensor that can be used for the motion sensor 145 is a device that converts an acceleration change in one direction into an electric signal and is widely used along with the development of MEMS (micro-electromechanical systems) technology.

The acceleration sensor measures the acceleration of a small value built in the airbag system of an automobile and recognizes the minute motion of the human hand and measures the acceleration of a large value used as an input means such as a game There are various kinds. Acceleration sensors are usually constructed by mounting two or three axes in one package. Depending on the usage environment, only one axis of Z axis is required. Therefore, when the acceleration sensor in the X-axis direction or the Y-axis direction is used instead of the Z-axis direction for some reason, the acceleration sensor may be mounted on the main substrate by using a separate piece substrate.

The gyro sensor is a sensor for measuring the angular velocity, and it can sense the direction of rotation with respect to the reference direction.

The output unit 150 is for outputting an audio signal, a video signal, or an alarm signal. The output unit 150 may include a display unit 151, an audio output module 153, an alarm unit 155, and a haptic module 157.

The display unit 151 displays and outputs information processed by the mobile terminal 100. For example, when the mobile terminal 100 is in the call mode, a UI (User Interface) or a GUI (Graphic User Interface) associated with a call is displayed. When the mobile terminal 100 is in the video communication mode or the photographing mode, the photographed or received images can be displayed individually or simultaneously, and the UI and the GUI are displayed.

Meanwhile, as described above, when the display unit 151 and the touch pad have a mutual layer structure to constitute a touch screen, the display unit 151 may be an input device capable of inputting information by a user's touch in addition to the output device Can also be used.

If the display unit 151 is configured as a touch screen, it may include a touch screen panel, a touch screen panel controller, and the like. In this case, the touch screen panel is a transparent panel that is attached to the outside, and can be connected to the internal bus of the mobile terminal 100. The touch screen panel keeps a watch on the contact result, and if there is a touch input, sends the corresponding signals to the touch screen panel controller. The touch screen panel controller processes the signals, and then transmits corresponding data to the controller 180 so that the controller 180 can determine whether the touch input has been made and which area of the touch screen has been touched.

The display unit 151 may be formed of an e-paper. Electronic paper (e-Paper) is a kind of reflective display, and has excellent visual characteristics such as high resolution, wide viewing angle and bright white background as conventional paper and ink. The electronic paper (e-paper) can be implemented on any substrate such as plastic, metal, paper, and the image is maintained even after the power is shut off, and the battery life of the mobile terminal 100 is long Can be maintained. As the electronic paper, a hemispherical twist ball filled with a telephone can be used, or an electrophoresis method and a microcapsule can be used.

In addition, the display unit 151 may be a liquid crystal display, a thin film transistor-liquid crystal display, an organic light-emitting diode, a flexible display, a three- (3D display). In addition, there may be two or more display units 151 according to the embodiment of the mobile terminal 100. For example, the mobile terminal 100 may include an external display unit (not shown) and an internal display unit (not shown) at the same time.

The audio output module 153 outputs audio data received from the wireless communication unit 110 or stored in the memory 160 in a call signal reception mode, a call mode or a recording mode, a voice recognition mode, a broadcast reception mode, In addition, the sound output module 153 outputs sound signals related to functions performed in the mobile terminal 100, for example, call signal reception tones, message reception tones, and the like. The sound output module 153 may include a speaker, a buzzer, and the like.

The alarm unit 155 outputs a signal for notifying the occurrence of an event of the mobile terminal 100. Examples of events that occur in the mobile terminal 100 include call signal reception, message reception, and key signal input. The alarm unit 155 outputs a signal for notifying the occurrence of an event in a form other than an audio signal or a video signal. For example, it is possible to output a signal in a vibration mode. The alarm unit 155 can output a signal to notify when a call signal is received or a message is received. Also, when the key signal is inputted, the alarm unit 155 can output the signal as the feedback to the key signal input. The user can recognize the occurrence of an event through the signal output by the alarm unit 155. A signal for notifying the occurrence of an event in the mobile terminal 100 may also be output through the display unit 151 or the sound output module 153. [

The haptic module 157 generates various tactile effects that the user can feel. A typical example of the haptic effect generated by the haptic module 157 is a vibration effect. When the haptic module 157 generates vibration with a haptic effect, the intensity and pattern of the vibration generated by the haptic module 157 can be converted, and the different vibrations can be synthesized and output or sequentially output.

In addition to the vibration, the haptic module 157 may be provided with a function of stimulating by a pin arrangement vertically moving with respect to the contact skin surface, an effect of stimulating air through the injection or suction force of the air through the injection port or the suction port, The effect of stimulation through contact of the electrode (eletrode), the effect of stimulation by the electrostatic force, and the effect of reproducing the cool / warm using the device capable of endothermic or exothermic can be generated. The haptic module 157 can be implemented not only to transmit the tactile effect through direct contact but also to feel the tactile effect through the muscular sense of the user's finger or arm. The haptic module 157 may include two or more haptic modules 157 according to the configuration of the mobile terminal 100.

The memory 160 may store a program for processing and controlling the control unit 180 and may store a function for temporarily storing input or output data (e.g., a phone book, a message, a still image, .

The memory 160 may be a flash memory type, a hard disk type, a multimedia card micro type, a card type memory (for example, SD or XD memory), a RAM , And a ROM. ≪ / RTI > In addition, the mobile terminal 100 may operate a web storage for storing the memory 150 on the Internet.

The interface unit 170 serves as an interface with all external devices connected to the mobile terminal 100. Examples of the external device connected to the mobile terminal 100 include a wired / wireless headset, an external charger, a wired / wireless data port, a memory card, a SIM (Subscriber Identification Module) card, a UIM An audio input / output (I / O) terminal, a video I / O (input / output) terminal, and an earphone. The interface unit 170 may receive data from the external device or supply power to the respective components in the mobile terminal 100 and may transmit data in the mobile terminal 100 to the external device .

The interface unit 170 may be a path through which the power from the cradle connected to the mobile terminal 100 is connected to the cradle when the mobile terminal 100 is connected to the cradle, And may be a passage to be transmitted to the terminal 100.

The controller 180 typically controls the operation of the respective units to control the overall operation of the mobile terminal 100. For example, voice communication, data communication, video communication, and the like. In addition, the control unit 180 may include a multimedia playback module 181 for multimedia playback. The multimedia playback module 181 may be configured in hardware in the controller 180 or separately from software in the controller 180. [

The voice recognition unit 182 drives the voice recognition engine to which the voice recognition algorithm is applied to recognize the voice of the user input to the microphone 123. [

That is, the voice recognition unit 182 converts the voice of the user input through the microphone 122 into digital data, pre-emphasizes the converted digital data, The end point is detected.

Then, the voice recognition unit 182 extracts voice feature values for the voice between the detected start point and end point, and matches the extracted voice feature values in the voice recognition database provided in the memory 160 And informs the controller 180 of the operation of the mobile terminal 100. At this time, the voice feature values may include at least one of a waveform of a voice input from the microphone 123, a format of a voice, and a pitch of a voice.

For example, if the operation that matches the voice feature value extracted by the voice recognition unit 182 in the voice recognition database is the "telephone directory ", the control unit 180 executes the" ) Display the "Phonebook" menu on the screen. If the operation that matches the voice feature value extracted by the voice recognition unit 182 in the voice recognition database is "text", the control unit 180 displays "text" on the screen of the display unit 151.

The voice recognition unit 182 may be provided in the mobile terminal 100 in the form of one "module" or "unit", or may be provided in the memory 160 in the form of software. The speech recognition unit 182 may be included in the control unit 180 and the control unit 180 may perform the operation of the speech recognition unit 182 in the same manner.

Meanwhile, the power supply unit 190 receives external power and internal power under the control of the controller 180 and supplies power required for operation of the respective components.

The mobile terminal 100 having such a configuration can be configured to be operable in a communication system capable of transmitting data through a frame or a packet, including a wired / wireless communication system and a satellite-based communication system. have.

FIG. 2 is a perspective view of a mobile terminal according to an embodiment of the present invention, and FIG. 3 is a rear perspective view of the mobile terminal shown in FIG. Hereinafter, with reference to FIG. 2 and FIG. 3, a mobile terminal according to the present invention will be described in terms of components according to the external appearance. Hereinafter, for convenience of description, a bar type mobile terminal having a front touch screen among various types of mobile terminals such as a folder type, a bar type, a swing type, a slider type, etc. will be described as an example. However, the present invention is not limited to the bar-type mobile terminal but can be applied to all types of mobile terminals including the above-mentioned types.

Referring to FIG. 2, the case constituting the appearance of the mobile terminal 100 is formed by the front case 100-1 and the rear case 100-2. Various electronic components are incorporated in the space formed by the front case 100-1 and the rear case 100-2.

The display unit 151, the first sound output module 153a, the first camera 121a, and the first through third user input units 130a, 130b, and 130c are connected to the main body, specifically, the front case 100-1 . A fourth user input portion 130d may be disposed on a side surface of the rear case 100-2 and a microphone (not shown) may be disposed on a lower surface of the rear case 100-2.

The display unit 151 may be constructed such that the touch pad is overlapped with the layer structure so that the display unit 151 operates as a touch screen so that information can be input by a user's touch.

The first acoustic output module 153a may be implemented in the form of a receiver or a speaker. The first camera 121a may be implemented in a form suitable for capturing an image or a moving image of a user. The microphone (not shown) may be implemented in a form suitable for receiving a user's voice, other sounds, and the like.

The first to fourth user input units 130a, 130b, 130c and 130d and the fifth user input unit 130e to be described later may collectively be referred to as a user input unit 130. The user input unit 130 may be a method tactile manner, any method can be employed.

For example, the user input unit 130 may be embodied as a dome switch or a touch pad capable of receiving a command or information by a user's pressing or touching operation, or may be a wheel, jog system, Or the like.

In a functional aspect, the first user input unit 130a is a menu key for inputting a command for calling a menu related to an application currently being executed, and the second user input unit 130b is a home key, The third user input unit 130c is for inputting a command for canceling the currently executing application as a back key.

Also, the fourth user input unit (i.e., the side key 130d) may operate as a key for switching between a sleep mode and a wake-up mode, or may operate as a special function within the mobile terminal 100 As a hot-key for activating the < / RTI > Here, the sleep mode indicates a state in which the backlight of the display unit 151 is turned off to save battery consumption of the mobile terminal, and the wake-up mode indicates a state in which the backlight of the display unit 151 is turned on.

3, a second camera 121b may be further mounted on the rear surface of the rear case 100-2, a fifth user input unit 130e may be disposed on a side surface of the rear case 100-2 And the interface unit 170 may be disposed on the lower surface of the rear case 100-2.

The second camera 121b has a photographing direction substantially opposite to that of the first camera 121a, and may have pixels different from those of the first camera 121a. A flash (not shown) and a mirror (not shown) may be additionally disposed adjacent to the second camera 121b. In addition, another camera may be installed adjacent to the second camera 121b to use it for shooting a three-dimensional stereoscopic image.

The flash illuminates the subject when the subject is photographed by the second camera 121b. The mirror enables the user to illuminate the user's own face or the like when the user intends to photograph (self-photograph) himself / herself using the second camera 121b.

A second sound output module (not shown) may be further disposed in the rear case 100-2. The second sound output module may implement the stereo function together with the first sound output module 153a, and may be used for talking in the speakerphone mode.

The interface unit 170 can be used as a path for exchanging data with an external device. An antenna for receiving broadcast signals (not shown) may be disposed in one area of the front case 100-1 and the rear case 100-2 in addition to the antenna for communication. The antenna may be installed to be capable of being drawn out from the rear case 100-2.

A power supply unit 190 for supplying power to the mobile terminal 100 may be mounted on the rear case 100-2. The power supply unit 190 may be a rechargeable battery, for example, and may be detachably coupled to the rear case 100-2 for charging or the like.

On the other hand, in the present embodiment, the second camera 121b and the like are disposed in the rear case 100-2, but the present invention is not limited thereto. Also, the first camera 121a may be rotatably formed so that the second camera 121b can be photographed up to the photographing direction, even if the second camera 121b is not separately provided.

In the foregoing, the configuration of the mobile terminal 100 according to the present invention has been described with reference to FIG. 1 to FIG. Hereinafter, a mobile terminal and its operation control method for easily photographing a desired subject based on various voice services provided in a picture taking mode according to a first embodiment of the present invention will be described in detail.

4 is a flowchart illustrating an operation of the mobile terminal according to the first embodiment of the present invention.

Referring to FIG. 4, the controller 180 sets a voice service function for the visually impaired according to a user command (S405). At this time, the talk back function of the Android, the voice over function of the iOS, and the like can be used as the voice service function to be set. In addition, a voice service function developed by the manufacturer of the mobile terminal may be used.

In setting the voice service function, the control unit 180 can activate the voice recognition unit 182 to enter the voice recognition mode. At this time, the mobile terminal 100 can not only output a voice service for a visually impaired person but also receive a voice command for controlling the mobile terminal.

After the voice service function is set, the controller 180 displays a home screen or menu screen including a plurality of icons on the display 151 according to a user command. At this time, the plurality of icons include icons corresponding to at least one application.

The control unit 180 executes a camera application (hereinafter referred to as a 'camera app') based on a user's touch input or a voice command (S410).

For example, in one embodiment, when the camera icon is touched according to a voice guidance on the icons of the home or menu displayed on the menu screen, the control unit 180 executes the camera app and notifies the user that the camera icon has been selected and / Lt; RTI ID = 0.0 > a < / RTI >

In addition, in another embodiment, the control unit 180 may control the camera application (e.g., a camera application or a camera application application) based on a user's voice command And may provide a voice service to notify the user that the camera icon has been selected and / or executed.

When the camera application is executed through the above process, the control unit 180 displays the preview image input through the camera 121 on the display unit 151 (S415).

If the direction of the camera is fixed or the auto focus is performed for a predetermined time while the preview image is being displayed, the controller 180 performs shape recognition on the object (or the subject) in the preview image do. At this time, the object to be subjected to shape recognition is not limited, but it is preferable that the object has a certain size and shape.

The shape recognition detects an object shape belonging to a predetermined model class from a preview image and restores the geometric information of the detected object shape, and a control point detection method and an object shape representation method are applied. The shape recognition technology applies currently used shape recognition technology, and its range is not limited, so a detailed description will be omitted for the sake of simplicity in this specification.

The control unit 180 provides a voice service on the preview image based on the shape recognition of the objects constituting the preview image (S420).

More specifically, the control unit 180 can guide the user to easily arrange and combine the subject by sequentially informing the information (i.e., name) about the subjects included in the preview image in accordance with a predetermined rule . In this case, the order of guiding the subject is the order in which the subjects of the preview image are stacked from the downward direction to the upward direction, the order in which the subjects are arranged from the left to the right direction of the preview image, The order in which they are arranged, the order according to the size of the subjects, or the order according to the distance between the subject and the photographer, and the like.

In addition, the control unit 180 allows the user to easily guess the position and shape of the objects by making the difference in height, strength, tone, etc. of the voice guiding the objects.

For example, when the height of the voice guiding the subject is "low ", the size of the subject is " large subject ", and when the height of voice is high, the subject is small. , And when the height of the voice is "normal ", it indicates that the size of the subject is a medium subject.

In addition, for example, when the tone of the voice guiding the subject is "soft sound", the shape of the subject is a "round shape subject", and when the tone of voice is "sharp sound", the shape of the subject is sharp Shaped object ", and when the tone of the voice is" normal ", it indicates that the shape of the subject is 'halfway between the round shape and the sharp shape'.

Further, for example, when the intensity of the voice guiding the subject is "a loud sound (i.e., strong sound) ", the distance between the user and the subject is a 'close subject' Sound ") indicates that the distance from the subject is a 'distant subject', and when the intensity of the voice is 'normal', the distance from the subject is 'medium.'

On the other hand, when earphone or headphone is mounted on the mobile terminal 100, stereo discrimination of voice guidance is possible. Accordingly, the controller 180 recognizes the position of the subject, provides voice guidance through the right earphone for the subject located in the right area of the subject review image, and provides voice guidance through the left earphone for the subject located in the left area can do.

In addition, if the number of the objects in the preview image is large and the type and location thereof are not well memorized after the voice service is provided, the user may touch the screen on which the preview image is displayed, . That is, when a predetermined touch input is received through the screen displaying the preview image, the control unit 180 guides the information about the object located at the touched point by voice. At the same time, the control unit 180 provides a vibration effect of a subject shape at a position where the subject is located, thereby allowing the user to easily guess the position and shape of the subject.

As described above, when the voice service related to the preview image is provided in real time, the visually impaired person can easily identify the information about the objects being photographed through the mobile terminal. Accordingly, when the user speaks the name of a subject to be photographed through the microphone 123 of the mobile terminal, the mobile terminal provides a voice service such that the subject to be photographed is located at the center of the preview image.

That is, when the user's voice signal is received through the microphone 123, the control unit 180 determines whether the voice signal detected through the voice recognition technology is a voice signal related to the subject in the preview image (S425). At this time, the preview image may include not only the preview image of the current time but also the preview image of the previous time.

If it is determined that the voice signal input through the microphone 123 is a voice signal related to the subject existing in the preview image, the controller 180 provides a voice service related to the position of the subject (S430). Accordingly, the user can roughly detect the position of the subject to be photographed in the preview image.

Upon receiving the voice guidance and starting to move the mobile terminal slowly (S435), the control unit 180 provides the voice guidance in real time so that the subject to be photographed is positioned at the center of the preview image. Accordingly, when the subject to be photographed is focused, the control unit 180 guides the voice of "the subject is focused" and ends the voice guidance. At this time, the control unit 180 can provide a voice effect and a vibration effect.

When the touch input is received through the focused object in the state that the preview image is being displayed, the controller 180 guides the information about the object located at the touched point by voice. At the same time, the control unit 180 provides a vibration effect of a subject shape at a position where the subject is located, thereby allowing the user to easily guess the position and shape of the subject.

On the other hand, if the voice signal is not received through the microphone 123, the controller 180 proceeds to step 445 without providing an audio guide regarding the position of the subject.

Thereafter, the control unit 180 determines whether a photographing signal for photographing the preview image displayed on the display unit 151 is received (S445). At this time, the photographing signal may be an input for touching the photographing button icon, or a voice command such as "photographing" or "kimchi ".

If it is determined that the preview signal has not been received and the preview image is changed in accordance with the direction of the camera, the controller 180 returns to step 420 and repeats the above-described processes.

On the other hand, if the photographing signal is received, the control unit 180 photographs the preview image and stores the photographed image in a preset memory area (S450). At this time, the controller 180 may link and store the image of the preview image and the data of the voice service of the preview image.

In addition, the control unit 180 provides a voice service for storing and / or sharing the photographed image (S455). Accordingly, the user can confirm where the photographed image is stored, and also can easily transmit the photographed image to a third party.

If the camera app is terminated (S460) after voice guidance regarding the storage and / or sharing, the controller 180 ends the above-described processes according to the present invention. On the other hand, if the camera app is not terminated, the control unit 180 moves to step 410 and displays the preview image, and then repeats the above-described processes.

5 to 12 are views for explaining the operation of a mobile terminal for taking an image based on voice guidance according to the first embodiment of the present invention. In the drawings, for convenience of explanation, the voice service output from the mobile terminal is shown as a square balloon, and the voice command input to the mobile terminal is shown as a round balloon.

5 is a view for explaining an operation of a mobile terminal for setting a voice service function for a visually impaired person.

5, when the environment setting icon 420 is selected in a state that the menu screen 510 is displayed on the display unit 151, the mobile terminal 100 displays the environment shown in FIG. 5 (b) And displays the setting screen 530 on the display unit 151. [

When the accessibility menu 540 is selected on the preference setting screen 530, the mobile terminal displays the accessibility setting screen 550 on the display unit 151 as shown in FIG. 5C. When the talkback menu 560 is selected on the accessibility setting screen 550, the mobile terminal displays a talkback setting screen 570 as shown in FIG. 5 (d) on the display unit 151 do.

If the off icon 580 is switched to an on icon (not shown) according to a user input in the talkback setting screen 570, the mobile terminal 100 activates a voice service function to provide voice feedback for a visually impaired person do. On the other hand, in the present embodiment, the talkback function of Android is used as the voice service function, but the present invention is not limited thereto.

After the voice service function is set, the mobile terminal 100 activates the voice recognition unit 182 to enter the voice recognition mode. Accordingly, the mobile terminal 100 can receive a voice command for controlling the mobile terminal as well as output a voice service for the visually impaired.

6 and 7 are diagrams illustrating operations of a mobile terminal that executes a camera application according to a user's touch input or voice command.

Referring to FIG. 6, after the voice service function is set, the mobile terminal 100 displays a menu screen 610 including a plurality of icons on the display 151 according to a user command. At this time, the plurality of icons include icons corresponding to at least one application and icons for execution of a specific function.

If the camera icon 620 is touched according to the voice guidance of the plurality of icons in the state that the menu screen 610 is displayed, the mobile terminal 100 displays a voice guidance 630 for informing the user that the camera icon is selected, Lt; / RTI >

Thereafter, the mobile terminal 100 displays the execution screen 630 of the selected camera app on the display unit 151, and provides a voice guidance 650 to inform the user that the camera application has been executed.

7, when a voice command 720 for executing a camera application is received through the microphone 123 while the menu screen 710 is displayed, the mobile terminal 100 displays an execution screen (730) on the display unit (151), and provides a voice guidance (740) to inform the user that "camera application has been executed ".

8 is a view for explaining the operation of the mobile terminal providing the voice service related to the preview image.

Referring to FIG. 8, when the direction of the camera is fixed or auto focus is performed for a predetermined time while the preview image 810 is displayed on the display unit 151, Performs shape recognition on the subjects in the preview image 810. [ In addition, when the preset icon or predetermined user input unit is selected, the mobile terminal 100 performs shape recognition on the objects in the preview image 810. [

The mobile terminal 100 provides the voice service 820 related to the preview image 810 based on the shape recognition. For example, the mobile terminal 100 provides audio guidance in the order of the table 811, the dish 813, the food 815, and the coffee 817 in the order in which the subjects included in the preview image are stacked. Accordingly, the user can easily guess the arrangement and combination of the objects included in the preview image.

In addition, the mobile terminal 100 allows the user to easily estimate the position and shape of the objects by making differences in the height, intensity, tone, etc. of the voice guiding the subjects of the preview image.

For example, since the 'table (811)' displayed on the preview image is 'a subject relatively large and close to the subject', voice guidance is provided with 'loud bass sound' so that the user can infer the position and shape of the table . Also, since the 'dish 813' displayed on the preview image is 'a relatively round shape and the subject is located close to the subject', voice guidance is provided as 'soft and loud', thereby allowing the user to set the position and shape of the dish So that it can be inferred.

In addition, since the 'food 815' displayed on the preview image is a 'relatively normal shape and a subject located close to the subject', the voice guidance is provided as a 'loud sound of the middle', thereby allowing the user to infer the position and shape of the food . In addition, since 'coffee 817' displayed on the preview image is 'a relatively normal shape and a subject located at a distant place', voice guidance is provided with 'small sound of medium sound', thereby allowing the user to infer the position and shape of coffee .

9 is a view for explaining an operation of a mobile terminal providing a voice service for a subject when a touch image is input to a preview image.

As shown in FIG. 8, when the number of the objects in the preview image is large and the type and location of the subject are not well memorized after the voice service regarding the preview image is provided, The user may be provided with a voice service related to the subject while moving slowly after touching.

9, when the touch input is received through the point where the food 815 of the preview image 810 is displayed, the mobile terminal 100 displays information about the food 815 located at the touched point To the voice (830). At the same time, the mobile terminal 100 provides a food-like vibration effect at the point where the food is located, so that the user can easily guess the position and shape of the subject.

As described above, when the voice service related to the preview image 810 is provided in real time, the visually impaired person can easily identify the information about the objects being photographed through the mobile terminal.

10 is a view for explaining an operation of a mobile terminal providing a voice service such that a subject to be photographed is positioned at the center of a preview image.

When the user speaks the name of the subject to be photographed through the microphone 123, the mobile terminal 100 controls the microphone 123 so that the subject corresponding to the voice input through the microphone 123 is positioned at the center of the preview image To provide a voice service.

10, when the voice signal 840 of "coffee" is received via the microphone 123 of the mobile terminal 100, the mobile terminal 100 transmits a voice (voice) regarding the position of the coffee 817 Service 850. < / RTI > Accordingly, the user can roughly detect the position of the subject to be photographed at present.

When the mobile terminal 100 starts moving slowly after receiving the voice guidance, the mobile terminal 100 provides the voice guidance 860 in real time so that the coffee 817 to be photographed is positioned at the center of the preview image. Accordingly, when the user intends to focus on the coffee to be photographed, the mobile terminal 100 outputs the voice "Focused on coffee" and then ends the voice guidance. At this time, the mobile terminal 100 can output the vibration effect together with the voice guidance.

11 is a view for explaining an operation of a mobile terminal that performs image capturing based on a voice command of a user.

11, when a voice signal (for example, kimchi, 870) indicating a photographing instruction is received in a state that the preview image 810 is displayed on the display unit 151, the mobile terminal 100 displays a preview image (810), and stores the photographed image in a predetermined memory area. In addition, the mobile terminal 100 provides a voice guidance 875 regarding the place where the photographed image is stored. In this way, the user can easily confirm where the photographed image is stored.

12 is a view for explaining an operation of a mobile terminal providing a voice service for sharing a photographed image with a third party.

Referring to FIG. 12, at the end of shooting, the mobile terminal 100 provides a voice guidance 880 for inquiring the user whether to share the photographed image with a third party. Upon receiving the voice signal 885 of "YES " from the user, the mobile terminal 100 provides a voice guidance 890 inquiring to whom to transmit. According to the voice guidance, when the user speaks the name of the person (for example, Young-Hee, 895) to be shared, the mobile terminal 100 displays the image photographed on the partner terminal based on the contact information of ' send. At this time, the photographed image may be transmitted through a message, an e-mail or a chat application or the like depending on the user's setting.

On the other hand, in the present embodiment, it is exemplified that various audio services are provided at the time of photographing, but it will be apparent to those skilled in the art that the present invention can also be applied to moving picture shooting.

As described above, the mobile terminal according to the first embodiment of the present invention provides the voice service related to the preview image in real time in the image photographing mode, so that the visually impaired can easily photograph the desired subject do.

13 is a flowchart illustrating an operation of the mobile terminal according to the second embodiment of the present invention.

Referring to FIG. 13, the controller 180 sets a voice service function for the visually impaired according to a user command (S1305). At this time, the talk back function of the Android, the voice over function of the iOS, and the like can be used as the voice service function to be set. In addition, a voice service function developed by the manufacturer of the mobile terminal may be used.

In setting the voice service function, the control unit 180 can activate the voice recognition unit 182 to enter the voice recognition mode. At this time, the mobile terminal 100 can not only output a voice service for a visually impaired person but also receive a voice command for controlling the mobile terminal.

After the voice service function is set, the controller 180 displays a home screen or menu screen including a plurality of icons on the display 151 according to a user command. At this time, the plurality of icons include icons corresponding to at least one application.

In a state where such a home or menu screen is displayed, the controller 180 executes a gallery application (hereinafter referred to as a 'gallery app') based on the user's touch input or a voice command (S1310).

For example, in one embodiment, when the gallery icon is touched according to a voice guidance on the icons of the home or menu displayed on the menu screen, the control unit 180 executes the gallery app and notifies the user that the gallery icon has been selected and / Lt; RTI ID = 0.0 > a < / RTI >

In another embodiment, the controller 180 may control the display of the gallery application (e.g., " Gallery "or" Gallery App Launch ") based on the user's voice command And to provide a voice service to notify the user that the gallery icon has been selected and / or executed.

When the gallery application is executed through this process, the control unit 180 displays a gallery screen including at least one folder on the display unit. If a desired folder is selected according to the voice guidance for the folders in the state that the gallery screen is displayed, the control unit 180 displays an image list screen including a plurality of thumbnail images stored in the folder on the display unit 151 (S1315).

If a desired thumbnail image is selected according to a voice guidance for a plurality of thumbnail images in the state that the image list screen is displayed (S1320), the control unit 180 displays the image corresponding to the selected thumbnail image on the display unit 151 (S1325). At the same time, the control unit 180 may provide a voice service for notifying the user that the selected thumbnail image is displayed. At this time, the image may be any image as well as a captured image.

Thereafter, the control unit 180 performs shape recognition on the objects of the image displayed on the display unit 151, and provides a voice service on the image based on the recognition. However, if the image displayed on the screen is the previously captured image, and the information about the voice service linked to the image is stored in the memory, the control unit 180 may skip the shape recognition on the image.

More specifically, the control unit 180 guides information (i.e., name) about the objects included in the image in order according to a predetermined rule so that the user can easily guess the arrangement and combination of the objects do. In this case, the order of guiding the objects is the order in which the objects of the image are stacked from the downward direction to the upward direction, the order in which the objects are arranged from the left to the right direction of the image, An order according to the size of the objects, an order according to the distance between the object and the photographer, and the like.

In addition, the control unit 180 allows the user to easily estimate the position and shape of the objects by making differences in the height, strength, and tone of the voice guiding the objects.

On the other hand, when earphone or headphone is mounted on the mobile terminal 100, stereo discrimination of voice guidance is possible. Accordingly, the control unit 180 can recognize the position of the object, provide the voice guidance through the right earphone for the object located in the right area of the image, and provide the voice guidance through the left earphone for the object located in the left area have.

In addition, if the number of objects existing in the image is large and the type and location thereof are not well memorized after the voice service is provided, the user may touch the screen on which the image is displayed, have.

That is, when the touch input is received through the screen on which the image is displayed (S1335), the control unit 180 guides the information about the object located at the touched point by voice (S1340). At the same time, the controller 180 provides the vibration effect of the object shape at the position where the object is located, thereby allowing the user to easily guess the position and shape of the object. Accordingly, the user can easily identify the contents of the image displayed on the display unit 151 through voice guidance.

After that, when the gallery app is terminated (S1345), the control unit 180 ends the process according to the present invention. On the other hand, if the gallery application is not terminated and the previous menu is selected (S1350), the control unit 180 moves to step 1315 again to display the image list screen, and then repeats the above-described processes.

FIGS. 14 to 16 are diagrams for explaining the operation of a mobile terminal that performs image viewing based on voice guidance according to a second embodiment of the present invention.

14 is a view for explaining the operation of the mobile terminal executing the gallery application according to the touch input of the user.

Referring to FIG. 14, after the voice service function is set, the mobile terminal 100 displays a menu screen 1410 including a plurality of icons on the display 151 according to a user command. At this time, the plurality of icons include icons corresponding to at least one application and icons for execution of a specific function.

When the gallery icon 1420 is touched according to the voice guidance for the application icons displayed on the menu screen 1410, the mobile terminal 100 provides the voice guidance 1430 for informing the user that the gallery icon is selected.

Then, the mobile terminal 100 displays an execution screen of the selected gallery app, that is, an image list screen 1440 including a plurality of thumbnail images, on the display unit 151. At this time, the controller 180 may provide a voice guidance (not shown) to notify the user that the gallery app has been executed.

When a specific thumbnail image 1450 is selected according to a voice guidance for a plurality of thumbnail images in the state that the image list screen 1440 is displayed, the mobile terminal 100 displays a thumbnail image 1450 for informing that the thumbnail image 1450 is selected And provides voice guidance 1460.

FIG. 15 is a view for explaining an operation of a mobile terminal providing a voice service related to an image displayed on a screen.

Referring to FIG. 15, when a specific thumbnail image 1450 is selected, the mobile terminal 100 displays an image 1510 corresponding to the selected thumbnail image on the display unit 151. At this time, the mobile terminal 100 may provide a voice service 1520 to inform the user that the selected thumbnail image is displayed.

The mobile terminal 100 performs shape recognition on the objects of the image 1510 displayed on the display unit 151 and provides a voice service 1530 related to the images based on the recognition. For example, the mobile terminal 100 provides voice guidance in the order of the table 1511, the dish 1513, the food 1515, and the coffee 1517 according to the order in which the objects included in the image 1510 are stacked do. Accordingly, the user can easily guess the arrangement and combination of the objects included in the image 1510.

In addition, the mobile terminal 100 allows the user to easily guess the positions and shapes of the objects by making differences in the height, strength, and tone of the voice guiding the objects.

For example, the 'table (811)' shown in the image is a 'relatively large and close subject' so that the voice guidance is provided with 'bass loud' so that the user can infer the position and shape of the table do. Also, since the 'dish (813)' shown in the image is a 'relatively round shape and close to the subject', the voice guidance is provided as 'soft and loud', thereby allowing the user to infer the position and shape of the dish .

In addition, since the 'food (815)' displayed in the image is a 'relatively normal shape and a nearby object', the voice guidance is provided as a 'loud sound of the middle', thereby allowing the user to infer the position and shape of the food . In addition, since the 'coffee (817)' displayed in the image is 'relatively normal shape and the subject is located at a remote location', voice guidance is provided as 'small sound of medium sound', thereby allowing the user to infer the position and shape of the coffee .

16 is a view for explaining an operation of a mobile terminal providing voice service on an object when touching an image.

As shown in FIG. 15, if the number of objects existing in the image is large and the type and location thereof are not well memorized after the voice service regarding the image is provided, the user touches the screen on which the corresponding image is displayed And then the voice service about the object may be provided while moving slowly.

16, when a touch input is received through the point at which the food 1515 is displayed in the image 1510, the mobile terminal 100 displays information about the food 1515 located at the touched point Voice 1540 is displayed. At the same time, the mobile terminal 100 provides a food-like vibration effect at the point where the food is placed, thereby allowing the user to easily guess the position and shape of the object.

As described above, when the voice service related to the image 1510 displayed on the display unit 151 is provided in real time, the user can easily identify the contents of the image displayed on the display unit 151.

As described above, the mobile terminal according to the second embodiment of the present invention provides the voice service related to the image displayed on the screen in real time in the image view mode, thereby allowing the visually impaired to easily So that they can recognize it.

Meanwhile, the present invention can be implemented as a code readable by a processor in a processor-readable recording medium provided in a mobile terminal. The processor-readable recording medium includes all kinds of recording apparatuses in which data that can be read by the processor is stored. Examples of the recording medium readable by the processor include a ROM, a RAM, a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, and the like, and also a carrier wave such as transmission over the Internet. In addition, the processor readable recording medium may be distributed over networked computer systems so that code readable by the processor in a distributed manner can be stored and executed.

While the present invention has been particularly shown and described with reference to exemplary embodiments thereof, it is to be understood that the invention is not limited to the disclosed exemplary embodiments, but, on the contrary, It will be understood by those skilled in the art that various changes and modifications may be made without departing from the spirit and scope of the present invention.

100: mobile terminal 110: wireless communication unit
120: A / V input unit 130: user input unit
140: sensing unit 150: output unit
151: display unit 160: memory
170: interface unit 180: control unit

Claims (6)

Displaying a preview image input through a camera on a display unit;
Performing shape recognition on the objects of the preview image;
Providing a voice service for the preview image; And
And photographing the preview image corresponding to a user's photographing command. The method according to claim 1,
Receiving a voice signal relating to a subject of the preview image;
Providing a voice service related to the position of the subject corresponding to the voice signal; And
Further comprising the step of providing an audio guide so that the subject is focused on the camera in accordance with the movement of the camera in the photographing direction. The method according to claim 1,
And providing a voice service relating to storage or sharing of the photographed image. The method according to claim 1,
Wherein the information about the objects included in the preview image is voiced in order according to a predetermined rule. 5. The method of claim 4,
A sequence in which the subjects are stacked in the upward direction below the preview image, a sequence in which the subjects are arranged in the left-to-right direction of the preview image, an order in which the subjects are arranged in the boundary direction in the center of the preview image, Wherein the order of the images is one of a sequence according to a size of the subject and a distance according to a distance between the subjects and the photographer. The method according to claim 1,
And outputting at least one of a height, an intensity, and a tone of a voice guiding the subject according to a position and a shape of the subjects included in the preview image.

Images