KR20200049744A - Portable apparatus and a screen control method thereof - Google Patents

Abstract

Provided are a portable device and a screen control method thereof, and more particularly, to a portable device that changes a screen of the portable device according to a result of recognizing characters in text selected to correspond to a continuous movement of a touch received in a text display area, and a screen control method thereof. Some embodiments of the present invention provide a portable device that changes a screen of the portable device according to control information corresponding to the result of recognizing characters in the text selected by the continuous movement of the touch received from text in the text display area, and a screen control method thereof. The screen control method of a portable device comprises the steps of: sequentially displaying a plurality of text display areas; detecting a touch in one text display area of the plurality of text display areas; detecting a continuous movement of the touch; obtaining character-recognized text in an area where the continuous movement of the touch is detected; obtaining a control command for changing the setting of the portable device corresponding to the obtained text; and changing the setting of the portable device based on the obtained control command.

Description

PORTABLE APPARATUS AND A SCREEN CONTROL METHOD THEREOF}

The following embodiments relate to a mobile device and a method for controlling a screen of a mobile device, and specifically, a mobile device and a mobile device that changes a screen in response to a continuous movement of a touch detected in a speech bubble displayed on the mobile device. It relates to a screen control method of the device.

With the development of technology, a mobile device can provide various services and functions through multitasking or various applications as well as communication and data transmission.

The portable device receives a user input by clicking a button (or key), touching through a touch panel (or touching gesture), voice through a microphone, and motion through a sensor. The user input may include one time or multiple times in succession (or by a combination of the user inputs described above). The portable device may display a screen (eg, a setting screen) corresponding to the received user input, or execute an application (eg, a camera application, etc.).

A method of controlling a screen of a portable device according to an embodiment of the present invention includes detecting a touch on a speech balloon containing text recognizable text on an application screen displayed on the portable device, and detecting continuous movement of the touch. , Recognizing the text in response to the continuous movement of the touch, and changing the application screen to another screen according to a control command corresponding to the text recognized text, wherein the text is the Includes all of the text in the speech bubble and some of the text.

According to an aspect of the present invention, the speech balloon may include at least one of a user's speech balloon and an opponent's speech balloon.

According to an aspect of the present invention, the continuous movement of the touch includes a single touch gesture and a multi-touch gesture, and may include flick, swipe, drag, and rotation.

According to another embodiment of the present invention, a screen control method for a mobile device includes detecting a touch in text that can be recognized in a text in a speech balloon on an application screen displayed on the mobile device, and displaying a cursor in the text in response to the touch. Step of detecting a continuous movement of the touch, Recognizing a part of the selected text from the cursor in response to the continuous movement of the touch, and a control command corresponding to the text recognized text And changing the application screen to another screen according to the reception of.

A portable device according to an embodiment of the present invention includes a communication unit that connects to the outside, a touch screen that displays an application screen including a speech balloon, and a processor that controls the communication unit and the touch screen. The text selected according to the continuous movement of the touch detected in the speech balloon containing recognizable text is transmitted to the outside through the communication unit, and the control information corresponding to the text recognition result of the text received through the communication unit Accordingly, it is controlled to change the application screen to another screen.

According to the present invention, a screen control method for a mobile device and a mobile device for changing to a different screen according to a text recognition result of a selected text corresponding to a continuous movement of a touch received from text in a speech balloon displayed on a chat application screen is provided. Can be provided.

A screen control method for a mobile device and a mobile device that changes to another screen through control information provided according to a text recognition result of text selected by a continuous movement of a touch received from text in a speech balloon displayed on a chat application screen Can be provided.

According to various embodiments of the present disclosure, the present invention is not limited to this, and the mobile device is changed to various different screens according to the result of character recognition of the selected text in response to the continuous movement of the touch received from the text in the speech balloon displayed on the chat application screen A method of controlling a screen of a device and a portable device may be provided.

1 is a schematic diagram showing an operation between a portable device and a server according to an embodiment of the present invention.
2 is a schematic block diagram showing a portable device and a server according to an embodiment of the present invention.
3 is a schematic flowchart illustrating a screen control method of a portable device according to an embodiment of the present invention.
4A to 4E are views illustrating an example of a method for controlling a screen of a portable device according to an embodiment of the present invention.
5A to 5F are diagrams illustrating an example of a method for controlling a screen of a portable device according to another embodiment of the present invention.
6A and 6B are views illustrating an example of a screen control method of a portable device according to another embodiment of the present invention.

Hereinafter, exemplary embodiments according to the present invention will be described in detail with reference to the contents described in the accompanying drawings. Also, a method of manufacturing and using the present invention will be described in detail with reference to the contents described in the accompanying drawings. The same reference numerals or symbols in each drawing denote parts or components that perform substantially the same function.

Terms including an ordinal number such as 'first', 'second', etc. may be used to describe various components, and the above-described components are not limited by the above-described terms. The above-mentioned terms may be used for the purpose of distinguishing one component from another component. For example, in the scope of the present invention, the first component may be referred to as the second component. Also, the second component may be referred to as a first component. The term 'and / or' includes a combination of a plurality of listed items or any one of a plurality of listed items.

An application according to an embodiment of the present invention refers to software used by a user running on an operating system (OS) for a computer (or portable device) or a mobile OS. For example, the application is a web browser, a camera application, a mobile payment application (or electronic payment application, payment application), a photo album application, a word processor, a spread Spread sheet, contacts application, calendar application, memo application, alarm application, SNS (Social Network System) application, call application, game store , A game application, a chat application, a map application, a music player or a video player.

The application according to an embodiment of the present invention may refer to software executed on a portable device or an external device (eg, a wearable device, a server, etc.) connected wirelessly or wiredly to the portable device. In addition, the application according to an embodiment of the present invention may mean software executed in a portable device in response to a received user input.

The content may be played or displayed through an executed application. For example, a video file or audio file played in a video player, an application, a game file played in a game application, a music file played in a music player, a photo file displayed in a photo album application, or a web displayed in a web browser It may include a page file, or payment information displayed in an electronic payment application (eg, mobile card number, payment price, product name, service name, or store name, etc.). In addition, the content includes a call screen (eg, caller ID, caller name, call start time, or caller video (or caller image) by video call) displayed in the call application. can do.

The content may include a running application screen and a user interface constituting the application screen. In addition, the content may include a single content or a plurality of content.

A widget means a mini-application, which is one of graphic user interfaces (GUIs) that more smoothly supports interaction between a user and an application or an OS. Examples include weather widgets, calculator widgets, and clock widgets.

In the embodiment of the present invention, 'bubble speech' may mean an area in which text input by a user and / or a counterpart through an executed application is displayed. The 'speech balloon' may be referred to as a 'text display area'.

In an embodiment of the present invention, 'user input' may be, for example, a user's button (or key) selection, a user's button (or key) pressed (or clicked), or a user's soft button (or soft key) touch , A touch that is received (or detected) by the user on the touch screen (or includes non-contact such as hovering), a touch gesture that is received (or detected) by the user on the touch screen (or includes non-contact such as hovering, user It can be used as a term including the voice of the user, the appearance of the user (for example, the user appears within the camera recognition range), or the motion of the user. It may be used as a term for pressing (or clicking) a key or touching a soft button (or a soft key).

Terms used in the present specification are used to describe examples, and are not intended to limit and / or limit the present invention. Singular expressions include plural expressions unless the context clearly indicates otherwise. In this specification, the term 'include' or 'have' is intended to indicate that there is a feature, number, step, operation, component, part, or a combination thereof described in the specification, or one or more other features or It should be understood that the presence or addition possibilities of numbers, steps, actions, components, parts or combinations thereof are not excluded in advance. The same reference numerals in each drawing denote members that perform substantially the same function.

1 is a schematic diagram showing an operation between a portable device and a server according to an embodiment of the present invention.

Referring to FIG. 1, the portable device 100 and one or more servers are displayed.

A touch screen 190 is positioned in a central area of the front surface 100a of the portable device 100. The touch screen 190 includes a flat touch screen, a single edge touch screen (not shown) extending from one side of the flat touch screen to a curved surface, and both sides extending from each side of the flat touch screen to a curved surface. It may include a dual edge touch screen. In addition, the touch screen 190 can be bent or bent through a thin and flexible printed circuit board (not shown).

In FIG. 1, a screen 191 of a chat application executed by user input is displayed on the touch screen 190. The chat application screen 191 may include a title area 191a, a display area 191b, and an input area 191c. When the input window 191c1 in the input area 191c is selected, a keypad 166 (see FIG. 2) may be displayed on the touch screen 190.

In the chat application, the text input by the caller may be displayed in a speech bubble, or text display area, 191b1, 191b11 in the display area 191b. Text can include words, sentences, or paragraphs. The above-described text may include consonants and vowels in Korean, alphabets in English, Japanese, Chinese, French, German, and other languages available in chat applications. The speech balloon can be referred to as a text display area.

The text entered by the user through the text input window 191c1 may be displayed on the speech balloon 191b2 of the display area 191b. The user may touch (or select) some of the text displayed on the speech balloons 191b1, 191b11, and 191b2, or input a continuous movement (or touch gesture) after the touch. The user can control the screen of the mobile device 100 through the touch (or continuous movement of the touch) of the text in the speech balloon (for example, re-booting, displaying a setting screen, running an application, etc.). have.

A first camera 151, one or more speakers 163, an illuminance sensor 171, and a proximity sensor 172 (see FIG. 2) may be positioned on the front surface 100a of the portable device 100. A second camera 152 (see FIG. 2) and a flash (flash, 153, see FIG. 2) may be located on a rear surface (not shown) of the portable device 100.

A home button 161a, a recently-launched app button 161b, and a return button 161c are positioned below the front surface 100a of the mobile device 100. The button 161 is not only a physical button (for example, a home button 161a, a power / lock button 161d, a volume button 161e, etc.), but also a touch button (for example, a home button 161a), recently executed It can be implemented as an app button (161b), a back button (161c). Also, the button 161 may be displayed along with text or other icons in the touch screen 190.

The power / lock button 161d and the volume button 161e may be located on the side of the portable device 100. The side surface of the portable device 100 connects the front surface 100a and the rear surface of the portable device 100, and may include a left side, a right side, a top, and a bottom.

One or a plurality of microphones 162 (see FIG. 2) may be positioned on the top, side, and / or bottom of the portable device 100. The connector 165 and / or the speaker 163 may be located on the top, side, and / or bottom of the portable device 100. Also, an insertion port (not shown) in which an input pen 167 (see FIG. 2) having a button (not shown) can be inserted may be located at the top or bottom of the portable device 100. The input pen 167 (see FIG. 2) may be stored inside the portable device 100 through an insertion port (not shown), and may be withdrawn from the portable device 100 for use.

The text recognition server 200 generates text data (for example, text, or code, etc.) generated through character recognition of an electrical signal (or a packet corresponding to an electrical signal) corresponding to the text selected in the portable device 100. ). The converted text data may be transmitted to the control server 300 via the mobile device 100 or directly to the control server 300.

The control information server 300 may convert the converted text data into control information recognizable by the mobile device 100 (eg, control commands for controlling the mobile device 100). The control information may be transmitted from the control information server 300 to the portable device 100. The detailed description of the character recognition server 200 and the control information server 300 will be described later.

2 is a schematic block diagram showing a portable device and a server according to an embodiment of the present invention.

Referring to FIGS. 1 and 2, the portable device 100 uses another device (eg, another portable device, or) using at least one of the mobile communication unit 120, the sub communication unit 130, and the connector 165. Server, etc.).

The portable device 100 may transmit data to or from another device through the communication unit 120 or 130 with the touch screen 190. The portable device 100 may have an input pen 167 and a touch screen 190 and transmit data to or from the outside through the communication unit 120 or 130.

The portable device 100 includes a control unit 110, a mobile communication unit 120, a sub communication unit 130, a multimedia unit 140, a camera 150, a positioning information receiver (155), and an input / output unit ( 160), a sensor unit 170, a storage unit 175 and a power supply unit 180. Also, the portable device 100 includes a touch screen 190 and a touch screen controller 195.

The control unit 110 may include a processor (processor, 111). In addition, the control unit 110 stores a signal or data input from the outside of the portable device 100 and a ROM (ROM, 112) in which a control program for controlling the portable device 100 is stored, or the portable device 100. A RAM (113) used as a storage area for various tasks performed may be further included.

The controller 110 controls the overall operation of the portable device 100 and the signal flow between the internal components 110 to 195 of the portable device 100 and processes data. The control unit 110 controls the supply of power to the internal components 120 to 195 using the power supply unit 180.

The processor 111 may include a graphic processing unit (GPU) (not shown) for graphic processing. In addition, the processor 111 may further include a sensor processor (not shown) for controlling the sensor or a communication processor (not shown) for controlling the communication.

The processor 111 may be implemented in the form of a system on chip (SoC) including a core (not shown) and a GPU. In addition, the processor 111 may include a single core, a dual core, a triple core, a quad core, and multiple cores thereof.

The processor 111, the ROM 112, and the RAM 113 may be interconnected by a bus.

The control unit 110 includes a mobile communication unit 120, a sub communication unit 130, a multimedia unit 140, a camera 150, a location information receiving unit 155, an input / output unit 160, a sensor unit 170, and storage The control unit 175, the power supply unit 180, the touch screen 190, and the touch screen controller 195 may be controlled.

In the exemplary embodiment of the present invention, the term “control unit of a portable device” may mean a processor 111, a ROM 112, and a RAM 113. Also, the term “control unit of the portable device” may mean the processor 111.

The mobile communication unit 120 may connect to other devices through a mobile communication network using one or more antennas under the control of the control unit 110.

The sub-communication unit 130 may connect to another device through the wireless LAN communication unit 131 and / or the short-range communication unit 132 using one or more antennas under the control of the control unit 110. The sub-communication unit 130 may include a wireless LAN antenna, an antenna for magnetic secure transmission (MST) for electronic payment, or an antenna for near field communication (NFC) communication.

The wireless LAN communication unit 131 may be wirelessly connected to the AP in a place where an access point (AP) is installed under the control of the controller 110. The wireless LAN communication unit 131 may support, for example, Wi-Fi communication.

Near field communication includes Bluetooth communication, Bluetooth low energy communication, infrared data association (IrDA), ultra-wideband communication (UWB), magnetic security transmission (MST) communication, and / or NFC communication. It may include.

The mobile device 100 may be one of the mobile communication unit 120, the wireless LAN communication unit 131, and the short-range communication unit 132, or the mobile communication unit 120, the wireless LAN communication unit 131, depending on the function and / or performance. It may include a combination of short-range communication unit 132. In various embodiments of the present invention, the term “communication unit” may include the mobile communication unit 120 and / or the sub communication unit 130.

The multimedia unit 140 may receive audio reproduction, video reproduction, and / or external broadcasting under the control of the controller 110.

The audio reproducing unit 141 is an audio source previously stored in the storage unit 175 of the portable device 100 or received from the outside under the control of the control unit 110 (for example, a file extension of mp3, wma, ogg Alternatively, audio files including wav) may be played using an audio codec.

According to various embodiments of the present disclosure, the audio playback unit 141 may play auditory feedback corresponding to another screen change according to the continuous movement of the touch received from the text in the speech balloon. For example, the audio reproducing unit 141 may receive auditory feedback (eg, a storage unit (eg, a storage unit) corresponding to another screen change according to the continuous movement of the touch received from the text in the pressure speech balloon under the control of the control unit 110. 175) can be played through the audio codec.

The video playback unit 142 is a digital video source previously stored in the storage unit 175 of the portable device 100 or received from the outside under the control of the control unit 110 (for example, the file extension is mpeg, mpg, video files including mp4, avi, mov, or mkv) can be played using a video codec.

According to various embodiments of the present invention, the video reproducing unit 142 may reproduce visual feedback corresponding to another screen change according to the continuous movement of the touch received from the text in the speech balloon. For example, visual feedback (eg, output of a video source stored in the storage unit 175) may be reproduced through a video codec under the control of the controller 110.

The broadcast receiving unit 143 is a broadcast signal (eg, a TV broadcast signal, a radio broadcast signal or a data broadcast signal) transmitted from an external broadcast station through an antenna (not shown) under the control of the control unit 110, and a broadcast addition Information (eg, an electronic program guide (EPG) or an electronic service guide (ESG)) may be received.

The multimedia unit 140 may include an audio playback unit 141 and a video playback unit 142 except for the broadcast reception unit 143 in response to the performance or structure of the portable device 100. In addition, the control unit 110 may be implemented to include at least one of the audio playback unit 141 and the video playback unit 142 of the multimedia unit 140.

The camera 150 may take a still image or a video under the control of the controller 110. The camera 150 includes a first camera 151 positioned on the front side and a second camera 152 positioned on the rear side. The first camera 151 or the second camera 152 may include an auxiliary light source (eg, a flash 153) that provides the amount of light required for shooting.

The camera 150 is adjacent to the first camera 151 and the first camera 151 located on the front of the portable device 100 (for example, a distance between two optical axes is greater than 5 mm and less than 80 mm). One additional camera (eg, a third camera (not shown)) may be further included. Also, the camera 150 may further include a form in which the first camera 151 and the third camera are implemented as one unit. The controller 110 may take a 3D still image or a 3D video using the first camera 151 and the third camera.

The camera 150 is adjacent to the second camera 152 and the second camera 152 located at the rear of the portable device 100 (for example, the distance between the optical axes is greater than 5 mm and less than 80 mm). An additional camera (eg, a fourth camera (not shown)) may be further included. In addition, the camera 150 may further include a form in which the second camera (not shown) and the fourth camera are implemented as one unit. The controller 110 may take a 3D still image or a 3D video using the second camera 152 and the fourth camera. The camera 150 may perform wide-angle shooting, telephoto shooting, or close-up shooting using an additional lens (not shown) detachably attached to a separate adapter (not shown).

The location information receiving unit 155 periodically receives signals (for example, orbit information of a satellite, time information of a satellite, and navigation messages) from a plurality of satellites (not shown) in orbit of the earth.

The portable device 100 may calculate the position of each satellite and the portable device 100 using signals received from a plurality of satellites, and may calculate a distance using a transmission / reception time difference. Through the triangulation, the position, time, or movement speed of the portable device 100 can be calculated. Additional satellites may be required for orbit correction or time correction.

In the case of indoors, the portable device 100 may detect the position or movement speed of the portable device 100 using a wireless AP. The detection of the location of the portable device 100 indoors may use a cell-ID method, an enhanced cell-ID method, or an angle of arrival (AoA) method. In addition, the portable device 100 may detect the position or movement speed of the portable device 100 located indoors using a wireless beacon (not shown).

The input / output unit 160 includes one or more buttons 161, one or more microphones 162, one or more speakers 163, one or more vibration motors 164, a connector 165, It may include at least one of the keypad 166 and the input pen 167.

1 and 2, the button 161 may receive a user input. The button 161 may include a home button 161a, a recently-launched app button 161b, and / or a return button 161c located on the lower front of the mobile device 100. The button 161 may include one or more volume buttons 161d and a power / lock button 161e positioned on the side of the portable device 100. Further, the button 161 may be implemented only with the home button 161a, the volume button 161d, and the power / lock button 161e.

Some of the buttons 161 (for example, 161a, 161b, 161c, and other buttons not shown) may be implemented as touch buttons as well as physical buttons. Also, some of the buttons 161a, 161b, and 161c may be displayed on the touch screen 190 in the form of text, images, or icons.

The control unit 110 may receive an electrical signal transmitted from the button 161 in response to reception of a user input. The controller 110 may detect a user input using a received signal (eg, a button 161 is pressed).

The shape, location, function, name, etc. of the button 161 shown in FIGS. 1 and 2 are examples for explanation, and it is not limited thereto, and it is possible to change, modify or modify the common knowledge in the relevant technical field. It can be easily understood by those who have it.

The microphone 162 generates a voice or sound received from the outside under the control of the controller 110 as an electrical signal. The electrical signal generated by the microphone 162 may be converted in an audio codec under the control of the control unit 110 and stored in the storage unit 175 or output through the speaker 163.

Referring to FIGS. 1 and 2, the microphone 162 may be located on one or more of the front, side, and / or rear of the portable device 100.

The speaker 163 outputs sound corresponding to various signals (for example, a wireless signal, a broadcast signal, an audio source, a video file or a picture taking) decoded by an audio codec under the control of the controller 110.

1 and 2, one or a plurality of speakers 163 may be located on the front, side, and / or rear of the portable device 100. Also, a plurality of speakers (not shown) may be located on the side of the portable device 100.

According to an embodiment of the present invention, the speaker 163 may output auditory feedback corresponding to another screen change according to the continuous movement of the touch received from the text in the speech balloon. For example, the speaker 163 may output auditory feedback corresponding to another screen change according to the continuous movement of the touch received from the text in the speech balloon under the control of the controller 110.

The vibration motor 164 converts an electrical signal into mechanical vibration under the control of the control unit 110. The vibration motor 164 may include a linear vibration motor, a bar type vibration motor, a coin type vibration motor, or a piezoelectric element vibration motor. Can be.

One or more vibration motors 164 may be located inside the portable device 100. The vibration motor 164 may vibrate the entire portable device 100 or may locally vibrate only a part.

According to an embodiment of the present invention, the vibration motor 164 may output tactile feedback corresponding to another screen change according to the continuous movement of the touch received from the text in the speech balloon. In addition, the vibration motor 164 may provide various tactile feedback (eg, intensity of vibration and duration of vibration) previously stored or received from the outside based on a control command of the controller 110.

The connector 165 may be used as an interface for connecting the portable device 100 and an external device (not shown) or a power source (not shown). For example, the connector 165 may include a micro USB type connector or a USB-C type connector.

The keypad 166 receives user input for control of the portable device 100. The keypad 166 has a button (or key), a virtual keypad (not shown) displayed in the touch screen 190, and a pressure touch keypad (not shown) corresponding to pressure (or force) applied to the user input. And a physical keypad (not shown) located in the lower area of the front surface of the portable device 100.

The keypad 166 may use a short-range communication, or may further include a separate keypad (or keyboard, not shown) connectable by wire.

The input pen 167 can be drawn in / out from one of the side and rear surfaces of the mobile device 100, and the handwriting / drawing displayed on the touch screen 190 or the touch screen 190 of the mobile device 100 by the user The objects (eg, menus, text, images, videos, shapes, icons, and shortcuts) that make up (or display) the application screen (eg, memo screen, note pad screen, calendar screen, etc.) ) And / or content (e.g. text file, image file, audio file, video file, payment information or web page), select (or touch), handwriting, or drawing (drawing, painting or sketching) can do.

The sensor unit 170 may detect the state of the portable device 100 and / or the state of the portable device 100 (eg, tilt (or angle), shaking, etc.) through one or more sensors.

The sensor unit 170 is a proximity sensor (171) that detects whether the user has access to the portable device (100), an illuminance sensor (172) that detects the amount of light around the portable device (100) Or, it may include a sensor that scans a user's fingerprint (finger print sensor 173).

At least one of the proximity sensor 171 and the illuminance sensor 172 may be located on the front surface of the portable device 100.

Fingerprint sensor 173 is a button (for example, a home button, etc.) located on the front of the portable device 100, a button (for example, a power button, etc.) located on the side of the portable device 100, or portable It may be located on the back of the device 100. In addition, the fingerprint sensor is located in a portion of the touch screen 190 of the mobile device 100 (eg, an area adjacent to the home button 161a, inside the touch screen 190, not shown) and the user You can also scan your fingerprint.

The sensor unit 170 is an acceleration sensor (acceleration sensor, not shown). It may further include a geomagnetic sensor (not shown) or a gyro sensor (not shown), a gravity sensor (not shown), or an altimeter (not shown). Further, the sensor unit 170 may further include a biosignal sensor (not shown) including a heart rate sensor (not shown).

The sensor included in the sensor unit 170 detects a user state (eg, body temperature, heart rate, brain wave, etc.) and the state of the portable device 100, and generates an electrical signal corresponding to the detection to the control unit 110 send. The sensor included in the sensor unit 170 may be easily added to, modified, or deleted according to the performance of the portable device 100 to those skilled in the art.

The storage unit 175 may store signals or data input / output corresponding to the operations of the components 110 to 195 under the control of the control unit 110. The storage unit 175 is a control program for controlling the portable device 100 or the control unit 110, a graphical user interface (GUI) related to an application provided by a manufacturer or downloaded from the outside, images corresponding to the GUI, and user information , Documents, databases or related data.

The storage unit 175 is output in response to another screen change according to the continuous movement of the touch received from the text in the speech bubble, and visual feedback (for example, a video source, etc.) recognized by the user is output from the speaker 163. The output may store auditory feedback (eg, a sound source, etc.) recognized by the user and a tactile feedback (eg, a haptic pattern, etc.) output by the vibration motor 164 and recognized by the user.

The storage unit 175 may store a feedback providing time (for example, 500 ms) of feedback provided to the user.

In the embodiment of the present invention, the term 'storage unit' refers to a memory card (eg, micro) mounted in the storage unit 175, the controller 110, the ROM 112, the RAM 113, or the portable device 100. SD card, etc., not shown). The storage unit 175 may include a non-volatile memory, a volatile memory, a hard disk drive (HDD), or a solid state drive (SSD).

The power supply unit 180 may supply power to the components 110 to 195 of the portable device 100 under the control of the control unit 110. The power supply unit 180 receives power input from an external power source (not shown) through a wired cable (not shown) connected to the connector 165 under the control of the controller 110, respectively. Components can be supplied. Also, the power supply unit 180 may charge one or more batteries 185 under the control of the control unit 110.

The power supply unit 180 may supply charged power of the battery 185 to an accessory device (for example, a watch or the like, not shown) through a cable. In addition, the power supply unit 180 uses a separate transmission coil (not shown) that is functionally connected to the battery 185 under the control of the control unit 110 to control other portable devices or accessory devices (not shown). It can be charged wirelessly. The wireless charging method may include, for example, a magnetic resonance method, an electromagnetic wave method, or a magnetic induction method.

The touch screen 190 is a GUI (graphical user interface) corresponding to various services (for example, voice calls, video calls, data transmission, broadcast reception, photo shooting, video viewing, or electronic payment including mobile payment). ) To the user. The touch screen 190 includes a touch panel (not shown) for receiving user input (eg, a touch, etc.) and a display panel (not shown) for screen display. The touch panel may include a capacitive touch panel, a resistive touch panel, or an ultrasonic touch panel.

The touch screen 190 may transmit analog signals corresponding to a single touch (or multi-touch) or single touch gesture (or multi-touch gesture) input through an application screen or a GUI to be executed, respectively, to the touch screen controller 195. . The touch screen 190 may receive a single touch or multi-touch through a user's body (eg, a finger including a thumb) or an input pen 167.

The touch screen 190 according to an embodiment of the present invention may output visual feedback corresponding to another screen change according to the continuous movement of the touch received from the text in the speech balloon. In an embodiment of the present invention, the display unit may include a touch screen 190.

When the user strongly touches the touch screen 190 (eg, exceeds a threshold), a pressure touch pad (force touch) located under the touch screen 190 and detecting pressure applied to the touch screen 190 A pad, or a pressure sensor, may be located on a pressure-detectable electrode (or electrode layer, including a force touch pad, not shown). Pressure (or touch) exceeding a threshold (eg, 50% of the capacitance or 50% of the voltage (or current) changed by the touch input to the touch screen 190) is received at the pressure touch pad In this case, the control unit 110 may detect a received pressure touch (or force touch).

The above-described threshold, electrostatic capacity and / or voltage (or current) is one embodiment, and changes according to the structure and / or function of the portable device 100 (eg, threshold change, detectable electrostatic capacity change, etc.) It can be easily understood by those skilled in the art.

The touch screen controller 195 converts an analog signal corresponding to a single touch (or multi-touch) or single touch gesture (or multi-touch gesture) received from the touch screen 190 into a digital signal and transmits it to the controller 110. . The controller 110 may calculate a touch position (for example, X and Y coordinates) of a touch received on the touch screen 190 using a digital signal received from the touch screen controller 195.

The control unit 110 may control the touch screen 190 using a digital signal received from the touch screen controller 195. For example, the control unit 110 may display a shortcut icon (or, also referred to as an icon) displayed on the touch screen 190 differently from other unselected shortcut icons or correspond to the selected shortcut icon in response to the input touch. The application (eg, a game application) may be executed to display a game application screen on the touch screen 190.

The character recognition server 200 receives a signal (or packet) corresponding to the continuous movement of the touch from the portable device 100 through a communication unit (not shown). The control unit (not shown) of the character recognition server 200 may analyze the received packet using a character recognition algorithm to perform character recognition.

The control unit of the character recognition server 200 converts the received electrical signal (or a packet corresponding to the electrical signal) into character recognition data (or text data) including phoneme, word, or sentence text using a character recognition algorithm. can do.

The control unit of the character recognition server 200 may transmit text data to the mobile device 200 and / or the control information server 300 through a communication unit (not shown).

The control unit (not shown) of the control information server 300 may determine (or select, convert, and generate) control information (eg, a control command) using text data. The control information may control a screen (or operation, function) of the portable device 100. The control unit of the control information server 300 may transmit control information to the portable device 200 through a communication unit (not shown).

In an embodiment of the present invention, the character recognition server 200 may be implemented as an integral type 100-1 with the portable device 100. The character recognition server 200 may be included in the portable device 100 (eg, an integrated portable device 100-1) as a component separate from the components 110 to 195 of the portable device 100. For example, the character recognition server 200 may be embedded in the storage unit 175 of the portable device 100 or may be implemented in a separate storage unit (not shown).

In another embodiment of the present invention, the separate character recognition server 200 and the control information server 300 may be embodied as an integral character 200-1 with the character recognition server 200. The integrated character recognition server 200 may convert the recognized text data into control information. The integrated character recognition server 200 may convert received text data into control information. The integrated character recognition server 200 may transmit control information to the portable device 100 through a communication unit.

Components of the portable device 100 shown in FIGS. 1 and 2 are added (eg, a separate touch screen is added) or deleted (eg, a separate touch screen) corresponding to the performance of the portable device 100. , Components indicated by dashed lines in FIG. 2, etc.) or may be easily changed to those skilled in the art.

3 is a schematic flowchart illustrating a screen control method of a portable device according to an embodiment of the present invention.

4A to 4E are views illustrating an example of a method for controlling a screen of a portable device according to an embodiment of the present invention.

5A to 5F are diagrams illustrating an example of a method for controlling a screen of a portable device according to another embodiment of the present invention.

In step S310 of FIG. 3, a chat is performed in a chat application displayed on the screen of the mobile device.

Referring to FIG. 4A, a chat application screen 191 is displayed on the touch screen 190 of the mobile device 100. The chat application screen 191 may include a title area 191a, a display area 191b, and an input area 191c. In the title area 191a, the name of the person to chat with (or, through the nickname, title, etc.) can be displayed.

The speech balloons 191b1 to 191b2 exchanged between the other party (one of the sender and the receiver) and the user may be displayed on the display area 191b. Text (eg, consonants, vowels, words, sentences, or paragraphs) input by the other party or the user may be displayed on the speech balloon. In addition, an icon (shortened icon), a symbol, an emoticon, or an image (eg, a small size such as a thumbnail image) may be further displayed on a speech balloon.

The user may input text through the text input window 191c1 of the input area 191c. In addition, through each input window 191c2, 191c3 of the input area 191c, the user can input an icon (shortcut icon), a symbol, an emoticon, or an image (for example, a small size such as a thumbnail image). Can be. When the user selects the input window 191c1, the keypad 166 may be displayed on the touch screen 190.

The user may input text by touching (or touching gesture) the keypad 166 using a part of the body including a thumb and / or an input pen 167. The user may input text by voice through the microphone 162. In addition, the user may input text in motion through the sensor 170.

Referring to FIG. 5A, which is another embodiment of the present invention, the chat is performed in the chat application displayed on the screen of the mobile device.

Speech balloons 191b1-1 to 191b2-1 exchanged between the other party (one of the sender and the receiver) and the user may be displayed on the display area 191b-1.

Chat proceeds in the chat application displayed on the screen of the mobile device of step S310 of FIG. 3 (corresponding to FIG. 5A). Chatting proceeds in the chat application displayed on the screen of the mobile device of step S310 of FIG. 3 (FIG. 4A) Corresponds to), so duplicate description is omitted.

In step S320 of FIG. 3, a touch is received in the text in the speech balloon.

4A and 4B, the user inputs a touch (or hovering included, 410) to the speech balloon 191b11. The touch 410 may include a tap or a double tap. The user may touch each of the speech balloons 191b1-1 to 191b2-1 displayed on the chat application screen. The user can touch any speech balloon displayed on the chat application screen (for example, a user speech balloon or a speech balloon of the other party (eg, Kim's team leader)).

The control unit (or processor) may detect the touch 410 using the touch screen 190 and the touch screen controller 195. The controller (or the processor) may calculate a touch position 410a corresponding to the touch 410 (for example, X1 coordinates and Y1 coordinates) by using an electrical signal received from the touch screen controller 195. The calculated touch position 410a may be referred to as an initial touch position.

The controller 110 stores the pressure touch information corresponding to the touch position 410a, the detection time of the touch (for example, 9:35 am), the pressure value of the touch, and an electrical signal corresponding to the touch (175) ). The touch 410 can be caused by, for example, one of the fingers including the thumb, or the input pen 167.

A cursor (not shown) may be displayed at the initial touch position 410a detected in the text to be recognized. The text to be recognized may be selected in response to a continuous movement of the touch from the displayed cursor (eg, movement from the initial position 410a to the last position 410b, 410-1).

The user inputs a pressure touch (not shown) into the speech balloon 191b11. The controller 110 may detect a pressure touch using a pressure touch panel (not shown), a touch screen 190 and / or a touch screen controller 195. The pressure touch can be detected through the pressure touch panel. The pressure touch panel may include, for example, the bottom of the touch screen 190, the interior of the touch screen 190, the bottom of the display panel (not shown), or the top of the touch panel (not shown). The pressure touch panel may be distinguished from the touch screen 190 separately. Also, the pressure touch panel may be included in the touch screen 190.

The pressure of the pressure touch can be detected through the pressure touch panel. The pressure touch panel may transmit an electrical signal corresponding to the input pressure (eg, in the case of voltage, between 0 V and 3 V, depending on the portable device) to the control unit 110. The control unit 110 may calculate the pressure using the received electrical signal. Also, the control unit 110 may map the pressure corresponding to the received electrical signal.

5A and 5B, which are other embodiments of the present invention, a touch is received in a text in a speech balloon.

The controller 110 may detect the touch 420 using the touch screen 190 and the touch screen controller 195. The control unit 110 may calculate a touch position 420a (eg, X2 coordinates and Y2 coordinates) corresponding to the touch 420 using the electric signal received from the touch screen controller 195. The calculated touch position 420a may be referred to as an initial touch position.

Touch is received in the text in the speech balloon in step S320 of FIG. 3 (corresponding to FIGS. 5A and 5B). Touch is received in the text in the speech balloon in step S320 in FIG. 3 (corresponding to FIGS. 4A and 4B) ), So a duplicate description is omitted.

In step S330 of FIG. 3, a continuous movement of the detected touch is received.

Referring to FIGS. 4C and 4D, a user may use a plurality of X and Y coordinates corresponding to a continuous movement of a touch on a speech balloon 191b11 (for example, a continuous touch (from 410a to 410b)), 410-1. ). In addition, the user may maintain the touch 410 for a set time (for example, 500 µs or less, changeable) at the initial touch position 410a. After maintaining the touch 410 at the initial touch position 410a, the user may have a plurality of X and Y coordinates corresponding to a continuous movement of the touch (eg, continuous touches (410a to 410b)), 410- 1) can be entered.

The control unit 110 may detect the continuous movement 410-1 of the touch in the speech balloon 191b11 using the touch screen 190 and the touch screen controller 195. The control unit 110 uses the electrical signal received from the touch screen controller 195 to continuously move the positions of the touch in the speech balloon 191b11 (for example, a plurality of X coordinates and a plurality of Y coordinates, 410a, 410b, And coordinates between 410a and 410b).

The continuous movement of the touch in the speech balloon 191b11 (for example, the movement from the initial position 410a to the last position 410b, 410-1) is a touch gesture input to the touch screen 190. Can mean The continuous movement of the touch in the speech balloon 191b11 (for example, moving from the initial position 410a toward the boundary (or boundary) between the speech balloon and the display area, 410-1) is input to the touch screen 190 Being a touch gesture.

In an embodiment of the present invention, the continuous movement of the touch 410-1 includes a swipe, drag, flick, or rotation starting from the initial position 410a. can do. Further, the continuous movement of the touch 410-1 may include continuous movement of the multi-touch starting from the initial location 410a and other locations (not shown) as well as the continuous movement of the single touch. .

4B to 4D, all text and some text in the speech balloon 191b11 in the speech balloon may be selected in response to the continuous movement 410-1 of the touch. In response to the continuous movement of the touch 410-1, selection of the text (or text to be recognized) from the initial touch position 410a may be displayed (or started).

In response to the continuous movement of the touch 410-1, text (or text to be recognized) that is a character recognition object may be selected from the initial touch position 410a. In response to the continuous movement of the touch 410-1, the starting position of the text (or text recognition target text), which is the character recognition target of the initial touch position 410a, and text that is not the character recognition target (or not selected) It can be distinguished (for example, changing the font color, changing the font thickness, or adding a font highlight color). In response to the continuous movement of the touch 410-1, the text (or text to be recognized) selected as a character recognition target selected from the cursor displayed at the initial touch position 410a is a character recognition non-target (or not selected). It can be distinguished from text (eg, changing font color, changing font thickness, or adding font highlight color).

The text (or text to be recognized) that is the character recognition target of the initial touch position 410a is a phoneme or word including a consonant, vowel, consonant and vowel located at the initial touch position 410a. It can contain.

The initial touch location 410a may be located at 'West' of 'Seoul Metropolitan City'. The text selected in response to the initial touch position 410a may be 'standing'. The text selected by the progress of the continuous movement 410-1 of the touch starting from the initial touch position 410a may be 'Seoul Metropolitan City'. When the first touch position 410a is detected at any position of 'Seoul Metropolitan City' in the speech balloon 191b11, it is selected by the progress of the continuous movement 410-1 of the touch starting from the first touch position 410a. The text may be 'Seoul Metropolitan City'.

The text 410s selected from the initial touch position 410a to the last touch position 410b in response to the continuous movement 410-1 of the touch may be 736-32, Yeoksam 1-dong, Gangnam-gu, Seoul. The text 410s selected from the initial touch position 410a to the last touch position 410b in response to the continuous movement 410-1 of the touch may be extended to 736-32 Yeoksam 1-dong, Gangnam-gu, Seoul. . In addition, the text selected in response to the continuous movement 410-1 of the touch from the first touch position 410a to the last touch position 410b is 'Gangnam-gu, Seoul' and 'Yeoksam 1-dong 736-32' on the next line. Can be extended to The text 410s finally selected according to the continuous movement 410-1 of the touch may be 736-32, Yeoksam 1-dong, Gangnam-gu, Seoul.

In addition, the text selected in response to the continuous movement 410-1 of the touch from the initial touch position 410a to the last touch position 410b is inputted and displayed in the speech balloon 191b11 (for example, 'Address can be 736-32, Yeoksam 1-dong, Gangnam-gu, Seoul'). The user may touch the initial touch position to be located in the speech balloon 191b11 for selecting text to be recognized.

The text finally selected according to the continuous movement of the touch (410-1) (for example, 'address is 736-32, Yeoksam 1-dong, Gangnam-gu, Seoul') is the storage unit 175 under the control of the control unit 110 It can be stored (or temporarily stored).

The user may be touch-off the touch (contact with the speech balloon 191b11, or the touch screen 190) at the last position 410b of the continuous movement 410-1 of the touch. The direction of the continuous movement 410-1 of the touch may be one of upper, lower, left and right.

5C and 5D, which are other embodiments of the present invention, a continuous movement of the detected touch is received.

The user inputs a plurality of X coordinates and Y coordinates 420-1 corresponding to continuous movements of a touch (for example, continuous touches 420a to 420b) into the speech balloon 191b11-1. In addition, the user may maintain the touch 420 for a set time (for example, 500 µs or less, changeable) at the initial touch position 420a. After maintaining the touch 420 at the initial touch position 420a, the user may have a plurality of X and Y coordinates corresponding to a continuous movement of the touch (eg, continuous touches (420a to 420b)), 420- 1) can be entered.

5B to 5D, text may be selected from the initial touch position 420a in response to a continuous movement 420-1 of the touch.

* The first touch position 420a may be located in the 'set' of the 'setting'. The text selected in response to the continuous movement 420-1 of the touch from the initial touch position 420a to the last touch position 420b may be extended to 'Wi-Fi' on the setting screen.

The user may touch-off the touch (contact between the speech balloon 191b11-1, or the touch screen 190) at the last position 420b of the continuous movement 420-1 of the touch.

In response to the continuous movement 420-1 of the touch in step S330 of FIG. 3, the selection of the text to be recognized as a character (corresponding to FIGS. 5B and 5D) is the continuous movement of the touch of step S330 of FIG. Corresponding to (410-1) is substantially similar to the selection of the text to be recognized as a character (corresponding to FIGS. 4B and 4D), a duplicate description is omitted.

In step S340 of FIG. 3, character recognition of the selected text is performed.

Referring to FIG. 4D, the controller 110 may transmit (or convert and transmit) a signal corresponding to the stored text to be received by the character recognition server 200 according to a wireless communication standard. In addition, the control unit 110 may control the communication unit 130 to transmit a packet including a signal corresponding to the text that is the object to be stored. The packet may be a wireless communication standard or a packet conforming to the standard of short-range communication.

The control unit (not shown) of the character recognition server 200 performs character recognition on the received packet using a character recognition algorithm. The character recognition algorithm may recognize a phoneme (word, or sentence) by dividing the text included in the packet into phoneme units, word units, or sentence units and comparing it with a reference pattern (or standard pattern). Character recognition algorithms may include pattern matching, structure analysis, feature matching, or stroke analysis. The character recognition algorithm described above may be used selectively or separately depending on the purpose (or purpose).

The storage unit (not shown) of the character recognition server 200 may store (update) a phoneme database corresponding to a specific phoneme (a specific word or a specific sentence). The controller of the character recognition server 200 may generate character data (corresponding to the selected text) using the recognized phoneme and a pre-stored database. The controller of the character recognition server 200 may generate character recognition data in the form of words or sentences.

The control unit of the character recognition server 200 may store the generated character data in a storage unit. The control unit of the character recognition server 200 may transmit the character data to the portable device 100 through the communication unit (not shown) before the control information (to be described later). The control unit of the character recognition server 200 may transmit text data to the control information server 300 through a communication unit (not shown).

The control unit of the control information server 300 may convert to control information (for example, a control command) using text data. The text data may be received from the text recognition server 200 or the integrated portable device 100-1. The control information may control a screen (or operation, function) of the portable device 100. Also, the control information may control a screen (or an operation or function) of the character recognition server 200.

The control information server 300 may include a control information database. The control information server 300 may store a control information database in a storage unit.

The control unit of the control information server 300 may determine (or select, convert, and generate) control information corresponding to the recognized text data using the stored control information database. The control information server 300 determines (or selects) control information (eg, the mobile device 100 is parsable) that controls the mobile device 100 using text data and a control information database. , Transform, create). For example, when a continuous movement of the touch is received in the text in the speech balloon 191d11, the portable device 100 may receive an electrical signal (eg, a digital signal, an analog signal, or the like) through the communication unit. , Packet) to the character recognition server 100. The character recognition server 200 may convert the received electrical signal (or packet) into text data through character recognition (eg, 736-32 Yeoksam 1-dong, Gangnam-gu, Seoul). The character recognition server 200 may determine (or select, convert, and generate) control information using text data.

The above-described control information may be displayed on a screen of a map application that is executed (or previously executed) corresponding to text data (for example, 736-32, Yeoksam 1-dong, Gangnam-gu, Seoul). The above-described control information is displayed on the screen of a web browser executing (or previously executed) a business name (shop (company) name) corresponding to text data (for example, 736-32 Yeoksam 1-dong, Gangnam-gu, Seoul) You can do it. The application corresponding to the above-described control information is not limited to this, and various applications may be executed according to a setting (or linked).

The control unit of the control information server 300 may transmit control information to the portable device 100 through the communication unit.

When character recognition is performed by the character recognition server 200, text corresponding to character recognition performed by the character recognition server 200 on the screen of the mobile device 100 (for example, 'during character recognition'), an image A video (corresponding to 'recognizing text') and a video including a symbol and flash (corresponding to 'recognizing text') may be displayed.

When control information is determined (or selected, converted, or generated) in the control information server 300, text corresponding to the control information determination in the control information server 300 on the screen of the mobile device 100 (for example, A 'determining control information'), an image (corresponding to 'determining control information'), a video including a symbol and a flash (corresponding to 'determining control information') may be displayed.

Table 1 below may be an example of control information.

No. Text to be recognized
Character recognition type Determined application name One 787-8, XX-gu, OO City
Near YY Station
ZZ University Main Gate address
Map application 2 AA Restaurant (BB Building)
Mutual Map application 3 Passport photo
Photo Photography application 4 Preferences
Wi-Fi screen Preferences screen … … … …
… … … …

Referring to [Table 1], the text to be recognized may represent the text to be recognized as the text selected by the continuous movement of the touch. A character recognition type may be determined according to a result of character recognition of the selected text to be recognized. The application (or application name, application type, or stored application path, etc.) executed in the mobile device 100 (or currently being executed) may be determined by the character recognition type. Items displayed in [Table 1] (eg, text to be recognized, text recognition type, determined application, etc.) each have a bit string (not shown) or a size (eg, bit, or byte). Can be. The control information corresponding to [Table 1] may be converted and stored in the control information server 300 as well as in the portable device 100 or the integrated portable device 100-1.

Items displayed in [Table 1] (for example, text to be recognized, text recognition type, determined application, etc.) are not limited thereto, and may be added, deleted, and changed.

Referring to [Table 1], the text to be recognized may represent the text to be recognized as the text selected by the continuous movement of the touch. A character recognition type may be determined according to a result of character recognition of the selected text to be recognized. The text to be recognized as a character may continuously improve the accuracy of the recognition result through artificial intelligence (AI) technology (for example, including deep learning).

Patterns between unlabeled data can be identified through unsupervised learning of text data written in various forms. The elements of the data are clustered based on the similarity between the data. Based on the learned data, the content of the text can be inferred according to the context and the user's intention, and even if it does not exactly match the type of the text to be recognized in the example given in [Table 1], the type of the text can be classified.

The character recognition type may determine an application (or application name, application type, or stored application path, etc.) to be executed (or currently being executed) in the portable device 100.

The portable device 100 and the character recognition server 200 may be implemented as an integrated portable device 100-1. Since the operation of the integrated portable device 100-1 is substantially similar to the operation of the separate portable device 100 and the character recognition server 200 described above, a duplicate description may be omitted. The character recognition server 200 and the control information server 300 may be implemented as an integrated character recognition server 200-1. Since the operation of the integrated character recognition server 200-1 is substantially similar to the operation of the separate character recognition server 200 and the operation of the control information server 300 described above, a duplicate description may be omitted.

Referring to FIG. 5D, which is another embodiment of the present invention, character recognition of the selected text proceeds. The controller 110 may transmit (or convert and transmit) a signal corresponding to the stored text to be received by the character recognition server 200 according to a wireless communication standard.

The control information server 300 or a character recognition server (or an integrated character recognition server) may transmit control information corresponding to a character recognition result to the mobile device 100.

Character recognition of the selected text in step S340 of FIG. 3 (corresponding to FIG. 5D) is substantially similar to character recognition of the selected text in step S340 (corresponding to FIG. 4D), and duplicate description is omitted. .

In step S350 of FIG. 3, the current screen is changed according to the character recognition result.

Referring to FIG. 4E, the control unit 110 of the portable device 100 may receive control information through the communication unit. The control unit 110 of the portable device 100 may receive control information from the control information server 300 through the communication unit. Also, the control unit 110 of the portable device 100 may receive control information from the integrated character recognition server 200-1 through the communication unit.

The control unit 110 of the portable device 100 may receive text data ('736-32 Yeoksam 1-dong, Gangnam-gu, Seoul') through a communication unit. Also, the control unit 110 of the portable device 100 may receive text data from the integrated character recognition server 200-1 through the communication unit. The portable device 100 may receive text data and control information together or separately.

The control unit 110 may parse the received control information. The control unit 110 may change the current screen 191 through parsing of control information. The controller 110 may change the current screen (eg, a chat application screen, 191) to another screen (eg, a map application screen, 192) through parsing of control information.

The above-described change of the current screen is not only the display of the changed application screen, but also the change of the application being executed (for example, the first text (e.g., the first text recognized in response to the continuous movement of the touch in the displayed map application) For example, the location of the setting screen (refer to FIG. 5E)) to the second screen (change to the position of the second text (Wi-Fi, see FIG. 5F) recognized as a character corresponding to the continuous movement of the touch). have.

In addition, the change of the current screen is the first screen 191 of the same application (for example, a map application), another screen (for example, a chat history of another party (for example, Kim, proxy, not shown)), (Not shown).

The controller 110 may determine a map application (eg, displayed as a map application in the title area 192a of the application) to be executed through parsing of control information. The controller 110 may determine the location of the banquet venue 192b1 according to the character recognition result in the map display area 192b of the map application screen 192 through parsing of control information.

The controller 110 optimizes the time required (for example, 10 minutes) between a user's current location (for example, an office) and a dinner location (191b1) on the map application screen 192 through parsing of control information, The optimal distance (eg, 4.5 km) can be displayed as a pop-up (192b2). Depending on the selection of the direction keys 192b21 and 192b22 in the pop-up 192b, the shortest distance or the shortest time may be displayed.

The controller 110 may provide a user with feedback corresponding to a screen change according to the continuous movement of the touch received from the text in the speech balloon. The control unit 110 may provide a user with feedback corresponding to a screen change according to a character recognition result of a text selected by a continuous movement of a touch received in a speech balloon.

The controller 110 may provide one of visual feedback, auditory feedback, and tactile feedback. In addition, the controller 110 may provide a user with a combination of visual feedback, auditory feedback, and tactile feedback.

Visual feedback is a visual effect corresponding to a screen change according to a continuous movement of a touch received from text in a speech bubble (for example, an animation effect such as a fade applied to a separate image or a separate image, not shown) It may include. The control unit 110 may display visual feedback on the touch screen 190.

The auditory feedback may include a sound corresponding to a screen change according to the continuous movement of the touch received from the text in the speech balloon. The control unit 110 may output auditory feedback through the speaker 163. The controller 110 may output auditory feedback through one speaker or a plurality of speakers.

The tactile feedback may include vibration corresponding to a screen change according to the continuous movement of the touch received from the text in the speech balloon. The controller 110 may output tactile feedback through the vibration motor 164. When there are multiple vibration motors 164, the controller 110 may selectively output one of the plurality of vibration motors for tactile feedback.

The time for providing feedback (for example, 500 s) provided to the user may be changed through an environment setting (not shown). Also, at least one of a plurality of feedbacks (eg, visual feedback, auditory feedback, and tactile feedback) provided in response to a screen change according to a continuous movement of a touch received from the text in the speech balloon may be selected.

5E and 5F, which are other embodiments of the present invention, the control unit 110 of the portable device 100 may receive control information through the communication unit. The control unit 110 of the portable device 100 may receive control information from the control information server 300 or the integrated character recognition server 200-1 through the communication unit. The control unit 110 of the portable device 100 may receive text data (“Wi-Fi from the setting screen”) through a communication unit from a character recognition server (or an integrated character recognition server 200). The portable device 100 may receive text data and control information together or separately.

The control unit 110 may parse the received control information. The control unit 110 may change the current screen 191-1 through parsing of control information. The controller 110 may change the current screen (eg, a chat application screen, 191-1) to another screen (eg, environment setting, 193) through parsing of control information. The above-described change of the current screen may include not only the display of the changed application screen, but also the change of the executed application.

The controller 110 may determine an environment setting screen (eg, displayed as settings in the title area 193a of the screen) to be executed through parsing of control information. The controller 110 may display an environment setting screen 193 including settings including the Wi-Fi 193b1 in the setting display area 193b through parsing of control information. The Wi-Fi screen 193b11 may be displayed sequentially or continuously on the display of the environment setting screen 193. The control unit 110 may directly display the Wi-Fi screen 193b11 through parsing of control information. The user can turn on (off) the Wi-Fi using the Wi-Fi on / off switch 193b11-1 in the Wi-Fi screen 193b11.

In another embodiment of the present invention, the settings included in the displayed environment setting screen 193 are screens (eg, sound and vibration, respectively) according to the result of character recognition of the text selected in the speech balloon. (display), edge screen, lock screen and security, battery, user manual, bluetooth, flight mode, mobile hotspot, data usage, Etc.) may be displayed. The screens of the remaining settings not displayed on the above-described environment setting screen 193 may be displayed according to a character recognition result of the text selected in the speech balloon. In addition, a screen of settings included in the quick panel (not shown) as well as the environment setting screen 193 may be displayed according to a character recognition result of the text selected in the speech balloon.

In another embodiment of the present invention, a function (or setting) of the mobile device 100 may be changed as well as a screen change. For example, according to the text recognition result of the text selected in the speech balloon (for example, 'change the mobile phone to the vibration mode at the time of the meeting'), the controller 110 responds to the received control information, and the mobile device 100 You can change the sound of the bell to vibration mode immediately. According to the text recognition result of the text selected in the speech balloon (for example, 'the screen becomes brighter in a dark place'), the control unit 110 brightens the screen brightness of the mobile device 100 in response to the received control information. Can be changed. In addition, according to the character recognition result of the text (for example, 'low battery level') selected within the speech balloon, the control unit 110 sets the battery mode of the mobile device 100 to a battery saving mode in response to the received control information. Can be changed immediately. The above-described function (or setting) change of the portable device 100 is not limited to this, and may be applied to a function (or setting) that the portable device can support.

In another embodiment of the present invention, the application executed according to the control information corresponding to the character recognition of the selected text in the speech bubble is an external device control application installed in the portable device 100 (for example, home appliances in the home) It may include a device (or electronic device), an office electronic device in an office, or an Internet of Things (IoT) device. The home appliance (or electronic device) in the home is a home network (or electronic device) linked to a home network (or electronic device), a refrigerator, a washing machine, a robot cleaner, an air conditioner, a display device including a TV, and audio including speakers. It may include a device, a door lock, a security device including a CCTV, a gas blocking device, a power blocking device, or a sensor device including a gas detection sensor, a fire monitoring sensor, a security sensor, and the like. Office electronic devices may include printers, copiers, speakers, or fax machines that are linked to an internal network (including intranets). In addition, the IoT device may include a light fixture, various sensors installed in the home, a door lock, and the like. IoT devices may be included in the home appliance.

6A and 6B are views illustrating an example of a screen control method of a portable device according to another embodiment of the present invention.

6A and 6B, chat is performed in the chat application 191 displayed on the screen of the mobile device. A speech balloon 191b1-2 to 191b2-2 exchanged between the other party (one of the sender and the receiver) and the user may be displayed on the display area 191b-2.

The user inputs a touch (or hovering, not shown) to the speech balloon 191b11-2. The continuous movement of the touch and / or touch input from the speech balloon 191b11-2 is substantially similar to the continuous movement of the touch and / or touch input in the steps S320 and S330 of FIG. 3 described above (for example, For example, a difference between a touch or a touch gesture) is omitted.

The text 430s selected in response to the continuous movement of the touch from the first touch position to the last touch position may be a 'strong air conditioner in the living room'. Since the text 430s selected in response to the continuous movement of the touch is substantially similar to the text 410s selected in response to the continuous movement of the touch in FIG. 4D (for example, the difference between the selected texts), the text 430s is overlapped. Description is omitted.

Since the progress of character recognition of the text 430s selected in response to the continuous movement of the touch is substantially similar to the progress of character recognition of the selected text in step S340 of FIG. 3, a duplicate description is omitted.

Referring to FIG. 6B, the control unit 110 of the portable device 100 may receive control information through the communication unit. The control unit 110 may parse the received control information. The control unit 110 may change the current screen 191-2 through parsing of control information. The control unit 110 changes the current screen (for example, a chat application screen, 191-2) to another screen (for example, an air conditioner control application, 194) through parsing of control information (or, Run).

The controller 110 may display (or execute) the air conditioner control application screen 194 displayed as the home air conditioner in the title area 194a through parsing of control information. The control unit 110 displays an operation mode (for example, strong wind, 194b1) and a current temperature (for example, 23 ° C.) in the air conditioner control application display area 194b of the air conditioner control application screen 194 through parsing of control information. 194b2) may be displayed. In addition, the user may select an operation mode (for example, strong wind) in the air conditioner control application display area 194b of the air conditioner control application screen 194.

The controller 110 may provide a user with feedback corresponding to a screen change (from the screen 191 to the screens 193 and 194) according to the continuous movement of the touch received from the text in the speech balloon. The controller 110 provides feedback corresponding to a screen change (from the screen 191 to the screens 193 and 194) according to the text recognition result of the text selected by the continuous movement of the touch received in the speech balloon. It can be provided to the user.

In step S350 of FIG. 3, when the current screen is changed according to the character recognition result, the screen control method of the portable device ends.

The methods according to the exemplary embodiment of the present invention may be implemented in the form of program instructions that can be executed through various computer means and recorded in a computer readable medium. The computer-readable medium may include program instructions, data files, data structures, or the like alone or in combination. For example, the computer-readable medium is volatile or non-volatile storage device, such as a storage device such as ROM, regardless of whether it is erasable or rewritable, or, for example, RAM, memory chip, device or integrated circuit It may be stored in a memory or a storage medium readable by a machine (for example, a computer) while being optically or magnetically recordable, for example, a CD, DVD, magnetic disk or magnetic tape. In addition, the computer readable program is stored in a computer readable storage medium of the server, and the computer program can be downloaded to a computing device over a network.

It will be appreciated that a memory that may be included in a portable device is an example of a machine readable storage medium suitable for storing programs or programs including instructions for implementing embodiments of the present invention. The program instructions recorded on the medium may be specially designed and configured for the present invention, or may be known and available to those skilled in computer software.

As described above, the present invention has been described by the above-described exemplary embodiments and drawings, but the present invention is not limited to the above-described exemplary embodiments, and those skilled in the art to which the present invention pertains from these descriptions. Various modifications and variations are possible.

Therefore, the scope of the present invention should not be limited to the described exemplary embodiments, but should be defined not only by the claims below, but also by the claims and equivalents.

100: mobile device 110: control unit
190: Touch screen 191: Application screen
191a: Title area 191b: Display area
191b1: speech balloon 191c: input area
410: Touch 410-1: Continuous movement of touch

Claims (18)

Sequentially displaying a plurality of text display areas corresponding to each of at least one transmission message transmitted through the application and at least one received message received through the application on an application screen displayed on the portable device;
Detecting a touch in a text display area of a plurality of text display areas displayed on the application screen;
Detecting a continuous movement of the touch;
Obtaining text recognized in a region in which the continuous movement of the touch is detected in the one text display region;
Obtaining a control command for changing a setting of the portable device corresponding to the obtained text; And
And changing settings of the portable device based on the obtained control command.
According to claim 1,
The plurality of text display areas include at least one of a text display area of a user corresponding to the at least one transmission text message and a text display area of a counterpart corresponding to the at least one received text message. .
According to claim 1,
The step of changing the settings of the portable device,
A method for controlling a screen of a mobile device, including changing to a vibration mode, changing a screen brightness, or changing to a power saving mode.

According to claim 1,
The continuous movement of the touch includes a single touch gesture and a multi-touch gesture, and a method of controlling a screen of a mobile device including flick, swipe, drag, and rotation.
According to claim 1,
The text recognized text,
A method for controlling a screen of a portable device obtained from one of the portable device and a character recognition server connected to the portable device in communication.
According to claim 1,
And displaying a setting screen corresponding to the acquired text.
The method of claim 6,
The setting screen,
A method of controlling a screen of a portable device, comprising a plurality of consecutive setting screens for setting the environment of the portable device.
According to claim 1,
The obtained control command is a screen control method of a mobile device that is received from the outside.
According to claim 1,
A method of controlling a screen of a mobile device, wherein text recognized as a character in the text display area is selected by continuous movement of the touch.
The method of claim 9,
A method of controlling a screen of a mobile device in which text selected by continuous movement of the touch in the one text display area is displayed differently from other text.
According to claim 1,
Further comprising the step of providing feedback corresponding to the change to the setting screen,
The feedback includes at least one of visual feedback, auditory feedback, and tactile feedback.
A communication unit connected to the outside;
touch screen; And
Includes; a processor for controlling the communication unit and the touch screen,
The processor,
A plurality of text display areas corresponding to each of at least one transmission text message transmitted through the application and at least one reception text message received through the application are sequentially displayed on an application screen,
Acquiring a character recognition result of text included in an area in which a continuous movement of a touch is detected in one text display area of the plurality of text display areas,
When the text recognition result of the text is transmitted to the outside through the communication unit, and control information corresponding to the text recognition result of the text is received through the communication unit, the setting of the portable device is changed based on the received control information. Mobile devices.
The method of claim 12,
And the processor controls to display one of all and some of the texts in the one text display area distinctly from other unselected texts based on the continuous movement of the touch.
The method of claim 12,
Further comprising a pressure touch pad,
The touch is a mobile device that is detected through the pressure touch pad.
The method of claim 12,
The processor,
A mobile device that changes the setting of the mobile device to a vibration mode, a screen brightness change, or a power saving mode based on the received control information.
The method of claim 12,
The text recognition result of the text,
A portable device obtained from one of a character recognition server connected to the portable device through the portable device and the communication unit.
The method of claim 12,
The processor,
A mobile device displaying a setting screen corresponding to the received control information on the touch screen.
The method of claim 17,
The setting screen,
A portable device comprising a plurality of consecutive environment setting screens for setting the environment of the portable device.

Images