JP2012226481A - Communication terminal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a communication terminal capable of determining a button area and performing tactile sensation feedback to a user even for an application in which an operation button and a tactile sensation feedback are not made to correspond to each other.SOLUTION: The communication terminal includes a touch sensor 16 which is arranged on a display part 14, a vibration generation part 20 which vibrates the touch sensor 16, a control part 12 which executes the application and displays an execution screen, a dummy application execution part 102 which executes a dummy application, and a button area determination part 26 which determines a button area on the execution screen based upon output change of the dummy application caused by false depression when a plurality of trial points are set and each trial point is falsely depressed. The control part 12 vibrates a touch surface 16A by the vibration generation part 20 on detecting depression on the touch surface 16A corresponding to the button area determined by the button area determination part 26 as a button area.

Description

  The present invention relates to a communication terminal.

  In recent years, an increasing number of communication terminals such as mobile phones and game machines adopt a touch sensor as an interface for input by a user. The interface using the touch sensor has an advantage that the user can operate intuitively.

  In an interface using a touch sensor, a user operates a communication terminal by touching a button or the like displayed on the screen. However, since such a touch operation does not generate tactile feedback, the user cannot determine whether or not a button press has been accepted based on the operational feeling, and it is difficult to perform an operation such as a blind touch.

  In order to give tactile feedback to a user, for example, a device that gives a tactile sensation such as a click feeling to a user's finger when a touch sensor receives an input has been proposed (for example, see Patent Document 1).

Japanese Patent No. 4633183

  According to the device disclosed in Patent Literature 1, when a touch operation on a button displayed on an interface using a touch sensor is received, vibration can be applied to a user's finger or the like. Thereby, the user can obtain a feeling like a click feeling, and an operation like a blind touch is also possible.

  However, in order to provide the vibration as described above, in the application executed on the communication terminal, it is necessary to associate the vibration to be provided to the user with the operation button displayed on the screen.

  In recent years, applications that are not specialized for a specific communication terminal, such as applications downloaded from Web pages, are increasing. It is not realistic to expect these applications to be associated with operation buttons and vibrations given to the user for all types of terminals.

  As a result, depending on the application to be used, there will be a mixture of those that do not correspond to the operation buttons and tactile feedback given to the user, and the user will feel uncomfortable because the operation feeling varies depending on the application. It is assumed that

  Therefore, an object of the present invention made in view of the above point is to determine a button area and provide tactile feedback to the user even for an application in which operation buttons and tactile feedback to be given to the user are not associated with each other. It is to provide a communication terminal.

A communication terminal according to the first aspect of achieving the above object is
A display unit;
A touch sensor disposed on the display unit;
A vibration generator for vibrating the touch surface of the touch sensor;
A control unit that executes an application and displays an execution screen of the application on the display unit;
A pseudo application execution unit that executes the same application as the application as a pseudo application;
A button area for determining a button area on the execution screen based on an output change caused by the pseudo pressing of the pseudo application when a plurality of trial points are set on the display unit and the trial points are pseudo pressed. A determination unit,
The control unit vibrates the touch surface by the vibration generation unit when detecting a press on the touch surface corresponding to the button region determined to be a button region by the button region determination unit. To do.

The communication terminal according to the second aspect is
In the communication terminal according to the first aspect,
The button area determination unit
A state holding unit that stores an output change caused by the pseudo pressing of the pseudo application in association with a trial point at which the output change is obtained;
And an area determination unit that determines an area where the output changes are the same and the positions of the trial points are adjacent to each other as the same button area.

The communication terminal according to the third aspect is
In the communication terminal according to the first aspect,
The button area determination unit
An input data holding unit that stores input data input to the application for a predetermined trial period;
An output data holding unit for storing the output data of the application corresponding to the input data for the trial period for the trial period;
An application state holding unit that stores the state of the application when the input data for the trial period is given as a pseudo application state for the trial period;
A state holding unit that stores an output change caused by the pseudo pressing of the pseudo application in association with a trial point at which the output change is obtained;
An area determination unit that determines that the output change is the same and the position of the trial point is adjacent to the same button area;
With
The controller is
The pseudo application execution unit is executed using the pseudo application state,
At the timing corresponding to the pseudo application state, input data for the trial period is input to the pseudo application being executed in the pseudo application state,
The pseudo-press is executed while the input data is being input to the pseudo application, and an output corresponding to the pseudo-press and an output of a timing corresponding to the output data stored in the output data holding unit And the output change caused by the pseudo-pressing of the pseudo application is given to the state holding unit.

The communication terminal according to the fourth aspect is
In the communication terminal according to any one of the first to third aspects,
A character extraction unit for extracting characters displayed in the same button area;
A storage unit that stores vibration information corresponding to the characters,
The control unit vibrates the touch surface by the vibration generation unit using vibration information corresponding to the extracted character.

A communication terminal according to the fifth aspect is
The communication terminal according to the fourth aspect further includes a load detection unit that determines whether or not pressing on the touch surface corresponds to pressing exceeding a threshold value,
The storage unit stores different vibration information depending on whether pressing on the touch surface exceeds a threshold,
The control unit vibrates the touch surface by the vibration generation unit using corresponding vibration information according to whether or not the pressing on the touch surface is a pressing exceeding a threshold value. is there.

  As described above, the solution of the present invention has been described as an apparatus. However, the present invention can be realized as a method, a program, and a storage medium storing the program, which are substantially equivalent to these. It should be understood that these are also included.

  According to the present invention, it is possible to determine a button area and give tactile feedback to a user even for an application in which operation buttons and tactile feedback given to the user are not associated with each other.

It is a functional block diagram which shows schematic structure of the communication terminal which concerns on 1st Embodiment of this invention. It is a schematic block diagram which shows a part of mounting structure of the communication terminal shown in FIG. It is a functional block diagram which shows schematic structure of the button area | region determination part which concerns on 1st Embodiment of this invention. It is a flowchart which shows the process sequence which determines the button area | region which concerns on 1st Embodiment of this invention. It is a flowchart which shows the process sequence which determines the button area | region which concerns on 1st Embodiment of this invention. It is a flowchart which shows the process sequence which determines the button area | region which concerns on 1st Embodiment of this invention. It is a figure which shows a mode that the button area | region which concerns on 1st Embodiment of this invention is determined. It is a flowchart which shows the process sequence which gives the vibration which concerns on 1st Embodiment of this invention. It is a functional block diagram which shows schematic structure of the communication terminal which concerns on 2nd Embodiment of this invention. It is a functional block diagram which shows schematic structure of the button area | region determination part which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process sequence which determines the button area | region which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process sequence which determines the button area | region which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process sequence which determines the button area | region which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process sequence which determines the button area | region which concerns on 2nd Embodiment of this invention. It is a flowchart which shows the process sequence which determines the button area | region which concerns on 2nd Embodiment of this invention. It is a figure which shows a mode that the button area | region which concerns on 2nd Embodiment of this invention is determined. It is a flowchart which shows the process sequence which gives the vibration which concerns on 2nd Embodiment of this invention.

  Embodiments of a communication terminal according to the present invention will be described below with reference to the drawings.

(First embodiment)
FIG. 1 is a functional block diagram illustrating a schematic configuration of the communication terminal 1. The communication terminal 1 includes a control unit 12, a display unit 14, a touch sensor 16, a load detection unit 18, a vibration generation unit 20, a storage unit 22, a communication unit 24, and a button area determination unit 26.

  The control unit 12 controls and manages the entire communication terminal 1 including each functional unit of the communication terminal 1, and is configured by a suitable processor such as a CPU.

  The display unit 14 is configured by, for example, a liquid crystal display (LCD) or an organic EL display. A touch sensor 16 that detects an input to the touch surface 16A (see FIG. 2) by a user's finger or the like is disposed on the front surface of the display unit 14.

  The touch sensor 16 detects pressing of the touch surface 16A with a finger, a stylus pen, or the like. The touch sensor 16 outputs known two-dimensional position information of a touch position such as a resistive film type, a capacitance type, and an optical type. It consists of a touch sensor and is arranged on the front surface of the display unit 14. The position information from the touch sensor 16 is supplied to the control unit 12. An object to be pressed onto the touch surface 16A such as a user's finger or stylus is hereinafter referred to as a “pressed object”. In addition, hereinafter, the user's finger is described as an example of the pressing target, but this is an example, and the pressing target is not limited to the user's finger. Note that it is not essential for the user's finger or the like to physically press the touch sensor 16 when the touch sensor 16 detects an input. For example, when the touch sensor 16 is optical, the touch sensor 16 detects a position where infrared rays are blocked by a finger or the like, and therefore can detect an input even when there is no physical depression.

  The load detection unit 18 detects a pressing load on the touch surface 16A of the touch sensor 16, and is configured using, for example, a piezoelectric element, a strain gauge sensor, or the like. The load detection unit 18 outputs detected pressure load data to the control unit 12. For example, when both the vibration generation unit 20 and the load detection unit 18 are configured using piezoelectric elements, the vibration generation unit 20 and the load detection unit 18 can be configured integrally with a common piezoelectric element. This is because the piezoelectric element has a property of generating electric power when pressure is applied and deforming when electric power is applied.

  The vibration generation unit 20 transmits vibration to the touch surface 16A of the touch sensor 16, and is configured using, for example, a piezoelectric element or an ultrasonic transducer. When the vibration generating unit 20 vibrates, vibration can be applied to a user's finger or the like pressing the touch sensor 16. In addition, the vibration generation part in this invention should just vibrate the touch surface 16A of the touch sensor 16 directly or indirectly, for example, let a vibration generation part be a vibration motor (eccentric motor), and touch sensor 16's The touch surface 16A may be directly vibrated, or the touch member 16A may be indirectly vibrated by vibrating the holding member that holds the touch sensor 16 or the housing of the communication terminal.

  In addition, when the control unit 12 changes the driving voltage applied to the piezoelectric element, the vibration generating unit 20 gives different vibrations to the user's finger. The controller 12 can change the frequency, amplitude, waveform, etc. of the drive voltage. The frequency and amplitude can be changed in magnitude, and the waveform can be a sine wave, a rectangular wave, a triangular wave, or the like.

  The storage unit 22 stores a vibration pattern presented by the vibration generation unit 20 and also functions as a work memory. Here, the vibration pattern is defined by the vibration method (frequency, phase, vibration interval, number of vibrations, etc.), vibration intensity (amplitude, etc.), and the like.

  The communication unit 24 wirelessly connects to a wireless communication network and executes communication such as telephone, electronic mail, and the Internet.

  The button area determination unit 26 determines a button area on the execution screen of the application displayed on the display unit 14. Details of the button area determination unit 26 will be described separately.

  2 shows a schematic configuration example of the mounting structure of the touch sensor 16, the vibration generating unit 20, and the display unit 14 shown in FIG. 1, FIG. 2 (a) is a cross-sectional view of the main part, and FIG. 2 (b) It is a principal part top view. The display unit 14 is housed and held in the housing 61. The touch sensor 16 is held on the display unit 14 via an insulator 62 made of an elastic member. Note that the touch sensor 16 is held on the display unit 14 via insulators 62 disposed at four corners outside the display area A of the display unit 14 indicated by virtual lines in FIG.

  The casing 61 is provided with an upper cover 63 so as to cover the surface area of the touch sensor 16 that is out of the display area A of the display unit 14. An elastic member is provided between the upper cover 63 and the touch sensor 16. A dustproof insulator 64 is provided.

  In the touch sensor 16, when the surface member having the touch surface 16 </ b> A is made of, for example, a transparent film or glass, the back member is made of glass or acrylic, and the touch surface 16 </ b> A is pressed via the insulator 64, The pressing portion is bent (distorted) by a small amount in accordance with the pressing force, and the entire touch sensor 16 including the back member is bent by a small amount.

  The vibration generating unit 20 has two piezoelectric elements 71. The piezoelectric elements 71 are mounted on the back surface of the touch sensor 16 in the vicinity of the two opposing sides covered by the upper cover 63 so as to bend and displace the touch sensor 16. The piezoelectric element 71 is preferably attached to the touch sensor 16 so that the bending direction of the touch sensor 16 when pressed and the bending direction of the piezoelectric element 71 coincide. In FIG. 2, since the touch sensor 16 has a rectangular shape in plan view, the two piezoelectric elements 71 are mounted along the long side in the vicinity of the two long sides facing each other. In FIG. 2B, the casing 61, the upper cover 63, and the insulator 64 shown in FIG. 2A are not shown.

  FIG. 3 is a functional block diagram showing a schematic configuration of the button area determination unit 26. The button area determination unit 26 includes a pseudo application execution unit 102, a pseudo pressing unit 104, a pseudo output unit 106, a pseudo transmission unit 108, a state holding unit 110, an area determination unit 112, and a character extraction unit 114.

  The pseudo application execution unit 102 executes the same application as the application executed by the control unit 12. Hereinafter, an application executed by the pseudo application execution unit 102 is referred to as a “pseudo application”. The pseudo application execution unit 102 can execute the pseudo application in parallel with the application executed by the control unit 12 in parallel, or can execute the pseudo application after the control unit 12 executes the application. You can also.

  The pseudo-pressing unit 104 sets a plurality of trial points in the display unit 14 and gives information indicating that the trial points are pseudo-pressed to the pseudo application. Hereinafter, the pseudo-pressing unit 104 pseudo-pressing the trial point is referred to as “pseudo-pressing”.

  The pseudo output unit 106 receives and stores data (hereinafter referred to as “pseudo output data”) output by the pseudo application when the pseudo pressing unit 104 performs pseudo pressing. The pseudo output data is, for example, display data (hereinafter referred to as “pseudo display data”). The pseudo display data is only stored as data in the pseudo output unit 106 and is not actually displayed on the display unit 14.

  The pseudo transmission unit 108 receives and stores transmission data (hereinafter referred to as “pseudo transmission data”) output by the pseudo application when the pseudo pressing unit 104 performs pseudo pressing. The pseudo transmission data is only stored as data in the pseudo transmission unit 108 and is not actually transmitted from the communication unit 24.

  The state holding unit 110 stores the change caused by the pseudo pressing of the pseudo pressing unit 104 in association with the coordinates of the trial point. For example, when it is determined that the pseudo display data and the display displayed on the display unit 14 are different, the state holding unit 110 stores the pseudo display data in association with the coordinates of the trial points. If the state holding unit 110 determines that the pseudo transmission data is stored in the pseudo transmission unit 108 by the pseudo pressing of the pseudo pressing unit 104, the state holding unit 110 stores the pseudo transmission data in association with the coordinates of the trial point.

  When the pseudo output data stored in the state holding unit 110 is the same and the coordinates of the trial points are adjacent, the area determination unit 112 determines that the areas constitute the same button, and the area Are areas constituting the same button.

  The character extraction unit 114 performs character recognition on an area determined to be the same button area, and when a character string having a recognition rate equal to or higher than a predetermined threshold exists, the character string having the highest recognition rate among them Is stored as a character string displayed in the same button area.

  A procedure for determining the button area in the application execution screen displayed on the display unit 14 while executing an application will be described with reference to the flowcharts shown in FIGS. 4 to 6 show one flowchart divided into three diagrams.

  The flowchart of the part shown in FIG. 4 is demonstrated. The control unit 12 executes the application (step S101). Subsequently, the pseudo application execution unit 102 executes the pseudo application (step S102).

  The control unit 12 determines whether or not the button area has already been determined by the button area determination unit 26 in the currently executing application (step S103). If it is determined in step S103 that the button area has not yet been determined, the control unit 12 determines whether or not the application is still being executed (step S104).

  If the control unit 12 determines in step S104 that the execution of the application has already ended, the control unit 12 ends the pseudo application being executed by the pseudo application execution unit 102 (step S105). When it is determined in step S104 that the application is continuously executed, the control unit 12 clears the data stored in the state holding unit 110 (step S106).

Subsequently, the button area determination unit 26 executes a pseudo-pressing repeated process. In the present embodiment, the case where there are N _max trial points will be described as an example (step S107). The control unit 12 determines whether or not the application state has been updated. If it is determined that the application state has not been updated, the control unit 12 exits the pseudo-pressing repeated process and returns to step S104 (step S108).

  When determining in step S108 that the application state has not been updated, the control unit 12 copies the application state to the pseudo application being executed by the pseudo application execution unit 102 (step S109). The pseudo-pressing unit 104 pseudo-presses the first trial point (step S110). The control unit 12 compares the pseudo display data output to the pseudo output unit 106 as a result of the pseudo press and the display data displayed on the display unit 14 (step S111).

  If the control unit 12 determines in step S111 that there is no difference between the pseudo display data and the display data displayed on the display unit 14, is the external transmission request data stored in the pseudo transmission unit 108? It is determined whether or not (step S112). If the control unit 12 determines in step S112 that the external access request data is not stored, the control unit 12 ends the first iteration and returns to step S107 (step S114).

  The control unit 12 determines that there is a difference between the pseudo display data and the display data displayed on the display unit 14 in step S11, or that the external access request data is stored in step S112. In this case, the state of the pseudo application is stored in the state holding unit 110 in association with the coordinates of the trial point, and the control unit 12 adds the registration number M to the combination of the state of the pseudo application and the coordinates of the trial point. Store in the holding unit 110. The control unit 12 increases the registration number M by one. Proceed to step S114 (step S113).

  The flowchart of the part shown in FIG. 5 is demonstrated. When the iterative process from step S107 to step S114 is completed, the area determination unit 112 checks the number of registrations M registered in the state holding unit 110. The number of registrations M corresponds to the number of trial points determined as “Yes” in step S111 or step S112. When the registration number M is 0, the area determination unit 112 returns to step S103 (step S121).

If the area determination unit 112 determines in step S121 that the registration number M is greater than 0, the area determination unit 112 clears the area information and sets P _cnt = 0 (step S122). The area determination unit 112 executes the repetition process for the number M of registrations, where L is a variable for the repetition process (steps S123 to S131). The region determination unit 112 sequentially reads the state P from the state holding unit 110 and compares it with the state L (step S125).

If it is determined in step S125 that the state L and the state P are equal, the region determination unit 112 calculates the distance between the trial point L and the region P (step S126), and whether the calculated distance is smaller than the threshold LNG _th . It is determined whether or not (step S127). When the region determination unit 112 determines that the distance calculated in step S127 is smaller than the threshold LNG _th , the region determination unit 112 determines that the position of the trial point L is in the vicinity of the region P and integrates the position of the trial point L into the region P. (Step S128).

  When determining that the state L and the state P are not equal in step S127, the region determination unit 112 increments the number of the state P by one and repeats the process of step S125. If the region determination unit 112 determines that the state L is not equal to any of the regions being integrated, the region determination unit 112 adds the state L as a new region and increases the number of regions P by one (step S130). The area determination unit 112 executes this repetition process for all trial points stored in the state holding unit 110.

The flowchart of the part shown in FIG. 6 is demonstrated. The character extraction unit 114 performs character extraction on the integrated button area (steps S141 to S146). The character extraction unit 114 extracts characters from the button area, and obtains a recognition rate Q of candidate characters (step S142). Character extraction unit 114, the recognition rate Q of the character is equal to or threshold _{Q th} is larger than (step S143).

Character extraction unit 114, if the recognition rate Q is determined the threshold Q _th larger in step S143, a character which is a candidate is a character that is described in the button area, corresponding to the character in the button area Vibration information is assigned (step S144). Character extraction unit 114, if the recognition rate Q is determined the threshold _{Q th} smaller than in step S143, allocates the vibration information of the generic to the button area (step S145).

  Fig.7 (a)-FIG.7 (c) show a mode that the communication terminal 1 determines a button area | region. FIG. 7A shows a case where the display unit 14 has three button areas (B1 to B3). FIG. 7B shows a state in which the pseudo pressing unit 104 has set 83 trial points T on the display unit 14. FIG. 7C shows a button area determination for the display unit 14 where the button area of FIG. 7A exists in a state where the pseudo-pressing unit 104 has set the trial point T shown in FIG. 7B. Indicates the execution result.

  In FIG. 7C, the button area determination unit 26 determines that the three areas of the buttons B4 to B6 are button areas. The areas of the buttons B4 to B6 correspond to the buttons B1 to B3, respectively, and when the touch sensor 16 detects pressing in any of the areas of the buttons B4 to B6, the vibration generator 20 Gives vibration according to the characters in the button area.

  A procedure in which the communication terminal 1 determines a button area and applies a vibration corresponding to a character displayed in the button area according to the strength of pressing will be described with reference to the flowchart shown in FIG.

  The control unit 12 executes the application (step S201). The button area determination unit 26 determines a button area in the display screen of the display unit 14 (step S202). The control unit 12 determines whether or not the application is still being executed (step S203).

  When it is determined in step S203 that the application is continuously executed, the control unit 12 determines whether or not the application execution screen displayed on the display unit 14 has been updated (step S204). If it is determined in step S204 that the application execution screen has been updated, the button area determination unit 26 executes button area determination for the updated execution screen (step S205).

  The control unit 12 determines whether or not the load detection unit 18 has detected a press that exceeds the threshold value (step S206). When the load detection unit 18 detects a press exceeding the threshold value in step S206, the control unit 12 determines whether or not the detected press position is included in the button area (step S207).

When it is determined in step S207 that the pressed position is included in the button area, the character extraction unit 114 extracts characters in the button area and reads vibration information corresponding to the extracted characters from the storage unit 22. The character extraction unit 114 reads general-purpose vibration information from the storage unit 22 when there is no character that matches the extracted character at a predetermined threshold value _Qth or more (step S208).

  The vibration generating unit 20 applies vibration based on the vibration information (step S209). When the load detection unit 18 does not detect pressing exceeding the threshold value in step S206, the control unit 12 determines whether the touch sensor 16 detects pressing. Here, it is determined that the pressing below the threshold value by the load detection unit 18 and the pressing detected by the touch sensor 16 is “contact” (step S210).

When the touch sensor 16 detects contact in step S210, the control unit 12 determines whether or not the detected contact position is included in the button area (step S211). If it is determined in step S211 that the contact position is included in the button area, the character extraction unit 114 extracts characters in the button area and reads vibration information corresponding to the extracted characters from the storage unit 22. The character extraction unit 114 reads general-purpose vibration information from the storage unit 22 when there is no character that matches the extracted character with a predetermined threshold value _Qth or more (step S212). The vibration generating unit 20 applies vibration based on the vibration information (step S213).

  As described above, the communication terminal 1 determines a button area in which an application is being executed for an arbitrary application, and displays a character displayed in the button area when a user's finger or the like is pressed in the button area. Corresponding vibration can be applied to the user's finger or the like.

(Second Embodiment)
In the second embodiment, the IPTV (Internet Protocol TV) that receives and reproduces the video distributed from the video distribution server, and the video phone that makes a call by transmitting and receiving audio data and video data in real time are used. A communication terminal capable of determining a button area even during execution of an application whose display screen changes regardless of presence or absence will be described.

  FIG. 9 is a functional block diagram illustrating a schematic configuration of the communication terminal 2. The communication terminal 2 includes a control unit 32, a display unit 34, a touch sensor 36, a load detection unit 38, a vibration generation unit 40, a storage unit 42, a communication unit 44, and a button area determination unit 46.

  Since the display unit 34, the touch sensor 36, the load detection unit 38, the vibration generation unit 40, the storage unit 42, and the communication unit 44 have the same functions as the corresponding functional blocks in the first embodiment, description thereof will be omitted.

  The control unit 32 controls and manages the entire communication terminal 2 including each functional unit of the communication terminal 2, and is configured by a suitable processor such as a CPU. Details will be described separately.

  The button area determination unit 46 determines a button area on the execution screen of the application displayed on the display unit 34. Details will be described with reference to FIG.

  FIG. 10 is a functional block diagram illustrating a schematic configuration of the button area determination unit 46. The button area determination unit 46 includes a pseudo application execution unit 202, a pseudo pressing unit 204, a pseudo output unit 206, a pseudo transmission unit 208, a state holding unit 210, an area determination unit 212, a character extraction unit 214, a pseudo time control unit 216, an application A state holding unit 218, an input data holding unit 220, and an output data holding unit 222 are provided.

  The simulated application execution unit 202, simulated pressing unit 204, simulated output unit 206, simulated transmission unit 208, state holding unit 210, area determination unit 212, and character extraction unit 214 are the same as the corresponding functional blocks of the first embodiment. Since it is a function, description is omitted.

  The pseudo time control unit 216 controls the pseudo time when the pseudo application execution unit 202 executes the pseudo application, and inputs the time input data stored in the input data holding unit 220 to the pseudo application. The time input data is input at the same timing.

  The application state holding unit 218 stores the state of the application executed by the control unit 32 for the trial period.

  The input data holding unit 220 stores the input data corresponding to the application state stored in the application state holding unit 218 for a preset trial period, that is, for the time for storing changes for all trial points. .

  The output data holding unit 222 stores output data for the trial period when data stored in the input data holding unit 220 is input to the application state stored in the application state holding unit 218.

  A procedure for determining the button area in the application execution screen displayed on the display unit 34 while executing the application will be described with reference to the flowcharts shown in FIGS. FIGS. 11 to 15 show one flowchart divided into five diagrams.

  The flowchart of the part shown in FIG. 11 is demonstrated. The control unit 32 executes the application (step S301). Subsequently, the pseudo application execution unit 202 executes the pseudo application (step S302).

The control unit 32 determines whether or not the button region has already been determined by the button region determination unit 46 in the currently executing application (step S303). If it is determined in step S303 that the button area has not yet been determined, the pseudo-pressing unit 204 determines the pressing trial start timing _Tt (step S304).

  The control unit 32 initializes the memory of the state holding unit 210, and sets the registration number M of the state holding unit 210 to M = 0 (step S305). The control unit 32 determines whether or not the application is still being executed (step S306). When determining in step S306 that the execution of the application has already ended, the control unit 32 ends the pseudo application being executed by the pseudo application execution unit 202 (step S307).

When it is determined in step S306 that the execution of the application is continued, the control unit 32 determines whether or not the trial start timing _Tt set in step S304 is reached (step S308). The control unit 32 stores the input data input to the application in the input data holding unit 220 for the trial period T _s from the trial start timing T _t , and also outputs the output data output from the application to the output data holding unit 222. (Step S309).

The flowchart of the part shown in FIG. 12 is demonstrated. The pseudo time control unit 216 sets the time of the pseudo time control unit 216 to the trial start timing T _t (step S321). The pseudo time control unit 216 sets the state of the pseudo application executed by the pseudo application execution unit 202 to the state of the trial start timing T _t (step S322). From this state, the pseudo time control unit 216 starts measuring time (step S323).

The control unit 32 reads input data corresponding to the measured time during the trial period T _s from the input data holding unit 220 and causes the pseudo application to input the input data (step S324). The control unit 32 stores the corresponding output data output from the pseudo application in the pseudo output unit 206 (step 325).

  The control unit 32 determines whether there is a difference between the output data stored in the output data holding unit 222 and the output data stored in the pseudo output unit 206 (Step S326). When it is determined that there is a difference in step S326, the control unit 32 excludes the trial point T determined to have a difference from the range in which the button region determination unit 46 determines the button region (step S327).

The flowchart of the part shown in FIG. 13 is demonstrated. The pseudo time control unit 216 sets the time of the pseudo time control unit 216 to the trial start timing T _t (step S342). The pseudo time control unit 216 sets the state of the pseudo application executed by the pseudo application execution unit 202 to the state of the trial start timing T _t (step S343).

From this state, the pseudo time control unit 216 starts measuring time (step S344). The pseudo-pressing unit 204 executes the pseudo-pressing, and gives the pseudo-pressing coordinate information to the pseudo application (step S345). The control unit 32 reads the input data corresponding to the time measured during the trial period T _s from the input data holding unit 220 and gives it to the pseudo application (step S346).

  The control unit 32 stores the corresponding output data output from the pseudo application in the pseudo output unit 206 (step S347). The control unit 32 determines whether there is a difference between the output data stored in the output data holding unit 222 and the output data stored in the pseudo output unit 206 (step S348).

When determining that there is a difference in step S348, the control unit 32 stores the trial point T and the output data determined to have a difference in the state holding unit 210, and increases the value of the registration number M by one (step S349). The control unit 32 repeats steps S341 to S350 until the N _max th trial point T is reached. When the repetitive processing from step S341 to S350 is completed, the control unit 32 checks whether or not the registration number M is M = 0 (step S351). When there is registration, that is, when the number of registrations M> 0, the control unit 32 clears the region information (P _cnt = 0) (step S352).

  The flowchart of the part shown in FIG. 14 is demonstrated. The region determination unit 212 reads the state stored in the state holding unit 210 and compares the read state (STAT (L)) with the state of the region being combined as a button region (STAT (P)) (Step S1). S363).

If it is STAT (L) = STAT (P) in step S363, the area determination unit 212 calculates the distance between the read coordinates (POINT (L)) and the area being integrated (AREA (P)). (Step S364). The area determination unit 212 determines whether or not the calculated distance is smaller than the threshold LNG _th (step S365). If the area determination unit 212 determines in step S365 that the calculated distance is smaller than the threshold LNG _th , the area determination unit 212 integrates the read coordinates (POINT (L)) into the area being integrated (AREA (P)).

If the read coordinates (POINT (L)) are not integrated into any of the integrated areas, the area determination unit 212 adds the read coordinates (POINT (L)) as a new area and adds the number of areas P _cnt Is updated (P _cnt ++) (step S368). The area determination unit 212 performs the processing from step S362 to S368 for all trial points T registered in the state holding unit 210 (steps S361 to S369). When the processing for all the trial points T registered in the state holding unit 210 is completed, the integration determination of the regions is repeated until the regions cannot be integrated by the button regions.

The flowchart of the part shown in FIG. 15 is demonstrated. The character extraction unit 214 performs character extraction on the integrated button area (steps S381 to S386). The character extraction unit 214 extracts characters from the button area, and obtains a recognition rate Q of candidate characters (step S382). Character extraction unit 214, the recognition rate Q of the character is equal to or threshold _{Q th} is larger than (step S 383).

Character extraction unit 214, if the recognition rate Q is determined the threshold Q _th greater in step S 383, a character which is a candidate is a character that is described in the button area, corresponding to the character in the button area Vibration information is assigned (step S384). Character extraction unit 214, if the recognition rate Q is determined the threshold _{Q th} smaller than in step S 383, allocates the vibration information of the generic to the button area (step S385).

  FIG. 16 is a diagram illustrating how a button area is determined in the second embodiment. 16A shows a display screen, FIG. 16B shows a trial point, and FIG. 16C shows the result of button area determination.

  FIG. 16A shows an example in which the display screen is divided into three areas of an upper stage, a middle stage, and a lower stage. In this example, a battery mark, an antenna mark, and the like are displayed on the upper stage. Further, in the middle row, the video screen of the video phone and the other party's video are displayed. The other party's video is constantly updated unless an operation such as pressing the pause button is performed. In the lower row, four operation buttons of off-hook (button B4), on-hook (button B1), high volume (button B2) and low volume (button B3) are displayed.

  FIG. 16B shows a state in which the pseudo pressing unit 204 sets the trial point T on the display unit 34. In this example, the pseudo-pressing unit 204 sets 83 trial points T on the display unit 34.

  FIG. 16C shows a state in which four button areas of the button B5 to the button B8 are determined as a result of the button area determination by the button area determination unit 46. The button B5 is a button area corresponding to off-hook (button B4), the button B6 is a button area corresponding to high volume (button B2), and the button B7 is a button area corresponding to low volume (button B3). The button B8 is a button area corresponding to on-hook (button B1).

  A procedure in which the communication terminal 2 determines a button area and applies a vibration corresponding to a character displayed in the button area in accordance with the strength of touching and pressing will be described with reference to the flowchart shown in FIG.

  The control unit 32 executes the application (step S401). The button area determination unit 46 determines a button area in the display screen of the display unit 34 (step S402). The control unit 32 determines whether or not the application is still being executed (step S403).

  When it is determined in step S403 that the application is continuously executed, the control unit 32 determines whether or not the application execution screen displayed on the display unit 34 has been updated (step S404). If it is determined in step S404 that the application execution screen has been updated, the button area determination unit 46 executes button area determination for the updated execution screen (step S405).

  The control unit 32 determines whether or not the load detection unit 38 has detected a press that exceeds the threshold value (step S406). When the load detection unit 38 detects a press exceeding the threshold value in step S406, the control unit 32 determines whether or not the detected press position is included in the button area (step S407).

If it is determined in step S407 that the pressed position is included in the button area, the character extraction unit 214 extracts characters in the button area and reads vibration information corresponding to the extracted characters from the storage unit. The character extraction unit 214 reads general-purpose vibration information from the storage unit 42 when there is no character that matches the extracted character with a predetermined threshold value _Qth or more (step S408).

  The vibration generating unit 40 applies vibration based on the vibration information (step S409). When the load detection unit 38 does not detect pressing exceeding the threshold value in step S406, the control unit 32 determines whether the touch sensor 36 detects pressing. Here, it is determined that the pressing below the threshold by the load detection unit 38 and the pressing detected by the touch sensor 36 is “contact” (step S410).

When the touch sensor 36 detects contact in step S410, the control unit 32 determines whether or not the detected contact position is included in the button area (step S411). If it is determined in step S411 that the contact position is included in the button area, the character extraction unit 214 extracts characters in the button area and reads vibration information corresponding to the extracted characters from the storage unit. The character extraction unit 214 reads general-purpose vibration information from the storage unit 42 when there is no character that matches the extracted character at a predetermined threshold value _Qth or more (step S412). The vibration generating unit 40 applies vibration based on the vibration information (step S413).

As described above, according to the second embodiment, the communication terminal 2 performs the pseudo time control on the output data between the time T _t and the time T _t + T _s and the pseudo application execution unit 202 by the pseudo time control unit 216. By determining the difference from the pseudo output data, the button area can be determined even for an application in which the application execution screen is updated without the user pressing the display unit 34. For example, the communication terminal 2 according to the second embodiment can determine the button area even in an application such as a shooting game where the screen is constantly updated regardless of the user's operation.

Further, the communication terminal 2 stores the input data between the time T _t and the time T _t + T _{s in} the input data holding unit 220, and gives the input data to the pseudo application execution unit 202 during the button area determination. As a result, even when the input data from the outside is interrupted (for example, freeze or disconnection), the button can be applied to an application such as an application (for example, a TV phone or an IPTV) that changes in the execution screen. Region determination can be performed.

  Although the present invention has been described based on the drawings and examples, it should be noted that those skilled in the art can easily make various modifications and corrections based on the present disclosure. Therefore, it should be noted that these variations and modifications are included in the scope of the present invention. For example, functions included in each member, each means, each step, etc. can be rearranged so as not to be logically contradictory, and a plurality of means, steps, etc. can be combined or divided into one. Is possible.

  In addition, the display units 14 and 34 and the touch sensors 16 and 36 in the description of the above embodiment are integrated by providing both functions of the display units 14 and 34 and the touch sensors 16 and 36 on a common substrate. You may make it comprise with an apparatus. As an example of the configuration of the device in which the functions of the display units 14 and 34 and the touch sensors 16 and 36 are integrated as described above, a plurality of photoelectric elements such as photodiodes are arranged in a matrix electrode array of pixels included in a liquid crystal panel. An example in which conversion elements are regularly mixed can be mentioned. This device displays an image with a liquid crystal panel structure, while reflecting the light of a backlight for liquid crystal display at the tip of a pen that touches and inputs a desired position on the surface of the panel. The reflected light is reflected by peripheral photoelectric conversion elements. The touch position can be detected by receiving the light.

  Further, “pressing” and “pressing” in the present invention may be a case where the pressing load by the user's finger or the like exceeds a predetermined threshold value, or may be a case where the touch is lightly touched. In particular, when an optical touch sensor is used, it is not necessary to press the touch sensor with a finger or the like because the infrared ray on the touch sensor is detected by a finger or a stylus pen.

  In the present invention, the meaning of the technical idea of expression such as detection of pressing “exceeding” the threshold is not necessarily a strict meaning, and includes both meanings including or not including the threshold. And For example, pressing “exceeding the threshold value” may imply not only a case where the pressing is larger than the threshold value but also a case where the pressing value is equal to the threshold value.

DESCRIPTION OF SYMBOLS 1 Communication terminal 2 Communication terminal 12 Control part 14 Display part 16 Touch sensor 18 Load detection part 20 Vibration generation part 22 Storage part 24 Communication part 26 Button area determination part 32 Control part 34 Display part 36 Touch sensor 38 Load detection part 40 Vibration generation Unit 42 storage unit 44 communication unit 46 button area determination unit 102 pseudo application execution unit 104 pseudo press unit 106 pseudo output unit 108 pseudo transmission unit 110 state holding unit 112 area determination unit 114 character extraction unit 202 pseudo application execution unit 204 pseudo press unit 206 Pseudo output unit 208 Pseudo transmission unit 210 State holding unit 212 Area determination unit 214 Character extraction unit 216 Pseudo time control unit 218 Application state holding unit 220 Input data holding unit 222 Output data holding unit

Claims (5)

A display unit;
A touch sensor disposed on the display unit;
A vibration generator for vibrating the touch surface of the touch sensor;
A control unit that executes an application and displays an execution screen of the application on the display unit;
A pseudo application execution unit that executes the same application as the application as a pseudo application;
A button area for determining a button area on the execution screen based on an output change caused by the pseudo pressing of the pseudo application when a plurality of trial points are set on the display unit and the trial points are pseudo pressed. A determination unit,
The control unit vibrates the touch surface by the vibration generation unit when detecting a press on the touch surface corresponding to the button region determined to be a button region by the button region determination unit. Communication terminal. The communication terminal according to claim 1,
The button area determination unit
A state holding unit that stores an output change caused by the pseudo pressing of the pseudo application in association with a trial point at which the output change is obtained;
A communication terminal comprising: an area determination unit that determines an area where the output changes are the same and the positions of the trial points are adjacent to each other as the same button area. The communication terminal according to claim 1,
The button area determination unit
An input data holding unit that stores input data input to the application for a predetermined trial period;
An output data holding unit for storing the output data of the application corresponding to the input data for the trial period for the trial period;
An application state holding unit that stores the state of the application when the input data for the trial period is given as a pseudo application state for the trial period;
A state holding unit that stores an output change caused by the pseudo pressing of the pseudo application in association with a trial point at which the output change is obtained;
An area determination unit that determines that the output change is the same and the position of the trial point is adjacent to the same button area;
With
The controller is
The pseudo application execution unit is executed using the pseudo application state,
At the timing corresponding to the pseudo application state, input data for the trial period is input to the pseudo application being executed in the pseudo application state,
The pseudo-press is executed while the input data is being input to the pseudo application, and an output corresponding to the pseudo-press and an output of a timing corresponding to the output data stored in the output data holding unit And providing the state holding unit with an output change caused by the pseudo pressing of the pseudo application. In the communication terminal in any one of Claims 1-3, Furthermore,
A character extraction unit for extracting characters displayed in the same button area;
A storage unit that stores vibration information corresponding to the characters,
The said control part vibrates the said touch surface by the said vibration generation part using the vibration information corresponding to the said extracted character, The communication terminal characterized by the above-mentioned. The communication terminal according to claim 4, further comprising a load detection unit that determines whether or not pressing on the touch surface corresponds to pressing exceeding a threshold value,
The storage unit stores different vibration information depending on whether pressing on the touch surface exceeds a threshold,
The control unit vibrates the touch surface by the vibration generation unit using corresponding vibration information according to whether or not the pressing on the touch surface is a pressing exceeding a threshold value. .

Images