JP2013041348A - Portable terminal, auxiliary information display program, and auxiliary information display method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To improve operability of a touch operation, even when a display becomes invisible by a hand performing the touch operation, since auxiliary display of the invisible part is appropriately performed.SOLUTION: A cellular phone 10 includes a display 30 provided with a touch panel 38, and the like. When a touch operation is performed, a hand and a finger performing the touch operation is photographed and the touch position is acquired. A processor 24 specifies, on the display 30, a shielded area shielded by the hand and the finger performing the touch operation, on the basis of the photographed image and the touch position. Auxiliary information is extracted from the display content of the specified shielded area, and a display area where the auxiliary information can be displayed is determined. The auxiliary information is thus displayed on the display area determined in this way.

Description

  The present invention relates to a mobile terminal, an auxiliary information display program, and an auxiliary information display method, and more particularly to a mobile terminal, an auxiliary information display program, and an auxiliary information display method having a touch panel.

An example of a portable terminal having a touch panel is disclosed in Patent Document 1. A touch panel is provided on the upper surface of a PDA (Personal Digital Assistant) of Patent Document 1, and a virtual keyboard is displayed on the input device. Further, when an arbitrary key is tapped on the virtual keyboard, the arbitrary key and the adjacent key are enlarged and displayed.
JP 2005-182487 [G06F 3/033, G06F 3/00, G06F 3/023, H03M 11/04]

  However, in the PDA of Patent Document 1, even if the display is enlarged, the display content is hidden by the finger or the hand depending on the finger to be tapped or the tapping position.

  Therefore, a main object of the present invention is to provide a novel portable terminal, auxiliary information display program, and auxiliary information display method.

  Another object of the present invention is to provide a portable terminal, an auxiliary information display program, and an auxiliary information display method that can further improve the operability of the touch operation.

  The present invention employs the following configuration in order to solve the above problems. The reference numerals in parentheses, supplementary explanations, and the like indicate the corresponding relationship with the embodiments described in order to help understanding of the present invention, and do not limit the present invention.

  1st invention is a portable terminal which has a display apparatus provided with the touch panel, the detection part which detects the touch operation with respect to a touch panel, and a touch panel, Comprising: Touch operation detected when the touch operation is detected by the detection part The identifying unit that identifies the shielding area in which the display device is shielded by, the extraction unit that extracts auxiliary information from the display content of the display device based on the shielding region identified by the identifying unit, and the shielding region identified by the identifying unit The mobile terminal includes a determination unit that determines a display area for displaying auxiliary information, and a display unit that displays the extracted auxiliary information in the display area determined by the determination unit.

  In the first invention, in the portable terminal (10: reference numeral exemplifying a corresponding part in the embodiment, the same applies hereinafter), the display device (30) is provided with a touch panel, and a touch operation on the touch panel is detected by a detection unit ( 36). When a touch operation on the touch panel is detected, the specifying unit (24, S11-S15) specifies a shielding area where the display device is shielded by a hand or a finger performing the touch operation, for example. The extraction unit (24, S17) extracts, for example, a character string displayed in the shielding area as auxiliary information. The determination unit (24, S21) determines a display area from the extracted content of the auxiliary information and an area (displayable area) that is not a shielding area. The display unit (24, S23) displays auxiliary information in the determined display area.

  According to the first invention, even if the display becomes invisible by the touch operation, the invisible part is appropriately auxiliary displayed, so that the operability of the touch operation can be further improved.

  A second invention is according to the first invention, further comprising a camera module that outputs an image, wherein the specifying unit is configured to shield the display device by the detected touch operation based on the image output by the camera module. Identify the shielded area.

  In the second invention, the camera module (40-44) includes an image sensor and a focus lens having a wide angle of view. An opening (OP3) leading to the camera module is provided so that a finger performing a touch operation can be photographed. And a specific part specifies the shielding area | region by touch operation based on the output image.

  According to the second invention, if the shielding area is specified only by the captured image, the processing until the shielding area is specified can be simplified.

  A third invention is dependent on the second invention, and the specifying unit has the display device shielded by the detected touch operation based on the touch operation detected by the detection unit and the image output by the camera module. Identify the occluded area.

  In the third aspect of the invention, the left and right hands performing the touch operation are determined based on the image output from the camera module, and the touch position in the touch operation is acquired to identify the shielding area.

  According to the third invention, the specifying unit can specify the shielding area with high accuracy by using the image and the touch operation.

  According to a fourth aspect of the present invention, there is provided a processor (24) of a portable terminal (10) having a touch panel (38), a detection unit (36) for detecting a touch operation on the touch panel, and a display device (30) provided with the touch panel. When the touch operation is detected by the identification unit (S11-S15) for identifying the shielding area where the display device is shielded by the detected touch operation, the display device based on the shielding area identified by the identification unit The extraction unit (S17) for extracting auxiliary information from the display content of the display, the determination unit (S21) for determining the display area for displaying auxiliary information based on the shielding area specified by the specifying unit, and the determination unit The auxiliary information display program causes the display area to function as a display unit (S23) that displays the extracted auxiliary information.

  In the fourth invention as well, as in the first invention, even if the display becomes invisible by the touch operation, the invisible part is appropriately auxiliary displayed, so that the operability of the touch operation can be further improved.

  5th invention is a supplementary information display method of a portable terminal (10) which has a touchscreen (38), the detection part (36) which detects the touch operation with respect to a touchscreen, and the display apparatus (30) provided with a touchscreen, When a touch operation is detected by the detection unit, a shielding area where the display device is shielded by the detected touch operation is identified (S11-S15), and the display content of the display device is based on the identified shielding area. The auxiliary information is extracted from (S17), the display area for displaying the auxiliary information is determined based on the specified shielding area (S21), and the extracted auxiliary information is displayed in the determined display area (S21). S23) is an auxiliary information display method.

  In the fifth invention as well, similar to the first invention, even if the display becomes invisible by the touch operation, the invisible portion is appropriately auxiliary displayed, so that the operability of the touch operation can be further improved.

  According to the present invention, even if the display becomes invisible by the touch operation, the invisible portion is appropriately auxiliary displayed, so that the operability of the touch operation can be further improved.

  The above object, other objects, features, and advantages of the present invention will become more apparent from the following detailed description of embodiments with reference to the drawings.

FIG. 1 is an illustrative view showing an electrical configuration of a mobile phone according to an embodiment of the present invention. FIG. 2 is an illustrative view showing one example of an appearance of the mobile phone shown in FIG. FIG. 3 is an illustrative view showing one example of a plurality of regions provided in the display shown in FIG. 4 is an illustrative view showing one example of a touch operation performed on a plurality of regions shown in FIG. FIG. 5 is an illustrative view showing one example of a configuration of a touch operation table stored in the RAM shown in FIG. 6 is an illustrative view showing another example of a touch operation performed on a plurality of areas shown in FIG. FIG. 7 is an illustrative view showing another example of the configuration of the touch operation table stored in the RAM shown in FIG. FIG. 8 is an illustrative view showing one example of how auxiliary information is displayed on the display shown in FIG. 1. FIG. 9 is an illustrative view showing one example of auxiliary information displayed on the display shown in FIG. FIG. 10 is an illustrative view showing another example of how auxiliary information is displayed on the display shown in FIG. FIG. 11 is an illustrative view showing one example of a memory map of the RAM shown in FIG. FIG. 12 is a flowchart showing an example of auxiliary information display processing of the processor shown in FIG.

  Referring to FIG. 1, a mobile phone 10 of this embodiment is a kind of mobile terminal, and includes a processor 24 called a computer or CPU. The processor 24 includes a wireless communication circuit 14, an A / D converter 16, a D / A converter 20, a key input device 26, a display driver 28, a flash memory 32, a RAM 34, a touch panel control circuit 36, and a camera control circuit 40. The illuminance sensor 46 and the proximity sensor 48 are connected. An antenna 12 is connected to the wireless communication circuit 14, a microphone 18 is connected to the A / D converter 16, and a speaker 22 is connected to the D / A converter 20. A display 30 is connected to the display driver 28. Further, a touch panel 38 is connected to the touch panel control circuit 36. An image sensor 42 and a focus lens 44 are connected to the camera control circuit 40.

  The processor 24 manages the overall control of the mobile phone 10. The RAM 34 is used as a work area (including a drawing area) or a buffer area of the processor 24. Content data such as characters, images, sounds, sounds and videos of the mobile phone 10 is recorded in the flash memory 32.

  The A / D converter 16 converts an analog audio signal for audio or sound input through the microphone 18 into a digital audio signal. The D / A converter 20 converts (decodes) the digital audio signal into an analog audio signal, and supplies the analog audio signal to the speaker 22 via an amplifier (not shown). Therefore, sound or sound corresponding to the analog sound signal is output from the speaker 22. The processor 24 can adjust the volume of the sound output from the speaker 22 by controlling the amplification factor of the amplifier.

  The key input device 26 includes a menu key, a call key, a call end key, and the like. Then, information (key data) on the keys operated by the user is input to the processor 24. Further, when each key included in the key input device 26 is operated, a click sound is generated. Therefore, the user can obtain an operational feeling for the key operation by listening to the click sound.

  The display driver 28 controls display on the display 30 which is a kind of display device under the instruction of the processor 24. The display driver 28 also includes a video memory (not shown) that temporarily stores image data to be displayed.

  The display 30 is provided with a backlight using LEDs as light sources. Then, the processor 24 adjusts the backlight based on the output of the illuminance sensor 46. The illuminance sensor 46 includes photodiodes and the like integrated in an array, and detects the illuminance around the mobile phone 10.

  The touch panel 38 is a capacitance method that detects a change in capacitance between electrodes caused by an object such as a finger approaching the surface. For example, it detects that one or more fingers touch the touch panel 38. To do. The touch panel 38 is provided on the display 30 and is a pointing device for instructing an arbitrary position within the screen. The touch panel control circuit 36 that functions as a detection unit detects a touch operation that is pressed, stroked, or touched within the effective touch range of the touch panel 38, and sends coordinate data indicating the position of the touch operation to the processor 24. Output. That is, the user can input an operation direction, a figure, or the like to the mobile phone 10 by pressing, stroking, or touching the surface of the touch panel 38 with a finger.

  Here, an operation in which the user touches the upper surface of the touch panel 38 with a finger is referred to as “touch”. On the other hand, an operation of releasing a finger from the touch panel 38 is referred to as “release”. Then, an operation in which the user touches the upper surface of the touch panel 38 and continuously releases is referred to as “touch and release”.

  Further, an operation of stroking the surface of the touch panel 38 is referred to as “slide”, and an operation of sliding after touching and releasing is referred to as “touch slide”. Furthermore, the operation of performing touch and release twice in succession is referred to as “double tap”, and the operation of touching two locations substantially simultaneously is referred to as “multi-touch”. The “touch operation” includes operations performed on the touch panel 38 such as touch, release, touch and release, slide, touch slide, double tap, and multi-touch.

  The coordinates indicated by the touch are referred to as “touch points” (touch start positions), and the coordinates indicated by the releases are referred to as “release points” (touch end positions).

  Note that the touch operation is not limited to a finger, and may be performed with a touch pen with a conductor attached to the tip. Further, as a detection method of the touch panel 38, a surface-type capacitance method may be employed, or a resistance film method, an ultrasonic method, an infrared method, an electromagnetic induction method, or the like may be used.

  The camera control circuit 40 is a circuit for taking a still image or a moving image with the mobile phone 10. For example, when an operation for executing a camera function is performed on the key input device 26, the processor 24 executes the camera function and activates the camera control circuit 40. The camera control circuit 40, the image sensor 42, and the focus lens 44 collectively function as a camera module.

  In the imaging area of the image sensor 42, a light receiving element corresponding to QVGA (320 × 240 pixels) is displayed. For this reason, when the image sensor 42 is irradiated with the optical image of the subject, a charge corresponding to the optical image of the subject, that is, a QVGA raw image signal is generated by photoelectric conversion in the imaging area.

  When the camera function is executed, the processor 24 activates an image sensor driver built in the camera control circuit 40 in order to display a real-time moving image of the subject, that is, a through image (preview image) on the display 30, and performs an exposure operation and The image sensor driver is instructed to perform a charge readout operation corresponding to the designated readout region.

  The image sensor driver performs exposure of the imaging surface of the image sensor 42 and reading of the charges generated by the exposure. As a result, a raw image signal is output from the image sensor 42. The output raw image signal is input to the camera control circuit 40. The camera control circuit 40 performs processes such as color separation, white balance adjustment, and YUV conversion on the input raw image signal, and outputs the YUV format. Generate image data. Then, the YUV format image data is input to the processor 24.

  The YUV format image data input to the processor 24 is stored (temporarily stored) in the RAM 34 by the processor 24. Further, the stored YUV format image data is converted into RGB format image data by the processor 24, and then supplied from the RAM 34 to the display driver 28. Then, RGB format image data is output to the display 30. As a result, a through image of QVGA representing the subject is displayed on the display 30.

  Further, the camera control circuit 40 calculates a focus evaluation value from the raw image signal and outputs the focus evaluation value to the processor 24. The processor 24 executes AF (Autofocus) processing based on the focus evaluation value output from the camera control circuit 40. When the AF process is executed, the camera control circuit 40 adjusts the lens position of the focus lens 44 under the instruction of the processor 24. As a result, a through image in which the subject is in focus is displayed on the display 30.

  Here, when a still image shooting operation is performed on the key input device 26, the processor 24 executes a still image main shooting process. That is, the processor 24 performs signal processing on the QVGA raw image signal output from the image sensor 42, temporarily stores it in the RAM 34, and executes recording processing on the flash memory 32. When the recording process is executed, image data is read from the RAM 34 through the processor 24. Then, the processor 24 associates the meta information with the read image data and stores it in the flash memory 32 as one image file. Further, the processor 24 outputs a sound for notifying that the main photographing process is being executed from a speaker (not shown).

  The meta information includes time information, image size, and model name (model number) of the mobile phone 10. When a memory card can be connected to the mobile phone 10, the image file may be stored in the memory card. Also, meta information associated with image data is stored in Exif format.

  The camera module is also used when a videophone function described later is executed. An opening OP3 (see FIG. 2) leading to the camera module is provided on the same surface as the display 30 so that the camera module can be used for the videophone function. Therefore, the camera module is sometimes called an in-camera.

  Although not shown, the proximity sensor 48 includes an infrared LED as a light emitting element and a photodiode as a light receiving element. The processor 24 calculates the distance of an object (for example, a user's face) close to the proximity sensor 48 (the mobile phone 10) from the change in the output of the photodiode. Specifically, the infrared LED emits infrared rays, and the photodiode receives infrared rays reflected from a face or the like. For example, when the photodiode is far from the user's face, the infrared light emitted by the infrared LED is not received by the photodiode, but when the user's face comes close to the proximity sensor 48, the infrared light emitted by the infrared LED is Reflected on the face and received by the photodiode. As described above, since the amount of received infrared light varies depending on whether the proximity sensor 48 is close to the user's face or not, the processor 24 uses the proximity sensor 48 based on the received light amount. The distance from the object to the object can be calculated.

  The wireless communication circuit 14 is a circuit for performing wireless communication by the CDMA method. For example, when the user instructs a telephone call (calling) using the key input device 26, the wireless communication circuit 14 executes a telephone call process under the instruction of the processor 24, and sends a telephone call signal via the antenna 12. Is output. The telephone call signal is transmitted to the other party's telephone through a base station and a communication network (not shown). Then, when an incoming call process is performed at the other party's telephone, a communicable state is established, and the processor 24 executes a call process.

  The normal call processing will be described in detail. The modulated audio signal transmitted from the other party's telephone is received by the antenna 12. The received modulated audio signal is demodulated and decoded by the wireless communication circuit 14. The received voice signal obtained by these processes is converted into an analog voice signal by the D / A converter 20 and then output from the speaker 22. On the other hand, the transmission voice signal captured through the microphone 18 is converted into a digital voice signal by the A / D converter 16 and then given to the processor 24. The transmission signal converted into the digital audio signal is subjected to encoding processing and modulation processing by the wireless communication circuit 14 under the instruction of the processor 24, and is output via the antenna 12. Therefore, the modulated voice signal is transmitted to the other party's telephone through the base station and the communication network.

  When a telephone call signal from the other party's telephone is received by the antenna 12, the radio communication circuit 14 notifies the processor 24 of an incoming call (incoming call). In response to this, the processor 24 controls the display driver 28 to display caller information (such as a telephone number) described in the incoming call notification on the display 30. At substantially the same time, the processor 24 outputs a ring tone (sometimes called a ringing melody or a ringing voice) from a speaker (not shown).

  When the user performs a response operation using the call key, the wireless communication circuit 14 executes a telephone call incoming process under the instruction of the processor 24. Further, the communicable state is established, and the processor 24 executes the normal call process described above.

  Further, when a call end operation is performed with the call end key after shifting to the call ready state, the processor 24 controls the wireless communication circuit 14 to transmit a call end signal to the other party. Then, after transmitting the call end signal, the processor 24 ends the call process. The processor 24 also ends the call process when a call end signal is received from the other party first. Furthermore, the processor 24 also ends the call process when a call end signal is received from the mobile communication network regardless of the call partner.

  In this embodiment, a hands-free call can also be performed. As described above, the videophone function can be executed by using the in-camera and the hands-free call.

  Note that the mobile phone 10 can execute the browser function by establishing communication with a network (not shown). The mobile phone 10 can also execute a memo pad function for creating and editing a document. Furthermore, with these functions, characters can be input using the virtual keyboard 66 (see FIG. 8A) having a QWERTY layout.

  FIG. 2 is an external view showing the external appearance of the surface of the mobile phone 10. Referring to FIG. 2, the mobile phone 10 has a straight shape and has a flat rectangular casing C. A microphone 18 (not shown) is built in the housing C, and an opening OP2 leading to the built-in microphone 18 is provided on one surface in the longitudinal direction of the housing C. Similarly, the speaker 22 (not shown) is built in the housing C, and the opening OP1 leading to the built-in speaker 22 is provided on the other surface in the longitudinal direction of the housing C.

  Each key included in the key input device 26 is provided on the surface of the casing C. The display 30 is attached so that the display surface (monitor screen) can be seen from the surface of the housing C. A touch panel 38 is provided on the display surface of the display 30.

  Further, the camera module is built in the housing C, and the opening OP3 leading to the image sensor 42 of the built-in camera module is provided on the upper left of the surface of the housing C. The illuminance sensor 44 and the proximity sensor 46 are provided at the upper right corner of the surface of the housing C so that the sensor surface is exposed. In the present embodiment, the illuminance sensor 44 and the proximity sensor 46 are provided adjacent to each other.

  For example, the user inputs a telephone number by performing a touch operation on a dial key displayed on the display 30, and performs a voice call operation using the call key. When the user ends the call, the user performs a call end operation using the end call key. Further, when the user performs the videophone function, the user performs a hands-free call while looking at the surface of the housing C, that is, the display 30 and the opening OP3.

  Further, the user operates the menu key to display a menu screen on the display 30. Further, the user performs menu selection and confirmation by touching the soft keys and menus displayed on the display 30.

  The camera module has a wide angle of view with respect to the center side (touch panel side) of the housing C so that the user's face is within the angle of view when the videophone function is executed. Has been.

  The antenna 12, wireless communication circuit 14, A / D 16, D / A 20, processor 24, display driver 28, flash memory 32, RAM 34, touch panel control circuit 36, camera control circuit 40, image sensor 42, and focus lens 44 are provided in a housing. Since it is built in the body C, it is not shown in FIG.

  Here, in the present embodiment, when a touch operation is performed, an area (shielded area) where the display of the display 30 is shielded by the touch operation is specified, and display contents (such as a character string) included in the shielded area are assisted. Extracted as information. The extracted auxiliary information is displayed in a display area determined based on the shielding area. Specific processing will be described below.

  First, referring to FIG. 3, in the display range of display 30, upper left is set as the first area, upper right is set as the second area, lower left is set as the third area, and lower right is set as the fourth area. In the present embodiment, the shielding area is specified based on the touch operation on these areas, and the display area for displaying the auxiliary information is determined.

  Next, when a touch operation is performed, the shielding area is specified based on the hand (left hand, right hand) that performed the touch operation and the area including the touch point in the touch operation. In this embodiment, the hand that has performed the touch operation is determined using pattern matching processing and skin color region extraction processing.

  As described above, the opening OP3 leading to the camera module is provided at the upper left of the surface of the housing C, and the camera module has a wide angle of view with respect to the touch panel side. That is, a finger or hand performing a touch operation can be photographed by the camera module. Therefore, it is possible to determine the right hand and the left hand that perform the touch operation by using the pattern matching process capable of recognizing the left hand and the right hand.

  Further, since the attachment position of the opening OP3 is deviated from the center of the housing C, a finger or a hand touching the touch panel 38 is reflected in an image differently between the right hand and the left hand. That is, when a finger performing a touch operation is photographed, the amount of the hand and the finger appearing is different. Therefore, it is also possible to determine the right hand and the left hand that perform the touch operation based on the size (amount) of the skin color area in the photographed image.

  In this embodiment, the determination result of the pattern matching process and the determination result based on the size of the skin color area are combined to determine the hand performing the touch operation. Specifically, in the turn matching process, a ratio (matching rate) that matches the left-hand pattern and a ratio that matches the right-hand pattern are calculated. At this time, if the difference between the match rates is equal to or greater than the threshold, the left and right hands are determined based on the match rate. On the other hand, if the difference in the match rate is less than the threshold value, the left and right hands are determined based on the size of the skin color area. In other embodiments, the determination of the skin color area may be given priority over the determination of pattern matching.

  Next, when the right and left sides of the hand performing the touch operation are determined, the shielding area is specified based on the touch position and the touch operation table. The touch operation table is a table in which a shielding area and a displayable area are registered in advance for an area including a touch position.

  For example, as shown in FIG. 4A, when the first area is touched with the left hand, the first area and the third area are touched by the hand performing the touch operation according to the first touch operation table for the left hand shown in FIG. It is specified as a shielding area to be shielded, and the second area and the fourth area are specified as displayable areas in which auxiliary information can be displayed. Further, as shown in FIG. 4B, when the second area is touched with the left hand, the second area, the third area, and the fourth area are specified as the shielding areas according to the first touch operation table of FIG. The first area is specified as the displayable area. Further, as shown in FIG. 4C, when the third area is touched with the left hand, the third area is specified as the shielding area according to the first touch operation table of FIG. 5, and the first area, the second area, and The fourth area is specified as the displayable area. Then, as shown in FIG. 4D, when the fourth area is touched with the left hand, the third area and the fourth area are specified as the shielding area according to the first touch operation table of FIG. The second area is specified as the displayable area.

  On the other hand, when the first area is touched with the right hand as shown in FIG. 6A, the first area, the third area, and the fourth area are occluded according to the second touch operation table for the right hand shown in FIG. And the second area is specified as the displayable area. Further, as shown in FIG. 6B, when the second area is touched with the right hand, the second area and the fourth area are specified as the shielding areas according to the second touch operation table, and the first area and the third area are specified. Is specified as a displayable area. Further, as shown in FIG. 6C, when the third area is touched with the right hand, the third area and the fourth area are specified as shielding areas, and the first area and the second area are specified as displayable areas. The Then, as shown in FIG. 6D, when the fourth area is touched with the right hand, the fourth area is specified as a shielding area, and the first area, the second area, and the third area are specified as displayable areas. The

  And the display content (for example, character string) of a shielding area is extracted as auxiliary information, and a display area is determined from the information amount. For example, if the auxiliary information is a character string and the number of characters fits in one displayable area, the displayable area with the smallest number is determined as the display area among the specified displayable areas. If the number of characters does not fit in one displayable area, among the specified displayable areas, the displayable area with the smallest number and the displayable area with the next smallest number are determined as display areas. However, when two or more displayable areas are not specified, the character size (font size) is changed so as to fit in one displayable area.

  Next, display of auxiliary information will be described with a specific example. For example, in the browser function, when a search character string is input at a certain search site, a virtual keyboard is displayed on the display 30. Referring to FIG. 8A, the display range of display 30 in a state where the browser function is executed includes a state display area 60 and a function display area 62. In the status display area 60, an icon (sometimes referred to as a pictograph) indicating the radio wave reception status by the antenna 12, an icon indicating the remaining battery capacity of the secondary battery, and the date and time are displayed.

  The function display area 62 displays data obtained from the server. Further, when a touch operation is performed on an input area where characters can be input, a soft key display area 64 including a soft key 66 having a QWERTY layout is displayed, and a cursor CU indicating a current input position (editing position) is displayed. Displayed in the input area.

  In this state, as shown in FIG. 8B, when the character key (E) in the third area is touched with the right hand, the third area and the fourth area including the virtual keyboard 66 are specified as the shielding area. . In addition, characters corresponding to each character key included in the virtual keyboard 66 are extracted as auxiliary information. Since the plurality of characters (auxiliary information) extracted in this way fit in one displayable area, the first area is determined as the display area. As a result, a character string indicating character keys of the virtual keyboard 66 is displayed in the first area. Further, in the auxiliary display shown in FIG. 8B, the character (E) corresponding to the touched character key is highlighted.

  As shown in FIG. 9, conventionally, depending on the touch position, the character string being input may be hidden by the auxiliary display. However, when auxiliary display is performed at a position different from the touch position as in the present embodiment, auxiliary information is displayed in the displayable area, so that the character string being input is not hidden by the auxiliary display. Therefore, in another embodiment, the display area may be determined after further determining whether the input area is hidden by the auxiliary display.

  In other embodiments, only the character (E) indicating the character key on which the touch operation is performed may be auxiliary displayed.

  Next, as shown in FIG. 10A, when searching for a character string from a document in the memo pad function, the display 30 has a sentence (character string) to be searched and a character string to be searched (searched character). An input area for inputting, and a search key for executing search processing are displayed. In this case, when a search character (“QWERTY”) is input and then the search key is touched, the corresponding character string is highlighted in the sentence. However, when the search key is touched with the left hand, as shown in FIG. 10B, the text included in the shielding area (the third area and the fourth area) is hidden with the left hand. Therefore, a character string (sentence) including the search result is extracted as auxiliary information. Further, since the text extracted as auxiliary information does not fit in one displayable area, the first area and the second area are determined as display areas. As a result, the text extracted for the first area and the second area is auxiliary displayed. Further, in the auxiliary display, the character string “QWERTY” as the search result is highlighted.

  Thus, even if the display becomes invisible by the touch operation, the invisible portion is appropriately auxiliary displayed, so that the operability of the touch operation can be further improved. Further, in this embodiment, the shielding area can be specified with high accuracy by using the captured image and the touch operation.

  However, in another embodiment, the shielding area may be specified only by the captured image by adjusting the angle of view of the camera module. In this case, since the shielding area can be specified only by the captured image, the process until the shielding area is specified can be simplified.

  FIG. 11 is a diagram showing a memory map of the RAM 34. The RAM 34 includes a program storage area 302 and a data storage area 304. A part of the program and data is read from the flash memory 32 all at once or partially and sequentially as necessary and stored in the RAM 34.

  The program storage area 302 stores a program for operating the mobile phone 10. For example, a program for operating the mobile phone 10 includes an auxiliary information display program 310. The auxiliary information display program 310 is a program for determining a position for displaying auxiliary information in accordance with a shielding area by a touch operation.

  Although not shown, the program for operating the mobile phone 10 includes a program for searching for broadcast waves.

  Subsequently, the data storage area 304 is provided with a touch buffer 330, an image buffer 332, an auxiliary information buffer 334, a display buffer 336, and a display coordinate buffer 338, and touch operation table data 340, touch coordinate map data 342, GUI data. 344 and text data 346 are stored. Further, the data storage area 304 is also provided with a touch flag 348 and a left / right flag 350.

  The touch buffer 330 temporarily stores the coordinates of the touch point, touch position, and release point output from the touch panel control circuit 36. The image buffer 332 temporarily stores an image output from the camera module. The auxiliary information buffer 334 temporarily stores the extracted auxiliary information. The display buffer 336 temporarily stores character strings and images displayed on the display 30. Display coordinates buffer 338 temporarily stores display coordinates such as character strings and images stored in display buffer 336.

  The touch coordinate map data 340 is data for associating the touch coordinates in the touch operation with the display coordinates of the display 30. That is, the processor 24 associates the result of the touch operation performed on the touch panel 38 with the display on the display 30 based on the touch coordinate map data 340.

  The touch operation table data 342 is data including a first touch operation table for the left hand shown in FIG. 5 and a second touch operation table for the right hand shown in FIG. The GUI data 344 is data for displaying a GUI such as the virtual keyboard 66 shown in FIG. Further, the text data 346 is data corresponding to the text shown in FIG. When the virtual keyboard 66 or text is displayed, GUI data 344 or text data 346 is stored in the display buffer 336.

  The touch flag 348 is a flag for determining whether or not a touch operation is performed on the touch panel 38. For example, the touch flag 348 is composed of a 1-bit register. When the touch flag 348 is turned on (established), a data value “1” is set in the register. On the other hand, when the touch flag 348 is turned off (not established), a data value “0” is set in the register.

  The left / right flag 350 is a flag indicating whether the hand performing the touch operation is the left hand or the right hand. The left / right flag 350 is turned on when the left hand is determined, and is turned off when the right hand is determined.

  Although not shown, the data storage area 304 stores image data displayed in a standby state, character string data, and the like, and is provided with counters and flags necessary for the operation of the mobile phone 10. It is done.

  The processor 24 performs a plurality of tasks in parallel under the control of the Linux (registered trademark) such as Android (registered trademark) and REX, and the auxiliary information display processing shown in FIG. 12 under the control of the other OS. Process.

  FIG. 12 is a flowchart of the auxiliary information display process. For example, when a function (application) for displaying a document or an image is executed on the mobile phone 10, an auxiliary information display process is executed. Then, in step S1, the processor 24 determines whether or not it has been touched. That is, it is determined whether the touch flag 348 is turned on. If “NO” in the step S1, that is, if it is not touched, it is determined whether or not an end operation is performed in a step S3. For example, it is determined whether or not the end call key has been operated to end the function being executed. If “YES” in the step S3, that is, if an operation for ending the function being executed is performed, the auxiliary information display process is ended. On the other hand, if “NO” in the step S3, that is, if the ending operation is not performed, the process returns to the step S1.

  If “YES” in the step S1, for example, if the user touches the touch panel, an image is taken in a step S5. That is, a shooting command is issued to the camera module. Further, the processor 24 stores the captured image in the image buffer 332. Subsequently, in step S7, image analysis processing is executed. For example, pattern matching processing and skin color region extraction processing are performed on the captured image stored in the image buffer 332. Subsequently, in step S9, it is determined whether or not a hand or a finger is included. That is, it is determined from the pattern matching result and the amount of the skin color area whether the hand is reflected in the photographed image.

  If “NO” in the step S9, the touch operation is performed with a touch pen, for example, and if the hand or the finger is not photographed, the process proceeds to the step S17. If “YES” in the step S9, for example, if a finger performing a touch operation is reflected on the photographed image, a left / right determination process is performed in a step S11. That is, it is determined whether the hand in the captured image is the left hand or the right hand from the pattern matching result and the amount of the skin color area. For example, as a result of pattern matching, if the difference between the matching rate with the left hand pattern and the matching rate with the right hand pattern is above a threshold value, the result with the higher matching rate is adopted. However, if the difference is less than the threshold value, right and left are determined based on the amount of the skin color area. Then, on / off of the left / right flag 350 is switched based on the left / right determination result.

  Subsequently, in step S13, the touch position is acquired. That is, the touch position is read from the touch buffer 330. Subsequently, in step S15, a shielding area is specified. For example, if the left / right flag 350 is on (left hand) and the touch position acquired from the touch buffer 330 is included in the fourth area, the shielding is performed based on the touch operation table data 342 (here, FIG. 5). Regions are identified as the third region and the fourth region. The processor 24 that executes the processes of steps S11 to S15 functions as a specifying unit.

  Subsequently, in step S17, auxiliary information is extracted. That is, of the character string stored in the display buffer 336 and the image, data included in the shielding area based on the display coordinate buffer 338 is read as auxiliary information. For example, as shown in FIG. 8B, if the GUI of the virtual keyboard 66 is included in the shielding area, a character string indicating the character key included in the virtual keyboard 66 is extracted from the display buffer 336 as auxiliary information. . As shown in FIG. 10B, if a sentence is displayed in the shielding area, a character string corresponding to the sentence is extracted from the display buffer 336 as auxiliary information. The extracted auxiliary information is stored in the auxiliary information buffer 334. However, if no character string, image, or the like is displayed in the shielding area, auxiliary information is not extracted in step S17. The processor 24 that executes the process of step S17 functions as an extraction unit.

  Subsequently, in step S19, it is determined whether or not auxiliary information has been extracted. That is, it is determined whether or not auxiliary information is stored in the auxiliary information buffer 334 as a result of executing the process of extracting auxiliary information in the process of step S17. If “NO” in the step S19, that is, if the auxiliary information is not extracted, the process returns to the step S1. On the other hand, if “YES” in the step S19, that is, if auxiliary information is extracted, the display area is determined in a step S21. That is, a displayable area necessary for displaying the stored auxiliary information is determined, and the display area is determined. For example, as shown in FIG. 10B, when the third area and the fourth area are specified as the shielding area and the text displayed in the shielding area is extracted as the auxiliary information, the first area and the second area An area is determined as a display area. In step S23, auxiliary information is displayed, and the process returns to step S1. That is, the auxiliary information stored in the auxiliary information buffer 334 is displayed in the display area thus determined.

  The processor 24 that executes the process of step S21 functions as a determination unit, and the processor 24 that executes the process of step S23 functions as a display unit.

  Further, when the shielding area is specified only by the image, the processes of steps S11 and S13 are omitted in the auxiliary information display process.

  Further, the auxiliary information to be displayed may be not only a character string but also an image or the like. Further, when displaying the auxiliary information, the auxiliary information may be enlarged and displayed. The function for displaying the auxiliary information is not limited to the browser function and the memo pad function, but may be other functions such as a mail function and an address book function.

  In another embodiment, the opening OP3 leading to the in-camera may be provided on the lower right side or the lower left side of the housing C in order to facilitate the extraction of the amount of the skin color area.

  In addition, when the output result of the illuminance sensor 46 or the proximity sensor 48 varies depending on the difference between the left and right hands performing the touch operation, the sensing results of the illuminance sensor 46 and the proximity sensor 48 are used for the left / right determination of the hand or finger performing the touch operation. May be.

  In other embodiments, the number of display ranges on the display 30 is not limited to four as shown in FIG. 3, but may be five or more. Moreover, when the shielding area shielded by the hand or the finger can be specified with high accuracy, the shape of the shielding area may be a shape like a hand or a finger.

  The communication system of the mobile phone 10 is a CDMA system, but an LTE (Long Term Evolution) system, a W-CDMA system, a GSM system, a TDMA system, an FDMA system, a PHS system, and the like may be employed.

  Further, the plurality of programs used in this embodiment may be stored in the HDD of the data distribution server and distributed to the mobile phone 10 via the network. Further, the storage medium may be sold or distributed in a state where a plurality of programs are stored in a storage medium such as an optical disk such as a CD, DVD, or BD (Blu-ray Disc), a USB memory, or a memory card. When a plurality of programs downloaded through the above-described server or storage medium are installed in a mobile phone having the same configuration as that of this embodiment, the same effect as that of this embodiment can be obtained.

  Further, the present embodiment is not limited to the mobile phone 10 but may be applied to so-called smartphones, tablet PCs, PDAs, and the like.

  The specific numerical values such as pixels mentioned in this specification are merely examples, and can be appropriately changed according to a change in product specifications.

DESCRIPTION OF SYMBOLS 10 ... Mobile phone 12 ... Antenna 14 ... Wireless communication circuit 24 ... Processor 26 ... Key input device 30 ... Display 34 ... RAM
36 ... Touch panel control circuit 38 ... Touch panel 40 ... Camera control circuit 42 ... Image sensor 44 ... Focus lens 46 ... Illuminance sensor 48 ... Proximity sensor

Claims (5)

A mobile terminal having a touch panel, a detection unit for detecting a touch operation on the touch panel, and a display device provided with the touch panel,
When a touch operation is detected by the detection unit, a specifying unit that specifies a shielding area where the display device is shielded by the detected touch operation;
An extraction unit for extracting auxiliary information from the display content of the display device based on the shielding area specified by the specification unit;
A determining unit that determines a display region for displaying the auxiliary information based on the shielding region specified by the specifying unit, and a display unit that displays the extracted auxiliary information in the display region determined by the determining unit A portable terminal. A camera module for outputting images;
The mobile terminal according to claim 1, wherein the specifying unit specifies a shielding area where the display device is shielded by a detected touch operation based on an image output from the camera module.   The said specific | specification part specifies the shielding area | region where the said display apparatus is shielded by the detected touch operation based on the touch operation detected by the said detection part, and the image which the said camera module outputs. Mobile devices. A processor of a portable terminal having a touch panel, a detection unit that detects a touch operation on the touch panel, and a display device provided with the touch panel.
When a touch operation is detected by the detection unit, a specifying unit that specifies a shielding area where the display device is shielded by the detected touch operation;
An extraction unit for extracting auxiliary information from the display content of the display device based on the shielding area specified by the specification unit;
A determining unit that determines a display region for displaying the auxiliary information based on the shielding region specified by the specifying unit, and a display unit that displays the extracted auxiliary information in the display region determined by the determining unit An auxiliary information display program that functions as An auxiliary information display method for a mobile terminal, comprising: a touch panel, a detection unit that detects a touch operation on the touch panel, and a display device provided with the touch panel.
When a touch operation is detected by the detection unit, a shielding area where the display device is shielded by the detected touch operation is specified,
Extracting auxiliary information from the display content of the display device based on the identified shielding area,
An auxiliary information display method for determining a display area for displaying the auxiliary information based on the identified shielding area, and displaying the extracted auxiliary information in the determined display area.

Images