JP2005216170A - Mobile terminal device and method for processing input to information processor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a mobile terminal device that enables an interface for an input process to be constructed from certain images on a display screen. <P>SOLUTION: A display part 15 displays images. A control part 20 extracts a plurality of objects by extracting the contours of the images displayed, and allocates the plurality of keys of a key input part 13 to the objects extracted. <P>COPYRIGHT: (C)2005,JPO&NCIPI

Description

  The present invention relates to a portable terminal device capable of displaying an image, such as a cellular phone, and an input processing method for an information processing device.

2. Description of the Related Art Conventionally, a mobile terminal device represented by a mobile phone is provided with a plurality of keys, and a user can select and execute various functions of the mobile phone by performing key operations on the plurality of keys. It is configured to be able to.
Such key operations have recently become more complex as the functions of mobile phones have increased, and various techniques have been proposed for an input method that allows a user to quickly find a function that the user wants to select.
For example, Patent Document 1 below discloses a technique that makes functions easy to recognize for the user by displaying the functions as graphics icons and displaying them hierarchically.

Japanese Patent Application Laid-Open No. 11-237942

  However, due to the recent progress in miniaturization of mobile terminal devices, there is a physical limit to increasing the size of the display screen, which makes it easy to understand icons whose functions are made into figures, and affects other display objects. Therefore, there is a demand for a technique for displaying the image so as not to give as much as possible.

  The present invention has been made in view of such circumstances, and an object thereof is to provide a portable terminal device and an information processing device input processing method capable of constructing an input processing interface from an arbitrary image on a display screen. is there.

  In order to achieve the above object, a first aspect of the present invention provides a display unit that displays at least an image, an operation unit, and a predetermined image with respect to the operation unit while displaying the predetermined image on the display unit. When an operation is performed, an object is extracted from the predetermined image, one function is associated with the extracted object, and when the object is designated by the operation means, the object is associated with the designated object. A portable terminal device having a control means for executing a given function.

  Preferably, the object is associated with a designated area including a display area of the object, and the control unit associates the designated area with the designated area when the designated means is designated by the operation means. It is determined that the object is specified.

  Preferably, the control means calculates a first rectangle that is the largest rectangle that can be accommodated in the display area of the object, and a second rectangle that is the smallest rectangle that can accommodate the display area of the object. A region in a third rectangle that is a rectangle having a size intermediate between the first rectangle and the second rectangle is associated with the object as a designated region.

  Preferably, the control unit displays a predetermined number of maximum display areas that can be displayed by the display unit when a predetermined operation is performed on the operation unit when a predetermined image is displayed on the display unit. Dividing into a plurality of regions, an object is extracted for each of the plurality of regions.

  In order to achieve the above object, a second aspect of the present invention is an input processing method of an information processing apparatus having a display unit and an operation unit, and a step of displaying a predetermined image on the display unit; Performing a predetermined operation on the operation unit, extracting an object from the predetermined image according to the predetermined operation, associating one function with the extracted object, and the operation unit The step of designating an object by the above and the step of executing a function associated with the designated object.

  According to the present invention, an interface for input processing can be constructed based on an arbitrary image on a display screen, and there is an advantage that operational comfort is improved for the user.

DESCRIPTION OF EMBODIMENTS Embodiments of a mobile terminal device according to the present invention will be described below with reference to the accompanying drawings.
FIG. 1 is a block diagram illustrating an example of a system configuration of a mobile phone 1 as a mobile terminal device of the present invention. FIG. 2 is a diagram illustrating an example of the appearance of the mobile phone 1.

As shown in FIG. 1, the mobile phone 1 includes a communication processing unit 11 including a transmission / reception antenna 111, a memory 12, a key input unit 13 as an operation unit, an audio processing unit 14 including a speaker 141 and a microphone 142, A display unit 15 as a display unit, a panel input unit 16, and a control unit 20 as a control unit are provided.
Hereafter, each component which comprises the mobile telephone 1, and the processing content which the mobile telephone 1 performs are described.

The communication processing unit 11 performs a wireless communication operation via the base station, for example, a telephone number calling operation or an e-mail transmission / reception operation.
The communication processing unit 11 is configured to include a transmission / reception antenna 111. In order to perform wireless communication using radio waves, the communication processing unit 11 modulates voice information, e-mail, and the like processed by the control unit 20, and a base (not shown) by the transmission / reception antenna 111. Send to the station.
Further, the communication processing unit 11 demodulates various information such as e-mail and voice information transmitted from the base station by radio and received by the transmission / reception antenna 111 and outputs the demodulated information to the control unit 20. For example, image data to be displayed on the display unit 15 to be described later may be acquired via wireless communication by the communication processing unit 11.
The transmission / reception antenna 111 is built in the main body of the mobile phone 1 and is not exposed to the outside in FIG.

The memory 12 includes a non-volatile memory such as an EEPROM, and stores a control program for transmitting and receiving calls and mails, an Internet browser, message data, an address book in which names and telephone numbers are registered.
Further, the memory 12 stores image data that is the basis of an object extraction process described later. The mobile phone 1 acquires the image data by wireless communication via the communication processing unit 11 and stores it in the memory 12, for example.
The memory 12 stores data related to the object extracted by the control unit 20. In the data related to the object, there are object contour position information, a designated area, and an assigned function, which will be described later, for each object.

The key input unit 13 has an end (end call) / power key, a start (call) key, a plurality of keys corresponding to numbers, and the like. When these keys are operated by the user, Input information is output to the control unit 20.
As shown in FIG. 2, the key input unit 13 arranges the plurality of keys on a 4 × 3 matrix.
In the present embodiment, when a user presses (operates) a predetermined key in the key input unit 13, an object extraction process, an allocation process, and the like, which will be described later, are started.

The audio processing unit 14 includes an audio processing circuit, and is connected to a speaker 141 that outputs audio for a call function and a microphone 142 that inputs audio.
The sound processing unit 14 performs predetermined processing on the sound collected by the microphone 142 and supplies the processed sound to the control unit 20.
Further, the audio processing unit 14 performs predetermined processing on the audio information supplied from the control unit 20 and outputs the audio information from the speaker 141.

  The display unit 15 includes a display device such as a liquid crystal display (LCD) and is configured to be able to display image data stored in the memory 12.

The panel input unit 16 is a so-called touch panel type input unit, and detects a position touched by the user by touching (touching) the LCD or the like of the display unit 15 and outputs the detected position to the control unit 20 as user input information. To do.
There are various touch-panel input methods, such as a capacitance method that detects the change in the surface charge of the display panel and a resistive film method that detects the position using the potential of the film surface attached to the display panel surface. It can be applied to the panel input unit 16.

  The control unit 20 is composed mainly of a microcomputer and controls the entire mobile phone 1. For example, the control unit 20 controls transmission / reception of various types of information in the communication processing unit 11, processes audio information for the audio processing unit 14, controls display of information on the display unit 15, the key input unit 13 and the panel input unit 16. The process according to the input information from, the access control with respect to the memory 12, etc. are performed.

Further, the control unit 20 performs an object extraction process based on the image data, an allocation process for the extracted object, and a process for determining a designated area of the object.
Hereinafter, the processing executed by the control unit 20 will be described in order.

Object Extraction Processing First, the control unit 20 displays an image on the display unit 15 based on the image data stored in the memory 12, and performs object extraction processing on the displayed image.
FIG. 3 is an example of an image for explaining the object extraction process.
For example, when the camera function is built in, the image that is the original data of the object extraction process may be an image obtained by shooting with the camera, an image attached to an e-mail, or an external medium such as a memory card. An image read from the Internet, an image obtained by data communication via a communication device (infrared communication, short-range communication, etc.), or an image downloaded from any WEB on the Internet may be used.
Such image data is temporarily stored in the memory 12, read from the memory 12 by the control unit 20, and displayed on the display unit 15. At that time, the image as the original data is enlarged / reduced and displayed as necessary so as to be optimized to the screen size of the display unit 15.

  In the object extraction process, an object (independent target) expressed on the image displayed on the display unit 15 is extracted from the image. For example, in the image illustrated in FIG. 3, the object OBJ1 (“sun”), the objects OBJ2 and OBJ3 (“cloud”), the objects OBJ4 and OBJ5 (“tree”), and the object OBJ6 (“house”) 6 Objects are included, and extracting these objects from the image is performed by object extraction processing.

The object extraction process is executed by performing an outline extraction process (edge extraction process) on the image displayed on the display unit 15.
The contour extraction process (edge extraction process) is performed by paying attention to the luminance value and color density of the image and detecting edge portions where these data (color data) change rapidly.
Specifically, a method for calculating the strength and direction of the contour from the magnitude and direction of the primary differentiation (gradient) of the color data based on the difference value of the color data between adjacent pixels, and further detecting the strength of the contour. Therefore, techniques such as a method using a second derivative (Laplacian) can be applied. In addition, contour extraction may be performed by focusing on similar colors from the displayed image.
By the object extraction process, each object in the image displayed on the display unit 15 is extracted.

The block-unit object extraction processing control unit 20 executes the above-described object extraction processing for each block obtained by dividing the display unit 15 into a predetermined number for object allocation processing described later.
FIG. 4 illustrates a case where the display unit 15 that displays the image illustrated in FIG. 3 is divided into 4 × 3 blocks. In the following description, the vertical block coordinate is X, the horizontal block coordinate is Y, and in the display unit 15, the block coordinate is expressed as a block B (X, Y).
In FIG. 4, blocks for object extraction processing are defined so as to match the 4 × 3 key layout (see FIG. 2) of the key input unit 13.

The object extraction processing is performed in the order of each block of B (1, 1), B (1, 2), B (1, 3), B (2, 1),.
In the example shown in FIG. 4, the object OBJ1 is extracted by performing the object extraction process in the block B (1, 1), and the object OBJ4 is extracted by performing the object extraction process in the block B (4, 1). The object OBJ6 is extracted by performing the object extraction process in the block B (4, 3).
When a plurality of objects are extracted in the same block, the largest object is extracted from the plurality of extracted objects.

In the object extraction process, the closed region is recognized as an object.
Since the object is straddling a plurality of blocks, if the object extraction process in one block is not recognized as a closed region, the object is ignored and not extracted.
For example, the object OBJ3 shown in FIG. 4 is not extracted as an object in the object extraction process for the block B (2, 3) because all the closed areas do not fit in the block B (2, 3).

  When the object extraction process is completed for each of the 12 blocks, the control unit 20 reduces the number of blocks and performs the object extraction process for each block again. Thereby, in the above 12-block division, it is possible to extract an object even when the object extraction process in one block is not recognized as a closed region or there is a block in which no object exists.

FIG. 5 illustrates a case where the display unit 15 that displays the image illustrated in FIG. 3 is divided into 2 × 2 blocks.
In FIG. 5, the objects OBJ1, OBJ4, and OBJ6 are not shown because they have already been extracted by the object extraction process in the 12 blocks.
As in the case of 12 blocks, the control unit 20 sequentially performs object extraction processing on the blocks B (1, 1) to B (2, 2), and the object OBJ2 is transferred from the block B (1, 1) to the block B. The object OBJ3 is extracted from (1, 2), and the object OBJ5 is extracted from the block B (2, 1).

  As described above, when an object is extracted for each block, the control unit 20 causes the memory 12 to store position information (contour position information) about the contour of each object.

When an object is extracted from the image displayed on the object assignment processing display unit 15, the control unit 20 performs a function assignment process on each extracted object.
The function allocation process can take various methods.

(A) Key allocation process of key input unit 13 As described above, in the object extraction process, the display unit 15 is configured to match the 4 × 3 key layout (see FIG. 2) of the key input unit 13. Since the screen is divided and executed for each block, the control unit 20 assigns the object extracted for each block to the corresponding key.
For example, in the 4 × 3 key arrangement of the key input unit 13 shown in FIG. 2, the control unit 20 determines that the uppermost key in the left column is a key for the number “1”. In the illustrated image, the object OBJ1 (“sun”) extracted from the block B (1, 1) is allocated.

The control unit 20 compares the input position information input from the panel input unit 16 with the contour position information of each object stored in the memory 12, and when the input position is within the area indicated by the contour position of the object. Then, it is determined that the key assigned to the corresponding object has been operated, and processing for the key is performed.
For example, in FIG. 4, when the user touches the “sun” portion of the image displayed on the display unit 15 with a finger or the like, the panel input unit 16 displays the “sun” of the image displayed on the display unit 15. An input position corresponding to the portion is detected, and the detected input position information is output to the control unit 20. The control unit 20 searches the memory 12 for an object whose input position is within the contour position of the object. As a result, the object OBJ1 is searched and input processing corresponding to the numeric key “1” assigned to the object OBJ1 is performed.

If objects cannot be extracted for some of the 4 × 3 blocks, the extraction process is performed again, for example, with a 2 × 2 block configuration as described above. The assigned object is assigned to the key corresponding to the block in which no object is extracted in the 4 × 3 block.
If all the keys of the key input unit 13 are not allocated even if the number of blocks is reduced, the user is notified via the display unit 15 and other images are extracted. You may comprise so that it may select.

(B) The assignment processing control unit 20 for a predetermined function assigns the extracted object not only to the above-described assignment processing to a single key but also as a combination of a plurality of keys of the key input unit 13 or as a menu button. It can also be assigned. At that time, the extracted objects are assigned by interactive processing with the user.
For example, after the object extraction process, functions that are frequently used by the user, such as setting the volume of the mobile phone 1, starting an application program such as a game, and setting an alarm clock are associated with each extracted object. These functions associated with each object are managed in the memory 12 in association with the object.
The image based on the object extraction process may be configured to be automatically set as a standby screen. When the user designates an image on the standby screen, the control unit 20 compares the input position information related to the designation with the contour position information of each object in the memory 12 to identify the object related to the designation. The function associated with the identified object is executed. For example, if the function “launch game” is assigned to the object OBJ4 (“Thu”) in the image shown in FIG. 3, the game is activated by designating the object OBJ4 on the screen. The control unit 20 performs control so that the display is switched to the game start screen.
In the object extraction stage, if the object has been extracted in a form corresponding to the same arrangement as the key input unit 13, the arrangement of the key input unit 13 and the position of the object are associated with each other without using the touch panel. By pressing each key on the key input unit 13, the control unit 20 may recognize that an associated object has been designated and execute a function assigned to the object.

As determination processing above specified area, the control unit 20 includes an input position information inputted from the panel input unit 16, compares the contour position information of each object memory 12 stores the input position of an object contour If it is within the area indicated by the position, it is determined that the key assigned to the corresponding object has been operated. In general, however, the shape of the object is not a simple shape such as a round shape or a square shape. It may be difficult for an object on the screen to touch. Therefore, the control unit 20 determines a designated area for determining that the input from the user is valid for each object.
That is, the control unit 20 compares the input position input from the panel input unit 16 with the designated area of each object, determines that an object having a designated area that includes the input position is designated, and designates the designated object. The input process according to is executed.
A method for calculating the designated area for each object will be described below.

FIG. 6 is a diagram for explaining a method of calculating the designated area of the object. In the drawing, an object OBJ represented by a solid line has already been extracted by the control unit 20, and its contour position information is stored in the memory 12.
First, the control unit 20 calculates the smallest rectangle that can accommodate the contour of the object based on the contour position information of the object. Specifically, the X and Y coordinates on the screen of the points P11 to P14 of the quadrangle REC1 in FIG. 6 are calculated.
Next, the control unit 20 calculates the maximum rectangle that can be accommodated in the contour range of the object based on the contour position information of the object. Specifically, the X and Y coordinates on the screen of the points P21 to P24 of the quadrangle REC2 in FIG. 6 are calculated.

Based on the points P11 to P14 and points P21 to P24 calculated as described above, points P31 to P34 that define the designated area of the object OBJ are calculated as in the following equations (1) to (8).

X (P31) = (X (P11) + X (P21)) / 2 (1)
Y (P31) = (Y (P11) + Y (P21)) / 2 (2)
X (P32) = (X (P12) + X (P22)) / 2 (3)
Y (P32) = (Y (P12) + Y (P22)) / 2 (4)
X (P33) = (X (P13) + X (P23)) / 2 (5)
Y (P33) = (Y (P13) + Y (P23)) / 2 (6)
X (P34) = (X (P14) + X (P24)) / 2 (7)
Y (P34) = (Y (P14) + Y (P24)) / 2 (8)

X (PN) and Y (PN) indicate the X and Y coordinates of the point PN, respectively.

The rectangle REC3 indicating the designated area is determined based on the points P31 to 34 calculated as described above, and the points P31 to P34 indicating the designated area are stored in the memory 12 in association with the objects.
In the above formulas (1) to (8), the quadrangular REC3 indicating the designated area is calculated so as to be the average size of the quadrangles REC1 and REC2 in FIG. 6, but is not limited to this. Any square may be used as long as it is positioned between the squares REC1 and REC2.

FIG. 7 shows an example of calculation of the designated area executed by the control unit 20.
In FIG. 7, the designated area (within the rectangle REC3 in the figure) of the object OBJ2 (“cloud”) extracted from the image illustrated in FIG. 3 is calculated.
When the input position input from the panel input unit 16 is within the range of the rectangle REC3 in FIG. 7, the control unit 20 determines that the object OBJ2 has been designated, and performs the input process assigned to the object OBJ2. Execute.

Heretofore, a plurality of processes executed by the control unit 20 have been described.
Next, a series of processing executed by the mobile phone 1 according to the present embodiment will be described in association with the flowchart shown in FIG.
The flowchart shown in FIG. 8 shows a series of processing from when a desired image is displayed by a user operation until function assignment is performed on objects in the image.
First, when a predetermined key of the key input unit 13 is pressed (operated) in a state where a desired image is displayed on the display unit 15 by a user operation (step ST10), the control unit 20 performs object extraction processing. Is started (step ST11).
The number of screen division blocks for object extraction processing is 4 × 3 12 blocks, and object extraction is performed for each block.

If all the objects cannot be extracted in 12 blocks, the range of the block is enlarged (step ST13), and the object extraction process is performed (step ST11). For example, an object is extracted by 2 × 2 4 blocks. At that time, objects already extracted in the object extraction process in 12 blocks are ignored.
During the above processing, every time an object is extracted from the image displayed on the display unit 15, the control unit 20 stores the extracted contour position information of each object in the memory 12.

When all the objects are extracted (step ST12), the control unit 20 calculates the designated area based on the contour position information of each object stored in the memory 12.
That is, the control unit 20 calculates the smallest quadrangle (REC1) that can accommodate the contour of the object based on the contour position information of the object (step ST14). Next, the control unit 20 calculates the maximum rectangle (REC2) that can be accommodated in the contour range of the object based on the contour position information of the object (step ST15).
Then, the control unit 20 calculates a quadrangle REC3 having an intermediate size based on the quadrangle REC1 calculated in Step ST14 and the quadrangle REC2 calculated in Step ST15, and sets the quadrangle REC3. Point coordinate data is stored in the memory 12 (step ST16).

When the designated area of each extracted object is calculated in steps ST14 to ST16, the control unit 20 next performs object allocation processing (step ST17).
For example, the control unit 20 assigns the object extracted for each block in step ST11 to the corresponding key.
Further, the control unit 20 may allocate the extracted object as a combination of a plurality of keys of the key input unit 13 or as a menu button by interactive processing with the user. As a result, each object is assigned to input processing from the user.
In step ST17, the object to which the input process is assigned is displayed separately from the object to which the input process is not assigned. For example, the display form such as the display color, brightness, and shade of the object is distinguished and displayed.

  After the processing of the flowchart shown in FIG. 8 is performed, when the user touches an object in the image displayed on the display unit 15 on the screen, the panel input unit 16 displays input position information corresponding to the touched position. Output to the control unit 20, the control unit 20 compares the contour position information of each object stored in the memory 12, and if the input position is within the specified area of the object, the input assigned to the corresponding object It is determined that processing (for example, key pressing) has been performed, and the input processing is performed.

As described above, according to the mobile phone 1 according to the present embodiment, by performing the above-described processing, a function is assigned to an object extracted based on an arbitrary image desired by the user, and the user's own man-machine An interface can be constructed.
Further, since the user can perform an input operation by directly specifying an object on the screen without using the key input unit, the comfort of the operation is improved.

According to the mobile phone 1 according to the present embodiment, the screen is divided into blocks, and an object is extracted for each divided block. Therefore, a plurality of objects scattered in a balanced manner on the screen can be extracted, and the above-described interface can be constructed. It's easy to do.
At this time, the block division mode is matched with the key arrangement of the key input unit, and the function corresponding to each key is assigned to the object extracted from each block, so that the user can easily recognize the key corresponding to the object.
Furthermore, if the object cannot be extracted in the divided block, the object is extracted again after reducing the number of divisions, so that the object that could not be extracted at the time of the first division can be extracted. It is possible to extract a plurality of objects scattered in a balanced manner on the screen, and it is easy to construct the interface.

  According to the mobile phone 1 according to the present embodiment, since the designated area of the extracted object is a quadrangle corresponding to the outline of the object, even if the object has a complicated shape, the object that has passed through the panel input unit 16 is displayed. Specification becomes easy.

  According to the mobile phone 1 according to the present embodiment, among the extracted objects, an object to which a function is assigned is displayed separately from an object to which no function is assigned. There is no wasteful operation of specifying.

The present invention is not limited to the contents of the above-described embodiment, and various modifications can be made without departing from the scope of the present invention.
For example, in the above-described embodiment, the touch panel type panel input unit has been described as the designation method for the screen from the user. However, it is also possible to apply input by a pointing device using a mouse, a cursor, or the like.
In the above embodiment, each extracted object is associated with the 4 × 3 key array of the key input unit, but may be associated with a soft key set on the screen.

1 is a block diagram illustrating an example of a system configuration of a mobile phone. It is the figure which illustrated an example of the external appearance of a mobile telephone. It is an example of the image which the display part of a mobile telephone displays. It is an example which divided | segmented the display part of the mobile telephone into 4x3. It is an example which divided | segmented the display part of the mobile telephone into 2x2. It is a figure for demonstrating the calculation method of the designated area | region of an object. It is an example of calculation of the designated area of an object. It is a flowchart for demonstrating the process of a mobile telephone.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Mobile telephone 11 ... Communication processing part 111 ... Transmission / reception antenna 12 ... Memory 13 ... Key input part 14 ... Audio | voice processing part 141 ... Speaker 142 ... Microphone 15 ... Display part 16 ... Panel input part 20 ... Control part

Claims (8)

Display means for displaying at least an image;
Operation means;
When a predetermined operation is performed on the operation unit while a predetermined image is displayed on the display unit, an object is extracted from the predetermined image, and one function is associated with the extracted object. And a control means for executing a function associated with the designated object when the object is designated by the operation means. An object is associated with a designated area that includes the display area of the object,
The control means includes
The portable terminal device according to claim 1, wherein when the designated area is designated by the operation means, it is determined that an object associated with the designated area is designated. The control means includes
Calculating a first rectangle that is the largest rectangle that can fit in the display area of the object and a second rectangle that is the smallest rectangle that can contain the display area of the object;
The portable terminal device according to claim 2, wherein an area in a third rectangle that is a rectangle having a size intermediate between the first rectangle and the second rectangle is associated with the object as a designated area. The control means includes
In a case where a predetermined image is displayed on the display means, when a predetermined operation is performed on the operation means, a maximum display area that can be displayed by the display means is divided into a predetermined number of areas. The mobile terminal device according to claim 1, wherein an object is extracted for each of the regions. The control means includes
The portable terminal device according to claim 4, wherein the predetermined number is decreased when an object cannot be extracted in at least one of the plurality of regions. The operating means comprises a plurality of keys,
Each region of the plurality of regions is divided corresponding to the plurality of keys,
The control means includes
The mobile terminal device according to claim 4, wherein when one key of the plurality of keys is operated, it is determined that an object extracted from an area corresponding to the one key is designated. The control means includes
The mobile terminal device according to claim 1, wherein an object associated with an input process is displayed separately from an object not associated with an input process. An input processing method for an information processing apparatus having a display unit and an operation unit,
Displaying a predetermined image on the display unit;
Performing a predetermined operation on the operation unit;
Extracting an object from the predetermined image in response to the predetermined operation;
Associating a function with the extracted object;
Designating an object by the operation unit;
And a step of executing a function associated with the designated object.

Images