WO2014030667A1

WO2014030667A1 - Information processing device, information processing system, and information processing method

Info

Publication number: WO2014030667A1
Application number: PCT/JP2013/072272
Authority: WO
Inventors: 俊治栗栖; 恭子増田; 直樹橋田; 結旗柘植; 幸子奈良
Original assignee: 株式会社エヌ・ティ・ティ・ドコモ
Priority date: 2012-08-21
Filing date: 2013-08-21
Publication date: 2014-02-27
Also published as: JPWO2014030667A1; JP5813881B2

Abstract

A communication terminal (1) comprises: a display unit (20) further comprising a display region; a designation means (31) for receiving a designation of a location within the display region: a determination means (12) for determining a range within the display region which includes the location; a display control means for displaying content in the display unit, and rendering an image object in the determined location; and a generation means (50) for generating, on the basis of a display description corresponding to the determined range of the content which is displayed in the display unit, information associated with the designated image object.

Description

Information processing apparatus, information processing system, and information processing method

The present invention relates to a GUI (Graphical User Interface).

In particular, on a device that has both input and display functions, such as a touch panel, the contents (display) displayed by the application by drawing (“sticking”) a predetermined image object (“sticker”) while the application is running There is a technique for setting events and actions related to (object). By imitating the sticking operation on the computer, the fact of the setting can be transmitted to the user in an easy-to-understand manner. For example, Patent Document 1 describes that a combination of an event and an action is flexibly set for such sticker sticking processing.

JP 2011-76508 A

Patent Document 1 describes setting an area in a screen where a sticker can be affixed among displayed contents. However, the user can only select a desired area from the predetermined pastable areas, and the position where the sticker is pasted and the target to which the sticker is pasted (display contents associated with the sticker) are limited. It was.

An object of the present invention is to allow a user to freely specify a sticking position and a sticking target of a seal.

In one aspect, the present invention provides a display unit having a display area, designation means for accepting designation of a position in the display area, determination means for determining a range in the display area including the position, and the display A display control means for displaying an image object at the determined position and a display control unit that displays the content at the determined position, and the display content corresponding to the determined range of the content displayed on the display unit. There is provided an information processing apparatus having generation means for generating information associated with an image object.
In a preferred aspect, the specifying means receives specification of two positions, and the determining means determines that a rectangular area including the two specified points as vertices is the range.
In another preferred embodiment, the determining means determines the center of the range as the position, and determines the size of the range based on the length of the period in which the position is specified.
In another preferred aspect, the specifying means continuously receives specification of a plurality of positions, and the determining means determines a boundary of the range based on a trajectory represented by the specified plurality of positions, The inside of the boundary is determined as the previous range.
In another preferable aspect, the determining unit determines an area including the position based on the content of the content displayed on the display unit, and the display control unit includes the determined area inside the determined area. Decide with a range.
In another preferred aspect, the apparatus further comprises second specifying means for accepting designation of an image object, wherein the determining means is when a length of a period during which the image object is designated by the second designation means is equal to or greater than a predetermined length. And determining one or a plurality of candidate areas based on the content of the content displayed on the display unit, and the display control means displays an image representing the determined one or a plurality of area candidates on the display unit. The determining means determines an area including the position specified by the specifying means among the displayed one or a plurality of area candidates as the range.
In another preferred aspect, the specifying means continuously receives specification of a plurality of positions, and the determining means determines a boundary of the range based on a trajectory represented by the specified plurality of positions, One or a plurality of areas that are specified on the basis of the content of the content displayed on the display unit and are included in the boundary are determined as the range.
In another aspect, the present invention provides a computer with a step of accepting designation of a position in a display area of a display unit, a step of determining a range within the display area including the position, and displaying content on the display unit And drawing an image object at the determined position, and generating information associated with the image object based on display content corresponding to the determined range of the content displayed on the display unit. A program for executing the above is provided.
In yet another aspect, the present invention further includes a step of accepting designation of a position in the display area of the display unit, a step of determining a range in the display area including the position, and displaying content on the display unit A step of drawing an image object at the determined position; and a generation step of generating information associated with the image object based on display content corresponding to the determined range of the content displayed on the display unit; An information processing method is provided.
In still another aspect, the present invention includes a display unit having a display area, first designation means for accepting designation of a position in the display area, second designation means for accepting designation of an image object, and the position. Determining means for determining a range within the display area; first display control means for displaying the content on the display unit; and drawing the designated image object at the determined position; and Generating means for generating information associated with the designated image object based on display contents corresponding to the determined range of the displayed content; and the generated information associated with the designated image object A storage unit for storing the content and information indicating the determined position, and an image object associated with the generated information. When the displayed image object is specified, the display of the content and the drawing of the image object are executed by the first display control means according to the information stored in association with the image object. There is provided an information processing apparatus having a second display control means.

According to the present invention, the user can freely specify the sticking position and sticking target of the seal.

The figure showing the function structure of the communication terminal. The figure showing the operation example of a sticking process. The figure showing the operation example which calls the affixed sticker. (A) to (f) are examples of the data structure of the seal definition information DST-a. (A) And (b) is an example of operation content (selection of a seal). (A) to (d) are examples of operation contents (selection of stickers to pasting process). (A) to (d) are examples of operation contents (selection of stickers to pasting process). Example of operation (determining the position and range of pasting). The flowchart of the operation | movement which determines the position and range of sticking. (A) to (c) are examples of the operation content (determining the position and range of application). The flowchart of the operation | movement which determines the position and range of sticking. (A) And (b) is an example of operation content (determination of the position and range of sticking). The flowchart of the operation | movement which determines the position and range of sticking. (A) And (b) is an example of operation content (determination of the position and range of sticking). The flowchart of the operation | movement which determines the position and range of sticking. (A) to (c) are examples of operation contents (determination of the position and range of application). The flowchart of the operation | movement which determines the position and range of sticking. Example of operation (determining the position and range of pasting). The flowchart of the operation | movement which determines the position and range of sticking. (A) And (b) is an example of operation content (determination of the position and range of sticking). The flowchart of the operation | movement which determines the position and range of sticking. The example of the structure of sticking process information DST2 and sticking process information DST. (A) to (c) are examples of screens related to the sticking process information DST and the seal calling process. An example of sticking process information DST2. Examples of paste processing and call processing. The example of sticking process information DST3. (A) to (c) are examples of screens related to the pasting process and the calling process. (A) is sticking process information DST4, (b), (c) is an example of the screen concerning a sticking process and a calling process. (A) is sticking process information DST5, (b), (c) is an example of the screen concerning a sticking process and a calling process. (A) is sticking process information DST6, (b), (c) is an example of the screen concerning a sticking process and a calling process. (A), (b) is an example of a screen with a sticker attached.

DESCRIPTION OF SYMBOLS 1 ... Communication terminal, 10 ... Control part, 11 ... 1st display control part, 12 ... 2nd display control part, 40 ... Determination part, 50 ... Generation part, 30. ..Specifying unit, 31... First specifying unit, 32... Second specifying unit, 60 ..Communication unit, 70... Voice input / output unit, 90. Application control unit

Hereinafter, embodiments of the present invention will be described with reference to the drawings.
1. Overview of Configuration FIG. 1 is a diagram illustrating a functional configuration of a communication terminal 1. The communication terminal 1 is an information processing apparatus including a processor, a storage device, and an information input / output device, such as a personal computer, a cellular phone, and a smartphone, and is determined as a display control unit 10, a display unit 20, and a designation unit 30. The unit 40, the generation unit 50, the communication unit 60, the voice input / output unit 70, and the storage unit 90 are functionally configured.

The storage unit 90 is a storage device such as a semiconductor memory or a hard disk, and the first display control unit 11 accesses and reads and writes information stored in the storage unit 90. The information stored in the storage unit 90 roughly includes an application program P, data DR, and pasting process information DST. Each application program (hereinafter referred to as an application) P is executed to implement various functions in the communication terminal 1 under the control of the application control unit 100. The application P includes a seal browsing application P1, a map application P2, a scheduler P3, a phone book application P4, a mail application P5, a history display application P6, and a browser P7. The seal browsing application P1 is an application that displays a list of pasted stickers on the display unit 20 and identifies what the user has selected from among them. The map application P2 reads the map data DR1 in the range designated by the user according to the user's operation, or acquires the map data DR1 from the server device via the communication unit 60, and displays it on the display unit 20. In addition, when a position on the screen (physical position) is designated, the map application P2 has a function of determining a position on the map displayed on the screen corresponding to the position. The scheduler P3 is an application for displaying a calendar and registering, browsing, and editing a schedule. The phone book application P4 is an application for registering, browsing, and editing a partner's name, telephone number, address, and other contact information. The mail application P5 is an application that creates, transmits, and receives e-mails. The history display application P6 is an application that stores and browses information (history) about calls and data communications made in the past. The browser P7 is an application that displays content described in a script language such as HTML. When any application is executed, the content of the displayed information and the position (or range of positions) of the information on the screen are set in response to a request from the first display control unit 11. It has a function to generate and provide information about the arrangement. Further, depending on the type of the data DR to be referenced, a position on the screen and position information generated uniquely for the content (for example, a position on the map) are generated in association with each other and temporarily or permanently stored in the storage unit 90. It has a function to memorize.

The data DR includes map data DR1, schedule data DR2, telephone directory data DR3, HTML data DR4, mail data DR5, and communication history data DR6, which are read or written when the corresponding application is activated or activated. (Update). DR1 includes coordinate data (latitude / longitude, etc.) and landmark information (building name, road name, etc.), and can be referred to during the execution of the map application P2. The schedule data DR2 includes schedule contents (text data, position information, etc.) associated with the date and time. The phone book data DR3 includes contact information such as a telephone number and an address associated with a person's name (text information). The HTML data DR4 includes web page data and setting information (such as cookies). For example, in the case where one web page includes a plurality of sub contents, information on the arrangement position of each sub content is included. Therefore, if the position on the screen is specified by the user even after the HTML data DR4 is interpreted by the browser P7 and the contents of the HTML data DR4 are displayed, the content displayed at the position is specified. The mail data DR5 includes received mail information (header, text, pasted file, etc.) and address information. The communication history data DR6 includes the telephone number of the other party that received the call, the telephone number of the other party, and information on the date and time when the incoming or outgoing call was made. Information such as name and address may be associated with the telephone number.

The pasting process information DST includes sticker definition information DST-a and pasting information DST-b. The seal definition information DST-a includes information necessary for the appearance of an image object called “seal”, the image object drawing process, and the operation based on the drawn image object. The pasting information DST-b is generated based on the contents of the pasting operation such as the pasting position and pasting range when drawing a sticker image (hereinafter referred to as “sticker pasting”), and an action is taken on the pasted sticker. Contains information to be read. In this embodiment, a plurality of types of seals are prepared in advance, and even the same type of seal can be applied to a plurality of locations. Note that the user may determine the content of the seal definition information DST-a. In this way, the operation of the seal can be freely defined.

Although not shown, the storage unit 90 stores an OS (operating system software) that controls the communication terminal 1, and the display control unit 10 is activated when the processor installed in the communication terminal 1 starts the OS. Functions of the determination unit 40, the generation unit 50, the application control unit 100, and the like are realized. Each application P operates on the OS.

The display control unit 10 includes a first display control unit 11 and a second display control unit 12 including a CPU, an image processor, and the like, and controls the display unit 20. Specifically, a display-related command is output to the application control unit 100, a display-related request or display data is received from the application control unit 100, and image data or a display command is output to the display unit 20. Regarding the display of the seal, the first display control unit 11 displays an image object (sticker) associated with the information generated by the generation unit 50 on the display unit 20. When the displayed image object is designated, the second display control unit 12 associates with the image object and controls the application control unit 100 according to the information stored in the storage unit 90 to display the content and the image. The second display control unit 12 is caused to execute drawing of the object.

The display unit 20 is a display device having a display area for displaying an image, a light source such as a light emitting element, an optical filter, a control circuit, and the like. For example, a liquid crystal display or an organic EL display having a region for displaying an image.

The designation unit 30 includes a first designation unit 31 and a second designation unit 32, and accepts designation of a position in the display area. The first designation unit 31 is an input device such as a touch sensor. In a preferred embodiment, the first designation unit 31 is formed integrally with the display unit 20 and constitutes the touch panel TP. The user specifies a predetermined action (for example, touching the screen) and the position on the screen pointed to. Further, not only the pointing position but also the time change of the pointing operation is detected. There are operations such as “tap operation”, “double tap operation”, “drag operation”, and “long press operation” depending on the duration of the operation for specifying the position and the time change of the specified position. That is, the first designation unit 31 detects the position (change). Information on the detected position (change) is output to the determination unit 40. Note that when inputting information via the first designation unit 31, it is not always necessary for the user's finger or other body part to touch the screen, and the user's finger or operator within a predetermined distance from the screen. It may be determined that an input operation has been performed by detecting the presence of a (touch pen, etc.).
The second designation unit 32 is a hardware button, an input key, a switch, or other mechanical input means that accepts an operation such as pressing by the user, and details of the operation (information on whether or not the button has been pressed, time of pressing, etc.) Are output to the determination unit 40.

The determination unit 40 is realized as a processor or a functional module, and determines a range (sticker pasting range) within the display area including the position received by the designation unit 30. Further, when the received position is provisional, the final position (sticker application position) is determined based on the range. That is, the designation unit 30 and the determination unit 40 specify the target sticker, the pasting position, and the pasting range to be pasted in cooperation or independently.
The generation unit 50 is realized as a processor or a functional module, and generates information to be associated with the designated image object based on the display content corresponding to the determined range of the content displayed on the display unit 20.

The communication unit 60 is realized by a wireless communication antenna, a signal processing circuit, or the like, and performs voice communication and data communication with other devices under the instruction of the display control unit 10.
The voice input / output unit 70 is realized by a microphone, a speaker, or the like, and is used by the user when making a call. The voice input / output unit 70 may be provided with a voice recognition circuit, and the voice input / output unit 70 may be provided with a function for realizing the designation of the sticking position and sticking range of the designation unit 30 based on the voice of the user. .
The application control unit 100 is realized by a processor or the like by a processor or the like. Under the control of the display control unit 10, the application P is activated, the data is transferred to the application P, and the application execution result (drawing request) display control unit 10. Output to.

2. Overview of Operation The operation of the present invention is broadly divided into a process related to sticking a sticker (hereinafter referred to as a sticker sticking process) and a process based on the sticker (hereinafter referred to as a seal calling process).
FIG. 2 is a diagram illustrating an outline of an operation example of the pasting process in the communication terminal 1. First, it is assumed that when the application P is being executed, the user wants to apply one sticker in association with the displayed content. At this time, the user first selects one seal to be pasted (S10). Next, the designation | designated part 30 and the determination part 40 determine the sticking position and sticking range of the said seal | sticker (S20). Specifically, first, the sticking position of the sticker (that is, on which position on the screen the sticker image is drawn) is determined (S21). Subsequently, the sticking range of the seal is determined (S22). The pasting range specifies which range (including all ranges) of the content displayed on the screen when pasting is associated with the seal. Note that the order of determination of the pasting position and the determination of the pasting range may be reversed or simultaneous. Further, the order of selecting the seal and determining the position and range of the seal may be reversed or simultaneous. In short, it is only necessary to specify a sticker to be pasted, a pasting position, and a pasting range.

Next, the communication terminal 1 draws a sticker at the determined position (S30). And the communication terminal 1 produces | generates sticking information (S40). It memorize | stores in the memory | storage part 90 as the produced | generated sticking process information DST (S50).

FIG. 3 is a diagram showing an outline of an example of the seal call processing. First, in response to a request from the user, the communication terminal 1 activates a first application for causing the user to select a seal (S100). The first application is, for example, a seal browsing application P1. When the user designates a sticker (S102), sticking information about the sticker is read (S104). The communication terminal 1 identifies and activates the application (second application) to be activated based on the read paste information (S106). Note that the execution timing of the sticking process of the sticker and the execution timing of the calling process of the sticker may not be independent as in this example. For example, the calling process may be executed following the completion of the sticking process. That is, S100 and S102 in FIG. 3 may be omitted, and the operation may be shifted to S104 in FIG. 3 after the end of S50 in FIG.

3. Detailed Configuration and Operation and Screen Example (1) Data Structure FIG. 4 shows an example of the data structure of the seal definition information DST-a. The seal definition information DST-a has a structure including data items DI1, DI2, and DI3 in association with a seal ID that identifies the type of seal (FIG. 5A). In the item DI1, information for designating image data for determining the appearance of the sticker (for example, a position in the storage unit 90 in which an image file in jpeg or bmp format is stored) is described. For convenience of explanation, not the image data name or the storage location of the image data but the appearance of the seal is drawn. In the example of FIG. 5, five types of seals (b) to (f), which are identified by

IDs

0001, 0002, 0003, 0005, and 0010, respectively, are defined. The contents of DI3 are different. Therefore, for example, even if the screen content at the time of pasting and the screen position to be pasted are the same, the generated pasting information may differ if the type of sticker to be pasted is different. As a result, the contents of the seal calling process are also different. By preparing a seal having a different function, the contents of the operation related to the pasted seal are diversified, and the range of selection by the user is expanded.

The item DI2 is information defining a method for generating pasting information. Specifically, it includes information for determining the pasting position and the association information included in the pasting information that is information to be provided to the cooperation destination application.
The information for determining the pasting position is, for example, whether absolute coordinates on the screen (such as the number of vertical and horizontal dots with the left end of the screen as the origin) are used as the pasting position, or the content to be pasted is a map In the case of information, if a position on the map (latitude, longitude, etc.) is adopted, or if one displayed content consists of a plurality of contents each having a display area (referred to as sub-contents), a predetermined sub-content This is information for designating whether to adopt a position (the number of dots in the vertical and horizontal directions of the screen) based on the arrangement position and a position (direction, distance, etc.) based on a predetermined character string included in the content.

The information for determining the linking information is information that defines what information in the display content should be linked to the pasted sticker for the determined pasting range. For example, after determining the pasting range, whether to generate image data of an image obtained by cutting out an area corresponding to the pasting range from the displayed image, or when the display content is map information, etc. Whether the coordinate data representing the area is used as the association information, or when the display content is a web page (HTML file), the link information (URL) of the content or sub-content is adopted as the association information. It is defined whether a character string (text data) is adopted as the association information. Further, as shown in (f) of the figure, DI2 is automatically determined at least one of the pasting position and the linking information according to the display content (in other words, the application displaying the content). Such information may be defined.
As described above, when the content of the operation to be pasted (that is, designation of the pasting position and pasting range) is confirmed, the content of pasting information generated by DI2 is determined. Note that the association information may include a plurality of contents. For example, in the case of FIG. 5C, information for specifying positions of two or more points may be included instead of the coordinates of one point on the map.

Item DI3 is information defining the application P linked with the seal. The application to be linked defines which application uses (for example, browses) the attached sticker. Specifically, an application to which the generated pasting information is delivered is defined. In the case where the seal definition information DST-a is a seal represented by (b) in the figure, it is defined that the application that is executed at the time of application is linked with the same application. For example, when the map application P2 is executed at the time of pasting, that is, when a sticker is pasted on the map, the sticker information of the sticker is handed over to the map app, and the map app P2 is activated to confirm the sticker again It is prescribed to do. In the case of the seal shown in FIG. 6C, the map application is linked regardless of the application executed at the time of pasting. Similarly, in the case of the seal shown in FIG. Or you may prescribe | regulate the candidate of a some cooperation application like the case of the figure (e). In this case, it may be linked with a plurality of applications, or one of the linked apps P may be finally determined by providing priority and conditions. Alternatively, the contents of the seal definition information DST-a may be described so that the cooperation destination application is determined according to the method for specifying the pasting position and the pasting range.

(2) Outline of operation regarding selection of seal, designation of pasting position and pasting range FIG. 5 is a diagram for explaining a seal selection process (S10 in FIG. 2). As shown in FIG. 5A, the communication terminal 1 displays content and detects a contact with a user's finger or the like and detects a contact position and a contact time, and other than the touch panel TP. As input means, hardware button groups BT1, BT2, BT3 are provided. FIG. 2A shows an example in which a map application is executed and a map image is displayed on the touch panel TP. In this state, when the user performs an operation for calling a predetermined sticker (for example, pressing one of the button group), the display control unit 10 displays images of all stickers registered in the storage unit 90. Pop-up images of the stickers ST1, ST2, ST3 and ST4 are displayed on a partial area PRT in the touch panel TP.

Subsequently, as shown in FIG. 6A, when the user taps the seal image to be pasted (ST1 in the example of the figure) with the finger F, the seal to be pasted is determined. The display control unit 10 draws a translucent pattern image on the entire screen so as to overlap the displayed map as shown in FIG. This is to inform the user that the current state has entered a state for accepting an operation for determining the sticking position and sticking range of the selected sticker, and to perform the next operation. The user designates a pasting position (C) and a pasting range SA on the touch panel TP ((c) in the figure). When the type of sticker to be pasted, the pasting position, and the pasting range are all determined, the drawing of the pattern image is stopped and the sticker ST1 is drawn on the map image as shown in FIG. Is attached to the map content). In the example shown in the figure, the pasting area SA is not drawn on the map, but this is because it is only necessary to inform the user that the sticker has been pasted. Therefore, in addition to the seal image, an image indicating the pasting range (for example, a boundary line of the range) may be drawn.

Alternatively, as shown in FIG. 7, the selection order of the seal, the designation of the pasting position, and the designation of the pasting range may be reversed. That is, the display control unit 10 automatically draws a pattern image when receiving a predetermined operation or without waiting for the operation from the state shown in FIG. To the state shown in FIG. The designation of the pasting position C and the pasting range SA is accepted, and the pasting position and the pasting range are determined. Subsequently, as shown in FIG. 6C, when the seal to be applied is selected, the type of the seal, the application position, and the application range are determined, and the seal is drawn on the map (FIG. 4D). .

(3) Details of operation for designating pasting position and pasting range Subsequently, some examples will be described for details of the operation for designating pasting position and pasting range. In any example, the order of selecting the seal type may be before or after the designation of the pasting position or pasting range. In addition, in the designation method shown below, multiple designation methods are facilitated according to the content of the user's operation, and it is determined in order whether the operation content matches the defined operation content. The position and range to be expressed by the contents are determined. However, only one or more of the operation examples described below may be accepted, and other operations may not be accepted, or the contents of operations for specifying other positions and ranges may be determined in advance. You may limit to. That is, each method of the pasting position and pasting range shown below may be used independently.
Hereinafter, in the drawings for explaining the operation content of the screen, the content displayed on the screen is appropriately discarded.

(A) Example 1
8 and 9 illustrate an example of an operation for designating a pasting position and a pasting range. In this example, two points on the screen are designated by the user's two fingers F1 and F2, a predetermined figure (for example, the same figure) with the two points as vertices and a line connecting the two points as a diagonal line (DL). (Rectangle represented by area A). And the center of this area | region is determined as a sticking position. FIG. 9 specifically shows this operation.

First, it is determined whether or not the touch panel TP has been touched (S500). When a touch is detected, it is determined whether or not two points are specified at the same time (two-point simultaneous pressing) (S510). If the two points are pressed simultaneously, the two points are set as vertices, and a rectangle having a diagonal line connecting the two points is drawn with a dotted line (S512). The reason for drawing with a dotted line is to inform the user that this area is provisional and can be changed by further operations. Then, every time the interval of the finger touching the screen is changed by the user, the provisional game drawing process is updated (S512). When all of the user's fingers are separated from the touch panel TP (S514; N), an area determined by the last two points touched is determined as the pasting range (S516). Subsequently, the center of the determined pasting area is determined as the pasting position (S518).
According to this example, it is possible to freely determine the pasting position (position where the sticker is actually displayed) and the pasting range (area defining the display content associated with the sticker) with a simple operation.

(B) Example 2
Another example of the operation for specifying the pasting position and the pasting range will be described with reference to FIGS. 10 and 11. As shown in FIG. 10A, the user first specifies ST1, and then specifies the pasting position. The designation of the sticking position may be performed by tapping the sticker image displayed in the region PRT to confirm the sticker, then releasing the finger and touching one point on the screen. The type of sticker and the sticking position may be determined by a series of operations by dragging the image on the touch panel TP and bringing it to the sticking position. Subsequently, as shown in FIG. 5B, the display control unit 10 determines the size of the temporary area (in the figure, a square) having a predetermined shape (square in the figure) according to the time for which the sticking position is long pressed. A1, A2, A3) will be expanded. And the determination part 40 determines the area | region (A4) at the time of a finger | toe separating as a sticking range like the figure (c).

Specifically, as shown in FIG. 11, when it is detected that a point on the screen is long pressed (S520; Y), the long pressed position is determined as a pasting position (S522). Then, the area of the provisional range is drawn so that the size increases with the elapsed time (S524). When it is detected that the finger is separated (S526; Y), the provisional range drawn at the moment of separation is fixed as the pasting range (S528).

(C) Example 3
Still another example of the operation of designating the pasting position and pasting range will be described with reference to FIGS.
As shown in FIG. 12A, after the user designates ST1, the user touches one point (S) on the screen, and traces the screen so that one closed space is formed by the trajectory TR (the screen remains touched). Change the position of your finger). Subsequently, the contour of the formed closed space is corrected according to a predetermined algorithm. FIG. 2B shows an example of a closed space (solid line) formed by TR and a corrected area A (broken line).

Specifically, as shown in FIG. 13, when it is determined that the detected position is continuously changed, the determination unit 40 determines that an operation of tracing the screen is being performed (S530; Y), and the display control unit. 10 starts drawing the trajectory and updates the drawing of the trajectory in accordance with the movement of the finger (S532). When the determination unit 40 determines that the closed region is formed by the trajectory (534; Y), the trajectory is corrected (S536), and the determination is made based on the trajectory in which the pasting range is corrected (S537). The designation unit 30 determines a pasting position based on the determined pasting range (S538).

(D) Example 4
Still another example of the operation of designating the pasting position and pasting range will be described with reference to FIGS. Here, it is assumed that the content to which the sticker is attached is a web page. Further, as shown in FIG. 14A, it is assumed that this content includes two sub-contents and is arranged in areas CA1 and CA2, respectively. For example, an image is displayed on CA1, and text data is displayed on CA2. Assume that the user taps a point on the content. Then, the designation unit 30 determines that the tap position is the pasting position, specifies the sub-content area CA1 including the tapped position, and determines that the entire area CA1 is the pasting range.

Specifically, as shown in FIG. 15, when the designation unit 30 detects that a point on the screen has been tapped (S540), the determination unit 40 determines that position as a pasting position (S542). Then, the determination unit 40 specifies the sub-content area including the content tap position based on the content, and determines the entire range of the specified area as the pasting range (S544). When there is no sub-content, it is determined that the entire content or the entire portion displayed on the screen of the content is the pasting range. In this example, since the position and the range are automatically determined by a single tap operation, the operation amount of the user related to the pasting process is reduced.

(E) Example 5
Still another example of the operation of designating the pasting position and pasting range will be described with reference to FIGS. In this example, only a part of the process is automated in determining the pasting range. Here, as shown in FIG. 16A, it is assumed that one content includes nine sub-contents, which are arranged in areas CA1 to CA9, respectively. First, the user designates ST1 and designates the pasting position by a tap operation or the like. When the pasting position is confirmed, a candidate for a region that can be designated as the pasting range is notified to the user. Specifically, the drawing mode of the outline of each area is changed (in the figure, it is changed from a solid line to a dotted line). The user designates at least one area candidate to be designated as the pasting range. For example, tap a point in the area to be specified. In the designated area, the drawing mode of the area is changed so that the user can know the designated fact. FIG. 4C shows that areas CA2, CA3, and CA8 are designated.

Specifically, as shown in FIG. 17, when the designation unit 30 detects that the hardware button has been pressed for a long time after selecting a sticker image or sticker displayed on a predetermined area PRT (S550; Y), the designation unit 30 The unit 30 receives an operation by the user and determines the sticking position of the seal (S551). Subsequently, when the browser P7 interprets the display content and confirms the existence of the sub-region, the boundary of all the sub-regions is specified, and the page displayed so that the display manner is different from the specified boundary is rewritten (S552). . A designation of a candidate range is accepted from the user (S554), and if there is a predetermined operation, the designated content is confirmed (S556). In this example, if the pasting position is designated, a pasting range candidate is designated, and the user can select the pasting range from there. Therefore, the operation burden related to the pasting range designation is reduced.

(F) Example 6
When the area is designated by freehand as in the example 4, not all the inner area of the determined area may be determined as the pasting area, but only a part of the area may be determined as the pasting area. Specifically, as shown in FIG. 18, when a map is displayed, landmark information (“B facility”, “A park”, “E store”, “D facility” is displayed in the area drawn by TR. In the case where there are image objects representing “C station”), an area in which these image objects are displayed may be extracted as candidates for the pasting range and selected by the user.

Specifically, as shown in FIG. 19, when an area is designated by freehand as in the case of Example 4 (S530-534; Y), the determination unit 40 sends the area information to the display control unit 10. The display control unit 10 outputs the landmark information located in the area from the application control unit 100 that controls the map application P2, and it can be seen that these landmark information are candidates that can be designated as the pasting range. The display unit 20 is caused to perform a simple drawing process (S560; dotted line area in FIG. 19). The user designates one or more candidate areas registered in association with the seal by tapping one point in the corresponding candidate area (S562). When the pasting range is determined, the determination unit 40 determines the pasting position based on the pasting range (S564). For example, when the selected pasting range is one, the center of the pasting range is determined as the pasting position. When a plurality of pasting ranges are designated, the representative point of each area (for example, the center of the area) is determined as the pasting position. Alternatively, as in the case of FIG. 4, the region drawn by freehand may be corrected to a circle or a rectangle, and the pasting position may be determined based on the corrected figure. For example, the corrected center of gravity position is determined as the pasting position.

(G) Example 7
Still another example of the operation of specifying the pasting position and the pasting range will be described with reference to FIGS. As shown in FIG. 20A, the user first selects one seal and designates one point (Pi) in the screen as a starting point. Subsequently, the sticker is dragged by tracing the position (the path formed by the drag operation is indicated by a dotted line). Then, the user designates the end point (Pd) of the drag operation. In this way, when the route of the drag operation of the seal is determined, the area AF is determined based on the route ((b) in the figure). For example, the area AF is an area having a predetermined width around the route.

Specifically, as shown in FIG. 21, after the user selects a seal from the region PRT, the user taps one point on the screen (S572; Y) and starts a drag operation (S573; Y). It is determined that the tap position is the starting point (S574). Thereafter, the display control unit 10 sequentially draws a route according to the drag operation (S575). When the designation unit 30 detects the end of the drag device (that is, the finger has been removed from the screen) (S576; Y), the determination unit 40 determines that the position is the end point (S577). The determining unit 40 determines the area AF using a predetermined algorithm based on the route, and determines this as the pasting range (S578). Subsequently, the determination unit 40 determines a pasting position based on the pasting range. In determining the pasting position, for example, an intermediate point on the route or the center of gravity of the area AF can be employed (S579).

As described above, when the pasting position and pasting range are determined by the user's operation (S10 and S20 in FIG. 2), a sticker image is drawn and pasting processing information DST is generated. An example of the pasting process information DST is shown in FIG. The sticking process information DST is generated every time a sticker sticking operation is performed, and is associated with an issue number for identifying each sticking operation, and the contents of the seal definition information DST-a determined by the type of sticker And paste information DST-b depending on the operation content. The pasting information DST-b includes data items DI4, DI5, DI6, and DI7. In DI4, information on the date and time when the pasting operation was performed is described. In DI5, an application P (for example, a map application P2 when a sticker is pasted on a map) that is being executed at the time of the pasting operation is described. DI6 and DI7 are generated by processing and editing the determined pasting position and pasting range according to information defined in DI2. After one sticking process information DST is generated for one sticking process in this way, when the sticker already attached is designated by the user or automatically, the sticking process information DST is read and linked. The pasting information DST-b is delivered to the previous application P.

(4) Seal Calling Process Hereinafter, the operation based on the pasted sticker (S100 to S106 in FIG. 3) will be described.
(A) Example 1
First, the user activates the seal browsing application P1. Then, the information of the pasting process information DST is read, and the pasted sticker and its date and time are displayed on the screen (FIG. 23 (a)). In this example, the five most recently attached stickers (ST1 to ST5) are displayed. For example, when the user wants to confirm the contents of the sticker ST3 (the contents of the sticking process is represented by the sticking process information DST11) and designates the sticker with a tap or the like, the display control unit 10 stores the sticker in the storage unit 90 DI6, which is information necessary for drawing a sticker, is determined by reading the stored pasting process information DST11, specifying that the application P as the cooperation destination is the map application P2, and managing the map application P2. , A drawing instruction of a sticker image including the description content of DI7 is output. The application control unit 100 activates the map application P2 and draws a sticker on the map. As a result, as shown in FIG. 6B, a sticker image is drawn at the position {latitude a0, longitude b0}, and the coordinates {a1, b1} and {a2, b2} of the landmark information associated with the sticker are drawn. The drawing process is performed so that can be recognized.

On the other hand, when ST1 (the pasting process content is represented by pasting process information DST12) is designated on the screen of FIG. 11A, information on DI6 and DI7 (that is, HTML file) is specified in the browser P7 designated as the cooperation destination. The coordinate data (300, 200) on the screen for displaying the sticker image is handed over.As a result, as shown in FIG. (C), the map application P2 is activated and the content is displayed according to the HTML file, In this example, by starting the seal browsing application P1, the user can check a pasted sticker, and from the date and time and the type of the seal, It becomes easy to specify the target seal.

(B) Example 2
In this example, the operation based on the automatically attached sticker is executed following the sticking process without starting the sticker browsing application P1. FIG. 24 shows pasting process information DST2, which is another example of pasting process information DST, corresponding to a pasting operation. The sticking process information DST2 includes sticker definition information DST-a2 and sticking information DST-b. The difference between the seal definition information DST-a2 and the seal definition information DST-a is that a data item DI8 is provided for describing information indicating whether or not the sticking process of the seal and the calling process of the seal are continuously performed. This is the point. In the example of the figure, it is defined that it is performed continuously. In the case of “continuous”, a command to immediately start the cooperation destination application P is generated.

FIG. 25 shows a screen when the sticking process for the sticker of the seal definition information DST2-a2 and the calling process of the sticker performed after the sticker is stuck and the sticking process information DST2 is generated are continuously performed. This is an example of transition. Now, assume that a history display application P6 that displays a list of incoming histories is activated in FIG. During the display of this screen, the user performs a predetermined operation to attach the sticker ST5. When the sticking position and sticking range are determined, a sticker ST5 is drawn on the screen ((c) in the figure), and sticking information DST-b is generated according to sticker definition information DST-a2, sticking position and sticking range. Is done. In this example, since the pasting position is designated close to the position where the character string “Oda △ male” is displayed, this character string is delivered by the application control unit 100 to the phone book application P4 that is the cooperation destination application. It will be. When the phone book application P4 is activated by the application control unit 100, the phone book application P4 performs screen display processing based on the pasted paste information DST-b. Specifically, only the information related to “Oda △ male” is read from the phone book data DR3 and displayed. At the same time, the sticker information pasted in association with “○ 田 △ 男” is also displayed at the same time (see display area AD1 in the figure). In this example, since the process of displaying the seal browsing is not performed, information related to the content of the pasting work is provided to the user.

(C) Example 3
FIG. 26 shows an example of the pasting process information DST3. The pasting process information DST3 has the same structure as that of the pasting process information DST, but the contents to be described are different. FIG. 27 is a screen transition diagram for explaining the pasting process and the calling process for the sticker of the pasting process information DST3.
As shown in FIG. 27 (a), it is assumed that the user has pasted the sticker ST6 at a position near “Oda △ male” while the history display application P6 is activated and the incoming call history is displayed. Then, ST6 is drawn, and paste processing information DST3 is generated. Subsequently, since the “operation definition” of the pasting processing information DST3 is “continuous”, the mail application P5 is automatically activated and “○ △ Δ” is delivered to the mail application P5. As a result, the received mail whose sender is “Oda △ male” is extracted and displayed ((c) in the figure). As shown in FIGS. 24 to 27, even if the display content, sticking position, sticking range, and the like at the time of sticking the sticker are the same, the subsequent operation can be made different by changing the type of the sticker.

(D) Example 4
FIG. 28A shows pasting process information DST4 which is another example of pasting process data. The pasting process information DST4 has the same structure as the pasting process information DST2 and pasting process information DST3, but is described as pasting process information DST1 to DST3, etc. in that the application to be linked depends on the “pasting contents”. The contents are different. As shown in FIG. 5B, when the user activates the scheduler P3 and the schedule information is displayed, the sticker is pasted at a desired position (in this example, near the character string “Yurakucho A-kan”). It is assumed that the seal is drawn ((c) in the figure). In this example, since the displayed content is text information, as shown in FIG. 5A, “Yurakucho A-kan” as the pasting position, and all text information in the screen as the pasting range, Each is generated. As for the delivery destination application, the pasting position is determined based on a character string “A hall in Yurakucho”. For example, the determination is made based on the analysis result of the meaning content of “Yurakucho A-kan” and the structure and format (length, number, combination of characters, etc.) of the character string. In the example of the figure, the generation unit 50 determines that the word “Yurakucho” means a place, and as a result, the cooperation destination application is determined to be a map application as shown in FIG. It is shown that.

(E) Example 5
FIG. 29 shows a process in the case where the same type of sticker is applied to different application positions on the same screen as FIG. FIG. 29A shows sticking processing information DST5 generated when sticking using this seal. In the pasting process information DST5, the part of the sticker definition information is the same as DS4, but the part of the pasting information is different. The reason for this will be described with reference to FIGS. As shown in FIG. 5B, the pasting position in this example is in the vicinity of the telephone number. In this case, it is determined from the format of the character string (here, the number string “03-****-****”) that this character string means a phone number, and the cooperation destination application is the phone book. It has been determined that As a result, as shown in FIG. 5C, the phone book application is automatically activated, and information on the contact that matches the character string is displayed. At this time, the sticker attached in association with the continuous production destination is also displayed (see area AD2).
Thus, even if it is the same kind of seal | sticker, subsequent operation | movement can be varied with the content of sticking operations, such as a sticking position.

(F) Example 6
30 shows another example of the pasting process information DST, pasting process information DST6 (FIG. (B)), and seal definition information DST-a3 (see FIG. An example of a)) is shown. The seal definition information DST-a3 with the seal ID “010” is different from the seal definition information DST-a in that a data item DI9 that defines the appearance of the seal is provided. As can be seen from FIG. 6B, in this example, when the web page is displayed on the browser P7, a sticker of the type specified by “010” is pasted, and the pasting range is a content represented by a predetermined URL, The pasting position is defined as address information included in the link destination. Now, assume that the user ends the display by the browser P7, starts the map application P2, and displays a map. Although the execution result of the browser P7 is not displayed on the screen, the browser P7 is activated in the background, and is in a state of performing a process of receiving a request from another application and returning a response. During the execution of the map application P2, the map application P2 requests the link information from the browser P7 at a predetermined timing.

When the user performs an operation for displaying an area including the address on the map, the map application P2 reads the pasted information regarding the position from DR1 together with the map data DR2, and displays the sticker ST10. The At this time, the text information of the link destination is reflected on the appearance of the sticker ST10 according to the information defined in DI9 of the sticking process information DST6. For example, a part of the text information is displayed in a sticker image. This is shown in FIG. (A) and (b) of the figure both display images in the vicinity of the designated address, but are examples of screens when the map application P2 is executed at different times. The appearance of the sticker ST10 may vary depending on the timing of information acquired from the browser P7.
In this way, by acquiring information that can be sequentially updated and reflecting it in the appearance of the sticker, it is possible to select the sticker and activate the application P (such as the browser P7) separately to display the web page. Information can be communicated to the user by the change of.

It should be noted that the attached sticker may be “peeled off”. That is, when the user performs a predetermined operation, the sticking process information DST of the seal is deleted from the storage unit 90.

4). Modifications The above embodiments can be modified as appropriate. Hereinafter, the viewpoint when performing the modification will be exemplified.
In at least one of designation of the sticking position and sticking range of the sticker, voice input may be used instead of input using a finger such as a touch operation or a button press. For example, in the method for determining the pasting position and pasting range shown in FIGS. 10 and 11, after the pasting position is designated using a finger (S522 in FIG. 11), the pasting range is determined based on the long press time. Instead, the size of the pasting range is determined according to the voice collected by the voice input / output unit 70. Specifically, the voice input / output unit 70 collects a user's utterance, performs voice analysis, and calculates a feature amount of the voice. The feature amount is, for example, information related to collected sound volume, pitch, other frequency characteristics, or temporal changes thereof. When receiving the feature amount from the voice input / output unit 70, the determination unit 40 determines the pasting range based on the feature amount. As an example of the method for determining the pasting range, the provisional pasting range is increased while a predetermined sound volume is detected, and the pasting range is determined when it is detected that the voice is interrupted. Alternatively, a temporary area having a size corresponding to the input volume is displayed, and an area having a size corresponding to the maximum volume measured by the voice input / output unit 70 in a predetermined period is determined as the pasting range.

Similarly, the selection of a seal may be performed using voice instead of a finger. For example, the type name of the seal is designated by voice input. Alternatively, the designation of a seal using a finger may be canceled or changed by voice input. For example, in the example of FIGS. 10 and 11, while the user designates a seal with a finger and drags it to a desired application position, the type name of the seal after the change or the designated seal is changed. Say the command to do. When the speech input / output unit 70 detects and interprets this utterance, the speech input / output unit 70 notifies the determination unit 40 of the content, and the determination unit 40 cancels or changes the type of the designated seal. May be performed.

Also, as in the examples of FIGS. 20 and 21, even when the pasting range depends on the start point and the end point, the pasting range can be determined by voice input. For example, when applying a sticker on a map, first, a sticker to be attached is selected using a finger. Subsequently, the user designates a pasting range by speaking a predetermined word. For example, when “route” is uttered, the voice input / output unit 70 notifies the determination unit 40 that the map input / output unit 70 uses the information related to the route that the map application P2 has as a pasting range designation method. Subsequently, when the user continues the drag operation and designates the end point, the determination unit 40 obtains information about a traffic route such as a road or a track corresponding to the drag operation trajectory from the map application P2, and based on this information. To determine the pasting range. For example, when the drag operation trajectory is compared with the position of the road and it is determined that the drag operation is performed along the road based on a predetermined matching algorithm, the portion from the start point to the end point on the corresponding map road is It is determined that it is within the pasting range. Thereby, even if the locus | trajectory of drag | drug operation is distorted, the sticking range becomes an orderly shape according to map information.

When the map application P2 has a function for searching and displaying a traffic route corresponding to the designated starting point and ending point, the conditions for the route (that is, the pasting range) may be further specified by voice input. For example, when the user utters “route by train” and designates the starting point and the ending point, the determination unit 40 determines the optimum route for traveling by train from the starting point to the ending point from the map application P2, and displays it on the screen. indicate. When the user performs a predetermined confirmation operation, the determination unit 40 determines that the displayed route is the pasting range. On the other hand, in the case of speaking “route on foot”, it is determined that an appropriate route for moving on foot from the starting point to the end point is the pasting range.

In this embodiment, how the function of the communication terminal 1 is implemented in hardware or software is not limited to the example shown in FIG. For example, some of the functions of the communication terminal 1 may be performed by another node using a network. In short, the information processing apparatus of the present invention includes a display unit having a display area, a designation unit that accepts designation of a position in the display area, a determination unit that determines a range in the display area including the position, The content is displayed on the display unit, the display control means for drawing the image object at the determined position, and the designation based on the display content corresponding to the determined range of the content displayed on the display unit It is only necessary to have generating means for generating information associated with the image object. Programs that implement this function on the hardware installed in the communication terminal 1 include magnetic recording media (magnetic tape, magnetic disk (HDD (Hard Disk Drive), FDD (Flexible Disk Drive), etc.)), optical recording medium (optical disk). (CD (Compact Disc), DVD (Digital Versatile Disc), etc.), a magneto-optical recording medium, a computer-readable recording medium such as a semiconductor memory, or the like, or can be downloaded via a network.

Claims

A display unit having a display area;
Designation means for accepting designation of a position in the display area;
Determining means for determining a range in the display area including the position;
Display control means for displaying content on the display unit and drawing an image object at the determined position;
An information processing apparatus comprising: a generating unit configured to generate information associated with the designated image object based on display content corresponding to the determined range of the content displayed on the display unit.
The designation means accepts designation of two positions,
The information processing apparatus according to claim 1, wherein the determination unit determines that a rectangular region including the two specified points as vertices is the range.
The information processing apparatus according to claim 1, wherein the determination unit determines the center of the range as the position, and determines the size of the range based on a length of a period in which the position is specified. .
The designation means continuously accepts designation of a plurality of positions,
2. The information according to claim 1, wherein the determination unit determines a boundary of the range based on a trajectory represented by the designated plurality of positions, and determines the inside of the boundary as a previous range. Processing equipment.
The determining means determines an area including the position based on the content displayed on the display unit,
The information processing apparatus according to claim 1, wherein the display control unit determines an inside of the determined area as the range.
A second specifying means for receiving an image object specification;
The determining unit may include one or more based on the content displayed on the display unit when the length of the period in which the image object is specified by the second specifying unit is equal to or greater than a predetermined length. Determine candidate areas,
The display control means displays an image representing the determined one or more region candidates on the display unit,
The information processing apparatus according to claim 1, wherein the determining unit determines, as the range, an area that includes a position specified by the specifying unit among the displayed one or more area candidates.
The designation means continuously accepts designation of a plurality of positions,
The determining means determines a boundary of the range based on a trajectory represented by the designated plurality of positions, and is an area included inside the boundary, the content being displayed on the display unit The information processing apparatus according to claim 1, wherein one or more areas specified based on content are determined as the range.
On the computer,
Receiving a designation of a position in the display area of the display unit;
Determining a range within the display area that includes the position;
Displaying content on the display unit, and drawing an image object at the determined position;
Generating information to be associated with the image object based on display content corresponding to the determined range of the content displayed on the display unit.
Receiving a designation of a position in the display area of the display unit;
Determining a range within the display area that includes the position;
Displaying content on the display unit, and drawing an image object at the determined position;
An information processing method comprising: a generation step of generating information associated with the image object based on display content corresponding to the determined range of content displayed on the display unit.
A display unit having a display area;
First designation means for accepting designation of a position in the display area;
A second designation means for accepting designation of an image object;
Determining means for determining a range in the display area including the position;
First display control means for displaying the content on the display unit and drawing the designated image object at the determined position, and display corresponding to the determined range of the content displayed on the display unit Generating means for generating information associated with the designated image object based on the content; information generated in association with the designated image object; the content; and information representing the determined position; A storage unit for storing
The image object associated with the generated information is displayed on the display unit, and when the displayed image object is designated, the display of the content and the image are performed according to the information stored in association with the image object. An information processing apparatus comprising: second display control means for causing the first display control means to perform drawing of an object.