JP2015170228A - Data processor, operation reception method, and content display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To diversify a multi-touch operation.SOLUTION: A data processor includes: a gesture event determination section for determining any one of a plurality of gesture events on the basis of one or more positions detected by a touch panel; and a display section determination section 72 for changing a display section on the basis of the gesture event determined during a gesture period in which the gesture events are continuously determined. The display section determination section 72 includes: a scale determination section 13 for determining a scale showing a degree to change the display section on the basis of the interval of two positions detected by the touch panel when a prescribed gesture event determined on the basis of the two positions is detected; and a conversion section 11 for converting the display section corresponding to an image for display displayed on a display at a point of time at which the gesture period is started by the determined scale.

Description

  The present invention relates to a data processing device, an operation reception method, and a content display program, and more particularly to a data processing device including a multi-touch touch panel, an operation reception method and a content display program executed by the data processing device.

  In recent years, in a data processing apparatus represented by a multifunction peripheral (MFP), as a user interface, there is an apparatus provided with a multi-touch compatible touch panel capable of detecting a plurality of positions designated by a user at the same time. For example, although it is a portable information device, Japanese Patent Application Laid-Open No. 2000-163031 discloses an electronic book including a display unit capable of displaying a map image, and the map is moved by moving two fingers in contact with the display unit. An electronic book is described in which an image enlargement instruction and an enlargement amount can be input, and a map image reduction instruction and a reduction amount can be input by moving two fingers close together.

Since the minimum value and the maximum value of the distance between the two fingers are fixed, there is a limit to the distance to move the two fingers. For this reason, in the conventional portable information device, the user may have to perform the same operation a plurality of times before the map image is enlarged or reduced to a desired size. For example, when the amount of enlargement or reduction is small relative to the distance to move two fingers, the same operation must be repeated several times, and conversely to the distance to move two fingers. When the enlargement amount or the reduction amount is large, there is a problem that it becomes difficult to adjust the distance to move the two fingers to a desired size in order to finely adjust the enlargement amount or reduction amount.
JP 2000-163031 A

  The present invention has been made to solve the above-described problems, and one of the objects of the present invention is to provide a data processing apparatus in which multi-touch operations are diversified.

  Another object of the present invention is to provide an operation receiving method in which multi-touch operations are diversified.

  Still another object of the present invention is to provide an operation reception program in which multi-touch operations are diversified.

  In order to achieve the above-described object, according to one aspect of the present invention, a data processing apparatus uses a display unit capable of displaying an image and a display image corresponding to a display part which is at least a part of the content as the display unit. In response to the display control means for displaying, the position detection means capable of detecting one or more positions indicated by the user on the display surface of the display means, and the one or more positions detected by the position detection means, Gesture event determination means for determining any one of a plurality of gesture events based on one or more positions, and a gesture event determined during a gesture period in which gesture events are continuously determined by the gesture event determination means Display portion determining means for changing the display portion based on the In the case where the gesture event determined by the event determination unit is a predetermined gesture event determined based on the two positions detected by the position detection unit, the degree of change of the display portion based on the interval between the two positions is indicated. Scale determining means for determining a scale, and conversion means for converting a display portion corresponding to the display image displayed on the display means at the time when the gesture period starts, with the determined scale.

  According to this aspect, a display image corresponding to the display portion of the content is displayed, and a gesture event is determined in response to detection of one or more positions indicated by the user, based on the two positions. When a predetermined gesture event to be determined is determined, a scale indicating a degree of changing the display portion is determined based on an interval between two positions, and the display portion corresponding to the display image is converted with the determined scale. The For this reason, the part displayed in content can be changed by changing the space | interval of two positions to instruct | indicate. As a result, it is possible to provide a data processing device in which multi-touch operations are diversified.

  Preferably, the scale determining means determines a larger scale as the interval between the two positions is larger.

  According to this aspect, the wider the interval between the two indicated positions, the greater the rate of change in the display portion. For this reason, the number of operations may be reduced.

  Preferably, the scale determining means determines a smaller scale as the interval between the two positions is larger.

  According to this aspect, the wider the interval between the two indicated positions, the smaller the rate of change in the display portion. For this reason, the user can easily make fine adjustments.

  Preferably, the scale determining unit determines a different scale for each type of gesture event determined by the gesture event determining unit.

  According to this aspect, since a different scale is determined for each type of gesture event, the type of change in the display portion can be made different for each gesture event.

  Preferably, when the gesture event determining unit determines the gesture event based on the temporal change in the interval between the two positions, the scale determining unit has a larger scale as the remaining distance that can change the interval between the two positions is larger. To decide.

  According to this aspect, the ratio of changing the display portion can be changed according to the interval between the designated positions with respect to the operation for changing the interval between the two designated positions.

  Preferably, when the gesture event is determined based on a temporal change in the relative direction of the two positions by the gesture event determining unit, the scale determining unit determines a larger scale as the interval between the two positions is larger. .

  According to this aspect, it is possible to vary the ratio of changing the display portion according to the interval between the designated positions with respect to the operation of changing the relative direction between the two designated positions.

  Preferably, when the gesture event determination unit determines the gesture event based on a temporal change in the relative direction of the two positions, the scale determination unit determines the smaller scale as the interval between the two positions is larger. .

  According to this aspect, it is possible to vary the ratio of changing the display portion according to the interval between the designated positions with respect to the operation of changing the relative direction between the two designated positions.

  Preferably, when the display period includes the front image at the end of the gesture period, the display part determining unit determines that the display part is the largest size of the display part in the area surrounded by the line in the content image. Further included is a correcting means for correcting to an area approximated by.

  According to this aspect, if the display portion is sandwiched between lines in the content image, the region is corrected to a region that is closest in size to the display portion. For this reason, all of the portions surrounded by the lines can be included in the display portion. For example, when a character is present in a region sandwiched between lines, the entire character can be displayed.

  Preferably, the gesture event determining means determines a gesture event at a predetermined time interval during the gesture period, and the display portion determining means is detected by the position detecting means when the predetermined gesture event is determined by the gesture event determining means. Timing determining means for determining a timing for determining the display portion at a time interval longer than the predetermined time interval based on the interval between the two positions to be further included.

  According to this aspect, based on the interval between the two positions, the timing for determining the display portion is determined as a time interval longer than a predetermined time interval at which the gesture event is determined. For this reason, since the frequency | count of producing | generating the display image reduces, the response time which reduces the processing load and displays the display image can be shortened.

  Preferably, the capture portion determining means for determining the capture portion including the display portion corresponding to the display image displayed on the display means at the time when the gesture period starts, and the content according to the determination of the capture portion And a temporary storage control means for temporarily storing the generated captured image, wherein the display part determining means is used as a scale. A capture rate determining means for determining a capture rate indicating a degree of the size of the capture portion; the capture portion determining means determines a capture portion having a size determined by the determined capture rate; and the display image generating means is temporarily stored. Based on captured images It generates a display image corresponding to the new display area determined by radical 113 determining means.

  According to this aspect, since the capture rate indicating the degree of the size of the captured image is determined based on the interval between the two positions, the degree of changing the size of the display portion is changed based on the interval between the two positions. The size of the captured image can be changed according to the degree of change in the size of the display portion. In other words, the size of the captured image can be prevented from becoming smaller than the size of the display portion, and the size of the captured image can be prevented from becoming too large relative to the display portion. As a result, the size of the captured image can be set to an appropriate size.

  According to another aspect of the present invention, an operation reception method is an operation reception method executed by a data processing apparatus including a display unit capable of displaying an image, and corresponds to a display portion that is at least a part of content. A display control step for displaying an image for display on the display means, a position detection step for detecting one or more positions indicated by the user on the display surface of the display means, and at least one position in the position detection step. A gesture event determining step for determining any one of a plurality of gesture events based on one or more positions in response to detection, and a gesture period in which gesture events are continuously determined in the gesture event determining step The display part determination step for changing the display part based on the gesture event determined during And the display part determining step includes two positions when the gesture event determined in the gesture event determining step is a predetermined gesture event determined based on the two positions detected in the position detecting step. A scale determination step for determining a scale indicating a degree of changing the display portion based on the interval, and a display portion corresponding to the display image displayed on the display means at the time when the gesture period starts with the determined scale. A conversion step for converting.

  If this aspect is followed, the operation reception method which diversified multi-touch operation can be provided.

  According to another aspect of the present invention, the operation reception program is an operation reception program that is executed by a computer that controls a data processing device including display means capable of displaying an image, and is at least a part of the content. In the display control step of displaying the display image corresponding to the display portion on the display means, the position detection step capable of detecting one or more positions designated by the user on the display surface of the display means, and 1 in the position detection step In response to detection of the above positions, a gesture event determining step for determining any one of a plurality of gesture events based on one or more positions, and gesture events are continuously determined in the gesture event determining step. Based on gesture events determined during the gesture period to be A display portion determination step for changing the display portion; and causing the computer to execute a display portion determination step, wherein the gesture event determined in the gesture event determination step is determined based on the two positions detected in the position detection step. In the case of a predetermined gesture event, a scale determining step for determining a scale indicating a degree of changing the display portion based on an interval between two positions, and a display image displayed on the display means when the gesture period starts Converting a display portion corresponding to で at a determined scale.

  If this aspect is followed, the operation reception program which diversified multi-touch operation can be provided.

1 is a perspective view showing an appearance of an MFP in one embodiment of the present invention. It is a top view which shows an example of an operation panel. 2 is a block diagram illustrating an example of an outline of a hardware configuration of an MFP. FIG. 2 is a diagram illustrating an example of a software architecture of a CPU provided in an MFP. FIG. 3 is a block diagram illustrating an example of functions of a CPU included in an MFP together with information stored in an HDD. FIG. It is a figure which shows an example of a master table. It is a block diagram which shows an example of a detailed structure of a display part determination part. It is a 1st figure which shows an example of a content. It is a 1st figure which shows an example of the image for a display. It is a 2nd figure which shows an example of a content. It is a 2nd figure which shows an example of the image for a display. It is a 1st flowchart which shows an example of the flow of a gesture event determination process. It is a 2nd flowchart which shows an example of the flow of a gesture event determination process. It is a flowchart which shows an example of the flow of a content display process. It is a flowchart which shows an example of the flow of a multi-touch corresponding | compatible process. It is a flowchart which shows an example of the flow of a rotation process. It is a flowchart which shows an example of the flow of a scaling process. It is a flowchart which shows an example of the flow of a movement process.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  FIG. 1 is a perspective view showing an external appearance of an MFP according to one embodiment. FIG. 2 is a plan view showing an example of the operation panel. Referring to FIGS. 1 and 2, MFP 100 includes a document reading unit 130 for reading a document, an automatic document feeder 120 for conveying a document to document reading unit 130, and document reading unit 130 reads a document. An image forming unit 140 for forming an image on a sheet or the like based on image data to be output, a sheet feeding unit 150 for supplying the sheet to the image forming unit 140, and an operation panel 160 as a user interface. .

  Operation panel 160 is provided on the upper surface of MFP 100. Operation panel 160 includes a display unit 161, a touch panel 165, and a hard key unit 167. The display unit 161 is, for example, a liquid crystal display device (LCD), and displays an instruction menu for a user, information about acquired image data, and the like. The hard key portion 167 includes four hard keys 167A to 167D each representing characters of “BOX”, “FAX”, “COPY”, and “ECO”. The touch panel 165 is a multi-touch compatible touch panel provided on the upper surface or the lower surface of the display unit 161 so as to overlap the display unit 161, and detects a position designated by the user on the display surface of the display unit 161. The operation in which the user instructs the touch panel 165 includes a multi-touch operation for instructing the touch panel 165 with a finger at the same time, and a single touch operation for instructing one place with a finger at a time.

  The automatic document feeder 120 automatically conveys a plurality of documents set on the document feeding tray one by one to a predetermined document reading position set on the platen glass of the document reading unit 130, and reads the document. The document whose document image is read by the unit 130 is discharged onto a document discharge tray. The document reading unit 130 includes a light source that irradiates light to the document conveyed to the document reading position and a photoelectric conversion element that receives light reflected from the document, and scans a document image corresponding to the size of the document. The photoelectric conversion element converts the received light into image data that is an electrical signal and outputs the image data to the image forming unit 140. The paper feed unit 150 conveys the paper stored in the paper feed tray to the image forming unit 140.

  The image forming unit 140 forms an image by a well-known electrophotographic method, and the image data after data processing is obtained by performing various data processing such as shading correction on the image data input from the document reading unit 130. Alternatively, an image is formed on a sheet conveyed by the sheet feeding unit 150 based on image data received from the outside.

  FIG. 3 is a block diagram illustrating an example of an outline of the hardware configuration of the MFP. Referring to FIG. 3, MFP 100 includes a main circuit 110. The main circuit 110 includes a CPU 111, a communication interface (I / F) unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory) 114, and a hard disk drive (HDD) 115 as a mass storage device. A facsimile unit 116 and an external storage device 117. CPU 111 is connected to automatic document feeder 120, document reading unit 130, image forming unit 140, sheet feeding unit 150, and operation panel 160, and controls the entire MFP 100.

  The ROM 113 stores a program executed by the CPU 111 or data necessary for executing the program. The RAM 114 is used as a work area when the CPU 111 executes a program. Further, the RAM 114 temporarily stores the read images continuously sent from the document reading unit 130.

  The facsimile unit 116 is connected to a public switched telephone network (PSTN), and transmits facsimile data to or receives facsimile data from the PSTN (Public Switched Telephone Networks). The facsimile unit 116 stores the received facsimile data in the HDD 115 or converts the received facsimile data into print data that can be printed by the image forming unit 140 and outputs the print data to the image forming unit 140. As a result, the image forming unit 140 forms an image on a sheet of facsimile data received by the facsimile unit 116. Further, the facsimile unit 116 converts the data read by the document reading unit 130 or the data stored in the HDD 115 into facsimile data, and transmits the facsimile data to a facsimile machine connected to the PSTN.

  Communication I / F unit 112 is an interface for connecting MFP 100 to a network. The communication I / F unit 112 communicates with other computers connected to the network using a communication protocol such as TCP (Transmission Control Protocol) or FTP (File Transfer Protocol). Note that the protocol for communication is not particularly limited, and an arbitrary protocol can be used. The network to which the communication I / F unit 112 is connected is, for example, a local area network (LAN), and the connection form may be wired or wireless. The network is not limited to a LAN, and may be a wide area network (WAN), a network using a public switched telephone network, or the like. Furthermore, the network is connected to the Internet. Therefore, MFP 100 can communicate with a computer such as a server connected to the Internet.

  The external storage device 117 is controlled by the CPU 111, and a CD-ROM (Compact Disk Read Only Memory) 118 or a semiconductor memory is mounted. The CPU 111 can access the CD-ROM 118 or the semiconductor memory via the external storage device 117. The CPU 111 loads the program recorded in the CD-ROM 118 or semiconductor memory mounted on the external storage device 117 into the RAM 114 and executes it. Note that the program executed by the CPU 111 is not limited to the program recorded on the CD-ROM 118, and the program stored in the HDD 115 may be loaded into the RAM 114 and executed. In this case, another computer connected to the network rewrites the program stored in HDD 115 of MFP 100 or adds a new program and writes it via the network connected to communication I / F unit 112. You may do it. Further, MFP 100 may download a program from another computer connected to the network and store the program in HDD 115. The program here includes not only a program directly executable by the CPU 111 but also a source program, a compressed program, an encrypted program, and the like.

  The medium for storing the program executed by the CPU 111 is not limited to the CD-ROM 118, but an optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC card, optical card, It may be a semiconductor memory such as a mask ROM, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM).

  Operation panel 160 includes a display unit 161 and an operation unit 163. The display unit 161 is a display such as a liquid crystal display (LCD) or an organic ELD (Electro-Luminescence Display), and displays an instruction menu for the user, information about acquired image data, and the like. The operation unit 163 includes a touch panel 165 and a hard key unit 167 including a plurality of keys. Each of the plurality of keys included in the hard key unit 167 includes a contact switch and is connected to the CPU 111. When the hard key is pressed by the operation user, the contact is closed and the circuit connected to the CPU 111 is closed. The hard key closes the circuit while being pressed by the operating user who operates the MFP 100, and opens the circuit when not pressed by the operating user.

  When a plurality of keys included in the hard key unit 167 are pressed, the operation unit 163 receives input of data such as instructions, characters, and numbers corresponding to the pressed keys. The touch panel 165 is provided on the upper surface or the lower surface of the display unit 161, and outputs the coordinates of the position designated by the operation user to the CPU 111. The touch panel 165 detects a position designated by the operation user with a finger or a stylus pen, and outputs the coordinates of the detected position to the CPU 111. The touch panel 165 is a multi-touch compatible touch panel, and outputs a plurality of coordinates corresponding to a plurality of positions simultaneously designated by the user to the CPU 111 when the user inputs a multi-touch operation. Further, the touch panel 165 outputs coordinates corresponding to a single position designated by the user to the CPU 111 when the user inputs a single touch operation.

  The touch panel 165 is preferably the same size or larger than the display surface of the display unit 161. Since the touch panel 165 is provided so as to overlap the display unit 161, the touch panel 165 indicates the coordinates of the position indicated by the operation user on the display surface of the display unit 161 when the operation user indicates the display surface of the display unit 161. It outputs to CPU111. For the touch panel 165, for example, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, or a capacitance method can be used, and the method is not limited.

  The operations that can be accepted by the operation unit 163 include an operation of pressing a plurality of keys of the hard key unit 167, a single touch operation that instructs the touch panel 165, and a multi-touch operation. Further, the hard key section 167 includes four hard keys 167A to 167D each representing characters of “BOX”, “FAX”, “COPY”, and “ECO”. Three hard keys 167A, 167B, and 167C, each of which represents the characters “BOX”, “FAX”, and “COPY”, are process switches to which a process switch operation for instructing the MFP 100 to switch processes is assigned. Key. A hard key 167D having characters “ECO” is a mode switching key to which a mode switching operation for instructing the MFP 100 to switch from the normal mode to the power saving mode having lower power consumption than the normal mode is assigned. is there. The operation unit 163 receives a process switching operation when any of the hard keys 167A, 167B, and 167C is pressed, and receives a mode switching operation when the hard key 167D is pressed.

  FIG. 4 is a diagram illustrating an example of a software architecture of a CPU provided in the MFP. Referring to FIG. 4, an operating system (OS) layer is formed in CPU 111, and an MFP process layer is formed in a layer above it. An application platform (PF) layer and an application layer are formed in this order on the MFP process layer.

  The OS layer includes a task for the CPU 111 to execute the OS program. A task belonging to the OS layer executes processing for controlling hardware resources of the MFP 100. Here, the hardware resources include the communication I / F unit 112, the ROM 113, the RAM 114, the HDD 115, the facsimile unit 116, the external storage device 117, the automatic document feeder 120, the document reading unit 130, the image forming unit 140, and the paper feeding unit 150. The operation panel 160 is included. A task belonging to the OS layer controls hardware resources in accordance with an operating command input from the MFP process layer. Further, the OS layer shares operation identification information for identifying each of a plurality of types of operations that can be accepted by the operation unit 163, which is one of hardware resources, with the MFP process layer. When the operation unit 163 detects an operation by the operating user, the task belonging to the group outputs operation identification information indicating the detected operation to the MFP process layer.

  The application layer includes a task for the CPU 111 to execute an application program. When a plurality of types of application programs are installed in MFP 100, a plurality of tasks that respectively execute the plurality of types of application programs may belong to the application layer.

  The task for executing the application program executes a plurality of types of processing determined by the application program. The multiple types of processing include processing for causing a task belonging to the MFP process layer to execute processing executed in the MFP process layer. The task that executes the application program outputs an application command when the task belonging to the MFP process layer executes processing. The application command is a command published as an API (Application Program Interface). Therefore, an application program that causes MFP 100 to execute processing can be easily created. “Public” means a state in which a third party other than the manufacturer that manufactures the MFP 100 can use it. Therefore, a third party can develop an application program that can be installed in MFP 100 using an application command.

  Further, the task for executing the application program specifies a process to be executed from among a plurality of types of processes defined by the application program based on an operation input by an operation user operating MFP 100, and executes the process. An operation input by an operation user who operates MFP 100 is accepted in the OS layer when the operation user operates operation unit 163 to input.

  The application PF layer is arranged between the application layer and the MFP process layer, and arbitrates a plurality of tasks belonging to the application layer, and a task for controlling an application command output by the plurality of tasks belonging to the application layer belongs. Specifically, the application PF layer accepts application commands output by each of a plurality of tasks belonging to the application layer, standardizes the received application commands by converting them into internal commands according to a predetermined rule, The command is output to the MFP process layer. For this reason, it is possible to cope with a case where versions are different among a plurality of application programs. In addition, the application PF layer determines any one of a plurality of tasks belonging to the application layer as a current state. Then, an operation received from the task belonging to the MFP process layer and received from the OS layer is output to the current task determined as the current state among a plurality of tasks belonging to the application layer.

  The operations input by the user accepted in the OS layer include a single touch operation in which the touch panel 165 is instructed with one finger and a multi-touch operation in which the touch panel 165 is instructed simultaneously with two or more fingers. The single touch operation and the multi-touch operation are detected by the operating system as touch events, respectively, and input from the application PF layer to the current task among a plurality of tasks belonging to the application layer. One or more touch events are converted into gesture events in the MFP process layer, and input from the application PF layer to the current task among a plurality of tasks belonging to the application layer. For this reason, the application layer shares one or more gesture events that can be accepted by the application program with the MFP process layer, and the task for executing the application program is executed from the task belonging to the application PF layer by the application program. One or more acceptable gesture events are input. The task that executes the application program executes a process corresponding to either a gesture event or a touch event among a plurality of types of processes.

  Application commands and internal commands are associated in advance. For example, a command correspondence table in which application commands and internal commands are associated with each other may be stored. An application command and an internal command may correspond one-to-one, or a set of two or more internal commands may correspond to one application command. Furthermore, one internal command or a set of a plurality of internal commands may correspond to a plurality of application commands having different versions. For this reason, it is possible to cope with a case where versions are different among a plurality of application programs. The internal command is a command that depends on the hardware resources of MFP 100. The internal command is not normally disclosed, but may be disclosed.

  The MFP process layer is arranged between the application PF layer and the OS layer, and a task for the CPU 111 to execute the MFP process program belongs to it. The MFP process layer converts an internal command output by a task belonging to the application PF layer into an operating command interpretable by the task belonging to the OS layer, and converts the operating command to a task belonging to the OS layer in order to control hardware resources. Output. In actuality, internal commands may be converted into one or more operating commands that can be executed by tasks belonging to the OS layer, but for the sake of explanation, internal commands and tasks belonging to the OS layer can be executed. The relationship with one or more operating commands will be described as one-to-one.

  As described above, the application program is an application program for causing MFP 100 to execute browsing processing, copy processing, scanning processing, printing processing, facsimile transmission / reception processing, data transmission processing, and the like. Among the processes executed by the CPU 111, the part related to the process for controlling the operation panel 160 will be mainly described.

  FIG. 5 is a block diagram illustrating an example of functions of the CPU included in the MFP, together with information stored in the HDD. The functions shown in FIG. 5 are functions realized by CPU 111 when CPU 111 provided in MFP 100 executes a program stored in ROM 113, HDD 115, or CD-ROM 118. Specifically, the CPU 111 is realized by the CPU 111 by executing an OS program, an MFP process program, an operation reception program, and an application program.

  Referring to FIG. 5, CPU 111 includes an application unit 51, an arbitration unit 53, an operation conversion unit 55, and an operating unit 57. The application unit 51 belongs to the application layer in the software architecture shown in FIG. The application unit 51 includes a browsing unit 61 that executes a browsing program that is an example of a content display program.

  The arbitration unit 53 and the operation conversion unit 55 are functions realized by a task in which the CPU 111 executes the operation reception program. The arbitration unit 53 belongs to the application PF layer in the software architecture shown in FIG. The operation conversion unit 55 belongs to the application PF layer in the software architecture shown in FIG. Although all the functions of the operation conversion unit 55 are functions of the application PF layer, at least some of the functions of the operation conversion unit 55 are functions of the application PF layer, and other parts are functions of the MFP process layer. Also good.

  When there are a plurality of tasks for executing the application program in the application unit 51, the arbitration unit 53 determines any one of the plurality of tasks as the current state. Hereinafter, the task that the arbitration unit 53 has determined to be in the current state is referred to as a current task. In the present embodiment, since the application unit 51 includes only the browsing unit 61, the browsing unit 61 is determined to be in the current task state.

  The arbitration unit 53 receives the application command output from the browsing unit 61 and outputs application identification information for identifying the task that has output the application command and the application command to the operation conversion unit 55.

  When a plurality of application commands are output from the browsing unit 61, the arbitrating unit 53 determines the order of the applications, and outputs the application commands to the operation conversion unit 55 in the determined order. For example, when a plurality of application commands cannot be executed simultaneously, the other application command is output to the operation conversion unit 55 after the execution of one application command by the operation conversion unit 55 is completed. Also, if the other app command cannot be executed until after one app command has been executed, the other app command can be executed even if the other app command is input prior to the other app command. Is output first.

  The operation conversion unit 55 receives the application command from the arbitration unit 53 and standardizes the application command by converting the application command into an internal command according to the command correspondence table. The command correspondence table associates one or more internal commands with one application command. The application commands included in the command correspondence table may include the same type of application commands with different versions. In this case, each different version of the application command is associated with one or more internal commands. For this reason, it is possible to install an application program in which application commands of different versions are described. When a new application command appears, the command correspondence table is updated with a command correspondence table in which one or more internal commands are associated with the newly appeared application command. Therefore, an application program describing a new application command can be installed.

  Further, the operation conversion unit 55 converts the standardized internal command into an operating command, and outputs the operating command to the operating unit 57. The operating command is a command that is predetermined between the operation conversion unit 55 and the operating unit 57 and that can be interpreted by the operating unit 57.

  The operating unit 57 is a function executed by a task for the CPU 111 to execute the OS program. The operating unit 57 belongs to the OS layer in the software architecture shown in FIG.

  The operating unit 57 receives an operating command output from the operation conversion unit 55 and controls hardware resources according to the operating command. As the hardware resources, here, the display unit 161, the touch panel 165 of the operation unit 163, and the HDD 115 will be described as an example. The operating unit 57 controls the display unit 161 to display an image on the display unit 161. In addition, the operating unit 57 controls the HDD 115 to store data in the HDD 115 and reads data stored in the HDD 115. Further, the operating unit 57 causes the touch panel 165 to detect a position designated by the user at a predetermined time interval, and acquires position information from the touch panel 165 at a predetermined time interval.

  Touch panel 165 detects the position instructed by the operation user when the operation user indicates the display surface of display unit 161, and outputs position information indicating the position of the display surface of display unit 161 to CPU 111. When the position information is input from the touch panel 165, the operating unit 57 outputs the position information to the operation conversion unit 55. The operating unit 57 may receive two or more pieces of position information from the touch panel 165 at a time. For example, when the operating user designates two different parts of the display surface of the display unit 161 with two fingers at the same time, two pieces of position information indicating the two positions of the designated display surface are received. When the operating unit 57 receives two or more pieces of position information from the touch panel 165 at a time, the operating unit 57 outputs the two or more pieces of position information to the operation conversion unit 55.

  The operation conversion unit 55 includes an association unit 81, a gesture event determination unit 83, a touch event generation unit 85, and a display unit control unit 87. The associating unit 81 associates each of a plurality of types of processing determined by the browsing program with one of a plurality of gesture events at the stage where the browsing program is installed. Specifically, the associating unit 81 generates a master table in which a gesture event that can be accepted by the browsing program is associated with each of a plurality of pieces of processing identification information determined by the browsing program, and stores the master table in the HDD 115.

  FIG. 6 is a diagram illustrating an example of the master table. Referring to FIG. 6, master table 91 includes a master record for each of a plurality of process identification information. The master record includes a process item and a gesture item. Processing identification information is set in the processing item, and gesture identification information for identifying a gesture event is set in the gesture item.

  For example, in the master record in which the process identification information “page feed” is set in the process item, the gesture identification information “swipe” is set in the gesture item. The process specified by the process identification information “page feed” is a process of changing the screen to the screen of the next page and displaying it. The gesture event specified by the gesture identification information “Swipe” corresponds to a user's operation of moving the finger to any one of up, down, left and right at a speed faster than a predetermined speed while instructing the touch panel 165. In the operation of moving the finger at a speed faster than a predetermined speed, the direction in which the finger is moved is determined. Therefore, the gesture event specified by the gesture identification information “swipe” includes the direction in which the finger is moved as a parameter.

  In the master record in which the process identification information “enlarge” is set in the process item, the gesture identification information “pinch out” is set in the gesture item. The process specified by the process identification information “enlargement” is a process for enlarging and displaying the screen. The gesture event specified by the gesture identification information “pinch out” corresponds to an operation of moving at least one of the designated two locations so that the interval between the two locations becomes long while designating the two locations on the touch panel 165. Since the operation of moving so that the interval between the two places becomes longer, the interval between the two places changes, the gesture event specified by the gesture identification information “pinch out” has the ratio of the change between the two places as a magnification. Include in parameter.

  In the master record in which the process identification information “reduction” is set in the process item, the gesture identification information “pinch-in” is set in the gesture item. The process specified by the process identification information “reduction” is a process of reducing the screen for display. The gesture event specified by the gesture identification information “pinch-in” corresponds to an operation of moving at least one of the designated two locations so that the interval between the two locations is shortened while designating the two locations on the touch panel 165. Since the operation of moving so that the interval between the two places becomes shorter, the interval between the two places changes, the gesture event specified by the gesture identification information “pinch-in” is a parameter with the ratio of the change between the two places as a magnification. Included.

  In the master record in which the process identification information “rotation” is set in the process item, the gesture identification information “rotation” is set in the gesture item. The process specified by the process identification information “rotation” is a process of rotating the screen and changing the screen direction. The gesture event specified by the gesture identification information “Rotation” is an operation to move at least one of the two designated locations while changing the orientation of a straight line connecting the two locations while designating two locations on the touch panel 165. Detected when input. For example, this is an operation of drawing an arc on one of two places as the center. Since there are clockwise and counterclockwise directions in which the arc is drawn, the gesture event specified by the gesture identification information “rotation” is included in a parameter whose direction is either clockwise or counterclockwise.

  In the master record in which the process identification information “scroll” is set in the process item, the gesture identification information “flick” is set in the gesture item. The process specified by the process identification information “scroll” is a process of scrolling the screen up / down / left / right. The gesture event specified by the gesture identification information “flick” corresponds to an operation in which the finger is moved at a speed slower than the threshold value in the operation of the swipe gesture event while instructing the touch panel 165, either vertically or horizontally. . Since the operation of moving the finger at a speed slower than the threshold value determines the direction in which the finger is moved, the gesture event specified by the gesture identification information “flick” includes the direction in which the finger is moved as a parameter.

  In the master record in which the process identification information “content selection” is set in the process item, the gesture identification information “double tap” is set in the gesture item. The process specified by the process identification information “content selection” is a process of selecting content.

  Returning to FIG. 5, the display control unit 87 converts the application command input from the browsing unit 61 into an operating command for causing the display unit 161 to display an image, and outputs the operating command to the operating unit 57. Accordingly, the display image generated by the browsing unit 61 is stored in the VRAM of the display unit 161, and at least a part of the display image is displayed on the display unit 161.

  The touch event generation unit 85 generates a touch event in response to position information input from the operating unit 57. Furthermore, the touch event generation unit 85 outputs the generated touch event to the gesture event determination unit 83 every time a touch event is generated. When a plurality of pieces of position information are input from the operating unit 57 at a time, the touch event generation unit 85 generates a plurality of touch events corresponding to the plurality of pieces of position information. When the touch event generation unit 85 generates one or more touch events, the touch event generation unit 85 outputs the generated one or more touch events to the gesture event determination unit 83 and the arbitration unit 53. The position information indicates a position on the display surface of the display unit 161 instructed by the operation user.

  In other words, the operation of the operation user instructing the display surface of the display unit 161 is an instruction start operation in which the user specifies an arbitrary position on the display surface of the display unit 161, in other words, while the user indicates the display surface of the display unit 161 , A moving operation for moving the position to be instructed while touching the display surface of the display unit 161, and an ending operation for the user to end the instruction on the display surface of the display unit 161. When the position information is input from the operating unit 57, the touch event generation unit 85 determines whether the operation by the operating user is an instruction start operation, a movement operation, or an end operation. As described above, the operating unit 57 outputs the position information every time the position information is output from the touch panel 165. Therefore, based on the continuity of the position information that is continuously input as time passes, It is determined whether the operation by the user is an instruction start operation, a move operation, or an end operation.

  The touch event generation unit 85 determines that the operation by the operating user is an instruction start operation with respect to the position information input after a predetermined time has passed without the position information being input from the operating unit 57, and the position information, A touch event including state identification information “Press” for identifying the instruction start operation is generated.

  When one or more pieces of position information are continuously input from the operating unit 57 after detecting the touch event of the state identification information “Press”, the touch event generation unit 85 operates the operation user for each piece of one or more pieces of position information. The operation according to is determined as a moving operation. The touch event generation unit 85 generates, for each of one or more pieces of position information continuously input from the operating unit 57, a touch event including the position information and state identification information “Move” for identifying the moving operation. Generate.

  After the touch event generating unit 85 detects the touch event of the state identification information “Move” and the position information is no longer input from the operating unit 57, the operation by the operating user is completed for the last input position information. The touch event including the position information input last and the state identification information “Release” for identifying the end operation is generated.

  More specifically, the touch event generation unit 85 is based on position information input from the operating unit 57 at the first time and position information input at a second time after a predetermined time from the first time. To determine the state of the touch event. The touch event generation unit 85 has a predetermined distance from the position specified by the position information input at the first time before the first time with respect to the position information input at the first time. When position information indicating a position within the range is not input, the state of the touch event for the position information input at the first time is determined as “Press”. When the position specified by the position information input at the first time does not exist within a predetermined distance from the position specified by the position information input at the second time, the touch event generation unit 85 The state of the touch event for the position information input at the second time is determined as “Press”.

  Further, the touch event generation unit 85 has a position specified by the position information input at the first time within a predetermined distance from the position specified by the position information input at the second time. First, the state of the touch event for the position information input at the second time is determined as “Move”. Further, the touch event generation unit 85 determines that the position information within a predetermined distance from the position specified by the position information input at the first time is not input at the second time. The state of the touch event for the position information input at the first time is determined as “Release”.

  The gesture event determination unit 83 determines a gesture event based on a plurality of touch events that are continuously input from the touch event generation unit 85. When one touch event is continuously input from the touch event generation unit 85 at a time, the gesture event determination unit 83 receives the second input after a predetermined time has elapsed since the first first touch event was input. The touch event is specified, and the gesture event is determined when the distance between the position specified by the first touch event and the position specified by the second touch event is equal to or greater than a predetermined distance. If the distance between the two positions is smaller than the predetermined distance, no gesture event is determined. The predetermined time can be arbitrarily determined. The predetermined time can be, for example, several times the interval at which the touch panel 165 detects the position. The predetermined distance can be arbitrarily determined, but is preferably 40 pixels, for example.

  The gesture event determination unit 83 determines a gesture event based on two or more touch events when two or more touch events are input from the touch event generation unit 85 at a time. Therefore, when the user inputs a multi-touch operation, the gesture event determination unit 83 determines a gesture event. Specifically, when the first touch event and the second touch event are input from the touch event generation unit 85 at the same time, the gesture event determination unit 83 continues to the first touch event. A gesture event is determined based on a set of a plurality of touch events input and a set of a plurality of touch events input continuously with respect to the second touch event. The plurality of touch events included in the set of the plurality of touch events that are continuously input with respect to the first touch event have the same position information or the position information is continuous. Continuous means that two pieces of position information are adjacent or indicate positions within a predetermined distance. The plurality of touch events that are continuously input with respect to the second touch event have the same position information or the position information is continuous. For this reason, the gesture event determination unit 83 receives a plurality of touch events that are input at a time as a set that is continuously input with respect to the first touch event and a input with respect to the second touch event. The position information included in them is determined.

  When one touch event is continuously input from the touch event generation unit 85 at a time, the gesture event determination unit 83 receives the second touch event after the first touch event is input. The state identification information of the first touch event may be “Press” or “Move”. The state information of the second touch event may be “Move” or “Release”. When the second touch event is input, the gesture event determination unit 83 calculates the distance between the two points as the instruction distance LB1 from the position information of the first touch event and the position information of the second touch event. If the designated distance LB1 is greater than or equal to the predetermined threshold value TH, the user's swipe operation is detected, and if the designated distance LB1 is shorter than the predetermined threshold value TH, the user's flick operation is detected.

  When detecting the swipe operation and the flick operation, the gesture event determination unit 83 calculates a direction from the position specified by the position information of the first touch event to the position specified by the position information of the second touch event. The direction closest to the calculated direction is determined from the four directions (up, down, left, and right) with reference to the display surface of the display unit 161. The gesture event determination unit 83 determines the direction and the instruction distance LB1 as parameters when detecting a swipe operation and a flick operation. When detecting a swipe operation, the gesture event determining unit 83 determines a gesture event including gesture identification information “swipe” and a parameter including a direction and an instruction distance LB1, and when detecting a flick operation, A gesture event including the gesture identification information “flick” and a parameter including the direction and the instruction distance LB1 is determined.

  When one touch event is continuously input at a time, a swipe operation by the user is detected by the first plurality of touch events, and a flick operation by the user is detected by the plurality of subsequent touch events. is there. In other words, when the gesture event determination unit 83 determines a plurality of types of gesture events from when the touch event of the state identification information “Press” is input to when the touch event of the state identification information “Release” is input. There is.

  When two touch events are input from the touch event generation unit 85 at a time, the gesture event determination unit 83 receives the second set after the first set of the first touch event and the second touch event are input. The first touch event and the second touch event are input. Here, the first touch event of the first set and the first touch event of the second set become a continuous set, and the second touch event of the first set and the second touch of the second set. A series of events. In this case, the state of each of the first touch event and the second touch event of the first set is “Press” or “Move”. Each state of the second set of the first touch event and the second touch event is “Move” or “Release”.

  In this case, the gesture event determination unit 83 determines the interval L1 between the two points and the straight line connecting the two points based on the position information included in each of the first touch event and the second touch event of the first set. The angle R1 is calculated. The angle of the straight line connecting the two points may be an angle formed by the straight line connecting the two points with the reference line with the horizontal direction of the display unit 161 as the reference line. The angle formed between the straight line connecting the two points and the reference line is an angle formed clockwise from the reference line. In addition, the gesture event determination unit 83 determines the interval L2 between the two points and the angle of the straight line connecting the two points based on the position information included in each of the first touch event and the second touch event of the second set. R2 is calculated.

  The gesture event determination unit 83 detects a pinch-out operation or a pinch-in operation by the user if the angle R1 and the angle R2 are the same and the interval L1 and the interval L2 are different. The angle R1 and the angle R2 being the same include the case where the difference between the angle R1 and the angle R2 is within a predetermined range. In order to detect the movement of the user's finger, an error is included. When the gesture event determination unit 83 detects a pinch-out operation or a pinch-in operation, the gesture event determination unit 83 calculates a value obtained by subtracting the interval L2 from the interval L1 as the instruction distance D. The gesture event determination unit 83 detects a pinch-in operation if the designated distance D is positive, and detects a pinch-out operation if the designated distance D is negative.

  The gesture event determination unit 83 determines the interval L1 and the designated distance D as parameters when detecting a pinch-out operation or a pinch-in operation. When the gesture event determination unit 83 detects a pinch-out operation, the gesture event determination unit 83 determines a gesture event including the gesture identification information “pinch-out” and a parameter including the interval L1 and the instruction distance D, and detects a pinch-in operation. In this case, a gesture event including the gesture identification information “Pinch In” and a parameter including the interval L1 and the instruction distance D is determined.

  The gesture event determination unit 83 detects a rotation operation by the user if the angles R1 and R2 are different and the interval L1 and the interval L2 are the same. The same interval L1 and interval L2 includes the case where the difference between the interval L1 and the interval L2 is within a predetermined range. The difference between the angle R1 and the angle R2 is when the difference between the angle R1 and the angle R2 is outside a predetermined range. In order to detect the movement of the user's finger, an error is included. When detecting a rotation operation, the gesture event determination unit 83 calculates a value obtained by subtracting the angle R2 from the angle R1 as the instruction angle R, and determines the interval L1 and the instruction angle R as parameters. If the command angle R is negative, the rotation direction is clockwise. If the command angle R is positive, the rotation direction is counterclockwise. When detecting a rotation operation, the gesture event determination unit 83 determines a gesture event including gesture identification information “rotation” and a parameter including an interval L1 and an instruction angle R.

  The gesture event determination unit 83 detects a swipe operation or a flick operation by the user if the angle R1 and the angle R2 are the same and the interval L1 and the interval L2 are the same. The angle R1 and the angle R2 being the same include the case where the difference between the angle R1 and the angle R2 is within a predetermined range. That the interval L2 and the interval L1 are the same includes the case where the difference between the interval L2 and the interval L1 is within a predetermined range. In order to detect the movement of the user's finger, an error is included. Gesture event determination unit 83 calculates a distance between two points as instruction distance LB2 based on position information included in each of the first set of first touch events and the second set of first touch events. . The indication distance LB2 may be calculated based on position information included in each of the first set of second touch events and the second set of second touch events.

  The gesture event determination unit 83 detects a swipe operation by the user if the designated distance LB2 is equal to or greater than a predetermined threshold value TH, and detects a flick operation by the user if the designated distance LB2 is shorter than the predetermined threshold value TH. When the gesture event determination unit 83 detects the swipe operation and the flick operation, the gesture event determination unit 83 includes the second set of first touch events from the position specified by the position information included in the first set of first touch events. The direction toward the position specified by the position information to be determined is calculated, and the direction closest to the calculated direction is determined from the four directions of upper, lower, left, and right with reference to the display surface of the display unit 161. When detecting a swipe operation and a flick operation, the gesture event determination unit 83 determines an interval L1, a direction, and an instruction distance LB2 as parameters. When gesture event determination unit 83 detects a swipe operation, gesture event determination unit 83 determines a gesture event including gesture identification information “swipe” and a parameter including interval L1, direction, and instruction distance LB2, and detects a flick operation. Includes determining a gesture event including gesture identification information “flick” and parameters including an interval L1, a direction, and an instruction distance LB2.

  When determining the gesture event, the gesture event determination unit 83 reads the master table 91 corresponding to the application identification information of the current task set by the arbitration unit 53 from the HDD 115. Specifically, an operating command for reading the master table 91 is output to the operating unit 57, and the operating unit 57 controls the HDD 115 to acquire the master table 91 to be read. Hereinafter, a case where the browsing unit 61 is the current task will be described.

  The gesture event determination unit 83 determines whether or not the gesture event is determined, and based on the gesture event, the application set in the current state by the arbitration unit 53 from among a plurality of types of processing determined in the master table 91. Identify the process associated with the program. Here, since the browsing unit 61 is set as the current task by the arbitrating unit 53, the gesture event determining unit 83 identifies the gesture event and the process associated with the master table 91. More specifically, the gesture event determination unit 83 extracts the master record in which the gesture identification information of the determined gesture event is set as the item of gesture from the master table 91, and the process item of the extracted master record Acquires the process identification information set in. When the process identification information is acquired, the gesture event determination unit 83 outputs the determined gesture event to the arbitration unit 53. However, when the process identification information is not acquired, the gesture event determination unit 83 does not output the determined gesture event to the arbitration unit 53. . Further, the gesture event determination unit 83 is a case where one touch event is continuously input from the touch event generation unit 85 at a time, and when the gesture event is not determined, the gesture event determination unit 83 indicates that the gesture event is not determined The signal is output to the arbitration unit 53.

  The gesture event determination unit 83 outputs only the gesture event corresponding to the process executable by the browsing unit 61 set in the current state by the arbitration unit 53 to the arbitration unit 53, and the arbitration unit 53 outputs the gesture event to the current task. Is output to the browsing unit 61. For this reason, it is possible to cause the browsing unit 61 that executes the browsing program to execute a gesture event that is an operation input to the touch panel 165 and is specified by a set of a plurality of touch events. In other words, at the stage of developing a browsing program, the MFP 100 can be developed without being aware of gesture events that can be accepted by the touch panel 165 in the MFP 100. For example, a browsing program that does not accept rotation operations can be developed.

  The arbitration unit 53 receives a gesture event from the gesture event determination unit 83. The arbitration unit 53 outputs the gesture event to the current task among a plurality of tasks for which the application unit 51 executes the application program. Here, since the application unit 51 includes only the browsing unit 61, the arbitrating unit 53 sets the browsing unit 61 in the current state. For this reason, the browsing unit 61 is the current task. The arbitration unit 53 outputs the gesture event to the browsing unit 61 in response to the gesture event input from the gesture event determination unit 83.

  The browsing unit 61 includes a content acquisition unit 71, a display part determination unit 72, a gesture period detection unit 73, a capture part determination unit 75, a capture image generation unit 76, a temporary storage control unit 77, and a display image generation unit. Part 78, correction part 74, and display control part 79.

  The content acquisition unit 71 acquires content from a server connected to the Internet, and outputs the acquired content to the display part determination unit 72, the capture image generation unit 76, and the display image generation unit 78. The content is, for example, a Web page described in a markup language such as HTML (HyperText Markup Language). The content acquisition unit 71 acquires a Web page and does not acquire another content such as another Web page linked to the Web page or image data. Note that the content acquisition unit 71 may acquire another content linked to the Web page. The content acquired by the content acquisition unit 71 may be a Web page identified by a URL (Uniform Resource Locator) specified by an operation user operating the MFP 100, or may be identified by a URL predetermined by the browsing unit 61. Web page may be used.

  The gesture period detection unit 73 detects the gesture period based on the gesture event input from the gesture event determination unit 83. When the gesture period detection unit 73 detects the start of the gesture period, the gesture period detection unit 73 outputs a start signal to the display part determination unit 72 and the capture part determination unit 75, and indicates the gesture period while detecting the gesture period. The period signal is output to the display image generation unit 78. The gesture period is a period in which gesture events are continuously determined by the gesture event determination unit 83 at predetermined time intervals.

  Specifically, the gesture event determination unit 83 determines a gesture event based on one or more touch events. After the gesture event determination unit 83 determines a gesture event, the touch event generation unit 85 continuously A touch event may be generated. For example, when the user inputs a swipe operation, a gesture event is determined by the gesture event determination unit 83 while the user is moving his / her finger. In this case, the touch event generation unit 85 generates a touch event of the state identification information “Move” until the user releases the finger from the touch panel 165. When the user releases the finger from the touch panel 165, the touch event generation unit 85 A touch event of state identification information “Release” is generated. After the gesture event determining unit 83 determines the gesture event of the gesture identification information “Swipe”, until the touch event of the state identification information “Release” is input from the touch event generation unit 85, the gesture event determining unit 83 stores the state identification information “Move”. Each time a touch event is input, a gesture event with gesture identification information “Swipe” is output. The gesture event determination unit 83 determines a gesture event every time a touch event is input from the touch event generation unit 85. Since the touch event generation unit 85 generates a touch event at a predetermined time interval, the gesture event determination unit 83 determines a gesture event at a predetermined time interval.

  The gesture period detection unit 73 detects the gesture period while gesture events are continuously input from the gesture event determination unit 83 at predetermined time intervals. The gesture period detection unit 73 detects the start of the gesture period when a gesture event is input from the gesture event determination unit 83, and detects the end of the gesture period when the gesture event is not input from the gesture event determination unit 83. The gesture period detection unit 73 detects the start of the gesture period when the touch event generation unit 85 generates a touch event of the state identification information “Press”, and generates a touch event of the state identification information “Release”. The end of the gesture period may be detected.

  The display part determination unit 72 determines at least a part of the Web page input from the content acquisition unit 71 as a display part. The display part determination unit 72 interprets the Web page and determines at least a part of an image corresponding to the Web page as a display part. The display part determination unit 72 determines the display part when content is acquired by the content acquisition unit 71 and when a gesture event is input from the gesture event determination unit 83. The display part determination unit 72 determines the part determined by default as the display part when the content is acquired by the content acquisition unit 71, in other words, when the content acquisition unit 71 downloads the Web page. The display portion determined by default may be determined by the Web page, or may be determined from the size of the image corresponding to the Web page and the size of the display unit 161. The display portion is a rectangular area and is specified by two diagonals. Therefore, the display portion includes two diagonal coordinates in the content image.

  The display part determination unit 72 determines the display part when a gesture event is input from the gesture event determination unit 83.

  FIG. 7 is a block diagram illustrating an example of a detailed configuration of the display portion determination unit. Referring to FIG. 7, display portion determination unit 72 includes a conversion unit 11, a scale determination unit 13, a timing determination unit 15, and a capture rate determination unit 17.

  The scale determination unit 13 receives a gesture event from the gesture event determination unit 83 and receives a start signal from the gesture period detection unit 73. The scale determination unit 13 includes a movement amount determination unit 31, an enlargement / reduction ratio determination unit 33, and a rotation amount determination unit 35. The movement amount determination unit 31 sets a gesture event whose gesture identification information is “swipe” or “flick” as a processing target. The enlargement / reduction ratio determination unit 33 sets a gesture event whose gesture identification information is “pinch-in” or “pinch-out” as a processing target. The rotation amount determination unit 35 sets a gesture event whose gesture identification information is “Rotation” as a processing target. The timing determination unit 15 and the capture rate determination unit 17 process gesture events whose gesture identification information is “pinch in”, “pinch out”, or “rotation”.

  The movement amount determination unit 31 determines a scale for determining the movement amount based on the interval L1 included as a parameter in the gesture event input from the gesture event determination unit 83 when the start signal is input. The scale that determines the amount of movement indicates the degree of distance by which the display portion is moved, and is the reference movement distance. The reference movement distance indicates a movement distance with respect to a predetermined unit distance. A movement distance table in which a reference movement distance is determined for each of a plurality of distances is stored in advance, and a reference movement distance determined corresponding to the interval L1 is determined with reference to the movement distance table. The reference movement distance is set to a larger value as the interval L1 is larger. The movement amount determination unit 31 outputs a reference movement distance corresponding to the interval L1 to the conversion unit 11.

  The enlargement / reduction ratio determination unit 33 determines a scale for determining a magnification based on the interval L1 included as a parameter in the gesture event input from the gesture event determination unit 83 at the time when the start signal is input. The scale that determines the magnification indicates the degree of magnification and is a reference magnification. The enlargement / reduction ratio determination unit 33 outputs the reference magnification corresponding to the interval L1 to the conversion unit 11. The reference magnification is a magnification with respect to a predetermined unit distance. A magnification table in which a magnification is determined for each of a plurality of distances is stored in advance, and a reference magnification determined corresponding to the interval L1 is determined with reference to the magnification table.

  The reference magnification is a reduction rate when the gesture event identification information is “pinch-in”, and is an enlargement rate when the gesture event identification information is “pinch-out”. For this reason, the magnification table includes a magnification table corresponding to the gesture identification information “pinch-in” and a magnification table corresponding to the gesture identification information “pinch-out”. In the magnification table corresponding to the gesture identification information “pinch out”, the reference magnification is larger than “1”, and a larger value is set as the interval L1 is larger. In the magnification table corresponding to the gesture identification information “Pinch-in”, the reference magnification is smaller than “1”, and the smaller the interval L1, the smaller the value is set. Since the size of the display unit 161 is constant, when the user changes the interval after instructing the display unit 161 with two fingers, there is a limit to the distance to move the finger. For example, when an instruction is given with a wide finger interval, when the interval is narrowed from that state, the distance that the finger can be moved (residual distance) compared to when an instruction is given with a narrow finger interval. Becomes longer. Also, when an instruction is given with a wide finger interval, when the interval is increased from that state, the distance that the finger can be moved (residual distance) compared to when an instruction is given with a narrow finger interval. Becomes shorter. Therefore, in the magnification table corresponding to the gesture identification information “pinch out”, the reference magnification is set to a larger value as the remaining distance is longer, and in the magnification table corresponding to the gesture identification information “pinch in”, the reference magnification is The longer the remaining distance, the smaller the value is set.

  The rotation amount determination unit 35 determines a scale for determining the rotation amount based on the interval L1 included as a parameter in the gesture event input from the gesture event determination unit 83 when the start signal is input. The scale that determines the amount of rotation indicates the degree of rotation and indicates the reference rotation angle. The reference rotation angle is a rotation angle with respect to a predetermined unit distance. A rotation angle table in which a reference rotation angle is determined for each of a plurality of distances is stored in advance, and a reference rotation angle determined corresponding to the interval L1 is determined with reference to the rotation angle table. The reference rotation angle is set to a larger value as the interval L1 is larger. The rotation amount determination unit 35 outputs a reference rotation angle corresponding to the interval L1 to the conversion unit 11.

  The timing determination unit 39 determines the timing for determining the display portion based on the interval L1 included as a parameter in the gesture event input from the gesture event determination unit 83 when the start signal is input. The timing for determining the display portion is the timing at which a gesture event is input by default. In other words, the gesture event is input at the same time interval as the time interval at which the touch event is generated. The timing determination unit 39 determines the time interval for determining the display portion based on the interval L1 included as a parameter in the gesture event input from the gesture event determination unit 83 at the time when the start signal is input. Specifically, the timing determination unit 39 determines the display portion when the interval L1 included as a parameter in the gesture event input from the gesture event determination unit 83 at the time when the start signal is input is larger than a predetermined value. The time interval is determined to be longer than a predetermined time interval at which a gesture event is input. Since the gesture event is input at the same time interval as the time interval at which the touch event is generated, here, the time interval is indicated by the number of gesture events input. Hereinafter, the number of gesture events that are values indicating time intervals is referred to as an inter-argument. A timing table in which an interval argument is determined for each of a plurality of distances is stored in advance, and the interval argument determined corresponding to the interval L1 is determined with reference to the timing table. In the timing table, the interval argument is set to a larger value as the interval L1 is larger. The timing determination unit 39 outputs an argument corresponding to the interval L1 to the conversion unit 11.

  The capture rate determination unit 37 determines the capture rate based on the interval L1 included as a parameter in the gesture event input from the gesture event determination unit 83 when the start signal is input. The capture rate indicates a magnification with respect to a capture size determined by default. A capture table in which a capture rate is determined for each of a plurality of distances is stored in advance, and the capture rate determined in correspondence with the interval L1 is determined with reference to the capture table. In the capture table, a larger capture rate is set as the interval L1 is larger. The capture rate determination unit 37 outputs a capture rate corresponding to the interval L1 to the capture portion determination unit 75.

  Note that the capture rate determination unit 37 may determine the capture rate only when the gesture event input from the gesture event determination unit 83 is pinch-in when the start signal is input. This is because in the case of other gesture events, the size of the capture portion does not need to be larger than the default size. In this case, the capture rate can be set to a larger value as the interval L1 that is the interval between the fingers when the multi-touch operation is first input is larger, in other words, the interval between the two fingers that the user indicates The larger the size, the larger the capture portion.

  The conversion unit 11 includes a moving unit 21, a scaling unit 23, and a rotating unit 25. The moving unit 21 sets a gesture event whose gesture identification information is “swipe” or “flick” as a processing target. The zoom unit 23 processes a gesture event whose gesture identification information is “pinch-in” or “pinch-out”. The rotation unit 25 processes a gesture event whose gesture identification information is “Rotation”.

  When the gesture event of the gesture identification information “flick” is input, the moving unit 21 sets the display portion determined before that in the direction included in the parameter from the specified distance LB1 included in the parameter. A new display part is determined by moving only. The distance by which the moving unit 21 moves the display part is a value obtained by multiplying the designated distance LB1 included in the parameter by a predetermined default moving distance. The default moving distance defines a moving distance per unit distance.

  When the reference movement distance is input from the movement amount determination unit 31, in other words, in the case of a multi-touch operation, the movement unit 21 sets the instruction distance LB <b> 1 included in the parameter included in the gesture event of the gesture identification information “flick”. The movement distance is calculated by multiplying the reference movement distance, and the display area determined before the gesture event is input is determined by moving the display area in the direction included in the parameter by the calculated movement distance. To do.

  Furthermore, when the reference movement distance is input from the movement amount determination unit 31 and the interval argument is input from the timing determination unit 39, the movement unit 21 receives the gesture event of the gesture identification information “flick” from the gesture event determination unit 83. Each time the gesture event of the gesture identification information “flick” is input without calculating the moving distance until the same number as the interval argument is input, the indication distance LB2 included in the parameter included in the gesture event Calculate the total. Then, when the same number of gesture events having the same number of pieces of gesture identification information “flick” as the interval arguments are input, the moving unit 21 calculates the moving distance by multiplying the total of the designated distances LB2 by the reference moving distance. The moving unit 21 calculates the moving distance by multiplying the total of the designated distance LB2 included in the parameter by the reference moving distance input from the moving amount determining unit 31, and is determined before the gesture event is input. The display part is determined by moving the display part in the direction included in the parameter by the calculated movement distance.

  When the gesture event of the gesture identification information “Swipe” is input, the moving unit 21 pages the page including the display portion determined before that in the direction specified by the parameter, The part corresponding to the display part is determined as a new display part.

  The scaling unit 23 receives the reference magnification from the enlargement / reduction ratio determination unit 33. When the gesture event having the gesture identification information “pinch out” or “pinch in” is input, the scaling unit 23 calculates the magnification by multiplying the reference distance D included in the parameter by the reference magnification, and the gesture event is input. A new display portion is determined by enlarging or reducing the display portion that has been determined before being enlarged by the reciprocal of the calculated magnification. As will be described later, the display image generated by the display image generation unit 78 is generated from the display portion in the capture image or the content image, and the size of the display image is determined by the size of the display unit 161 and is constant. is there. For this reason, when the display image is reduced, the size of the display portion is enlarged, and when the display image is enlarged, the size of the display portion is reduced. Since the magnification calculated by multiplying the instruction distance D by the reference magnification is a value for the display image, when the size of the display portion is enlarged or reduced, it is calculated by the reciprocal of the magnification.

  When the reference magnification is input from the enlargement / reduction ratio determination unit 33 and the interval argument is input from the timing determination unit 39, the scaling unit 23 receives the gesture identification information “pinch out” or “pinch in” from the gesture event determination unit 83. Each time a gesture event with the gesture identification information “Pinch out” or “Pinch in” is input without calculating the magnification until the same number of gesture events are input, the parameters included in that gesture event The total of the designated distances D included in is calculated. Then, when the same number of gesture identification information “pinch out” or “pinch in” gesture events as the number of interval arguments are input, the scaling unit 23 multiplies the sum of the indicated distances D by a reference magnification to obtain a magnification. A new display portion is determined by calculating and enlarging or reducing the display portion determined before the gesture event is input by the reciprocal of the calculated magnification.

  The rotation unit 25 receives the reference rotation angle from the rotation amount determination unit 35. When a gesture event having the gesture identification information “Rotation” is input, the rotation unit 25 calculates the rotation angle by multiplying the reference angle R included in the parameter by the reference rotation angle, and before the gesture event is input. The determined display portion is determined as a new display portion by rotating the calculated display portion by the calculated rotation angle.

  Furthermore, when the reference rotation angle is input from the rotation amount determination unit 35 and the interval argument is input from the timing determination unit 39, the rotation unit 25 receives the gesture event of the gesture identification information “rotation” from the gesture event determination unit 83. Each time a gesture event of the gesture identification information “Rotation” is input without calculating the rotation angle until the same number as the interval argument is input, the indication angle R included in the parameter included in the gesture event Calculate the total. Then, when the same number of gesture events of the gesture identification information “Rotation” as the interval arguments are input, the zoom unit 23 calculates the rotation angle by multiplying the total of the instruction angles R by the reference rotation angle, A new display portion is determined by rotating the display portion determined before the gesture event is input by the calculated rotation angle.

  When the gesture event having the gesture identification information “double tap” is input, the display part determination unit 72 determines the display part determined before that as the display part.

  Returning to FIG. 5, the capture portion determination unit 75 receives the display portion from the display portion determination unit 72 and the start signal from the gesture period detection unit 73. The capture portion determination unit 75 determines a capture portion including a display portion that is determined before the gesture period starts. The display part determined before the start of the gesture period is a display part corresponding to the display image displayed on the display unit 161 when the gesture period starts. When the start signal is input from the gesture period detection unit 73, the capture part determination unit 75 determines that the gesture period has started. The capture portion is at least a part of the content image and includes a display portion. The capture portion has a larger area than the display portion.

  An example of a method for determining the size of the capture portion will be described. The default size of the capture portion is determined based on the size of the display unit 161, the magnification of the content image, and the free space of the RAM 114. Here, the size of the image is indicated by the number of pixels in each of the vertical direction and the horizontal direction. The size of the display portion 161 is indicated by the number of pixels in the vertical direction and the horizontal direction of the display portion 161. The magnification of the content image is determined by the size of the display portion of the content image and the size of the display portion 161, and the horizontal pixel count of the display portion of the content image is divided by the horizontal pixel count of the display portion 161. The value is taken. In the present embodiment, when the content is acquired, the magnification of the content image is “1”, and the magnification of the content image varies when a pinch-out operation or a pinch-in operation is input by the user. Is the value to be

  When the size of the display unit 161 is S (unit: number of pixels), the magnification of the content image is b, and the free capacity of the RAM 114 is m (unit: number of pixels), the capture portion determination unit 75 is m / (b * S) The following maximum integer n is obtained, and a value obtained by multiplying the size S of the display unit 161 by n is determined as the default size of the capture portion. Here, the unit of the free capacity of the RAM 114 is the number of pixels, but this is because the number of bytes per pixel varies depending on the content image. When the storage capacity is indicated in bytes, a value obtained by dividing the storage capacity M (byte) by the number of bytes per pixel may be used.

  When the capture rate is input from the display portion determination unit 72, the capture portion determination unit 75 changes the size of the capture portion to a value obtained by multiplying the capture size by the default size.

  When the capture part determining unit 75 determines the size of the capture part, the capture part determining unit 75 determines an area having the determined size and including the display part as the capture part. When the display part is located at the end of the content image, the capture part is determined so that all of the capture part includes at least part of the content image and includes all of the display part. . The capture part determination unit 75 outputs the determined capture part to the capture image generation unit 76. The capture portion is a rectangular area, and is specified by two diagonals. For this reason, the capture portion includes two diagonal coordinates in the content image.

  The capture image generation unit 76 receives a web page as content from the content acquisition unit 71 and receives a capture portion from the capture portion determination unit 75, and generates a capture image based on the web page. Specifically, the capture image generation unit 76 analyzes the web page and generates an image corresponding to the capture portion of the web page in response to the capture portion input from the capture portion determination unit 75. The captured image generation unit 76 outputs the generated image to the temporary storage control unit 77 as a capture image. Since the capture portion determination unit 75 determines a capture portion at the start of the gesture period, the capture image generation unit 76 generates a capture image at the start of the gesture period.

  When the capture image is input from the capture image generation unit 76, the temporary storage control unit 77 stores the capture image in the RAM 114 and notifies the display image generation unit 78 that the capture image has been stored. For example, the address or file name of the captured image stored in the RAM 114 is notified. Here, a case where the file name of the captured image is notified will be described as an example.

  The display image generation unit 78 receives a web page as a content from the content acquisition unit 71, receives a display portion from the display portion determination unit 72, receives a gesture period signal from the gesture period detection unit 73, and temporarily stores control. The file name of the captured image is input from the unit 77. The display image generation unit 78 is configured to display the content when the content acquisition unit 71 acquires the content, when the gesture event is determined during the gesture period in which the gesture period signal is input, and when the gesture period ends. Generate an image. The display image generation unit 78 outputs the generated display image to the correction unit 74.

  Specifically, the display image generation unit 78 generates a display image corresponding to the display portion based on the captured image stored in the RAM 114 during the gesture period. A capture image having a file name input from the temporary storage control unit 77 is read from the RAM 114, and a region specified by the display portion input from the display portion determination unit 72 during the gesture period is displayed in the captured image. Extract as The display image generation unit 78 generates a display image based on the Web page input from the content acquisition unit 71 when the Web page is acquired by the content acquisition unit 71 or when the gesture period ends. To do. The web page is analyzed to generate an image corresponding to the display portion of the web page as a display image. The display image generation unit 78 determines that the gesture period ends when the gesture period signal is not input after the gesture period signal is input from the gesture period detection unit 73.

  The correction unit 74 analyzes the display image and determines whether or not the display image includes a front image. For example, if a display image is binarized and two or more straight lines are detected, it is determined that a table image is included. When the straight line in the display image does not overlap the boundary of the display image, the correction unit 74 changes the size of the display portion so that the straight line closest to the periphery overlaps the boundary. The display portion is changed by enlarging or reducing the display portion until the display image has a size delimited by a straight line included in the captured image or the content image. The correction unit 74 outputs the changed display portion to the display image generation unit 78.

  When the display portion after the change is input from the correction unit 74, the display image generation unit 78 corresponds to the display portion after the change based on the captured image stored in the RAM 114 during the gesture period. A display image to be generated is generated. Further, the display image generation unit 78 analyzes the Web page input from the content acquisition unit 71 when the Web page is acquired by the content acquisition unit 71 or when the gesture period ends, and displays the Web page. An image corresponding to the portion is generated as a display image. The display image generation unit 78 outputs the generated display image to the display control unit 79.

  The display control unit 79 displays the display image on the display unit 161 in response to the display image input from the display image generation unit 78. Specifically, the display control unit 79 outputs an application command to the arbitration unit 53 in order to display the display image on the display unit 161. Thus, the operating unit 57 stores the display image in a VRAM (Video RAM) included in the display unit 161, and the display image is displayed on the display unit 161.

  Next, the captured image will be described. FIG. 8 is a first diagram illustrating an example of content. Referring to FIG. 8, the content is map image data stored in the server. Map image 401 includes a region 403, a region 405, and a region 407 indicated by dotted lines. A region 403 indicates a display portion, a region 405 indicates a first capture portion corresponding to the region 403 that is a display portion, and a region 407 indicates a second capture portion corresponding to the region 403 that is a display portion. . The region 407 corresponding to the second capture portion is larger in size than the region 405 corresponding to the first capture portion.

  FIG. 9 is a first diagram illustrating an example of a display image. Referring to FIG. 9, display image 413 corresponds to display portion 403 in map image 401 shown in FIG. 8.

  Referring to FIG. 8 again, when the display image 413 is displayed on the display unit 161 and the interval between the two fingers when the user starts the pinch-in multi-touch operation is the first interval. The first capture portion region 405 is determined, and the second capture portion region 407 is determined when the interval between the two fingers is a second interval wider than the first interval.

  The reference magnification is set to a smaller value when the interval between the two fingers at the start of the pinch-in multi-touch operation is the first interval than when the interval is the second interval. For this reason, even if the distance between Japanese fingers is the same, the distance between the two fingers at the time of the first instruction is the first interval than the second interval. The reduction ratio of the image 413 is reduced. For this reason, when the interval between the two fingers at the first instruction time in the pinch-in operation is the first interval, the second capture portion region 407 having a size larger than that of the first capture portion region 405 is determined. Thus, when the display image is reduced, the display portion after the display portion is enlarged can be prevented from exceeding the size of the area 407 of the second capture portion. Further, by determining the first capture portion area 405 having a smaller size than the second capture portion area 407 when the interval between the two fingers at the first instruction time in the pinch-in operation is the first interval, The size of the capture part can be made as small as possible. For this reason, since the amount of data to be processed can be reduced, the load on the CPU 111 can be reduced and the response time can be shortened.

  FIG. 10 is a second diagram illustrating an example of content. The content shown in FIG. 10 includes a table. FIG. 11 is a second diagram illustrating an example of a display image. Referring to FIG. 11, the display image is a part of the table that is the content shown in FIG. Further, the boundary of the display image overlaps with a straight line whose periphery forms a table. For this reason, since the character surrounded by two straight lines is displayed, it is possible to easily recognize the character, and in the pinch-in operation, it is easy to adjust the distance between the fingers at the position where the character is displayed. be able to.

  FIG. 12 is a first flowchart illustrating an example of the flow of the gesture event determination process. FIG. 13 is a second flowchart illustrating an example of the flow of the gesture event determination process. The gesture event determination process is a process executed by CPU 111 when CPU 111 of MFP 100 executes an operation reception program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIGS. 12 and 13, CPU 111 determines whether a touch event has been detected (step S01). When the user indicates touch panel 165, a touch event is detected based on the position detected by touch panel 165. The process waits until a touch event is detected (NO in step S01). If a touch event is detected (YES in step S01), the process proceeds to step S02. In other words, the gesture event determination process is a process executed on condition that a touch event is detected when the user instructs the touch panel 165.

  In step S02, it is determined whether or not the detected touch event is single. If a single touch event is detected, the process proceeds to step S03. If a plurality of touch events are detected, the process proceeds to step S17.

  In step S03, the process branches depending on the state of the touch event. If the state of the touch event is “Press”, the process proceeds to step S04. If the state is “Move”, the process proceeds to step S09. If the state is “Release”, the process proceeds to step S05. In step S04, timer T1 is started and the process returns to step S01. The timer T1 measures the elapsed time since the touch event whose state is “Press” is detected.

  In step S09, it is determined whether the value of timer T1 is equal to or less than threshold value TT1. If the value of timer T1 is equal to or smaller than threshold value TT1, the process proceeds to step S10; otherwise, the process proceeds to step S16. In step S10, the designated distance LB1 is calculated. When the process proceeds to step S10, a touch event whose state information is “Press” or “Move” is detected before the touch event detected in step S01. Here, the touch event detected before the touch event detected in step S01 is referred to as a first touch event, and the touch event detected in step S01 is referred to as a second touch event. In step S10, the distance between two positions specified by the position information included in each of the first touch event and the second touch event is calculated as an instruction distance LB1.

  In step S11, it is determined whether or not the designated distance LB1 is greater than or equal to the threshold value TL. If instructed distance LB1 is equal to or greater than threshold value TL, the process proceeds to step S12; otherwise, the process proceeds to step S16. In step S16, an undetermined signal indicating that the touch event is not determined is output, and the process ends. A touch event indicating a position away from the position indicated by the “Press” touch event by the threshold TL or more is not detected within the threshold time TT1 after the touch event having the state “Press” is detected. In the case, the process ends without determining the gesture event. This is because the gesture event is not determined when the position to be instructed does not move even after the time of the threshold value TT1 has elapsed since the user instructed. A gesture event that indicates the same position for a predetermined time may be determined.

  When the process proceeds to step S12, the state of the touch event is “Move”, and another touch event is input immediately before the touch event. The state of another touch event includes a case of “Press” and a case of “Move”. In step S12, an instruction direction is determined. A direction from the position specified by the position information included in another touch event input immediately before to the position specified by the position information included in the touch event detected in step S01 is determined. The direction determined here is the closest direction among the four directions, top, bottom, left, and right, based on the display surface of the display unit 161.

  In the next step S13, it is determined whether or not the designated distance LB1 is greater than or equal to a threshold value TH. If instructed distance LB1 is equal to or greater than threshold value TH, the process proceeds to step S14; otherwise, the process proceeds to step S15.

  In step S14, the gesture event is determined to be swipe, and the process proceeds to step S33. The gesture event includes gesture identification information “swipe”, the parameter determined in step S12, and the parameter including the instruction distance LB1 calculated in step S10. In step S15, the gesture event is determined to be flicked, and the process proceeds to step S33. The gesture event includes gesture identification information “flick”, a parameter determined in step S12, and a parameter including the instruction distance LB1 calculated in step S10.

  When the process proceeds to step S05, the state of the touch event detected in step S01 is “Release”. When the state of the touch event is “Release”, another touch event is input immediately before the touch event. The state of another touch event includes a case of “Press” and a case of “Move”. In step S05, the process branches depending on the state of another touch event input immediately before. If the state of another touch event input immediately before is “Press”, the process proceeds to step S06; otherwise, the process returns to step S01.

  In step S06, it is determined whether or not the same position information as the position information included in the touch event detected in step S01 is stored in RAM 114. If stored in RAM 114, the process proceeds to step S07; otherwise, the process proceeds to step S08. In addition, the same position information here includes the case where the distance between the two positions respectively specified by the two position information is within a predetermined length. In order to detect an operation in which the user points to the same position, an error is included.

  In step S08, the position information included in the touch event detected in step S01 is temporarily stored in RAM 114, and the process proceeds to step S34. When the process proceeds to step S08, the state of the touch event input immediately before is “Press”, the state of the touch event input next is “Release”, and the same position information is stored in the RAM 114. This is the case when it is not stored. In other words, the user inputs the first tap operation.

  In step S07, the gesture event is determined to be a double tap, and the process proceeds to step S34. When the process proceeds to step S07, the state of the touch event input immediately before is “Press”, the state of the touch event input in step S01 is “Release”, and the same position information is stored in the RAM 114. Is stored. In other words, after the user inputs the first tap operation, the user taps the same position as the first tap operation.

  The process proceeds to step S16 when two touch events are detected. In step S16, it is determined whether or not the state of at least one of the two touch events is “Move”. If the state of at least one of the two touch events is “Move”, the process proceeds to step S17. If not, the process returns to step S01.

  When the process proceeds to step S17, two touch events whose state information is “Press” or “Move” are detected before the two touch events detected in step S01. Here, the two touch events detected before the two touch events detected in step S01 are referred to as a first touch event and a second touch event of the first set, and are detected in step S01. The two touch events are referred to as a second set of first touch events and second touch events. Here, it is assumed that the first touch event of the first set and the first touch event of the second set are consecutive, and the second touch event of the first set and the second touch of the second set. A series of events. In step S18, the interval L1 is calculated. A distance between two positions specified by position information included in each of the first touch event and the second touch event of the first set is calculated as an interval L1.

  In step S19, an instruction angle R is calculated. An angle R1 of a straight line connecting two points is calculated based on position information included in each of the first and second touch events of the first set. The angle of the straight line connecting the two points may be an angle formed by the straight line connecting the two points with the reference line with the horizontal direction of the display unit 161 as the reference line. An angle R2 of a straight line connecting the two points is calculated based on position information included in each of the second set of the first touch event and the second touch event. Then, a difference obtained by subtracting the angle R2 from the angle R1 is calculated as the instruction angle R.

  In the next step S20, it is determined whether or not the absolute value of the indicated angle R calculated in step S19 is larger than a predetermined threshold value TR. This is to determine whether or not the angle R1 has changed. If the absolute value of command angle R is greater than predetermined threshold value TR, the process proceeds to step S21; otherwise, the process proceeds to step S22. In step S21, the gesture event is determined as rotation, and the process proceeds to step S34. The gesture event generated here includes gesture identification information “rotation” and a parameter including the interval L1 calculated in step S18 and the instruction angle R calculated in step S19.

  In step S22, the designated distance D is calculated. Based on the position information included in each of the second set of the first touch event and the second touch event, an interval L2 between the two points is calculated, and the interval L2 is subtracted from the interval L1 calculated in step S18. The value is calculated as the indicated distance D.

  In the next step S23, it is determined whether or not the absolute value of the indicated distance D calculated in step S22 is greater than a predetermined threshold value TD. This is to determine whether or not the interval L1 has changed. If the absolute value of indicated distance D is greater than predetermined threshold value TD, the process proceeds to step S24; otherwise, the process proceeds to step S27. In step S24, it is determined whether or not the indicated distance D is a positive value. If the instruction distance D is a positive value, the process proceeds to step S25; otherwise, the process proceeds to step S26. In step S25, the gesture event is determined to be pinch-in, and the process proceeds to step S34. The gesture event includes gesture identification information “Pinch In” and a parameter including the interval L1 calculated in step S18 and the instruction distance D calculated in step S22. In step S26, the gesture event is determined to be pinch out, and the process proceeds to step S34. The gesture event includes gesture identification information “pinch out” and a parameter including the interval L1 calculated in step S18 and the instruction distance D calculated in step S22.

  When the process proceeds to step S27, two touch events whose state information is “Press” or “Move” are detected before the two touch events detected in step S01. In step S27, the designated distance LB2 is calculated. A distance between two positions specified by position information included in each of the first touch event of the first group and the first touch event of the second group is calculated as an instruction distance LB2. The distance between the two positions specified by the position information included in the second touch event of the first group and the second touch event of the second group is calculated as the instruction distance LB2. Also good.

  In step S28, it is determined whether or not indicated distance LB2 is greater than or equal to threshold value TL. If instructed distance LB2 is greater than or equal to threshold value TL, the process proceeds to step S29; otherwise, the process proceeds to step S33. In step S33, an undetermined signal indicating that the touch event is not determined is output, and the process ends. If two touch events indicating positions separated by the threshold value TL or more are not detected, the process ends without determining a gesture event. This is because the gesture event is not determined when the position designated by the user does not move. A gesture event that indicates the same position for a predetermined time may be determined.

  In step S29, an instruction direction is determined. A direction from the position specified by the position information included in the first touch event of the first set to the position specified by the position information included in the second touch of the first touch event is determined. The direction determined here is the closest direction among the four directions, top, bottom, left, and right, based on the display surface of the display unit 161.

  In the next step S30, it is determined whether or not the designated distance LB2 is greater than or equal to a threshold value TH. If instructed distance LB2 is equal to or greater than threshold value TH, the process proceeds to step S31; otherwise, the process proceeds to step S32.

  In step S31, the gesture event is determined to be swipe, and the process proceeds to step S34. The gesture event includes gesture identification information “swipe” and a parameter including the interval L1 calculated in step S18 and the instruction distance LB2 calculated in step S27. In step S32, the gesture event is determined to be flicked, and the process proceeds to step S34. The gesture event includes gesture identification information “flick” and a parameter including the interval L1 calculated in step S18 and the instruction distance LB2 calculated in step S27.

  In step S34, it is determined whether or not a process corresponding to the determined gesture event is defined. Judgment is made based on whether or not there is a process associated with a gesture event corresponding to the application program set in the current state among a plurality of types of processes defined in the master table 91 stored in the HDD 115. To do. If a process is associated with the determined gesture event in the master table 91, it is determined that a process corresponding to the determined gesture event is defined corresponding to the application program set in the current state. . If a process corresponding to the determined gesture event is defined, the process proceeds to step S35; otherwise, the process proceeds to step S36. In step S35, the determined gesture event is output, and the process ends. In step S36, an undetermined signal is output and the process ends.

  FIG. 14 is a flowchart illustrating an example of the flow of content display processing. The content display process is a process executed by CPU 111 when CPU 111 provided in MFP 100 executes a content display program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIG. 14, CPU 111 acquires content (step S51). Specifically, a Web page is downloaded as content from a server connected to the Internet. The web page is a web page identified by a URL specified by an operating user who operates MFP 100. Further, it may be a web page identified by a predetermined URL. Further, the content is not limited to a Web page, but may be image data.

  In step S52, a display portion defined as a default in the acquired Web page is determined. In step S53, a display image corresponding to the display portion is generated based on the Web page acquired in step S51. Then, the generated display image is displayed on the display unit 161 (step S54), and the process proceeds to step S55. By storing the display image in the VRAM of the display unit 161, the display image is displayed on the display unit 161.

  In step S55, it is determined whether or not a gesture vent has been determined. In the gesture event determination process described above, it is determined that a gesture event has been determined when a gesture event is output. If a gesture event is determined, the process proceeds to step S56; otherwise, the process proceeds to step S68.

  In step S56, it is determined whether or not the gesture event is a multi-touch operation. The gesture event corresponding to the multi-touch operation includes the interval L1 in the parameter, but the gesture event corresponding to the single touch operation does not include the interval L1. If the gesture event corresponds to a multi-touch operation, the process proceeds to step S62; otherwise, the process proceeds to step S57. In step S62, a multi-touch compatible process is executed, and the process proceeds to step S63. Although details of the multi-touch compatible process will be described later, this is a process of generating a display image. In step S63, it is determined whether or not a display image has been generated as a result of the multi-touch handling process being executed. If the display image is generated, the process proceeds to step S64. If not, the process returns to step S55.

  In step S57, the process branches depending on the gesture event. Here, the gesture events corresponding to the single touch operation are a flick operation and a swipe operation. If the gesture event corresponds to a flick operation, the process proceeds to step S58. If the gesture event corresponds to a swipe operation, the process proceeds to step S59.

  In step S58, the movement distance is calculated, and the process proceeds to step S60. The movement distance is calculated by multiplying the instruction distance LB1 included in the parameter of the gesture event by the default movement distance. The default movement distance indicates a movement distance determined as a default, and indicates a movement distance per unit distance. In the next step S60, the display portion is moved by the movement distance in the direction included in the parameter of the gesture event, and the process proceeds to step S61. In step S59, the content is switched to the next Web page, and the process proceeds to step S61.

  In step S61, a display image is generated from the Web page that is the content acquired in step S51, and the process proceeds to step S64. When the process proceeds from step S60, a display image corresponding to the display portion after being moved in step S60 is generated in the Web page. When the process proceeds from step S59, a display image corresponding to the display portion at the same position as the position of the display portion in the page before switching is generated in the Web page after switching.

  In step S64, it is determined whether the display image includes a front image. For example, if a display image is binarized and a straight line in the horizontal direction is detected, it is determined that a table image is included. If the display image includes a front image, the process proceeds to step S65; otherwise, the process proceeds to step S67. In step S65, it is determined whether or not the straight line in the display image is a boundary of the display image. If the straight line in the display image is not the boundary of the display image, the process proceeds to step S66. If the line is a boundary, step S66 is skipped and the process proceeds to step S67. In step S66, the size of the display portion is changed so that the straight line in the display image overlaps the boundary of the display image, and the process proceeds to step S67. The display portion is changed by enlarging or reducing the display portion until the display image has a size delimited by a straight line included in a capture image or content image described later. Note that the boundary of the display image may overlap the straight line in the table by moving the display part without changing the size of the display part.

  In step S67, the display image is displayed on display 161, and the process proceeds to step S68. By storing the display image in the VRAM of the display unit 161, the display image is displayed on the display unit 161. In step S68, it is determined whether an instruction to end the display of content has been received. For example, if a key to which an instruction to end the content display program is assigned among a plurality of keys included in the hard key unit 167, an instruction to end the display of content is accepted. If an instruction to end the display of content is accepted, the process ends. If not, the process returns to step S55.

  FIG. 15 is a flowchart illustrating an example of the flow of multi-touch compatible processing. The multi-touch handling process is a process executed in step S62 of the content display process shown in FIG. Before the step S62 is executed, the display image is displayed, and the gesture event corresponding to the multi-touch operation is determined. Referring to FIG. 15, CPU 111 determines whether or not a gesture period is started (step S71). If the gesture event is not determined at a predetermined time before the gesture event is determined in step S55 of FIG. 14, it is determined that the gesture period starts. If the gesture period is started, the process proceeds to step S72; otherwise, the process proceeds to step S78.

  In step S72, the capture rate is determined. The capture rate is determined based on the interval L1 included in the gesture event parameter. The capture rate indicates a magnification with respect to a capture size determined as a default. A capture table in which a capture rate is determined for each of a plurality of distances is stored in advance, and the capture rate determined in correspondence with the interval L1 is determined with reference to the capture table. In the capture table, a larger capture rate is set as the interval L1 is larger.

  In step S73, the capture portion including the display portion corresponding to the display image, which has a size obtained by multiplying the capture size by the default size, is determined. In the next step S74, a capture image corresponding to the capture portion determined in step S73 is generated. Of the Web page acquired in step S51, an image of a portion corresponding to the capture portion determined in step S73 is generated as a capture image. Then, the generated captured image is temporarily stored in the RAM 114 (step S75), and the process proceeds to step S76.

  In step S76, the interval argument M is determined. With reference to the timing table in which the interval argument is determined for each of the plurality of distances, the interval argument determined corresponding to the interval L1 is determined. In the timing table, the interval argument is set to a larger value as the interval L1 is larger.

  In step S77, an initial value “1” is set in variable C, and the process proceeds to step S78. The variable C is a variable for counting the number of gesture events to be determined.

  In step S78, the process branches depending on the type of gesture event. If the gesture event includes event identification information “rotation”, the process proceeds to step S79. If the gesture event includes event identification information “pinch-in” or “pinch-out”, the process proceeds to step S80. If the information includes “flick”, the process proceeds to step S81. If the gesture event includes event identification information “swipe”, the process proceeds to step S82.

  In step S79, a rotation process is executed, and the process returns to the content display process. In step S80, the scaling process is executed, and the process returns to the content display process. In step S81, the movement process is executed, and the process returns to the content display process. In step S82, the content page is switched, and the process returns to the content display process.

  FIG. 16 is a flowchart illustrating an example of the flow of the rotation process. The rotation process is a process executed in step S79 of FIG. Referring to FIG. 16, the value of the total rotation angle, which is a variable, is set to a value obtained by adding instruction angle R included in the gesture event parameter to the total rotation angle (step S101). The initial value of the total rotation angle is “0”. In the next step S102, it is determined whether or not the variable C is equal to the interval argument M. If variable C is equal to interval argument M, the process proceeds to step S103; otherwise, the process proceeds to step S107. In step S107, 1 is added to variable C, and the process is returned to the multi-touch support.

  In step S103, the reference rotation angle is determined from the interval L1 included in the gesture event parameter. A reference rotation angle determined in correspondence with the interval L1 is determined with reference to a rotation angle table in which a reference rotation angle is determined for each of a plurality of distances. The reference rotation angle is set to a larger value as the interval L1 is larger.

  Then, the rotation angle is calculated by multiplying the total rotation angle by the reference rotation angle (step S104). In the next step S105, the display portion is rotated by the rotation angle calculated in step S104. In the next step S106, an image corresponding to the display portion of the captured image stored in the RAM 114 is set as a display image, and the process is returned to the multi-touch compatible process.

  FIG. 17 is a flowchart illustrating an example of the flow of the scaling process. The scaling process is a process executed in step S80 of FIG. Referring to FIG. 17, the value of the distance total that is a variable is set to a value obtained by adding the instruction distance D included in the parameter of the gesture event to the distance total (step S111). The initial value of the total distance is “0”. In the next step S112, it is determined whether or not the variable C is equal to the interval argument M. If variable C is equal to interval argument M, the process proceeds to step S113; otherwise, the process proceeds to step S117. In step S117, 1 is added to variable C, and the process returns to the multi-touch support.

  In step S113, the reference magnification is determined from the interval L1 included in the gesture event parameter. A reference magnification determined corresponding to the interval L1 is determined with reference to a magnification table that defines a reference magnification for each of a plurality of distances. In the magnification table corresponding to the gesture identification information “Pinch-in”, the reference magnification is smaller than “1”, and the smaller the interval L1, the smaller the value is set. A larger value is set as the interval L1 is larger. In the magnification table corresponding to the gesture identification information “pinch out”, the reference magnification is larger than “1”, and a larger value is set as the interval L1 is larger.

  Then, the magnification is calculated by multiplying the total distance by the reference magnification (step S114). In the next step S115, the display portion is enlarged or reduced at the magnification calculated in step S114. In the next step S116, an image corresponding to the display portion of the captured image stored in the RAM 114 is set as a display image, and the process is returned to the multi-touch compatible process.

  FIG. 18 is a flowchart illustrating an example of the flow of movement processing. The movement process is a process executed in step S81 in FIG. Referring to FIG. 18, the value of the moving total that is a variable is set to a value obtained by adding the designated distance LB2 included in the parameter of the gesture event to the moving total (step S121). The initial value of the moving total is “0”. In the next step S122, it is determined whether or not the variable C is equal to the interval argument M. If variable C is equal to interval argument M, the process proceeds to step S123; otherwise, the process proceeds to step S127. In step S127, 1 is added to the variable C, and the process is returned to the multi-touch correspondence.

  In step S123, the reference movement distance is determined from the interval L1 included in the parameters of the gesture event. With reference to a movement distance table in which a reference movement distance is determined for each of a plurality of distances, a reference movement distance determined corresponding to the interval L1 is determined. The reference movement distance is set to a larger value as the interval L1 is larger.

  Then, the movement distance is calculated by multiplying the movement total by the reference movement distance (step S124). In the next step S125, the display portion is moved in the direction included in the parameter of the gesture event by the movement distance calculated in step S124. In the next step S126, an image corresponding to the display portion of the content image acquired in step S51 is set as a display image, and the process returns to the multi-touch compatible process.

  In the embodiment described above, in the rotation operation, the angle at which the display portion is rotated with respect to the instruction angle R is increased as the distance L1 between the two fingers to be instructed first is larger. You may make it make the angle which rotates a display part with respect to instruction | indication angle R small, so that the space | interval L1 of two fingers is wide. In this case, the display portion rotates only slightly even when the distance of finger movement is long. For this reason, it is effective when finely adjusting the rotation angle. It may be set in advance whether the angle at which the display portion is rotated with respect to the instruction angle R is increased or decreased as the interval L1 is wider. Further, by comparing the interval L1 with a threshold value, it may be determined whether the angle at which the display portion is rotated with respect to the instruction angle R is increased or decreased as the interval L1 is wider. For example, when the interval L1 is equal to or larger than the threshold value, the angle at which the display portion is rotated with respect to the instruction angle R is decreased as the interval L1 is larger. You may make it enlarge the angle which rotates a display part with respect to the instruction | indication angle R. FIG.

  Further, in the pinch-in operation, the enlargement ratio for enlarging the display portion with respect to the designated distance D is increased as the distance L1 between the two fingers designated first is larger, but the distance L1 between the two fingers designated first is increased. The larger the size, the smaller the enlargement ratio for enlarging the display portion with respect to the instruction distance D. In this case, even if the finger movement distance is long, the display portion is enlarged only a little. Therefore, it is effective when finely adjusting the size of the display portion. Similarly, in the pinch-out operation, the reduction ratio for reducing the display portion with respect to the designated distance D is reduced as the distance L1 between the two designated fingers first becomes smaller. As the interval L1 is narrower, the reduction ratio for reducing the display portion with respect to the instruction distance D may be increased. In this case, the display part is reduced only slightly even when the distance of finger movement is long. Therefore, it is effective when finely adjusting the size of the display portion. It may be set in advance whether the enlargement ratio or the reduction ratio is increased or decreased with respect to the instruction distance D as the interval L1 is wider. Further, by comparing the interval L1 with a threshold value, the larger the interval L1, the larger or smaller the enlargement rate or reduction rate of the display portion with respect to the indicated distance D may be determined. Good. For example, in the case of a pinch-out operation, when the interval L1 is greater than or equal to the threshold value, the larger the interval L1 is, the smaller the enlargement ratio with respect to the indicated distance D is. You may make it enlarge the enlargement rate with respect to the instruction | indication distance D so that. By comparing the interval L1 with a threshold value, in the case of a pinch-in operation, when the interval L1 is equal to or greater than the threshold value, the reduction rate with respect to the indicated distance D increases as the interval L1 increases, When the distance is smaller, the reduction ratio with respect to the instruction distance D may be decreased as the interval L1 is increased.

  Further, in the flick operation, the distance to move the display portion with respect to the instruction distance LB2 is increased as the distance L1 between the two fingers specified first is larger. However, the distance L1 between the two fingers specified first is larger. The larger the distance, the smaller the distance to move the display portion relative to the instruction distance LB2. In this case, the display portion moves only slightly even when the distance of finger movement is long. Therefore, it is effective when finely adjusting the position of the display portion. It may be set in advance whether the movement amount is increased or decreased with respect to the instruction distance LB2 as the interval L1 is wider. Further, by comparing the interval L1 with a threshold value, it may be determined whether the amount of movement of the display portion with respect to the instruction distance LB2 is increased or decreased as the interval L1 is wider. For example, when the distance L1 is equal to or greater than the threshold value, the amount of movement of the display portion with respect to the instruction distance LB2 is reduced as the distance L1 is larger. The amount by which the display portion is moved with respect to the instruction distance LB2 may be increased.

  As described above, MFP 100 according to the present embodiment functions as a data processing device, displays a display image corresponding to a display portion of a Web page that is content, and detects one or more positions designated by the user. When determining a gesture event and determining a predetermined gesture event determined based on two positions, a scale indicating a degree of changing a display portion based on an interval L1 between the two positions The display portion corresponding to the display image is converted with the determined kale. For this reason, the part displayed in content can be changed by changing the space | interval L1 of two positions which a user instruct | indicates. As a result, multi-touch operations are diversified.

  In addition, since the capture rate for determining the size of the captured image is determined based on the interval L1 between the two positions designated by the user, the size of the captured image is determined when the enlargement rate of the display portion is determined based on the interval L1. , Can be changed according to magnification. In other words, the size of the captured image can be prevented from becoming smaller than the size of the display portion, and the size of the captured image can be prevented from becoming too large with respect to the size of the display portion. As a result, the size of the captured image can be set to an appropriate size.

  In the above-described embodiment, the MFP 100 has been described as an example of the data processing apparatus. However, the operation reception method that causes the MFP 100 to execute the processes illustrated in FIGS. It goes without saying that the invention can be understood as an operation reception program for execution.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

<Appendix>
(1) When the predetermined gesture event is a gesture event that causes the conversion unit to enlarge the size of the display portion, the scale determination unit has a large enlargement ratio with a large residual distance that can change the interval between the two positions. The data processing device according to claim 5, wherein the data processing device is determined as a scale.
(2) In the case where the predetermined gesture event is a gesture event that causes the conversion unit to reduce the size of the display portion, the scale determination unit reduces the smaller the larger the remaining distance in which the interval between the two positions can be changed. The data processing apparatus according to claim 5, wherein the rate is determined as a scale.
(3) (Scroll) In the case where the gesture determining event is determined by the gesture event determining unit, the gesture event whose position changes with time while maintaining the relative direction of the two positions, The data processing apparatus according to claim 4, wherein a larger scale is determined as the interval between the two positions is larger.
(4) The display image generating means is configured to display the display image based on the content stored outside before the gesture event is determined by the gesture event determining means or when the gesture period ends. The data processing apparatus according to claim 10, which generates an image.

100 MFP, 110 Main circuit, 111 CPU, 112 Communication I / F unit, 113 ROM, 114 RAM, 115 HDD, 116 Facsimile unit, 117 External storage device, 120 Automatic document feeder, 130 Document reading unit, 140 Image forming unit , 150 paper feed unit, 160 operation panel, 161 display unit, 163 operation unit, 165 touch panel, 167 hard key unit, 21 conversion unit, 23 scale determination unit, 25 timing determination unit, 25 movement unit, 27 capture rate determination unit, 27 Scaling unit, 29 Rotating unit, 31 Movement amount determining unit, 33 Enlarging / contracting rate determining unit, 35 Rotating amount determining unit, 37 Capture rate determining unit, 39 Timing determining unit, 51 Application unit, 53 Arbitration unit, 55 Operation converting unit , 57 Operating part, 61 Browsing part, 71 Content acquisition 72 Display portion determination unit, 73 Gesture period detection unit, 74 Correction unit, 75 Capture portion determination unit, 76 Capture image generation unit, 77 Temporary storage control unit, 78 Display image generation unit, 79 Display control unit, 81 Part, 83 gesture event determination part, 85 touch event generation part, 87 display part control part, 91 master table.

Claims (12)

Display means capable of displaying images;
Display control means for displaying on the display means a display image corresponding to a display portion which is at least a part of the content;
Position detection means capable of detecting one or more positions designated by a user on the display surface of the display means;
Gesture event determining means for determining any one of a plurality of gesture events based on the one or more positions in response to detection of one or more positions by the position detecting means;
Display part determining means for changing the display part based on a gesture event determined during a gesture period in which gesture events are continuously determined by the gesture event determining means,
When the gesture event determined by the gesture event determination unit is a predetermined gesture event determined based on the two positions detected by the position detection unit, the display portion determination unit determines the interval between the two positions. Scale determining means for determining a scale indicating the degree of change of the display portion based on:
A data processing apparatus comprising: conversion means for converting a display portion corresponding to the display image displayed on the display means at the time when the gesture period starts, with the determined scale.   The data processing apparatus according to claim 1, wherein the scale determining unit determines a larger scale as the interval between the two positions is larger.   The data processing apparatus according to claim 1, wherein the scale determining unit determines a smaller scale as the interval between the two positions is larger.   The data processing apparatus according to claim 1, wherein the scale determining unit determines a different scale for each type of gesture event determined by the gesture event determining unit.   In the case where a gesture event is determined based on a temporal change in the interval between the two positions by the gesture event determining unit, the scale determining unit is larger as a residual distance that can change the interval between the two positions is larger. The data processing apparatus according to claim 4, wherein a scale is determined.   The scale determining unit determines a larger scale as the interval between the two positions is larger when the gesture event is determined by the gesture event determining unit based on a temporal change in the relative direction of the two positions. The data processing device according to claim 4 or 5.   When the gesture event is determined based on a temporal change in the relative direction of the two positions by the gesture event determining unit, the scale determining unit determines a smaller scale as the interval between the two positions is larger. The data processing device according to claim 4 or 5.   If the display image includes a front image at the end of the gesture period, the display part determination unit determines the display part as the display part of the area surrounded by a line in the content image. The data processing apparatus according to any one of claims 1 to 7, further comprising correction means for correcting to an area having the closest size. The gesture event determining means determines a gesture event at a predetermined time interval during the gesture period,
When the predetermined gesture event is determined by the gesture event determining unit, the display part determining unit determines a timing for determining the display part based on an interval between two positions detected by the position detecting unit. The data processing device according to claim 1, further comprising timing determination means for determining a time interval longer than the predetermined time interval. Capture portion determining means for determining a capture portion including a display portion corresponding to the display image displayed on the display means at the time when the gesture period starts;
Capture image generating means for generating a capture image corresponding to the determined capture portion based on the content in response to the determination of the capture portion;
Temporary storage control means for temporarily storing the generated captured image, and
The display portion determining means includes a capture rate determining means for determining a capture rate indicating a degree of the size of the capture portion as the scale,
The capture portion determining means determines a capture portion having a size determined by the determined capture rate,
The display image generation means generates a display image corresponding to a new display portion determined by the display portion determination means based on the temporarily stored captured image. The data processing apparatus described. An operation receiving method executed by a data processing apparatus having a display means capable of displaying an image,
A display control step of displaying on the display means a display image corresponding to a display portion that is at least a part of the content;
A position detecting step capable of detecting one or more positions designated by a user on the display surface of the display means;
A gesture event determining step of determining any one of a plurality of gesture events based on the one or more positions in response to detection of one or more positions in the position detecting step;
A display part determining step of changing the display part based on a gesture event determined during a gesture period in which gesture events are continuously determined in the gesture event determining step,
In the display portion determination step, when the gesture event determined in the gesture event determination step is a predetermined gesture event determined based on the two positions detected in the position detection step, an interval between the two positions A scale determining step for determining a scale indicating a degree of changing the display portion based on
A conversion step of converting a display portion corresponding to the display image displayed on the display means at the time when the gesture period starts, with the determined scale. An operation reception program that is executed by a computer that controls a data processing apparatus including display means capable of displaying an image,
A display control step of displaying on the display means a display image corresponding to a display portion that is at least a part of the content;
A position detecting step capable of detecting one or more positions designated by a user on the display surface of the display means;
A gesture event determining step of determining any one of a plurality of gesture events based on the one or more positions in response to detection of one or more positions in the position detecting step;
Causing the computer to execute a display part determining step of changing the display part based on a gesture event determined during a gesture period in which gesture events are continuously determined in the gesture event determining step,
In the display portion determination step, when the gesture event determined in the gesture event determination step is a predetermined gesture event determined based on the two positions detected in the position detection step, an interval between the two positions A scale determining step for determining a scale indicating a degree of changing the display portion based on
An operation reception program, comprising: a conversion step of converting a display portion corresponding to a display image displayed on the display means at the time when the gesture period starts with the determined scale.

Images