JP2014160426A - Data processing apparatus, content display method, and content display program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To reduce incorrect operation by a user.SOLUTION: An MFP comprises: a content display unit 51 that displays a display part of contents; and a gesture event determination unit 93 that determines at least one of a plurality of gesture events on the basis of one or more positions detected by a touch panel. The content display unit 51 includes a display image creating unit 71 that creates a display image for displaying the display part on the basis of the gesture event. The display image creating unit 71 comprises: a page switching unit 83 that changes a page to be displayed when a first gesture event is determined; a scroll unit 85 that changes a display part within a page when a second gesture event is determined; and a specific state page switching unit 86 that changes a page to be displayed when the display part cannot be changed when the second gesture event is determined.

Description

  The present invention relates to a data processing device, a content display method, and a content display program, and more particularly to a data processing device provided with a touch panel, a content display method and a content display program executed by the data processing device.

  In recent years, data processing apparatuses represented by MFPs (MFPs) include apparatuses having a touch panel as a user interface. By the touch panel, various operations can be detected based on the difference in the operation that the user designates with a finger.

  For example, Japanese Patent Laid-Open No. 2011-170603 discloses at least one of a display unit that displays data information, a contact input unit that performs input by touching the display unit, and an input from the contact input unit. The contact detection unit that detects the contact point of the simultaneous input described above, the acquisition unit that acquires the number of contact points of the simultaneous input detected by the contact detection unit, and a processing unit that performs different processing at least according to the number of contact points And when the contact detection unit detects that the at least one or more simultaneous input contact points are moved in one direction, the data information displayed on the display unit A portable terminal is described that is configured to display and move with a movement amount corresponding to the number of contact points of the simultaneous input acquired by the acquisition unit.

  Also, different operations can be detected based on the difference in the speed at which the user moves the position indicated by the finger using the touch panel. For example, a flick operation is detected when the speed at which the user moves the position indicated by the finger is lower than a predetermined speed, and a swipe operation is detected when the speed is higher than a predetermined speed.

On the other hand, when the user operates the MFP, the posture is upright. Therefore, when the touch panel is mounted on the MFP, the operation surface of the touch panel is often directed upward. For this reason, the direction in which the user touches the touch panel with the finger is above the touch panel. In the operation of touching and moving the touch panel with a finger from above, it may be difficult to adjust the speed of moving the finger. For example, the movement of the finger on the touch panel may be performed at different speeds when using only the wrist and when using an arm, waist, or the like in addition to the wrist. For this reason, when the user tries to input the swipe operation, there is a problem that the MFP may detect the flick operation.
JP 2011-170603 A

  The present invention has been made to solve the above-described problems, and one of the objects of the present invention is to provide a data processing apparatus capable of reducing erroneous operations by a user and improving operability. It is.

  Another object of the present invention is to provide a content display method capable of reducing erroneous operations by a user and improving operability.

  Still another object of the present invention is to provide a content display program capable of reducing erroneous operations by a user and improving operability.

  In order to achieve the above-described object, according to one aspect of the present invention, a data processing device detects display means capable of displaying an image, and one or more positions indicated by a user on the display surface of the display means. Content display for displaying at least a part of a display part image of one page to be displayed among a plurality of pages included in the content. And a gesture event determining means for determining at least one of a plurality of gesture events based on the one or more positions in response to detection of one or more positions by the position detecting means. The means includes a table for displaying the display portion based on at least one gesture event determined by the gesture event determination means. Display image generating means for generating a display image, and display control means for displaying the generated display image on the display means. The display image generating means determines the first gesture event by the gesture event determining means. When the second gesture event is determined by the page switching means for changing the page to be displayed to another page and the gesture event determining means, the scroll means for changing the display portion in the page to be displayed. And a specific state page switching means for changing the display target page to another page when the display part cannot be changed by the scroll means when the second gesture event is determined by the gesture event determining means. .

  According to this aspect, when at least a part of the display portion of the content is displayed and the first gesture event is determined, the page to be displayed is changed to another page, and the second gesture event is determined When the display part is changed in the display target page and the second gesture event is determined, if the display part cannot be changed in the display target page, the display target page is changed to another page. Is done. For this reason, when the display image cannot be changed when the second gesture event is determined, the same display image as that when the first gesture event is determined can be displayed. For this reason, the user does not need to input again the operation corresponding to the first gesture event. As a result, it is possible to provide a data processing apparatus capable of reducing erroneous operations by the user and improving operability.

  Preferably, the specific state page switching unit is configured to display the gesture period even when the display unit can be changed by the scroll unit at the start of the gesture when the gesture period in which one or more positions are continuously detected by the position detection unit starts. If there is a period during which the display part cannot be changed by the scroll means by the time when the gesture is ended, the page to be displayed is changed to another page when the gesture ends.

  According to this aspect, even if the display part can be changed in the display target page at the start of the gesture, there is a period in which the display part cannot be changed in the display target page by the end of the gesture. In such a case, the page to be displayed is changed to another page when the gesture ends. For this reason, even if the display part can be changed in the page to be displayed at the start of the gesture, the display image of the display part of another page can be displayed at the end of the gesture period. As a result, when the display portion cannot be changed, even when the user inputs an operation for changing the operation display portion, the display image with the page switched can be displayed.

  Preferably, in the gesture period, the scroll unit changes the display part until the display part cannot be changed in the display target page every time the second gesture event is determined by the gesture event determination unit.

  According to this aspect, in the gesture period, each time the second gesture event is determined, the display portion is changed until the display portion cannot be changed in the display target page. For this reason, in the gesture period, the display portion can be changed in the display target page and the display image can be displayed, and the display portion can be displayed in the display target page by one operation input by the user. And the display image of the display portion after the page is switched can be displayed.

  Preferably, the gesture event determining means is detected by the flick determining means for determining the first gesture event when the moving speed of one position detected by the position detecting means is smaller than the threshold, and the position detecting means. Swipe determining means for determining a second gesture event when the speed at which one position moves is equal to or greater than a threshold value.

  According to this aspect, the first gesture event is determined when the moving speed of one position detected by the position detecting means is smaller than the threshold value, and the second gesture event is determined when the moving speed is equal to or higher than the threshold value. The Therefore, the user can input different operations only by changing the moving speed of the designated position.

  According to another aspect of the present invention, the content display method is a content display method executed by a data processing device, the data processing device including a display unit capable of displaying an image, and a display surface of the display unit. A position detecting unit capable of detecting one or more positions designated by the user, and displaying at least a part of an image of a display portion of one page to be displayed among a plurality of pages included in the content as the display unit A content display step for displaying, and a gesture event determining step for determining at least one of a plurality of gesture events based on the one or more positions in response to detection of the one or more positions by the position detecting means. The content processing step is executed by the data processing device, and the content display step is at least determined in the gesture event determination step. Display image generation step including a display image generation step for generating a display image for displaying a display portion based on one gesture event, and a display control step for displaying the generated display image on a display means. When the first gesture event is determined in the gesture event determining step, the page switching step for changing the display target page to another page, and the second gesture event is determined in the gesture event determining step, When the second gesture event is determined in the scroll step for changing the display portion in the page to be displayed and the gesture event determining step, if the display portion cannot be changed in the scroll step, the display target page is separated. No ba Including a specific state page switching step of changing the di- and.

  According to this aspect, it is possible to provide a content display method capable of reducing erroneous operations by the user and improving operability.

  Preferably, the specific state page switching step includes the gesture period even when the display portion can be changed in the scroll step at the gesture start time when the gesture period in which one or more positions are continuously detected by the position detection unit starts. If there is a period during which the display portion cannot be changed in the scroll step by the time when the gesture is ended, the step of changing the page to be displayed to another page at the time of the end of the gesture is included.

  Preferably, the scroll step includes a step of changing the display portion in the gesture period until the display portion cannot be changed in the display target page every time the second gesture event is determined in the gesture event determination step. .

  Preferably, the gesture event determining step is detected by the flick determining step of determining the first gesture event when the moving speed of one position detected by the position detecting unit is smaller than the threshold, and the position detecting unit. And a swipe determination step of determining a second gesture event when the speed at which one position moves is equal to or greater than a threshold value.

  According to another aspect of the present invention, the content display program is a content display program that is executed by a computer that controls the data processing device. The data processing device includes: a display unit that can display an image; An image of at least a part of a display portion of one page to be displayed among a plurality of pages included in the content. A content display step for displaying the image on the display means, and a gesture event determination for determining at least one of the plurality of gesture events based on the one or more positions in response to the detection of the one or more positions by the position detection means And the step of causing the computer to execute the content display step determines the gesture event. A display image generating step for generating a display image for displaying a display portion based on at least one gesture event determined in step, a display control step for displaying the generated display image on a display means, And when the first gesture event is determined in the gesture event determination step, the display image generation step includes a page switching step for changing the display target page to another page, and a second gesture in the gesture event determination step. When the event is determined, the display portion cannot be changed in the scroll step when the second gesture event is determined in the scroll step for changing the display portion in the page to be displayed and the gesture event determination step. Optionally comprises a specific state page switching step of changing the page to be displayed on a separate page.

  According to this aspect, it is possible to provide a content display program capable of reducing erroneous operations by the user and improving operability.

1 is a perspective view showing an appearance of an MFP in one embodiment of the present invention. It is a top view which shows an example of an operation panel. 2 is a block diagram illustrating an example of an outline of a hardware configuration of an MFP. FIG. 2 is a diagram illustrating an example of a software architecture of a CPU provided in an MFP. FIG. 3 is a block diagram illustrating an example of functions of a CPU included in an MFP, together with information stored in an HDD. FIG. It is a figure which shows an example of a master table. It is a flowchart which shows an example of the flow of a gesture event determination process. It is a flowchart which shows an example of the flow of a multiple position detection process. It is a flowchart which shows an example of the flow of a content display process. It is a flowchart which shows an example of the flow of a display image update process. It is a flowchart which shows an example of the flow of a priority display process. It is a flowchart which shows an example of the flow of a specific state page switching process. It is a 1st figure which shows an example of a screen. It is a 2nd figure which shows an example of a screen. It is a 3rd figure which shows an example of a screen. It is a 4th figure which shows an example of a screen. It is a 5th figure which shows an example of a screen.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  FIG. 1 is a perspective view showing an external appearance of an MFP according to one embodiment. FIG. 2 is a plan view showing an example of the operation panel. Referring to FIGS. 1 and 2, MFP 100 is an example of a data processing device, and includes a document reading unit 130 for reading a document, an automatic document transport device 120 for transporting a document to document reading unit 130, and An image forming unit 140 for forming an image on a sheet or the like based on image data output by the document reading unit 130 reading the document, a sheet feeding unit 150 for supplying the image forming unit 140 with a sheet, and a user interface As an operation panel 160.

  Operation panel 160 is provided on the upper surface of MFP 100. Operation panel 160 includes a display unit 161, a touch panel 165, and a hard key unit 167. The display unit 161 is, for example, a liquid crystal display device (LCD), and displays an instruction menu for a user, information about acquired image data, and the like. When the user operates the MFP 100, the user is in an upright posture, so that the display surface of the display unit 161 and the operation surface of the touch panel 165 are arranged facing upward. This is because the user can easily visually recognize the display surface of the display unit 161 and the user can easily perform an operation of pointing the touch panel with a finger.

  The hard key portion 167 includes four hard keys 167A to 167D each representing characters of “BOX”, “FAX”, “COPY”, and “ECO”. The touch panel 165 is a multi-touch compatible touch panel provided on the upper surface or the lower surface of the display unit 161 so as to overlap the display unit 161, and detects a position designated by the user on the display surface of the display unit 161. The operation in which the user instructs the touch panel 165 includes a multi-touch operation for instructing the touch panel 165 with a finger at the same time, and a single touch operation for instructing one place with a finger at a time.

  The automatic document feeder 120 automatically conveys a plurality of documents set on the document feeding tray one by one to a predetermined document reading position set on the platen glass of the document reading unit 130, and reads the document. The document whose document image is read by the unit 130 is discharged onto a document discharge tray. The document reading unit 130 includes a light source that irradiates light to the document conveyed to the document reading position and a photoelectric conversion element that receives light reflected from the document, and scans a document image corresponding to the size of the document. The photoelectric conversion element converts the received light into image data that is an electrical signal and outputs the image data to the image forming unit 140. The paper feed unit 150 conveys the paper stored in the paper feed tray to the image forming unit 140.

  The image forming unit 140 forms an image by a well-known electrophotographic method, and the image data after data processing is obtained by performing various data processing such as shading correction on the image data input from the document reading unit 130. Alternatively, an image is formed on a sheet conveyed by the sheet feeding unit 150 based on image data received from the outside.

  FIG. 3 is a block diagram illustrating an example of an outline of the hardware configuration of the MFP. Referring to FIG. 3, MFP 100 includes a main circuit 110. The main circuit 110 includes a CPU 111, a communication interface (I / F) unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory) 114, and a hard disk drive (HDD) 115 as a mass storage device. A facsimile unit 116 and an external storage device 117. CPU 111 is connected to automatic document feeder 120, document reading unit 130, image forming unit 140, sheet feeding unit 150, and operation panel 160, and controls the entire MFP 100.

  The ROM 113 stores a program executed by the CPU 111 or data necessary for executing the program. The RAM 114 is used as a work area when the CPU 111 executes a program. Further, the RAM 114 temporarily stores the read images continuously sent from the document reading unit 130.

  The facsimile unit 116 is connected to the public switched telephone network (PSTN) and transmits facsimile data to the PSTN or receives facsimile data from the PSTN. The facsimile unit 116 stores the received facsimile data in the HDD 115 or converts the received facsimile data into print data that can be printed by the image forming unit 140 and outputs the print data to the image forming unit 140. As a result, the image forming unit 140 forms an image on a sheet of facsimile data received by the facsimile unit 116. Further, the facsimile unit 116 converts the data read by the document reading unit 130 or the data stored in the HDD 115 into facsimile data, and transmits the facsimile data to a facsimile machine connected to the PSTN.

  Communication I / F unit 112 is an interface for connecting MFP 100 to a network. The communication I / F unit 112 communicates with other computers connected to the network using a communication protocol such as TCP (Transmission Control Protocol) or FTP (File Transfer Protocol). Note that the protocol for communication is not particularly limited, and an arbitrary protocol can be used. The network to which the communication I / F unit 112 is connected is, for example, a local area network (LAN), and the connection form may be wired or wireless. The network is not limited to the LAN, and may be a network using a wide area network (WAN), a public switched telephone network, or the like. Furthermore, the network is connected to the Internet. Therefore, MFP 100 can communicate with a computer such as a server connected to the Internet.

  The external storage device 117 is controlled by the CPU 111, and a CD-ROM (Compact Disk Read Only Memory) 118 or a semiconductor memory is mounted. The CPU 111 can access the CD-ROM 118 or the semiconductor memory via the external storage device 117. The CPU 111 loads the program recorded in the CD-ROM 118 or semiconductor memory mounted on the external storage device 117 into the RAM 114 and executes it. Note that the program executed by the CPU 111 is not limited to the program recorded on the CD-ROM 118, and the program stored in the HDD 115 may be loaded into the RAM 114 and executed. In this case, another computer connected to the network rewrites the program stored in HDD 115 of MFP 100 or adds a new program and writes it via the network connected to communication I / F unit 112. You may do it. Further, MFP 100 may download a program from another computer connected to the network and store the program in HDD 115. The program here includes not only a program directly executable by the CPU 111 but also a source program, a compressed program, an encrypted program, and the like.

  The medium for storing the program executed by the CPU 111 is not limited to the CD-ROM 118, but an optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC card, optical card, It may be a semiconductor memory such as a mask ROM, EPROM (Erasable Programmable ROM), or EEPROM (Electrically EPROM).

  Operation panel 160 includes a display unit 161 and an operation unit 163. The display unit 161 is a display such as a liquid crystal display (LCD) or an organic ELD (Electro-Luminescence Display), and displays an instruction menu for the user, information about acquired image data, and the like. The operation unit 163 includes a touch panel 165 and a hard key unit 167 including a plurality of keys. Each of the plurality of keys included in the hard key unit 167 includes a contact switch and is connected to the CPU 111. When the hard key is pressed by the operation user, the contact is closed and the circuit connected to the CPU 111 is closed. The hard key closes the circuit while being pressed by the operating user who operates the MFP 100, and opens the circuit when not pressed by the operating user.

  When a plurality of keys included in the hard key unit 167 are pressed, the operation unit 163 receives input of data such as instructions, characters, and numbers corresponding to the pressed keys. The touch panel 165 is provided on the upper surface or the lower surface of the display unit 161, and outputs the coordinates of the position designated by the operation user to the CPU 111. The touch panel 165 detects a position designated by the operation user with a finger or a stylus pen, and outputs the coordinates of the detected position to the CPU 111. The touch panel 165 is a multi-touch compatible touch panel, and outputs a plurality of coordinates corresponding to a plurality of positions simultaneously designated by the user to the CPU 111 when the user inputs a multi-touch operation. Further, the touch panel 165 outputs coordinates corresponding to a single position designated by the user to the CPU 111 when the user inputs a single touch operation.

  The touch panel 165 is preferably the same size or larger than the display surface of the display unit 161. Since the touch panel 165 is provided so as to overlap the display unit 161, the touch panel 165 indicates the coordinates of the position indicated by the operation user on the display surface of the display unit 161 when the operation user indicates the display surface of the display unit 161. It outputs to CPU111. For the touch panel 165, for example, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, or a capacitance method can be used, and the method is not limited. The operations that can be accepted by the operation unit 163 include an operation of pressing a plurality of keys of the hard key unit 167, a single touch operation that instructs the touch panel 165, and a multi-touch operation.

  FIG. 4 is a diagram illustrating an example of a software architecture of a CPU provided in the MFP. Referring to FIG. 4, an operating system (OS) layer is formed in CPU 111, and an MFP process layer is formed in a layer above it. An application platform (PF) layer and an application layer are formed in this order on the MFP process layer.

  The OS layer includes a task for the CPU 111 to execute the OS program. A task belonging to the OS layer executes processing for controlling hardware resources of the MFP 100. Here, the hardware resources include the communication I / F unit 112, the ROM 113, the RAM 114, the HDD 115, the facsimile unit 116, the external storage device 117, the automatic document feeder 120, the document reading unit 130, the image forming unit 140, and the paper feeding unit 150. The operation panel 160 is included. A task belonging to the OS layer controls hardware resources in accordance with an operating command input from the MFP process layer. Further, the OS layer shares operation identification information for identifying each of a plurality of types of operations that can be accepted by the operation unit 163, which is one of hardware resources, with the MFP process layer. When the operation unit 163 detects an operation by the operating user, the task belonging to the group outputs operation identification information indicating the detected operation to the MFP process layer.

  The application layer includes a task for the CPU 111 to execute an application program. When a plurality of types of application programs are installed in MFP 100, a plurality of tasks that respectively execute the plurality of types of application programs may belong to the application layer.

  The task for executing the application program executes a plurality of types of processing determined by the application program. The multiple types of processing include processing for causing a task belonging to the MFP process layer to execute processing executed in the MFP process layer. The task that executes the application program outputs an application command when the task belonging to the MFP process layer executes processing. The application command is a command published as an API (Application Program Interface). Therefore, an application program that causes MFP 100 to execute processing can be easily created. “Public” means a state in which a third party other than the manufacturer that manufactures the MFP 100 can use it. Therefore, a third party can develop an application program that can be installed in MFP 100 using an application command.

  Further, the task for executing the application program specifies a process to be executed from among a plurality of types of processes defined by the application program based on an operation input by an operation user operating MFP 100, and executes the process. An operation input by an operation user who operates MFP 100 is accepted in the OS layer when the operation user operates operation unit 163 to input.

  The application PF layer is arranged between the application layer and the MFP process layer, and arbitrates a plurality of tasks belonging to the application layer, and a task for controlling an application command output by the plurality of tasks belonging to the application layer belongs. Specifically, the application PF layer accepts application commands output by each of a plurality of tasks belonging to the application layer, standardizes the received application commands by converting them into internal commands according to a predetermined rule, The command is output to the MFP process layer. For this reason, it is possible to cope with a case where versions are different among a plurality of application programs. In addition, the application PF layer determines any one of a plurality of tasks belonging to the application layer as a current state. Then, an operation received from the task belonging to the MFP process layer and received from the OS layer is output to the current task determined as the current state among a plurality of tasks belonging to the application layer.

  The operations input by the user accepted in the OS layer include a single touch operation in which the touch panel 165 is instructed with one finger and a multi-touch operation in which the touch panel 165 is instructed simultaneously with two or more fingers. The single touch operation and the multi-touch operation are detected by the operating system as mouse events, respectively, and input from the application PF layer to the current task among a plurality of tasks belonging to the application layer. One or more mouse events are converted into gesture events in the MFP process layer, and input from the application PF layer to the current task among a plurality of tasks belonging to the application layer. For this reason, the application layer shares one or more gesture events that can be accepted by the application program with the MFP process layer, and the task for executing the application program is executed from the task belonging to the application PF layer by the application program. One or more acceptable gesture events are input. The task for executing the application program executes a process corresponding to the gesture event among a plurality of types of processes.

  Application commands and internal commands are associated in advance. For example, a command correspondence table in which application commands and internal commands are associated with each other may be stored. An application command and an internal command may correspond one-to-one, or a set of two or more internal commands may correspond to one application command. Furthermore, one internal command or a set of a plurality of internal commands may correspond to a plurality of application commands having different versions. For this reason, it is possible to cope with a case where versions are different among a plurality of application programs. The internal command is a command that depends on the hardware resources of MFP 100. The internal command is not normally disclosed, but may be disclosed.

  The MFP process layer is arranged between the application PF layer and the OS layer, and a task for the CPU 111 to execute the MFP process program belongs to it. The MFP process layer converts an internal command output by a task belonging to the application PF layer into an operating command interpretable by the task belonging to the OS layer, and converts the operating command to a task belonging to the OS layer in order to control hardware resources. Output. In actuality, internal commands may be converted into one or more operating commands that can be executed by tasks belonging to the OS layer, but for the sake of explanation, internal commands and tasks belonging to the OS layer can be executed. The relationship with one or more operating commands will be described as one-to-one.

  As described above, the application program causes MFP 100 to execute content display processing for displaying content such as Web pages and image data, copy processing, scan processing, print processing, facsimile transmission / reception processing, data transmission processing, and the like. Although it is a program, in the following description, the part relevant to the process which controls the operation panel 160 is mainly demonstrated among the processes which an application program makes CPU111 perform.

  FIG. 5 is a block diagram illustrating an example of functions of the CPU included in the MFP, together with information stored in the HDD. The functions shown in FIG. 5 are functions realized by CPU 111 when CPU 111 provided in MFP 100 executes a program stored in ROM 113, HDD 115, or CD-ROM 118. Specifically, the CPU 111 is realized by the CPU 111 by executing an OS program, an MFP process program, an operation reception program, and an application program. Here, the function of CPU 111 when a content display program is installed as an application program in MFP 100 is shown.

  Referring to FIG. 5, CPU 111 includes a content display unit 51, an arbitration unit 53, an operation conversion unit 55, and an operating unit 57. The content display unit 51 belongs to the application layer in the software architecture shown in FIG. The content display unit 51 is a function formed in the CPU 111 when the CPU 111 executes a content display program.

  The arbitration unit 53 and the operation conversion unit 55 are functions formed in the CPU 111 by a task in which the CPU 111 executes the operation reception program. The arbitration unit 53 belongs to the application PF layer in the software architecture shown in FIG. The operation conversion unit 55 belongs to the application PF layer in the software architecture shown in FIG. Although all the functions of the operation conversion unit 55 are functions of the application PF layer, at least some of the functions of the operation conversion unit 55 are functions of the application PF layer, and other parts are functions of the MFP process layer. Also good.

  When the CPU 111 executes another application program in addition to the content display unit 51 to form a plurality of tasks in the application layer, the arbitration unit 53 determines any one of the plurality of tasks as the current state. To do. Hereinafter, the task that the arbitration unit 53 has determined to be in the current state is referred to as a current task. In the present embodiment, since only the content display unit 51 that executes the content display program is included in the application layer, the content display unit 51 is determined to be in the current state.

  The arbitration unit 53 receives the application command output from the content display unit 51, and outputs application identification information for identifying the task that has output the application command and the application command to the operation conversion unit 55.

  When a plurality of application commands are output from the content display unit 51, the arbitration unit 53 determines their order and outputs them to the operation conversion unit 55 in order. For example, when a plurality of application commands cannot be executed simultaneously, the other application command is output to the operation conversion unit 55 after the execution of one application command by the operation conversion unit 55 is completed. Also, if the other app command cannot be executed until after one app command has been executed, the other app command can be executed even if the other app command is input prior to the other app command. Is output first.

  The operation conversion unit 55 receives the application command from the arbitration unit 53 and standardizes the application command by converting the application command into an internal command according to the command correspondence table. The command correspondence table associates one or more internal commands with one application command. The application commands included in the command correspondence table may include the same type of application commands with different versions. In this case, each different version of the application command is associated with one or more internal commands. For this reason, it is possible to install an application program in which application commands of different versions are described. When a new application command appears, the command correspondence table is updated with a command correspondence table in which one or more internal commands are associated with the newly appeared application command. Therefore, an application program describing a new application command can be installed.

  Further, the operation conversion unit 55 converts the standardized internal command into an operating command, and outputs the operating command to the operating unit 57. The operating command is a command that is predetermined between the operation conversion unit 55 and the operating unit 57 and that can be interpreted by the operating unit 57.

  The operating unit 57 is a function formed in the CPU 111 when the CPU 111 executes the OS program. The operating unit 57 belongs to the OS layer in the software architecture shown in FIG.

  The operating unit 57 receives an operating command output from the operation conversion unit 55 and controls hardware resources according to the operating command. As the hardware resources, here, the display unit 161, the touch panel 165 of the operation unit 163, and the HDD 115 will be described as an example. The operating unit 57 controls the display unit 161 to display an image on the display unit 161. In addition, the operating unit 57 controls the HDD 115 to store data in the HDD 115 and reads data stored in the HDD 115. Further, the operating unit 57 causes the touch panel 165 to detect a position designated by the user at a predetermined time interval, and acquires position information from the touch panel 165 at a predetermined time interval.

  Touch panel 165 detects the position instructed by the operation user when the operation user indicates the display surface of display unit 161, and outputs position information indicating the position of the display surface of display unit 161 to CPU 111. When the position information is input from the touch panel 165, the operating unit 57 outputs the position information to the operation conversion unit 55. The operating unit 57 may receive two or more pieces of position information from the touch panel 165 at the same time. For example, when the operating user designates two different parts of the display surface of the display unit 161 with two fingers at the same time, two pieces of position information indicating the two positions of the designated display surface are received. When the operating unit 57 simultaneously receives two or more pieces of position information from the touch panel 165, the operating unit 57 outputs the two or more pieces of position information to the operation conversion unit 55.

  The operation conversion unit 55 includes an association unit 91, a gesture event determination unit 93, a mouse event generation unit 95, and a display unit control unit 97. The associating unit 91 associates each of a plurality of types of processes defined by the content display program with one of a plurality of gesture events at the stage where the content display program is installed. Specifically, the associating unit 91 generates a master table in which a gesture event that can be received by the content display program is associated with each of a plurality of pieces of processing identification information determined by the content display program, and stores the master table in the HDD 115.

  FIG. 6 is a diagram illustrating an example of the master table. Referring to FIG. 6, master table 99 includes a master record for each of a plurality of process identification information. The master record includes a process item and a gesture item. Processing identification information is set in the processing item, and gesture identification information for identifying a gesture event is set in the gesture item.

  For example, in the master record in which the process identification information “page feed” is set in the process item, the gesture identification information “swipe” is set in the gesture item. The process specified by the process identification information “page feed” is a process of changing the screen to a screen of another page and displaying it. The gesture event specified by the gesture identification information “Swipe” corresponds to a user's operation of moving the finger to any one of up, down, left and right at a speed faster than a predetermined speed while instructing the touch panel 165. In the operation of moving the finger at a speed faster than a predetermined speed, the direction in which the finger is moved is determined. Therefore, the gesture event specified by the gesture identification information “swipe” includes the direction in which the finger is moved as a parameter.

  In the master record in which the process identification information “enlarge” is set in the process item, the gesture identification information “pinch out” is set in the gesture item. The process specified by the process identification information “enlargement” is a process for enlarging and displaying the screen. The gesture event specified by the gesture identification information “pinch out” corresponds to an operation of moving at least one of the designated two locations so that the interval between the two locations becomes long while designating the two locations on the touch panel 165. Since the operation of moving so that the interval between the two locations becomes longer, the interval between the two locations changes, the gesture event specified by the gesture identification information “pinch out” has the ratio of the change between the two locations as a magnification, The magnification is included as a parameter.

  In the master record in which the process identification information “reduction” is set in the process item, the gesture identification information “pinch-in” is set in the gesture item. The process specified by the process identification information “reduction” is a process of reducing the screen for display. The gesture event specified by the gesture identification information “pinch-in” corresponds to an operation of moving at least one of the designated two locations so that the interval between the two locations is shortened while designating the two locations on the touch panel 165. Since the operation of moving so that the interval between the two locations is shortened, the interval between the two locations changes, the gesture event specified by the gesture identification information “pinch-in” has the ratio of the change between the two locations as a magnification. Includes magnification as a parameter.

  In the master record in which the process identification information “rotation” is set in the process item, the gesture identification information “rotation” is set in the gesture item. The process specified by the process identification information “rotation” is a process of rotating the screen and changing the screen direction. The gesture event specified by the gesture identification information “Rotation” is an operation to move at least one of the two designated locations while changing the orientation of a straight line connecting the two locations while designating two locations on the touch panel 165. Detected when input. For example, this is an operation of drawing an arc on one of two places as the center. Since there are clockwise and counterclockwise directions for drawing the arc, the gesture event specified by the gesture identification information “rotation” includes the rotation direction and the rotation angle as parameters.

  In the master record in which the process identification information “scroll” is set in the process item, the gesture identification information “flick” and “double flick” are set in the gesture item. The process specified by the process identification information “scroll” is a process of scrolling the screen up / down / left / right. The gesture event specified by the gesture identification information “flick” corresponds to an operation in which the finger is moved at a speed slower than the threshold value in the operation of the swipe gesture event while instructing the touch panel 165, either vertically or horizontally. . The operation to move the finger at a speed slower than the threshold value determines the direction in which the finger is moved. Therefore, the gesture event specified by the gesture identification information “flick” uses the direction in which the finger is moved and the movement distance as parameters. Including. The gesture event specified by the gesture identification information “double flick” is slower than the threshold value in the gesture event operation of swiping two fingers either up, down, left, or right while pointing the touch panel 165 with two fingers. It corresponds to the operation to move in the same direction at the same time at the speed. The operation of moving two fingers simultaneously in the same direction at a speed slower than the threshold value determines the direction in which the finger is moved. Therefore, the gesture event specified by the gesture identification information “double flick” The parameters include the direction in which the two are moved in the same direction and the movement distance.

  Returning to FIG. 5, the display control unit 97 converts the application command input from the content display unit 51 into an operating command for causing the display unit 161 to display an image, and outputs the operating command to the operating unit 57. Thereby, the display image generated by the content display unit 51 is stored in the VRAM of the display unit 161, and at least a part of the display image is displayed on the display unit 161.

  The mouse event generation unit 95 generates a mouse event in response to position information input from the operating unit 57. Further, every time a mouse event is generated, the mouse event generation unit 95 outputs the generated mouse event to the gesture event determination unit 93. When a plurality of pieces of position information are simultaneously input from the operating unit 57, the mouse event generation unit 95 generates a plurality of mouse events corresponding to the plurality of pieces of position information. When generating one or more mouse events, the mouse event generation unit 95 outputs the generated one or more mouse events to the gesture event determination unit 93. The position information indicates a position on the display surface of the display unit 161 instructed by the operation user.

  In other words, the operation of the operation user instructing the display surface of the display unit 161 is an instruction start operation in which the user specifies an arbitrary position on the display surface of the display unit 161, in other words, while the user indicates the display surface of the display unit 161. , A moving operation for moving the position to be instructed while touching the display surface of the display unit 161, and an ending operation for the user to end the instruction on the display surface of the display unit 161. When the position information is input from the operating unit 57, the mouse event generation unit 95 determines whether the operation by the operating user is an instruction start operation, a movement operation, or an end operation. As described above, the operating unit 57 outputs the position information every time the position information is output from the touch panel 165. Therefore, based on the continuity of the position information that is continuously input as time passes, It is determined whether the operation by the user is an instruction start operation, a move operation, or an end operation.

  The mouse event generation unit 95 determines that the operation by the operating user is an instruction start operation with respect to the position information input after a predetermined time has passed without the position information being input from the operating unit 57, and the position information, A mouse event including state identification information “Press” for identifying the instruction start operation is generated.

  When one or more pieces of position information are continuously input from the operating unit 57 after detecting the mouse event of the state identification information “Press”, the mouse event generation unit 95 operates the operation user for each piece of one or more pieces of position information. The operation according to is determined as a moving operation. For each of one or more pieces of position information continuously input from the operating unit 57, the mouse event generation unit 95 generates a mouse event including the position information and state identification information “Move” for identifying the moving operation. Generate.

  When the mouse event generation unit 95 detects the mouse event of the state identification information “Move” and the position information is no longer input from the operating unit 57, the operation by the operating user is completed for the last input position information. It is determined as an operation, and a mouse event including the position information last input and state identification information “Release” for identifying the end operation is generated.

  More specifically, the mouse event generation unit 95 is based on position information input from the operating unit 57 at the first time and position information input at a second time after a predetermined time from the first time. To determine the state of the mouse event. The mouse event generation unit 95 has a predetermined distance from the position specified by the position information input at the first time before the first time with respect to the position information input at the first time. If position information indicating a position within the range is not input, the state of the mouse event for the position information input at the first time is determined as “Press”. When the position specified by the position information input at the first time does not exist within a predetermined distance from the position specified by the position information input at the second time, the mouse event generation unit 95 The state of the mouse event for the position information input at the second time is determined as “Press”.

  In addition, the mouse event generation unit 95 is configured such that the position specified by the position information input at the first time is within a predetermined distance from the position specified by the position information input at the second time. The mouse event state for the position information input at the second time is determined as “Move”. Further, the mouse event generation unit 95 determines that the position information within a predetermined distance from the position specified by the position information input at the first time is not input at the second time. The state of the mouse event for the position information input at the first time is determined as “Release”.

  The gesture event determination unit 93 determines a gesture event based on a plurality of mouse events input continuously from the mouse event generation unit 95. When one mouse event is continuously input from the mouse event generating unit 95 simultaneously, the gesture event determining unit 93 is a second mouse that is input after a predetermined time has elapsed since the first first mouse event was input. When an event is specified and the distance between the position specified by the first mouse event and the position specified by the second mouse event is equal to or greater than a predetermined distance, the gesture event is determined. If the distance between the positions is smaller than the predetermined distance, no gesture event is determined. The predetermined time can be arbitrarily determined. The predetermined time can be, for example, several times the interval at which the touch panel 165 detects the position. The predetermined distance can be arbitrarily determined, but is preferably 40 pixels, for example.

  When two or more mouse events are simultaneously input from the mouse event generating unit 95, the gesture event determining unit 93 determines a gesture event based on the two or more mouse events. Therefore, when the user inputs a multi-touch operation, the gesture event determination unit 93 determines a gesture event. Specifically, when the first mouse event and the second mouse event are simultaneously input from the mouse event generation unit 95, the gesture event determination unit 93 inputs continuously for the first mouse event. The gesture event is determined based on the first set of mouse events to be performed and the second set of mouse events input in succession to the second mouse event. The plurality of mouse events included in the first group have the same position information or continuous position information. Continuous means that two pieces of position information indicate positions within a predetermined distance. The plurality of mouse events included in the second group have the same position information or continuous position information. For this reason, the gesture event determination unit 93 determines whether a plurality of mouse events input simultaneously belong to the first group or the second group based on the position information included in them.

  When the mouse event is continuously input from the mouse event generation unit 95, the gesture event determination unit 93 has “Press” as the state identification information of the mouse event input first. When the mouse event whose state identification information is “Move” is input, the gesture event determination unit 93 receives the position information of the mouse event whose state identification information is “Press” and the mouse event whose state identification information is “Move”. The distance between the two points is calculated from the position information, and the time from the detection of the mouse event whose state identification information is “Press” until the detection of the mouse event whose state identification information is “Move” is calculated. The command speed is calculated from the measured distance. If the instruction speed is equal to or higher than the predetermined threshold TV, a swipe operation by the user is detected. If the moving speed is lower than the predetermined threshold TV, a flick operation by the user is detected. When a plurality of mouse events are input, a user's flick operation may be detected by the first plurality of mouse events, and a user's swipe operation may be detected by a plurality of subsequent mouse events. When detecting the flick operation and the swipe operation, the gesture event determination unit 93 calculates a direction from the position specified by the first position information to the position specified by the subsequent position information, and displays the display surface of the display unit 161. The direction closest to the calculated direction among the four directions (up, down, left, and right) is determined as a reference direction, and a gesture event including the specified direction as a parameter is determined. When detecting a swipe operation, the gesture event determination unit 93 determines a gesture event including gesture identification information “swipe” and a parameter of the pointing direction. When detecting a flick operation, the gesture event determination unit 93 outputs a gesture event including gesture identification information “flick”, a parameter in a pointing direction, and a parameter indicating a moving distance. The movement distance is a distance between two points calculated from the position information of the mouse event whose state identification information is “Press” and the position information of the mouse event whose state identification information is “Move”.

  When two mouse events are simultaneously input from the mouse event generation unit 95, the gesture event determination unit 93 has a state of “Press” for each of the two mouse events input first. In this case, the distance L1 between the two points and the angle R1 of the straight line connecting the two points are calculated based on the position information included in each of the two mouse events. The angle of the straight line connecting the two points may be the angle formed by the straight line connecting the two points with the reference line with the horizontal direction of the display unit 161 as the reference line. The angle formed between the straight line connecting the two points and the reference line is an angle formed clockwise from the reference line. Then, the gesture event determination unit 93 calculates the distance L2 between the two points and the angle R2 of the straight line connecting the two points based on the positional information of two mouse events that are input simultaneously.

  The gesture event determination unit 93 detects a pinch-out operation or a pinch-in operation by the user if the distance L2 and the distance L1 are different. If the distance L2 is larger than the distance L1, a pinch-out operation is detected. If the distance L2 is smaller than the distance L1, a pinch-in operation is detected. When detecting a pinch-out operation or a pinch-in operation, the gesture event determination unit 93 calculates a magnification by dividing the distance L2 by the distance L1, determines the calculated magnification as a parameter, and includes a gesture event including the magnification parameter To decide. When detecting a pinch-out operation, the gesture event determination unit 93 determines a gesture event including gesture identification information “pinch-out” and a magnification parameter. When detecting a pinch-in operation, the gesture event determination unit 93 determines a gesture event including gesture identification information “pinch-in” and a magnification parameter.

  The gesture event determination unit 93 detects a rotation operation by the user when the difference between the angle R1 and the angle R2 is equal to or greater than a predetermined threshold value TR. The threshold value R is a predetermined value, for example, preferably 90 degrees or less. When detecting the rotation operation, the gesture event determination unit 93 determines the rotation angle and the rotation direction by subtracting the angle R1 from the angle R2, and determines the determined rotation angle and the rotation direction as parameters. If the value obtained by subtracting the angle R1 from the angle R2 is positive, the rotational direction is clockwise. If the value obtained by subtracting the angle R1 from the angle R2 is negative, the rotational direction is counterclockwise. When detecting a rotation operation, the gesture event determination unit 93 determines a gesture event including gesture identification information “rotation”, a rotation angle parameter, and a rotation direction parameter.

  In addition, when the first and second mouse events whose state identification information is “Move” are simultaneously input, the gesture event determination unit 93 corresponds to the position information of the first mouse event and the first mouse event. The first movement direction is calculated from the position information of the mouse event whose state identification information is “Press”, and the position information of the second mouse event and the state identification information corresponding to the second mouse event are “Press”. A second movement direction is calculated from the position information of the mouse event. A direction from the position specified by the position information of the mouse event whose state identification information is “Press” to the position specified by the position information of the mouse event whose state identification information is “Move” is calculated as the moving direction. If the movement directions corresponding to the first mouse event and the second mouse event are the same, a double flick operation by the user is detected. Whether the direction corresponding to each of the first mouse event and the second mouse event is the same or not is determined if the difference in the moving direction corresponding to each of the first mouse event and the second mouse event is within a predetermined range. Good. When detecting a double-flick operation, the gesture event determination unit 93 outputs a gesture event including gesture identification information “double-flick”, an instruction direction parameter indicating a movement direction, and a movement distance parameter. The movement distance is a distance between the position specified by the position information of the first mouse event and the position specified by the position information of the mouse event whose state identification information corresponding to the first mouse event is “Press”. do it.

  Therefore, the gesture event determination unit 93 may determine a plurality of types of gesture events when a set of two mouse events is input simultaneously. Gesture event of gesture identification information “Pinch out”, Gesture event of gesture identification information “Rotation”, Gesture event of gesture identification information “Pinch in”, Gesture event of gesture identification information “Rotation”, Gesture identification information A gesture event of “pinch out” and a gesture event of gesture identification information “double flick”, a gesture event of gesture identification information “pinch in”, and a gesture event of gesture identification information “double flick”.

  When determining the gesture event, the gesture event determining unit 93 reads the master table 99 corresponding to the application identification information of the current task set by the arbitrating unit 53 from the HDD 115. Specifically, an operating command for reading the master table 99 is output to the operating unit 57, and the operating unit 57 controls the HDD 115 to acquire the master table 99 to be read. Hereinafter, a case where the content display unit 51 is the current task will be described.

  Based on the gesture event, the gesture event determining unit 93 determines the current that is set in the current state by the arbitrating unit 53 from among a plurality of types of processing defined in the master table 99 based on the gesture event. Identify the process associated with the task. Here, since the content display unit 51 is set as the current task by the arbitration unit 53, the gesture event determination unit 93 specifies the process associated with the gesture event and the master table 99. More specifically, the gesture event determination unit 93 extracts, from the master table 99, a master record in which the gesture identification information of the determined gesture event is set as an item of gesture, and an item of processing of the extracted master record Acquires the process identification information set in. When the process identification information is acquired, the gesture event determination unit 93 outputs the determined gesture event to the arbitration unit 53. However, when the process identification information is not acquired, the gesture event determination unit 93 does not output the determined gesture event to the arbitration unit 53. . In addition, when determining a plurality of gesture events, the gesture event determining unit 93 outputs all the gesture events whose processing is specified among the plurality of gesture events to the arbitrating unit 53. The gesture event output by the gesture event determination unit 93 includes gesture identification information for identifying the gesture event. In addition, the gesture event determination unit 93 is a case where mouse events are continuously input from the mouse event generation unit 95, and when the gesture event is not determined, an undetermined signal indicating that the gesture event is not determined is an arbitration unit To 53.

  The gesture event determination unit 93 outputs only gesture events corresponding to processing that can be executed by the content display unit 51, which is the current task set in the current state by the arbitration unit 53, to the arbitration unit 53. The gesture event is output to the content display unit 51 which is the current task. For this reason, it is possible to cause the content display unit 51 that executes the content display program to execute a gesture event that is an operation input to the touch panel 165 and is specified by a set of a plurality of mouse events. In other words, at the stage of developing the content display program, the MFP 100 can be developed without being aware of the gesture events that can be received by the touch panel 165 in the MFP 100. For example, a browsing program that does not accept rotation operations can be developed.

  The arbitration unit 53 receives the gesture event from the gesture event determination unit 93 and outputs the gesture event to the content display unit 51 set to the current state. The arbitration unit 53 outputs the gesture event to the content display unit 51 in response to the gesture event input from the gesture event determination unit 93. When a plurality of tasks are formed in the application layer by the CPU 111 executing another application program in addition to the content display unit 51, the arbitration unit 53 sets the gesture event to the task in the current state among the plurality of tasks. A gesture event is output from the determination unit 93.

  The content display unit 51 includes a content acquisition unit 61, a gesture period detection unit 63, a display image generation unit 65, and a display control unit 67.

  The content acquisition unit 61 acquires content and outputs the acquired content to the display image generation unit 65. The content includes data stored in an external server and data stored in the HDD 115, and the type of data is not limited. For example, when the content is a web page stored in an external web server and described in a markup language such as HTML (HyperText Markup Language), the content acquisition unit 61 controls the communication I / F unit 112. A web page is acquired by receiving a web page from a web server. If another content is linked to the Web page, the content acquisition unit 61 acquires the other content. The content acquired by the content acquisition unit 61 may be a Web page identified by a URL (Uniform Resource Locator) specified by an operation user operating the MFP 100, or may be identified by a URL predetermined in the content display program. Web page may be used. When the content is application data or image data stored in the HDD 116, the content acquisition unit 61 acquires the content by reading the application data or image data from the HDD 116.

  The gesture period detection unit 63 receives a gesture event from the gesture event determination unit 93. The gesture period detection unit 63 detects the gesture period based on the gesture event input from the gesture event determination unit 93, and displays the gesture period signal indicating the gesture period while detecting the gesture period. The data is output to the generation unit 65. The gesture period is a period in which gesture events are continuously determined by the gesture event determination unit 93. The gesture event input from the gesture event determination unit 93 has a gesture identification information of “swipe”, “flick”, “pinch out”, “pinch in”, “rotation”, “double flick”, and “double tap”. That's it.

  Specifically, the gesture event determining unit 93 determines a gesture event based on one or more mouse events. After the gesture event determining unit 93 determines a gesture event, the mouse event generating unit 95 continuously Mouse events may be generated. For example, when a user inputs a pinch-in operation, a gesture event is determined by the gesture event determination unit 93 while the user is moving two fingers. In this case, a mouse event of the state identification information “Move” is generated by the mouse event generation unit 95 until the user releases the finger from the touch panel 165, and when the user releases the finger from the touch panel 165, the mouse event generation unit 95 A mouse event of the state identification information “Release” is generated. After the gesture event determination unit 93 determines the gesture event of the gesture identification information “Pinch-in”, the gesture event determination unit 93 stores the state identification information “Move” until the mouse event of the state identification information “Release” is input from the mouse event generation unit 95. Each time a mouse event is input, a gesture event with gesture identification information “Pinch In” is output. The gesture period detection unit 63 detects the gesture period while the same gesture event is continuously input from the gesture event determination unit 93.

  When the user inputs a rotation operation, the gesture event having the gesture identification information “rotation” is determined by the gesture event determination unit 93 while the two users are moving two fingers. In some cases, the interval between fingers may fluctuate. For example, when the finger interval is narrowed, the gesture event determining unit 93 determines the gesture event having the gesture identification information “Pinch In”. The gesture event determination unit 93 outputs two gesture events when determining two gesture events after the gesture period starts. Even in this case, the gesture period detection unit 63 detects the start of the gesture period when the first gesture event is input from the gesture event determination unit 93, and one or more gesture events are not input from the gesture event determination unit 93. The end of the gesture period is detected.

  The display image generation unit 65 receives content from the content acquisition unit 61, receives a gesture period signal from the gesture period detection unit 63, and receives one or more gesture events from the gesture event determination unit 93. The display image generation unit 65 generates a display image for displaying content based on one or more gesture events input from the gesture event determination unit 93. When the content is composed of a plurality of pages, the display image generation unit 65 generates at least a part of an image of a page to be displayed among the plurality of pages as a display image. In other words, the display image generated by the display image generation unit 65 includes a part or all of a single page, and does not include a plurality of pages. The display image generation unit 65 includes a display target change unit 71, a display direction change unit 73, a state storage unit 75, and a priority display unit 77.

  When the content is input from the content acquisition unit 61, the display image generation unit 65 determines at least a part of the content as a display part, and generates a display image for displaying the display part. The display portion that the display image generation unit 65 determines at the time when content is input from the content acquisition unit 61 may be determined in advance. For example, the entire content may be set as a default display portion. If the content is composed of a plurality of pages, the entire first page may be set as the default display portion.

  The state storage unit 75 temporarily stores in the RAM 114 state information for specifying a display portion at the start of the gesture when the gesture period starts. The state information includes the position and size (magnification) of the display part in the content. When the content is composed of a plurality of pages, the state information includes a page number that identifies the page.

  When the first type of gesture event is input from the gesture event determining unit 93, the display target changing unit 71 changes the display part and generates a display image for displaying the changed display part. The first type of gesture events are gesture events of gesture identification information “Pinch In”, “Pinch Out”, “Swipe”, “Flick”, and “Double Flick”. Display target changing unit 71 includes a scaling unit 81, a page switching unit 83, a scroll unit 85, and a specific state page switching unit 86.

  When the gesture event having the gesture identification information “Pinch In” or “Pinch Out” is input from the gesture event determination unit 93, the scaling unit 81 changes the size of the display part. The gesture event of the gesture identification information “Pinch In” or “Pinch Out” includes a magnification as a parameter. The scaling unit 81 changes the size of the display part at a magnification included as a parameter in the gesture event of the gesture identification information “Pinch In” or “Pinch Out”.

  When the content includes a plurality of pages, and the gesture event of the gesture identification information “swipe” is input from the gesture event determination unit 93, the page switching unit 83 newly adds pages before and after the page to which the display portion belongs. The page to which the correct display part belongs is determined. The page before the page to which the display part belongs is the last page when the page to which the display part belongs is the first first page. The page after the page to which the display part belongs is the first page when the page to which the display part belongs is the last page. The page switching unit 83 specifies the page to which the display part belongs by the status information. The gesture event of the gesture identification information “Swipe” includes an instruction direction as a parameter. The page switching unit 83 determines a page determined by the instruction direction included as a parameter in the gesture event of the gesture identification information “Swipe”. Here, as an example, a case where the content includes the first to Nth pages (N is an integer of 2 or more) and the display portion is included in the Mth page (M is an integer of 1 to N) will be described as an example. To do. When the pointing direction included in the gesture event of the gesture identification information “Swipe” indicates “up” or “left”, the page switching unit 83 determines the (M−1) th page as the page to which the new display portion belongs. . When the pointing direction included in the gesture event of the gesture identification information “Swipe” indicates “down” or “right”, the page switching unit 83 determines the (M + 1) th page as the page to which the new display portion belongs.

  When the gesture event having the gesture identification information “flick” or “double flick” is input from the gesture event determination unit 93, the scroll unit 85 moves the display part in the content. The gesture event of the gesture identification information “flick” or “double flick” includes a parameter indicating a direction and a parameter indicating a moving distance. The scroll unit 85 translates the display portion by the movement distance in the direction included as a parameter in the gesture event of the gesture identification information “flick” or “double flick”. When the display part is located at the end of the content, the distance that the display part can be moved may be shorter than the movement distance determined by the parameter. In this case, the scroll unit 85 moves the display portion by a movable distance in the direction included as a parameter, and outputs a specific state signal indicating that the display portion has not been moved by the moving distance to the specific state page switching unit 86. The specific state signal includes a pointing direction included as a parameter in the gesture event of the gesture identification information “flick” or “double flick”.

  When the content includes a plurality of pages and the specific state signal is input from the scroll unit 85 during the gesture period, the specific state page switching unit 86 displays the display portion at the end of the gesture period when the gesture period ends. The pages before and after the page to which the page belongs belong to the page to which the new display portion belongs. When the gesture period detection unit 63 changes from the state in which the gesture period signal is input to the state in which no gesture period signal is input, the end of the gesture period is detected. The specific state page switching unit 86 specifies the page to which the display part belongs by the state information. The specific state page switching unit 86 determines a page determined by the instruction direction included in the specific state signal. Specifically, when the direction included in the specific state signal indicates “up” or “left”, the page before the page to which the display portion belongs is determined as the page to which the new display portion belongs. When the direction included in the specific state signal indicates “down” or “right”, the page after the page to which the display portion belongs is determined as the page to which the new display portion belongs.

  When the gesture event having the gesture identification information “Rotation” is input from the gesture event determining unit 93, the display direction changing unit 73 changes the direction of the display part. The gesture event of the gesture identification information “Rotation” includes a rotation angle parameter and a rotation direction parameter. The display direction changing unit 73 changes the direction of the display part by rotating the display part by the rotation angle in the rotation direction included as a parameter in the gesture event of the gesture identification information “Rotation”.

When the first type gesture event and the second type gesture event are input from the gesture event determination unit 93 during the gesture period, the priority display unit 77 uses the state information based on the second type gesture event. Change the identified display part. The first type of gesture events are gesture events of gesture identification information “Pinch In”, “Pinch Out”, “Swipe”, “Flick”, and “Double Flick”. The second type of gesture event is a gesture event having gesture identification information “Rotation”. The priority display unit 77 changes the direction of the display part determined by the state information. The direction of the display portion is changed by rotating the display portion by the rotation angle in the rotation direction included as a parameter in the gesture event of the gesture identification information “Rotation” last input from the gesture event determination unit 93 during the gesture period. <Gesture period>
The display image generation unit 65 sets the changed page as a processing target when the page is switched by the page switching unit 83 during the gesture period, and uses the state information when the page is not switched by the page switching unit 83. The specified page is the target of processing. In addition, when the display portion is changed by the scaling unit 81 and the scroll unit 85, the display image generation unit 65 generates a display image for displaying the changed display portion in the processing target page. In addition, when the display direction changing unit 73 changes the direction of the display portion, the display image generating unit 65 displays a display image for changing the direction of the display portion on the processing target page. Generate. Furthermore, the display image generation unit 65 is configured to process the page to be processed when the display portion is changed by the scaling unit 81 and the scroll unit 85 and when the display direction change unit 73 changes the direction of the display portion. The display image for displaying the display part after the change in is changed in direction.

<At the end of the gesture period>
The display image generation unit 65 displays a display part for displaying the display portion with the changed page as a processing target when the page is switched by the specific state page switching unit 86 at the end of the gesture period when the gesture period ends. An image is generated. The display part may be a predetermined part of the page. Further, the position and size in the page of the display portion determined at the end of the gesture period may be used.

  When the direction of the display part is changed by the priority display part 77 at the end of the gesture period when the gesture period ends, the display image generation unit 65 changes the direction of the display part on the processing target page. A display image for display is generated.

  The display control unit 67 displays the display image on the display unit 161 in response to the display image input from the display image generation unit 65. Specifically, the display control unit 67 outputs an application command to the arbitration unit 53 in order to display the display image on the display unit 161. Thus, the operating unit 57 stores the display image in a VRAM (Video RAM) included in the display unit 161, and the display image is displayed on the display unit 161.

  FIG. 7 is a flowchart illustrating an exemplary flow of a gesture event determination process. The gesture event determination process is a process executed by CPU 111 when CPU 111 of MFP 100 executes an operation reception program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIG. 7, CPU 111 determines whether a mouse event has been detected (step S01). If the user indicates touch panel 165, a mouse event is detected based on the position detected by touch panel 165. The process waits until a mouse event is detected (NO in step S01). If a mouse event is detected (YES in step S01), the process proceeds to step S02. In other words, the gesture event determination process is a process executed on the condition that the user instructs the touch panel 165 and a touch event is detected.

  In step S02, it is determined whether or not the detected mouse event is single. If a single mouse event is detected, the process proceeds to step S03. If a plurality of mouse events are detected, the process proceeds to step S21. In step S21, a multiple position detection process is executed, and the process ends. Details of the multiple position detection process will be described later.

  In step S03, the process branches depending on the state of the mouse event. If the mouse event state is “Press”, the process proceeds to step S04. If the mouse event state is “Release”, the process proceeds to step S05. If the mouse event state is “Move”, the process proceeds to step S12. In step S04, the position information included in the mouse event detected in step S01 is stored in RAM 114 as a start position, and the process ends.

  When the process proceeds to step S05, the state of the mouse event detected in step S01 is “Release”. When the state of the mouse event is “Release”, another mouse event is input immediately before the mouse event. The state of another mouse event may be “Press” or “Move”. In step S05, the process branches depending on the state of another mouse event input immediately before. If the state of another mouse event input immediately before is “Press”, the process proceeds to step S06; otherwise, the process proceeds to step S11. In step S11, an undetermined signal is output and the process ends.

  In step S06, it is determined whether or not the same position information as the position information included in the mouse event detected in step S01 is stored in RAM 114. If stored in RAM 114, the process proceeds to step S07; otherwise, the process proceeds to step S11. In addition, the same position information here includes the case where the distance between the two positions respectively specified by the two position information is within a predetermined length. In order to detect an operation in which the user points to the same position, an error is included.

  In step S07, the gesture event is determined to be a double tap, and the process proceeds to step S08. When the process proceeds to step S07, the state of the mouse event input immediately before is “Press”, the state of the mouse event input in step S01 is “Release”, and the same position information is stored in the RAM 114. Is stored. In other words, after the user inputs the first tap operation, the user taps the same position as the first tap operation.

  In step S08, it is determined whether or not a process corresponding to the determined gesture event is defined. Judgment is made based on whether or not there is a process associated with a gesture event corresponding to the application program set in the current state among a plurality of types of processes defined in the master table 99 stored in the HDD 115. To do. If a process is associated with the determined gesture event in the master table 99, it is determined that a process corresponding to the determined gesture event is defined corresponding to the application program set in the current state. . If a process corresponding to the determined gesture event is defined, the process proceeds to step S09; otherwise, the process proceeds to step S11. In step S09, the gesture event of the gesture identification information “double tap” determined in step S07 is output, and the process ends. In step S11, an undetermined signal is output and the process ends.

  When the process proceeds to step S10, the state of the mouse event input immediately before is “Press”, the state of the next mouse event input is “Release”, and the same position information is stored in the RAM 114. This is the case when it is not stored. In other words, the user inputs the first tap operation. In step S10, the position information included in the mouse event detected in step S01 is stored in RAM 114 as a start position, and the process proceeds to step S11. In step S11, an undetermined signal is output and the process ends.

  When the process proceeds to step S12, the mouse event state is “Move”, and another mouse event is input immediately before the mouse event. The state of another mouse event may be “Press” or “Move”. In step S12, it is determined whether or not the moving distance L is greater than or equal to a threshold value TH. If the movement distance L is greater than or equal to the threshold value TL, the process proceeds to step S13. If not, the process ends. The distance between the position specified by the start position stored in step S04 and the position specified by the position information included in the mouse event detected in step S01 is calculated as the movement distance L.

  In the next step S13, an instruction direction is determined. A direction from the position specified at the start position stored in step S04 to the position specified by the position information included in the mouse event detected in step S01 is determined as the designated direction. The designated direction is the closest direction among the four directions, top, bottom, left, and right, with the display surface of the display unit 161 as a reference.

  Then, the instruction speed is calculated. An instruction speed is calculated based on the movement distance calculated in step S12. For example, a value obtained by dividing the movement distance calculated in step S12 by the time from when the mouse event having the state “Press” is detected until the mouse event is detected in step S01 is set as the instruction speed. Further, the instruction speed may be calculated from the mouse event detected immediately before and the mouse event detected in step S01.

  In step S15, it is determined whether the indicated speed is equal to or higher than threshold value TV. If the instruction speed is equal to or higher than threshold TV, the process proceeds to step S16; otherwise, the process proceeds to step S17. In step S16, the gesture event is determined to be swipe, and the process proceeds to step S18. In step S17, the gesture event is determined to be flicked, and the process proceeds to step S18.

  In step S 18, it is determined whether or not the process corresponding to the determined gesture event is defined by master table 99 stored in HDD 115. If the process corresponding to the gesture event is defined, the process proceeds to step S19. If not, the process proceeds to step S20. In step S19, the gesture event of the gesture identification information “swipe” determined in step S16 or the gesture event of the gesture identification information “flick” determined in step S17 is output, and the process ends. In step S20, an undetermined signal is output and the process ends.

  FIG. 8 is a flowchart illustrating an example of the flow of the multiple position detection process. The multiple position detection process is a process executed in step S21 of the gesture event determination process shown in FIG. The multiple position detection process is a process executed when two mouse events are detected. Referring to FIG. 8, CPU 111 branches the process depending on the state of two mouse events. If at least one of the two mouse events is “Release”, the process proceeds to step S54. If “Press”, the process proceeds to step S32. If “Move”, the process proceeds to step S33. In step S32, the position information included in each of the two mouse events is stored in the RAM 114 as the start position, and the process ends.

  In step S33, it is determined whether or not the state of at least one of the two mouse events is “Move”. If the state of at least one of the two mouse events is “Move”, the process proceeds to step S34; otherwise, the process proceeds to step S54. In step S54, an undetermined signal is output, and the process returns to the gesture event detection process.

  In step S34, the rotation angle is calculated. Based on a predetermined reference side of the display surface of the display unit 161, an angle R1 formed by a straight line connecting the two positions specified at the start position stored in step S32 with the reference side is calculated. In addition, an angle R2 is calculated that a straight line connecting two positions specified by position information included in each of the two detected mouse events makes a reference side. Then, the rotation angle R is calculated by subtracting the angle R1 from the angle R2. It is determined whether or not the absolute value of the rotation angle R is greater than or equal to a threshold value TR. If the absolute value of rotation angle R is greater than or equal to threshold value TR, the process proceeds to step S36; otherwise, the process proceeds to step S40. In step S36, a rotation direction and a rotation angle are determined. The rotation direction is determined by the sign of the rotation angle R. The rotation angle is determined as the rotation angle R. In step S37, the gesture event is determined as rotation, and the process proceeds to step S38.

  In step S38, it is determined whether or not the process corresponding to the determined gesture event is defined by master table 99 stored in HDD 115. If the process corresponding to the gesture event is defined, the process proceeds to step S39. If not, step S39 is skipped and the process proceeds to step S40. In step S39, the gesture event of the gesture identification information “rotation” determined in step S37 is output.

  In step S40, the change amount of the distance between the designated positions is calculated. A distance L1 between the two positions specified by the start position stored in step S32 is calculated. In addition, a distance L2 between the two positions specified by the position information included in each of the two detected mouse events is calculated. The difference between the distance L2 and the distance L1 is calculated as the change amount of the distance between the indicated positions.

  In the next step S41, it is determined whether or not the distance has changed. If the change amount of the distance between the indicated positions is equal to or longer than a predetermined length, it is determined that the distance has changed. If the distance has changed, the process proceeds to step S43; otherwise, the process proceeds to step S48.

  In step S43, the magnification is determined. A value obtained by dividing the distance L2 by the distance L1 is determined as the magnification. In the next step S44, it is determined whether or not the magnification is 1 or larger. If the magnification is greater than 1, the process proceeds to step S44; otherwise, the process proceeds to step S45.

  In step S44, the gesture event is determined to be pinch out, and the process proceeds to step S46. In step S45, the gesture event is determined to be pinch-in, and the process proceeds to step S46.

  In step S <b> 46, it is determined whether or not the process corresponding to the determined gesture event is defined by master table 99 stored in HDD 115. If the process corresponding to the gesture event is defined, the process proceeds to step S47. If not, step S47 is skipped and the process proceeds to step S48. In step S47, the gesture event of the gesture identification information “pinch out” determined in step S44 or the gesture event of the gesture identification information “pinch in” determined in step S45 is output.

  In step S48, it is determined whether the moving directions of the two points are the same. The direction in which the position specified by the position information included in each of the two gesture events moves is compared. Specifically, the direction from one of the two positions specified at the start position stored in step S32 toward the position specified by the position information included in the corresponding one of the two detected mouse events is determined. , One moving direction is calculated. Further, the other direction is moved from the other of the two positions specified at the start position stored in step S32 toward the position specified by the position information included in the corresponding other of the two detected mouse events. Calculate as direction. If the calculated two directions are the same, the process proceeds to step S49; otherwise, the process proceeds to step S53.

  In step S49, an instruction direction is determined. The moving direction calculated in step S48 is determined as the designated direction. In step S50, the gesture event is determined to be a double flick, and the process proceeds to step S51. In step S 51, it is determined whether or not the process corresponding to the determined gesture event is defined by master table 99 stored in HDD 115. If the process corresponding to the gesture event is defined, the process proceeds to step S52. If not, step S52 is skipped and the process proceeds to step S53. In step S52, the gesture event of the gesture identification information “double flick” determined in step S50 is output.

  In step S53, it is determined whether a gesture event has been output. If at least one of step S39, step S47, and step S52 is executed, it is determined that a gesture event has been output. If at least one gesture event has been output, the process returns to the gesture event detection process; otherwise, the process proceeds to step S54. In step S54, an undetermined signal is output, and the process returns to the gesture event detection process.

  FIG. 9 is a flowchart illustrating an example of the flow of content display processing. The content display process is a process executed by CPU 111 when CPU 111 provided in MFP 100 executes a content display program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIG. 9, CPU 111 acquires content (step S61). For example, when the content is a Web page stored in an external Web server, the communication I / F unit 112 is controlled to receive the Web page from the Web server. If the content is application data or image data stored in the HDD 116, the application data or image data is read from the HDD 116.

  In step S62, a display target page is determined. If the content acquired in step S51 is a plurality of pages, the first first page is determined as the display target page, and if the content is a single page, the entire content is determined as the display target page. In step S63, a display portion is determined. At least a part of the display target page is determined as a display part. Here, a case where the entire display target page is determined as a display portion will be described as an example.

  In the next step S64, a display image corresponding to the display portion of the display target page determined in step S62 is generated. Then, the generated display image is displayed on the display unit 161 (step S65), and the process proceeds to step S66. By storing the display image in the VRAM of the display unit 161, the display image is displayed on the display unit 161.

  In step S66, the state information is stored. The state information includes the page number of the display target page, the position and size of the display portion in the page. In the next step S67, the non-scrollable flag is set to OFF. Then, it is determined whether or not a gesture vent has been determined (step S68). In the gesture event determination process described above, it is determined that a gesture event has been determined when a gesture event is output. The process waits until a gesture event is determined (NO in step S68). If a gesture event is determined (YES in step S68), the process proceeds to step S69.

  In step S69, the start time of the gesture period is set. The time at that time is set at the start of the gesture period. Then, a display image update process is executed (step S70), and the process proceeds to step S71. Although details of the display image update process will be described later, this is a process of updating the display image based on the gesture event detected in step S68 or step S71.

  In step S71, it is determined whether or not a gesture vent has been determined. If a gesture event is determined, the process returns to step S70; otherwise, the process proceeds to step S72. In step S72, it is determined whether an undetermined signal has been detected. An undetermined signal output in the gesture event determination process described above is detected. If an undetermined signal is detected, the process proceeds to step S73; otherwise, the process returns to step S71.

  In step S73, the end of the gesture period is set. The time at that time is set at the end of the gesture period. And a priority display process (step S74) and a specific state page switching process (step S75) are performed, and a process is advanced to step S76. Details of the priority display process and the specific state page switching process will be described later. In step S76, it is determined whether an instruction to end the display of content has been received. For example, if a key to which an instruction to end the content display program is assigned among a plurality of keys included in the hard key unit 167, an instruction to end the display of content is accepted. If an instruction to end display of contenton is received, the process ends. If not, the process returns to step S66.

  FIG. 10 is a flowchart illustrating an example of the flow of the display image update process. The display image update process is a process executed in step S70 of the content display process shown in FIG. In FIG. 9, the display target page and the display portion are determined and the gesture event is detected before step S70 is executed. Referring to FIG. 10, CPU 111 determines whether the detected gesture event is a pinch-out or pinch-in gesture event (step S81). If a gesture event with the gesture identification information “Pinch Out” or a gesture event with the gesture identification information “Pinch In” is detected, the process proceeds to Step S82; otherwise, the process proceeds to Step S83. In step S82, a magnification for changing the display portion is set. The magnification included as a parameter in the gesture event is set to a magnification for changing the display portion.

  In step S83, it is determined whether or not the detected gesture event is rotation. If a gesture event having the gesture identification information “Rotation” is detected, the process proceeds to step S84; otherwise, the process proceeds to step S86. In step S84, the rotation direction and rotation angle are acquired. The rotation direction and rotation angle included as parameters in the gesture event are acquired. In the next step S85, the acquired rotation direction and rotation angle are set to the rotation direction and rotation angle for generating the display image, and the process proceeds to step S86.

  In step S86, it is determined whether or not the detected gesture event is a flick or double flick gesture event. If a gesture event with gesture identification information “flick” or a gesture event with gesture identification information “double flick” is detected, the process proceeds to step S87; otherwise, the process proceeds to step S92. In step S87, the scroll direction and distance are acquired. The direction and the movement distance included as parameters in the gesture event are acquired. In the next step S88, it is determined whether or not the display portion can be changed. Specifically, it is determined whether or not the display portion can be translated by the movement distance acquired in the direction acquired in step S87. In the display target page, if there is a region having a width of the movement distance acquired in the direction acquired in step S87 with respect to the region where the display portion before the translation is arranged, the display portion can be changed. to decide. If it is determined that the display portion can be changed, the process proceeds to step S89; otherwise, the process proceeds to step S90.

  In step S89, the direction and movement distance acquired in step S87 are set to the scroll direction and movement distance for changing the display portion, and the process proceeds to step S92. On the other hand, in step S90, the non-scrollable flag is set to ON, and the process proceeds to step S91. In step S91, the direction acquired in step S87 is stored in RAM 114 as the scroll direction, and the process proceeds to step S92.

  In step S92, it is determined whether or not the detected gesture event is a swipe. If a gesture event having the gesture identification information “Swipe” is detected, the process proceeds to step S93; otherwise, the process proceeds to step S94. In step S93, the page feed direction is set, and the process proceeds to step S94. To do. The page turning direction is determined based on the direction included as a parameter in the gesture event. If the direction indicates “up” or “left”, the page turning direction indicating the previous page is determined. If the direction indicates “down” or “right”, the page turning direction indicating the subsequent page is determined.

  In step S94, the display portion is changed. The display part at the start of the gesture period is specified from the state information, and the specified display part is determined by the magnification set in step S82, the scroll direction and the moving distance set in step S89, and the page feed direction set in step S93. Based on the change. The state information includes a page number for specifying the display target page, the position of the display portion, and the size of the display portion.

  If step S93 is executed and the page feed direction is set, the page number of the display target page is changed to the page number of the page determined by the page feed direction. If the page feed direction indicates the previous page, the page number of the previous page is changed. If the page with the page number determined by the status information is the first page, the page number is changed to the last page. If the page feed direction indicates a subsequent page, the page number of the subsequent page is changed. If the page with the page number determined by the status information is the last page, the page number is changed to the page number of the first first page.

  When step S82 is executed and the magnification is set, the size of the display portion is changed to a size determined by the magnification. When step S89 is executed and the scroll direction and the movement distance are set, the position of the display portion is changed to a position translated by the movement distance in the direction determined by the scroll direction.

  In step S95, a display image is generated. In the content, an image of a portion corresponding to the display portion changed in step S95 is generated as a display image. When step S85 is executed and the rotation direction and rotation angle are set, an image obtained by rotating the direction of the display portion by the rotation angle in the rotation direction is set as a display image. The rotation angle may be classified for each predetermined range, the angle for rotating the display portion may be determined for each classified range in advance, and the rotation angle may be rotated by a predetermined angle with respect to the range to which the rotation angle belongs. For example, 90 degrees is set as an angle for rotating the display part in a range where the rotation angle is greater than 0 degree and 30 degrees or less, and 180 degrees is set as an angle for rotating the display part in a range where the rotation angle is greater than 30 degrees and 60 degrees or less. Make sure to define. When the rotation angle belongs to a range greater than 0 degree and 30 degrees or less, an image obtained by rotating the display portion by 90 degrees is used as a display image, and the rotation angle belongs to a range greater than 30 degrees and 60 degrees or less. Further, an image obtained by rotating the display portion by 180 degrees is set as a display image.

  In step S96, the generated display image is displayed on display unit 161, and the process returns to the content display process.

  FIG. 11 is a flowchart illustrating an example of the flow of priority display processing. The priority display process is a process executed in step S74 of the content display process shown in FIG. Referring to FIG. 11, CPU 111 determines whether or not a rotation operation exists during the gesture period (step S101). If a gesture event having the gesture identification information “Rotation” is detected during the gesture period, it is determined that a rotation operation has been performed. If there is a rotation operation, the process proceeds to step S102; otherwise, the process returns to the content display process.

  In step S102, it is determined whether a pinch-out operation or a pinch-in operation exists during the gesture period. If a gesture event having the gesture identification information “pinch out” is detected during the gesture period, it is determined that a pinch out operation has been performed. If a gesture event having the gesture identification information “Pinch-in” is detected during the gesture period, it is determined that a pinch-in operation has been performed. If at least one of the pinch-out operation and the pinch-in operation exists during the gesture period, the process proceeds to step S103. If neither the pinch-out operation or the pinch-in operation exists, the process returns to the content display process.

  In step S103, the display part is specified from the state information. This is because a display image is generated based on the display portion and the magnification at the start of the gesture period. In the next step S104, the rotation direction and rotation angle are set. The rotation direction and rotation angle included as parameters in the gesture event of the gesture identification information “rotation” detected last in the gesture period are set to the rotation direction and rotation angle for generating the display image.

  In step S105, a display image is generated based on the display portion specified in step S103 and the rotation angle in the rotation direction set in step S104. Specifically, an image obtained by rotating the image of the display portion specified in step S103 by the rotation angle in the rotation direction set in step S104 is set as a display image. The rotation angle is classified for each predetermined range, the angle for rotating the display portion is determined for each classified range in advance, and the rotation angle is rotated by a predetermined angle with respect to the range to which the rotation angle belongs. Good. For example, 90 degrees is set as an angle for rotating the display part in a range where the rotation angle is greater than 0 degree and 30 degrees or less, and 180 degrees is set as an angle for rotating the display part in a range where the rotation angle is greater than 30 degrees and 60 degrees or less. Make sure to define. When the rotation angle belongs to a range greater than 0 degree and 30 degrees or less, the display portion is rotated by 90 degrees, and when the rotation angle belongs to a range greater than 30 degrees and less than 60 degrees, the display portion is 180 degrees. Just rotate.

  In the next step S105, the generated display image is displayed on the display unit 161, and the process returns to the content display process.

  FIG. 12 is a flowchart illustrating an example of the flow of the specific state page switching process. The specific state page switching process is a process executed in step S75 of the content display process shown in FIG. Referring to FIG. 12, CPU 111 determines whether or not the non-scrollable flag is set to ON (step S111). The non-scrollable flag is set to OFF as an initial value at the start of the gesture period in step S67 of the content display process shown in FIG. 9, and scrolling can be performed in step S90 of the display image update process shown in FIG. Set to ON if not possible. If the non-scrollable flag is set to ON, the process proceeds to step S112. If not, the process returns to the content display process.

  In step S112, the page feed direction is determined from the scroll direction stored in the RAM 114. The scroll direction is stored in the RAM 114 when it is not possible to scroll in step S91 of the display image update process shown in FIG. If the scroll direction indicates “up” or “left”, the page feed direction is determined to indicate the previous page. If the scroll direction is “down” or “right”, the page shift direction is determined to indicate the next page. To do.

  In the next step S113, the display target page is changed. The page number of the display target page is changed to the page number of the page determined in the page feed direction. If the page feed direction indicates the previous page, the page number of the previous page is changed. If the page with the page number determined by the status information is the first page, the page number is changed to the last page. If the page feed direction indicates a subsequent page, the page number of the subsequent page is changed. If the page with the page number determined by the status information is the last page, the page number is changed to the page number of the first first page.

  In the next step S114, a display portion is determined. The display portion may be a predetermined portion for the page, or may be determined by the position and size of the display portion determined before changing the display target page.

  In the next step S115, a display image is generated. In the content, an image of a portion corresponding to the display portion of the display target page after being changed in step S113 is generated as a display image. Then, the generated display image is displayed on the display unit 161 (step S116), and the process returns to the content display process.

  Next, a case where priority display processing is executed will be described using a screen displayed on the display unit 161. FIG. 13 is a first diagram illustrating an example of a screen. Referring to FIG. 13, screen 301 includes a display image 303. The display image 303 is an entire image of one page of application data. In order to rotate the display image 303 to the right by 90 degrees while the screen 301 shown in FIG. 13 is being displayed on the display unit 161, the user may input a rotation operation. Specifically, the user may touch the touch panel with the thumb and index finger at the same time, and perform an operation of drawing an arc so that the position touched by the index finger is rotated clockwise around the position touched by the thumb. However, in the middle of inputting the rotation operation, the distance between the position touched by the thumb and the position touched by the index finger may be shortened. User operations in this case are accepted by MFP 100 as rotation operations and pinch-in operations.

  FIG. 14 is a second diagram illustrating an example of the screen. The screen 301A shown in FIG. 14 is displayed when the screen 301 shown in FIG. 13 is displayed and a rotation operation and a pinch-in operation are accepted as user operations, and processing corresponding to both of them is executed. Shows the screen. Referring to FIG. 14, screen 301A includes a display image 303A. The display image 303A is an image obtained by rotating the display screen 303 included in the screen 301 illustrated in FIG. 13 by 90 degrees clockwise and reducing the size to 50%.

  When MFP 100 according to the present embodiment accepts a rotation operation and a pinch-in operation in one gesture period, MFP 100 accepts the rotation operation and displays a display image as a result of not accepting the pinch-in operation when the gesture period ends. . Therefore, FIG. 14 that is an intermediate image is not necessarily displayed, and the screen shown in FIG. 15 described later may be displayed directly from the screen shown in FIG. In that case, the screen flicker may be reduced.

  FIG. 15 is a third diagram illustrating an example of the screen. A screen 301B shown in FIG. 15 is a screen that is displayed after the user finishes the operation after accepting a rotation operation and a pinch-in operation as a user operation while the screen 301 shown in FIG. 13 is displayed. Indicates. Referring to FIG. 15, screen 301B includes a display image 303B. The display image 303B is an image obtained by rotating the display image 303 included in the screen 301 illustrated in FIG. 13 by 90 degrees in the clockwise direction. The display image 303B has the same size as the display image 303.

  In the present embodiment, when a rotation operation and a pinch-in operation or a pinch-out operation are accepted during the gesture period, the display portion is changed during the gesture period and the direction of the display unit after the change is changed. The gesture identification information corresponding to the pinch-in operation or the pinch-out operation is determined after the gesture event of the “rotation” gesture identification information corresponding to the rotation operation is determined after the gesture period is started. The display portion may not be changed based on the “pinch-in” or “pinch-out gesture event. This allows the user to be notified that the rotation operation has priority over the pinch-in operation or the pinch-out operation. .

  Next, a case where the specific state page switching process is executed will be described using a screen displayed on the display unit 161. FIG. 16 is a fourth diagram illustrating an example of the screen. Referring to FIG. 16, screen 311 includes a display image 313. The display image 313 is a partial image below the first page of application data. When the screen 311 shown in FIG. 16 is displayed on the display unit 161, when the user inputs a downward scroll operation for moving the position touched by the finger 315 in the arrow direction in the drawing, the MFP 100 performs the scroll operation. Accept. However, since the display image 313 is a part below the first page of the application data, it cannot be scrolled downward. Therefore, when MFP 100 receives a scroll operation but cannot scroll, the page is switched in a direction corresponding to the scroll direction, and the screen shown in FIG. 17 is displayed.

  FIG. 17 is a fifth diagram illustrating an example of the screen. Referring to FIG. 17, screen 311A includes a display image 313A. The display image 313A is a partial image above the second page of the application data.

  As described above, MFP 100 according to the present embodiment functions as a data processing device and displays at least a part of a display portion of content. When the user touches and points on the touch panel 165, the first gesture event corresponding to the process of temporarily storing the state information for specifying the display part at the time of starting the gesture and changing the display part during the gesture period When the second gesture event that does not change the display portion is determined, a display image for displaying the display portion specified by the stored state information is generated and displayed based on the second gesture event. To do. Here, the first gesture event is determined by a flick operation, a double flick operation, a pinch-out operation, a pinch-in operation, and a swipe operation by the user. The second gesture event is determined by a rotation operation by the user. For this reason, the process for generating the display image based on the second gesture event during the gesture period has priority over the process for generating the display image based on the first gesture event. It can be made to correspond.

  In addition, when determining the first gesture event, the MFP 100 generates and displays a display image in which the display portion in the content is changed. When determining the second gesture event, the MFP 100 displays the display portion in which the direction of the display portion is changed. Generate and display an image. Since changing the direction of the display part has priority over changing the display part in the content, the user can easily input an operation to change the direction of the display part without changing the display part. it can.

  In addition, during the gesture period, the display part in the content is changed and the display image in which the direction of the display part is changed is displayed. When the gesture period ends, the display image in which the direction of the display part is changed is displayed. To do. For this reason, the user can continue the operation while confirming the display image.

  Further, after the second gesture event is determined after the gesture period is started, the display portion is not changed. Therefore, it is possible to notify the user that the second gesture event has priority over the first gesture event.

  Further, when a pinch-in operation or a pinch-out operation is accepted, the first gesture event is detected, the size of the display part is changed, and when a swipe operation is accepted, the first gesture event is detected, and the position of the display part in the content is detected. change. Therefore, the display image can be changed to an image obtained by enlarging or reducing a portion in the content, or an image of an arbitrary portion in the content.

  Furthermore, when two positions are detected at the same time, a pinch-in operation or a pinch-out operation in which the distance between the two positions detected at a predetermined time interval changes, or a difference between the movement directions of the two positions is a predetermined value. When a double flick operation within the range is accepted, a first gesture event is determined, two positions are detected at the same time, and a rotation operation in which the slope of a straight line passing through the two positions detected at predetermined time intervals changes by a predetermined angle is performed. If accepted, the second gesture event is determined. For this reason, the user can input a plurality of types of operations with two fingers.

  In addition, when a gesture event corresponding to a swipe operation is determined, the page is switched. When a gesture event corresponding to a flick operation or a double flick operation is determined, a display portion is changed in the page, and a flick operation or a double flick is performed. When the gesture event corresponding to the operation is determined, the page is switched when the display portion cannot be changed in the page. For this reason, when the display part cannot be changed in the page, the user switches the page when inputting the flick operation or the double flick operation, and therefore the user does not need to input the swipe operation again. As a result, the operation by the user can be simplified.

  Further, even if the display part can be changed within the page to be displayed at the start of the gesture, if there is a period during which the display part cannot be changed by the end of the gesture, the page is switched at the end of the gesture. For this reason, the display image of the display part of another page can be displayed at the end of the gesture period.

  In addition, every time a gesture event corresponding to a flick operation or a double flick operation is determined in the gesture period, the display portion is changed until the display portion cannot be changed in the display target page. For this reason, in the gesture period, the display portion can be changed in the display target page and the display image can be displayed, and the display portion can be displayed in the display target page by one operation input by the user. And the display image of the display portion after the page is switched can be displayed.

  In the above-described embodiment, MFP 100 has been described as an example of a data processing apparatus. However, a content display method that causes CPU 111 that controls MFP 100 to execute the processes illustrated in FIGS. It goes without saying that the invention can be understood as a content display program for causing the CPU 111 to execute the method.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

  DESCRIPTION OF SYMBOLS 100 MFP, 51 Content display part, 53 Arbitration part, 55 Operation conversion part, 57 Operating part, 61 Content acquisition part, 63 Gesture period detection part, 65 Display image generation part, 67 Display control part, 71 Display object change part, 73 display direction change unit, 75 status storage unit, 77 priority display unit, 81 scaling unit, 83 page switching unit, 85 scroll unit, 86 specific status page switching unit, 91 association unit, 93 gesture event determination unit, 95 mouse Event generation unit, 97 display unit control unit, 99 master table, 110 main circuit, 111 CPU, 112 communication I / F unit, 113 ROM, 114 RAM, 115 HDD, 116 facsimile unit, 117 external storage device, 120 automatic document conveyance Device, 130 Document reading unit, 140 images Generating unit, 150 a sheet feeding unit, 160 operation panel, 161 display unit, 163 operation unit, 165 touch panel, 167 hard key unit.

Claims (9)

Display means capable of displaying images;
Position detection means capable of detecting one or more positions designated by a user on the display surface of the display means;
Control means,
The control means includes content display means for displaying an image of at least a part of a display portion of one page to be displayed among a plurality of pages included in the content on the display means,
Gesture event determining means for determining at least one of a plurality of gesture events based on the one or more positions in response to detection of one or more positions by the position detecting means;
The content display means includes a display image generation means for generating a display image for displaying the display portion based on at least one gesture event determined by the gesture event determination means;
Display control means for displaying the generated display image on the display means,
The display image generating means, when a first gesture event is determined by the gesture event determining means, a page switching means for changing the display target page to another page;
When a second gesture event is determined by the gesture event determining means, scroll means for changing a display part in the display target page;
In the case where the second gesture event is determined by the gesture event determining means, if the display portion cannot be changed by the scroll means, the specific state page switching means for changing the display target page to another page, Including data processing apparatus.   The specific state page switching unit is configured to display the gesture even if the scroll unit can change a display portion at a gesture start time when a gesture period in which one or more positions are continuously detected by the position detection unit starts. The page to be displayed is changed to another page at the end of the gesture when there is a period during which the display part cannot be changed by the scroll means by the end of the gesture at which the period ends. Data processing equipment.   The scroll means changes the display part until the display part cannot be changed in the display target page every time the second gesture event is determined by the gesture event determination means in the gesture period. The data processing apparatus according to claim 2. The gesture event determining means is a flick determining means for determining a first gesture event when a moving speed of one position detected by the position detecting means is smaller than a threshold value;
4. The data according to claim 1, further comprising: a swipe determining unit that determines a second gesture event when a moving speed of one position detected by the position detecting unit is equal to or higher than a threshold value. Processing equipment. A content display method executed by a data processing device,
The data processing device includes display means capable of displaying an image;
Position detection means capable of detecting one or more positions indicated by a user in the display surface of the display means,
A content display step of displaying, on the display means, an image of at least a part of a display portion of one page to be displayed among a plurality of pages included in the content;
A gesture event determining step for determining at least one of a plurality of gesture events based on the one or more positions in response to detection of one or more positions by the position detecting means; Let it run
The content display step includes: a display image generation step for generating a display image for displaying the display portion based on at least one gesture event determined in the gesture event determination step;
A display control step of displaying the generated display image on the display means,
The display image generation step includes a page switching step for changing the display target page to another page when the first gesture event is determined in the gesture event determination step;
When a second gesture event is determined in the gesture event determining step, a scrolling step of changing a display part in the display target page;
In the case where the second gesture event is determined in the gesture event determining step, if the display part cannot be changed in the scrolling step, the specific state page switching step for changing the display target page to another page, Including content display method.   In the specific state page switching step, even if the display portion can be changed in the scroll step at the start of a gesture at which a gesture period in which one or more positions are continuously detected by the position detecting unit is started, The method includes a step of changing the page to be displayed to another page at the end of the gesture when there is a period during which the display portion cannot be changed in the scroll step by the end of the gesture when the period ends. 5. The content display method according to 5.   The scrolling step is a step of changing the display part in the gesture period until the display part cannot be changed in the display target page every time the second gesture event is determined in the gesture event determining step. The content display method according to claim 6, comprising: The gesture event determining step includes a flick determining step of determining a first gesture event when a moving speed of one position detected by the position detecting unit is smaller than a threshold value;
The content according to any one of claims 5 to 7, further comprising: a swipe determining step of determining a second gesture event when a moving speed of one position detected by the position detecting means is equal to or higher than a threshold value. Display method. A content display program executed by a computer that controls the data processing device,
The data processing device includes display means capable of displaying an image;
Position detection means capable of detecting one or more positions indicated by a user in the display surface of the display means,
A content display step of displaying on the display means an image of at least a part of a display portion of one page to be displayed among a plurality of pages included in the content;
Causing the computer to execute a gesture event determining step of determining at least one of a plurality of gesture events based on the one or more positions in response to detection of one or more positions by the position detecting means. ,
The content display step includes: a display image generation step for generating a display image for displaying the display portion based on at least one gesture event determined in the gesture event determination step;
A display control step of displaying the generated display image on the display means,
The display image generation step includes a page switching step for changing the display target page to another page when the first gesture event is determined in the gesture event determination step;
When a second gesture event is determined in the gesture event determining step, a scrolling step of changing a display part in the display target page;
In the case where the second gesture event is determined in the gesture event determining step, if the display part cannot be changed in the scrolling step, the specific state page switching step for changing the display target page to another page, Including content display program.

Images