JP2014106807A - Data processing apparatus, operation reception method, and browsing program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To appropriately execute an application program different from a browsing program when a content includes a description for executing the application program.SOLUTION: A data processing apparatus includes: a browsing unit 61 that displays a display image corresponding to a content; a mouse event generation unit 85 that generates a mouse event showing one or more positions designated by a user; and a gesture event determination unit 83 that determines one gesture event on the basis of one or more of pairs of mouse events. The browsing unit 61 includes: a display image creation unit 71 that creates a display image having an embedded image created through the execution of an application program included in the content arranged in an embedding area; and a selection unit 75 that selects the mouse event when the mouse events show positions in the embedding area, and selects the gesture event when the mouse events show positions outside the embedding area.

Description

  The present invention relates to a data processing device, an operation reception method, and a browsing program, and more particularly, to a data processing device including a multi-touch compatible touch panel, an operation reception method and a browsing program executed by the data processing device.

  In recent years, in a data processing apparatus represented by a multifunction peripheral (MFP), as a user interface, there is an apparatus provided with a multi-touch compatible touch panel capable of detecting a plurality of positions designated by a user at the same time. On the other hand, some MFPs can install an application program, and a user can customize the MFP by installing an application program according to a usage pattern. This application program has a specification that can accept a multi-touch operation in which a user simultaneously designates a plurality of positions and a single touch operation in which a user designates one position at a time and a multi-touch operation as a user interface. There is a specification that cannot accept single touch operation but can accept it. For this reason, when a plurality of application programs are executed simultaneously, it is necessary to switch between accepting an operation input by the user as a multi-touch operation or a single touch operation.

  For example, Japanese Patent Application Laid-Open No. 2009-2111704 discloses a method for responding to a mouse event in a multi-touch device, the step of displaying one or more views, and one or more each of which is associated with a specific view. Executing the above software elements; associating a multi-touch flag or exclusive touch flag with individual views; receiving one or more touches in the one or more views; Based on the value of the flag and the exclusive touch flag, one or more mouse events each describing a received touch are transmitted to one or more of the one or more views associated with the one or more views that received the touch. Selectively sending to the software element. How to symptoms are described.

However, depending on the application program, an image (view) output by the application program may include an area for receiving a multi-touch operation and an area for receiving a single touch operation. Therefore, in the conventional method, depending on the view, an operation input by the user is switched between accepting as a multi-touch operation or accepting as a single touch operation, so that the multi-touch operation is accepted in one view (image). There is a problem that it is not possible to cope with a case where a region and a region for receiving a single touch operation are included.
JP 2009-2111704 A

  The present invention has been made to solve the above-described problems, and one of the objects of the present invention is to appropriately execute an application program when the content includes a description for executing an application program different from the browsing program. It is to provide a data processing device capable of doing so.

  Another object of the present invention is to provide an operation reception method capable of receiving one operation by a user as an operation suitable for each of a browsing program and an application program.

  Still another object of the present invention is to provide a browsing program capable of accepting one operation by a user as an operation suitable for each of the browsing program and the application program.

  In order to achieve the above-described object, according to one aspect of the present invention, a data processing device detects display means capable of displaying an image, and one or more positions indicated by a user on the display surface of the display means. A touch panel capable of controlling, and a control unit. The control unit executes a browsing program, acquires content described in a markup language from the outside, and at least a part of a display image corresponding to the acquired content. Corresponding to the input gesture event among the plurality of types of processing in response to the input of any one of the plurality of gesture events corresponding to the plurality of types of processing defined by the browsing program. One or more positions are detected by the browsing means for executing the type of processing and the touch panel. Accordingly, based on a set of mouse event generation means for generating a mouse event including position information indicating each of one or more positions, and one or more mouse events generated successively by the mouse event generation means, Gesture event determining means for determining any one of a plurality of gesture events, and the browsing means includes a description for displaying an embedded image in which content is generated by executing an application program different from the browsing program. If included, display image generating means for generating a display image in which an embedded image generated by executing an application program is arranged in an embedded area defined by content, and if the display image includes an embedded area, a mouse event The position indicated by the mouse event input from the generation means Selection for selecting a mouse event to be input if it is within the embedded area, and selecting a gesture event to be input from the gesture event determining means if the position indicated by the mouse event input from the mouse event generating means is outside the embedded area Means.

  According to this aspect, in response to detection of one or more positions designated by the user, a mouse event including position information indicating each of the one or more positions is generated, and the one or more generated continuously Based on the set of mouse events, any one of a plurality of gesture events is determined, and the content received from the outside displays an embedded image generated by executing an application program different from the browsing program When a description is included, a display image in which an embedded image is arranged in an embedded area defined by the content is generated. When the display image includes an embedded area, a mouse event and a gesture are based on the position indicated by the mouse event. One of the events is selected. For this reason, one operation by the user is detected as a mouse event and a gesture event, the detected gesture event is made to correspond to the browsing program, and the mouse event is made to correspond to the application program. For this reason, when the browsing program corresponds to a gesture event and the application program corresponds to a mouse event, the application program can be appropriately executed. As a result, it is possible to provide a data processing apparatus capable of appropriately executing an application program even when the content includes a description for executing an application program different from the browsing program.

  Preferably, the browsing means further includes instruction means for outputting the mouse event to a task for executing the application program and updating the embedded image displayed in the embedded area when the mouse event is selected.

  If this aspect is followed, the task which performs an application program can be made to perform a process according to a mouse event.

  Preferably, when a gesture event is selected, the browsing means executes a process corresponding to the selected gesture event among a plurality of kinds of processes, thereby displaying the display image to be displayed on the display means. Further included is a determining means for determining the portion.

  According to this aspect, the browsing program can be executed according to the gesture event.

  Preferably, the control means further includes association means for associating a plurality of gesture events corresponding to a plurality of types of processing defined by the browsing program corresponding to the browsing program when the browsing program is installed. The means is that a gesture event corresponding to a set of one or more mouse events continuously generated by the mouse event generation means is associated with one of a plurality of types of processing corresponding to the browsing program by the association means. Determine the gesture event on the condition.

  According to this aspect, when a browsing program is installed, a plurality of gesture events corresponding to a plurality of types of processing defined by the browsing program are associated, so only a gesture event corresponding to each of the plurality of types of processing defined by the browsing program is associated. Can be detected.

  According to another aspect of the present invention, an operation reception method includes display means capable of displaying an image, and a touch panel capable of detecting one or more positions designated by the user on the display surface of the display means. An operation reception method executed by a data processing apparatus that executes a browsing program, acquires content described in a markup language from the outside, and displays at least a part of a display image corresponding to the acquired content A browsing step for displaying on the means, a mouse event generating step for generating a mouse event including position information indicating each of the one or more positions in response to detection of one or more positions by the touch panel, and a mouse event generating step. Multiple gesture events based on a set of one or more mouse events generated in succession A gesture event determining step for determining any one of them, and the browsing step includes an application program when the content includes a description for displaying an embedded image generated by executing an application program different from the browsing program. A display image generation step for generating a display image in which an embedded image generated by executing is placed in an embedded region defined by the content, and if the display image includes an embedded region, it is generated by a mouse event generation step. If the position indicated by the mouse event is within the embedded area, the mouse event is selected. If the position indicated by the mouse event generated by the mouse event generating step is outside the embedded area, the gesture determined in the gesture event determining step is selected. Event Selecting comprises a selection step, the.

According to this aspect, one operation by the user is detected as a mouse event and a gesture event, and the detected gesture event is made to correspond to the browsing program, and the mouse event is made to correspond to the application program. For this reason, when the browsing program corresponds to a gesture event and the application program corresponds to a mouse event, one operation by the user can be accepted as an operation suitable for each of the browsing program and the application program. As a result, it is possible to provide an operation receiving method capable of receiving one operation by the user as an operation suitable for each of the browsing program and the application program.

Preferably, when the mouse event is selected in the selection step, the browsing step outputs the mouse event to a task that executes the application program, and further includes an instruction step for updating the embedded image displayed in the embedded region. Including.

  If this aspect is followed, the task which performs an application program can be made to perform a process according to a mouse event.

  Preferably, when a gesture event is selected in the selection step, the browsing step displays the display image on the display unit by executing a type of processing corresponding to the selected gesture event among a plurality of types of processing. And a determining step for determining a display portion for.

  According to this aspect, the browsing program can be executed according to the gesture event.

  Preferably, the method further includes an association step of associating a plurality of gesture events corresponding to a plurality of types of processing defined by the browsing program corresponding to the browsing program when the browsing program is installed. On condition that a gesture event corresponding to a set of one or more mouse events generated continuously in the event generation step is associated with one of a plurality of types of processing corresponding to the browsing program in the association step. Determining a gesture event.

  According to this aspect, it is possible to detect only gesture events corresponding to a plurality of types of processing determined by the browsing program.

  According to still another aspect of the present invention, the browsing program is a browsing program executed by a computer that controls the data processing device, and the data processing device includes a display unit capable of displaying an image, and a display of the display unit. A mouse event including position information indicating each of one or more positions is generated in response to detection of one or more positions on the touch panel that can be detected by the user and one or more positions indicated by the user. Mouse event generation means, and gesture event determination means for determining any one of a plurality of gesture events based on a set of one or more mouse events generated successively by the mouse event generation means. Execute a browsing program and write in markup language from the outside. A browsing step for displaying at least a part of a display image corresponding to the acquired content on a display means is executed by a computer, and the browsing step is executed by an application program different from the browsing program. A display image generating step for generating a display image in which an embedded image generated by executing an application program is arranged in an embedded region defined by the content when the description includes a description for displaying the generated embedded image When the display image includes an embedded area, if the position indicated by the mouse event generated by the mouse event generating means is within the embedded area, the mouse event is selected and indicated by the mouse event generated by the mouse event generating step. The position to be embedded is an embedded area If comprising a selection step of selecting a gesture event determined in the gesture event determining step.

  According to this aspect, one operation by the user is detected as a mouse event and a gesture event, and the detected gesture event is made to correspond to the browsing program, and the mouse event is made to correspond to the application program. For this reason, when the browsing program corresponds to a gesture event and the application program corresponds to a mouse event, one operation by the user can be accepted as an operation suitable for each of the browsing program and the application program. As a result, it is possible to provide a browsing program that can accept one operation by the user as an operation suitable for each of the browsing program and the application program.

  Preferably, when the mouse event is selected in the selection step, the browsing step outputs the mouse event to a task that executes the application program, and further includes an instruction step for updating the embedded image displayed in the embedded region. Including.

  Preferably, when a gesture event is selected in the selection step, the browsing step displays the display image on the display unit by executing a type of processing corresponding to the selected gesture event among a plurality of types of processing. And a determining step for determining a display portion for.

1 is a perspective view showing an appearance of an MFP in one embodiment of the present invention. It is a top view which shows an example of an operation panel. 2 is a block diagram illustrating an example of an outline of a hardware configuration of an MFP. FIG. 2 is a diagram illustrating an example of a software architecture of a CPU provided in an MFP. FIG. 3 is a block diagram illustrating an example of functions of a CPU included in an MFP, together with information stored in an HDD. FIG. It is a figure which shows an example of a master table. FIG. 5 is a diagram illustrating an example of a flow of processing performed by a user and an operation performed by an MFP. It is a flowchart which shows an example of the flow of a gesture event determination process. It is a flowchart which shows an example of the flow of a browsing process. It is a figure which shows an example of a display image. It is a 1st figure which shows an example of the display part in a display image. It is a 2nd figure which shows an example of the display part in a display image. It is a 3rd figure which shows an example of the display part in a display image.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  FIG. 1 is a perspective view showing an external appearance of an MFP according to one embodiment. FIG. 2 is a plan view showing an example of the operation panel. Referring to FIGS. 1 and 2, MFP 100 includes a document reading unit 130 for reading a document, an automatic document feeder 120 for conveying a document to document reading unit 130, and document reading unit 130 reads a document. An image forming unit 140 for forming an image on a sheet or the like based on image data to be output, a sheet feeding unit 150 for supplying the sheet to the image forming unit 140, and an operation panel 160 as a user interface. .

  Operation panel 160 is provided on the upper surface of MFP 100. Operation panel 160 includes a display unit 161, a touch panel 165, and a hard key unit 167. The display unit 161 is, for example, a liquid crystal display device (LCD), and displays an instruction menu for a user, information about acquired image data, and the like. The hard key portion 167 includes four hard keys 167A to 167D each representing characters of “BOX”, “FAX”, “COPY”, and “ECO”. The touch panel 165 is a multi-touch compatible touch panel provided on the upper surface or the lower surface of the display unit 161 so as to overlap the display unit 161, and detects a position designated by the user on the display surface of the display unit 161. The operation in which the user instructs the touch panel 165 includes a multi-touch operation for instructing the touch panel 165 with a finger at the same time, and a single touch operation for instructing one place with a finger at a time.

  The automatic document feeder 120 automatically conveys a plurality of documents set on the document feeding tray one by one to a predetermined document reading position set on the platen glass of the document reading unit 130, and reads the document. The document whose document image is read by the unit 130 is discharged onto a document discharge tray. The document reading unit 130 includes a light source that irradiates light to the document conveyed to the document reading position and a photoelectric conversion element that receives light reflected from the document, and scans a document image corresponding to the size of the document. The photoelectric conversion element converts the received light into image data that is an electrical signal and outputs the image data to the image forming unit 140. The paper feed unit 150 conveys the paper stored in the paper feed tray to the image forming unit 140.

  The image forming unit 140 forms an image by a well-known electrophotographic method, and the image data after data processing is obtained by performing various data processing such as shading correction on the image data input from the document reading unit 130. Alternatively, an image is formed on a sheet conveyed by the sheet feeding unit 150 based on image data received from the outside.

  FIG. 3 is a block diagram illustrating an example of an outline of the hardware configuration of the MFP. Referring to FIG. 3, MFP 100 includes a main circuit 110. The main circuit 110 includes a CPU 111, a communication interface (I / F) unit 112, a ROM (Read Only Memory) 113, a RAM (Random Access Memory) 114, and a hard disk drive (HDD) 115 as a mass storage device. A facsimile unit 116 and an external storage device 117. CPU 111 is connected to automatic document feeder 120, document reading unit 130, image forming unit 140, sheet feeding unit 150, and operation panel 160, and controls the entire MFP 100.

  The ROM 113 stores a program executed by the CPU 111 or data necessary for executing the program. The RAM 114 is used as a work area when the CPU 111 executes a program. Further, the RAM 114 temporarily stores the read images continuously sent from the document reading unit 130.

  The facsimile unit 116 is connected to the public switched telephone network (PSTN) and transmits facsimile data to the PSTN or receives facsimile data from the PSTN. The facsimile unit 116 stores the received facsimile data in the HDD 115 or converts the received facsimile data into print data that can be printed by the image forming unit 140 and outputs the print data to the image forming unit 140. As a result, the image forming unit 140 forms an image on a sheet of facsimile data received by the facsimile unit 116. Further, the facsimile unit 116 converts the data read by the document reading unit 130 or the data stored in the HDD 115 into facsimile data, and transmits the facsimile data to a facsimile machine connected to the PSTN.

  Communication I / F unit 112 is an interface for connecting MFP 100 to a network. The communication I / F unit 112 communicates with other computers connected to the network using a communication protocol such as TCP (Transmission Control Protocol) or FTP (File Transfer Protocol). Note that the protocol for communication is not particularly limited, and an arbitrary protocol can be used. The network to which the communication I / F unit 112 is connected is, for example, a local area network (LAN), and the connection form may be wired or wireless. The network is not limited to the LAN, and may be a network using a wide area network (WAN), a public switched telephone network, or the like. Furthermore, the network is connected to the Internet. Therefore, MFP 100 can communicate with a computer such as a server connected to the Internet.

  The external storage device 117 is controlled by the CPU 111, and a CD-ROM (Compact Disk Read Only Memory) 118 or a semiconductor memory is mounted. The CPU 111 can access the CD-ROM 118 or the semiconductor memory via the external storage device 117. The CPU 111 loads the program recorded in the CD-ROM 118 or semiconductor memory mounted on the external storage device 117 into the RAM 114 and executes it. Note that the program executed by the CPU 111 is not limited to the program recorded on the CD-ROM 118, and the program stored in the HDD 115 may be loaded into the RAM 114 and executed. In this case, another computer connected to the network rewrites the program stored in HDD 115 of MFP 100 or adds a new program and writes it via the network connected to communication I / F unit 112. You may do it. Further, MFP 100 may download a program from another computer connected to the network and store the program in HDD 115. The program here includes not only a program directly executable by the CPU 111 but also a source program, a compressed program, an encrypted program, and the like.

  The medium for storing the program executed by the CPU 111 is not limited to the CD-ROM 118, but an optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC card, optical card, It may be a semiconductor memory such as a mask ROM, an EPROM (Erasable Programmable ROM), or an EEPROM (Electrically EPROM).

  Operation panel 160 includes a display unit 161 and an operation unit 163. The display unit 161 is a display such as a liquid crystal display (LCD) or an organic ELD (Electro-Luminescence Display), and displays an instruction menu for the user, information about acquired image data, and the like. The operation unit 163 includes a touch panel 165 and a hard key unit 167 including a plurality of keys. Each of the plurality of keys included in the hard key unit 167 includes a contact switch and is connected to the CPU 111. When the hard key is pressed by the operation user, the contact is closed and the circuit connected to the CPU 111 is closed. The hard key closes the circuit while being pressed by the operating user who operates the MFP 100, and opens the circuit when not pressed by the operating user.

  When a plurality of keys included in the hard key unit 167 are pressed, the operation unit 163 receives input of data such as instructions, characters, and numbers corresponding to the pressed keys. The touch panel 165 is provided on the upper surface or the lower surface of the display unit 161, and outputs the coordinates of the position designated by the operation user to the CPU 111. The touch panel 165 detects a position designated by the operation user with a finger or a stylus pen, and outputs the coordinates of the detected position to the CPU 111. The touch panel 165 is a multi-touch compatible touch panel, and outputs a plurality of coordinates corresponding to a plurality of positions simultaneously designated by the user to the CPU 111 when the user inputs a multi-touch operation. Further, the touch panel 165 outputs coordinates corresponding to a single position designated by the user to the CPU 111 when the user inputs a single touch operation.

  The touch panel 165 is preferably the same size or larger than the display surface of the display unit 161. Since the touch panel 165 is provided so as to overlap the display unit 161, the touch panel 165 indicates the coordinates of the position indicated by the operation user on the display surface of the display unit 161 when the operation user indicates the display surface of the display unit 161. It outputs to CPU111. For the touch panel 165, for example, a resistive film method, a surface acoustic wave method, an infrared method, an electromagnetic induction method, or a capacitance method can be used, and the method is not limited.

  The operations that can be accepted by the operation unit 163 include an operation of pressing a plurality of keys of the hard key unit 167, a single touch operation that instructs the touch panel 165, and a multi-touch operation. Further, the hard key section 167 includes four hard keys 167A to 167D each representing characters of “BOX”, “FAX”, “COPY”, and “ECO”. Three hard keys 167A, 167B, and 167C, each of which represents the characters “BOX”, “FAX”, and “COPY”, are process switches to which a process switch operation for instructing the MFP 100 to switch processes is assigned. Key. A hard key 167D having characters “ECO” is a mode switching key to which a mode switching operation for instructing the MFP 100 to switch from the normal mode to the power saving mode having lower power consumption than the normal mode is assigned. is there. The operation unit 163 receives a process switching operation when any of the hard keys 167A, 167B, and 167C is pressed, and receives a mode switching operation when the hard key 167D is pressed.

  FIG. 4 is a diagram illustrating an example of a software architecture of a CPU provided in the MFP. Referring to FIG. 4, an operating system (OS) layer is formed in CPU 111, and an MFP process layer is formed in a layer above it. An application platform (PF) layer and an application layer are formed in this order on the MFP process layer.

  The OS layer includes a task for the CPU 111 to execute the OS program. A task belonging to the OS layer executes processing for controlling hardware resources of the MFP 100. Here, the hardware resources include the communication I / F unit 112, the ROM 113, the RAM 114, the HDD 115, the facsimile unit 116, the external storage device 117, the automatic document feeder 120, the document reading unit 130, the image forming unit 140, and the paper feeding unit 150. The operation panel 160 is included. A task belonging to the OS layer controls hardware resources in accordance with an operating command input from the MFP process layer. Further, the OS layer shares operation identification information for identifying each of a plurality of types of operations that can be accepted by the operation unit 163, which is one of hardware resources, with the MFP process layer. When the operation unit 163 detects an operation by the operating user, the task belonging to the group outputs operation identification information indicating the detected operation to the MFP process layer.

  The application layer includes a task for the CPU 111 to execute an application program. When a plurality of types of application programs are installed in MFP 100, a plurality of tasks that respectively execute the plurality of types of application programs may belong to the application layer.

  The task for executing the application program executes a plurality of types of processing determined by the application program. The multiple types of processing include processing for causing a task belonging to the MFP process layer to execute processing executed in the MFP process layer. The task that executes the application program outputs an application command when the task belonging to the MFP process layer executes processing. The application command is a command published as an API (Application Program Interface). Therefore, an application program that causes MFP 100 to execute processing can be easily created. “Public” means a state in which a third party other than the manufacturer that manufactures the MFP 100 can use it. Therefore, a third party can develop an application program that can be installed in MFP 100 using an application command.

  Further, the task for executing the application program specifies a process to be executed from among a plurality of types of processes defined by the application program based on an operation input by an operation user operating MFP 100, and executes the process. An operation input by an operation user who operates MFP 100 is accepted in the OS layer when the operation user operates operation unit 163 to input.

  The application PF layer is arranged between the application layer and the MFP process layer, and arbitrates a plurality of tasks belonging to the application layer, and a task for controlling an application command output by the plurality of tasks belonging to the application layer belongs. Specifically, the application PF layer accepts application commands output by each of a plurality of tasks belonging to the application layer, standardizes the received application commands by converting them into internal commands according to a predetermined rule, The command is output to the MFP process layer. For this reason, it is possible to cope with a case where versions are different among a plurality of application programs. In addition, the application PF layer determines any one of a plurality of tasks belonging to the application layer as a current state. Then, an operation received from the task belonging to the MFP process layer and received from the OS layer is output to the current task determined as the current state among a plurality of tasks belonging to the application layer.

  The operations input by the user accepted in the OS layer include a single touch operation in which the touch panel 165 is instructed with one finger and a multi-touch operation in which the touch panel 165 is instructed simultaneously with two or more fingers. The single touch operation and the multi-touch operation are detected by the operating system as mouse events, respectively, and input from the application PF layer to the current task among a plurality of tasks belonging to the application layer. One or more mouse events are converted into gesture events in the MFP process layer, and input from the application PF layer to the current task among a plurality of tasks belonging to the application layer. For this reason, the application layer shares one or more gesture events that can be accepted by the application program with the MFP process layer, and the task for executing the application program is executed from the task belonging to the application PF layer by the application program. One or more acceptable gesture events are input. The task that executes the application program executes a process corresponding to either a gesture event or a mouse event among a plurality of types of processes.

  Application commands and internal commands are associated in advance. For example, a command correspondence table in which application commands and internal commands are associated with each other may be stored. An application command and an internal command may correspond one-to-one, or a set of two or more internal commands may correspond to one application command. Furthermore, one internal command or a set of a plurality of internal commands may correspond to a plurality of application commands having different versions. For this reason, it is possible to cope with a case where versions are different among a plurality of application programs. The internal command is a command that depends on the hardware resources of MFP 100. The internal command is not normally disclosed, but may be disclosed.

  The MFP process layer is arranged between the application PF layer and the OS layer, and a task for the CPU 111 to execute the MFP process program belongs to it. The MFP process layer converts an internal command output by a task belonging to the application PF layer into an operating command interpretable by the task belonging to the OS layer, and converts the operating command to a task belonging to the OS layer in order to control hardware resources. Output. In actuality, internal commands may be converted into one or more operating commands that can be executed by tasks belonging to the OS layer, but for the sake of explanation, internal commands and tasks belonging to the OS layer can be executed. The relationship with one or more operating commands will be described as one-to-one.

  As described above, the application program is an application program for causing MFP 100 to execute browsing processing, copy processing, scanning processing, printing processing, facsimile transmission / reception processing, data transmission processing, and the like. Among the processes executed by the CPU 111, the part related to the process for controlling the operation panel 160 will be mainly described.

  FIG. 5 is a block diagram illustrating an example of functions of the CPU included in the MFP, together with information stored in the HDD. The functions shown in FIG. 5 are functions realized by CPU 111 when CPU 111 provided in MFP 100 executes a program stored in ROM 113, HDD 115, or CD-ROM 118. Specifically, the CPU 111 is realized by the CPU 111 by executing an OS program, an MFP process program, an operation reception program, and an application program. Here, the function of CPU 111 when first application program and second application program are installed as application programs in MFP 100 is shown.

  Referring to FIG. 5, CPU 111 includes an application unit 51, an arbitration unit 53, an operation conversion unit 55, and an operating unit 57. The application unit 51 belongs to the application layer in the software architecture shown in FIG. The application unit 51 includes a browsing unit 61 that executes a browsing program that is a first application program, and a sub-execution unit 63 that executes a second application program different from the browsing program.

  The browsing unit 61 includes a display image generation unit 71, a determination unit 73, a selection unit 75, and an instruction unit 77. The display image generation unit 71 downloads a Web page, which is content written in a markup language such as HTML (HyperText Markup Language), from a server connected to the Internet, interprets the Web page, and corresponds to the Web page A display image is generated, and at least a part of the display image is displayed on the display unit 161. The web page may be a web page identified by a URL (Uniform Resource Locator) specified by an operating user who operates MFP 100, or a web page identified by a predetermined URL in browsing unit 61. Also good.

  Specifically, the display image generation unit 71 outputs an application command to the arbitration unit 53 in order to display at least a part of the display image corresponding to the Web page on the display unit 161. As a result, a display image is stored in a VRAM (Video RAM) included in the display unit 161 by the operating unit 57 described later, and at least a part of the display image is displayed on the display unit 161. The part displayed on the display part 161 in a display image can be defined arbitrarily. For example, it may be determined in advance, may be determined by a Web page, or may be determined from the size of the display image and the size of the display unit 161.

  When the Web page acquired from the server includes a description for displaying an embedded image generated by executing a second application program different from the browsing program, the display image generating unit 71 performs sub-execution. An activation instruction is output to the unit 63. Thereby, the sub execution part 63 is formed in the CPU 111. When the Web page includes a description for displaying an embedded image, the display image generation unit 71 does not place an image in the embedded region defined by the Web page in the display image. The embedding area is an area in which the embedding image is arranged in the display image.

  The activation instruction includes position information indicating the position of the embedded area in the display image. In addition, the activation instruction may include content identification information that identifies embedded content different from the Web page. The content identification information is a network address, for example, and is preferably a URL.

  The sub execution unit 63 is activated in response to an activation instruction input from the display image generation unit 71. The sub-execution unit 63 executes a program described in an object-oriented programming language such as JAVA (registered trademark) or JAVASCRIPT (registered trademark). The sub-execution unit 63 causes a server that is an external computer connected via a network to execute a second application program that is different from the browsing program and a second application program that is different from the browsing program. There are cases.

  When the secondary execution unit 63 causes the server connected to the network to execute the second application program, the secondary execution unit 63 may be included in the browsing unit 61 or may not be included in the browsing unit 61. The sub-execution unit 63 is included in the browsing unit 61, for example, when the sub-execution unit 63 executes an application program described in JAVASCRIPT (registered trademark). In this case, the application program described in JAVASCRIPT (registered trademark) is described in the browsing program that is the first application program. The case where the secondary execution unit 63 is not included in the browsing unit 61 is, for example, the case where the secondary execution unit 63 executes an application program described in JAVA (registered trademark). In this case, the application program described in JAVA (registered trademark) is a program different from the browsing program that is the first application program.

  The sub-execution unit 63 acquires an image of content determined by the second application program executed by the server as an embedded image. The sub-execution unit 63 transmits a processing request to the server to cause the server to execute a process predetermined by the second application program. Then, the sub-execution unit 63 receives an image generated when the server executes processing according to the processing request from the server, and acquires the received image as an embedded image. If the content to be displayed is determined by the server, the processing request need not include content identification information. Conversely, the content is specified on the Web page, a processing request including the content identification information included in the activation instruction is transmitted to the server, the content specified by the content identification information is processed by the server, and the content image is displayed. You may make it reply. The server on which the sub-execution unit 63 executes the process may be the same as the server from which the browsing unit 61 has downloaded the web page, or may be a different server.

  Further, when the sub-execution unit 63 executes a second application program different from the browsing program, the second application program is an application program different from the browsing program, for example, a still image or a moving image that is content. A reproduction program to be generated. The sub-execution unit 63 executes the reproduction program that is the second application program, and causes the CPU 111 to execute processing for reproducing the image of the content specified by the content identification information included in the activation instruction.

  The sub-execution unit 63 displays the content image received from the server or the content image generated by executing the second application program on the display unit 161 as an embedded image. Specifically, the sub-execution unit 63 outputs an application command to the arbitration unit 53 in order to place the embedded image in the embedded region of the display image. Further, the sub-execution unit 63 outputs a display request to the browsing unit 61. As a result, the embedded image is stored in the embedded region in the display image stored in the VRAM included in the display unit 161 by the operating unit 57 described later. The application command includes position information included in the execution instruction.

  Note that the application programs executed by the browsing unit 61 and the sub-execution unit 63 are not limited to these. In order to cause the MFP 100 to execute copy processing, scan processing, print processing, facsimile transmission / reception processing, data transmission / reception processing, and the like, It may be a program for customizing a user interface or the like.

  The arbitration unit 53 and the operation conversion unit 55 are functions realized by a task in which the CPU 111 executes the operation reception program. The arbitration unit 53 belongs to the application PF layer in the software architecture shown in FIG. The operation conversion unit 55 belongs to the application PF layer in the software architecture shown in FIG. Although all the functions of the operation conversion unit 55 are functions of the application PF layer, at least some of the functions of the operation conversion unit 55 are functions of the application PF layer, and other parts are functions of the MFP process layer. Also good.

  When there are a plurality of tasks for executing the application program in the application unit 51, the arbitration unit 53 determines any one of the plurality of tasks as the current state. Hereinafter, the task that the arbitration unit 53 has determined to be in the current state is referred to as a current task. In the present embodiment, the application unit 51 includes a browsing unit 61 and a sub-execution unit 63. Since the sub-execution unit 63 is activated by the browsing unit 61, the browsing unit 61 is determined to be in the current task state.

  The arbitration unit 53 receives an application command output from each of the browsing unit 61 and the sub-execution unit 63, and outputs application identification information and an application command for identifying the task that has output the application command to the operation conversion unit 55.

  When a plurality of application commands are output from the browsing unit 61 and the sub-execution unit 63, the arbitration unit 53 determines their order and outputs them in order to the operation conversion unit 55. For example, when a plurality of application commands cannot be executed simultaneously, the other application command is output to the operation conversion unit 55 after the execution of one application command by the operation conversion unit 55 is completed. Also, if the other app command cannot be executed until after one app command has been executed, the other app command can be executed even if the other app command is input prior to the other app command. Is output first.

  The operation conversion unit 55 receives the application command from the arbitration unit 53 and standardizes the application command by converting the application command into an internal command according to the command correspondence table. The command correspondence table associates one or more internal commands with one application command. The application commands included in the command correspondence table may include the same type of application commands with different versions. In this case, each different version of the application command is associated with one or more internal commands. For this reason, it is possible to install an application program in which application commands of different versions are described. When a new application command appears, the command correspondence table is updated with a command correspondence table in which one or more internal commands are associated with the newly appeared application command. Therefore, an application program describing a new application command can be installed.

  Further, the operation conversion unit 55 converts the standardized internal command into an operating command, and outputs the operating command to the operating unit 57. The operating command is a command that is predetermined between the operation conversion unit 55 and the operating unit 57 and that can be interpreted by the operating unit 57.

  The operating unit 57 is a function executed by a task for the CPU 111 to execute the OS program. The operating unit 57 belongs to the OS layer in the software architecture shown in FIG.

  The operating unit 57 receives an operating command output from the operation conversion unit 55 and controls hardware resources according to the operating command. As the hardware resources, here, the display unit 161, the touch panel 165 of the operation unit 163, and the HDD 115 will be described as an example. The operating unit 57 controls the display unit 161 to display an image on the display unit 161. In addition, the operating unit 57 controls the HDD 115 to store data in the HDD 115 and reads data stored in the HDD 115. Further, the operating unit 57 causes the touch panel 165 to detect a position designated by the user at a predetermined time interval, and acquires position information from the touch panel 165 at a predetermined time interval.

  Touch panel 165 detects the position instructed by the operation user when the operation user indicates the display surface of display unit 161, and outputs position information indicating the position of the display surface of display unit 161 to CPU 111. When the position information is input from the touch panel 165, the operating unit 57 outputs the position information to the operation conversion unit 55. The operating unit 57 may receive two or more pieces of position information from the touch panel 165 at the same time. For example, when the operating user designates two different parts of the display surface of the display unit 161 with two fingers at the same time, two pieces of position information indicating the two positions of the designated display surface are received. When the operating unit 57 simultaneously receives two or more pieces of position information from the touch panel 165, the operating unit 57 outputs the two or more pieces of position information to the operation conversion unit 55.

  The operation conversion unit 55 includes an association unit 81, a gesture event determination unit 83, a mouse event generation unit 85, and a display control unit 87. The associating unit 81 associates each of a plurality of types of processing determined by the browsing program with one of a plurality of gesture events at the stage where the browsing program is installed. Specifically, the associating unit 81 generates a master table in which a gesture event that can be accepted by the browsing program is associated with each of a plurality of pieces of processing identification information determined by the browsing program, and stores the master table in the HDD 115.

  FIG. 6 is a diagram illustrating an example of the master table. Referring to FIG. 6, master table 91 includes a master record for each of a plurality of process identification information. The master record includes a process item and a gesture item. Processing identification information is set in the processing item, and gesture identification information for identifying a gesture event is set in the gesture item.

  For example, in the master record in which the process identification information “page feed” is set in the process item, the gesture identification information “swipe” is set in the gesture item. The process specified by the process identification information “page feed” is a process of changing the screen to the screen of the next page and displaying it. The gesture event specified by the gesture identification information “Swipe” corresponds to a user's operation of moving the finger to any one of up, down, left and right at a speed faster than a predetermined speed while instructing the touch panel 165. In the operation of moving the finger at a speed faster than a predetermined speed, the direction in which the finger is moved is determined. Therefore, the gesture event specified by the gesture identification information “swipe” includes the direction in which the finger is moved as a parameter.

  In the master record in which the process identification information “enlarge” is set in the process item, the gesture identification information “pinch out” is set in the gesture item. The process specified by the process identification information “enlargement” is a process for enlarging and displaying the screen. The gesture event specified by the gesture identification information “pinch out” corresponds to an operation of moving at least one of the designated two locations so that the interval between the two locations becomes long while designating the two locations on the touch panel 165. Since the operation of moving so that the interval between the two places becomes longer, the interval between the two places changes, the gesture event specified by the gesture identification information “pinch out” has the ratio of the change between the two places as a magnification. Include in parameter.

  In the master record in which the process identification information “reduction” is set in the process item, the gesture identification information “pinch-in” is set in the gesture item. The process specified by the process identification information “reduction” is a process of reducing the screen for display. The gesture event specified by the gesture identification information “pinch-in” corresponds to an operation of moving at least one of the designated two locations so that the interval between the two locations is shortened while designating the two locations on the touch panel 165. Since the operation of moving so that the interval between the two locations is shortened, the interval between the two locations changes, the gesture event specified by the gesture identification information “Pinch In” is a parameter in which the interval between the two locations is the ratio of the change. Included.

  In the master record in which the process identification information “rotation” is set in the process item, the gesture identification information “rotation” is set in the gesture item. The process specified by the process identification information “rotation” is a process of rotating the screen and changing the screen direction. The gesture event specified by the gesture identification information “Rotation” is an operation to move at least one of the two designated locations while changing the orientation of a straight line connecting the two locations while designating two locations on the touch panel 165. Detected when input. For example, this is an operation of drawing an arc on one of two places as the center. Since there are clockwise and counterclockwise directions in which the arc is drawn, the gesture event specified by the gesture identification information “rotation” is included in a parameter whose direction is either clockwise or counterclockwise.

  In the master record in which the process identification information “scroll” is set in the process item, the gesture identification information “flick” is set in the gesture item. The process specified by the process identification information “scroll” is a process of scrolling the screen up / down / left / right. The gesture event specified by the gesture identification information “flick” corresponds to an operation in which the finger is moved at a speed slower than the threshold value in the operation of the swipe gesture event while instructing the touch panel 165, either vertically or horizontally. . Since the operation of moving the finger at a speed slower than the threshold value determines the direction in which the finger is moved, the gesture event specified by the gesture identification information “flick” includes the direction in which the finger is moved as a parameter.

  Returning to FIG. 5, the display control unit 87 converts the application command input from the browsing unit 61 or the sub-execution unit 63 into an operating command for causing the display unit 161 to display an image, and outputs the operating command to the operating unit 57. . Thereby, the display image generated by the browsing unit 61 and the embedded image generated by the sub-execution unit 63 are stored in the VRAM of the display unit 161, and at least a part of the display image is displayed on the display unit 161.

  The mouse event generation unit 85 generates a mouse event in response to position information input from the operating unit 57. Furthermore, every time a mouse event is generated, the mouse event generation unit 85 outputs the generated mouse event to the browsing unit 61 via the gesture event determination unit 83 and the arbitration unit 53. The mouse event generation unit 85 generates a plurality of mouse events corresponding to each of the plurality of pieces of position information when a plurality of pieces of position information are simultaneously input from the operating unit 57. When generating one or more mouse events, the mouse event generation unit 85 outputs the generated one or more mouse events to the gesture event determination unit 83 and the browsing unit 61. The position information indicates a position on the display surface of the display unit 161 instructed by the operation user.

  In other words, the operation of the operation user instructing the display surface of the display unit 161 is an instruction start operation in which the user specifies an arbitrary position on the display surface of the display unit 161, in other words, while the user indicates the display surface of the display unit 161. , A moving operation for moving the position to be instructed while touching the display surface of the display unit 161, and an ending operation for the user to end the instruction on the display surface of the display unit 161. When the position information is input from the operating unit 57, the mouse event generation unit 85 determines whether the operation by the operating user is an instruction start operation, a movement operation, or an end operation. As described above, the operating unit 57 outputs the position information every time the position information is output from the touch panel 165. Therefore, based on the continuity of the position information that is continuously input as time passes, It is determined whether the operation by the user is an instruction start operation, a move operation, or an end operation.

  The mouse event generation unit 85 determines that the operation by the operating user is an instruction start operation for the position information input after a predetermined time has passed without the position information being input from the operating unit 57, and the position information, A mouse event including state identification information “Press” for identifying the instruction start operation is generated.

  When one or more pieces of position information are continuously input from the operating unit 57 after detecting the mouse event of the state identification information “Press”, the mouse event generation unit 85 operates the operation user for each piece of one or more pieces of position information. The operation according to is determined as a moving operation. For each of one or more pieces of position information continuously input from the operating unit 57, the mouse event generation unit 85 generates a mouse event including the position information and state identification information “Move” for identifying the moving operation. Generate.

  When the mouse event generation unit 85 detects the mouse event of the state identification information “Move” and the position information is no longer input from the operating unit 57, the operation by the operating user is completed for the last input position information. It is determined as an operation, and a mouse event including the position information last input and state identification information “Release” for identifying the end operation is generated.

  More specifically, the mouse event generation unit 85 is based on position information input from the operating unit 57 at the first time and position information input at a second time after a predetermined time from the first time. To determine the state of the mouse event. The mouse event generation unit 85 has a predetermined distance from the position specified by the position information input at the first time before the first time with respect to the position information input at the first time. If position information indicating a position within the range is not input, the state of the mouse event for the position information input at the first time is determined as “Press”. When the position specified by the position information input at the first time does not exist within a predetermined distance from the position specified by the position information input at the second time, the mouse event generation unit 85 The state of the mouse event for the position information input at the second time is determined as “Press”.

  In addition, the mouse event generation unit 85 has a case where the position specified by the position information input at the first time is within a predetermined distance from the position specified by the position information input at the second time. The mouse event state for the position information input at the second time is determined as “Move”. Further, the state detection unit 76, when position information within a predetermined distance from the position specified by the position information input at the first time is not input at the second time, The state of the mouse event for the position information input at 1 is determined as “Release”.

  The gesture event determination unit 83 determines a gesture event based on a plurality of mouse events input continuously from the mouse event generation unit 85. Gesture event determination unit 83 has a plurality of mice that are continuously input to the first mouse event when two of the first mouse event and the second mouse event are input simultaneously from mouse event generation unit 85. A gesture event is determined based on a first set of events and a second set of multiple mouse events that are input sequentially for the second mouse event. The gesture event determining unit 83 outputs the determined gesture event to the process specifying unit 79. The plurality of mouse events included in the first group have the same position information or continuous position information. Continuous means that two pieces of position information indicate adjacent positions. The plurality of mouse events included in the second group have the same position information or continuous position information. For this reason, the gesture event determination unit 83 determines whether a plurality of mouse events input simultaneously belong to the first group or the second group based on the position information included in them.

  When the mouse event is continuously input from the mouse event generation unit 85, the gesture event determination unit 83 firstly inputs the state identification information of the mouse event as “Press”. When the mouse event whose state identification information is “Move” is input, the gesture event determination unit 83 receives the position information of the mouse event whose state identification information is “Press” and the mouse event whose state identification information is “Move”. The distance between two points is calculated from the position information, and if the distance is greater than or equal to a predetermined threshold TH, the user's swipe operation is detected, and if the distance is shorter than the predetermined threshold TH, the user's flick operation is detected. To do. When a plurality of mouse events are input, the user's swipe operation may be detected by the first plurality of mouse events, and the user's flick operation may be detected by the subsequent plurality of mouse events. When detecting the swipe operation and the flick operation, the gesture event determining unit 83 calculates a direction from the position specified by the first position information to the position specified by the subsequent position information, and displays the display surface of the display unit 161. The direction closest to the calculated direction is determined as a parameter among the four directions of up, down, left, and right as a reference, and gesture identification information including the parameter is determined. When a swipe operation is detected, the gesture event of the gesture identification information “Swipe” includes a parameter indicating the direction, and the gesture event of the gesture identification information “Flick” includes a parameter indicating the direction.

  When two mouse events are input simultaneously from the mouse event generation unit 85, the state of each of the two mouse events input first is “Press”. In this case, the distance L1 between the two points and the angle R1 of the straight line connecting the two points are calculated based on the position information included in each of the two mouse events. The angle of the straight line connecting the two points may be an angle formed by the straight line connecting the two points with the reference line with the horizontal direction of the display unit 161 as the reference line. The angle formed between the straight line connecting the two points and the reference line is an angle formed clockwise from the reference line. Then, the gesture event determination unit 83 calculates the distance L2 between the two points and the angle R2 of the straight line connecting the two points based on the positional information of the two mouse events that are successively input thereafter.

  The gesture event determination unit 83 detects a pinch-out operation or a pinch-in operation by the user if the angle R1 and the angle R2 are the same and the distance L2 is different from the distance L1. The angle R1 and the angle R2 being the same include the case where the difference between the angle R1 and the angle R2 is within a predetermined range. In order to detect the movement of the user's finger, an error is included. If the distance L2 is larger than the distance L1, a pinch-out operation is detected. If the distance L2 is smaller than the distance L1, a pinch-in operation is detected. When detecting a pinch-out operation or a pinch-in operation, the gesture event determination unit 83 calculates a magnification by dividing the distance L2 by the distance L1, determines the calculated magnification as a parameter, and determines gesture identification information including the parameter. decide. When a pinch-out operation is detected, the gesture event of the gesture identification information “Pinch-out” includes a parameter indicating a magnification. When a pinch-in operation is detected, the gesture event of the gesture identification information “Pinch-in” is a magnification. The parameter that indicates is included.

  The gesture event determination unit 83 detects a rotation operation by the user when the angles R1 and R2 are different. The difference between the angle R1 and the angle R2 is when the difference between the angle R1 and the angle R2 is outside a predetermined range. In order to detect the movement of the user's finger, an error is included. When detecting a rotation operation, the gesture event determination unit 83 determines a rotation direction by subtracting the angle R1 from the angle R2, determines the determined rotation direction as a parameter, and determines gesture identification information including the parameter. . If the value obtained by subtracting the angle R1 from the angle R2 is positive, the rotational direction is clockwise. If the value obtained by subtracting the angle R1 from the angle R2 is negative, the rotational direction is counterclockwise. The gesture identification information “Rotation” includes a parameter indicating the rotation direction.

  When determining the gesture event, the gesture event determination unit 83 reads the master table 91 corresponding to the application identification information of the current task set by the arbitration unit 53 from the HDD 115. Specifically, an operating command for reading the master table 91 is output to the operating unit 57, and the operating unit 57 controls the HDD 115 to acquire the master table 91 to be read. Hereinafter, a case where the browsing unit 61 is the current task will be described.

  The gesture event determination unit 83 determines whether or not the gesture event is determined, and based on the gesture event, the application set in the current state by the arbitration unit 53 from among a plurality of types of processing determined in the master table 91. Identify the process associated with the program. Here, since the browsing unit 61 is set as the current task by the arbitrating unit 53, the gesture event determining unit 83 identifies the gesture event and the process associated with the master table 91. More specifically, the gesture event determination unit 83 extracts the master record in which the gesture identification information of the determined gesture event is set as the item of gesture from the master table 91, and the process item of the extracted master record Acquires the process identification information set in. When the process identification information is acquired, the gesture event determination unit 83 outputs the determined gesture event to the arbitration unit 53. However, when the process identification information is not acquired, the gesture event determination unit 83 does not output the determined gesture event to the arbitration unit 53. .

  The gesture event determination unit 83 outputs only the gesture event corresponding to the process executable by the browsing unit 61 set in the current state by the arbitration unit 53 to the arbitration unit 53, and the arbitration unit 53 outputs the gesture event to the current task. Is output to the browsing unit 61. For this reason, it is possible to cause the browsing unit 61 that executes a browsing program to execute a gesture event that is an operation input to the touch panel 165 and is specified by a set of a plurality of mouse events. In other words, at the stage of developing a browsing program, the MFP 100 can be developed without being aware of gesture events that can be accepted by the touch panel 165 in the MFP 100. For example, a browsing program that does not accept rotation operations can be developed.

  The arbitration unit 53 receives a mouse event from the mouse event generation unit 85, receives a gesture event from the gesture event determination unit 83, and outputs them to the current task among a plurality of tasks for which the application unit 51 executes the application program. . Here, the application unit 51 includes a browsing unit 61 and a sub-execution unit 63. Since the sub-execution unit 63 is activated by the browsing unit 61, the arbitration unit 53 sets the browsing unit 61 to the current state. For this reason, the browsing unit 61 is the current task. The arbitration unit 53 outputs the mouse event to the browsing unit 61 in response to the mouse event input from the mouse event generation unit 85. The arbitration unit 53 outputs the gesture event to the browsing unit 61 in response to the gesture event input from the gesture event determination unit 83.

  The browsing unit 61 receives a mouse event output from the mouse event generation unit 85 from the arbitration unit 53, and receives a gesture event output from the gesture event determination unit 83 from the arbitration unit 53. Since the gesture event is determined from a group of one or more mouse events, the gesture event and the group of one or more mouse events that determine the gesture event are a single touch operation input by the operating user or a single operation. This occurs simultaneously for multiple touch operations.

  Browsing unit 61 includes a plurality of browsing programs defined by a browsing program in accordance with a single touch operation or a multi-touch operation that is input to an area other than the embedded region among single-touch operations or multi-touch operations input to touch panel 165 by an operation user operating MFP 100. The process to be executed is identified from the types of processes, and the process is executed.

  The plurality of types of processing executed by the browsing unit 61 includes processing for changing an image displayed on the display unit 161. The process of changing the screen is, for example, a process of changing an image to be displayed to an image of another content, or a part of the image that is displayed on the display unit 161 when the image is displayed on the display unit 161. Scroll processing for changing the image, page turning processing, enlargement processing for enlarging and displaying the image, reduction processing for reducing and displaying the image, and rotation processing for changing the direction of the image.

  Sub-execution unit 63 executes processing in accordance with a single touch operation or a multi-touch operation that is input into the embedding area among a single touch operation or a multi-touch operation that is input to touch panel 165 by the user who operates MFP 100. . Specifically, when the sub-execution unit 63 executes a second application program different from the browsing program, a process to be executed from among a plurality of types of processes defined by the second application program different from the browsing program And specify the specified process. In this case, the plurality of types of processing executed by the sub-execution unit 63 includes processing for changing an image displayed on the display unit 161. The process of changing the screen is, for example, a process of changing an image to be displayed to an image of another content, or a part of the image that is displayed on the display unit 161 when the image is displayed on the display unit 161. Scroll processing for changing the image, page turning processing, enlargement processing for enlarging and displaying the image, reduction processing for reducing and displaying the image, and rotation processing for changing the direction of the image.

  In addition, when the sub-execution unit 63 causes a server, which is an external computer connected via a network, to execute a second application program different from the browsing program, a mouse event corresponding to a single touch operation or a multi-touch operation A processing request containing is sent to the server.

  The selection unit 75 included in the browsing unit 61 selects either a mouse event or a gesture event. When a gesture event is input to the selection unit 75, one or more mouse events for determining a gesture event are input before that. When a mouse event is input from the arbitration unit 53, the selection unit 75 determines whether or not the position indicated by the position information included in the mouse event is within an embedded region in the display image generated by the display image generation unit 71. Determine whether. The selection unit 75 selects a mouse event if the position indicated by the position information included in the mouse event is within the embedded area, but selects the mouse event if the position indicated by the position information included in the mouse event is outside the embedded area. Do not select. The selection unit 75 selects a gesture event if the position indicated by the position information included in the mouse event is outside the embedded area, but if the position indicated by the position information included in the mouse event is within the embedded area. Do not select a gesture event. The selection unit 75 outputs the mouse event to the instruction unit 77 when selecting a mouse event, and outputs the gesture event to the determination unit 73 when selecting a gesture event.

  When a mouse event is input from the selection unit 75, the instruction unit 77 outputs the mouse event to the sub-execution unit 63. When the sub-execution unit 63 causes the server, which is an external computer connected via the network, to execute a second application program different from the browsing program, in response to a mouse event being input from the instruction unit 77 Then, a processing request including a mouse event is transmitted to the server, and the embedded image is updated with the image received from the server. Specifically, an application command for overwriting an image received from the server in the VRAM included in the display unit 161 is output to the display control unit 87. Thereby, the embedding area of the display image stored in the VRAM included in the display unit 161 is overwritten with the image received from the server.

  In addition, when the secondary execution unit 63 executes a second application program different from the browsing program, the secondary execution unit 63 selects one of a plurality of types of processing determined by the second application program based on a mouse event. Identify the process to be executed and execute the process. For example, if the position specified by the position information included in the mouse event is within the button image included in the embedded image, the process associated with the button image is specified. For example, if the image is an enlargement button image, processing for enlarging the image is executed, and the embedded image is updated with the image after enlargement. Specifically, an application command for overwriting the updated image in the VRAM included in the display unit 161 is output to the display control unit 87. As a result, the display image embedding area stored in the VRAM included in the display unit 161 is overwritten with the updated image.

  When a gesture event is input from the selection unit 75, the determination unit 73 executes a type of process corresponding to the gesture event among a plurality of types of processes determined by the browsing program. Specifically, a display portion to be displayed on the display unit 161 in the display image is determined. The determination unit 73 outputs an application command for causing the display unit 161 to display the determined display portion to the display control unit 87. Thereby, the image of the part determined by the determination part 73 among the display images memorize | stored in VRAM with which the display part 161 is provided is displayed on the display part 161. FIG. For example, if the gesture event input from the selection unit 75 corresponds to the process identification information “scroll”, the part displayed on the display unit 161 in the display image is slid from the part to the top, bottom, left, or right. The part is displayed on the display unit 161.

  Next, in MFP 100 according to the present embodiment, for example, a case where a Web page is downloaded from server 300 connected to the Internet and a display image, which is an image of the Web page, is displayed, a user operation and processing executed by MFP 100 An example of the flow will be described.

  FIG. 7 is a diagram illustrating an example of a user operation and a flow of processing executed by the MFP. Referring to FIG. 7, in order from the left, the user operation, the processing of the operating unit 57, the processing of the sub-execution unit 63, the processing of the browsing unit 61, and the processing of the server 300 are shown. Shows the flow of time from top to bottom.

  First, when the user inputs an operation for starting a browsing program, the operation unit 57 accepts the operation. Here, a case where the URLs of Web pages stored in the server 300 are specified at the same time will be described. In accordance with a user operation, the operating unit 57 causes the CPU 111 to execute the browsing program, thereby starting the browsing unit 61 that is a task for the CPU 111 to execute the browsing program. In addition, the operating unit 57 outputs the URL input by the user to the browsing unit 61.

  The browsing unit 61 transmits to the server 300 a content request that is a request for transmitting the Web page specified by the URL input from the operating unit 57 to the server 300 specified by the URL. When the server 300 receives the content request, it returns a Web page specified by the URL, so that the browsing unit 61 receives the Web page from the server 300. Further, the browsing unit 61 analyzes the Web page, generates a display image corresponding to the Web page, and outputs a command to store the display image in the VRAM and a command to display the display part to the operating unit 57. Accordingly, the operating unit 57 stores the display image in the VRAM of the display unit 161 and displays the display part on the display unit 161.

  Here, it is assumed that the Web page includes a description of a command for starting a JAVA (registered trademark) program. For this reason, when the browsing program 61 causes the CPU 111 to execute the JAVA (registered trademark) program, the sub-execution unit 63, which is a task for the CPU 111 to execute the JAVA (registered trademark) program, is activated.

  Here, it is assumed that processing for causing the server 300 to transmit an image of content stored in the JAVA (registered trademark) program is determined. The sub-execution unit 63 transmits a processing request, which is a request for executing processing for transmitting a content image, to the server. Accordingly, the server 300 returns a content image, so the sub-execution unit 63 receives the content image received from the server 300 as an embedded image. Further, the sub execution unit 63 stores the embedded image in the VRAM of the display unit 161 and outputs a display request to the browsing unit 61. Accordingly, the operating unit 57 stores the display image in the VRAM of the display unit 161. In addition, the browsing unit 61 outputs an instruction to display the display portion to the operating unit 57, and the operating unit 57 displays the display portion of the display image in which the embedded image is arranged in the embedded area on the display unit 161. . Therefore, if the display portion includes the embedded region, the embedded image is displayed on the display unit 161.

  Next, when the user inputs a single touch operation or a multi-touch operation, the operation unit 57 accepts the operation. The operation conversion unit 55 generates a mouse event corresponding to a single touch operation or a multi-touch operation, and determines a gesture event based on one or more mouse events. The mouse event and the gesture event are input to the browsing unit 61.

  The browsing unit 61 makes processing different depending on whether the position indicated by the position information included in the mouse event is within the embedded area or outside the embedded area. The browsing unit 61 outputs the mouse event to the sub-execution unit 63 when the position indicated by the position information included in the mouse event is within the embedded region. The sub-execution unit 63 transmits a processing request including a mouse event to the server 300. When the server 300 receives the processing request, the server 300 executes processing determined by the second application program executed by the server 300 in correspondence with the position in the embedded image specified by the mouse event included in the processing request. Returns the content image after processing. For example, when processing for enlarging an image is defined by the second application program, an image obtained by enlarging the content image is returned. The sub-execution unit 63 receives an image of content received from the server 300 as an embedded image. Further, the sub execution unit 63 stores the embedded image in the VRAM of the display unit 161 and outputs a display request to the browsing unit 61. Accordingly, the operating unit 57 stores the display image in the VRAM of the display unit 161. In addition, the browsing unit 61 outputs an instruction to display the display portion to the operating unit 57, and the operating unit 57 displays the display portion of the display image in which the embedded image is arranged in the embedded area on the display unit 161. . Therefore, if the display portion includes the embedded region, the embedded image after being enlarged by the server 300 is displayed on the display unit 161.

  On the other hand, if the position indicated by the position information included in the mouse event is outside the embedded area, processing based on the gesture event is executed. In this case, no mouse event is output to the sub-execution unit 63. Processing based on the gesture event is defined by a browsing program. For example, if the process based on the gesture event is a process of scrolling downward, an instruction to display the display part after moving the display part of the display image downward is output to the operating unit 57. The operating unit 57 displays a display part of the display image on the display unit 161. Therefore, the display part after the display image is moved is displayed on the display unit 161.

  FIG. 8 is a flowchart illustrating an example of the flow of gesture event determination processing. The gesture event determination process is a process executed by CPU 111 when CPU 111 of MFP 100 executes an operation reception program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIG. 8, CPU 111 determines whether a mouse event has been detected (step S01). If the user indicates touch panel 165, a mouse event is detected based on the position detected by touch panel 165. The process waits until a mouse event is detected (NO in step S01). If a mouse event is detected (YES in step S01), the process proceeds to step S02.

  In step S02, it is determined whether or not the detected mouse event is single. If a single mouse event is detected, the process proceeds to step S03. If a plurality of mouse events are detected, the process proceeds to step S13.

  In step S03, the process branches depending on the state of the mouse event. If the mouse event state is “Press”, the process returns to step S01. If “Mouse”, the process proceeds to step S04. If “Release”, the process proceeds to step S09. When the process proceeds to step S04, the state of the mouse event is “Move”, and another mouse event is input immediately before the mouse event. The state of another mouse event may be “Press” or “Move”. In step S04, an instruction direction is determined. A method is determined for going from the position specified by the position information included in another mouse event input immediately before to the position specified by the position information included in the mouse event detected in step S01. The direction determined here is the closest direction among the four directions, top, bottom, left, and right, based on the display surface of the display unit 161.

  In the next step S05, the designated distance L is calculated. The distance between the position specified by the position information included in another mouse event input immediately before and the position specified by the position information included in the mouse event detected in step S01 is calculated as the instruction distance L. . Then, it is determined whether or not the designated distance L is greater than or equal to the threshold value TH (step S06). If instructed distance L is equal to or greater than threshold value TH, the process proceeds to step S07; otherwise, the process proceeds to step S08.

  In step S07, the gesture event is determined to be swipe, and the process returns to step S01. In step S08, the gesture event is determined to be flicked, and the process returns to step S01.

  When the process proceeds to step S09, the state of the mouse event detected in step S01 is “Release”. When the state of the mouse event is “Release”, another mouse event is input immediately before the mouse event. The state of another mouse event may be “Press” or “Move”. In step S09, the process branches depending on the state of another mouse event input immediately before. If the state of another mouse event input immediately before is “Press”, the process proceeds to step S10; otherwise, the process returns to step S01.

  In step S10, it is determined whether or not the same position information as the position information included in the mouse event detected in step S01 is stored in the RAM 114. If stored in RAM 114, the process proceeds to step S11; otherwise, the process proceeds to step S12. In addition, the same position information here includes the case where the distance between the two positions respectively specified by the two position information is within a predetermined length. In order to detect an operation in which the user points to the same position, an error is included.

  In step S12, the position information included in the mouse event detected in step S01 is temporarily stored in RAM 114, and the process returns to step S01. When the process proceeds to step S12, the state of the mouse event input immediately before is “Press”, the state of the next mouse event input is “Release”, and the same position information is stored in the RAM 114. This is the case when it is not stored. In other words, the user inputs the first tap operation.

  In step S11, the gesture event is determined to be a double tap, and the process returns to step S01. When the process proceeds to step S11, the state of the mouse event input immediately before is “Press”, the state of the mouse event input in step S01 is “Release”, and the same position information is stored in the RAM 114. Is stored. In other words, after the user inputs the first tap operation, the user taps the same position as the first tap operation.

  The process proceeds to step S13 when two mouse events are detected. In step S13, it is determined whether or not the state of at least one of the two mouse events is “Move”. If the state of at least one of the two mouse events is “Move”, the process proceeds to step S14; otherwise, the process returns to step S01.

  In step S14, an angle is calculated. Based on a predetermined reference side of the display surface of the display unit 161, an angle between a straight line connecting two positions specified by position information included in each of the two mouse events and the reference side is calculated. In the next step S15, it is determined whether or not there is a change in the angle calculated in step S14. If the difference between the angle calculated in step S14 and the angle calculated based on the other two mouse events input immediately before is greater than or equal to a predetermined angle, it is determined that the angle has changed. If the angle has changed, the process proceeds to step S16; otherwise, the process proceeds to step S17. In step S16, the gesture event is determined as rotation, and the process returns to step S01.

  In step S17, the distance is calculated. A distance between two positions specified by position information included in each of two mouse events is calculated. In the next step S18, it is determined whether or not there is a change in the distance calculated in step S14. If the difference between the distance calculated in step S17 and the distance calculated based on the other two mouse events input immediately before is greater than or equal to a predetermined length, it is determined that the distance has changed. If the distance has changed, the process proceeds to step S19; otherwise, the process returns to step S01.

  In step S19, the magnification is determined. A value obtained by dividing the distance calculated in step S18 by the distance calculated based on the other two mouse events input immediately before is determined as the magnification. If the magnification is greater than 1, the process proceeds to step S21. If not, the process proceeds to step S22.

  In step S21, the gesture event is determined to be pinch out, and the process returns to step S01. In step S22, the gesture event is determined to be pinch-in, and the process returns to step S01.

  FIG. 9 is a flowchart illustrating an example of the flow of browsing processing. The browsing process is a process executed by CPU 111 when CPU 111 of MFP 100 executes a browsing program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIG. 9, CPU 111 acquires content (step S31). Specifically, a Web page is downloaded as content from a server connected to the Internet. The web page is a web page identified by a URL specified by an operating user who operates MFP 100. Further, it may be a web page identified by a predetermined URL.

  In step S32, it is determined whether or not the acquired Web page includes a description of an embedded image (step S32). The description of the embedded image is a description for displaying an embedded image generated by executing a second application program different from the browsing program. If the Web page acquired from the server includes the description of the embedded image, the process proceeds to step S33; otherwise, the process proceeds to step S35. In step S35, a display image is generated according to the description of the Web page, and the process proceeds to step S36. Specifically, the display image is stored in a VRAM included in the display unit 161.

  In step S33, a display image including an embedded area is generated. Specifically, the display image including the embedded region is stored in the VRAM included in the display unit 161. As a result, the display image generated by the task that executes the browsing program is stored in the VRAM included in the display unit 161. However, the embedded area of the display image is not stored depending on the task that executes the browsing program.

  In step S34, the application is activated and the process proceeds to step S36. According to the description of the Web page, the CPU 111 or the server executes a second application program that is different from the browsing program, and an embedded image is generated by a task that executes the second application program. Specifically, when the second application program is executed by the server using a JAVA (registered trademark) script described on a Web page, the server sends a processing request for generating content to the server, and the content is returned from the server. Are stored in the embedded area of the display image stored in the VRAM included in the display unit 161. As a result, an image generated by the server executing the second application program is stored as an embedded image in the embedded region of the display image stored in the VRAM included in the display unit 161. When the Web page includes an instruction to start a second application program such as a reproduction program that can be executed by the CPU 111, the CPU 111 executes the second application program, and a task for the CPU 111 to execute the second application program is generated. The image is stored in the embedded area of the display image stored in the VRAM included in the display unit 161. As a result, the embedded image generated by the task in which the CPU 111 executes the second application program is stored in the embedded region of the display image stored in the VRAM included in the display unit 161.

  As described above, the task that executes the browsing program rewrites the display image, and either the CPU 111 or the server rewrites the embedded area with the image generated by the task that executes the second application program. In other words, the embedded area can be updated regardless of the task for executing the browsing program with the image generated by the task for executing either the second application program or the server or the CPU 111.

  In step S36, a display image is displayed. The display image stored in the VRAM of the display unit 161 is displayed on the display unit 161. At this time, the display image may be partially displayed or the entire display image may be displayed.

  In step S37, it is determined whether a mouse event has been input. When the operating user indicates the display surface of the display unit 161 with a finger, the position indicated by the touch panel 165 is detected, and a mouse event including the position information is input. The process waits until a mouse event is input (NO in step S37). If a mouse event is input (YES in step S37), the process proceeds to step S38.

  In step S38, it is determined whether or not the mouse event is an operation for the embedded area. If the position specified by the position information included in the mouse event is within the embedded area in the display image, it is determined that the mouse event is an operation for the embedded area. If the mouse event is an operation on the embedded region, the process proceeds to step S39; otherwise, the process proceeds to step S40.

  In step S39, a mouse event is output to the task that executes the second application program started in step S34, and the process proceeds to step S43. Specifically, when the second application program is executed by the server by a JAVA (registered trademark) script described in a Web page, a processing request including a mouse event is transmitted to the server, and an image of content returned from the server And the received image is stored in the embedding area of the display image stored in the VRAM included in the display unit 161. When the Web page includes an instruction to start a second application program such as a playback program that can be executed by the CPU 111, a task for executing the second application program is formed in the CPU 111. For this reason, the CPU 111 outputs a mouse event to a task that executes the second application program, causes the CPU 111 to execute a process based on the mouse event, and displays an image of content generated by the task that the CPU 111 executes the second application program. 161 is stored in the embedded area of the display image stored in the VRAM included in 161.

  The task for the server or CPU 111 to execute the second application program executes a process specified by the mouse event from among a plurality of processes determined by the second application program when a mouse event is input. Specifically, among the plurality of processes determined by the second application program, the process assigned to the position specified by the position information included in the mouse event is executed. For example, if the embedded image includes a button to which the enlargement process is assigned and the position specified by the position information included in the mouse event is within the button to which the enlargement process is assigned, the enlargement process is executed. Then, the embedded area in the display image stored in the VRAM included in the display unit 161 is updated with the embedded image after the enlargement process. For this reason, only the image of the embedding area included in the display image displayed on the display unit 161 is enlarged and displayed.

  On the other hand, in step S40, it is determined whether or not a gesture event has been determined. The gesture event determination process shown in FIG. 8 is executed, and it is determined whether or not a gesture event has been determined based on the mouse event input in step S37. If a gesture event is determined, the process proceeds to step S41. If not, the process proceeds to step S42. In step S42, the mouse event input in step S37 is suspended, and the process proceeds to step S43. Note that the mouse event input in step S37 may be discarded.

  In step S41, a process corresponding to the gesture event is executed, and the process proceeds to step S43. Of the plurality of processes defined by the browsing program, the process specified by the gesture event is executed. Specifically, if the gesture event is a flick, the portion of the display image that is displayed on the display unit 161 is moved in the direction determined by the gesture event. More specifically, an area displayed on the display unit 161 in the display image stored in the VRAM included in the display unit 161 is moved in a direction determined by the gesture event, and the moved region is displayed on the display unit 161. . For this reason, the portion of the display image displayed on the display unit 161 is changed.

  In step S43, it is determined whether an instruction to end the display of content has been received. For example, if a key to which an instruction to end the browsing program is assigned among a plurality of keys included in the hard key unit 167, an instruction to end the display of content is accepted. If an instruction to end the display of content is accepted, the process ends. If not, the process returns to step S37.

  FIG. 10 is a diagram illustrating an example of a display image. The display image shown in FIG. 10 corresponds to a Web page downloaded from a server connected to the Internet. Here, the Web page will be described by taking as an example a case where the contents for introducing a store that sells products are described. Referring to FIG. 10, display image 300 includes two embedded regions 310 and 311 in which embedded images generated by a second application program different from the browsing program are arranged on a Web page. The embedded images arranged in the embedded areas 310 and 311 are generated when a server connected to the Internet executes a second application program installed on the server.

  In the embedded area 310, an image of a map showing the location of the store is arranged as an embedded image, and in the embedded area 311, an image displaying a list of products sold by the store is arranged as an embedded image. The Regions 301 and 303 indicated by dotted lines indicate display portions displayed on the display unit 161 in the display image 300. The dotted lines indicating the areas 301 and 303 are given for the sake of explanation, and do not actually exist in the display image 300.

  FIG. 11 is a first diagram illustrating an example of a display portion in the display image 300. Referring to FIG. 11, display portion 331 is a portion indicated by region 301 in display image 300 shown in FIG. 10. Display portion 331 includes an embedded region 310 arranged in display image 300. The embedding area includes a button group 313 in which four arrows (up, down, left and right) are represented, and a button group 315 in which symbols “+” and “−” are represented. In the button groups 313 and 315, processing is assigned to each button by the second application program installed in the server. Here, if the user points with a finger the button in which the “+” symbol of the button group 315 is represented, a mouse event including position information indicating the position within the button in which the “+” symbol is represented is generated. . Then, the mouse event is transmitted to a server that generates an image arranged in the embedding area 310, and processing for enlarging the image is executed by the server. Then, the embedded area 310 is updated with the image after the enlargement process is executed by the server.

  FIG. 12 is a second diagram illustrating an example of a display portion in the display image 300. Referring to FIG. 12, display portion 331 is a portion indicated by region 301 in display image 300 shown in FIG. 10. Display portion 331 includes an embedded region 310 arranged in display image 300. The embedded image displayed in the embedded region 310 is an enlarged image as compared with the image arranged in the embedded region 310 shown in FIG. However, the part other than the embedded region 310 in the display part 331 of the display image 300 shown in FIG. 12 is the same as the display part 331 shown in FIG.

  Furthermore, if the operating user touches a portion other than the embedded region 310 in the display portion 331 of the display image 300 shown in FIG. 12 with a finger and inputs a flick operation, a gesture event occurs. In this case, a mouse event also occurs, but the mouse event is discarded because the position touched by the operating user is a portion other than the embedded region 310 in the display portion 331 of the display image 300 shown in FIG. Corresponding to the flick gesture event, a scrolling process for changing the display portion of the display image is defined by the browsing program. Here, a case where the scroll process is executed and the region 301 in the display image 300 is changed to the region 303 will be described as an example.

  FIG. 13 is a third diagram illustrating an example of a display portion of 300 in the display image. Referring to FIG. 13, the display portion 341 of the display image 300 displayed on the display portion 161 after the flick operation is input at the stage where the display portion 331 shown in FIG. 12 is displayed on the display portion 161 is shown. Yes. The display portion 341 is a portion indicated by a region 303 in the display image 300 shown in FIG. Display portion 341 includes an embedded region 311 arranged in display image 300. The embedded image displayed in the embedded region 311 is the same as the image arranged in the embedded region 311 shown in FIG.

  As described above, MFP 100 according to the present embodiment functions as a data processing device, and positions indicating one or more positions in response to detection of one or more positions instructed by a user on touch panel 169. A mouse event including information is generated, and any one of a plurality of gesture events is determined based on a set of one or more mouse events generated in succession. On the other hand, when the Web page received from the outside includes a description for displaying an embedded image generated by executing an application program different from the browsing program, the embedded image is stored in the embedded area defined by the Web page. The arranged display image is generated. If the display image includes an embedded area, either a mouse event or a gesture event is selected based on the position indicated by the mouse event. For this reason, one operation that the user designates the display surface of the display unit 161 is detected as one or more mouse events and gesture events, and the browsing program and the application program are respectively detected from the detected mouse events and gesture events. A suitable one can be accepted. As a result, even when the Web page includes a description for executing an application program different from the browsing program, the application program can be appropriately executed.

  In addition, the user detects one operation for designating the display surface of the display unit 161 as one or more mouse events and gesture events, and the detected gesture events are made to correspond to the browsing program, and one or more mouse events are applied to the application. Correspond to the program. For this reason, when the browsing program corresponds to a gesture event and the application program corresponds to a mouse event, the application program can be appropriately executed.

  When a mouse event is selected, the mouse event is output to a task that executes the application program, and the embedded image displayed in the embedded area is updated. For this reason, it is possible to cause a task that executes an application program to execute processing according to a mouse event.

  Further, when selecting a gesture event, a process corresponding to the selected gesture event is executed among a plurality of processes that can be executed by the browsing program. For this reason, the process which determines the display part for displaying on the display part 161 among display images can be performed. A task that executes a browsing program can execute processing according to a gesture event.

  Furthermore, since the master data 91 associated with a plurality of gesture events corresponding to a plurality of types of processing defined by the browsing program is stored in the HDD 115 at the stage where the browsing program is installed, it corresponds to each of a plurality of types of processing defined by the browsing program. Only gesture events can be detected.

  In the above-described embodiment, MFP 100 has been described as an example of a data processing apparatus. However, an operation reception method for causing CPU 111 that controls MFP 100 to execute the processes illustrated in FIGS. It goes without saying that the invention can be understood as a browsing program for causing the CPU 111 to execute the method.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

100 MFP, 51 application section, 53 arbitration section, 55 operation conversion section, 57 operating section, 61 browsing section, 63 secondary execution section, 71 display image generation section, 73 determination section, 75 selection section, 76 state detection section, 77 instruction Unit, 79 process identification unit, 81 association unit, 83 gesture event determination unit, 85 mouse event generation unit, 87 display control unit, 91 master table, 110 main circuit, 111 CPU, 112 communication I / F unit, 113 ROM, 114 RAM, 115 HDD, 116 facsimile unit, 117 external storage device, 120 automatic document feeder, 130 document reading unit, 140 image forming unit, 150 sheet feeding unit, 160 operation panel, 161 display unit, 163 operation unit, 165 touch panel 167 Hard key part.

Claims (11)

Display means capable of displaying images;
A touch panel capable of detecting one or more positions designated by a user in the display surface of the display means;
Control means,
The control means executes a browsing program, acquires content described in a markup language from the outside, displays at least a part of a display image corresponding to the acquired content on the display means, and the browsing In response to the input of any one of a plurality of gesture events respectively corresponding to a plurality of types of processing determined by the program, the type of processing corresponding to the input gesture event is executed among the plurality of types of processing. Browsing means to
Mouse event generation means for generating a mouse event including position information indicating each of the one or more positions in response to detection of one or more positions by the touch panel;
Gesture event determining means for determining any one of a plurality of gesture events based on a set of one or more mouse events generated successively by the mouse event generating means,
When the content includes a description for displaying an embedded image generated by executing an application program different from the browsing program, the browsing means displays the embedded image generated by executing the application program. Display image generating means for generating a display image arranged in an embedding area defined by content;
When the display image includes the embedded region, if the position indicated by the mouse event input from the mouse event generation unit is within the embedded region, the input mouse event is selected, and the mouse event generation unit And a selecting means for selecting a gesture event inputted from the gesture event determining means if the position indicated by the mouse event inputted from is outside the embedding area.   The browsing means further includes an instruction means for outputting the mouse event to a task for executing the application program and updating an embedded image displayed in the embedded area when the mouse event is selected. The data processing apparatus according to claim 1.   When the gesture event is selected, the browsing means displays the display image on the display means by executing a process corresponding to the selected gesture event among the plurality of kinds of processes. The data processing apparatus according to claim 1, further comprising a determination unit that determines a display portion for the display. The control means further includes association means for associating a plurality of gesture events corresponding to the plurality of types of processing defined by the browsing program corresponding to the browsing program when the browsing program is installed,
The gesture event determination unit is configured to process a plurality of types of processing in which gesture events corresponding to a set of one or more mouse events continuously generated by the mouse event generation unit correspond to the browsing program by the association unit. The data processing apparatus according to claim 1, wherein the gesture event is determined on the condition that the gesture event is associated with any one of the above. Display means capable of displaying images;
An operation receiving method executed by a data processing device comprising: a touch panel capable of detecting one or more positions indicated by a user on a display surface of the display means,
A browsing step of executing a browsing program, acquiring content described in a markup language from the outside, and displaying at least a part of a display image corresponding to the acquired content on the display means;
A mouse event generating step for generating a mouse event including position information indicating each of the one or more positions in response to detection of the one or more positions by the touch panel;
A gesture event determining step of determining any one of a plurality of gesture events based on a set of one or more mouse events generated in succession in the mouse event generating step,
In the browsing step, when the content includes a description for displaying an embedded image generated by executing an application program different from the browsing program, the embedded image generated by executing the application program is displayed as the embedded image. A display image generation step for generating a display image arranged in an embedded region defined by the content;
When the display image includes the embedded region, the mouse event is selected if the position indicated by the mouse event generated by the mouse event generating step is within the embedded region, and is generated by the mouse event generating step. And a selection step of selecting the gesture event determined in the gesture event determination step if the position indicated by the mouse event is outside the embedded region.   In the browsing step, when the mouse event is selected in the selection step, the mouse event is output to a task for executing the application program, and an instruction step for updating an embedded image displayed in the embedded region The operation accepting method according to claim 5, further comprising:   In the browsing step, when a gesture event is selected in the selection step, the display means of the display image is executed by executing a process corresponding to the selected gesture event among the plurality of types of processes. The operation accepting method according to claim 5, further comprising a determining step of determining a display part to be displayed on the screen. An association step of associating a plurality of gesture events corresponding to the plurality of types of processing defined by the browsing program corresponding to the browsing program at the stage of installing the browsing program;
In the gesture event determination step, the gesture event corresponding to a set of one or more mouse events generated continuously in the mouse event generation step corresponds to the plurality of types of processing corresponding to the browsing program in the association step. The operation reception method according to claim 5, further comprising the step of determining the gesture event on condition that it is associated with any one of the above. A browsing program executed by a computer that controls a data processing device,
The data processing device includes:
Display means capable of displaying images;
A touch panel capable of detecting one or more positions designated by a user in the display surface of the display means;
Mouse event generation means for generating a mouse event including position information indicating each of the one or more positions in response to detection of one or more positions by the touch panel;
Gesture event determining means for determining any one of a plurality of gesture events based on a set of one or more mouse events generated continuously by the mouse event generating means,
Causing the computer to execute a browsing step of executing a browsing program, acquiring content described in a markup language from the outside, and displaying at least a part of a display image corresponding to the acquired content on the display means ,
In the browsing step, when the content includes a description for displaying an embedded image generated by executing an application program different from the browsing program, the embedded image generated by executing the application program is displayed as the embedded image. A display image generation step for generating a display image arranged in an embedded region defined by the content;
When the display image includes the embedded area, the mouse event is selected if the position indicated by the mouse event generated by the mouse event generating means is within the embedded area, and is generated by the mouse event generating step. And a selection step of selecting the gesture event determined in the gesture event determination step if the position indicated by the mouse event is outside the embedded region.   In the browsing step, when the mouse event is selected in the selection step, the mouse event is output to a task for executing the application program, and an instruction step for updating an embedded image displayed in the embedded region The browsing program according to claim 9, further comprising:   In the browsing step, when a gesture event is selected in the selection step, the display means of the display image is executed by executing a process corresponding to the selected gesture event among the plurality of types of processes. The browsing program according to claim 9, further comprising a determining step of determining a display portion to be displayed on the screen.

Images