JP7010345B2 - Operation device, information processing system, information processing method and program - Google Patents

Description

The present invention relates to an operating device, an information processing system , an information processing method and a program.

Conventionally, as an operation panel for operating an image forming apparatus, for example, an operating device equipped with an OS (Operating System) such as Android (registered trademark) has been used. By using the reading function (talkback function) provided in Android or the like, this operating device can read aloud a message corresponding to a UI (User Interface) part such as an operation button on the screen operated by the user. As a result, the operating device can notify the user of the contents of the UI parts on the screen operated by the user by voice. By using such an operating device, even a visually impaired person can operate the image forming apparatus.

FIG. 1 is a diagram showing an example of an operation screen of a copy application displayed on a conventional operation device. On the operation screen of FIG. 1, UI parts such as an operation button for setting a color mode, printing paper, and the number of copies to be printed, and a start button (start) for starting copying are displayed. Since these UI parts are generated by the operating device, when used by a visually impaired person, the operating device may recognize the UI part operated by the user and read out the message set in the UI part. can. For example, the operating device reads out the message "Start copying" when the start button is operated. In the operation screen of FIG. 1, each UI part is large and simply arranged so that it can be easily used by visually impaired people.

On the other hand, in the conventional image forming apparatus not provided with the above-mentioned operation device, the image of the operation screen generated by the image forming apparatus is displayed on the operation screen operation panel. For a healthy person user who is accustomed to using such a conventional operation screen, it is considered that the operation screen displayed by the conventional image forming apparatus is easier to use than the operation screen as shown in FIG. In consideration of such user convenience, recent operation devices are configured to receive an image of a conventional operation screen generated by an image forming device and display the received image.

However, when the image of the operation screen received from the image forming apparatus is displayed, the conventional operation device cannot recognize the UI part included in this image and cannot read out the message corresponding to the UI part. was there. As a result, in the conventional operation device, when the image received from the image forming device is displayed, even if the UI parts included in the image are operated, the message cannot be output by voice, which is useful for the visually impaired. It became difficult.

The present invention has been made in view of the above problems, and when an image received from an electronic device is operated in an operation device provided with an operation panel that displays an image received from the electronic device, a message is sent. The purpose is to make it readable.

The operation device according to the embodiment is an operation device for operating an electronic device, which is an operation screen for operating the electronic device, and has a plurality of image elements that can be operated by a user. When the user operates a specific image element among the plurality of image elements possessed by the first area and the display unit for displaying the operation screen including the area and the second area, each of the plurality of image elements. When a message corresponding to the specific image element is output by voice among the messages associated with, and the user operates any of the plurality of image elements possessed by the second area, the plurality of image elements are displayed. On the other hand, it is provided with an audio output unit that outputs a message commonly set by audio.

According to each embodiment of the present invention, in an operation device provided with a touch panel that displays an image generated by an electronic device, a message can be read aloud when the image generated by the electronic device is touched. ..

The figure which shows an example of the operation screen of the copy application displayed on the conventional operation device. The figure which shows an example of the hardware composition of an electronic device system. The figure which shows an example of the software structure of the electronic device system. The figure which shows an example of the functional structure of an electronic device system. The figure which shows an example of the operation image including an image. The figure which shows an example of the operation image including the MFP image. The figure which shows an example of the operation image including the MFP image. The figure which shows an example of the 1st message table. The figure which shows an example of the 2nd message table. The figure which shows an example of a message setting table. The sequence diagram which shows an example of the operation of the electronic device system which concerns on 1st Embodiment. The sequence diagram which shows an example of the operation of the electronic device system which concerns on 2nd Embodiment. The sequence diagram which shows an example of the operation of the electronic device system which concerns on 3rd Embodiment. The sequence diagram which shows an example of the operation of the electronic device system which concerns on 3rd Embodiment.

Hereinafter, each embodiment of the present invention will be described with reference to the accompanying drawings. Regarding the description of the specification and the drawings according to each embodiment, the components having substantially the same functional configuration are designated by the same reference numerals, and the superimposed description will be omitted.

(First Embodiment)
The electronic device system according to the first embodiment will be described with reference to FIGS. 2 to 11. The electronic device system according to the present embodiment is a system composed of an electronic device and an operating device for a user to operate the electronic device. Electronic devices include, but are limited to, for example, facsimile machines, printer devices, scanner devices, copy devices, multi-function peripherals (MFPs), refrigerators, washing machines, televisions, microwave ovens, and vacuum cleaners. I can't. In the following, a case where the electronic device is an MFP will be described as an example.

First, the hardware configuration of the electronic device system according to the present embodiment will be described. FIG. 1 is a diagram showing an example of a hard way air configuration of an electronic device system according to the present embodiment. The electronic device system of FIG. 1 includes an MFP (electronic device) 1, an operating device 2, and a communication path 3.

The MFP 1 has a facsimile function, a printer function, a scanner function, a copy function, and the like, and operates according to an instruction received from the operation device 2. The MFP 1 includes a CPU (Central Processing Unit) 101, a ROM (Read Only Memory) 102, a RAM (Random Access Memory) 103, and an HDD (Hard Disk Drive) 104. Further, the MFP 1 includes a communication I / F (interface) 105, a connection I / F 106, an engine unit 107, and a bus 108.

The CPU 101 comprehensively controls the operation of the MFP 1 by executing the program stored in the ROM 102 or the HDD 104, and realizes various functions of the MFP 1.

The ROM 102 stores various programs and data. The RAM 103 provides a working area for the CPU 101. The HDD 104 stores various programs and data.

The communication I / F 105 is an interface for connecting to the network N and communicating with an external device. The network N is a LAN (Local Area Network), a WAN (Wide Area Network), the Internet, or the like. External devices include other MFPs, server computers, client computers, and the like.

The connection I / F 106 is an interface for connecting to the communication path 3 and communicating with the operation device 2.

The engine unit 107 is one or a plurality of devices for realizing a facsimile function, a printer function, a scanner function, a copy function, and the like. The engine unit 107 includes, for example, an image reading unit (scanner) that reads an existing image, an image forming unit (plotter) that prints an image on a print medium, a fax unit that performs fax communication, and the like. Further, the engine unit 107 may include a sorting unit (finisher) for sorting printed print media, an automatic document feeder (ADF: Auto Document Feeder) for automatically feeding the print media, and the like.

The bus 108 interconnects the CPU 101, the ROM 102, the RAM 103, the HDD 104, the communication I / F 105, the connection I / F 106, and the engine unit 107.

The operation device 2 receives information and instructions according to the operation from the user, and transmits the received information and instructions to the MFP 1. The operation device 2 includes a CPU 201, a ROM 202, a RAM 203, a flash memory 204, a communication I / F 205, a connection I / F 206, an external connection I / F 207, an operation panel 208, and a speaker 209.

The CPU 201 comprehensively controls the operation of the operating device 2 by executing the program stored in the ROM 202 or the flash memory 204, and realizes various functions of the operating device 2.

The ROM 202 stores various programs and data. The RAM 203 provides a work area for the CPU 201. The flash memory 204 stores various programs and data.

The communication I / F 205 is an interface for connecting to the network N and communicating with an external device. The network N is a LAN, a WAN, the Internet, or the like. External devices include other MFPs, server computers, client computers, and the like.

The connection I / F 206 is an interface for connecting to the communication path 3 and communicating with the MFP 1.

The external connection I / F 207 is an interface for connecting an external device so as to be communicable. The external device is, for example, an IC (Integrated Circuit) card reader, a microphone, a keyboard, and the like.

The operation panel 208 includes a display such as a liquid crystal display (LCD) or an organic EL (Electro Luminescence) display, and a touch panel provided on the display. The operation panel 208 receives information and instructions according to the user's operation. Further, the operation panel 208 displays various information such as information corresponding to the received operation, the operation status of the MFP 1, and the setting status. The operation panel 208 may include an operation unit such as a hardware key and a display unit such as a lamp.

The speaker 209 outputs a message described later by voice. The speaker 209 may be connected to the MFP 1 via the external connection I / F 207.

The bus 210 interconnects the CPU 201, ROM 202, RAM 203, flash memory 204, communication I / F 205, connection I / F 206, external connection I / F 207, operation panel 208, and speaker 209. do.

The communication path 3 connects the MFP 1 and the operating device 2 so as to be communicable. The communication standard of the communication path 3 is arbitrary, and may be a wired communication standard such as USB (Universal Serial Bus) or a wireless communication standard such as WiFi. Further, electric power may be supplied from the MFP 1 to the operating device 2 via the communication path 3.

Next, the software configuration of the electronic device system according to the present embodiment will be described. FIG. 3 is a diagram showing an example of a software configuration of an electronic device system according to the present embodiment.

The MFP 1 includes an application layer 1001, a service layer 1002, and an OS layer 1003. Each of these layers is composed of various software (programs) stored in the ROM 102 and the HDD 104. When the CPU 101 executes the software of each layer, the function of each layer is realized.

The application layer 1001 provides the function of the MFP 1 to the user by operating the hardware. Hereinafter, the software (application software) constituting the application layer 1001 is referred to as an MFP application. As the MFP application, a facsimile application that provides a facsimile function, a printer application that provides a printer function, a scanner application that provides a scanner function, a copy application that provides a copy function, and the like can be considered. Further, the MFP 1 transmits image information for displaying the operation screen for operating these applications on the operation device 2 to the operation device 2. The MFP 1 operates the application according to the operation of the user received via the operation screen displayed on the operation device 2 based on the transmitted image information.

The service layer 1002 is interposed between the application layer 1001 and the OS layer 1003, and provides an interface for operating the hardware of the MFP 1 to the application layer 1001. Specifically, the service layer 1002 receives an operation request for hardware from the application layer 1001, arbitrates the received operation request, converts the operation request into a command that can be interpreted by the OS layer 1003, and converts the converted command into an OS. Pass to layer 1003. As the operation request for the hardware, for example, an image reading request for the image reading unit, an image printing request for the image forming unit, and the like can be considered. In the present embodiment, the service layer 1002 also provides an interface for using the hardware of the MFP 1 for the application layer 2001 of the operating device 2.

The OS layer 1003 is configured by the OS of the MFP1 and provides a basic function for controlling the hardware of the MFP1. When the OS layer 1003 executes the command received from the service layer 1002, the hardware of the MFP 1 executes the operation requested by the MFP application. In the present embodiment, the OS of the MFP 1 is a different type of OS from the OS of the operating device 2 described later, and may not have a reading function. The OS of the MFP1 is, for example, Linux (registered trademark), but the OS is not limited thereto.

The operation device 2 includes an application layer 2001, a service layer 2002, and an OS layer 2003. Each of these layers is composed of various software (programs) stored in the ROM 202 and the HDD 204. The function of each layer is realized by the CPU 201 executing the software of each layer.

The application layer 2001 provides the function of the operation device 2 to the user by operating the hardware. Hereinafter, the software (application software) constituting the application layer 2001 is referred to as an operation application. The operation application includes an emulator application for emulating the MFP application and a UI application that provides a UI. By using the interface provided by the service layer 1002 of the MFP 1, these operation applications can control the hardware and each function of the MFP 1 without going through the application layer 1001 of the MFP 1.

The service layer 2002 intervenes between the application layer 2001 and the OS layer 2003, and provides the application layer 2001 with an interface for operating the hardware of the operating device 2. Specifically, the service layer 2002 receives an operation request for hardware from the application layer 2001, arbitrates the received operation request, converts the operation request into a command that can be interpreted by the OS layer 2003, and converts the converted command into an OS. Pass to layer 2003. As the operation request for the hardware, for example, an image reading request for the image reading unit, an image printing request for the image forming unit, and the like can be considered.

The OS layer 2003 is configured by the OS of the operating device 2 and provides a basic function for controlling the hardware of the operating device 2. When the OS layer 2003 executes the command received from the service layer 2002, the hardware of the operation device 2 executes the operation requested by the operation application. In the present embodiment, the OS of the operating device 2 is a different type of OS from the OS of the MFP 1, and has a message reading function. The OS of the operating device 2 is, for example, Android (registered trademark), but is not limited thereto.

As described above, in the present embodiment, the MFP 1 and the operating device 2 are equipped with different OSs. Therefore, the communication between the MFP 1 and the operating device 2 is performed not as interprocess communication in the same device but as communication between different devices. By communication between the MFP 1 and the operating device 2, commands corresponding to the information and instructions received by the operating device 2 via the operation application are transmitted from the operating device 2 to the MFP 1, and the MFP 1 responds to the operation from the user. The action is performed. Further, by communication between the MFP 1 and the operating device 2, information such as the operating status and the setting status of the MFP 1 is notified from the MFP 1 to the operating device 2.

Each program included in the MFP 1 and the operating device 2 may be recorded on a computer-readable recording medium as a file in an installable format or an executable format. Examples of such a recording medium include a CD (Compact Disk) -ROM, a flexible disk (FD), a CD-R, a DVD (Digital Versatile Disk), and a USB memory. Further, each program included in the MFP 1 and the operating device 2 may be provided or distributed via a network such as the Internet. Further, each program included in the MFP 1 and the operating device 2 may be provided or distributed by incorporating it into a non-volatile recording medium in advance.

Next, the functional configuration of the electronic device system according to the present embodiment will be described. FIG. 4 is a diagram showing an example of the functional configuration of the electronic device system according to the present embodiment.

The MFP 1 includes a communication unit 11, an image information generation unit 12, and a control unit 13.

The communication unit 11 is realized by the CPU 101 executing a program and controlling the connection I / F 106. The communication unit 11 communicates with the operation device 2 and transmits / receives information to / from the operation device 2.

The image information generation unit 12 is realized by the CPU 101 executing a program. When the user requests to start the MFP application, the image information generation unit 12 displays image information (hereinafter referred to as "MFP image information") for displaying an MFP image (first image) based on the state of the MFP1. To generate. The MFP image is an image according to the state of the MFP1. Generating MFP image information is equivalent to generating an MFP image.

The image information generation unit 12 stores in advance data (image data, text data, etc.) of a plurality of image elements (icons, buttons, frames, messages, etc.) that can form an MFP image. The image information generation unit 12 responds to the state of the MFP 1 by combining these image elements based on the state of the MFP 1 (that is, selecting the image element to be used and determining the arrangement of the selected image element). Generate an MFP image. The MFP image information includes data of each image element constituting the MFP image and arrangement information indicating the arrangement of each image element. The arrangement of each image element is represented by, for example, coordinates on the screen.

The control unit 13 is realized by the CPU 101 executing a program. The control unit 13 controls the hardware of the MFP1 and the MFP application in response to a request from the operating device, and also detects the state of the hardware of the MFP1 and the MFP application.

The operation device 2 includes a communication unit 21, an image information generation unit 22, a coordinate acquisition unit 23, a display unit 24, an application management unit 25, a message storage unit 26, a message setting unit 27, and a voice output unit 28. And.

The communication unit 21 is realized by the CPU 201 executing a program and controlling the connection I / F 106. The communication unit 11 communicates with the operation device 2 and transmits / receives information to / from the operation device 2.

The image information generation unit 22 is realized by the CPU 201 executing a program. The image information generation unit 22 generates image information (hereinafter referred to as “operation image information”) of the operation image (second image). The operation image is an image displayed on the screen of the operation panel 208 (the display portion of the display included in the operation panel 208). Generating operational image information is equivalent to generating operational images.

The image information generation unit 22 stores in advance data (image data, text data, etc.) of a plurality of image elements (icons, buttons, frames, messages, etc.) that can configure an operation image. The image information generation unit 22 generates and stores an operation image by combining these image elements (that is, selecting an image element to be used and determining the arrangement of the selected image element). The operation image information includes data of each image element constituting the operation image and arrangement information indicating the arrangement of each image element. The arrangement of each image element is represented by, for example, coordinates on the screen. Further, the image information generation unit 22 stores the operation image information of the operation image displayed on the display unit 24, which will be described later, in the RAM 203, and identifies the image element arranged at the coordinates where the user touches the operation panel 208.

FIG. 5 is a diagram showing an example of an operation image. The operation image of FIG. 5 is an operation image of the home screen, and is composed of banners arranged at the top and bottom of FIG. 5 and start buttons arranged at predetermined positions for starting an application. ..

The banner is an image element that is always included in the operation image, and the user ID logged in to the MFP 1 is described. In the example of FIG. 5, the user ID is "AAA".

On the banner arranged at the top, a logout button and an energy saving mode button are arranged in order from the left. The logout button is a button for logging out from MFP1. The energy saving mode button is a button for shifting the MFP 1 to the energy saving mode.

On the banner arranged at the bottom, a status confirmation button, a back button, a home button, a menu button, and an end button (stop) are arranged in order from the left. The status confirmation button is a button for displaying the status of the MFP1. The back button is a button for displaying the screen previously displayed. The home button is a button for displaying the home screen. The menu button is a button for displaying the operation menu of the running application. The end button is a button for terminating the processing being executed. Further, the operating device 2 may include a bird key that provides a function corresponding to each button displayed on the banner, and the user may operate the hard key. In this case, the button corresponding to the hard key does not have to be displayed on the banner.

The start button includes a start button for starting the operation application and a start button for starting the MFP application. The activation button for activating the operation application includes a copy button, a scan button, a fax button, a printer button, an application 1 button, and an application 2 button. Further, the start button for starting the MFP application includes a copy button (conventional screen) and a system setting button. When the user touches the start button and requests the operation device 2 to start the application, the operation device 2 displays the operation screen of the application corresponding to the touched start button.

For example, when the user touches the copy button, the copy application (operation application) is activated and the operation image of the copy application as shown in FIG. 1 is displayed. As described above, when the user touches an image element (UI part) included in this operation image, a message corresponding to the touched image element is read aloud.

As described above, it is conceivable that the user wants to use the screen of the MFP application instead of the screen of the operation application as shown in FIG. In such a case, the user may touch the start button of the MFP application.

When the user touches the start button of the MFP application and requests the operation device 2 to start the MFP application, the image information generation unit 22 determines the MFP as an image element based on the state of the MFP 1 and the MFP image information. Generates image information for operation images including images.

FIG. 6 is a diagram showing an example of an operation image including an MFP image. FIG. 6 is an operation image displayed when the system setting button (start button of the MFP application) of FIG. 5 is touched. The operation image Im2 of FIG. 6 is composed of a banner and an MFP image Im1 arranged inside the banner. The MFP image Im1 includes a button (fax initial setting button, etc.) for performing various settings (fax initial setting, etc.), a numeric keypad, a reset button, a start button, and the like. A healthy person user can operate the MFP1 by touching a button included in the MFP image Im1.

FIG. 7 is a diagram showing another example of the operation image including the MFP image. The operation image Im2 of FIG. 7 shows the operation image Im2 when an error has occurred in the MFP1. The operation image Im2 of FIG. 7 is composed of a banner and an MFP image Im1 arranged inside the banner. The MFP image Im1 includes a numeric keypad, a reset button, a start button, and the like. Further, the MFP image Im1 displays a countermeasure for the error. A healthy person user can cancel the error and restore the MFP1 by executing the countermeasure described in the MFP image Im1.

Since the MFP image Im1 of FIGS. 6 and 7 is an image generated by the MFP1, a message for each image element included in the MFP image Im1 is not set. Therefore, even if the user touches the image element, the message corresponding to the touched image element is not read aloud. Therefore, when the operation image Im2 of the MFP application is displayed, the visually impaired person may not be able to grasp the state of the electronic device system and may not be able to use the electronic device system. Therefore, the operating device 2 according to the present embodiment sets a message for the entire MFP image Im1 so that the visually impaired can use the electronic device system.

For example, the operating device 2 sets the message "Unsupported function has been activated. Tap the home button to return to the home screen." To the MFP image Im1 in FIG. When the user touches the MFP image Im1, the above message is read aloud. By touching the home button according to this message, the user (visually impaired person) can display the home screen as shown in FIG. 5, which can also be used by the visually impaired person. Since the home button arranged on the banner is generated by the operation device 2, the corresponding message is read aloud when the user touches it. Therefore, the visually impaired can search for the home button by relying on the message and touch the home button.

In addition, the operating device 2 sets the message "An error has occurred. Please contact the administrator." For the MFP image Im1 in FIG. 7. As a result, the user (visually impaired person) can grasp that an error has occurred in the MFP 1 and take appropriate measures to clear the error.

The coordinate acquisition unit 23 is realized by the CPU 201 executing a program and controlling the touch panel included in the operation panel 208. The coordinate acquisition unit 23 acquires the coordinates on the screen of the operation panel 208 touched by the user.

The display unit 24 is realized by the CPU 201 executing a program and controlling the display included in the operation panel 208. The display unit 24 displays the operation image generated by the image information generation unit 22 on the screen.

The application management unit 25 controls the activation and termination of the operation application and the MFP application and the switching of the operation image according to the instruction from the user.

The message storage unit 26 is realized by a ROM 202, a RAM 203, a flash memory 204, and the like. The message storage unit 26 stores a preset message for each image element that can form an operation image. Image elements that set the message include a start button, an end button, a start button for launching the MFP app, a settings button, a numeric keypad, a reset button, a home button, and at least a part of the frame of the operation image (for example, a tab). Can be considered.

FIG. 8 is a diagram showing an example of a first message table that stores a message preset for each image element that can form an operation image. In the first message table of FIG. 8, the image element ID for identifying the image element, the name of the image element, and the message set for the image element are associated with each other. In the example of FIG. 8, the image element C1 whose image element ID is "C1" is a "copy button" (start button) for starting the copy application (operation application), and "starts the copy application". Message is set. The image element for setting the message and the message set for each image element are not limited to the above example.

Further, the message storage unit 26 stores a message preset for each state of the MFP1. As the state of the MFP1 for setting a message, a state in which an error such as a jam or a system call has occurred (error state) and a normal state (a state in which no error has occurred) can be considered. The message may be set for each type of error that is occurring. Further, the message may be set according to the type of the started MFP application and the type of the MFP image displayed on the screen.

FIG. 9 is a diagram showing an example of a second message table that stores a message preset for each state of the MFP 1. In the second message table of FIG. 9, the state ID for identifying the state of the MFP 1, the content of the state, and the message set for the state are associated with each other. In the example of FIG. 9, the state S1 in which the state ID is "S1" is a state in which "error 1" has occurred in MFP1, and the message "error 1 has occurred" is set. In this way, it is preferable to set a message notifying the occurrence and type of the error for the error state. In addition, the state S3 in which the state ID is "S3" is the state (normal state) of "the system setting application (MFP application) is running" of MFP1, and "the unsupported function has been activated. Please tap the button to return to the home screen. " As described above, it is preferable to set a message for displaying a screen (home screen or the like) that can be used by a visually impaired person for the normal state. The state in which the message is set and the message set in each state are not limited to the above example. For example, a message may be set for the error state to notify the corrective action for the error that has occurred.

The message setting unit 27 sets a message for each image element constituting the operation image based on the operation image information. Specifically, the message setting unit 27 searches the message storage unit 26 using the image element ID of each image element included in the operation image information as a search key, and extracts the message set for each image element. The message setting unit 27 stores the extracted message in association with the image element ID and the arrangement information. As a result, a message is set for each image element constituting the operation image.

In the present embodiment, the message setting unit 27 treats the MFP image as one of the image elements (that is, one image) constituting the operation image. That is, the message setting unit 27 sets one message for the entire MFP image. Specifically, the message setting unit 27 searches the message storage unit 26 using the state of the MFP 1 as a search key, and extracts the message set for the state of the MFP 1. The message setting unit 27 stores the extracted message in association with the MFP image. As a result, a message corresponding to the state of the MFP 1 is set for the MFP image included in the operation image. In the following, it is assumed that the image element of the operation image includes the MFP image.

FIG. 10 is a diagram showing an example of a message setting table for storing a message set for each image element of the operation image. In the message setting table of FIG. 10, the image element ID, the name of the image element, the arrangement information of the image element, and the message set for the image element are associated with each other. In the example of FIG. 10, the image element C2 whose image element ID is "C2" is a start button, and the message "Start processing" is set. Further, the image element Im1 whose image element ID is "Im1" is an MFP image, and the message "Error 1 has occurred" is set.

The audio output unit 28 is realized by the CPU 201 executing a program (OS) and controlling the speaker 209. The voice output unit 28 outputs a message set for the image element of the operation image touched by the user by voice.

Next, the operation of the electronic device system according to the present embodiment will be described. FIG. 11 is a sequence diagram showing an example of the operation of the electronic device system according to the present embodiment. In the following, it is assumed that a message is set for the MFP image included in the operation image according to the presence / absence of an error (error state or normal state).

When the user touches the operation screen displayed on the operation panel 208, the coordinate acquisition unit 23 first calculates and acquires the coordinates on the operation screen touched by the user (step S101). The coordinate acquisition unit 23 passes the coordinate information indicating the acquired coordinates to the image information generation unit 22 (step S102).

When the image information generation unit 22 receives the coordinate information, the image of the operation image touched by the user is based on the received coordinate information and the arrangement information of each image element of the operation image stored in the RAM 203. The element is specified (step S103). The image information generation unit 22 specifies an image element arranged at the coordinates indicated by the coordinate information as an image element touched by the user. Here, it is assumed that the system setting button for starting the system setting application is touched on the home screen of FIG.

When the image information generation unit 22 identifies that the image element touched by the user is a system setting button for activating the system setting application, the image information generation unit 22 notifies the application management unit 25 to that effect (step S104).

When notified that the start button of the system setting application is touched, the application management unit 25 starts the emulator application for emulating the system setting application (step S105). Further, since the system setting application to be started is the MFP application, the application management unit 25 transmits a start request for starting the system setting application to the MFP 1 via the communication unit 21 (step S106).

When the control unit 13 of the MFP 1 receives the start request of the system setting application via the communication unit 11, the control unit 13 activates the system setting application and detects the state of the system setting application in the MFP 1 (step S107). For example, it detects the status of hardware and software related to the system setting application. The control unit 13 passes the state information indicating the state of the system setting application to the image information generation unit 12, and requests the generation of the MFP image for displaying the operation screen of the system setting application on the operation device 2 (step S108). As described above, the state indicated by the state information shall be either an error state or a normal state.

The image information generation unit 12 generates an MFP image of the system setting application according to the state indicated by the state information in response to the generation request from the control unit 13 (step S109). The image information generation unit 12 generates an MFP image of the system setting application in the error state when the state information indicates an error state, and generates an MFP image of the MFP system setting application in the normal state when the state information indicates a normal state. do. The image information generation unit 12 passes the MFP image information of the generated MFP image to the control unit 13 (step S110). The control unit 13 transmits the MFP image information and the state information to the operation device 2 via the communication unit 11 (step S111).

The application management unit 25 of the operation device 2 receives the MFP image information and the state information transmitted from the MFP 1 via the communication unit 21 and passes them to the image information generation unit 22 to request the generation of the operation image of the system setting application. (Step S112). The image information generation unit 22 is displayed on the banner and the MFP image according to the state of the system setting application based on the passed MFP image information and state information in response to the generation request from the application management unit 25. Generate an operation image of the system setting application including an image element other than the MFP image such as a home button (step S113). The image information generation unit 22 responds to the application management unit 25 with the generated operation image (step S114). The application management unit 25 passes the responded operation image information and the state information to the message setting unit 27, and requests the setting of the message (step S115). Here, in order to display the operation image including the MFP image received from the MFP 1, the application management unit 25 requests the message setting unit 27 to acquire a message from the second message table. That is, the application management unit 25 stores whether or not the operation image to be displayed includes the MFP image received from the MFP and the acquisition destination of the message in association with each other.

The message setting unit 27 acquires a message corresponding to the state indicated by the passed state information from the second message table of the message storage unit 26 in response to the setting request from the application management unit 25. The message setting unit 27 associates the acquired message with the ID, name, and arrangement information of the pixel element included in the passed operation screen information, and sets the acquired message in the message setting table of the message storage unit 26 (step S116). .. For example, if the system settings app is in the normal state, the "unsupported function has been activated. The home screen is displayed by tapping the home button." A message such as "Please return to" is acquired, and this message is set in the message setting table of the message storage unit 26 in association with the pixel element ID, the name, and the arrangement information included in the operation screen information. On the other hand, if the system setting application is in an error state, a message such as "An error has occurred. Please contact the administrator." Corresponding to the state of "Error 1" in the second message table is acquired. It is set in the message setting table. Further, the application management unit 25 may acquire a message from the second message table based on the state information and pass it to the message setting unit 27.

When the message setting unit 27 receives the operation image information, the message setting unit 27 refers to the message storage unit 26 and sets a message for each image element included in the operation image except the MFP image. For example, for the home button displayed on the banner, a message such as "Launch the home application." Is registered in the message setting table.

Subsequently, the application management unit 25 passes the operation image information of the operation image to the display unit 24 (step S117). When the display unit 24 receives the operation image information, the display unit 24 displays the operation image on the screen based on the received operation image information (step S118).

After that, when the user touches the screen of the operation panel 208, the coordinate acquisition unit 23 calculates and acquires the coordinates on the screen touched by the user (step S119). The coordinate acquisition unit 23 passes the coordinate information indicating the acquired coordinates to the message setting unit 27 (step S120).

When the message setting unit 27 receives the coordinate information, the message setting unit 27 inputs the image element of the operation image touched by the user based on the received coordinate information and the arrangement information of each image element (including the MFP image) of the operation image. Identify. The image information generation unit 22 specifies an image element arranged at the coordinates indicated by the coordinate information as an image element touched by the user. Then, the message setting unit 27 refers to the message setting table and determines the message set for the specified image element as the message to be output by voice (step S121). The message setting unit 27 passes the determined message (text data) to the voice output unit 28 (step S122).

When the voice output unit 28 receives the message, the voice output unit 28 outputs the received message by voice from the speaker 209 (step S123). As a result, when the user touches the MFP image, a message is output by voice according to the state of the system setting application to be started. Further, although omitted in the example of FIG. 11, after step S119, the application management unit 25 is notified that the image element has been touched, as in steps S102 to S104. The application management unit 25 controls the activation and termination of the operation application and the MFP application and the switching of the operation image according to the touched image element. Further, when the operation image generated based on the MFP image transmitted from the MFP 1 is displayed, the application management unit 25 starts the operation application and the MFP application even if it is notified that the image element is touched. And may be controlled so as not to end or switch the operation image. This makes it possible to prevent erroneous operation by a visually impaired person.

As described above, according to the present embodiment, the operating device 2 is activated with respect to the state of the MFP1 indicating whether or not the MFP application is activated with respect to the MFP image generated by the MFP1 (electronic device). You can set a message according to the status of the MFP application. Therefore, when the MFP image displayed on the screen of the operation panel 208 is touched by the user, the operation device 2 can output (read) a message set for the MFP image by voice. Since the user of the operating device 2 can obtain information (message) about an image element including the MFP image as voice, even if the user is visually impaired, the MFP 1 can be easily operated by using the operating device 2. It can be operated.

Although omitted in the example of FIG. 11, the ID, name, and arrangement information of the image element of the start button of the system setting application on the home screen and the message corresponding to this start button set in the first message table. Is set in the message setting table. Therefore, when the user touches the start button of the system setting application, the voice output unit 28 outputs a message set for the start button (for example, "start the system setting application") by voice. Is preferable. The voice output of this message is realized by the same processing as in steps S120 to S123.

Further, in the example of FIG. 11, since it is assumed that the MFP system setting is started in the background, the system setting application is not started. When the system setting application is not started, the MFP1 may execute the start processing of the system setting application in response to the application start request. The electronic device system according to the present embodiment operates in the same manner as in FIG. 11 even in the case of an MFP application other than the system setting application.

(Second Embodiment)
The electronic device system according to the second embodiment will be described with reference to FIG. In the first embodiment, the operation when the MFP application is started by the operation of the user has been described. In this embodiment, the operation when the MFP application is terminated by the user's operation will be described. The configuration of the electronic device system according to the present embodiment is the same as that of the first embodiment.

FIG. 12 is a sequence diagram showing an example of the operation of the electronic device system according to the present embodiment. In the following, it is assumed that the system setting application is running on the emulator application at the start of the operation, and the operation image including the MFP image in which the message is set is displayed on the screen.

When the user touches the screen of the operation panel 208, the coordinate acquisition unit 23 first calculates and acquires the coordinates on the screen touched by the user (step S201). The coordinate acquisition unit 23 passes the coordinate information indicating the acquired coordinates to the image information generation unit 22 (step S202).

When the image information generation unit 22 receives the coordinate information, the image information generation unit 22 identifies the image element of the operation image touched by the user based on the received coordinate information and the arrangement information of each image element of the operation image (step S203). ). The image information generation unit 22 specifies an image element arranged at the coordinates indicated by the coordinate information as an image element touched by the user. Here, it is assumed that the home button for displaying the home screen is touched. Displaying the home screen is equivalent to terminating the system setting application running on the emulator.

When the image information generation unit 22 identifies that the image element touched by the user is the home button, the image information generation unit 22 notifies the application management unit 25 to that effect (step S204). When the home button is a hard key, the image information generation unit 22 notifies the application management unit 25 when it specifies that the home button has been touched by the user.

When the application management unit 25 is notified that the home button has been touched, since the activated system setting application is an MFP application, the application management unit 25 requests termination to terminate the system setting application via the communication unit 21. Is transmitted to the MFP 1 (step S205). When the control unit 13 of the MFP 1 receives the termination request of the system setting application via the communication unit 11, the control unit 13 terminates the system setting application (step S206).

Further, when the application management unit 25 is notified that the home button has been touched, the application management unit 25 requests the message setting unit 27 to delete the message (step S207). When the message setting unit 27 is requested to delete the message, the message setting unit 27 deletes the message of each image element (including the MFP image) set in the message setting table (step S208). It is not necessary to delete the message corresponding to the image element that is always displayed on the banner.

Further, the application management unit 25 requests the display unit 24 to hide the operation image (step S209). When the display unit 24 is requested to hide the operation image, the display unit 24 hides the operation image of the displayed system setting application (step S210).

Further, the application management unit 25 terminates the emulator application of the system setting application (step S211), activates the home screen application for displaying the home screen (step S212), and notifies the image information generation unit 22 to that effect. (Step S213). Further, although omitted in the example of FIG. 12, in step S211 the application management unit 25 transmits a system setting application termination request to the MFP 1 via the communication unit 21. When the control unit 13 of the MFP 1 receives the end request via the communication unit 11, the system setting application is terminated.

When the image information generation unit 22 is notified of the activation of the home screen application, the image information generation unit 22 generates an operation image of the home screen application (step S214). The image information generation unit 22 responds to the application management unit 25 with the operation image information of the generated operation image (step S215). The application management unit 25 passes the responded operation image information to the message setting unit 27 and requests the setting of the message (step S216). Here, in order to display the operation image that does not include the MFP image received from the MFP 1, the application management unit 25 requests the message setting unit 27 to acquire a message from the first message table.

Upon receiving the operation image information, the message setting unit 27 acquires a message corresponding to each image element included in the operation image from the first message table of the message storage unit 26. Then, the message setting unit 27 sets the ID, name, and arrangement information of each image element in the message setting table in association with the acquired message (step S217). For example, set a message such as "Launch the copy application" to the copy button. The operation image does not include the MFP image.

Further, the application management unit 25 passes the operation image information to the display unit 24 (step S218). When the display unit 24 receives the operation image information, the display unit 24 displays the operation image on the screen based on the received operation image information (step S219).

After that, when the user touches the screen of the operation panel 208, the message set for the touched image element is output by voice.

As described above, according to the present embodiment, when the user touches the button for terminating the MFP application, that is, when the user instructs the user to terminate the MFP application, the display unit 24 includes the MFP image. Hide the operation image. Further, the message setting unit 27 deletes the message set for each image element of the operation image. Then, a new operation image corresponding to the user's operation is displayed, and a new message is set for each image element of the operation image. As described above, according to the present embodiment, a new operation image can be automatically generated and a new message can be automatically set in the new operation image according to an instruction from the user.

In the example of FIG. 12, after the original operation image is hidden, a new operation image is generated and displayed on the screen, but the original operation image is displayed until the new operation image can be displayed. It may be displayed on the screen. The activation of the home screen application (step S210) can be executed at any timing after being notified of the touch of the home button (step S204 or later).

Further, the electronic device system according to the present embodiment operates in the same manner as in FIG. 12 in the case of the MFP application other than the system setting application. The image information generation unit 22 may store the generated operation image of the home application in the RAM 203, and after being notified of the touch of the home button, may acquire the operation image from the RAM 203.
(Third Embodiment)
The electronic device system according to the third embodiment will be described with reference to FIG. In the first embodiment, a case where a new message is set at the timing of receiving an operation from the user has been described. In the present embodiment, a case where a new message is automatically set at the timing when the state of the MFP 1 changes will be described. The configuration of the electronic device system according to the present embodiment is the same as that of the first embodiment.

FIG. 13 is a sequence diagram showing an example of the operation of the electronic device system according to the present embodiment. In the following, it is assumed that an error has occurred in the MFP1 while the MFP1 is executing a predetermined function. For example, when a user uses a copy application to instruct the operation screen of the copy application to execute a copy and the MFP1 executes copying in response to this user instruction, an error such as a jam occurs. It is assumed that it is.

When an error occurs in the MFP 1, the control unit 13 of the MFP 1 detects the occurrence of the error (step S301). That is, the control unit 13 detects that the state of the MFP 1 has changed from the normal state to the error state.

When the control unit 13 detects a change in the state, the control unit 13 passes the state information to the image information generation unit 12 (step S302).

Upon receiving the status information, the image information generation unit 12 generates an MFP image of the error application for displaying the operation procedure for canceling the error according to the status (error status) indicated by the status information (step S303). ).

The image information generation unit 12 passes the MFP image information of the generated MFP image to the control unit 13 (step S304). The control unit 13 transmits the MFP image information and the state information to the operation device 2 via the communication unit 11 (step S305).

The application management unit 25 of the operation device 2 receives the MFP image information and the status information via the communication unit 21. Since the received status information indicates the error status, the emulator application for emulating the error application is started (step S306). Then, the application management unit 25 passes the MFP image information to the image information generation unit 22 and requests the generation of the operation image of the error application (step S307). The image information generation unit 22 generates an operation image including the MFP image based on the passed MFP image information (step S308). The image information generation unit 22 responds to the application management unit 25 with the operation image information of the generated operation image (step S309). The application management unit 25 passes the response operation image information and the state information to the message setting unit 27, and requests the setting of the message (step S310). Here, in order to display the operation image including the MFP image received from the MFP 1, the application management unit 25 requests the message setting unit 27 to acquire a message from the second message table.

In response to the setting request from the application management unit 25, the message setting unit 27 responds to the message "An error has occurred. Please contact the administrator." Is acquired from the second message table of the message storage unit 26, and the acquired message is associated with the ID, name, and arrangement information of the pixel element included in the passed operation screen information, and the message storage unit 26 Set in the message setting table (step S311).

When the message setting unit 27 receives the operation image information, the message setting unit 27 refers to the message storage unit 26 and sets a message for each image element included in the operation image except the MFP image.

Subsequently, the application management unit 25 passes the operation image information of the operation image to the display unit 24 (step S312). When the display unit 24 receives the operation image information, the display unit 24 displays the operation image on the screen based on the received operation image information (step S313).

After that, when the user touches the screen of the operation panel 208, the message set for the touched image element is output by voice.

FIG. 14 is a sequence diagram showing an example of the operation of the electronic device system according to the present embodiment. In the following, it is assumed that the error that occurred while the MFP1 is executing a predetermined function is cleared. For example, it is assumed that the user instructs the execution of the copy by using the copy application, and the error such as the jam that occurs while the MFP1 is executing the copy is cleared.

When the error of the MFP 1 is cleared, the control unit 13 of the MFP 1 detects the clearing of the error (step S401). That is, the control unit 13 detects that the state of the MFP 1 has changed from the error state to the normal state.

When the control unit 13 detects a change in the state, the control unit 13 transmits the state information indicating the detected state (normal state) of the MFP 1 to the operation device 2 via the communication unit 11 (step S402). The application management unit 25 of the operation device 2 receives the status information via the communication unit 21.

When the error is cleared and the state information indicating the normal state is received, the application management unit 25 requests the message setting unit 27 to delete the message (step S403). When the message setting unit 27 is requested to delete the message, the message setting unit 27 deletes the message of each element (including the MFP image) set in the message setting table (step S404). The application management unit 25 requests the display unit 24 to hide the operation image (step S405). When the display unit 24 is requested to hide the operation image, the display unit 24 hides the operation image of the displayed system setting application (step S406).

Further, the application management unit 25 determines that the error has been canceled based on the received status information, terminates the emulator application (step S407), and further activates the application when the error occurs (step S408). Here, since an error occurred when the user operated the operation screen of the copy application and the copy was executed, the copy application is started and the image information generation unit 22 is notified to that effect (step S409).

When the image information generation unit 22 is notified of the activation of the copy application, the image information generation unit 22 generates an operation image of the copy application (step S410). The image information generation unit 22 responds to the application management unit 25 with the operation image information of the generated operation image (step S411). The application management unit 25 passes the responded operation image information to the message setting unit 27 and requests the setting of the message (step S412). Here, in order to display the operation image that does not include the MFP image received from the MFP 1, the application management unit 25 requests the message setting unit 27 to acquire a message from the first message table.

Upon receiving the operation image information, the message setting unit 27 acquires a message corresponding to each image element included in the operation image from the first message table of the message storage unit 26. Then, the message setting unit 27 sets the ID, the name, and the arrangement information of each image element in the message setting table in association with the acquired message (step S413). For example, set a message such as "Start copying" to the start button for starting copying.

When the message setting unit 27 receives the operation image information, the message setting unit 27 refers to the message storage unit 26 and sets a message for each image element included in the operation image except the MFP image.

Further, the application management unit 25 passes the operation image information to the display unit 24 (step S410). When the display unit 24 receives the operation image information, the display unit 24 displays the operation image on the screen based on the received operation image information (step S415).

After that, when the user touches the screen of the operation panel 208, the message set for the touched image element is output by voice.

As described above, according to the present embodiment, when the error state of the MFP 1 is detected, the image information generation unit 12 of the MFP 1 generates a new MFP image according to the error state. The image information generation unit 22 of the operation device 2 generates a new operation image including a new MFP image. Further, the message setting unit 27 generates a new message according to the error state for the new MFP image. As described above, according to the present embodiment, when an error occurs in the MFP 1, a new operation image corresponding to the error state is automatically generated, and a new message corresponding to the error state is set in the MFP image. be able to.

Then, when the error state of the MFP 1 is canceled, the image information generation unit 22 of the operation device 2 generates an operation image of the application when the error occurs. Further, the message setting unit 27 sets a message for the generated operation image. As described above, according to the present embodiment, the operation image of the application at the time of the error is automatically generated at the timing when the error state of the MFP1 is canceled, and the message according to the normal state after the error is canceled. Can be set in the MFP image. The image information generation unit 22 may store the generated operation image of the application when an error occurs in the RAM 203, and acquire the operation image from the RAM 203 when the error state is cleared.

In each of the above embodiments, the case where the user touches the screen has been described as an example, but the user's operation is not limited to the touch. The user's operation may be tap, flip, swipe, pinch, multi-touch, or a combination of these operations. In either case, the message can be output by voice according to the user's operation.

The present invention is not limited to the configurations shown here, such as combinations with other elements in the configurations and the like described in the above embodiments. These points can be changed without departing from the spirit of the present invention, and can be appropriately determined according to the application form thereof.

1: Multifunction device (MFP)
2: Operation device 3: Communication path N: Network 11: Communication unit 12: Image information generation unit 12
13: Control unit 21: Communication unit 22: Image information generation unit 23: Coordinate acquisition unit 24: Display unit 25: Application management unit 26: Message storage unit 27: Message setting unit 28: Voice output unit

Japanese Patent Application Laid-Open No. 2003-323081

Claims (10)

An operating device for operating electronic devices
An operation screen for operating the electronic device, a display unit for displaying an operation screen including a first area and a second area having a plurality of image elements that can be operated by the user, respectively.
When the user operates a specific image element among the plurality of image elements included in the first area, the message corresponding to the specific image element among the messages associated with each of the plurality of image elements. And when the user operates any of the plurality of image elements of the second area, the voice output unit that outputs a message commonly set for the plurality of image elements by voice.
An operating device equipped with. The audio output unit is
When the user operates any of the plurality of image elements included in the second area, a message that is commonly set for the plurality of image elements and is corresponding to the state of the electronic device is voiced. The operation device according to claim 1 to be output. The state of the electronic device is one of a plurality of states including a state in which an error has occurred in the electronic device and a state in which a specific application is started.
The audio output unit is
When the state of the electronic device is a state in which an error has occurred in the electronic device, one of the information indicating the occurrence of the error, the information indicating the type of the error, and the information indicating the coping method of the error. Alternatively, a request for voice output of a message notifying a plurality of information and a message prompting an operation for displaying a specific screen when the state of the electronic device is a state in which a specific application is started. Item 2. The operating device according to item 2. Further provided with a receiving unit for receiving the first image from the electronic device,
The display unit is
The operation device according to any one of claims 1 to 3, which displays the operation screen including the second area based on the first image. The operating device according to claim 4, wherein the second area is an area where the first image is displayed, and the first area is an area where the first image is not displayed. Claims 1 to 1 to further include a setting unit for setting a message for each of the plurality of image elements included in the first region and setting a common message for each of the plurality of image elements included in the second region. The operating device according to any one of claims 5. The setting unit is
The operation device according to claim 6, wherein when the user operates a specific image element included in the first region, a common message set for a plurality of image elements included in the second region is deleted. .. An information processing system in which an electronic device and an operating device for operating the electronic device are communicably connected to each other.
The electronic device is
A control unit that controls execution of processing in the electronic device based on a request from the operating device is provided.
The operating device is
An operation screen for operating the electronic device, a display unit for displaying an operation screen including a first area and a second area having a plurality of image elements that can be operated by the user, respectively.
When the user operates a specific image element among the plurality of image elements included in the first area, the message corresponding to the specific image element among the messages associated with each of the plurality of image elements. And an audio output unit that outputs a message commonly set for the plurality of image elements by audio when the user operates any of the plurality of image elements included in the second area. Information processing system to prepare. A method executed by an information processing system in which an electronic device and an operating device for operating the electronic device are communicably connected to each other.
The electronic device is
Controlling the execution of processing in the electronic device based on the request from the operating device,
The operating device is
An operation screen for operating the electronic device, the operation screen including the first area and the second area having a plurality of image elements that can be operated by the user, respectively, is displayed.
When the user operates a specific image element among the plurality of image elements included in the first area, the message corresponding to the specific image element among the messages associated with each of the plurality of image elements. Is an information processing method that outputs a message commonly set for the plurality of image elements by voice when the user operates any of the plurality of image elements included in the second area. For computers for operating electronic devices,
A display step of displaying an operation screen for operating the electronic device, which includes a first area and a second area having a plurality of image elements that can be operated by the user, respectively.
When the user operates a specific image element among the plurality of image elements included in the first area, the message corresponding to the specific image element among the messages associated with each of the plurality of image elements. When the user operates any of the plurality of image elements possessed by the second area, the output process of outputting a message commonly set for the plurality of image elements by voice and
A program to execute.

Images