JP2020077239A - Portable device and operation support program - Google Patents

Abstract

To provide a portable device and an operation support program capable of increasing the content that can be displayed on a screen displayed on a display device of an electronic device when supporting the operation of the electronic device.SOLUTION: The portable device that supports the operation of an MFP having a display unit includes a camera, a screen acquisition unit that acquires a screen displayed on the display unit of the MFP from an image captured by the camera, and an operation notification unit that provides a notification of a next operation to be performed. The operation notification unit provides a notification of the next operation to be performed which is associated with the screen itself, acquired by the screen acquisition unit, in help data indicating the screen itself and the next operation to be performed in association with each other (S80).SELECTED DRAWING: Figure 9

Description

  The present invention relates to a mobile device and an operation support program that support the operation of an electronic device including a display device.

  Conventionally, as a mobile device that supports the operation of an electronic device including a display device, an identification code included in a screen displayed on the display device of the electronic device is acquired from an image captured by a capturing device of the mobile device, and the identification code is used. It is known that the associated help screen is displayed on the display device of the mobile device (for example, refer to Patent Document 1).

JP, 2016-045635, A

  However, in the conventional mobile device, it is necessary to include the identification code in the screen displayed on the display device of the electronic device, and the content that can be displayed on the screen displayed on the display device of the electronic device is the identification code. There is a problem of decreasing.

  Therefore, it is an object of the present invention to provide a portable device and an operation support program capable of increasing the content that can be displayed on the screen displayed on the display device of the electronic device when supporting the operation of the electronic device. ..

  A mobile device of the present invention is a mobile device that supports operation of an electronic device including a display device, and includes a photographing device and a screen displayed on the display device of the electronic device from an image photographed by the photographing device. A screen acquisition unit for acquiring and an operation notification unit for notifying the operation to be performed next, and the operation notification unit, in the help data indicating the screen itself and the operation to be performed next in association with each other, It is characterized in notifying the operation to be performed next, which is associated with the screen itself acquired by the screen acquisition unit.

  With this configuration, the mobile device of the present invention notifies the next operation to be performed, which is associated with the screen itself displayed on the display device of the electronic device in the help data, which is acquired from the image captured by the imaging device. Therefore, even if the identification code for specifying the next operation to be performed is not included in the screen displayed on the display device of the electronic device, the next operation can be notified. Therefore, the mobile device of the present invention can increase the content that can be displayed on the screen displayed on the display device of the electronic device when supporting the operation of the electronic device.

  The mobile device of the present invention includes a display device, and the operation notification unit displays on the display device of the mobile device an image in which the operation to be performed next is superimposed on the image captured by the imaging device. May notify the operation to be performed next.

  With this configuration, the mobile device of the present invention displays an image obtained by superimposing an operation to be performed next on the image captured by the imaging device on the display device of the mobile device, thereby improving the operability of the electronic device. be able to.

  An operation support program according to the present invention is an operation support program for supporting an operation of an electronic device including a display device, and includes a portable device including an image capturing device, the image of the electronic device including the image captured by the image capturing device. A screen acquisition unit that acquires a screen displayed on a display device and an operation notification unit that notifies the operation to be performed next are realized, and the operation notification unit includes the screen itself and the operation to be performed next. In the help data indicating the above items in association with each other, the operation to be performed next, which is associated with the screen itself acquired by the screen acquisition unit, is notified.

  With this configuration, the mobile device that executes the operation support program of the present invention is associated with the screen itself displayed on the display device of the electronic device in the help data, which is acquired from the image captured by the image capturing device. Since the operation to be performed is notified, the operation to be performed next can be notified even if the identification code for specifying the operation to be performed next is not included in the screen displayed on the display device of the electronic device. it can. Therefore, the portable device that executes the operation support program of the present invention can increase the content that can be displayed on the screen displayed on the display device of the electronic device when supporting the operation of the electronic device.

  The mobile device and the operation support program of the present invention can increase the contents that can be displayed on the screen displayed on the display device of the electronic device when supporting the operation of the electronic device.

1 is a block diagram of a system according to an embodiment of the present invention. 2 is an external perspective view of the MFP shown in FIG. 1. FIG. 3 is a block diagram of the MFP shown in FIG. 2. FIG. (A) It is a front view of the portable device shown in FIG. FIG. 4B is a rear view of the mobile device shown in FIG. It is a block diagram of the portable device shown in FIG. It is a figure which shows an example of the help data shown in FIG. It is a figure which shows an example of the help data specific information shown in FIG. 6 is a flowchart of part of the operation of the mobile device shown in FIG. 5 when notifying the next operation. 9 is a continuation of the flowchart shown in FIG. 8. FIG. 6A is a diagram showing an example of a screen displayed on the display unit of the MFP, which is included in the image captured by the camera shown in FIG. 5. (B) It is a figure which shows an example of the image displayed on the display part shown in FIG.

  Embodiments of the present invention will be described below with reference to the drawings.

  First, the configuration of a system according to an embodiment of the present invention will be described.

  FIG. 1 is a block diagram of a system 10 according to this embodiment.

  As shown in FIG. 1, the system 10 includes an MFP (Multifunction Peripheral) 20 as an electronic device. In addition to the MFP 20, the system 10 may include at least one MFP having the same configuration as the MFP 20.

  The system 10 includes a mobile device 30 such as a tablet or a smartphone. In addition to the mobile device 30, the system 10 may include at least one mobile device having the same configuration as the mobile device 30.

  FIG. 2 is an external perspective view of the MFP 20. FIG. 3 is a block diagram of the MFP 20.

  As shown in FIGS. 2 and 3, the MFP 20 includes an operation unit 21 which is an input device such as a button for inputting various operations, and a display device such as an LCD (Liquid Crystal Display) for displaying various information. A display unit 22, a printer 23 that is a printing device that prints an image on a recording medium such as paper, a scanner 24 that is a reading device that reads an image from a document, an external facsimile device (not shown), and a public telephone line. A fax communication unit 25, which is a fax device that performs fax communication via a communication line such as a wireless LAN, and a network such as a LAN (Local Area Network) or the Internet, or directly by a wired or wireless connection without a network. A communication unit 26 that is a communication device that communicates with the apparatus, a storage unit 27 that is a non-volatile storage device that stores various types of information, such as a semiconductor memory or an HDD (Hard Disk Drive), and control that controls the entire MFP 20. And part 28.

  At least a part of the operation unit 21 constitutes a touch panel together with the display unit 22.

  The control unit 28 includes, for example, a CPU (Central Processing Unit), a ROM (Read Only Memory) that stores programs and various data, and a RAM (Random Memory) that is used as a work area of the CPU of the control unit 28. Access Memory). The CPU of the control unit 28 executes the program stored in the storage unit 27 or the ROM of the control unit 28.

  FIG. 4A is a front view of the mobile device 30. FIG. 4B is a rear view of the mobile device 30. FIG. 5 is a block diagram of the mobile device 30.

  As shown in FIGS. 4 and 5, the mobile device 30 includes an operation unit 31 which is an operation device such as a button for inputting various operations, and a display which is a display device such as an LCD for displaying various information. The unit 32, a microphone 33 that is a voice input device to which voice is input, a speaker 34 that is a voice output device that outputs voice, a camera 35 as a photographing device, and a network such as a LAN or the Internet, or A communication unit 36 that is a communication device that directly communicates with an external device by wire or wirelessly without using a network, and a storage unit that is a non-volatile storage device such as a semiconductor memory or an HDD that stores various information. 37 and a control unit 38 that controls the entire mobile device 30.

  At least a part of the operation unit 31 constitutes a touch panel together with the display unit 32.

  The display unit 32 is arranged on the front side of the mobile device 30. The camera 35 is arranged on the back side of the mobile device 30, and photographs the outside of the back side of the mobile device 30.

  The storage unit 37 stores an operation support program 37a for supporting the operation of the MFP. The operation support program 37a may be installed in the mobile device 30 at the manufacturing stage of the mobile device 30, or may be additionally installed in the mobile device 30 from the network.

  The storage unit 37 stores help data 37b for supporting the operation of the MFP. In addition to the help data 37b, the storage unit 37 can store one or more help data having the same structure as the help data 37b. The storage unit 37 stores help data for each series of operating procedures in the MFP.

  The storage unit 37 stores help data identification information 37c for identifying the help data stored in the storage unit 37.

  The storage unit 37 stores display unit shape information 37d in which the model name of the MFP and the shape of the display unit of the MFP are associated with each other for each model of the MFP.

  The control unit 38 includes, for example, a CPU, a ROM that stores programs and various data, and a RAM that is a memory used as a work area of the CPU of the control unit 38. The CPU of the control unit 38 executes the program stored in the storage unit 37 or the ROM of the control unit 38.

  The control unit 38 executes the operation support program 37a to acquire the screen displayed on the display unit of the MFP from the image captured by the camera 35, and the operation of the MFP to be performed next ( Hereinafter, the operation notification unit 38b that notifies the "next operation") is realized.

  FIG. 6 is a diagram showing an example of the help data 37b.

  As shown in FIG. 6, the help data 37b is data indicating the screen itself displayed on the display unit of the MFP and the next operation for this MFP in association with each other. Help data 37b is data including all screens displayed on the display unit of the MFP in a series of operation procedures corresponding to help data 37b. It should be noted that the help data 37b includes only the last screen displayed on the display unit of the MFP in the series of operation procedures corresponding to the help data 37b, since there is no next operation, and therefore only the screen is included. Note that the image of the broken line drawn in the next operation in FIG. 6 is not displayed on the display unit 32 when the next operation is displayed on the display unit 32.

  FIG. 7 is a diagram showing an example of the help data identification information 37c.

  As shown in FIG. 7, the help data identification information 37c associates a combination of the model name of the MFP with the type of a series of operation procedures for the MFP (hereinafter referred to as “help content”) and the ID of the help data with each other. Is the data shown below.

  Next, the operation of the system 10 will be described.

  8 and 9 are flowcharts of the operation of the mobile device 30 when notifying the next operation.

  The user can instruct execution of the operation support program 37a via the operation unit 31 of the mobile device 30. When the control unit 38 of the mobile device 30 is instructed to execute the operation support program 37a, the control unit 38 executes the operation support program 37a, thereby performing the operations shown in FIGS. 8 and 9.

  As shown in FIGS. 8 and 9, the operation notification unit 38b includes a model name of a target MFP (hereinafter referred to as “target MFP”) for which the user wants to be notified of the next operation, and a notification of the next operation by the user. A notification prompting the user to input the desired help content (hereinafter referred to as “target help content”) to the microphone 33 by voice is displayed by the display unit 32 and the voice output by the speaker 34. Execute (S51). Therefore, the user can input the model name of the target MFP and the target help content into the microphone 33 by voice. For example, the user speaks to the microphone 33, "I want to set the model XX as XXX", and the model name "XX" of the target MFP and the target help content "XX". “×” can be input to the microphone 33 by voice.

  After the processing of S51, the operation notification unit 38b determines whether or not voice is input to the microphone 33 until it is determined that voice is input to the microphone 33 (S52).

  When the operation notifying unit 38b determines in S52 that the voice is input to the microphone 33, the voice notifying the voice input to the microphone 33 recognizes the model name of the target MFP and the target MFP from the voice input to the microphone 33. It is determined whether or not the combination with the help content has been acquired (S53). Here, the operation notification unit 38b uses the combination stored in the help data identification information 37c as the combination of the model name of the target MFP and the target help content.

  If the operation notification unit 38b determines in S53 that the combination of the model name of the target MFP and the target help content could not be acquired, the operation notification unit 38b may acquire the combination of the model name of the target MFP and the target help content. The notification of the failure is executed by the display of the display unit 32 and the output of the sound by the speaker 34 (S54), and the process of S51 is executed.

  When the operation notification unit 38b determines in S53 that the combination of the model name of the target MFP and the target help content can be acquired, the operation notification unit 38b acquires the combination of the model name of the target MFP and the target help content acquired by voice recognition. A notification prompting the user to give a voice response as to whether or not is correct is executed by the display on the display unit 32 and the voice output by the speaker 34 (S55). Therefore, if the combination of the model name of the target MFP and the target help content notified from the mobile device 30 is the same as the one that the user himself / herself has input by voice, the user is, for example, “correct”. By speaking to the microphone 33, the fact that the combination of the model name of the target MFP and the target help content notified from the mobile device 30 is correct can be answered by voice to the mobile device 30. On the other hand, if the combination of the model name of the target MFP and the target help content notified from the mobile device 30 is not the same as what the user himself / herself has input by voice, the user, for example, “is different”. By speaking to the microphone 33, it is possible to reply to the mobile device 30 by voice that the combination of the model name of the target MFP and the target help content notified from the mobile device 30 is incorrect.

  After the processing of S55, the operation notification unit 38b determines whether or not voice is input to the microphone 33 until it is determined that voice is input to the microphone 33 (S56).

  When the operation notification unit 38b determines in S56 that voice is input to the microphone 33, the voice input to the microphone 33 is recognized by voice, and the model name of the target MFP and the target help content acquired by voice recognition are input. It is determined whether or not a voice answer as to whether or not the combination is correct can be obtained (S57).

  When the operation notification unit 38b determines in S57 that the answer could not be acquired, the operation notification unit 38b executes the notification that the answer could not be acquired by the display on the display unit 32 and the sound output from the speaker 34. Then (S58), the process of S55 is executed.

  When the operation notification unit 38b determines in S57 that the answer can be acquired, the operation notification unit 38b determines whether or not the answer that the combination of the model name of the target MFP and the target help content acquired by voice recognition is correct is acquired. Yes (S59).

  If the operation notification unit 38b determines in S59 that the combination of the model name of the target MFP and the target help content acquired by voice recognition is not correct, the operation notification unit 38b executes the process of S51.

  When the operation notifying unit 38b determines in S59 that the answer that the combination of the model name of the target MFP and the target help content acquired by voice recognition is correct, the operation notification unit 38b determines that the target MFP of the target MFP acquired by voice recognition is acquired. The combination of the model name and the target help content is determined (S60), and the ID of the help data associated with the combination determined in S60 in the help data specifying information 37c is acquired (S61).

  Next, the operation notification unit 38b acquires the help data specified by the ID acquired in S61 from the storage unit 37 (S62).

  Next, the screen acquisition unit 38a activates the camera 35 (S71), and starts displaying the image captured by the camera 35 on the display unit 32 (S72).

  Next, the screen acquisition unit 38a issues a notification prompting the user to include the screen displayed on the display unit of the target MFP in the image captured by the camera 35 displayed on the display unit 32. This is executed by the display by 32 and the output of the sound by the speaker 34 (S73). Therefore, by adjusting the position of the mobile device 30, the user can include the screen displayed on the display unit of the target MFP in the image captured by the camera 35 displayed on the display unit 32. it can.

  Next, the image acquisition unit 38a includes the display unit of the target MFP in the image captured by the camera 35 until it determines that the image captured by the camera 35 includes the display unit of the target MFP. It is determined whether or not (S74). Here, the screen acquisition unit 38a is imaged by the camera 35 by performing image matching with the shape of the display unit of the target MFP, which is associated with the model name determined in S60 in the display unit shape information 37d. In the displayed image, the display unit of the target MFP can be specified.

  When the screen acquisition unit 38a determines in S74 that the image captured by the camera 35 includes the display unit of the target MFP, the screen acquisition unit 38a displays the display unit of the target MFP included in the image captured by the camera 35. The displayed screen is acquired (S75).

  Next, the operation notification unit 38b determines by image matching whether or not the screen acquired in S75 is included in the help data acquired in S62 (S76).

  If the operation notification unit 38b determines in S76 that the screen acquired in S75 is not included in the help data acquired in S62, the operation notification unit 38b notifies that no screen in the target help content is displayed on the display unit of the MFP. This is executed by the display on the display unit 32 and the sound output from the speaker 34 (S77). Therefore, the user can display any screen of the target help content on the display unit of the MFP by operating the target MFP via the operation unit of the target MFP.

  The screen acquisition unit 38a executes the process of S74 after the process of S77.

  If the operation notification unit 38b determines in S76 that the screen acquired in S75 is included in the help data acquired in S62, the operation notification unit 38b displays the screen acquired in S75 in the series of operation procedures corresponding to the help data acquired in S62. It is determined whether or not it is the last screen displayed on the display unit of S based on the screen acquired in S75 and the help data acquired in S62 (S78).

  When the operation notification unit 38b determines in S78 that the screen acquired in S75 is not the last screen, the operation notification unit 38b specifies the next operation associated with the screen acquired in S75 in the help data acquired in S62 (S79).

  Next, the operation notification unit 38b notifies the next operation specified in S79 by displaying an image in which the next operation specified in S79 is superimposed on the image captured by the camera 35 on the display unit 32 (S80). . That is, the operation notification unit 38b displays the next operation specified in S79 by AR (Augmented Reality). For example, when the screen acquired in S75 is the screen shown in FIG. 10A and the help data acquired in S62 is the help data 37b shown in FIG. 6, the operation notification unit 38b displays the screen in FIG. 10B. The image shown is displayed on the display unit 32. Therefore, the user can operate the target MFP via the operation unit of the target MFP with reference to the next operation displayed on the display unit 32. The target MFP changes the image displayed on the display unit of the target MFP by being operated through the operation unit of the target MFP.

  The screen acquisition unit 38a executes the process of S74 after the process of S80.

  When the operation notification unit 38b determines in S78 that the screen acquired in S75 is the last screen, the operation notification unit 38b notifies the display unit 32 and the voice of the speaker 34 that all the operations in the target help content are completed. The output is executed (S81). Therefore, the user can give an instruction to end the execution of the operation support program 37a via the operation unit 31 of the mobile device 30.

  After the processing of S81, the operation notification unit 38b determines whether or not the end of execution of the operation support program 37a is instructed until it is determined that the end of execution of the operation support program 37a is instructed (S82).

  When the operation notification unit 38b determines in S82 that the execution end of the operation support program 37a is instructed, the operation notification unit 38b ends the operation of the camera 32 (S83), and then starts the process of ending the execution of the operation support program 37a. (S84), the operation shown in FIGS. 8 and 9 is ended.

  As described above, the mobile device 30 notifies the next operation to be performed, which is associated with the screen itself displayed on the display unit of the MFP, which is acquired from the image captured by the camera 35, in the help data. As a result (S80), the next operation can be notified even if the identification code for specifying the next operation is not included in the screen displayed on the display unit of the MFP. Therefore, the mobile device 30 can increase the content that can be displayed on the screen displayed on the display unit of the MFP when supporting the operation of the MFP.

  Since the portable device 30 notifies the operation to be performed next, which is associated with the help data in the screen itself displayed on the display unit of the MFP, which is acquired from the image captured by the camera 35, the portable device 30 should perform the next operation. The operation can be recognized by the user in real time.

  Since the mobile device 30 displays, on the display unit 32, an image obtained by superimposing an operation to be performed next on the image captured by the camera 35, the ease of operation of the MFP can be improved. The mobile device 30 may display the next operation to be performed on the display unit 32 by a method other than AR.

  In the present embodiment, the mobile device ends the execution of the operation support program 37a after instructing the end of the execution of the operation support program 37a after determining in S78 that the screen acquired in S75 is the last screen. To do. However, if the mobile device determines in S78 that the screen acquired in S75 is the last screen, the mobile device may automatically terminate the execution of the operation support program 37a.

  The help data, help data specifying information, and display section shape information are stored in the storage section of the mobile device in the present embodiment. However, at least one of the help data, the help data specifying information, and the display section shape information may be stored in a place other than the storage section of the mobile device, such as on the cloud.

  In the present embodiment, the mobile device simultaneously acquires both the model name of the target MFP and the target help content. However, the mobile device may acquire the model name of the target MFP and the target help content one by one.

  In the present embodiment, in the mobile device, the model name of the target MFP and the target help content are input by voice. However, in the mobile device, at least one of the model name of the target MFP and the target help content may be input by a method other than voice. For example, in the mobile device, at least one of the model name of the target MFP and the target help content may be input from the operation unit of the mobile device.

  Although the electronic device of the present invention is the MFP in the present embodiment, it may be an image forming apparatus other than the MFP, such as a printer-dedicated machine, a copy-dedicated machine, a fax-dedicated machine, or a scanner-dedicated machine. For example, a PC (Personal) An electronic device other than the image forming apparatus, such as a computer).

20 MFP (electronic device)
22 Display (display device)
30 mobile device 32 display unit (display device)
35 Camera (shooting device)
37a Operation support program 37b Help data 38a Screen acquisition unit 38b Operation notification unit

Claims (3)

A mobile device that supports the operation of an electronic device including a display device,
A shooting device,
A screen acquisition unit that acquires a screen displayed on the display device of the electronic device from an image captured by the imaging device,
An operation notification unit for notifying the operation to be performed next,
The operation notification unit is associated with the screen itself acquired by the screen acquisition unit in help data indicating the screen itself and the operation to be performed next, and the operation to be performed next. A mobile device characterized by notifying. Equipped with a display device,
The operation notification unit notifies the operation to be performed next by displaying on the display device of the portable device an image in which the operation to be performed next is superimposed on the image captured by the image capturing device. The mobile device according to claim 1, wherein: An operation support program for supporting the operation of an electronic device including a display device,
A mobile device equipped with a shooting device,
A screen acquisition unit that acquires a screen displayed on the display device of the electronic device from an image captured by the imaging device,
And an operation notification unit for notifying the operation to be performed next,
The operation notification unit is associated with the screen itself acquired by the screen acquisition unit in help data indicating the screen itself and the operation to be performed next, and the operation to be performed next. An operation support program characterized by notifying.

Images