WO2019150529A1

WO2019150529A1 - Portable terminal device and information display method

Info

Publication number: WO2019150529A1
Application number: PCT/JP2018/003438
Authority: WO
Inventors: 清水　宏; 橋本　康宣; 尚久高見澤; 吉澤　和彦
Original assignee: マクセル株式会社
Priority date: 2018-02-01
Filing date: 2018-02-01
Publication date: 2019-08-08

Abstract

An objective of the present invention is to provide a portable terminal device and information display method for enabling guidance of a designated operation button or panel operation without error by displaying operation guidance in correspondence with a variety of subject devices to be operated. To this end, provided is a portable terminal device for displaying information associated with an operation of a subject device to be operated, said portable terminal device configured to comprise: a photography part for photographing the subject device; a display device for displaying a photographic image photographed by the photography part; a first matching part for identifying the subject device from the photographic image; a second matching part for identifying a state of the subject device from the photographic image; an acquisition part for acquiring operation information on the basis of the identified subject device information and the identified state of the subject device; and a control part for displaying the operation information in superposition on the photographic image.

Description

Portable terminal device and information display method

The present invention relates to a portable terminal device and an information display method for displaying information related to operation of an object to be operated.

General household electrical appliances such as refrigerators, washing machines, and air conditioners have become more difficult to handle and troubleshoot operation guidance due to the increasing complexity of operation methods as touch panels with operation buttons become more popular and sophisticated. ing.

As a background art in this technical field, there is JP-A-2015-228072 (Patent Document 1). In Patent Document 1, the AR tag displayed on the operation panel of the elevator is photographed together with the operation panel by a smartphone equipped with an imaging device and a display device, the photographed image is displayed on the display device, and the displayed AR tag and An elevator information display system is disclosed in which guidance information, presence / absence information, advertisement / advertisement information, which is information specified by an AR tag, is superimposed and displayed on the call button image of the operation panel. .

Japanese Patent Laying-Open No. 2015-228072

In Patent Document 1, the object is specialized in an elevator panel, and the elevator is specified by an AR tag. For this reason, application to different target devices is not considered, and there is a problem that application to general household electrical appliances is not possible, for example. Furthermore, since the content to be superimposed is simply information specified by the AR tag, information related to the operation of different target devices is not considered.

An object of the present invention is to provide a portable terminal device and an information display method that can definitely guide specified operation buttons and panel operations by displaying operation guidance corresponding to various target devices that are actually operated. Is to provide.

In view of the above-described background art and problems, the present invention is a mobile terminal device that displays information related to operation of a target device to be operated, and includes a shooting unit that images the target device, and a shooting unit A display device that displays a captured image captured by the first imaging unit, a first matching unit that identifies a target device from the captured image, a second matching unit that identifies a state of the target device from the captured image, and the identified target device A configuration is provided that includes an acquisition unit that acquires operation information based on information and the state of the specified target device, and a control unit that displays the operation information superimposed on a captured image.

According to the present invention, it is possible to provide a portable terminal device and an information display method that can definitely guide specified operation buttons and panel operations corresponding to various target devices that are actually operated.

It is a block diagram of the information display system which consists of the operation target apparatus in Example 1, and the portable terminal device which image | photographs it. 1 is a configuration block diagram of a mobile terminal device in Embodiment 1. FIG. FIG. 10 is a diagram illustrating a display example of operation guidance when an operation error occurs in the target device in the first embodiment. It is a figure explaining the operation | movement which acquires the operation manual of the object apparatus in Example 1 from a server. FIG. 10 is a diagram illustrating a display example of countermeasure explanation guidance when the target device cannot be identified from a captured image in the first embodiment. 6 is a flowchart of operation support sequence processing for displaying operation guidance in the first embodiment. FIG. 10 is a diagram illustrating a display example of operation guidance in the second embodiment. FIG. 10 is a diagram illustrating an example of superimposed display on operation buttons in the third embodiment. FIG. 10 is a diagram illustrating an example of superimposed display on an operation button when there is an operation button with a partial notation or an operation button with a faint notation according to the third embodiment. FIG. 10 is a diagram illustrating an example of superimposed display on operation buttons in the fourth embodiment.

Hereinafter, embodiments of the present invention will be described with reference to the drawings.

FIG. 1 is a configuration diagram of an information display system including an operation target device and a mobile terminal device that photographs the device in the present embodiment. FIG. 1A shows a state in which the mobile terminal device 100 is photographing the operation panel 210 of the operation target device 200, and is an example when the operation target device 200 is a copier. The operation panel 210 includes a power switch (POWER) 211, a copy execution button (COPY) 212, a menu button (MENU) 213, and a system information display unit (READY) 214. FIG. 1B shows a shooting screen of the mobile terminal device 100.

The basic configuration in the present embodiment is that the mobile terminal device has a matching unit that captures an operation target device that is actually operated and specifies an operation target object from the captured image, and operates the specified operation target device. Information is acquired, operation buttons and panels of the operation target device are superimposed on the captured image, and operation guidance obtained from the acquired operation information is displayed.

In FIG. 1B, the mobile terminal device 100 identifies the operation target device 200 from an image obtained by capturing the operation target device 200, acquires operation information of the identified operation target device 200, and captures the operation target device. An example is shown in which explanation guidance for the operation buttons is displayed superimposed on an image of 200 operation panels. In other words, each operation button is superimposed and displayed in the form of a balloon as indicated by 111 to 114. Reference numeral 115 denotes a model number display of the specified operation target device 200.

The mobile terminal device 100 identifies the operation target from these captured images, acquires operation information of the specified operation target, and superimposes operation guidance on the image of the operation panel of the captured operation target device. An application program that executes a series of processes to be displayed is installed in advance, and a series of processes can be executed by executing the application program.

FIG. 2 is a configuration block diagram of the mobile terminal device in the present embodiment. As shown in FIG. 2, the mobile terminal device 100 in the present embodiment is basically a computer system. That is, the CPU 2000 is a central information processing apparatus, and a touch panel that detects that a camera 2010 that captures an image, a display device such as a liquid crystal panel, and an icon displayed on the bus 2000 are touched on the bus 2001. In addition to the display 2020, the hardware SW 2040 that accepts input from a switch such as a power switch or a shutter button of the camera, a program executed by the CPU 2000, basic data, personal information of the photographer or the owner of the processing device Flash memory 2080 storing an address book for managing the destination of image data sent by e-mail, SD-RAM 2090 which is a work area used by a program executed by CPU 2000, and an external for storing captured images as image files Memo 2101, an external memory interface 2100 that interfaces the external memory 2101 to the CPU, a GPS 2110 that acquires position information, an NFC (Near Field Communication) 2050 that directly communicates with touched terminals, and a Bluetooth (registered trademark) that performs short-range wireless communication ) Module 2060, a wireless communication with the outside, for example, a first wireless communication unit that connects to the Internet 2300, for example, a portable communication unit 2160 such as LTE, a second wireless communication unit that performs wireless communication with the outside A wireless LAN module 2150, a gyro 2070 that detects the tilt, direction, movement, and the like of the mobile terminal device 100, an acceleration sensor 2130, a microphone 2120 that converts user voice and the like into voice data, and the like are connected. It also has a projector module 2140 that projects an image for an embodiment to be described later. The gyro 2070 and the acceleration sensor 2130 can be used for camera shake correction of the mobile terminal device 100. In addition, you may have a speaker for an audio | voice output, and you may further provide other sensors, such as an atmospheric pressure sensor. The external memory in which the image file is stored may be an internal memory in the processing device. Further, a flash memory storing personal information and an address book and an external memory storing image files may be used as a common memory. Further, a configuration not shown in the figure such as a digital broadcast receiving function and an electronic money settlement function may be further added.

Further, the mobile terminal device 100 may be a feature phone, a smartphone, a tablet terminal, a wearable terminal, or the like. Further, it may be a PDA (Personal Digital Assistant) or a notebook PC. Further, it may be a digital still camera, a video camera capable of shooting moving images, a portable game machine, or other portable digital devices.

FIG. 3 is a diagram for explaining a display example of operation guidance when an operation error occurs in the target device in the present embodiment. In FIG. 3, first, when an operation error occurs in a home appliance that does not have a terminal cooperation function or the like by communication, as shown in FIG. The operation panel 210 of the operation target device 200 displaying the error code 215 is started. Then, the model of the target device is specified from the captured image by pattern matching. Then, from the detected model and the error code indicating the error operation that indicates the status of the target device, the recovery operation method is read from the manual stored in the main unit memory or server, and the operation panel button displayed on the camera through An operation instruction and an action by the operation button are indicated by superimposing on the button to be operated. That is, as shown in FIG. 3 (A), “Error code: P04 Paper jam occurred in paper feed unit. Please open the left paper feed gate” is displayed on the balloon-shaped superimposed display 116, and the operation The arrow display 117 is performed. FIG. 3B shows a state where the target device is photographed by turning to the left side. For example, by displaying an operation instruction such as “Turn to the left and photograph the sheet feeding gate portion” Display the left image. Then, on the photographed image, a balloon-shaped superimposed display 118 shows “Error code: P04 Paper jam occurs at the paper feed unit. Remove the jammed paper, close the paper feed gate. It automatically returns. ”Is displayed and an operation instruction is displayed.

In this way, when a certain operation button is pressed and a predetermined display appears, a plurality of operation steps of pressing the next operation button are also followed. In addition, a step-by-step sequence corresponding to the operation is guided, and after pressing a certain button, the operation is guided to the next operation. Using these, a procedure by a plurality of operations for fault diagnosis is also guided. In addition, the operation includes elements such as opening the lid of the device and removing the panel. After that, a new captured image is displayed on the internal structure, and an instruction of a work procedure to be subsequently performed is also given. For example, when the front panel is shot again after opening the side panel, the user's destination is guided by an arrow display or the like so that the side panel side is shot.

In addition to the character display (error code), the lamp blink pattern / color, error sound scale (frequency) / pattern, noise, etc. are also used as information sources for acquiring the status including errors of the target device. May be. In addition, the shape of the superimposed display is not limited to the shape of the above-described balloon, and any display that can correspond to the operation button may be used.

In the above description, an actual operation button is operated while viewing the operation guidance on the imaging screen on the mobile terminal device 100. However, when the operation button on the mobile terminal device 100 is touched, the actual operation button is touched. You may make it perform the simulation of operation | movement when an operation button is operated. Moreover, you may make it perform the operation | movement when an actual operation button is operated by touching the operation button on the portable terminal device 100. FIG.

FIG. 4 is a diagram for explaining the operation of obtaining the operation manual of the target device from the server in the present embodiment. In FIG. 4, a mobile terminal device 100, for example, an application for obtaining an operation manual called “Smart Manual” is activated to photograph the operation panel 210 of the operation target device 200. Then, the model of the target device is specified from the captured image by pattern matching. Then, an operation manual is obtained from the server 2410 of the manufacturer 2400 of the target device via the Internet 2300 from the detected model. When the operation error of FIG. 3 occurs, the above-mentioned “appliance for failure of home appliances” is set so that the operation manual acquisition operation is automatically executed without starting up the operation manual acquisition application one by one. It may be incorporated.

FIG. 5 is a diagram illustrating a display example of countermeasure explanation guidance when the target device cannot be identified from the captured image in the present embodiment. In FIG. 5, when the target device cannot be identified from the captured image, as shown in FIG. 5A, the mobile terminal device 100 displays “model name: unknown” device name or model name on the main body shown in the superimposed display 119. Please take a photo. ”Is displayed and prompts the user to take a photo at a specific location. Then, as shown in FIG. 5B, the mobile terminal device 100 can identify the target device by photographing the device name 216, and displays “Model has been confirmed: CP01AX” shown in the superimposed display 120. . In addition to the device name 216, photographing of the entire target device, photographing of the model display portion 217, and photographing of the QR code (registered trademark) 218 may be prompted. In addition, input of a model may be prompted. Alternatively, assuming that the QR code reflected in the screen shot by the mobile terminal 100 is an ID representing a device, decoding is performed, and when it is determined that the device is the device ID, the ID is used. The operation may be shifted to a specific operation.

FIG. 6 is a flowchart of operation support sequence processing for displaying operation guidance in the present embodiment. In FIG. 6, first, in step 5010, the mobile terminal device captures an operation target device to be actually operated, and in step 5020, from the captured image, the operation panel shape, overall shape, device name, and model name of the operation target device. Is compared with the database of each product, and matching processing for specifying the operation target device is performed. This database may be stored in the mobile terminal device, or may be obtained from an external database via a network. As another method of specifying the operation target device, the operation target device may be narrowed down from the purchase history of the device held in the mobile terminal device, or the installation location of the operation target device is detected by the GPS in the mobile terminal device. The installation location may be specified from a correspondence database of installation locations and devices. For example, if it is a washing machine, since it should be installed in the washing machine place, an apparatus can be specified from the positional relationship.

In step 5030, the operation target device is specified. In step 5040, it is determined whether the specification is completed. If the specification is not completed, it is determined in step 5050 whether a timeout has occurred. Returning to step 5010, in accordance with the guidance for handling the case where the target device cannot be specified as described with reference to FIG. 5, the operation target device is photographed again and repeated until the target device is specified. The sequence process is terminated (step 5300).

When the identification is completed in step 5040, in step 5060, the portable terminal device takes an image of the operation panel of the operation target device to be operated, and in step 5070, the operation is performed to specify the status of the operation target device from the captured image. The display on the panel is compared with a database of display information on the panel for each product state, and matching processing is performed. In step 5080, the display content on the operation panel is specified. In step 5090, the display content on the operation panel is determined. For example, if the display content is Ready, the operation guidance obtained from the acquired operation information is displayed by superimposing the operation button or panel of the operation target device shown in FIG. indicate. Further, a response method as shown in step 5110 is displayed according to the error code. For example, in the case of a paper jam of the error code P04 shown in FIG. 3, an instruction to open the side panel shown in step 5200 is displayed, and an instruction to take out the jammed paper is displayed in step 5210. In step 5220, it is determined whether or not a timeout has occurred. If it is within the predetermined time, the process returns to step 5010 to repeat the operation support sequence process. When a time-out occurs in step 5220, the operation support sequence process is terminated (step 5300).

As described above, according to the present embodiment, the target device to be actually operated is photographed, and the operation button and the panel are superimposed on the photographed image to display the operation guidance. There is no need to take action between the two, and it is definitely possible to guide the specified operation buttons and panel operations.

This example describes an example in which a user's finger pressing a button is tracked and superimposed display is performed according to the position of the finger.

FIG. 7 is a diagram for explaining a display example of operation guidance in the present embodiment. 7A, FIG. 7A shows a state in which the mobile terminal device 100 is photographing the operation panel 210 of the operation target device 200, the position of the user's finger is recognized by pattern matching, and the fingertip is When a predetermined operation button is touched, explanation guidance for the operation button is displayed. In FIG. 7A, it is assumed that the fingertip touches the copy execution button (COPY) 212. As explanation guidance of the touched operation button, “Copy by pressing the execution button” on the superimposed display 112 in the form of a balloon. “Start” is superimposed. As described above, since the explanation guidance for all the operation buttons is not displayed at a time as in the first embodiment, only the explanation guidance for the touched operation button is displayed, so that unnecessary display is not performed. There is an advantage that the display screen is easy to see. If the fingertip approaches before touching a predetermined operation button, an explanation guidance for the operation button may be displayed.

Furthermore, the operation target device 200 may actually operate by pressing a button actually displayed on the screen of the mobile terminal device 100. When operating the mobile terminal device 100 for the first time, guidance is displayed. However, when the same button is operated for the second time or later, the guidance display may be omitted.

Also, as shown in FIG. 7B, when the copy operation is currently being executed and an attempt is made to accidentally press the power switch (POWER) 211, a “warning: power SW copy” is displayed as in the superimposed display 121 in the form of a balloon. The action is being executed. Pressing this button causes the copy operation to stop urgently. "Is displayed in a superimposed manner, alerting you when you try to press the wrong place, and guiding you to perform the correct operation. Further, an x icon may be superimposed as in 122, and a circle icon may be superimposed if it is correct. If you try to press the wrong place, you may be alerted with a buzzer or a lamp. Further, a finger contour with a different color may be superimposed and displayed so that a green finger contour is superimposed when trying to press the correct location, and a red finger contour is superimposed when trying to press the wrong location.

As described above, according to the present embodiment, the operation guidance of the operation button to be operated can be displayed by performing the superimposed display according to the position of the operated finger, and the operation of the wrong operation button can be suppressed. Therefore, it is possible to guide the correct operation.

In this embodiment, an example will be described in which the function and specification of the operation button are superimposed on an operation button that does not have a display, such as a small device in which it is difficult to provide a space for displaying the operation button.

FIG. 8 is a diagram for explaining an example of superimposed display on the operation buttons in the present embodiment. FIG. 8 shows a display screen in which the mobile terminal device 100 is photographing the operation target device 200. The operation target device 200 does not display the operation buttons 224 to 228, for example, as shown in the superimposed display 123. It is a pencil case style camera that is a small device. As described in the first and second embodiments, the mobile terminal device 100 identifies the operation target device 200 from the image obtained by capturing the operation target device 200, acquires the operation information of the identified operation target device 200, and performs shooting. The function and specification of the operation button are superimposed and displayed on the image of the operated device 200 that has been operated. For example, a function display “shutter button” indicated by a balloon-shaped superimposed display 124 is displayed on the operation button 224, and a function display “power supply SW 3” indicated by a balloon-shaped superimposed display 125 is displayed on the operation button 225. Press and hold for 2 seconds: ON Press and hold for 1 second: OFF, and the function display “Still image / video switching” indicated by a balloon-shaped superimposed display 126 for the operation button 226 LED green: stationary Display mode red: moving image mode ”, a function display“ LED power ON display status display ”indicated by a balloon-shaped superimposed display 127 is displayed for the operation button 227, and a balloon is displayed for the operation button 228. The specification display “lens fixed focus f = 10 mm F4.0” indicated by the superimposed display 128 of the shape is displayed.

FIG. 9 is a diagram for explaining an example of superimposed display on operation buttons when there are operation buttons that are not partially described or operation buttons that are notated in the present embodiment. FIG. 9 shows a display screen in which the mobile terminal device 100 is photographing the operation panel 210 of the operation target device 200, and a part of the notation of the power switch (POWER) 211 and the copy execution button (COPY) 212. This shows the case where the fading has occurred. In this case, as described in the first and second embodiments, the mobile terminal device 100 identifies the operation target device 200 from the image obtained by photographing the operation target device 200, and captures the operation manual and the photographing of the identified operation target device 200. Compare the images, acquire the information of the operation buttons, estimate the operation buttons with faint notation, and superimpose them on the image of the operation panel 210 of the operation target device 200 that has been photographed, and superimpose the functions and specifications of the operation buttons indicate. At this time, the description is displayed all at once, so that all operations can be understood at a glance.

As described above, according to the present embodiment, since the function and specification of the operation button can be superimposed on the operation button that is not displayed, the user can use the function and specification of the operation button without using the operation manual. There is a feature that can be confirmed.

In the first to third embodiments, the operation guidance is superimposed on the captured image of the operation target device captured by the mobile terminal device 100. In this embodiment, an example in which the operation guidance is displayed on the operation target device will be described.

FIG. 10 is a diagram for explaining an example of superimposed display on the operation buttons in the present embodiment. In FIG. 10, as described in the first to third embodiments, the mobile terminal device 100 specifies the operation target device 200 from an image obtained by capturing the operation target device 200 and acquires operation information of the specified operation target device 200. To do. The mobile terminal device 100 includes a projector module that projects an image, and the operation button operation guidance obtained from the acquired operation information is transferred to the operation panel 210 and the operation buttons of the actual operation target device 200. Projection is performed using a projector module, and the images are superimposed and displayed in the form of balloons as indicated by 111 to 114. 300 is the projection light, and 310 is the projection image.

Thus, when operating the operation buttons, the operation can be performed in a state where the operation guidance is superimposed, so that a reliable operation can be executed.

Although the embodiments have been described above, the present invention is not limited to the above-described embodiments, and includes various modifications. For example, the above-described embodiments have been described in detail for easy understanding of the present invention, and are not necessarily limited to those having all the configurations described. For example, each element in the embodiment may be executed by software processing, or may be configured by hardware such as various integrated circuits and other circuits. Of course, it may be composed of a combination of software and hardware. Further, a part of the configuration of one embodiment can be replaced with the configuration of another embodiment, and the configuration of another embodiment can be added to the configuration of one embodiment. Moreover, it is also possible to add, delete, and replace other configurations for a part of the configuration of each embodiment.

100: portable terminal device, 200: operation target device, 210: operation panel, 111 to 114, 116, 118, 121, 124 to 128: superimposed display, 215: error code, 117: arrow display, 310: projection image

Claims

A mobile terminal device that displays information related to operation of a target device to be operated,
An imaging unit for imaging the target device;
A display device for displaying a photographed image photographed by the photographing unit;
A first matching unit that identifies the target device from the captured image;
A second matching unit that identifies the state of the target device from the captured image;
An acquisition unit for acquiring operation information based on the specified target device information and the state of the specified target device;
A portable terminal device comprising a control unit that superimposes and displays the operation information on the captured image.
The mobile terminal device according to claim 1,
The first matching unit recognizes an operation panel shape and an overall shape of the target device, a device name and a model name from the photographed image, and performs a matching process in comparison with a database of each product to identify the target device. ,
The second matching unit recognizes the state of the target device from the photographed image, and specifies the state of the target device by performing a matching process in comparison with a database of each product. apparatus.
The mobile terminal device according to claim 1 or 2,
The control unit superimposes and displays the operation information corresponding to the operation button display of the target device of the captured image.
The mobile terminal device according to any one of claims 1 to 3,
The portable terminal device, wherein the control unit is preinstalled with an application program for executing a series of processes for the superimposed display.
The mobile terminal device according to any one of claims 1 to 4,
The portable terminal device characterized in that the state of the target device is an error code.
An information display method for displaying information related to operation of a target device to be operated,
Shoot the target device,
Display the captured image,
Identify the target device from the captured image,
Identify the state of the target device from the captured image,
Obtaining operation information of the target device based on the specified target device information and the state of the specified target device;
An information display method, wherein the operation information is superimposed on the captured image.
The information display method according to claim 6,
The identification of the target device is performed by recognizing the operation panel shape and overall shape of the target device, the device name and model name from the captured image, and performing a matching process in comparison with the database of each product,
The identification of the state of the target device is performed by recognizing the state of the target device from the captured image and performing matching processing in comparison with a database of each product.
The information display method according to claim 6 or 7,
The information display method according to claim 1, wherein the superimposed display displays the operation information in correspondence with operation button display of the target device of the captured image.
An information display method according to any one of claims 6 to 8,
An information display method, wherein the superimposed display is executed by an application program that executes a series of processes for superimposed display.
An information display method according to any one of claims 6 to 9,
The information display method, wherein the state of the target device is an error code.