JP2013161246A - Portable terminal, control program of portable terminal, and display system including portable terminal - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a portable terminal that enables a user to easily grasp a physical position relationship of each device when executing data processing using the plurality of devices on a network.SOLUTION: A portable terminal 2 includes: a position specification unit for specifying a position of an own machine; a posture specification unit for specifying a posture of the own machine; and a display unit for displaying data flow information A2 indicating the flow of data with executable data processing by data processing devices (a scanner 3, and printers 4, 5), which are detected based on the position and the posture specified by the position specification unit and the posture specification unit, existing within a predetermined distance from the own machine and existing in a specification direction of being viewed from the own machine.

Description

  The present invention relates to a mobile terminal having a function of specifying the position and orientation of its own device, a control program for the mobile terminal, and a display system including the mobile terminal.

  In recent office environments, not only a PC used by each person but also a large number of network devices such as printers and scanners shared by all are installed. When executing printing or scanning in such an office environment, it is necessary to specify a network device using an IP address or a device ID linked to the IP address. It is often difficult to identify individual network devices from the above.

  For example, when performing a scan and print process using a combination of a scanner and a printer, the user needs to specify the IP addresses of both devices on the PC. It is difficult to grasp the positional relationship and the like. For this reason, it has been pointed out that the installation location of the scanner for reading the original and the printer for outputting the printed matter must be checked one by one, which increases the burden on the user. This problem becomes prominent in an office environment where more network devices are installed.

  On the other hand, although it is possible to scan and print using an MFP (Multi-Function Peripheral) that has various functions such as scanning and printing, you can enter a login password or print preview on the operation panel of the MFP. Since there is a danger of being peeped from the side when displaying, there is a possibility that a security problem may occur.

  In relation to the above problem, the following Patent Document 1 searches for a printing device that exists in the vicinity of the current position of the mobile terminal, and the icon of the found printing device is centered on the current position of the mobile terminal. There has been proposed a server device that publishes a web page that is arranged at an appropriate position on a map image. However, even if the server device of Patent Document 1 is used, the user can only grasp the positional relationship between the mobile terminal and the printing device on the two-dimensional map image, and in which direction the printing device is located. It is not always easy to specify whether it is arranged.

JP 2004-234218 A

  The present invention has been made in view of the above-described problems of the prior art, and an object of the present invention is to allow a user to physically execute each device when executing data processing using a plurality of devices on a network. To provide a mobile terminal capable of easily grasping the positional relationship, a control program for the mobile terminal, and a display system including the mobile terminal.

  The above object of the present invention is achieved by the following means.

  (1) A position specifying unit for specifying the position of the own device, a posture specifying unit for specifying the posture of the own device, and data detected based on the position and posture specified by the position specifying unit and the posture specifying unit. Data flow information indicating the flow of data associated with data processing that can be executed by a data processing device that is within a predetermined distance from the own device and that exists in a specific direction as seen from the own device. And a display unit for displaying.

  (2) The image processing apparatus further includes an image capturing unit that captures an image in the specific direction, and the display unit displays the data flow information superimposed on an image captured by the image capturing unit. The portable terminal as described in.

  (3) The display unit displays an icon representing the data processing device superimposed on the image captured by the imaging unit when the data processing device is not reflected in the image captured by the imaging unit. The portable terminal according to (2) above.

  (4) Any one of the above (1) to (3), wherein the display unit displays communication method information indicating a data communication method associated with the data processing in association with the data flow information. The portable terminal as described in.

  (5) The display unit displays an operation button for receiving an operation instruction for the data processing in association with the data flow information, according to any one of (1) to (4) above. Mobile devices.

  (6) The mobile terminal according to (5), wherein the display unit displays an operation guidance for the data processing when the operation instruction is received by the operation button.

  (7) The display unit displays an icon representing the data processing device when the data processing device does not exist within a predetermined distance from the own device or does not exist in a specific direction viewed from the own device. Is displayed in association with the data flow information. The mobile terminal according to any one of (1) to (6) above.

  (8) The terminal device according to any one of (1) to (7), wherein the data flow information is an annotation image including arrows indicating a data flow associated with the data processing.

  (9) A mobile terminal control program, which is specified by the procedure (A) for specifying the position of the own device, the procedure (B) for specifying the posture of the own device, and the procedures (A) and (B). A data processing device that is detected based on the position and orientation, and that can be executed by a data processing device that exists within a predetermined distance from the own device and that exists in a specific direction as seen from the own device. A control program for causing a mobile terminal to execute a procedure (C) for displaying data flow information indicating a data flow associated with the mobile terminal.

  (10) The portable terminal is further caused to execute the procedure (a) for capturing an image in the specific direction, and in the procedure (C), the data flow information is superimposed on the captured image in the procedure (a). The control program according to (9), wherein the control program is displayed.

  (11) In the procedure (C), when the data processing device is not shown in the captured image in the procedure (a), an icon representing the data processing device is superimposed on the captured image in the procedure (a). The control program according to (10) above, characterized by being displayed together.

  (12) In the procedure (C), any one of the above (9) to (11), wherein communication method information indicating a data communication method associated with the data processing is displayed in association with the data flow information. The control program according to one.

  (13) In the procedure (C), an operation button for receiving an operation instruction for the data processing is displayed in association with the data flow information, and any one of the above (9) to (12) The control program described in 1.

  (14) In the procedure (C), when the operation instruction is received by the operation button, an operation guidance for the data processing is displayed.

  (15) In the procedure (C), when the data processing device does not exist within a predetermined distance from the own device or does not exist in a specific direction viewed from the own device, the data processing device is The control program according to any one of (9) to (14) above, wherein an icon to be displayed is displayed in association with the data flow information.

  (16) The control program according to any one of (9) to (15), wherein the data flow information is an annotation image including arrows indicating a data flow associated with the data processing.

  (17) A computer-readable recording medium on which the control program according to any one of (9) to (16) is recorded.

  (18) A display system including a mobile terminal and a server that can communicate with each other, wherein the mobile terminal specifies a position specifying unit that specifies a position of the own device and an attitude specifying unit that specifies the posture of the own device. And the server is located within a predetermined distance from the portable terminal and is identified from the portable terminal based on the position and posture specified by the position specifying unit and the posture specifying unit. A data processing unit having a detection unit that detects a data processing device existing in the direction of the data, wherein the portable terminal includes data flow information indicating a data flow associated with data processing executable by the data processing device detected by the detection unit. A display system further comprising a display unit for displaying.

  According to the present invention, information indicating a data flow associated with data processing that can be executed by a device that exists within a predetermined distance from a mobile terminal and that is present in a specific direction viewed from the mobile terminal is displayed on the display unit of the mobile terminal. Is done. Therefore, according to the present invention, the physical positional relationship between each device on the network can be easily and intuitively grasped, and therefore when performing various data processing using a plurality of devices on the network. The burden on users is greatly reduced.

It is a block diagram which shows the structure of the display system containing the portable terminal which concerns on one Embodiment of this invention. It is the perspective view which looked at the display system containing the portable terminal concerning one embodiment of the present invention from one direction. It is a block diagram which shows the structure of AR server which concerns on one Embodiment of this invention. It is a block diagram which shows notionally various functions of AR server which concerns on one Embodiment of this invention. It is a block diagram which shows the structure of the portable terminal which concerns on one Embodiment of this invention. It is the schematic which shows the apparatus information table which concerns on one Embodiment of this invention. It is the schematic which shows the data processing information table which concerns on one Embodiment of this invention. It is a sequence chart which shows exchange of the signal in the display system containing the portable terminal which concerns on one Embodiment of this invention. It is a sequence chart which shows exchange of the signal in the display system containing the portable terminal which concerns on one Embodiment of this invention. It is a sequence chart which shows exchange of the signal in the display system containing the portable terminal which concerns on one Embodiment of this invention. It is the schematic which shows an example of AR image displayed with the portable terminal which concerns on one Embodiment of this invention. It is the schematic which shows an example of AR image displayed with the portable terminal which concerns on one Embodiment of this invention. It is a flowchart which shows the procedure of AR image display processing of the portable terminal which concerns on one Embodiment of this invention. It is a flowchart which shows the procedure of AR image display processing of the portable terminal which concerns on one Embodiment of this invention. It is the schematic which shows an example of AR image displayed with the portable terminal which concerns on one Embodiment of this invention. It is the schematic which shows an example of AR image displayed with the portable terminal which concerns on one Embodiment of this invention.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
<System configuration>
FIG. 1 is a block diagram illustrating an example of a display system including a mobile terminal according to an embodiment of the present invention. As shown in FIG. 1, this system includes an AR (Augmented Reality) server 1, a mobile terminal 2, and data processing devices 3 to 6 that can execute various data processing. It is connected so that it can communicate. However, the mobile terminal 2 is connected to the network N via a wireless LAN router, a wireless base station, or the like. In addition, the mobile terminal 2 can directly communicate with other devices on the network N by wireless communication technology such as Bluetooth or Wi-Fi (Wireless Fidelity). Here, the network N is a LAN (Local Area Network) according to a standard such as Ethernet (registered trademark), Token Ring, FDDI (Fiber Distributed Data Interface), or a WAN (Wide Area Network) in which LANs are connected by a dedicated line. It is.

  FIG. 2 is a perspective view when the display system according to the example of FIG. 1 is viewed from one direction. In the example of FIG. 2, the AR server 1 and the data processing device 6 are not drawn in the perspective view because they are arranged in another room separated from other devices by a wall. 1 and 2, the data processing device 3 is a scanner, the data processing devices 4 and 5 are printers, and the data processing device 6 is a file server. However, the types and the number of devices constituting this system are not limited to the examples in FIGS. The detailed configuration of each device will be described in order below.

  FIG. 3 is a block diagram showing a configuration of the AR server 1 according to the present embodiment. As shown in FIG. 3, the AR server 1 includes a control unit 11, a storage unit 12, and a communication interface 13, which are connected via a bus 10 for exchanging signals. The detailed configuration of each part will be described in order below.

  The control unit 11 is a CPU (Central Processing Unit), and realizes various functions of the AR server 1 by controlling the operation of each unit according to a program. Various functions of the AR server 1 will be described later. The storage unit 12 includes a RAM (Random Access Memory) that temporarily stores data as a work area of the CPU, a ROM (Read Only Memory) that stores various programs and parameters for controlling the basic operation of the AR server 1, And an HDD (Hard Disk Drive) for storing various programs including the OS (Operating System), parameters, and the like. In particular, the storage unit 12 stores a device information table and a data processing information table, which will be described later (see FIGS. 6 and 7). The communication interface 13 is an interface for performing bidirectional communication with other devices on the network N.

  FIG. 4 is a block diagram conceptually showing various function units realized by the control unit 11. As shown in FIG. 4, the control unit 11 includes functional units such as a state acquisition unit 11 a, a login authentication unit 11 b, a device detection unit 11 c, and a data extraction unit 11 d. These functions will be described in order below. The status acquisition unit 11a acquires the status (status) of each data processing device by periodically communicating with each data processing device via the network N. The login authentication unit 11b determines whether or not to log in to the AR server 1 according to the login notification from the mobile terminal 2.

  The device detection unit 11 c detects a data processing device existing within a predetermined distance from the mobile terminal 2 in response to a command from the mobile terminal 2. More specifically, the device detection unit 11c is based on the position information received from the mobile terminal 2 and the position coordinates of each data processing stored in the storage unit 12, and the distance from the mobile terminal 2 to each data processing device. Is calculated. Hereinafter, the data processing device detected by the device detection unit 11c may be referred to as a “proximity device”.

  The data extraction unit 11 d extracts a part of the device information table, the data processing information table (see FIGS. 6 and 7) and the like in the storage unit 12 in response to a command from the mobile terminal 2 and transmits the extracted information to the mobile terminal 2. More specifically, the data extraction unit 11d creates a list of proximity devices detected by the device detection unit 11c and transmits the list to the mobile terminal 2, or creates a list of data processes that can be executed by a combination of a plurality of proximity devices. And transmitted to the mobile terminal 2.

  FIG. 6 is a schematic diagram of the device information table T1 stored in the storage unit 12. As shown in FIG. 6, the device information table T1 includes “device ID”, “device name”, “device type”, “IP address”, and “position” corresponding to each of the data processing devices 3 to 6 on the network N. "Coordinates" are retained. Hereinafter, each row of the device information table T1 may be referred to as “device information”.

  FIG. 7 is a schematic diagram of the data processing table T2 stored in the storage unit 12. As shown in FIG. 7, the data processing table T2 includes a “processing ID”, a “master device ID”, and a “slave device ID” corresponding to each of data processes that can be executed by a combination of a plurality of data processing devices on the network N. , “Communication method”, and “processing content”. Here, the “process ID” is an ID assigned to each data process, and the “main apparatus ID” is an input side apparatus (that is, a data transmission side apparatus) of a data flow associated with each data process. The “slave device ID” is the device ID of the output side device (that is, the data reception side device) of the data flow accompanying each data processing, and the “communication method” is the master device It is a communication method (LAN, BlueTooth, USB, etc.) with the slave device, and “processing content” is the content of each data processing (scanning, copying, printing, etc.). Hereinafter, each row of the data processing table T2 may be referred to as “data processing information”.

  FIG. 5 is a block diagram illustrating a configuration of the mobile terminal 2 according to the present embodiment. The mobile terminal 2 is a smart phone, a PDA (Personal Digital Assistant), a tablet PC (Personal Computer), a notebook PC, or the like. As shown in FIG. 5, the mobile terminal 2 includes a control unit 21, a storage unit 22, a display unit 23, an input unit 24, an imaging unit 25, a position specifying unit 26, a posture specifying unit 27, a device detecting unit 28, and a communication interface 29. These are connected to each other via a bus 20 for exchanging signals. The detailed configuration of each part will be described in order below.

  The control unit 21 is a CPU, and controls the operation of each unit according to a program and executes various arithmetic processes. The storage unit 22 stores a RAM that temporarily stores data as a work area of the CPU, a ROM that stores various programs and parameters for controlling basic operations of the mobile terminal 2, and various programs and parameters including an OS. An EEPROM (Electrically Erasable Programmable Read-Only Memory) is provided. In particular, the storage unit 22 stores an application for displaying a composite image in which various annotation images are superimposed on a real-time image of the imaging unit 25 using a known AR technique. This application is hereinafter referred to as an AR application. 11 and 12 are schematic diagrams illustrating an example of a composite image displayed by the AR application according to the present embodiment. Details of the AR application will be described later.

  The display unit 23 is a display device such as a liquid crystal display or an organic EL display. The input unit 24 is an input device provided with a pointing device such as a touch panel and a fixed key such as a keyboard. The imaging unit 25 is a digital camera including an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS (Complementary Metal Oxide Semiconductor) image sensor. The mobile terminal 2 can display the real-time image captured by the imaging unit 25 on the display unit 23. The mobile terminal 2 can also switch the real-time image being displayed to a still image at the timing indicated by the user. Moreover, it is preferable that the imaging | photography part 25 is provided with the zoom function so that a distant image can be image | photographed.

  The position specifying unit 26 specifies the position of the mobile terminal 2 using GPS (Global Positioning System). The “position” of the mobile terminal 2 here refers to the position coordinates of the mobile terminal 2 in a predetermined three-dimensional coordinate system. The posture identifying unit 27 identifies the posture of the mobile terminal 2 using a gyroscope, an acceleration sensor, or the like. The “posture” of the mobile terminal 2 here refers to a direction when the mobile terminal 2 is viewed from a predetermined observation point. The position specifying unit 26 and the posture specifying unit 27 specify the position and posture of the mobile terminal 2 by analyzing a two-dimensional marker such as a QR (Quick Response) code included in the shot image of the shooting unit 25. There may be. In addition, in the form using GPS as the position specifying unit 26, there is a great possibility that the position specifying unit 26 is used in an environment in which radio waves from satellites cannot be acquired (for example, indoors without windows). In this case, the position of the timing is obtained by correcting the positioning information position coordinates obtained at the most recent time based on the information obtained after that time from the gyroscope, the acceleration sensor, or the like constituting the posture specifying unit 27. Can be identified.

  The configuration of the position specifying unit 26 and the posture specifying unit 27 is not limited to the above example. For example, a beacon is installed at a plurality of locations in a room, and a receiving unit that receives radio waves or infrared rays from these beacons is provided. It is good also as providing in the portable terminal 2 and specifying the position and attitude | position of an own machine from the received information. In this case, the position specifying unit 26 and the posture specifying unit 27 share a single receiving unit. Further, the procedure required for position specification by the position specifying unit 26 and the procedure required for posture specification by the posture specifying unit 27 can be realized by a common single procedure.

  The communication interface 28 is an interface for performing bidirectional communication with other devices on the network N via a wireless LAN router, a wireless base station, or the like. Note that the mobile terminal 2 can also communicate directly with other devices without using the network N by a wireless communication technology such as Wi-Fi or Bluetooth (registered trademark).

  Next, the data processing devices 3 to 6 according to the present embodiment will be described. First, the data processing device 3 is a scanner, which converts reflected light when an original image set on a platen is irradiated with light into an electrical signal by an image pickup device such as a CCD image sensor, and the original based on the electrical signal. Digital image data is generated and transmitted to another device on the network N. Hereinafter, the data processing device 3 may be referred to as a scanner 3.

  The data processing devices 4 and 5 are printers, and print output images based on digital data received from other devices on the network N on paper using an electrophotographic printing technique. The data processing devices 4 and 5 may be referred to as printers 4 and 5 below. The printing method employed by the printers 4 and 5 is not limited to the electrophotographic method, and may be other printing methods such as an impact method, a thermal transfer method, and an ink jet method. The data processing device 6 is a file server, stores digital data received from other devices on the network N in a storage area such as an HDD, and stores digital data in the storage area in other devices on the network N. Or send. Hereinafter, the data processing device 6 may be referred to as a file server 6.

In the display system according to the present embodiment, a combination of the scanner 3 (device name: scanner A) as a main device and the printers 4 and 5 (device names: printers A and B) as slave devices is used for “copying” an original image. Can be executed. Similarly, in this system, a “scan” of a document image can be executed by a combination in which the scanner 3 (device name: scanner A) is the main device and the file server 6 (device name: server A) is the slave device. Yes, it is possible to execute “printing” of digital data by a combination in which the file server 6 (device name: server A) is the main device and the printer 5 (device name: printer B) is the slave device (see FIG. 7).
<System operation>
Next, an outline of the operation of the display system according to the present embodiment will be described. FIGS. 8 to 10 show signal exchanges when a copy of a document image is executed by a combination in which the scanner 3 (device name: scanner A) is the main device and the printer 4 (device name: printer A) is the slave device. It is a sequence chart. As shown in FIG. 8, when the user instructs the mobile terminal 2 to start the AR application 1, the mobile terminal 2 starts the AR application and displays a login screen to the AR server 1. 8 to 9, the AR server 1 periodically acquires the status of the scanner 3 and the printer 4 at regular intervals.

  Subsequently, when the user inputs the ID and password on the login screen and presses the login button, the portable terminal 2 notifies the AR server 1 of login. Then, the AR server 1 performs login authentication and notifies the portable terminal 2 of the login authentication result. When the login is permitted, the mobile terminal 2 activates the camera (imaging unit 25) and starts displaying a real-time image (continuing to FIG. 9).

  Next, as shown in FIG. 9, when the mobile terminal 2 identifies the position of the mobile terminal 2 and transmits the position information to the AR server 1, the mobile terminal 2 is within a predetermined distance of the mobile terminal 2 based on the received position information. , And a list of detected proximity devices is transmitted to the portable terminal 2. Thereafter, the portable terminal 2 specifies the position of the own device and further specifies the posture of the own device.

  Subsequently, the mobile terminal 2 calculates the display position of the proximity device in the real-time image, acquires the state of the proximity device from the AR server 1, determines the display mode of the information indicating the state of the proximity device, and Information indicating the state of the proximity device is displayed superimposed on the real-time image using the AR technology. Hereinafter, information indicating the state of the proximity device is referred to as “state information”. FIG. 11 is a schematic diagram showing the state information A1 of the proximity devices 3 to 6 displayed superimposed on the real-time image. The procedure shown in this paragraph is repeated for all the proximity devices in the list acquired from the AR server 1.

  Subsequently, the portable terminal 2 displays information indicating a data flow associated with data processing that can be executed by a combination of a plurality of proximity devices, overlaid on a real-time image, using a known AR technology. Hereinafter, information indicating a data flow accompanying the data processing is referred to as “data flow information”. Subsequently, the mobile terminal 2 displays information indicating the data flow communication method associated with the data processing in association with the data flow information in the real-time image. Hereinafter, the information indicating the communication method of the data flow is referred to as “communication method information”. Subsequently, the mobile terminal 2 displays an operation button for accepting an operation instruction related to the above data processing in association with the data flow information in the real-time image (continuing to FIG. 10). Note that the procedure shown in this paragraph is repeated for all data processes that can be executed by a combination of proximity devices in the list acquired from the AR server 1.

  FIG. 11 is a schematic diagram showing the data flow information A2, the communication method information A3, and the operation button B that are displayed superimposed on the real-time image. The processing procedure of the mobile terminal 2 in FIG. 9 is shown in detail in the flowcharts of FIGS.

  As shown in FIG. 10, when the user subsequently presses the operation button B displayed on the mobile terminal 2, the information indicating the operation guidance of the data processing corresponding to the operation button B pressed on the mobile terminal 2 is displayed in real time. The information is displayed in association with the neighboring proximity device (main device). Hereinafter, the information indicating the operation guidance of the data is referred to as “operation guidance information”. FIG. 12 is a schematic diagram showing an example of the operation guidance information A4 displayed so as to be superimposed on the real-time image. The operation guidance information A4 in the example of FIG. 12 is displayed when the operation button B displayed in association with the data flow information A2 associated with data processing with the scanner 3 as the main device and the printer 4 as the slave device is pressed. . When the user presses the “OK” button included in the operation guidance information A4, the mobile terminal 2 highlights the proximity device (secondary device) in the real-time image. At this time, the mobile terminal 2 notifies the AR server 1 of information indicating that the copy is made with the scanner 3 as the scanner A as the main device and the printer 4 as the printer A as the slave device. Then, the AR server 1 instructs the scanner 3 to shift to a mode for realizing a copy function with the printer 4 as an output destination.

  Subsequently, when the user sets an original on the scanner 3 and presses the start button, the scanner 3 reads the original image to generate digital data, and transmits the digital data to the printer 4. Then, the output image is printed based on the scan data received by the printer 4, and after the printing is finished, the AR server 1 is notified of the printing end. Thereafter, the portable terminal 2 acquires the status information of the printer 4 from the AR server 1 and displays a message indicating that the copying of the document image by the scanner 3 and the printer 4 has been completed. Subsequently, when the user instructs the mobile terminal 2 to end the AR application, the mobile terminal 2 notifies the AR server 1 of logout. When the AR server 1 completes the logout process, the mobile terminal 2 ends the AR application.

  Next, FIGS. 13 and 14 are flowcharts showing a procedure of processing (AR image display processing) of displaying a composite image in which various annotation images are superimposed on a real-time image of the photographing unit 25 in the mobile terminal 2 according to the present embodiment. It is. The algorithms shown in the flowcharts of FIGS. 13 and 14 are stored as a control program in the ROM of the storage unit 22, read out to the RAM, and executed by the CPU. Prior to the AR image display processing, it is assumed that the AR application is activated by the mobile terminal 2 and the real-time image of the photographing unit 25 is displayed on the display unit 23 (see FIG. 8).

As shown in FIG. 13, first, the mobile terminal 2 specifies its own location and transmits it to the AR server (S101), and acquires a list of proximity devices of its own from the AR server 1 (S102). And the portable terminal 2 repeats the process of following S104-S111 about each of the neighboring apparatuses contained in the list acquired by S102, and if the process about all the proximity apparatuses is completed (YES of S103), it will transfer to S112 of FIG. In S104, the mobile terminal 2 acquires device information (see FIG. 6) of the nth proximity device (n = 1, 2, 3,...) That is the current processing target from the AR server 1. Subsequently, the mobile terminal 2 specifies the posture of the own device (S105), and further specifies the position of the own device (S106). And the portable terminal 2 is a real-time image of the nth proximity device based on the position coordinates of the nth proximity device included in the device information acquired in S104 and the position and orientation of the own device specified in S105 and S106. The position coordinates at are calculated (S107). The position coordinates calculated in S107 are used when the state information A1 is displayed superimposed on the real-time image in S111 described later. Note that the mobile terminal 2 analyzes the shape of the QR code included in the real-time image of the imaging unit 24 or the shape and color of the proximity device itself instead of acquiring the position coordinates of the proximity device from the AR server 1 in S104. Thus, the position coordinates of the proximity device can be specified.

  Subsequently, the mobile terminal 2 determines whether or not the nth device exists in a specific direction viewed from the mobile terminal 2 based on the position coordinates calculated in S107 (S108). More specifically, in S108, based on the position coordinates calculated in S107, it is determined whether or not the nth proximity device is included within the angle of view of the photographing unit 25. If the n-th device is present in a specific direction (YES in S108), the mobile terminal 2 proceeds to S109 to s111, which will be described later, and displays the state information A1 of the n-th proximity device superimposed on the real-time image. To do. On the other hand, when the n-th proximity device does not exist in a specific direction (NO in S108), the mobile terminal 2 returns to S103, and proceeds to the process for the n + 1-th device (S104 to S111).

  In S109, the mobile terminal 2 acquires the state of the nth proximity device from the AR server 1. And the portable terminal 2 determines the display mode of state information A1 which should be superimposed and displayed on a real-time image according to the distance to the nth proximity device, the state of the nth proximity device, etc. (S110). The “display mode” of the state information here refers to the size, color, transparency, shape, animation effect, etc. of the state information A1 to be displayed superimposed on the real-time image. The portable terminal 2 can change the display mode of the state information according to the distance from the own device to the proximity device, the state of the proximity device, and the like.

  Thereafter, the portable terminal 2 displays the state information A1 of the n-th device superimposed on the real-time image using a known AR technique (S111). At this time, the portable terminal 2 displays the state information A1 in the display mode determined in S110 (continuing to FIG. 14). FIG. 11 is a schematic diagram illustrating an example of the state information A1 displayed in S111. The state information A1 according to the example of FIG. 11 is an annotation image including a balloon indicating the device name (eg, “scanner A”) and the state (eg, “standby”) of the proximity device, and the proximity device in the real-time image. Displayed in association.

  In the example of FIG. 11, the file server 6 (device name: server A) is a proximity device of the mobile terminal 2 and is included in the angle of view of the photographing unit 25, but is viewed from the mobile terminal 2. Because it is placed on the opposite side of the wall, it is not included in the real-time image. Therefore, the portable terminal 2 displays an icon I representing the file server 6 superimposed on the real-time image so that the user can intuitively understand the position of the file server 6, and the file server 6 is displayed on the icon I. Are displayed in association with each other.

  Further, when a proximity device that was originally within the angle of view has moved out of the angle of view due to a change in the position or orientation of the mobile terminal 2, an icon representing the proximity device is displayed superimposed on the real-time image. It is preferable to display the state information of the proximity device in association with the icon. In that case, the icon representing the proximity device is arranged near the outer periphery of the display area of the real-time image so that the user can intuitively understand that the proximity device has gone out of the angle of view of the photographing unit 25. It is preferred that

  Referring to FIG. 14, the mobile terminal 2 acquires from the AR server 1 a list of data processes that can be executed by the combination of proximity devices in the list acquired in S102 (S112). And the portable terminal 2 repeats the process of the following S114-S121 about each of the data processing contained in the list acquired by S112.

  In S114, the mobile terminal 2 acquires data processing information (see FIG. 7) corresponding to the m-th (m = 1, 2, 3,...) Data processing that is the current processing target. Subsequently, the mobile terminal 2 specifies the posture of the own device (S115), and further specifies the position of the own device (S116). Then, the mobile terminal 2 acquires device information (see FIG. 6) corresponding to the master device ID and the slave device ID included in the data processing information acquired in S114 from the AR server 1 (S117). For example, when the data processing information whose processing ID is “1” in the data processing information table T2 of FIG. 7 is acquired in S114, in S117, the scanner 3 (device name: scanner A) and the slave device that are the master device are used. The device information of a certain printer 4 (device name: printer A) is acquired.

  Subsequently, the mobile terminal 2 performs the m-th data processing based on the position coordinates of the main device and the slave device included in the device information acquired in S117 and the position and orientation of the own device specified in S114 and S115. The position coordinates in the real-time image of the device and the slave device are calculated (S118). The position coordinates calculated in S118 are used when displaying the data flow information superimposed on the real-time image in S119.

  Subsequently, the portable terminal 2 displays the data flow information A2 associated with the m-th data process superimposed on the real-time image using a known AR technique (S119). FIG. 11 is a schematic diagram illustrating an example of the data flow information A2 displayed in S119. The data flow information A2 in the example of FIG. 11 is an annotation image that has an arrow shape starting from the m-th data processing master device and ending with the slave device, and is associated with the master device and slave devices in the real-time image. It is displayed.

  FIG. 15 is a schematic diagram illustrating another example of the data flow information A2. The data flow information A2 in the example of FIG. 15 is an annotation image made up of a cloud-like solid arranged between the m-th data processing main device and the slave device. However, the data flow information according to the present embodiment is not limited to the examples of FIGS. 11 and 15, and a balloon indicating a serial number (for example, master device = 1, slave device = 2) assigned to the master device and the slave device, It may be an animation or the like showing a data flow from the master device to the slave device. It is also possible to obtain data flow information by highlighting the main device and the slave device in the real-time image with the same color.

  In S119, the display size, display color, etc. of the data flow information may be changed according to the usage history of each data process recorded in the AR server 1 or the like (eg, the number of uses, the use time, etc.). . In addition, the display contents can be simplified by displaying only the data flow information corresponding to the data processing whose past use frequency exceeds a certain level.

  Subsequently, the mobile terminal 2 refers to the “communication method” included in the data processing information acquired in S114 and associates the communication method information A3 of the data flow associated with the m-th data processing with the data flow information A1. Display (S120). FIG. 11 is a schematic diagram illustrating an example of the communication method information A3 displayed in S120. The communication method information A3 in the example of FIG. 11 is an annotation image made up of symbols indicating the communication method (Bluetooth) of the data flow accompanying the mth data processing, and is displayed near the middle point of the data flow information A2 made up of an arrow shape. Has been.

  Subsequently, the mobile terminal 2 uses the well-known AR technology to execute an operation button B for receiving an operation instruction for executing the m-th data processing or displaying an operation guidance for the data processing in real time. The image is superimposed and displayed (S121). FIG. 11 is a schematic diagram illustrating an example of the operation button B displayed in S121. The operation button B in the example of FIG. 11 is arranged adjacent to the data flow information A2 having an arrow shape, and the user touches the operation button B to display the operation guidance A4 for the data processing. Can do.

  FIG. 12 is a schematic diagram illustrating an example of operation guidance A4 for data processing displayed on a real-time image. In the example of FIG. 12, the operation guidance A4 is displayed in association with the main device (“scanner A”) of the data processing (“Copy”) in the real-time image. Note that the method of giving an operation instruction to the mobile terminal 2 by the user is not limited to touching the operation button, but the user slides his / her finger from the main device to the slave device in the real-time image or is mounted on the mobile terminal 2. An operation instruction may be given by inputting sound into a microphone.

  After that, the portable terminal 2 proceeds to the process for the (m + 1) th data process (S114 to S121), and when the process for all the data processes in the list acquired in S112 is completed (YES in S113), the AR image display process End (end).

  As described above, according to the display system according to the present embodiment, it accompanies data processing that can be executed by a data processing device that exists within a predetermined distance from the mobile terminal 2 and that exists in a specific direction viewed from the mobile terminal 2. Since the data flow information A2 indicating the data flow is displayed on the display unit 23 of the portable terminal 2, according to the display system according to the present embodiment, the physical positional relationship of each data processing device on the network N can be easily set. In addition, since it can be grasped intuitively, the burden on the user when executing various data processing using a plurality of data processing devices on the network N is greatly reduced.

  The present invention is not limited to the above-described embodiments, and can be variously modified within the scope of the claims. For example, in the above embodiment, the mobile terminal displays data flow information associated with data processing that can be realized by a combination of a plurality of data processing devices, but can be executed by a combination of the mobile terminal and each data processing device. When data processing exists, data flow information accompanying the data processing can be displayed. FIG. 16 is a schematic diagram illustrating an example of data flow information A2 associated with data processing that can be executed by each combination of the mobile terminal 2, the scanner 3, and the printer 4.

  In the example of FIG. 16, image data can be printed by a combination of the portable terminal 2 and the printer 4, and a document image can be scanned by a combination of the scanner 3 and the portable terminal 2. Therefore, in this example, as the data flow information A2 corresponding to scanning (“Scan”) and printing (“Print”), the arrow shape from the scanner 3 to the lower side of the display area in the real-time image, and the lower side of the display area Annotation images each having an arrow shape reaching the printer 4 in the real-time image are displayed. However, an icon corresponding to the mobile terminal 2 is displayed at a predetermined position on the real-time image instead of the arrow shape starting from the lower side of the display area of the real-time image as shown in FIG. It is good also as displaying the annotation image which consists of an arrow shape.

  Further, the mobile terminal according to the present invention is not limited to the smart phone or the like shown in the above embodiment, and may be a goggle type display device such as a head mounted display that displays an AR image superimposed on a real space scene. . Furthermore, in the above embodiment, data flow information associated with data processing realized by two data processing devices is displayed. However, the data flow information according to the present invention is data realized by three or more data processing devices. It may indicate a data flow accompanying processing.

1 AR server,
11 Control unit,
11a state acquisition unit,
11b Login authentication unit,
11c device detector,
11d data extraction unit,
12 storage unit,
13 Communication interface,
2 mobile devices,
21 control unit,
22 storage unit,
23 display section,
24 input section,
25 Shooting department,
26 Location specifying part,
27 Posture specifying part,
28 communication interface,
3 Data processing device (scanner),
4 Data processing device (printer),
5 Data processing device (printer),
6 Data processing device (file server),
A1 status information,
A2 data flow information,
A3 communication method information,
A4 operation guidance information,
B Operation button,
I icon,
N network,
T1 device information table,
T2 data processing table.

Claims (18)

A position identifying unit that identifies the position of the aircraft,
A posture identifying unit that identifies the posture of the aircraft,
A data processing device detected based on the position and posture specified by the position specifying unit and the posture specifying unit, the data processing device being within a predetermined distance from the own device, and a specific direction seen from the own device A display unit that displays data flow information indicating a data flow associated with data processing that can be executed by the data processing apparatus existing in the mobile terminal. A photographing unit that photographs the image in the specific direction;
The mobile terminal according to claim 1, wherein the display unit displays the data flow information superimposed on an image captured by the imaging unit.   The display unit displays an icon representing the data processing device in a superimposed manner on an image captured by the imaging unit when the data processing device is not reflected in an image captured by the imaging unit. 2. The mobile terminal according to 2.   The mobile terminal according to any one of claims 1 to 3, wherein the display unit displays communication method information indicating a communication method of data accompanying the data processing in association with the data flow information.   The mobile terminal according to claim 1, wherein the display unit displays an operation button for receiving an operation instruction for the data processing in association with the data flow information.   The portable terminal according to claim 5, wherein the display unit displays an operation guidance for the data processing when the operation instruction is received by the operation button.   The display unit displays an icon representing the data processing device when the data processing device does not exist within a predetermined distance from the own device or does not exist in a specific direction viewed from the own device. The mobile terminal according to claim 1, wherein the mobile terminal is displayed in association with flow information.   The terminal device according to claim 1, wherein the data flow information is an annotation image including arrows indicating a data flow associated with the data processing. A control program for a mobile terminal,
A procedure (A) for identifying the position of the aircraft;
Procedure (B) for identifying the attitude of the aircraft,
A data processing device that is detected based on the position and orientation specified in the steps (A) and (B), and that is present within a predetermined distance from the own device and that is viewed from the own device A control program for causing a portable terminal to execute a procedure (C) for displaying data flow information indicating a data flow associated with data processing that can be executed by a data processing apparatus existing in the mobile terminal. Causing the portable terminal to further execute a procedure (a) of taking an image in the specific direction;
The control program according to claim 9, wherein in the procedure (C), the data flow information is displayed by being superimposed on the captured image obtained in the procedure (a).   In the procedure (C), when the data processing device is not shown in the photographed image in the procedure (a), an icon representing the data processing device is superimposed on the photographed image in the procedure (a). The control program according to claim 10.   The control according to any one of claims 9 to 11, wherein in the step (C), communication method information indicating a communication method of data accompanying the data processing is displayed in association with the data flow information. program.   The control program according to any one of claims 9 to 12, wherein in the step (C), an operation button for receiving an operation instruction for the data processing is displayed in association with the data flow information.   14. The control program according to claim 13, wherein in the step (C), an operation guidance for the data processing is displayed when the operation instruction is received by the operation button.   In the step (C), when the data processing device does not exist within a predetermined distance from the own device or does not exist in a specific direction viewed from the own device, an icon representing the data processing device is displayed. The control program according to claim 9, wherein the control program is displayed in association with the data flow information.   The control program according to any one of claims 9 to 15, wherein the data flow information is an annotation image including arrows indicating a data flow associated with the data processing.   The computer-readable recording medium with which the control program as described in any one of Claims 9-16 was recorded. A display system including a mobile terminal and a server that can communicate with each other,
The mobile terminal has a position specifying unit that specifies the position of the own device, and a posture specifying unit that specifies the posture of the own device,
The server is data existing within a predetermined distance from the mobile terminal and in a specific direction viewed from the mobile terminal based on the position and orientation specified by the position specifying unit and the attitude specifying unit. A detection unit for detecting the processing device;
The mobile terminal further includes a display unit that displays data flow information indicating a data flow associated with data processing that can be executed by the data processing device detected by the detection unit.