JP2014174507A - Augmented reality display device, information terminal, multi display system, and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a highly convenient multi display system.SOLUTION: A multi display system includes an information terminal which displays a real screen, and an augmented reality display device which displays a virtual screen which is an augmented reality display screen different from the real screen. The augmented reality display device detects position range of the real screen displayed on the information terminal, and controls a display position of the virtual screen so that the detected position range of the real screen and the position range of the virtual screen do not overlap.

Description

  The present invention relates to a technique for realizing a multi-display system by augmented reality display.

  A so-called multi-display system that displays different contents on a plurality of display devices is used in various situations.

  For example, an English sentence is displayed on one display of a multi-display system configured by a computer connected to two displays, and a part of the displayed English document designated by a user with a mouse or a touch pen A form of use is known in which a Japanese translation of the text is displayed on the other display.

  In the case of such a system, it is a display screen of two displays as a real environment perceived by a person.

  In recent years, a so-called augmented reality (AR) technology for extending a real environment perceived by a person using a computer has been developed and is used in various fields. Hereinafter, the display screen of the real environment is referred to as “real screen”, and the display screen of augmented reality is referred to as “virtual screen”.

  For example, in Patent Literature 1, a part of an augmented reality display screen (“virtual space display screen” in Patent Literature 1) is displayed on a display device worn by the user, and the display device is displayed according to the movement of the user's head. In the display system in which the part to be displayed is moved, when the part to be displayed on the display is moved, the part to be displayed on the display is moved from one end of the virtual space display screen to the other end. Control is performed so as to move from the other end to the virtual space display screen. In this system, in a system such as that described above, when a display that performs augmented reality display can display only a portion of the entire display content (virtual space display screen) at a time, the virtual space display can be performed while viewing the display portion of the display. This is intended to facilitate the search for a desired position on the screen.

  In addition, when a character displayed on the game play screen (real screen) is pointed, the real screen displays the name and attributes of the character as a virtual screen in the vicinity of the character on the real screen using augmented reality display technology. Also known is a system that displays images on top of each other.

JP 2001-337645 A

  When all the multi-display systems are realized in a real environment, it is necessary to secure a physical space for a plurality of display devices, which is inconvenient. In particular, in the case of a tablet terminal or the like that is assumed to move, it is inconvenient at the time of movement in addition to a weight problem to connect other display devices in addition to the touch panel provided in itself.

  Therefore, it is conceivable to display one or more display screens of a plurality of display screens displayed by the multi-display system on a virtual screen by using augmented reality display technology. However, when all the multi-display systems are realized in a real environment, since all the display screens are real screens displayed on a display device that physically exists, these display screens do not overlap each other. On the other hand, when one or more display screens of a plurality of images displayed by the multi-display system are displayed on a virtual screen, the real screen and the virtual screen may overlap, which is inconvenient for the user. May be. This problem is also particularly noticeable when the multi-display system includes a tablet terminal that is assumed to move.

  The present invention has been made in view of the above circumstances, and provides a technique for realizing a highly convenient multi-display system.

  One aspect of the present invention is an augmented reality display device that displays an augmented reality display screen (virtual screen) different from a display screen (real screen) of an information terminal. The augmented reality display device includes a real screen position detection unit and a control unit.

  The real screen position detection unit detects a position range of the real screen, and the control unit prevents the position range detected by the real screen position detection unit and the position range of the virtual screen from overlapping each other. The display position of the virtual screen is controlled.

  Another aspect of the present invention is an information terminal. The information terminal includes a display unit and a real screen position presentation unit.

  The real screen position presentation unit causes an augmented reality display device that displays an augmented reality display screen (virtual screen) different from a display screen (real screen) by the display unit to detect a position range of the real screen. Presents real screen position information that can be used.

  In addition, the augmented reality display device and the information terminal of the above aspect are displayed by replacing them with a method, the system including the augmented reality display device or the information terminal, and the processing by the augmented reality display device or the information terminal are executed on the computer A program to be displayed and a recording medium storing these programs are also effective as an aspect of the present invention.

  According to the technology of the present invention, a highly convenient multi-display system can be realized.

1 is a diagram showing a multi-display system according to a first embodiment. It is a figure which shows the augmented reality display apparatus in the multi-display system shown in FIG. It is a figure which shows the real screen position detection part in the augmented reality display apparatus shown in FIG. FIG. 3 is a diagram illustrating an example of a positional relationship between a real screen and a virtual screen displayed by the multi-display system illustrated in FIG. 1 (part 1). FIG. 3 is a diagram illustrating an example of a positional relationship between a real screen and a virtual screen displayed by the multi-display system illustrated in FIG. 1 (part 2). FIG. 3 is a diagram showing an example of a positional relationship between a real screen and a virtual screen displayed by the multi-display system shown in FIG. 1 (No. 3). FIG. 4 is a diagram showing an example of a positional relationship between a real screen and a virtual screen displayed by the multi-display system shown in FIG. 1 (No. 4). FIG. 10 is a diagram showing an example of a positional relationship between a real screen and a virtual screen displayed by the multi-display system shown in FIG. 1 (No. 5). It is a figure which shows the multi-display system concerning 2nd Embodiment.

    Embodiments of the present invention will be described below with reference to the drawings. For clarity of explanation, the following description and drawings are omitted and simplified as appropriate. It should be understood by those skilled in the art that each element described in the drawings as functional blocks for performing various processes can be realized in various forms by a combination of hardware and software (program). It is not limited to any of the above. Note that, in each drawing, the same element is denoted by the same reference numeral, and redundant description is omitted as necessary.

  Further, the above-described program can be stored using various types of non-transitory computer readable media and supplied to a computer. Non-transitory computer readable media include various types of tangible storage media. Examples of non-transitory computer-readable media include magnetic recording media (for example, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (for example, magneto-optical disks), CD-ROM (Read Only Memory) CD-R, CD -R / W, semiconductor memory (for example, mask ROM, PROM (Programmable ROM), EPROM (Erasable PROM), flash ROM, RAM (Random Access Memory)). The program may also be supplied to the computer by various types of transitory computer readable media. Examples of transitory computer readable media include electrical signals, optical signals, and electromagnetic waves. The temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.

<First Embodiment>
FIG. 1 shows a multi-display system 100 according to a first embodiment of the present invention. The multi-display system 100 includes an information terminal 110 and an augmented reality display device 200.

  In the present embodiment, as an example, the information terminal 110 is a tablet terminal, and includes a touch panel 120 and a housing 130. A mark 140 is provided on the housing 130. The tablet terminal 110 has a wireless communication function, for example, and can perform wireless communication with the augmented reality display device 200.

  The mark 140 indicates to the augmented reality display device 200 that it can support the AR device, and is, for example, a generally known AR marker. In the present embodiment, the mark 140 has a positional relationship with the touch panel 120, and the pattern includes information indicating the size of the display screen (real screen) by the touch panel 120.

  The tablet terminal 110 receives input from an input unit such as a finger or a touch pen via the touch panel 120. In the present embodiment, the tablet terminal 110 can determine whether the information input from the above-described input means is information to itself, information to the augmented reality display device 200, and the input information is extended. When it is determined that the information is information for the reality display device 200, the information is transmitted to the augmented reality display device 200.

  Specifically, whether the information to the tablet terminal 110 itself or the information to the augmented reality display device 200 is, for example, information input with one finger is information to the tablet terminal 110 and input with two fingers. It can be determined that the processed information is information for the augmented reality display device 200.

  Alternatively, a predetermined position range of the touch panel 120 may be set in advance, and it may be determined that information input by input within the range is information to the augmented reality display device 200. In this case, in order to reduce the influence on the display screen of the touch panel 120, it is preferable that the predetermined position range is as small as possible as any one of the four corners of the touch panel 120.

  FIG. 2 shows an augmented reality display device 200. The augmented reality display device 200 displays an augmented reality screen (virtual screen), and is used by being worn on the user's head, for example. As shown in FIG. 2, the augmented reality display device 200 includes a communication unit 210, a real screen position detection unit 220, a control unit 230, and a virtual screen display unit 240.

  The communication unit 210 is a functional block responsible for communication with the tablet terminal 110 wirelessly, and transfers information to the control unit 230 when information is received from the tablet terminal 110.

  The real screen position detection unit 220 detects the position range of the display screen (real screen) by the touch panel 120 of the tablet terminal 110 and supplies the detected position range to the control unit 230. A configuration example of the real screen position detection unit 220 and a specific detection method example of the real screen position range by the real screen position detection unit 220 will be described later.

  The control unit 230 displays a virtual screen on the virtual screen display unit 240 so that the real screen position range and the virtual screen position range do not overlap according to the real screen position range detected by the real screen position detection unit 220. The display position of is indicated. The virtual screen display unit 240 displays a virtual screen at a display position designated by the control unit 230.

  In the present embodiment, control unit 230 further controls display contents to be displayed on virtual screen display unit 240 based on information from tablet terminal 110 received by communication unit 210.

  For example, there are two types of information from the tablet terminal 110: display data and instruction data. In the case of display data, the control unit 230 transfers the display data to the virtual screen display unit 240 for display.

  The instruction data may be, for example, data indicating a scroll instruction or an enlargement / reduction instruction. In the case of these instruction data, the control unit 230 controls the virtual screen display unit 240 so that the virtual screen is displayed with the display content or display method as instructed by the instruction data.

  FIG. 3 is a diagram illustrating a configuration example of the real screen position detection unit 220. In this example, the real screen position detection unit 220 includes an imaging unit 222 and an image analysis unit 224.

  The imaging unit 222 is a camera that faces in the same direction as the line of sight of the user wearing the augmented reality display device 200. When the user wants to display both the real screen and the virtual screen, it is considered that the user wears the augmented reality display device 200 and the line of sight faces the tablet terminal 110. Therefore, the imaging unit 222 is also suitable for the tablet terminal 110, and the image captured by the imaging unit 222 includes the image of the tablet terminal 110.

  The image analysis unit 224 determines the position range of the real screen displayed on the touch panel 120 of the tablet terminal 110 by analyzing the image captured by the imaging unit 222.

  In the present embodiment, specifically, the image analysis unit 224 first detects the mark 140 from the image captured by the imaging unit 222 and identifies the position of the mark 140. Next, the image analysis unit 224 determines the position range of the real screen based on the positional relationship between the mark 140 and the touch panel 120 and the size of the display screen (real screen) of the touch panel 120 indicated by the pattern of the mark 140. calculate.

  The flow of processing in the multi-display system 100 will be described in time series with reference to specific examples.

  First, in order to configure the multi-display system 100, the user wears the augmented reality display device 200 and turns on the augmented reality display device 200. Then, the orientation of the head is adjusted so as to look at the tablet terminal 110.

  In response to the augmented reality display device 200 being turned on, the imaging unit 222 starts imaging. The image analysis unit 224 detects the mark 140 on the image acquired by the imaging unit 222. When the mark 140 is detected, the image analysis unit 224 identifies the position, determines the size of the real screen indicated by the pattern, and notifies the control unit 230 of it.

  When notified from the image analysis unit 224 that the mark 140 has been detected, the control unit 230 transmits a pairing request to the tablet terminal 110 via the communication unit 210.

  When the tablet terminal 110 receives a pairing request from the augmented reality display device 200, the tablet terminal 110 executes a sequence for permitting communication with the augmented reality display device 200 and sends a signal indicating the completion of pairing to the augmented reality display device 200. reply.

  Thereby, the tablet terminal 110 and the augmented reality display device 200 can communicate with each other.

  Here, it is assumed that the user inputs an instruction with one finger via the touch panel 120, starts browsing software installed on the tablet terminal 110, and starts browsing an English homepage. This home page is provided with a “display Japanese translation in parallel” button, and when the user presses this button, the Japanese translation data is also transmitted to the tablet terminal 110.

  When the user presses the button during browsing, the tablet terminal 110 transfers the Japanese translation data transmitted from the server device displaying the home page to the augmented reality display device 200.

  In the augmented reality display device 200, when the Japanese translation data is received from the tablet terminal 110, the control unit 230 causes the virtual screen display unit 240 to display the Japanese translation data from the top, for example.

  Thereafter, it is assumed that the user operates the touch panel 120 with one finger to scroll the English homepage. In response to this, the tablet terminal 110 transmits a similar scroll instruction to the augmented reality display device 200.

  The control unit 230 of the augmented reality display device 200 controls the virtual screen display unit 240 to scroll the display content of the virtual screen according to the scroll instruction.

  Next, since the user wants to enlarge a part of the Japanese translation, the touch panel 120 is instructed to enlarge with two fingers. The tablet terminal 110 determines that this instruction is an instruction to the augmented reality display device 200 and transmits the instruction to the augmented reality display device 200 without enlarging the real screen displayed by the touch panel 120.

  The control unit 230 of the real screen position detection unit 220 controls the virtual screen display unit 240 to perform an enlarged display on the virtual screen in response to an enlargement instruction from the tablet terminal 110.

  An example of the positional relationship between the real screen by the tablet terminal 110 and the virtual screen by the augmented reality display device 200 will be described.

  4 to 7 show examples in which virtual screens are displayed on the right side, left side, upper side, and lower side of the real screen, respectively. As long as the position ranges of the real screen and the virtual screen do not overlap, the positional relationship between the virtual screen and the real screen is not limited.

  For example, when the augmented reality display device 200 can directly accept an input from the user, it is preferable to make the positional relationship between the virtual screen and the real screen constant according to the input method.

  For example, the augmented reality display device 200 measures the distance from the touch panel 120 to the real screen and understands the movement of the finger on the touch panel 120 as an input to the tablet terminal 110, while moving the finger between the touch panel 120 and the user. Can be understood as an instruction to itself. In such a case, it is preferable to always display the virtual screen above the real screen so that there is no misunderstanding.

  Further, the augmented reality display device 200 may display a plurality of virtual screens on one real screen depending on the application. FIG. 8 shows an example in which one virtual screen is displayed on each of the upper and right sides of the real screen. Of course, the size of the virtual screen is not limited.

  Furthermore, according to the positional relationship between the tablet terminal 110 and the user, the positional relationship between the virtual screen and the real screen may be adjusted so that the user can easily see. Further, when an operation of shaking the tablet terminal 110 by the user is detected, the display contents may be changed, switched, or the positional relationship may be changed between the real screen and the virtual screen. For example, when the tablet terminal 110 detects an operation of shaking the tablet terminal 110 in the right direction, the virtual screen displayed in the left direction with respect to the real screen may be switched to be displayed in the right direction. The display content displayed on the screen may be displayed on the virtual screen, and the display content displayed on the virtual screen may be displayed on the real screen. For this purpose, the tablet terminal 110 may be provided with a sensor for detecting vibration and a sensor for detecting the direction of vibration.

  For example, the augmented reality display device 200 has means for detecting whether or not the user is facing directly in front of the tablet terminal 110, and when the user is facing directly in front of the tablet terminal 110, as shown in FIG. The virtual screen is displayed on the right side of the real screen. However, when the user moves the tablet terminal 110 to the right side of his / her own, displaying the virtual screen on the right side of the real screen as usual still makes it difficult for the user to see. In this case, as shown in FIG. 5, the augmented reality display device 200 preferably changes the display position of the virtual screen so that the virtual screen is on the left side of the real screen.

  According to the multi-display system 100 of the present embodiment, the augmented reality display device 200 includes the real screen position detection unit 220, and thus can determine the position range of the real screen, and the positions of the real screen and the virtual screen. The range can be displayed so as not to overlap. Therefore, it is not necessary to provide another display device that displays a real screen other than the touch panel 120, and a lightweight and space-saving multi-display system can be realized.

  Furthermore, even when the tablet terminal 110 moves, the augmented reality display device 200 can follow the display position of the real screen, which is highly convenient.

  Further, since communication is possible between the tablet terminal 110 and the augmented reality display device 200, the real screen and the virtual screen can be linked, and the augmented reality display device 200 can be operated via the tablet terminal 110. it can.

<Second Embodiment>
In the multi-display system 100 according to the first embodiment, the pairing of the tablet terminal 110 and the augmented reality display device 200 and the detection of the real screen position range by the augmented reality display device 200 are both marks that are AR markers. 140 is used. Separate marks may be used for the pairing mark and the mark for detecting the position range of the real screen. An example of this will be described using the second embodiment.

  FIG. 9 shows a multi-display system 300 according to the second embodiment. The multi-display system 300 includes a tablet terminal 310 and an augmented reality display device 400 that can communicate with each other.

  Similar to the tablet terminal 110, the tablet terminal 310 includes a touch panel 120 and a housing 130, and the housing 130 is provided with a mark 140 that is an AR marker for pairing. Furthermore, the tablet terminal 310 includes a real screen position presentation unit 350 that presents real screen position information that allows the augmented reality display device 400 to detect the position range of the display screen (real screen) on the touch panel 120.

  The augmented reality display device 400 uses the real screen position information presented by the real screen position presentation unit 350 in place of the mark 140 to detect the real screen position range, except that the augmented reality display in the multi-display system 100 is used. Similar to the apparatus 200.

  The real screen position presentation unit 350 of the tablet terminal 310 displays a mark 340 as real screen position information on the touch panel 120 after the pairing between the tablet terminal 310 and the augmented reality display device 400 is completed. The display position of the mark 340 on the touch panel 120 is constant, and the size of the touch panel 120 can be indicated by the pattern. The mark 340 is, for example, a QR code (registered trademark) (QR: Quick Response).

  When displaying the virtual screen, the augmented reality display device 400 detects the QR code 340 from the captured image and specifies the position range of the real screen. As described above, except for this point, the augmented reality display device 400 is the same as the augmented reality display device 200 in the multi-display system 100.

  The multi-display system 300 according to the second embodiment can obtain the same effects as the multi-display system 100 according to the first embodiment.

  The present invention has been described above based on the embodiment. The embodiment is an exemplification, and various modifications, increases / decreases, and combinations may be made to the above-described embodiments without departing from the gist of the present invention. It will be understood by those skilled in the art that modifications in which these changes, increases / decreases, and combinations are also within the scope of the present invention.

  For example, in the multi-display system 300, the QR code 340 may be used for pairing the tablet terminal 310 and the augmented reality display device 400 without using the mark 140.

  Further, the present invention is not limited to the above-described embodiment, and a method that does not use the mark 140 or the QR code 340 may be used. For example, when a predetermined mark or frame displayed on the real screen is detected, the tablet terminal may start pairing with the augmented reality display device.

In addition, a barcode may be used instead of the QR code 340.
Further, instead of the QR code 340, a frame along the outer edge of the display screen of the touch panel 120 may be displayed. In this case, the augmented reality display device side only has to detect the position of the frame and determine the range including the frame and surrounded by the frame as the position range of the real screen.

  Of course, instead of being displayed on the touch panel 120, the frame may be provided on a portion of the casing that defines the outer frame of the touch panel 120.

DESCRIPTION OF SYMBOLS 100 Multi-display system 110 Tablet terminal 120 Touch panel 130 Case 140 Mark 200 Augmented reality display device 210 Communication part 220 Real screen position detection part 222 Imaging part 224 Image analysis part 230 Control part 240 Virtual screen display part 300 Multi-display system 310 Tablet terminal 340 QR code 350 Real screen position presentation unit 400 Augmented reality display device

Claims (18)

An augmented reality display device that displays a virtual screen that is a display screen different from a real screen that is a display screen of an information terminal,
A real screen position detection unit for detecting a position range of the real screen;
Augmented reality display comprising: a control unit that controls the display position of the virtual screen so that the position range detected by the real screen position detection unit and the position range of the virtual screen do not overlap. apparatus. A communication unit for communicating with the information terminal;
The controller is
The augmented reality display device according to claim 1, further comprising: controlling display contents and / or a display method of the virtual screen according to information from the information terminal received by the communication unit. It is used by wearing on the user's head,
The real screen position detector
An imaging unit facing in the same direction as the line of sight of the user wearing the augmented reality display device;
The augmented reality display device according to claim 1, further comprising: an image analysis unit that determines a position range of the real screen by analyzing an image captured by the imaging unit. The image analysis unit further analyzes the complexity of a portion other than the real screen in the image captured by the imaging unit,
4. The augmented reality according to claim 3, wherein the control unit controls the display position of the virtual screen so as to be displayed in a position range having the smallest complexity obtained by the image analysis unit. 5. Display device. The image analysis unit
A predetermined mark displayed at a predetermined position of the real screen or provided at a predetermined position of a casing unit that defines an outer frame of the real screen is detected from an image captured by the imaging unit. The augmented reality display device according to claim 3, wherein a position range of the real screen is determined.   The augmented reality display device according to claim 5, wherein the predetermined mark includes information that can indicate a size of the real screen.   The augmented reality display device according to claim 6, wherein the predetermined mark is a bar code or a QR code (registered trademark) (QR: Quick Response). An information terminal comprising a display unit;
A multi-display system having an augmented reality display device that displays a virtual screen that is a display screen different from a real screen that is a display screen of the display unit of the information terminal,
The augmented reality display device includes:
A real screen position detection unit for detecting a position range of the real screen;
A multi-display system comprising: a control unit that controls a display position of the virtual screen so that the position range detected by the real screen position detection unit and the position range of the virtual screen do not overlap. . The information terminal and the augmented reality display device can communicate with each other,
The information terminal
It has an input part,
When the information input by the input unit is information to the augmented reality display device, the information is transmitted to the augmented reality display device,
The control unit of the augmented reality display device includes:
The multi-display system according to claim 8, wherein the display content of the virtual screen is controlled in accordance with information received from the information terminal. A display unit;
Real screen position information capable of detecting a position range of the real screen on an augmented reality display device that displays a virtual screen that is a display screen of an augmented reality different from the real screen that is a display screen by the display unit. An information terminal comprising a real screen position presentation unit for presentation. An input section;
A communication unit that communicates with the augmented reality display device;
11. The information according to claim 10, wherein when the information input by the input unit is information to the augmented reality display device, the information is transmitted to the augmented reality display device via the communication unit. Information terminal.   The information terminal according to claim 10 or 11, wherein the real screen position presentation unit displays a predetermined mark at a predetermined position on the real screen as the real screen position information.   The augmented reality display device according to claim 12, wherein the predetermined mark includes information that can indicate a size of the real screen.   14. The augmented reality display device according to claim 13, wherein the predetermined mark is a bar code or a QR code (registered trademark) (QR: Quick Response). When displaying a virtual screen that is different from the real screen that is the display screen of the information terminal,
Detect the real screen position range,
A program causing a computer to execute a process of controlling a display position of the virtual screen so that the detected position range and the position range of the virtual screen do not overlap.   The program according to claim 15, further causing a computer to execute a process of controlling display contents of the virtual screen in accordance with information received from the information terminal. Display on the computer display,
A process of presenting real screen position information capable of detecting a position range of the real screen to an augmented reality display device that displays a virtual screen that is a display screen different from the real screen that is a display screen by the display unit. A program that is executed by the computer.   Furthermore, when the information input from the input unit of the computer is information to the augmented reality display device, the computer is caused to execute a process of transmitting the information to the augmented reality display device. Item 18. The program according to Item 17.

Images