JP2021043752A - Information display device, information display method, and information display system - Google Patents

Abstract

To determine a display pattern of a virtual object based on a combination of a plurality of different AR markers simultaneously detected, and add an AR content based on a point of focus of a user on the virtual object.SOLUTION: An information display device includes: a data transmission and reception part which acquires a real video including a plurality of AR markers; a data analysis part which determines tag information, the type and number of the AR markers in the real video and determines a first display pattern for a first virtual object based on the type and number of AR markers; an object creation part which acquires, from a predetermined database, first additional information as the first virtual object based on the tag information of the AR markers and creates the first virtual object based on the first additional information; and a diffusion reality management part which creates, in accordance with the determined first display pattern, a first diffusion reality video obtained by superposing the created first virtual object on the real video and outputs the first diffusion reality video.SELECTED DRAWING: Figure 2

Description

The present invention relates to an information display device, an information display method, and an information display system.

In recent years, an image superimposition technique called augmented reality (AR) has been developed in which additional information is superposed on a real space and presented to a user. The superimposed additional information can be visualized by displaying various types of display objects such as text, icons, or animations in front of the user.

AR technology is widely used in fields such as entertainment, marketing, tourism, and industry, and various products and services using AR technology have been proposed. For example, as an example of using AR technology, Japanese Patent Application Laid-Open No. 2014-127148 (Patent Document 1) states that "a trigger recognition unit that acquires an image and recognizes predetermined trigger information contained in the image". And, the content acquisition unit that acquires the content including the augmented reality information corresponding to the situation at the time when the captured image is captured or the acquisition status of the past content, and the predetermined trigger information recognized by the trigger recognition unit. And, an information processing device provided with.

Japanese Unexamined Patent Publication No. 2014-127148

The above-mentioned Patent Document 1 describes a technique of detecting a plurality of different AR markers in order and displaying different AR contents depending on the order in which the AR markers are detected. According to the invention of Patent Document 1, it is possible to provide a new AR content different from the AR content detected in the past by giving the plurality of AR markers positional or temporal interlocking.

However, in the invention of Patent Document 1, it is not considered to determine the display position and the display format of the virtual object by the combination of a plurality of different AR markers detected at the same time, and the user pays attention to the specific virtual object. Further addition of AR content based on this has not been considered.

Therefore, the present invention determines the display position and display format of the virtual object to be displayed in the real space based on the types, numbers, positions, etc. of a plurality of different AR markers detected at the same time, and determines the display position and display format of the virtual object to be displayed in the real space. An object of the present invention is to provide an information display device for adding other virtual objects based on a point of interest.

In order to solve the above problems, one of the typical inventions is an information display device, which is a data transmission / reception unit that acquires a real image including a plurality of AR markers from a camera, and an AR marker in the real image. A data analysis unit that determines the tag information, type and number, and determines the first display pattern for the first virtual object based on the type and number of AR markers, and the first based on the tag information of the AR marker. According to the object creation unit that acquires the first additional information that becomes one virtual object from the virtual object information management database and creates the first virtual object based on the first additional information, and the determined first display pattern. Includes an augmented reality management unit that creates and outputs a first augmented reality image in which the created first virtual object is superimposed on the actual image.

According to the present invention, the display position and display format of a virtual object to be displayed in the real space are determined based on the types, numbers, positions, etc. of a plurality of different AR markers detected at the same time, and the user with respect to a specific virtual object It is possible to provide an information display device that adds other virtual objects based on the point of interest.
Issues, configurations and effects other than those described above will be clarified by the description of the following embodiments.

It is a block diagram of the computer system for carrying out embodiment of this invention. It is a figure which shows the system configuration which concerns on embodiment of this invention. It is a figure which shows an example of the terminal apparatus which concerns on embodiment of this invention. It is a figure which shows an example of the virtual object which is displayed when the AR marker which concerns on embodiment of this invention is recognized. It is a figure which shows another example of the virtual object which is displayed when the AR marker which concerns on embodiment of this invention is recognized. It is a figure which shows another example of the virtual object which is displayed when the AR marker which concerns on embodiment of this invention is recognized. It is a figure which shows another example of the virtual object which is displayed when the AR marker which concerns on embodiment of this invention is recognized. It is a figure which shows an example of the AR marker / virtual object management table which concerns on embodiment of this invention. It is a figure which shows an example of the 1st additional information management table which concerns on embodiment of this invention. It is a figure which shows an example of the hidden information management table which concerns on embodiment of this invention. It is a figure which shows an example of the 2nd additional information management table which concerns on embodiment of this invention. It is a flowchart which shows the flow of the information display processing which concerns on embodiment of this invention. It is a flowchart which shows the flow of attention point analysis processing which concerns on embodiment of this invention. It is a flowchart which shows the flow of the process which determines the display pattern of the virtual object which concerns on embodiment of this invention. It is a figure which shows an example of the coordinates of the four corners of the AR marker which concerns on embodiment of this invention. It is a figure which shows an example of the base coordinates of the virtual object which concerns on embodiment of this invention. It is a figure which shows an example of the display pattern of the virtual object which concerns on embodiment of this invention. It is a figure which shows another example of the display pattern of the virtual object which concerns on embodiment of this invention. It is a figure which shows one aspect of the specific example which applied the information display means which concerns on embodiment of this invention to the maintenance work of a server. It is a figure which shows another aspect of the specific example which applied the information display means which concerns on embodiment of this invention to the maintenance work of a server. It is a figure which shows another aspect of the specific example which applied the information display means which concerns on embodiment of this invention to the maintenance work of a server. It is a figure which shows an example which acquires necessary data from a database when the information display means which concerns on embodiment of this invention is applied to maintenance work of a server. It is a figure which shows another example which acquires necessary data from a database when the information display means which concerns on embodiment of this invention is applied to maintenance work of a server. It is a figure which shows another example which acquires necessary data from a database when the information display means which concerns on embodiment of this invention is applied to maintenance work of a server. It is a figure which shows an example of the operation which fixes the virtual object which concerns on embodiment of this invention.

Hereinafter, embodiments of the present invention will be described with reference to the drawings. The present invention is not limited to this embodiment. Further, in the description of the drawings, the same parts are indicated by the same reference numerals.
(Hardware configuration)

First, with reference to FIG. 1, a computer system 300 for carrying out the embodiment of the present disclosure will be described. The mechanisms and devices of the various embodiments disclosed herein may be applied to any suitable computing system. The main components of the computer system 300 include one or more processors 302, memory 304, terminal interface 312, storage interface 314, I / O (input / output) device interface 316, and network interface 318. These components may be interconnected via memory bus 306, I / O bus 308, bus interface unit 309, and I / O bus interface unit 310.

The computer system 300 may include one or more general purpose programmable central processing units (CPUs) 302A and 302B collectively referred to as processors 302. In one embodiment, the computer system 300 may include a plurality of processors, and in another embodiment, the computer system 300 may be a single CPU system. Each processor 302 may execute an instruction stored in memory 304 and include an onboard cache.

In certain embodiments, the memory 304 may include a random access semiconductor memory, a storage device, or a storage medium (either volatile or non-volatile) for storing data and programs. The memory 304 may store all or part of the programs, modules, and data structures that perform the functions described herein. For example, the memory 304 may store the information display application 350. In certain embodiments, the information display application 350 may include an instruction or description that executes a function described below on the processor 302.

In certain embodiments, the information display application 350 replaces or in addition to a processor-based system, a semiconductor device, chip, logic gate, circuit, circuit card, and / or other physical hardware device. It may be implemented in hardware via. In certain embodiments, the information display application 350 may include data other than instructions or descriptions. In certain embodiments, a camera, sensor, or other data input device (not shown) may be provided to communicate directly with the bus interface unit 309, processor 302, or other hardware of the computer system 300. ..

The computer system 300 may include a processor 302, a memory 304, a display system 324, and a bus interface unit 309 that communicates between the I / O bus interface unit 310. The I / O bus interface unit 310 may be connected to an I / O bus 308 for transferring data to and from various I / O units. The I / O bus interface unit 310, via the I / O bus 308, is a plurality of I / O interface units 312, 314, 316, also known as an I / O processor (IOP) or I / O adapter (IOA). And 318 may be communicated.

The display system 324 may include a display controller, display memory, or both. The display controller can provide video, audio, or both data to the display device 326. The computer system 300 may also include devices such as one or more sensors configured to collect the data and provide the data to the processor 302.

For example, the computer system 300 includes a biometric sensor that collects heart rate data, stress level data, etc., an environment sensor that collects humidity data, temperature data, pressure data, etc., and a motion sensor that collects acceleration data, exercise data, etc. May include. Other types of sensors can also be used. The display system 324 may be connected to a stand-alone display screen, a display device 326 such as a television, tablet, or portable device.

The I / O interface unit has a function of communicating with various storages or I / O devices. For example, the terminal interface unit 312 may be a user output device such as a video display device or a speaker TV, or a user input device such as a keyboard, mouse, keypad, touchpad, trackball, button, light pen, or other pointing device. Such a user I / O device 320 can be attached. The user inputs input data and instructions to the user I / O device 320 and the computer system 300 by operating the user input device using the user interface, and receives the output data from the computer system 300. May be good. The user interface may be displayed on the display device via the user I / O device 320, reproduced by the speaker, or printed via the printer, for example.

The storage interface 314 is an array of disk drives or other storage device configured to appear as a single disk drive, although one or more disk drives or a direct access storage device 322 (usually a magnetic disk drive storage device). It may be). In certain embodiments, the storage device 322 may be implemented as any secondary storage device. The contents of the memory 304 are stored in the storage device 322 and may be read out from the storage device 322 as needed. The I / O device interface 316 may provide an interface to other I / O devices such as printers and fax machines. The network interface 318 may provide a communication path so that the computer system 300 and other devices can communicate with each other. This communication path may be, for example, network 330.

In certain embodiments, the computer system 300 is a device that receives a request from another computer system (client) that does not have a direct user interface, such as a multi-user mainframe computer system, a single user system, or a server computer. There may be. In other embodiments, the computer system 300 may be a desktop computer, a portable computer, a laptop computer, a tablet computer, a pocket computer, a telephone, a smartphone, or any other suitable electronic device.

Next, the configuration of the information display system according to the embodiment of the present invention will be described with reference to FIG.

FIG. 2 is a diagram showing a configuration of an information display system 200 according to an embodiment of the present invention. As shown in FIG. 2, the information display system 200 mainly includes data acquisition devices 210A and 210B, a control device 220, and a terminal device 240. The data acquisition devices 210A and 210B, the control device 220, and the terminal device 240 may be connected to each other via a communication network (not shown) such as the Internet.

The data acquisition devices 210A and 210B are functional units for collecting information on the surrounding environment and transmitting the collected information to the control device 220. The data acquisition devices 210A and 210B may be installed in a place where the target data can be easily acquired. For example, when data on the temperature of the server is acquired, each of the data acquisition devices 210A and 210B may be installed inside the server.
Note that FIG. 2 shows a configuration including two data acquisition devices 210A and 210B as an example, but the present invention is not limited to this, and the number of data acquisition devices may be one. It may be 3 or more.

As shown in FIG. 2, the data acquisition devices 210A and 210B include sensor units 212A and 212B and data transmission / reception units 214A, respectively. The sensor units 212A and 212B apply some scientific principle to the mechanical, electromagnetic, thermal, acoustic, and chemical properties of natural phenomena and artificial objects, or the spatial and temporal information indicated by them, by applying some scientific principle to humans. It is a device that replaces the signal of another medium that is easy for the machine to handle.
The sensor units 212A and 212B are, for example, a digital sensor that collects CPU and memory usage data, a biometric sensor that collects heart rate data, stress level data, etc., humidity data, temperature data, water pressure data, pressure data, illuminance data, etc. An environmental sensor that collects data, a motion sensor that collects acceleration data, motion data, and the like may be included, but the sensor unit according to the present invention is not limited to this.

The data transmission / reception units 214A and 214B are functional units for transmitting the information acquired from the sensor units 212A and 212B to the control device 220. The data transmission / reception units 214A and 214B may transmit the information acquired by the sensor units 212A and 212B to the control device 220 via a communication network (not shown) such as the Internet. The data transmission / reception units 214A and 214B are not limited to data transmission, and may receive a command from the control device 220 and control the sensor units 212A and 212B based on the received instruction.

The control device 220 is a device for managing information received from the data acquisition devices 210A and 210B, and generating and providing AR contents (virtual objects, etc.) output to the terminal device. The control device 220 is configured as a remote server device that is installed at a location away from the data acquisition device and the terminal device, and is connected to the data acquisition device and the terminal device via a communication network (not shown) such as the Internet. You may.

As shown in FIG. 2, the control device 220 includes a data transmission / reception unit 222, a data analysis unit 224, a storage unit 226, an object creation unit 228, an augmented reality management unit 230, and a line-of-sight analysis unit 232.

The data transmission / reception unit 222 is a functional unit for performing information communication with the data acquisition device described above and the terminal device 240 described later. The data transmission / reception unit 222 can, for example, transmit an instruction for specifying the data acquisition frequency to the data acquisition device, or distribute the created AR content to the terminal device.

The data analysis unit 224 is a functional unit for analyzing various data received from the data acquisition device and the terminal device. The data analysis unit 224 may specify, for example, an AR marker appearing in a real space image received from a terminal device described later, or analyze sensor data received from a data acquisition device.
The AR marker here means an image having a fixed pattern, which serves as a sign functioning as a trigger for displaying AR contents. The AR marker may be, for example, a QR code (registered trademark) or another predetermined image. Further, in general, each AR marker is associated with a tag that uniquely identifies the AR marker and defines what kind of additional information should be displayed as AR content. After that, when the AR marker is detected, additional information to be the AR content specified in the tag of the AR marker is read from a predetermined storage medium (for example, the storage unit 226 described later) and displayed.

The storage unit 226 is a storage device for storing various data used by each functional unit of the information display system 200. The storage unit 226 may be any storage medium such as a flash memory or a hard disk drive. The storage unit 226 may store, for example, an AR marker / virtual object management table, a first additional information management table, a hidden information management table, and a second additional information management table, which will be described later.

The object creation unit 228 is a functional unit for generating a virtual object to be superimposed on the image in the real space. The object creation unit 228 may acquire additional information corresponding to the tag of the AR marker specified by the data analysis unit from the storage unit 226 and generate a virtual object according to the information. For example, when the AR content associated with the tag of the AR marker is "temperature display of server A", the object creation unit 228 acquires the temperature information of server A from the storage unit 226 described above, and the information is obtained. You may generate a virtual object (a balloon showing a temperature value, a graph, an animation, etc.) for displaying.
As described above, when creating a virtual object, the object creation unit 228 determines a display pattern that specifies the display position and arrangement format of the virtual object based on the number, type, and position of the detected AR markers. You may. The process of determining the display pattern of the virtual object will be described later.

The augmented reality management unit 230 is a functional unit that creates an augmented reality image in which a virtual object generated by the object creation unit 228 is superimposed on an image in real space, and provides the augmented reality image to a terminal device. The augmented reality management unit 230 may create an augmented reality image according to the display pattern of the virtual object determined based on the number, type, and position of the detected AR markers.

The line-of-sight analysis unit 232 measures and tracks the location of the user's viewpoint (where the user is looking) and the movement of the eyeball with respect to the head in the image displayed on the image display unit 244 of the terminal device 240, which will be described later. It is a functional part for. The line-of-sight analysis unit 232 can specify the coordinates of the point (hereinafter, the point of interest) that the user is paying attention to in the image by measuring the position of the user's viewpoint and the movement of the eyeball. As will be described later, in the present invention, the content, the outer material position, and the display format of the virtual object superimposed on the real space may be displayed based on the information of the coordinates of the point of interest.
The line-of-sight analysis unit 232 here is, for example, a means for attaching a special contact lens to the eye and measuring its movement, a means for measuring using an optical technique, or a means for arranging an electrode around the eye and eyeball. Any means such as a means for measuring an electric potential generated from a muscle or the like for moving the lens may be used.

The terminal device 240 is a device for providing the user with an augmented reality image in which a virtual object is superimposed on an image in the real space. The terminal device 240 may be a computing device such as a head-mounted display, a smartphone, or a tablet, and is not particularly limited as long as it is a device capable of displaying an augmented reality image.
As shown in FIG. 2, the terminal device 240 may include a data transmission / reception unit 241, a camera unit 242, and a video display unit 244.

The data transmission / reception unit 241 is a functional unit for performing information communication with the control device 222 described above. The data transmission / reception unit 241 may transmit, for example, an image of the real space captured by the camera unit 242, which will be described later, to the control device 222, or may receive the augmented reality image generated by the control device 222.

The camera unit 242 is a functional unit for capturing an image in the real space. The camera unit 242 may continuously capture an image in the real space and dynamically transmit the captured image to the control device 222 in real time.

The image display unit 244 is a display for displaying an augmented reality image in which a virtual object is superimposed on an image in the real space to the user.

Next, an example of the terminal device according to the embodiment of the present invention will be described with reference to FIG.

FIG. 3 is a diagram showing an example of the terminal device 240 according to the embodiment of the present invention.

As described above, the terminal device 240 according to the embodiment of the present invention that provides the user with an augmented reality image in which a virtual object is superimposed on an image in the real space may be implemented as, for example, a mobile terminal 240A or a head-mounted display 240B. Good.

The mobile terminal 240 is a computing device held by the user. The mobile terminal 240 may be, for example, a smartphone, a tablet, or the like. By configuring the terminal device 240 as a mobile terminal 240A, the user can view the augmented reality image on a small and lightweight device.

The head-mounted display 240B is a display device worn on the user's head. The head-mounted display 240B may be, for example, a monocular type that covers only one eye or a binocular type that covers both eyes, and the shape and display method may be appropriately selected. By configuring the terminal device 240 as the head-mounted display 240B, the user's hands are freed, so that the work can be performed more freely.

The form of the terminal device 240 described above may be appropriately selected based on the purpose and environment in which it is used. For example, when the information display means according to the embodiment of the present invention is used to display information on meals and events at a tourist spot, the terminal device 240 is implemented as a small and lightweight mobile terminal 240A. This may be preferable from the viewpoint of user convenience. On the other hand, when the information display means according to the embodiment of the present invention is used for maintenance work of piping, it is a viewpoint of work efficiency of an operator that the terminal device 240 is mounted as a head-mounted display 240B. May be preferable.

Next, with reference to FIGS. 4 to 7, an example of an augmented reality image displayed when the AR marker is recognized by using the information display means according to the embodiment of the present invention will be described.

4 to 7 are diagrams showing an example of an augmented reality image displayed when the AR marker is recognized by using the information display means according to the embodiment of the present invention. As described above, the information display means according to the embodiment of the present invention determines the display pattern of the virtual object to be displayed in the real space based on the types, numbers, positions, etc. of the plurality of different AR markers detected at the same time. The display pattern referred to here is information that specifies the format of the virtual object (for example, text, graph, animation, etc.) and the position (for example, plane coordinates) for displaying the virtual object in the base real image.
Hereinafter, an example of an image captured by the terminal device and a virtual object displayed according to the AR marker appearing in the image will be described.

FIG. 4 is a diagram showing an input image 420 including one AR marker 425 and an augmented reality image 440 including a virtual object 445 created in response to the AR marker 425. When the input image 420 shown in FIG. 4 is captured and the AR marker 425 is recognized by the data analysis unit described above, the augmented reality image 440 including the virtual object 445 is created by the object creation unit and the augmented reality management unit, and the terminal device. It is displayed on the video display section of.

As described above, the information display means according to the embodiment of the present invention has a display pattern (that is, display) of a virtual object to be displayed in the real space based on the types, numbers, positions, etc. of a plurality of different AR markers detected at the same time. Position and display format) are determined. For example, when only one AR marker is recognized as shown in FIG. 4, a virtual object showing the information associated with the AR marker in detail can be displayed. Therefore, in the information display means according to the embodiment of the present invention, for example, when the number of recognized AR markers is only one, a virtual object (for example, virtual object 445) that shows the information associated with the AR markers in detail. ) May be displayed.
As a result, the user can see the detailed information of the AR content corresponding to the recognized AR marker.

FIG. 5 is a diagram showing an input image 520 including three AR markers and an augmented reality image 540 created according to the three AR markers. When the input image 520 shown in FIG. 5 is captured and the three AR markers are recognized by the above-mentioned data analysis unit, an augmented reality image 540 including three virtual objects is created by the object creation unit and the augmented reality management unit. , Displayed on the video display of the terminal device.

As shown in FIG. 5, for example, when three AR markers are recognized, a virtual object created from the information associated with each recognized AR marker may be displayed. In this case, the amount of information of the virtual object may be adjusted depending on the area and resolution of the screen of the image display unit of the terminal device. For example, if the information associated with the three AR markers is shown in detail and the screen is crowded and difficult to see, a virtual object that more concisely shows the information associated with the AR markers can be created and displayed.

FIG. 6 is a diagram showing an input video 620 including six AR markers and an augmented reality video 640 created according to the six AR markers. When the input video 620 shown in FIG. 6 is captured and the six AR markers are recognized by the above-mentioned data analysis unit, the augmented reality video 640 including a virtual object that collectively displays additional information corresponding to the six AR markers. Is created by the object creation unit and the augmented reality management unit, and is displayed on the video display unit of the terminal device.

When a large number of AR markers are recognized at the same time, if the information associated with each AR marker is created and displayed as an individual virtual object, the display of the video display unit will be crowded with the virtual object, making the screen difficult to see. May become. Therefore, in the information display means according to the embodiment of the present invention, for example, when the number of recognized AR markers exceeds a predetermined number (for example, four or more), the information associated with each AR marker is displayed as an individual virtual object. Instead of displaying as, the information associated with each AR marker may be displayed as one integrated virtual object (see Augmented Reality Video 640).
This makes it easier to see the AR content to be displayed, and the screen area of the video display unit of the terminal device can be used more efficiently.

FIG. 7 is a diagram showing an example of changing a virtual object displayed according to a user's point of interest. As described above, the information display means according to the embodiment of the present invention can create and display a virtual object based on a user's point of interest. For example, as shown in FIG. 7, when the line-of-sight analysis unit of the terminal device (for example, the line-of-sight analysis unit 232 shown in FIG. 2) determines that the user's point of interest is directed to the virtual object 710, the virtual object The virtual object 720 created from the hidden information (described later) embedded in the object 710 may be displayed. Then, when the line-of-sight analysis unit determines that the user's point of interest has moved from the virtual object 710 to the virtual object 730, the virtual object 740 created from the hidden information embedded in the virtual object 730 is displayed. May be good.
As a result, the user can display the AR content of interest only by moving his / her eyes without using his / her hands.

Next, the information management table used in the information display processing according to the embodiment of the present invention will be described with reference to FIGS. 8 to 11.

FIG. 8 is a diagram showing an example of the AR marker / virtual object management table 800 according to the embodiment of the present invention. The AR marker / virtual object management table 800 shown in FIG. 8 is a table showing the tag 820 of the AR marker corresponding to the additional information, the line-of-sight attention state 830, and the virtual object format 840 for each additional information 810 that becomes a virtual object. is there. When the tag 820 of a specific AR marker is detected in the captured real image, the object creation unit described above converts the additional information 810 and the format 840 of the virtual object corresponding to the detected AR marker tag 820 into the AR marker / virtual. It can be specified based on the object management table 800.
The virtual object format 840 shown in FIG. 8 specifies in which format the virtual object should be displayed, and each of "A", "B", "C", and "D" is, for example, text. A label that specifies an arbitrary format such as a graph, an image, or a moving image.

FIG. 9 is a diagram showing an example of the first additional information management table 900 according to the embodiment of the present invention. The first additional information management table 900 shown in FIG. 9 is a table that stores additional information that becomes a virtual object (for example, a first virtual object created based on an AR marker). When a tag of a specific AR marker is detected in the captured real image and additional information corresponding to the tag of the AR marker is specified (for example, based on the AR marker / virtual object management table 800 shown in FIG. 8), Next, the object creation unit described above acquires the specified additional information from the first additional information management table 900. After that, the object creation unit may create a virtual object according to the additional information acquired from the first additional information management table 900.
In the first additional information management table 900 shown in FIG. 9, information such as the CPU temperature, temperature, humidity, and event of the server is exemplified as additional information of the virtual object, but the present invention is not limited thereto. The additional information of the virtual object may be appropriately configured according to the purpose and environment in which the information display means according to the present invention is used.

FIG. 10 is a diagram showing an example of the hidden information management table 1000 according to the embodiment of the present invention. As will be described later, the virtual object superimposed on the augmented reality image is divided into a plurality of areas, and different hidden information is embedded in each area. This non-display information is information that is not displayed in the video and is detected by the line-of-sight analysis unit. When the line-of-sight analysis unit detects that the user's point of interest is directed to the area of the virtual object, a table in which additional information specified by the hidden information embedded in the area is specified (for example, described later). A new virtual object (for example, a second virtual object) is created according to the additional information obtained from the second additional information management table 1100) according to the display pattern specified by the hidden information, and augmented reality. Add to the video.
For example, when the line-of-sight analysis unit determines that the user's point of interest is directed to the area of "CPU temperature", the line-of-sight analysis unit determines the CPU usage rate for each processor according to the conditions of the area, for example, the second additional information described later. A virtual object acquired from the management table 1100 and showing the acquired information in a tabular format may be created and added to the augmented reality image.

FIG. 11 is a diagram showing an example of the second additional information management table 1100 according to the embodiment of the present invention. FIG. 11 is a table that stores additional information specified by hidden information embedded in the area of the user's attention when the user's attention is detected. The second additional information shown in FIG. 11 is a new virtual object (for example, a second, third, fourth, etc. virtual object) when the first virtual object being displayed is the target of the user's attention. Information for creating an object). Further, as shown in FIG. 11, the second additional information may include information such as CPU usage rate and event information for each processor of the server.
In the second additional information management table 1100 shown in FIG. 11, information such as the CPU usage rate of the server and events is exemplified as additional information, but the present invention is not limited thereto, and the additional information is the present invention. It may be appropriately configured according to the purpose and environment in which the information display means according to the above is used.

Next, with reference to FIG. 12, the processing flow of information display according to the embodiment of the present invention will be described.

FIG. 12 is a flowchart showing a flow of information display processing 1200 according to the embodiment of the present invention.

First, in step S1201, the data transmission / reception unit of the control device (for example, the data transmission / reception unit 222 of the control device 220 shown in FIG. 2) receives a real image from the terminal device (for example, the terminal device 240 shown in FIG. 2). .. This real image may be, for example, an image taken by a camera unit of a terminal device (for example, 242 shown in FIG. 2).
It should be noted that this video may be a live distribution continuously shot in real time.

Next, in step S1202, the data analysis unit of the control device (for example, the data analysis unit 224 of the control device 220 shown in FIG. 2) analyzes the type and number of AR markers in the video. Here, the data analysis unit detects the AR marker embedded in the received video by using, for example, an existing image processing means, determines the type of the detected AR marker, and calculates the number for each type. May be good.

Next, in step S1203, the object creation unit acquires the display information of the virtual object corresponding to the AR marker from the storage unit based on the type and number of the detected AR markers. Here, the object creation unit identifies additional information corresponding to the detected AR marker by using the above-mentioned AR marker / virtual object management table (for example, the AR marker / virtual object management table 800 shown in FIG. 8). , The additional information may be acquired from the first additional information management table (for example, the first additional information management table 900 shown in FIG. 9).

Next, in step S1204, the object creation unit determines the display pattern of the virtual object based on the type and number of detected AR markers. Here, the object creation unit may determine the virtual object format corresponding to the detected AR marker by using the AR marker / virtual object management table described above. For example, to explain an example with reference to FIG. 8, when the additional information "1" is determined as a result of the processing of FIG. 14 described later, the object creation unit determines the virtual object format of "C". ..
The details of steps S1203 to S1204 will be described later with reference to FIG.

Next, in step S1205, the object creation unit creates a virtual object based on the additional information of the virtual object acquired from the storage unit in step S1203 and the display pattern determined in step S1204. For example, when the additional information acquired from the storage unit indicates "server 1 CPU temperature: 60 degrees" and the display pattern of the virtual object indicates "text format", the object creation unit indicates "server 1 CPU temperature: 60 degrees". You may create a virtual object that shows "" in a tabular format.

Next, in step S1206, the object creation unit synthesizes the virtual object created in step S1205 and the real image received in step S1201 to generate an augmented reality image in which the virtual object is superimposed on the real image.

Next, in step S1207, the augmented reality image generated in step S1206 is transmitted to the terminal device. The terminal device that has received the augmented reality image displays the augmented reality image on the image display unit.

Next, in step S1208, it is determined whether or not the user has a point of interest. If the user's attention is directed to the virtual object, this process proceeds to step S1208. If the user's attention is not directed to the virtual object, this process ends.
As a result, a desired virtual object can be created according to a display pattern determined based on the types, numbers, positions, etc. of a plurality of different AR markers detected at the same time.

Next, with reference to FIG. 13, the processing flow of the point of interest analysis according to the embodiment of the present invention will be described.

FIG. 13 is a flowchart showing a flow of the attention point analysis process 1300 according to the embodiment of the present invention.

First, in step S1301, the line-of-sight analysis unit of the control device (for example, the line-of-sight analysis unit 232 of the control device 220 shown in FIG. 2) has the coordinates of the user's line of sight from the terminal device (for example, the terminal device 240 shown in FIG. 2). By receiving the information indicating the above and analyzing the information, the coordinates of the point of interest in the augmented reality image are calculated. The coordinates here may be, for example, general plane coordinates.

Next, in step S1302, the line-of-sight analysis unit determines whether or not the coordinates of the point of interest calculated in step S1301 are included in the area of the virtual object superimposed on the augmented reality image. If the coordinates of the user's point of interest are included in the area of the virtual object, this process proceeds to step S1303. If the coordinates of the user's point of interest are not included in the virtual object area, this process ends.

Next, in step S1303, the line-of-sight analysis unit acquires the hidden information embedded in the area including the point of interest. As described above, this hidden information indicates additional information that becomes an additional virtual object (for example, a second, third, fourth, etc. virtual object) and a condition for designating a display pattern of the virtual object. Is.

Next, in step S1304, the line-of-sight analysis unit determines the display pattern of the additional virtual object based on the acquired non-display information. This display pattern may be determined according to the conditions specified by the acquired non-display information. For example, when the hidden information specifies that the additional information is displayed in the "graph format", the line-of-sight analysis unit may determine the display pattern of the additional virtual object as the "graph format".

Next, in step S1305, the object creation unit acquires additional information to be an additional virtual object from the storage unit. Specifically, the object creation unit may acquire the information specified by the hidden information from the additional information management table (for example, the second additional information management table 1100 shown in FIG. 11).

Next, in step S1306, the object creation unit creates an additional virtual object (for example, a second virtual object) according to the additional information acquired from the additional information management table. As mentioned above, this additional virtual object may provide detailed information about the area that contains the user's attention. For example, if the user's attention is included in the area showing the CPU usage of the server, a graph showing the time series of the CPU usage of the server may be created as an additional virtual object.

Next, in step S1307, the object creation unit synthesizes the additional virtual object and the augmented reality image (that is, the augmented reality image on which the first virtual object is already superimposed). Specifically, the object creation unit may superimpose the additional virtual object created in step S1306 on a location that matches the coordinates of the user's point of interest.

Next, a process of determining the display pattern of the virtual object according to the embodiment of the present invention will be described with reference to FIG.

FIG. 14 is a flowchart showing a flow of processing 1400 for determining a display pattern of a virtual object according to an embodiment of the present invention. The process 1400 shown in FIG. 14 corresponds to steps S1203 to S1204 in the process 1200 shown in FIG. 12, and shows the details of the process of determining the display pattern of the virtual object.

First, in step S1401, the object creation unit sets an unanalyzed flag for all the AR markers identified in the real image. The unanalyzed flag is a label indicating that the virtual object including the additional information corresponding to the AR marker has not been created. For example, when the AR marker 1, the AR marker 2, the AR marker 3, and the AR marker 4 are specified, the object creation unit assigns the AR marker 1, the AR marker 2, the AR marker 3, and the AR marker 4 to each of them. On the other hand, set the unanalyzed flag.
Note that this unanalyzed flag may be stored, for example, in the AR marker / virtual object management table described above (not shown).

Next, in step S1402, the object creation unit selects one AR marker from the AR markers for which the unanalyzed flag is set. The AR marker selected here may be any AR marker. For example, when the unanalyzed flag is set for the AR marker 1, the AR marker 2, the AR marker 3, and the AR marker 4, the object creation unit may select the AR marker 1.

Next, in step S1403, the object creation unit refers to the AR marker / virtual object management table described above and acquires additional information corresponding to the selected AR marker. For example, when the AR marker 1 is selected, the object creation unit acquires additional information 1, 7, and 9.

Next, in step S1404, the object creation unit sets an unanalyzed flag for each of the acquired additional information. As described above, this unanalyzed flag is a label indicating that the virtual object containing the additional information has not been created, and may be stored in the AR marker / virtual object management table (not shown).

Next, in step S1405, the object creation unit selects one of the additional information for which the unanalyzed flag is set. The additional information selected here may be arbitrary. For example, when the unanalyzed flag is set for the additional information 1, 7, and 9, the object creation unit may select the additional information 1.

Next, in step S1406, the object creation unit refers to the AR marker / virtual object management table, compares the AR marker corresponding to the selected additional information with the unanalyzed AR marker, and corresponds to the selected additional information. It is determined whether the AR marker to be used is included in the unanalyzed AR marker (that is, it is confirmed whether or not the AR marker is unanalyzed). If the AR marker corresponding to the selected additional information is included in the unanalyzed AR marker, the present process proceeds to step S1407, and if not, the process proceeds to step S1408.
For example, when the selected additional information is the additional information 1 and the unanalyzed AR markers are the AR markers 1, 2, 3, and 4, the AR marker 1 corresponding to the additional information 1 is included in the unanalyzed AR marker. Therefore, this process proceeds to step S1408.

Next, in step S1407, the additional information determined not to be included in the unanalyzed AR marker is determined not to satisfy the condition, and a label indicating that fact is displayed (for example, in the AR marker / virtual object management table). ) Set.

Next, in step S1408, the additional information determined to be included in the unanalyzed AR marker is determined to satisfy the condition, and a label indicating that fact is displayed (for example, in the AR marker / virtual object management table). ) Set.

Next, in step S1409, the object creation unit determines whether or not additional information for which the unanalyzed flag is set remains. If unanalyzed additional information remains, this process returns to step S1405, and the processes from steps S1405 to steps S1408 are performed on the unanalyzed additional information. If no unanalyzed additional information remains, this process proceeds to step S1410. This process may be repeated until all the additional information for which the unanalyzed flag is set is processed.
As an example, when the additional information 1, 7, and 9 are acquired in step S1403 and only the additional information 1 has been determined in step S1406, the additional information 7 and the additional information 9 still remain as unanalyzed data. Therefore, this process returns to step S1405, and the processes from step S1405 to step S1408 are performed for each of the additional information 7 and the additional information 9. As a result, the additional information 1 and the additional information 7 are classified as satisfying the conditions, and the additional information 9 is classified as not satisfying the conditions.

Next, in step S1410, the object creation unit determines whether or not there is a plurality of information classified as satisfying the conditions. If there are a plurality of information classified as satisfying the conditions, the present process proceeds to step S1412, and if not, the present process proceeds to step S1411.
For example, if the additional information 1 and the additional information 7 are classified as satisfying the conditions, the result of the determination in step S1410 is "Yes", so this process proceeds to step S1412.

Next, in step S1411, since the object creation unit does not have a plurality of information classified when the condition is satisfied, one additional information classified when the condition is satisfied is set as "information to be used".

Next, in step S1412, the object creation unit refers to the AR marker / virtual object management table and sets additional information corresponding to the most AR markers as “information to be used”. For example, as shown in the AR marker / virtual object management table of FIG. 8, the additional information 1 corresponds to one AR marker of the AR marker 1, but the additional information 7 corresponds to the AR marker 1, the AR marker 2, and the AR marker 2. Since it corresponds to the three AR markers of the AR marker 3, the additional information 7 corresponding to more AR markers is set as "information to be used".

Next, in step S1413, the object creation unit changes the flag of the AR marker corresponding to the additional information set as "information to be used" from "unanalyzed" to "analyzed". For example, when the additional information 7 is set as "information to be used", the flags of the three AR markers corresponding to the additional information 7, AR marker 1, AR marker 2, and AR marker 3 are changed from "Unanalyzed". Change to "analyzed".

Next, in step S1414, the data analysis unit analyzes the actual image of the AR marker corresponding to the additional information set as the "information to be used", and specifies the coordinates of the four corners of each AR marker. For example, when the additional information 7 is set as "information to be used", the coordinates of the three AR markers corresponding to the additional information 7 are specified.

Next, in step S1415, the object creation unit calculates the base coordinates that define the position to display the virtual object based on the coordinates of the specified AR marker. The base coordinates are coordinates that specify the coordinates of the four corners of the area for displaying the virtual object. In order to calculate the base coordinates of the virtual object, the object creation unit sets the coordinates that will be the base coordinates among the coordinates of all AR markers that correspond to the additional information set as "information to be used" as follows. It may be specified.
OBJX1: Minimum value of X coordinate of all AR markers, OBJY1: Maximum value of Y coordinate of all AR markers OBJX2: Maximum value of X coordinate of all AR markers, OBJY2: Maximum value of Y coordinate of all AR markers OBJX3: Minimum value of X coordinate of all AR markers, OBJY3: Minimum value of Y coordinate of all AR markers OBJX4: Maximum value of X coordinate of all AR markers, OBJY4: Minimum value of Y coordinate of all AR markers An example of the processing in this step will be described later.

Next, in step S1416, the object creation unit determines whether or not an unanalyzed AR marker remains. If an unanalyzed AR marker remains, this process returns to step S1402 and repeats the processes of steps S1402 to S1415. If no unanalyzed AR marker remains, this process ends. After that, as described above in step S1205 of process 1200 shown in FIG. 12, a virtual object is created and superimposed on the base coordinates specified in step S1415 of this process.
As an example, when the AR marker 1, the AR marker 2, the AR marker 3, and the AR marker 4 are specified in step S1401, the AR markers 1 to 3 are analyzed as a result of the above-described processing, and the AR markers 1 to 3 are analyzed. A virtual object containing additional information is created, but the AR marker 4 is still set with an unanalyzed flag. Therefore, after the processing of the AR markers 1 to 3 is completed, the result of the determination in step S1416 becomes "NO", and this processing returns to step S1402. Subsequently, steps S1402 to S1415 are performed on the AR marker 4. As a result, a virtual object including additional information corresponding to all the AR markers specified in the real image is created and superimposed on the augmented reality image.

Next, an example of specifying the coordinates of the four corners of the AR marker will be described with reference to FIG.

FIG. 15 is a diagram showing an example of specifying the coordinates of the four corners of the AR marker according to the embodiment of the present invention. As described above, the base coordinates for displaying the virtual object are specified based on the coordinates of the four corners of each AR marker detected in the real image. In order to specify the coordinates of the four corners of the AR marker, the above-mentioned data analysis unit may perform edge detection such as, for example, the Canny method, other primary methods, threshold processing and continuous processing, secondary methods, and differential edge detection. By using means or other existing image processing means, the coordinates of the four corners of the detected AR marker may be specified.
For example, as shown in FIG. 15, the data analysis unit identifies the plane coordinates of each of the four corners of the AR marker 1510, the AR marker 1520, and the AR marker 1530.

Next, an example of the base coordinates of the virtual object according to the embodiment of the present invention will be described with reference to FIG.

FIG. 16 is a diagram showing an example of base coordinates of a virtual object according to an embodiment of the present invention. The base coordinates are coordinates that specify the coordinates of the four corners of the area for displaying the virtual object, and are calculated by the calculation method shown in step S1415 of FIG. 14 based on the coordinates of the four corners of the AR marker detected in the real image. It is calculated.
For example, the base coordinates of the virtual object shown in FIG. 16 are (OBJX1, OBJY1), (OBJX2, OBJY2), (OBJX3, OBJY3), and (OBJX4, OBJY4).

The display pattern indicating the format and position of the virtual object according to the embodiment of the present invention is determined based on the types, numbers, positions, and the like of a plurality of different AR markers detected at the same time. Further, as described above, the coordinates of the four corners of the area for displaying the virtual object are calculated based on the coordinates of the four corners of all the detected AR markers. Hereinafter, an example of a virtual object display pattern according to an embodiment of the present invention will be described with reference to FIGS. 17 to 18.

FIG. 17 is a diagram showing an example of a display pattern of a virtual object when AR markers are arranged side by side. As shown in FIG. 17, when the AR marker 1710, the AR marker 1720, and the AR marker 1730 are arranged side by side, the object creation unit specifies the coordinates of the four corners of the AR markers 1710, 1720, and 1730, respectively. When the base coordinates of the virtual object 1750 are calculated from these coordinates, the base coordinates of the created virtual object 1750 are the lower left corner of the AR marker 1710, the upper left corner of the AR marker 1710, and the lower right corner of the AR marker 1730. And the coordinates correspond to the upper right corner of the AR marker 1730. As a result, the virtual object 1750 created according to the AR markers 1710, 1720, and 1730 will be displayed in the area occupied by the AR markers 1710, 1720, and 1730.

FIG. 18 is a diagram showing an example of a display pattern of virtual objects when AR markers are arranged in a disorderly manner. As shown in FIG. 17, when the AR markers 1810, the AR markers 1820, and the AR markers 1830 are arranged in a disorderly manner, the object creation unit specifies the coordinates of the four corners of the AR markers 1810, 1820, and 1830, respectively. When the base coordinates of the virtual object 1850 are calculated from these coordinates, the base coordinates of the created virtual object 1850 are the left corner of the AR marker 1810, the lower corner of the AR marker 1820, the upper corner of the AR marker 1830, and The coordinates corresponding to the residence are on the right side of the AR marker 1830. As a result, the virtual object 1850 created according to the AR markers 1810, 1820, 1830 will be displayed in the square area surrounding the AR markers 1810, 1820, and 1830.

Next, a specific example in which the information display means according to the embodiment of the present invention is applied to the maintenance work of the server will be described with reference to FIGS. 19 to 21.

FIG. 19 shows a reality image 1920 including one AR marker 1925 attached to a server arranged in a server rack, and an augmented reality image 1940 including a virtual object 1945 created in response to the AR marker 1925. It is a figure. When the reality image 1920 shown in FIG. 19 is captured and the AR marker 1925 is recognized by the data analysis unit described above, the augmented reality image 1940 including the virtual object 1945 is created by the object creation unit and the augmented reality management unit, and the terminal device. It is displayed on the video display section of.

As described above, when the data analysis unit recognizes the AR marker 1925, the data analysis unit determines the display pattern of the virtual object based on the number, type, and position of the AR marker. Next, the object creation unit acquires the information corresponding to the AR marker 1925 from the storage unit and creates a virtual object 1945 indicating this information. After that, the augmented reality management unit displays the augmented reality image 1940 superimposed on the reality image 1920 on the image display unit of the terminal device according to the display pattern determined by the data analysis unit.

For example, as shown in FIG. 19, the virtual object 1945 created according to the AR marker may show information about CPU temperature, air temperature, and humidity. In this way, the operator who performs the maintenance work of the server can easily confirm the information indicating the state of the server without manually diagnosing the server.

FIG. 20 is a diagram showing a reality image 2020 including six AR markers attached to six servers arranged in a server rack and an augmented reality image 2040 created according to the six AR markers. is there. When the reality image 2020 shown in FIG. 20 is photographed and the six AR markers are recognized by the above-mentioned data analysis unit, the augmented reality image 2040 including the virtual object 2045 is created by the object creation unit and the augmented reality management unit. It is displayed on the video display of the terminal device.

As described above, when the number of recognized AR markers exceeds a predetermined number (for example, four or more), it is preferable to display the information associated with each AR marker as one collective virtual object. Sometimes. Therefore, as shown in FIG. 20, when the number of recognized AR markers exceeds a predetermined number, the object creation unit corresponds to the recognized AR markers and is based on the information stored in the storage unit. , You may create a virtual object 2045 that shows the state (normal or abnormal) of the entire server rack.
In this way, the worker who performs the maintenance work of the server can easily check the state of the entire server rack without diagnosing each server.

FIG. 21 is a diagram showing a specific example in which the point of interest analysis according to the embodiment of the present invention is applied to the maintenance work of the server rack.

In this specific example, it is assumed that the augmented reality image including the virtual object corresponding to the AR marker has already been created and displayed. For example, as shown in FIG. 21, it is assumed that virtual objects 2120A, 2120B, and 2120C indicating the status (normal or abnormal) of the server are displayed for each of a plurality of servers arranged in the server rack. When the line-of-sight analysis unit (for example, the line-of-sight analysis unit 232 shown in FIG. 2) determines that the user's point of interest is directed to the virtual object 2120A corresponding to one server, the line-of-sight analysis unit (that is, the virtual object 2120A) Virtual objects 2140 indicating CPU temperature, temperature, humidity, etc., which are additional information about the CPU, may be added to the augmented reality image 2110.

Next, when the line-of-sight analysis unit determines that the user's attention has shifted to the virtual object 2140 indicating the CPU of the server, the information indicating the usage rate of each core of the CPU may be displayed as the virtual object 2142. Next, when the line-of-sight analysis unit determines that the point of interest of the user has shifted to the virtual object 2142 indicating the usage rate of the core, the graph showing the transition of the usage rate of the core may be displayed as the virtual object 2144. .. Further, when the line-of-sight analysis unit determines that the point of interest of the user has shifted to the virtual object 2144, which indicates the transition of the core usage rate, the virtual object 2146 indicating the work diary corresponding to this transition may be displayed.

In this way, by using the points of interest calculated by the user's line of sight, the user can operate the virtual object displayed in the augmented reality image only with the line of sight, so both hands can be used freely, and work such as server rack maintenance can be performed. It can be done more easily.

Next, an example of acquiring necessary data from the database when the information display means according to the embodiment of the present invention is applied to the maintenance work of the server will be described with reference to FIGS. 22 to 24.

FIG. 22 includes a real image 2220 including an AR marker 2225 attached to a server arranged in a server rack, and a virtual object 2255 created when the user's attention point 2235 is directed to the AR marker 2225. It is a figure which shows the augmented reality image 2250.

As described above, the AR marker and the virtual object superimposed on the augmented reality image are divided into a plurality of areas, and different hidden information is embedded in each area. This non-display information specifies additional information to be a virtual object, a storage destination of the additional information, and a display pattern of the virtual object.
When the line-of-sight analysis unit determines that the user's attention point 2235 is directed to a specific area of the AR marker 2225, the table in which the information specified in the hidden information embedded in the area is specified is specified. (For example, the virtual object 2255 obtained from the additional information management table 1100 described later) and corresponding to the information (CPU temperature, temperature, humidity, etc.) is displayed in a display pattern (for example, tabular format) specified by the non-display information. Create according to and add to the augmented reality video 2250.

FIG. 23 is a diagram showing an example of an area included in the virtual object 2320. As shown in FIG. 23, the virtual object 2320 is divided into three areas 2322, 2324, and 2326. In each area, additional information used to create different virtual objects and hidden information that specifies the format of the additional information are embedded.
For example, when the line-of-sight analysis unit determines that the user's attention point is directed to the area 2322 where the CPU temperature information is displayed, the above-mentioned additional information management is performed according to the non-display information embedded in the area 2322. You may create a virtual object 2330 that acquires the CPU usage rate for each CPU process name from the table 1100 up to the past 1 hour and displays the average value in the form of a table.

Further, similarly to the virtual object 2320, the newly created virtual object 2330 is divided into a plurality of areas 2332, 2334, and 2336, and hidden information is embedded in each area. Therefore, as shown in FIG. 24, when the line-of-sight analysis unit determines that the user's point of interest is directed to a specific area of the virtual object 2330, the new virtual object is based on the non-display information of that area. 2340 may be created.
For example, as shown in FIG. 24, when the line-of-sight analysis unit determines that the user's point of interest is directed to the area 2332 where the information of the specific process xx1 is displayed, the non-embedded area 2332. According to the display information, a virtual object 2340 that displays the state of the time change of the CPU usage rate of the process xx1 in a graph format may be additionally created from the additional information management table 1100 described above.

Next, an example of the operation of fixing the virtual object will be described with reference to FIG. 25.

FIG. 25 is a diagram showing an example of an operation of fixing a virtual object according to an embodiment of the present invention. When using the information display means according to the embodiment of the present invention, it may be desirable to display a plurality of virtual objects of different AR markers at the same time. For example, when the information display means according to the present invention is applied to server maintenance work, a user may want to compare information (CPU temperature, near temperature, temperature, etc.) of a plurality of servers. Therefore, in one of the aspects according to the present invention, the user can display a virtual object related to another AR marker in a state where the virtual object being displayed is fixed by a specific operation.
Here, the operation for fixing the virtual object may be, for example, a wink (blink of one eye).

As an example, as shown in FIG. 25, the virtual object is fixed by the user performing a specific operation such as winking while the virtual object 2520 showing the information of one server is displayed. The user then shifts the focus to another AR marker to display a new virtual object 2530 for the server of interest. In this way, since the user can simultaneously view the status of a plurality of servers, the work efficiency of server maintenance can be improved.

Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments, and various modifications can be made without departing from the gist of the present invention.

210A, 210B Data acquisition device 212A, 212B Sensor unit 214A, 214B Data transmission / reception unit 220 Control device 222 Data transmission / reception unit 224 Data analysis unit 226 Storage unit 228 Object creation unit 230 Augmented reality management unit 232 Line-of-sight analysis unit 240 Terminal device 241 Data transmission / reception Unit 242 Camera unit 244 Video display unit

Claims (7)

It is an information display device
A data transmitter / receiver that acquires a real image including multiple AR markers from the camera,
A data analysis unit that determines the tag information, type and number of AR markers in the real image, and determines the first display pattern for the first virtual object based on the type and number of AR markers.
Based on the tag information of the AR marker, the first additional information to be the first virtual object is acquired from the virtual object information management database, and the first virtual object is created based on the first additional information. Object creation part and
An augmented reality management unit that creates and outputs a first augmented reality image in which the created first virtual object is superimposed on the reality image according to the determined first display pattern.
An information display device characterized by. The information display device is
It further includes a line-of-sight analysis unit that determines the line-of-sight position of the user in the first augmented reality image.
When it is determined that the line-of-sight position is directed to the first virtual object,
The data analysis unit
Based on the first virtual object, a second display pattern for the second virtual object is determined.
The object creation unit
Based on the first virtual object, the second additional information to be the second virtual object is acquired from the virtual object information management database, and the second virtual object is obtained based on the second additional information. make,
The augmented reality management department
According to the second display pattern, a second augmented reality image is created and output by superimposing the created second virtual object on the first augmented reality image.
The information display device according to claim 1. The first virtual object is
It is divided into multiple areas
In each area, the second additional information that becomes the second virtual object and the non-display information that specifies the display pattern of the second virtual object are embedded.
The data analysis unit
Using the hidden information embedded in the area to which the user's line of sight is directed, the second display pattern for the second virtual object is determined.
The information display device according to claim 2, wherein the information display device is characterized by the above. The line-of-sight analysis unit
When a predetermined user action for the first virtual object is detected,
The display position and display format of the first virtual object are fixed.
The information display device according to claim 2, wherein the information display device is characterized by the above. The first display pattern specifies the display coordinates and the display format of the first virtual object.
The information display device according to claim 1. An information display method executed by an information display device.
The information display method is
The process of acquiring a real image including multiple AR markers from the camera,
A step of determining the tag information, type and number of AR markers in the real image, and determining a first display pattern for the first virtual object based on the type and number of AR markers.
Based on the tag information of the AR marker, the first additional information to be the first virtual object is acquired from the virtual object information management database, and the first virtual object is created based on the first additional information. And the process to do
A process of creating and outputting a first augmented reality image in which the created first virtual object is superimposed on the reality image according to the determined first display pattern.
Information display method including. An information display system in which a data acquisition terminal, a control device, and a terminal device are connected via a communication network.
The data acquisition terminal is
A sensor unit that acquires sensor data related to the surrounding environment,
Including a first data transmission / reception unit that transmits the sensor data to the control device.
The terminal device is
A camera unit that acquires real-life images including multiple AR markers,
A second data transmission / reception unit that transmits the real image including the plurality of AR markers to the control device, and
Including an image display unit that displays an augmented reality image received from the control device.
The control device is
A third data transmission / reception unit that receives the real image including the plurality of AR markers from the terminal device and receives the sensor data from the data acquisition terminal.
A virtual object information management database that stores the sensor data received from the data acquisition terminal as additional information for generating a virtual object, and a virtual object information management database.
The tag information, type and number of AR markers in the real image including the plurality of AR markers are determined, and the first display pattern for the first virtual object is determined based on the types and numbers of the AR markers. Data analysis department and
Based on the tag information of the AR marker, the first additional information to be the first virtual object is acquired from the virtual object information management database, and the first virtual object is obtained based on the first additional information. The object creation part to be created and
An augmented reality management unit that creates a first augmented reality image by superimposing the created first virtual object on the reality image according to the determined first display pattern and outputs the first augmented reality image to the terminal device.
An information display system featuring.

Images