JP6248732B2 - Augmented reality display device, augmented reality display system, augmented reality display method, and augmented reality display program - Google Patents

Description

  The present invention relates to an augmented reality display device, an augmented reality display system, an augmented reality display method, and an augmented reality display program. In particular, the present invention relates to an augmented reality display device that displays information related to a subject, and an augmented reality display including the augmented reality display device. The present invention relates to an augmented reality display method and an augmented reality display program executed by the system, the augmented reality display device.

  In recent years, augmented reality technology has been developed. For example, Japanese Patent Laid-Open No. 9-33271 discloses imaging means such as a camera or a camera-integrated VTR device, display means for displaying video and data taken by the imaging means, and the imaging means on the earth at each time. Detection means for detecting the position and direction of the object by satellite positioning, storage means for storing map data, three-dimensional position coordinates on the earth of a predetermined object among the objects displayed on the display means, the name of the object, etc. Control means for reading the attribute data from the storage means, converting the three-dimensional position coordinates into two-dimensional position coordinates on the display means, and displaying the attribute data on the two-dimensional position coordinates on the display means An imaging apparatus characterized by this is described.

However, in a conventional imaging apparatus, the three-dimensional position coordinates of a predetermined object on the earth must be stored in association with the object. Further, there is a problem that the processing load is heavy because the three-dimensional position coordinates must be converted into the two-dimensional position coordinates on the display means.
JP-A-9-33271

  The present invention has been made to solve the above-described problems, and one of the objects of the present invention is to provide an augmented reality display device capable of executing a predetermined process when a predetermined subject is imaged. Is to provide.

  Another object of the present invention is to provide an augmented reality display system capable of displaying related information registered by an arbitrary user in association with a subject.

  Still another object of the present invention is to provide an augmented reality display method capable of executing a predetermined process when a predetermined subject is imaged.

  Still another object of the present invention is to provide an augmented reality display program capable of executing a predetermined process when a predetermined subject is imaged.

  In order to achieve the above object, according to an aspect of the present invention, an augmented reality display device captures an object and outputs a first image; and an object in the acquired first image; Augmented reality information acquisition means for acquiring augmented reality information including a second image having the same subject, and augmented reality information is relative to the subject in the second image in addition to the second image. Including region information indicating the display region determined by the position and related information related to the subject, the relative position with respect to the subject in the first image being the same as the relative position between the subject and the display region in the second image The actual display area determining means for determining the actual display area, the display control means for displaying the related information included in the augmented reality information in the determined actual display area, and the second image and the first image are different. Different part extraction to extract different parts And means, if different part by different part extraction unit is extracted, and the process execution means for executing predetermined processing, the.

  According to this aspect, augmented reality information including a second image having the same subject as the subject in the first image obtained by capturing the subject is acquired, and the relative position with respect to the subject in the first image is the second position. A real display area having the same relationship as the relative position between the subject in the image and the display area is determined, and related information is displayed in the real display area. For this reason, the related information associated with the subject in the first image is obtained from the augmented reality information including the second image having the same subject as the subject in the first image obtained by capturing the subject. It is displayed at a relative position determined with respect to the inside subject. As a result, information related to the subject in the captured image can be displayed at a predetermined position with respect to the subject. In addition, when different portions are extracted between the second image and the first image, a predetermined process is executed, so that the predetermined process can be executed when a predetermined subject is imaged. An augmented reality display device can be provided.

  Preferably, the process execution unit includes an update control unit that updates the second image of the augmented reality information with the first image.

  According to this aspect, when a predetermined subject is imaged, the second image included in the augmented reality information is updated with the first image obtained by imaging, so that a portion different from the subject, for example, the background Etc., the image included in the augmented reality information can be updated to the latest state.

  Preferably, the process execution unit includes a notification unit that notifies the user that the second image is different from the first image.

  According to this aspect, it is possible to notify the user that a portion different from the subject, for example, the background or the like has changed. Further, it is possible to notify the user that the image included in the augmented reality information is updated to the latest state.

  Preferably, the different portion extraction unit compares the part of the first image output by the imaging unit by imaging the subject with at least a part of the second image included in the acquired augmented reality information. including.

  According to this aspect, since a part of the first image is compared with at least a part of the second image included in the augmented reality information, the entire first image and the second image do not match each other. However, it can be determined whether or not the subject is the same.

  Preferably, the different portion extraction unit includes a dividing unit that divides the first image into a plurality of partial images, a comparison unit that compares each of the plurality of partial images with a part of the second image, and a plurality of partial images. A different portion determining means for determining, as a different portion, a partial image surrounded by a plurality of partial images similar to a part of the second image and not similar to a part of the second image; Including.

  According to this aspect, each of the plurality of partial images obtained by dividing the first image is compared with a part of the second image, and a plurality of partial images similar to a part of the second image among the plurality of partial images. A partial image that is surrounded by and is not similar to a part of the second image is determined as a different part. Therefore, it can be determined that the first image and the second image have the same subject but are different as a whole.

  Preferably, the augmented reality information further includes position information indicating a geographical position, and the augmented reality display device further includes position acquisition means for acquiring a geographical position where the augmented reality display device actually exists as a current position, and the augmented reality information The acquisition means further includes a position reference acquisition means for acquiring augmented reality information associated with a geographical position within a predetermined range from the current position acquired based on the acquired current position.

  According to this aspect, the augmented reality information is acquired because the geographic position of the subject in the second image acquires the augmented reality information existing within a predetermined range from the geographic position at the time of capturing the subject. Becomes easy.

  Preferably, the augmented reality information acquisition unit acquires the augmented reality information based on the first image output by the imaging unit when the current position cannot be acquired by the position acquisition unit.

  According to this aspect, augmented reality information can be acquired even when the current position cannot be acquired.

  Preferably, the image processing apparatus further includes display means for displaying an image, and the display control means includes composite image generation means for generating a composite image in which related information included in the augmented reality information is superimposed on the first image. The image is displayed on the display means.

  According to this aspect, a composite image in which related information is superimposed on the first image is generated, and the composite image is displayed. Therefore, related information can be associated with the subject in the first image.

  Preferably, the display unit is further provided with a transmissive display unit arranged at a position determined with reference to the optical axis of the imaging unit, and the actual display area determination unit displays the display unit based on the first image output by the imaging unit. The real display area is determined in the plane, and the display control means displays the related information included in the augmented reality information in the real form display area in the display plane of the display means.

  According to this aspect, the real display area is determined in the display surface of the display unit based on the first image obtained by imaging the subject, and the related information is displayed in the real display area in the display surface of the display unit. For this reason, related information can be displayed in association with the subject that the user sees through the display means.

  Preferably, communication control means capable of communicating with a server storing augmented reality information is further provided, and the augmented reality information acquisition means acquires augmented reality information from the server.

  If this aspect is followed, since augmented reality information is acquired from the server which stores augmented reality information, the related information registered by the user different from the user of an augmented reality display apparatus can be displayed.

  According to another aspect of the present invention, an augmented reality display system is an augmented reality display system including the above-described augmented reality display device, and a basic information registration device and an augmented reality information registration device that can communicate with a server. The basic information registration device includes a basic imaging unit that images a subject and outputs an image, a basic position acquisition unit that acquires a geographical position where the basic information registration device actually exists, and a basic imaging unit. A basic information registration unit for registering basic information including the second image and the current position acquired by the basic position acquisition unit in the server, and an image to be captured and output as a second image; The real information registration device includes a position receiving unit that receives designation of a geographical position, and basic information including position information in which the received geographical position is within a predetermined range from a server. Obtaining basic information acquisition means, generation-side display means for displaying a second image included in the acquired basic information, display area reception means for receiving designation of a display area in the second image, and acquisition of related information The augmented reality information generating means for generating augmented reality information by adding the related information acquisition means, the acquired related information, and the area information indicating the display area received by the display area accepting means to the basic information, and the server Registration means for registering the generated augmented reality information.

  According to this aspect, the basic information is registered by the basic information registration device, and the augmented reality information in which the related information and the region information are added to the basic information is registered by the augmented reality information registration device. Therefore, it is possible to display related information registered by the user of the augmented reality display device or a user other than the user. As a result, it is possible to provide an augmented reality display system that can display related information registered by an arbitrary user in association with a subject.

  Preferably, the augmented reality information registration apparatus further includes a document reading unit that optically reads a document and outputs image data converted into electronic data, and the related information acquisition unit relates image data output by the document reading device. Obtain as information.

  According to this aspect, image data obtained by reading a document can be used as related information.

  Preferably, the augmented reality information registration device further includes data storage means for storing data, and the related information acquisition means acquires data specified by the user from the data stored in the data storage means as related information. .

  According to this aspect, data specified by the user can be used as related information.

  According to still another aspect of the present invention, an augmented reality display method includes an imaging step of acquiring a first image obtained by imaging a subject, and a second subject having the same subject as the subject in the acquired first image. The augmented reality information acquisition step for acquiring augmented reality information including the image of the image and the augmented reality information indicate a display area determined at a position relative to the subject in the second image in addition to the second image. A real display area is determined that includes area information and related information related to the subject, and whose relative position to the subject in the first image is the same as the relative position of the subject and the display region in the second image. A real display area determining step, a display control step for displaying related information included in the augmented reality information in the determined real display area, and a different part for extracting a different part between the second image and the first image Extraction step Including a flop, when the different part in different part extraction step is extracted, and the process execution step of executing predetermined processing, the.

  According to this aspect, it is possible to provide an augmented reality display method capable of executing a predetermined process when a predetermined subject is imaged.

  According to still another aspect of the present invention, the augmented reality display program obtains a first image obtained by imaging a subject, and a second subject having the same subject as the subject in the obtained first image. The augmented reality information acquisition step for acquiring augmented reality information including the image of the image and the augmented reality information indicate a display area determined at a position relative to the subject in the second image in addition to the second image. A real display area is determined that includes area information and related information related to the subject, and whose relative position to the subject in the first image is the same as the relative position of the subject and the display region in the second image. A real display area determining step, a display control step for displaying related information included in the augmented reality information in the determined real display area, and a different part for extracting a different part between the second image and the first image Extraction Steps and, when a different part in different part extraction step is extracted, to execute a process executing step of executing predetermined processing, to the computer.

  According to this aspect, it is possible to provide an augmented reality display program capable of executing a predetermined process when a predetermined subject is imaged.

It is a figure which shows an example of the whole outline | summary of the augmented reality display system in embodiment of this invention. 1 is a perspective view showing an external appearance of an MFP in the present embodiment. 2 is a block diagram illustrating an example of a hardware configuration of an MFP according to the present embodiment. FIG. It is a block diagram which shows an example of the outline | summary of the hardware constitutions of the portable information device in this Embodiment. It is a block diagram which shows an example of the hardware constitutions of HMD in this Embodiment. It is a block diagram which shows an example of the hardware constitutions of the server in this Embodiment. It is a figure which shows an example of the format of augmented reality information. It is a block diagram which shows an example of the outline | summary of the function of CPU with which the portable information device which functions as an augmented reality display apparatus is provided. It is a block diagram which shows an example of the detailed function of a different part extraction part. It is a figure which shows an example of a background image. It is a figure which shows an example of a captured image. It is a figure which shows an example of a synthesized image. It is a figure which shows an example of the format of basic information. It is a block diagram which shows an example of the function of CPU201 with which the portable information device 200 which functions as a basic information registration apparatus is provided. It is a block diagram which shows an example of the function of CPU with which MFP which functions as an augmented reality information registration apparatus is provided. It is a block diagram which shows an example of the function which CPU with which a server is provided has. It is a flowchart which shows an example of the flow of a basic information registration process. It is a flowchart which shows an example of the flow of an augmented reality information registration process. It is a flowchart which shows an example of the flow of an augmented reality information management process. It is a flowchart which shows an example of the flow of an augmented reality display process. It is a flowchart which shows an example of the flow of a different part judgment process.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

<First Embodiment>
FIG. 1 is a diagram showing an example of an overall outline of an augmented reality display system according to an embodiment of the present invention. Referring to FIG. 1, an augmented reality display system 1 includes a wireless station 3, an MFP (Multi Function Peripheral) 100, portable information devices 200, 200 </ b> A, 200 </ b> B, and a personal computer (hereinafter referred to as a “computer”). 3, a head mounted display (hereinafter referred to as “HMD”) 400, and a server 500.

  MFP 100 functions as an image processing apparatus, and includes a document reading function for reading a document, an image forming function for forming an image on a recording medium such as paper based on image data, and a facsimile transmission / reception function for transmitting / receiving facsimile data. ing. In the first embodiment, the MFP 100 will be described as an example of an image processing apparatus. However, if the apparatus has a function of processing an image instead of the MFP 100, for example, a PC 300, a server 500, or the like. It may be.

  The PC 300 is a general computer. The PC 300 is installed with a printer driver program for controlling the MFP 100.

  The portable information devices 200, 200A, and 200B are examples of information processing devices. The portable information devices 200, 200A, and 200B are used by being carried by users, such as PDAs (Personal Digital Assistants) and smartphones, and have a function of storing data and a wireless LAN function. In addition, portable information devices 200, 200A, and 200B can install application programs in the same manner as PC 300. Since the portable information devices 200, 200A, and 200B have the same hardware configuration and functions, the portable information device 200 will be described as an example here unless otherwise specified.

  The HMD 400 has a shape of glasses and is used by being worn by a user. The HMF 400 has an imaging function for imaging a subject, a display function for displaying an image on the lens portion of the glasses, and a wireless LAN function. A user wearing the HMD 400 can visually recognize the subject through the lens and at the same time visually recognize the image displayed on the lens.

  The network 2 is a local area network (LAN), and the connection form may be wired or wireless. Further, the network 2 is not limited to a LAN, and may be a network using a public switched telephone network (Public Switched Telephone Networks). Furthermore, the network 2 is connected to a wide area network (WAN) such as the Internet.

  The MFP 100 can transmit and receive data to and from the wireless station 3, the PC 300, and the server 500 via the network 2. The wireless station 3 is a relay device of the network 2 and communicates with the portable information devices 200, 200A, 200B and the HMD 400 having a wireless LAN communication function to connect the portable information devices 200, 200A, 200B and the HMD 400 to the network 2. To do. Therefore, each of portable information devices 200, 200 </ b> A, 200 </ b> B and HMD 400 can send and receive data to / from MFP 100, PC 300, and server 500 via wireless station 3 and network 2.

  FIG. 2 is a perspective view showing an appearance of the MFP according to the present embodiment. FIG. 3 is a block diagram illustrating an example of the hardware configuration of the MFP according to the present embodiment. 2 and 3, MFP 100 includes a main circuit 110, a document reading unit 130 for reading a document, an automatic document transport device 120 for transporting a document to document reading unit 130, and a document reading unit. An image is formed by an image forming unit 140 for forming an image on a sheet or the like based on image data output by the manuscript 130 reading and outputting a document, a paper feeding unit 150 for supplying the image to the image forming unit 140, and an image. A post-processing unit 155 that processes paper and an operation panel 160 as a user interface are included.

  The post-processing unit 155 executes a sorting process for sorting and discharging one or more sheets on which images are formed by the image forming unit 140, a punching process for punch hole processing, and a staple process for driving staple needles.

  The main circuit 110 includes a CPU 111, a communication interface (I / F) unit 112, a ROM 113, a RAM 114, a hard disk drive (HDD) 115 as a mass storage device, a facsimile unit 116, and a CD-ROM 118. External storage device 117. CPU 111 is connected to automatic document feeder 120, document reading unit 130, image forming unit 140, sheet feeding unit 150, and operation panel 160, and controls the entire MFP 100.

  The ROM 113 stores a program executed by the CPU 111 or data necessary for executing the program. The RAM 114 is used as a work area when the CPU 111 executes a program. The RAM 114 temporarily stores read data (image data) continuously sent from the document reading unit 130.

  Operation panel 160 is provided on the upper surface of MFP 100 and includes a display unit 161 and an operation unit 163. The display unit 161 is a display device such as a liquid crystal display (LCD) or an organic ELD (Electro-Luminescence Display), and displays an instruction menu for the user, information about acquired image data, and the like. The operation unit 163 includes a hard key unit 167 including a plurality of keys, and accepts input of various instructions, data such as characters and numbers by user operations corresponding to the keys. The operation unit 163 further includes a touch panel 165 provided on the display unit 161.

  Communication I / F unit 112 is an interface for connecting MFP 100 to network 2. CPU 111 communicates with portable information devices 200, 200 </ b> A, 200 </ b> B, PC 300, HMD 400, and server 500 via communication I / F unit 112 to transmit / receive data. The communication I / F unit 112 can communicate with a computer connected to the Internet via the network 2.

  The facsimile unit 116 is connected to the public switched telephone network (PSTN) and transmits facsimile data to the PSTN or receives facsimile data from the PSTN. The facsimile unit 116 stores the received facsimile data in the HDD 115 or outputs it to the image forming unit 140. The image forming unit 140 prints the facsimile data received by the facsimile unit 116 on a sheet. Further, the facsimile unit 116 converts the data stored in the HDD 115 into facsimile data, and transmits the facsimile data to a facsimile machine connected to the PSTN.

  The external storage device 117 is loaded with a CD-ROM (Compact Disk ROM) 118. The CPU 111 can access the CD-ROM 118 via the external storage device 117. The CPU 111 loads the program recorded on the CD-ROM 118 attached to the external storage device 117 to the RAM 114 and executes it. The medium for storing the program executed by the CPU 111 is not limited to the CD-ROM 118, but an optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC card, optical card, It may be a semiconductor memory such as a mask ROM, EPROM (Erasable Programmable ROM), or EEPROM (Electrically EPROM).

  Further, the program executed by the CPU 111 is not limited to the program recorded on the CD-ROM 118, and the program stored in the HDD 115 may be loaded into the RAM 114 and executed. In this case, another computer connected to the network 2 may rewrite the program stored in the HDD 115 of the MFP 100, or may write a new program. Further, MFP 100 may download a program from another computer connected to network 2 and store the program in HDD 115. The program here includes not only a program directly executable by the CPU 111 but also a source program, a compressed program, an encrypted program, and the like.

  In the augmented reality display system 1, data is transmitted and received between the MFP 100, the portable information devices 200, 200 </ b> A, 200 </ b> B, the PC 300, the HDM 400, and the server 500. As a protocol for transmitting and receiving data, any protocol can be used as long as the transmission source can be specified in the receiving apparatus. As a protocol for transmitting and receiving data, for example, HTTP (Hyper Text Transfer Protocol), FTP (File Transfer Protocol), SMPT (Simple Mail Transfer Protocol), POP (Post Office Protocol), and the like are used.

  FIG. 4 is a block diagram illustrating an example of an outline of a hardware configuration of the portable information device according to the present embodiment. Referring to FIG. 4, portable information device 200 is connected to CPU 201 for controlling overall portable information device 200, camera 202, flash memory 203 for storing data in a nonvolatile manner, and call unit 205. A wireless communication unit 204; a display unit 206 that displays information; an operation unit 207 that accepts user operations; a wireless LAN I / F 208; a position detection unit 209; an orientation detection unit 210; and an external storage device 211. Including.

  The wireless communication unit 204 performs wireless communication with a mobile phone base station connected to a telephone communication network. The wireless communication unit 204 connects the portable information device 200 to a telephone communication network and enables a call using the call unit 205. The radio communication unit 204 decodes a voice signal obtained by demodulating the radio signal received from the mobile phone base station and outputs the decoded signal to the call unit 205. Further, the wireless communication unit 204 encodes the voice input from the call unit 205 and transmits it to the mobile phone base station. The call unit 205 includes a microphone and a speaker, outputs the voice input from the wireless communication unit 204 from the speaker, and outputs the voice input from the microphone to the wireless communication unit 204. Further, the wireless communication unit 204 is controlled by the CPU 201, connects the portable information device 200 to an electronic mail server, and transmits and receives electronic mail.

  The camera 202 includes a lens and a photoelectric conversion element. The light focused by the lens is imaged on the photoelectric conversion element. The photoelectric conversion element photoelectrically converts the received light and outputs image data to the CPU 201. The photoelectric conversion element is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, or the like.

  The display unit 206 is a display device such as a liquid crystal display (LCD) or an organic ELD, and displays an instruction menu for a user, information about acquired image data, and the like. The operation unit 207 includes a plurality of keys, and accepts input of various instructions, data such as characters and numbers by user operations corresponding to the keys.

  The wireless LAN I / F 208 is an interface for communicating with the wireless station 3 and connecting the portable information device 200 to the network 3. By registering the IP (Internet Protocol) addresses of HMD 200, server 500, MFP 100, and PC 300 in portable information device 200, portable information device 200 can communicate with HMD 400, server 500, MFP 100, and PC 300. Data transmission / reception becomes possible. In this embodiment, a case where portable information device 200 communicates with MFP 100, PC 300, HMD 400, and server 500 using wireless LAN I / F 208 will be described as an example, but communication is performed using another communication method. You may do it. Specifically, when portable information device 200, MFP 100, PC 300, HMD 400, and server 500 are equipped with a short-range wireless device such as Bluetooth (registered trademark), for example, portable information device 200 is MFP 100, PC 300, HMD 400. , One-to-one communication with any of the servers 500 may be performed. In addition, the portable information device 200 is connected to any one of the MFP 100, the PC 300, the HMD 400, and the server 500 by a wire such as a USB (Universal Serial Bus) cable, and the portable information device 200 is connected to the MFP 100, the PC 300, the HMD 400, the server. You may make it communicate with either of 500 one to one.

  The flash memory 203 stores a program executed by the CPU 201 or data necessary for executing the program. The CPU 201 loads a program recorded in the flash memory 203 to a RAM included in the CPU 201 and executes it.

  The position detection unit 209 detects the current position of the portable information device 200. Specifically, the position detection unit 209 is a global positioning system receiver, and measures the current position by receiving radio waves from a plurality of GPS satellites. The position detection unit 209 outputs a value indicating the measured current position, for example, latitude and longitude to the CPU 201.

  The direction detection unit 210 detects the direction in which the camera 202 of the portable information device 200 faces. Specifically, the orientation detection unit 210 is a geomagnetic sensor that measures geomagnetism. The direction detection unit 210 outputs the detected direction to the CPU 201. The direction detection unit 210 is installed in the portable information device 200 so as to be able to detect the imaging direction of the camera 202, in other words, the direction in which the optical axis of the lens of the camera 202 faces. The direction detection unit 210 is not limited to a geomagnetic sensor, and a gyro sensor or the like may be used.

  The external storage device 211 is detachable from the portable information device 200, and a CD-ROM 211A storing a remote operation program can be attached thereto. The CPU 201 can access the CD-ROM 211A via the external storage device 211. The CPU 201 can load a remote operation program recorded on the CD-ROM 211A attached to the external storage device 211 into a RAM included in the CPU 201 and execute it.

  Note that the program recorded in the flash memory 203 or the CD-ROM 211A has been described as a program executed by the CPU 201, but a program in which another computer connected to the network 2 rewrites the program stored in the flash memory 203, Alternatively, it may be a new program written additionally. Further, the portable information device 200 may be a program downloaded from another computer connected to the network 2. The program here includes not only a program directly executable by the CPU 201 but also a source program, a compressed program, an encrypted program, and the like.

  The medium for storing the program executed by the CPU 201 is not limited to the CD-ROM 211A, and may be a semiconductor memory such as an optical disk (MO / MD / DVD), an IC card, an optical card, a mask ROM, an EPROM, or an EEPROM. Good.

  FIG. 5 is a block diagram showing an example of the hardware configuration of the HMD in the present embodiment. Referring to FIG. 5, HMD 400 in the present embodiment includes a CPU 401 for controlling the entire HMD 400, a camera 402, a flash memory 403 for storing data in a nonvolatile manner, and a display unit 404 for displaying information. And wireless LAN I / F 405.

  The camera 402 includes a lens and a photoelectric conversion element. The light focused by the lens is imaged on the photoelectric conversion element. The photoelectric conversion element photoelectrically converts the received light and outputs image data to the CPU 401. The photoelectric conversion element is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, or the like.

  The display unit 404 is a liquid crystal display (LCD) made of a transparent member, and is embedded in the lens portion of the HMD 400. The display surface of the display unit 404 is arranged at a position determined with reference to the optical axis of the camera 402. Specifically, the imaging range of the camera 402 is set to be the same as the field of view when the user wears the HMD 400. For this reason, the image obtained by imaging with the camera 404 is the same as the image in the field of view actually viewed by the user. For this reason, the position in the user's field of view can be specified from the image output by the camera 402. The display surface of the display unit 404 is set to be the same as the field of view when the user wears the HMD 400 in the lens portion. For this reason, an image can be displayed at an arbitrary position in the user's field of view.

  The wireless LAN I / F 405 is an interface for communicating with the wireless station 3 and connecting the HMD 400 to the network 2. By registering the IP (Internet Protocol) address of the server 500 in the flash memory 403, the HMD 400 can communicate with the server 500 and transmit / receive data. In this embodiment, the case where the HMD 400 communicates with the server 500 using the wireless LAN I / F 405 will be described as an example, but communication may be performed using another communication method. Specifically, when the HMD 400 and the server 500 are mounted with a short-range wireless device such as Bluetooth (registered trademark), for example, the HMD 400 may communicate with the server 500 on a one-to-one basis.

  The flash memory 403 stores a program executed by the CPU 401 or data necessary for executing the program. The CPU 401 loads the program recorded in the flash memory 403 to the RAM provided in the CPU 401 and executes it.

  FIG. 6 is a block diagram illustrating an example of a hardware configuration of the server according to the present embodiment. Referring to FIG. 6, server 500 has a CPU 501 for controlling server 500 as a whole, ROM 502 for storing a program to be executed by CPU 501, RAM 503 used as a work area for CPU 501, and non-volatile storage of data. HDD 504 for storing information, a communication unit 505 for connecting CPU 501 to network 2, a display unit 506 for displaying information, and an operation unit 507 for receiving an input of a user operation.

  In the augmented reality display system 1 according to the present embodiment, the augmented reality information is stored in the server 500, and the portable information device 200 functioning as the augmented reality display device downloads the augmented reality information from the server 500 and displays it.

  FIG. 7 is a diagram illustrating an example of a format of augmented reality information. Referring to FIG. 7, the augmented reality information includes a position information item, a content item, a related information item, and a region information item. In the position information item, position information indicating a geographical position is set. Content is set in the content item. Here, an example will be described in which content is set as a background image, which is a captured image obtained by capturing a subject at a position specified by position information set in the position information item. In the related information item, related information related to the object in the content is set. Here, information related to the subject in the background image set in the content item is set. In the area information item, area information indicating a display area determined by a relative position with respect to an object in the content is set. Here, area information indicating the position of the area defined by the relative position with respect to the subject in the background image set in the content item is set.

  Although a specific example of the method for storing the augmented reality information in the server 500 will be described later, the augmented reality information is stored in the server 500 in advance. Here, when the whiteboard is imaged in the conference room A, the position information indicating the position of the conference room A at the time of imaging is set in the position information item, and the captured image captured using the whiteboard as a subject is displayed. An extension that is set as a content item, an image that is a meeting material is set as a related information item, and an area that includes the entire drawing surface of the whiteboard included as a subject in the captured image is set as an area information item A case where real information is registered in the server 500 will be described as an example.

  FIG. 8 is a block diagram illustrating an example of an overview of the functions of the CPU provided in the portable information device 200 that functions as an augmented reality display device. The functions shown in FIG. 8 are functions formed in the CPU 201 when the CPU 201 included in the portable information device 200 functioning as an augmented reality display device executes the augmented reality display program stored in the flash memory 203. Referring to FIG. 8, the CPU 201 includes an imaging control unit 251, a position acquisition unit 253 that acquires the current geographical position of the portable information device 200, and an augmented reality information acquisition unit that acquires augmented reality information from the server 500. 255, a display control unit 257 that controls the display unit 161, a different part extraction unit 271, and a process execution unit 273.

  The imaging control unit 251 includes an imaging control unit 259 that controls the camera 202. The imaging control unit 259 controls the camera 202, acquires a captured image output by the camera 202 imaging a subject, and extracts the acquired captured image from the display control unit 257, the augmented reality information acquisition unit 255, and the difference portion extraction. Output to the unit 271. The display control unit 257 controls the display unit 206 in response to the captured image input from the imaging control unit 259 and causes the display unit 206 to display the captured image.

  The position acquisition unit 253 controls the position detection unit 209, causes the position detection unit 209 to measure the current position, and acquires position information indicating the current position output by the position detection unit 209. The position acquisition unit 253 outputs the acquired position information to the augmented reality information acquisition unit 255.

  The augmented reality information acquisition unit 255 acquires augmented reality information from the server 500. The augmented reality information acquisition unit 255 includes a position reference acquisition unit 261 and a subject reference acquisition unit 263. The position reference acquisition unit 261 transmits an augmented reality information transmission request including the position information input from the position acquisition unit 253 to the server 500 via the wireless LAN I / F 208. The IP address of server 500 may be stored in advance in flash memory 203 of portable information device 200. Although details of the operation of the server 500 that receives the augmented reality information transmission request will be described later, the extended information including the position information indicating the position existing within a predetermined range from the position specified by the position information included in the augmented reality information transmission request. Real information is extracted, and one or more extracted augmented reality information is returned. When the wireless LAN I / F 408 receives one or more augmented reality information from the server 500, the augmented reality information acquisition unit 255 outputs the one or more augmented reality information to the subject reference acquisition unit 263.

  The subject reference acquisition unit 263 receives a captured image from the imaging control unit 259 and receives one or more augmented reality information from the position reference acquisition unit 261. The subject reference acquisition unit 263 extracts, from one or more augmented reality information input from the position reference acquisition unit 261, augmented reality information including a background image in which the subject is the same as the captured image input from the imaging control unit 259. To do. The subject reference acquisition unit 263 extracts a characteristic shape from each of the captured image and the background image set in the content item of the augmented reality information, and the same or similar shape as the characteristic shape of the captured image is the background. If it is included in the characteristic shape of the image, the augmented reality information is determined to be augmented reality information including a background image having the same subject as the captured image. The subject reference acquisition unit 263 outputs augmented reality information including a background image having the same subject as the captured image input from the imaging control unit 259 to the display control unit 257 and the different portion extraction unit 271. The subject reference acquisition unit 263 displays the display control unit 257 when there is no augmented reality information including a background image in which the captured image and the subject are the same among the one or more augmented reality information input from the position reference acquisition unit 261. And the augmented reality information is not output to the different part extraction unit 271.

  For example, by generating a binarized image obtained by binarizing the image or an edge image obtained by differentiating the image, the contour of the subject included in the image can be extracted. What is necessary is just to extract the shape of the extracted outline as a characteristic shape. Whether the characteristic shapes are the same or not includes a similar case because the sizes may be different. The edge image is an image in which the value of a pixel whose density difference between adjacent pixels is equal to or greater than a predetermined threshold is “1” and the value of a pixel smaller than the predetermined threshold is “0”. For example, when the background image and the captured image include a whiteboard as a subject, each image includes the outline of the whiteboard as a characteristic shape.

  Note that if the position detection unit 209 cannot measure the current position, the position acquisition unit 253 cannot acquire position information. In this case, the position reference acquisition unit 261 receives all the augmented reality information stored therein from the server 500 and outputs it to the subject reference acquisition unit 263. In addition, the object reference acquisition unit 263 transmits an augmented reality information transmission request including the captured image input from the imaging control unit 259 to the server 500 via the wireless LAN I / F 208, and the server 500 transmits the augmented reality information transmission request. The augmented reality information in which the captured image and the subject included in the image are the same may be extracted and returned. In this case, the subject reference acquisition unit 263 receives the augmented reality information returned from the server 500 and outputs it to the display control unit 257.

  The display control unit 257 displays the captured image on the display unit 206 in response to the input of the captured image from the imaging control unit 259. When the augmented reality information is input from the augmented reality information acquisition unit 255, the display control unit 257 displays the related information set in the related information item included in the augmented reality information on the display unit 206.

  The display control unit 257 includes a real display area determination unit 258. Based on the area information set in the area information item of the augmented reality information, the reality display area determining unit 258 specifies the relative position with respect to the subject in the background image set in the content item of the augmented reality information, and performs imaging. An area present at the specified relative position is determined as a real display area with reference to the subject in the image. In addition, the real display area determination unit 258 outputs a real display area for displaying related information in the captured image to the update control unit 277 described later.

  The display control unit 257 displays related information set in the related information item of the augmented reality information in the actual display area determined in the captured image. Specifically, the display control unit 257 generates a composite image obtained by combining the images of the related information in the actual display area in the captured image, and displays the generated composite image on the display unit 206.

  The real display area determination unit 258 specifies an area specified by the area information set in the area information item in the background image set in the content item included in the augmented reality information, and includes the captured image in the captured image. Thus, an area similar to the area specified in the background image may be determined as an area for displaying related information.

  Here, when the user captures an image using the camera 202 of the portable information device 200 in the conference room A with the whiteboard as the subject, the location information indicating the location of the conference room A is set in the location information item, and the whiteboard is the subject. The background image captured in this way is set as the content item, the image that is the conference material is set as the related information, and the display area including the entire drawing surface of the whiteboard in the background image is set as the area information item A case where the augmented reality information is received from the server 500 will be described as an example. In this case, a synthesized image obtained by synthesizing the image of the conference material as related information on the drawing surface of the whiteboard in the captured image is displayed on the display unit 206. Actually, even if no character or the like is written on the drawing surface of the whiteboard or an image is not projected, the display unit 206 of the portable information device 200 displays the whiteboard in the captured image. A composite image obtained by combining the images of the conference materials is displayed in the drawing surface area.

  The different portion extraction unit 271 receives a captured image from the imaging control unit 251 and receives augmented reality information from the augmented reality information acquisition unit 255. The different part extraction unit 271 determines whether or not the background image and the captured image included in the augmented reality information are partially different. When determining that the background image and the captured image are partially different, the different portion extraction unit 271 outputs an update instruction including the captured image and the augmented reality information to the process execution unit 273.

  When the update instruction is input from the different part extraction unit 271, the process execution unit 273 executes a predetermined process. The process execution unit 273 includes a notification unit 275 and an update control unit 277. The notification unit 275 notifies the user that the augmented reality information needs to be updated in response to the update instruction input from the different part extraction unit 271. For example, a message “Do you want to update augmented reality information with this image?” Is displayed superimposed on the image displayed on the display unit 206. When the user inputs an instruction to update augmented reality information to the operation unit 207, the notification unit 275 outputs an update instruction including the captured image and the augmented reality information to the update control unit 277.

  The update control unit 277 receives an update instruction including a captured image and augmented reality information from the notification unit 275, and receives an area for displaying related information in the captured image from the display control unit 257. The update control unit 277 updates the content item of the augmented reality information with the captured image, updates the area information item of the augmented reality information to the actual display region, and issues an augmented reality information update request including the augmented reality information after the update. The data is transmitted to the server 500 via the wireless LAN I / F 208.

  FIG. 9 is a block diagram illustrating an example of detailed functions of the different portion extraction unit 271. Referring to FIG. 9, different portion extraction unit 271 includes a division unit 281, a comparison unit 283, and a different portion determination unit 285.

  The dividing unit 281 divides the captured image input from the imaging control unit 259 into a plurality of partial images. Although the number of divisions is not limited, for example, it is preferable to divide into a plurality of equal parts in the vertical and horizontal directions.

  The comparison unit 283 compares, for each of the plurality of partial images, at least a part of the background image set in the content item of the augmented reality information. The comparison unit 283 compares at least a part of the partial image and the background image by pattern matching. If the partial image is similar or identical to at least a part of the background image, the comparison unit 283 determines the partial image as a similar portion and outputs a partial image number for identifying the similar portion to the different portion determination unit 285. If the background image does not include the same or similar portion as the partial image, the background image is determined as a dissimilar portion, and the partial image number of the dissimilar portion is output to the different portion determining unit 285.

  The different part determination unit 285 determines a dissimilar part as a different part when there is a dissimilar part surrounded by the similar part in the plurality of partial images. The different part determination unit 285 outputs a notification instruction to the notification unit 275 when determining the different part.

  FIG. 10 is a diagram illustrating an example of a background image. The background image shown in FIG. 10 is a background image set in the item of augmented reality information content registered in the server 500, and is a background image captured in the conference room A using a whiteboard as a subject. Referring to FIG. 10, background image 600 includes white board 601, clock 603, and MFP 605 as subjects. The white board 601 includes a drawing surface 601.

  FIG. 11 is a diagram illustrating an example of a captured image. The captured image shown in FIG. 11 is an image obtained by capturing an image of the whiteboard 601 as a subject in the portable information device 200 in the conference room A. Referring to FIG. 11, captured image 700 includes white board 601, clock 603, MFP 605, and poster 607 as subjects. The white board 601 includes a drawing surface 601.

  The captured image shown in FIG. 11 is an image obtained by capturing in the conference room A, and the captured position is the conference room A. The captured image 700 includes a whiteboard 601 as a subject.

  FIG. 12 is a diagram illustrating an example of a composite image. The composite image illustrated in FIG. 12 is generated when the captured image illustrated in FIG. 11 is acquired in the portable information device 200 and the augmented reality information including the background image illustrated in FIG. Shows the composite image displayed. Referring to FIG. 12, related information 610 is included in the area of drawing surface 601A of whiteboard 601 in the captured image shown in FIG.

  As described above, when the user carrying the portable information device 200 images the whiteboard 601 as a subject in the conference room A, the augmented reality information including the background image 600 shown in FIG. 10 is acquired from the server 500, and FIG. The composite image shown is displayed on the display unit 206 of the portable information device 200. Therefore, actually, even if nothing is drawn on the drawing surface 601A of the whiteboard 601, an image of related information set in the related information item of the augmented reality information prepared in advance is synthesized on the drawing surface 601A. The composite image can be displayed on the display unit 206 of the portable information device 200.

  The background image 700 includes a poster 607 as a subject, but the captured image 600 does not include it. Therefore, the background image set in the augmented reality information content item acquired from the server 500 is updated in the captured image 700 shown in FIG. 11, and the display region set in the region information item is captured image. 700 is updated with area information indicating the area of the drawing surface 601A in 700, and the updated augmented reality information is registered in the server 500. Therefore, in the augmented reality information registered in the server 500, even when the background image set in the content item is captured and the state of the place changes, the background image changes after the state changes. Since the captured image is updated with the captured image, the augmented reality information registered in the server 500 can be maintained in the latest state.

  There are various methods for registering augmented reality information in the server 500. Here, the portable information device 200 registers basic information, which is a part of augmented reality information, in the server 500, and the MFP 100 stores the basic information. A case where the augmented reality information is registered in the server 500 using the above will be described as an example. In this case, portable information device 200 functions as a basic information registration device that registers basic information in server 500, and MFP 100 functions as an augmented reality information registration device that registers augmented reality information in server 500.

  FIG. 13 is a diagram illustrating an example of a format of basic information. Referring to FIG. 13, the basic information includes an item of position information that is a part of augmented reality information and an item of content.

  FIG. 14 is a block diagram illustrating an example of a function of the CPU 201 included in the portable information device 200 that functions as a basic information registration device. The functions shown in FIG. 14 are functions formed in the CPU 201 when the CPU 201 included in the portable information device 200 functioning as a basic information registration device executes a basic information registration program stored in the flash memory 203. In addition, the same code | symbol is attached | subjected to the same function as the function shown in FIG. 8, and the overlapping description is not repeated.

  Referring to FIG. 14, CPU 201 included in portable information device 200 that functions as a basic information registration device includes a position acquisition unit 253, an imaging control unit 251, and a basic information registration unit 287. The imaging control unit 251 acquires a captured image that the camera 202 captures and outputs a subject, and outputs the acquired captured image to the basic information registration unit 287. The position acquisition unit 253 acquires position information indicating the current position output by the position detection unit 209 when the camera 202 images the subject, and outputs the acquired position information to the basic information registration unit 287.

  The basic information registration unit 287 sets the position information input from the position acquisition unit 253 in the item of position information. Basic information is generated by setting a captured image input from the imaging control unit 259 as a content item, and a basic information registration request including the generated basic information is transmitted to the server 500 via the wireless LAN I / F 208. To do.

  For example, if a user carries the portable information device 200 and images a whiteboard existing in the conference room A as a subject in the conference room A, the basic information including the location information of the conference room A and the whiteboard as the subject And the captured image are registered in the server 500.

  FIG. 15 is a block diagram illustrating an example of functions of a CPU provided in an MFP that functions as an augmented reality information registration apparatus. The functions shown in FIG. 15 are functions formed in the CPU 111 when the CPU 111 provided in the MFP 100 functioning as an augmented reality information registration apparatus executes the augmented reality information registration program stored in the ROM 113, the HDD 115, and the CD-ROM 118. is there. Referring to FIG. 15, CPU 111 includes a generation content acquisition unit 51, a region determination unit 53, a related information acquisition unit 55, an augmented reality information generation unit 57, and a registration unit 59.

The generation-side content acquisition unit 51 includes a position reception unit 61 and a basic information acquisition unit 63.
The position receiving unit 61 receives a geographical position designated by the user. The position receiving unit 61 displays a two-dimensional map on the display unit 161 and receives a position indicated by the user on the displayed map. For example, when the location of the conference room A is indicated on the displayed map, the geographical position of the conference room A is received. The position reception unit 61 outputs position information indicating the received geographical position to the basic information acquisition unit 63. The position information indicating the geographical position of the conference room A is indicated by the latitude and longitude of the conference room A.

  The basic information acquisition unit 63 acquires basic information from the server 500 in response to position information input from the position reception unit 61, and sends the acquired basic information to the region determination unit 53 and the augmented reality information generation unit 57. Output. The basic information acquisition unit 63 transmits a basic information transmission request including the position information input from the position reception unit 61 to the server 500 via the wireless LAN I / F 208. The details of the server 500 that receives the basic information transmission request will be described later. Among the registered basic information, position information indicating a position within a predetermined range from the position specified by the position information included in the basic information transmission request. Reply with basic information including. The basic information acquisition unit 63 acquires basic information that the wireless LAN I / F 208 receives from the server 500. For example, when the user instructs the meeting room A, the basic information including the position information indicating the geographical position of the meeting room A and including the background image captured using the whiteboard as a subject is acquired.

  The region determination unit 53 includes a generation-side content display unit 65, a display region reception unit 67, and a preview unit 69. The generation-side content display unit 65 displays the background image set in the content item of the basic information on the display unit 161 in response to the input of the basic information from the basic information acquisition unit 63. Here, a case will be described as an example where the user instructs the location of the conference room A on the map, and a background image captured using the whiteboard as a subject is displayed on the display unit 161. The generation-side content display unit 65 outputs the background image to the display area receiving unit 67 and the preview unit 69.

  The display area receiving unit 67 receives a display area specified by the user. The display area is an area in the background image. If the user operates the operation unit 163 to point to a part of the background image displayed on the display unit 161, the display region receiving unit 67 receives the region designated by the user as the display region. The display area may be, for example, a point or an area surrounded by an arbitrary shape. Here, the display area is a rectangular area. The display area is an area for displaying related information described later. The related information is information related to the subject included in the background image. Therefore, the display area is an area including the subject included in the background image, or an area within a predetermined range from the subject included in the background image. For this reason, if the user indicates a part of the subject in the background image, a rectangular region including the subject may be accepted as the display region. The display area receiving unit 67 outputs the received display area to the preview unit 69 and the augmented reality information generating unit 57. For example, when the display area is a rectangular area in the background image, the coordinates of the two diagonal vertices of the rectangular area may be used.

  The preview unit 69 receives a background image from the generation-side content display unit 65, receives a display area from the display region receiving unit 67, and receives related information from the related information acquisition unit 55 described below. The preview unit 69 generates a composite image obtained by combining the related information image in the display area of the background image, and displays the generated composite image on the display unit 161. The user can confirm the position where the related information is displayed in the background image by looking at the related information displayed superimposed on the background image. In order to change the position where the related information is displayed, the operation unit 163 may be operated to input an instruction to change the display area. Thereby, the display area receiving unit 67 receives the changed display area.

  The related information acquisition unit 55 acquires the related information and outputs the acquired related information to the preview unit 69 and the augmented reality information generation unit 57. The related information is information determined by the user, and is a character, a symbol, a graphic or a photograph, or a combination thereof.

  The related information acquisition unit 55 includes an input reception unit 71, a document reading control unit 73, and a data acquisition unit 75. The input receiving unit 71 receives characters or symbols that the user inputs to the operation unit 163 as related information. The document reading control unit 73 receives, as related information, an image output by the document reading unit 130 reading a document. The data acquisition unit 75 receives data specified by the user among the data stored in the HDD 115 as related information. The data stored in the HDD 115 includes data such as a document or a graphic created by another computer, data such as a photograph obtained by imaging with a digital camera, and data downloaded from a server connected to the Internet. Here, a case where a user uses a sheet on which an image of data for presentation is formed for use in a conference as a document, causes the document reading unit 130 to read the image, and uses an image output from the document reading unit 130 as related information. Explained as an example.

  The augmented reality information generation unit 57 receives basic information from the basic information acquisition unit 63, receives a display area from the display area reception unit 67, and receives related information from the related information acquisition unit 55. The augmented reality information generation unit 57 sets the position information included in the basic information in the item of position information, sets the background image included in the basic information as the item of content, and the related information input from the related information acquisition unit 55 Is set in the related information item, and the augmented reality information is generated by setting the region information input from the display region receiving unit 67 as the region information item. The augmented reality information generation unit 57 transmits an augmented reality information registration request including the generated augmented reality information to the server 500 via the wireless LAN I / F 208.

FIG. 16 is a block diagram illustrating an example of functions of a CPU included in the server. The functions illustrated in FIG. 16 are functions formed in the CPU 501 when the CPU 501 included in the server 500 executes the augmented reality information management program stored in the ROM 502 or the HDD 504. Referring to FIG. 16, CPU 501 includes basic information registration request receiving unit 551, basic information registering unit 553, basic information transmission request receiving unit 555, basic information extracting unit 557, basic information transmitting unit 559, extended Reality information registration request reception unit 561, augmented reality information registration unit 563, augmented reality information transmission request reception unit 565, augmented reality information extraction unit 567, augmented reality information transmission unit 569, and augmented reality information update request reception unit 571, augmented reality information update unit 573,
including.

  The basic information registration request receiving unit 551 receives the basic information included in the basic information registration request in response to the communication unit 505 receiving the basic information registration request transmitted from the portable information device 200 that functions as the basic information registration device. The information is output to the information registration unit 553. The basic information registration unit 553 registers the basic information by storing the basic information in the HDD 504.

  Basic information transmission request receiving unit 555 extracts basic information from the location information included in the basic information transmission request in response to communication unit 505 receiving a basic information transmission request transmitted by MFP 100 functioning as an augmented reality information registration device. Output to the unit 557. The basic information extracting unit 557 indicates position information indicating a position within a predetermined range from the position information included in the basic information transmission request from the basic information stored in the HDD 504 in response to the input of the position information. Basic information including is extracted. The basic information extraction unit 557 outputs the extracted basic information to the basic information transmission unit 559. Basic information transmission unit 559 transmits the basic information extracted by basic information extraction unit 557 to MFP 100 that has transmitted the basic information transmission request via communication unit 505.

  The augmented reality information registration request reception unit 561 receives the augmented reality information registration request included in the augmented reality information registration request when the communication unit 505 receives the augmented reality information registration request transmitted by the MFP 100 functioning as an augmented reality information registration device. Is output to the augmented reality information registration unit 563. The augmented reality information registration unit 563 registers the augmented reality information by storing the augmented reality information in the HDD 504.

  The augmented reality information transmission request receiving unit 565 is a position included in the augmented reality information transmission request when the communication unit 505 receives the augmented reality information transmission request transmitted by the portable information device 200 functioning as an augmented reality display device. The information is output to the augmented reality information extraction unit 567. The augmented reality information extraction unit 567 selects a position within a predetermined range from the position information included in the augmented reality information transmission request from the augmented reality information stored in the HDD 504 in response to the input of the position information. Augmented reality information including the position information shown is extracted. The augmented reality information extraction unit 567 outputs the extracted augmented reality information to the augmented reality information transmission unit 569. The augmented reality information transmission unit 569 transmits the augmented reality information extracted by the augmented reality information extraction unit 567 to the portable information device 200 that has transmitted the augmented reality information transmission request via the communication unit 505.

  The augmented reality information update request receiving unit 571 receives the augmented reality information update request transmitted from the portable information device 200 that functions as the augmented reality display device by the communication unit 505, and stores the augmented reality information stored in the HDD 504. Among them, the augmented reality information having the same position information and related information as the augmented reality information included in the augmented reality information update request is updated with the augmented reality information included in the augmented reality information update request. Specifically, of the augmented reality information stored in the HDD 504, augmented reality information having the same position information and related information as the augmented reality information included in the augmented reality information update request is included in the augmented reality information update request. Update with current reality information.

  FIG. 17 is a flowchart illustrating an example of the flow of basic information registration processing. The basic information registration process is a process executed by the CPU 201 when the CPU 201 included in the portable information device 200 functioning as a basic information registration device executes a basic information registration program stored in the flash memory 203. With reference to FIG. 17, the CPU 201 controls the camera 202 to image a subject (step S01). When the user indicates a shutter key provided in the operation unit 207, the CPU 201 causes the camera 202 to image the subject. In the next step S02, a captured image output by the camera 202 is acquired, and the process proceeds to step S03. In step S03, position information indicating the current position output by the position detection unit 209 at the time when the camera 202 images the subject is acquired.

  Then, basic information is generated (step S04). The basic information is generated by setting the captured image acquired in step S02 as a content item of basic information and setting the positional information acquired in step S03 as a positional information item.

  In the next step S05, the generated basic information is registered in the server 500. Specifically, a basic information registration request including basic information is transmitted to the server 500 via the wireless LAN I / F 208.

  For example, if the user carries the portable information device 200 and takes an image of a whiteboard existing in the conference room A as a subject in the conference room A, the captured image using the whiteboard as a subject, the position information of the conference room A, Is registered in the server 500.

  FIG. 18 is a flowchart illustrating an example of the flow of the augmented reality information registration process. The augmented reality information registration process is a process executed by the CPU 111 when the CPU 111 included in the MFP 100 functioning as an augmented reality information registration apparatus executes the augmented reality information registration program stored in the ROM 113, the HDD 115, and the CD-ROM 118. is there. With reference to FIG. 18, the CPU 111 displays a two-dimensional map on the display unit 161 (step S <b> 11). Display a map downloaded from a computer connected to the Internet. Further, when a map is stored in advance in the HDD 115, the map may be displayed.

  In the next step S12, designation of the position is accepted. The process waits until a position is accepted (NO in step S12). If a position is accepted (YES in step S12), the process proceeds to step S13. A position indicated by the user using the operation unit 163 in the map is received.

  In step S13, position information indicating the accepted position is acquired, and the process proceeds to step S14. Here, a case where the location of the conference room A is instructed in the displayed map will be described as an example. In this case, the latitude and longitude of the meeting room A are acquired as position information indicating the geographical position of the meeting room A.

  In step S14, basic information corresponding to the position information acquired in step S13 is acquired. Specifically, a basic information transmission request including position information is transmitted to server 500 via communication I / F unit 112. In the server 500 that receives the basic information transmission request, among the basic information registered in the server 500, position information indicating a position within a predetermined range from the position specified by the position information included in the basic information transmission request. Reply with basic information including. In the next step S <b> 15, basic information received by the wireless LAN I / F 208 from the server 500 is acquired. For example, the basic information registered in the server 500 by the portable information device 200 functioning as a basic information registration device, including location information indicating the geographical location of the conference room A, and a background imaged using a whiteboard as a subject Get basic information including images.

  In step S16, the background image set in the content item of the basic information is displayed on the display unit 161. As a result, a background image captured using the whiteboard as a subject is displayed on the display unit 161. Then, the display area designated by the user is received (step S17). The display area is an area in the background image. If the user operates the operation unit 163 to point to a part of the background image displayed on the display unit 161, the region designated by the user is accepted as the display region. The display area may be a point or an area surrounded by an arbitrary shape. Here, the display area is a rectangular area. If the user indicates a part of the subject in the background image, a rectangular area including the subject may be accepted as the display area.

  In step S18, it is determined whether a document reading instruction has been accepted. When operation unit 163 detects a document reading instruction input by the user, it accepts the document reading instruction. If a document reading instruction is accepted, the process proceeds to step S19. If not, the process proceeds to step S21. In step S19, the original reading unit 130 is caused to read the original. Then, the image obtained by reading the document is set as related information (step S20, the process proceeds to step S25).

  In step S21, it is determined whether or not a character has been input. When operation unit 163 detects a character input by the user, it determines that a character has been input. If a character has been input, the process proceeds to step S22; otherwise, the process proceeds to step S23. In step S22, the character string detected by the operation unit 163 is set as related information, and the process proceeds to step S25.

  In step S23, it is determined whether designation of data has been accepted. When the operation unit 163 detects an operation for designating data input by the user, the data designation is accepted. If the designation of data is accepted, the process proceeds to step S24. If not, the process returns to step S19. In step S24, the data specified by the user among the data stored in HDD 115 is false, and the process proceeds to step S25. The data stored in the HDD 115 includes data such as a document or a graphic created by another computer, data such as a photograph obtained by imaging with a digital camera, and data downloaded from a server connected to the Internet. Here, a case where a user uses a sheet on which an image of data for presentation is formed for use in a conference as a document, causes the document reading unit 130 to read the image, and uses an image output from the document reading unit 130 as related information. Explained as an example.

  In step S25, the related information is displayed as a preview. Specifically, the image of the related information set in any of step S20, step S22, and step S24 is displayed in the display area accepted in step S17 in the background image displayed on display unit 161 in step S16. To do. In step S26, it is determined whether or not permission from the user has been accepted. If permission is accepted (YES in step S26), the process proceeds to step S28; otherwise (NO in step S26), the process proceeds to step S27. In step S27, the correction of the display area is accepted, and the process returns to step S25. When the process proceeds from step S27 to step S25, in step S25, an image of related information is displayed in the display area after being corrected in step S27.

  In step S28, augmented reality information is generated, and the process proceeds to step S29. Each of the position information and the background image included in the basic information acquired in step S14 is set in the position information item and the content item, and the related information set by the shift between step S20, step S22 and step S24 is related information. The augmented reality information is generated by setting the display area received in step S17 as the area information item.

  In the next step S29, the generated augmented reality information is registered in the server 500, and the process ends. Specifically, an augmented reality information registration request including augmented reality information is transmitted to the server 500 via the wireless LAN I / F 208.

  FIG. 19 is a flowchart illustrating an example of the flow of the augmented reality information management process. The augmented reality information management process is a process executed by the CPU 501 by the CPU 501 included in the server 500 executing the augmented reality information management program stored in the ROM 502 or the HDD 504. Referring to FIG. 19, CPU 501 determines whether or not a basic information registration request has been received (step S71). If communication unit 505 receives a basic information registration request from portable information device 200 functioning as a basic information registration device, the process proceeds to step S72; otherwise, the process proceeds to step S73. In step S72, the basic information included in the received basic information registration request is stored in HDD 504 to register the basic information, and the process proceeds to step S73.

  In step S73, it is determined whether a basic information transmission request has been received. If communication unit 505 receives a basic information transmission request from MFP 100 functioning as an augmented reality information generation device, the process proceeds to step S74; otherwise, the process proceeds to step S77. In step S74, position information included in the received basic information transmission request is acquired. Then, basic information including position information indicating a position within a predetermined range from the position determined by the acquired position information is extracted from the basic information stored in the HDD 504 (step S75). In step S76, the extracted basic information is transmitted to MFP 100 that has transmitted the basic information transmission request via communication unit 505, and the process proceeds to step S77.

  In step S77, it is determined whether an augmented reality information registration request is received. If communication unit 505 receives an augmented reality information registration request from MFP 100 functioning as an augmented reality information generation device, the process proceeds to step S78; otherwise, the process proceeds to step S79. In step S78, augmented reality information is registered by storing augmented reality information included in the received augmented reality information registration request in HDD 504, and the process proceeds to step S79.

  In step S79, it is determined whether an augmented reality information transmission request is received. If communication unit 505 receives an augmented reality information transmission request from portable information device 200 functioning as an augmented reality display device, the process proceeds to step S80; otherwise, the process proceeds to step S83.

  In step S80, the position information set in the item of position information of the augmented reality information included in the received augmented reality information transmission request is acquired. Then, the augmented reality information including the position information indicating the position within a predetermined range from the position determined by the acquired position information is extracted from the augmented reality information stored in the HDD 504 (step S81).

  In the next step S82, the extracted augmented reality information is transmitted to the portable information device 200 that has transmitted the augmented reality information transmission request via the communication unit 505, and the process proceeds to step S83.

  In step S83, it is determined whether an augmented reality information update request has been received. If communication unit 505 receives an augmented reality information update request from portable information device 200 functioning as an augmented reality display device, the process proceeds to step S84. If not, the process ends. In step S84, the augmented reality information specified by the augmented reality information update request is updated, and the process ends. Specifically, of the augmented reality information stored in the HDD 504, augmented reality information having the same position information and related information as the augmented reality information included in the augmented reality information update request is included in the augmented reality information update request. Update with current reality information.

  FIG. 20 is a flowchart illustrating an example of the flow of the augmented reality display process. The augmented reality display process is a process executed by the CPU 201 when the CPU 201 included in the portable information device 200 executes the augmented reality display program stored in the flash memory 203. Referring to FIG. 20, the CPU 201 controls the camera 202 to image a subject (step S31). When the user indicates a shutter key provided in the operation unit 207, the CPU 201 causes the camera 202 to image the subject. The process waits until the subject is imaged (NO in step S31). If the subject is imaged (YES in step S31), the process proceeds to step S32. In step S32, a captured image output by camera 202 is acquired, and the process proceeds to step S33. In step S33, the imaging direction is acquired. The direction detection unit 210 is controlled to cause the direction detection unit 210 to detect the imaging direction of the camera 202. In the next step S34, the position detection unit 209 is controlled to acquire position information indicating the current position output by the position detection unit 209 when the camera 202 images the subject.

  And augmented reality information is acquired (step S35). The augmented reality information transmission request including the position information acquired in step S34 is transmitted to the server 500 via the wireless LAN I / F 208, and the augmented reality information returned by the server 500 is acquired.

  The augmented reality information transmission request includes the imaging direction of the camera 202 in addition to the position information indicating the current position. In the server 500, the position information indicating the position within the range determined by the current position and the acquired orientation is obtained. The specified augmented reality information may be extracted. The imaging direction of the camera 202 controls the azimuth detection unit 210, causes the azimuth detection unit 210 to detect the imaging direction of the camera 202, and acquires the azimuth output when the azimuth detection unit 210 images the camera 202. When acquiring a plurality of augmented reality information in step S35, the plurality of augmented reality information acquired in step S35 is obtained by changing the imaging direction of the camera 202 of the portable information device 200 after acquiring the plurality of augmented reality information. Augmented reality information in which position information indicating a position within a range determined by the current position and the acquired orientation is determined may be extracted from the actual information.

  In the next step S36, it is determined whether or not the subject of the captured image captured in step S31 and acquired in step S32 matches the subject of the background image included in the augmented reality information. If a characteristic shape is extracted from each of the captured image and the background image and the same or similar shape as the characteristic shape of the captured image is included in the characteristic shape of the background image, the captured image and the augmented reality It is determined that the information subject is the same. For example, by generating a binarized image obtained by binarizing the image or an edge image obtained by differentiating the image, the contour of the subject included in the image can be extracted. What is necessary is just to extract the shape of the extracted outline as a characteristic shape. Whether the characteristic shapes are the same or not includes a similar case because the sizes may be different.

  If it is determined in step S36 that the subject is the same, the process proceeds to step S37; otherwise, the process proceeds to step S44. In step S44, the captured image is displayed on display 161, and the process proceeds to step S43.

  In step S37, the actual display area in the captured image is specified. Specifically, based on the area information set in the area information item of augmented reality information, the relative position with respect to the subject in the background image set in the content item of augmented reality information is specified, and An area present at the specified relative position is determined as an actual display area with reference to the subject.

  In the next step S38, a synthesized image is generated by synthesizing the captured image with the related information image in the actual display area in the identified captured image. In the next step S39, the generated composite image is displayed on the display unit 161, and the process proceeds to step S40.

  In step S40, a different portion determination process is executed. Although the details of the different part determination process will be described later, it is a process of determining whether or not there is a different part between the captured image and the background image. In the next step S41, the process is branched depending on the result of the different portion determination process. If it is determined that a different part exists, the process proceeds to step S42; otherwise, the process proceeds to step S43.

  In step S42, the augmented reality information is updated, and the process proceeds to step S43. Specifically, the content item of augmented reality information acquired in step S35 is updated with the captured image acquired in step S32, and the item of region information is updated with the actual display region specified in step S37. An augmented reality information update request including the augmented reality information after update is transmitted to the server 500 via the wireless LAN I / F 208.

  In step S43, it is determined whether an instruction to end the augmented reality display process has been received. If an end instruction is accepted, the process ends. If not, the process returns to step S31.

  If there are a plurality of augmented reality information in which the subject is determined to be the same in step S36, in step S37, a composite image obtained by synthesizing the related information images of the plurality of augmented reality information is displayed. When selecting from a plurality of related information images, a combined image obtained by combining only the selected one related information image may be displayed. In step S42, an augmented reality information update request for each of the plurality of augmented reality information may be transmitted to the server 500.

  FIG. 21 is a flowchart illustrating an example of the flow of the different portion determination process. The different portion determination process is a process executed in step S40 of FIG. Referring to FIG. 21, CPU 201 divides the captured image into a plurality of partial images (step S51). The number of divisions is not limited, but it is preferable to divide the number into a plurality of equal parts in the vertical and horizontal directions.

  Then, one of the plurality of partial images is selected as a processing target (step S52). In the next step S53, the partial image to be processed is compared with at least a part of the background image. For example, at least a part of the partial image and the background image are compared by pattern matching. Then, the process branches depending on the comparison result (step S54). If the background image includes the same or similar part as the partial image, the process proceeds to step S55; otherwise, the process proceeds to step S56. In step S55, “similar” is set for the partial image to be processed, and the process proceeds to step S57. In step S56, “not similar” is set for the partial image to be processed, and the process proceeds to step S57. . In step S57, it is determined whether there is a partial image that is not selected as a processing target. If there is an unprocessed partial image, the process returns to step S52; otherwise, the process proceeds to step S58. Hereinafter, a partial image set to “similar” is referred to as a similar portion, and a partial image set to “dissimilar” is referred to as a dissimilar portion.

  In step S58, it is determined whether there is a different portion. Specifically, when a dissimilar part surrounded by a similar part exists in a plurality of partial images, the dissimilar part is determined as a different part. If it is determined that a different portion exists, the process proceeds to S59; otherwise, the process proceeds to step S60. In step S59, it is determined that there is a different part, and the process returns to the augmented reality display process. In step S60, it is determined that there is no different part, and the process returns to the augmented reality display process.

<Modification>
In the embodiment described above, the portable information device 200 has been described as an example of the augmented reality display device. However, the HMD 400 may function as the augmented reality display device. In this case, the CPU 401 included in the HMD 400 has the same function as that of the CPU 201 included in the portable information device 200 illustrated in FIG. 8, but the display control unit 257 is different.

  The display control unit 257 included in the CPU 401 of the HMD 400 is input from the augmented reality information acquisition unit 255 without displaying the captured image on the display unit 206 in response to the input of the captured image from the imaging control unit 259. Based on the augmented reality information, the related information set in the related information item included in the augmented reality information is displayed on the display unit 404. The user wearing the HMD 400 actually sees the same image as the captured image through the display surface of the display unit 404. Therefore, an area corresponding to the actual display area determined in the captured image by the actual display area determining unit 258 is specified on the display surface of the display unit 404, and the related information is stored in the area specified on the display surface of the display unit 404. The image of is displayed.

  Here, if the user wearing the HMD 400 views the white board in the conference room A, the white board becomes the subject of the camera 402. In this case, position information indicating the position of the conference room A is set in the position information item, a background image captured using the whiteboard as a subject is set in the content item, and an image that is a meeting material is related information. The augmented reality information that is set and the area including the entire drawing surface of the whiteboard in the background image is set in the area information item is received from the server 500. When the image of the conference material, which is related information, is displayed on the display unit 404, the user displays the image on the display surface of the display unit 404 on the drawing surface of the whiteboard that is seen through the display surface of the display unit 404. You will see an image of the meeting material.

  In the modification, the CPU 401 provided in the HMD 400 executes the augmented reality display process shown in FIG. 20, but steps S38, S39, and S41 are different. In step S38, the CPU 401 included in the HMD 400 specifies an area on the display surface of the display unit 404 corresponding to the actual display area in the captured image specified in step S37 without generating a composite image. Then, in the next step S39, an image of related information is displayed in the area in the display surface of the specified display unit 404, and the process proceeds to step S40. In step S44, the process proceeds to step S40 without displaying anything.

  As described above, portable information device 200 according to the present embodiment functions as an augmented reality display device, and has a background that has the same subject as the subject in the captured image (first image) obtained by capturing the subject. Augmented reality information including the image (second image) is acquired from the server 500, and the relative display position of the subject in the captured image is the same as the relative position of the subject in the background image and the display area. And the related information is displayed in the actual display area. For this reason, since the related information related to the subject in the captured image by the augmented reality information is displayed at the relative position determined with respect to the subject in the captured image, information related to the subject in the captured image is displayed. Can be displayed at a predetermined position with respect to the subject.

  In addition, when a different portion that is different between the background image and the captured image is extracted, the background image of the augmented reality information is updated with the captured image, so that the user is notified. Etc., the image included in the augmented reality information can be updated to the latest state.

  Further, when a different portion that is different between the background image and the captured image is extracted, the user is notified, so that the user can be notified that a portion different from the subject, for example, the background or the like has changed. Further, it is possible to notify the user that the image included in the augmented reality information is updated to the latest state.

  Further, by comparing a part of the captured image (first image) with at least a part of the background image (second image) included in the augmented reality information, a different part that differs between the background image and the captured image is obtained. Since extraction is performed, it is possible to determine whether or not the subject is the same even if the entire captured image and background image do not match.

  Further, each of the plurality of partial images obtained by dividing the captured image is compared with a part of the background image, and is a partial image surrounded by a plurality of partial images similar to a part of the background image among the plurality of partial images, Since a partial image that is not similar to a part of the background image is determined as a different portion, it can be determined that the captured image and the background image are the same subject but different as a whole.

  Furthermore, since the augmented reality information in which the geographical position of the subject in the background image exists within a predetermined range from the geographical position at the time of capturing the subject is acquired, the processing target is selected from a plurality of augmented reality information. The augmented reality information can be narrowed down, and the augmented reality information including the background image having the same subject as the subject in the background image can be easily obtained.

  Even when the current position cannot be obtained, if the subject in the captured image is compared with the subject in the background image, the subject in the background image is the same as the subject in the background image from among a plurality of augmented reality information. Augmented reality information including a background image can be extracted. For this reason, even if it is a case where a portable information device is located in the place which cannot receive the signal from a GPS satellite, augmented reality information can be acquired.

  In addition, since the composite image in which the related information is superimposed on the captured image is generated and displayed, the related information can be associated with the subject in the captured image.

  When the HMD 400 functions as an augmented reality display device, a real display area is determined on the display surface of the display unit 404 based on a captured image (first image) obtained by capturing an image of the subject with the camera 402, and related information is displayed. Since the information is displayed in the actual display area in the display surface of the display unit 404, the related information can be displayed in association with the subject that the user sees through the display surface of the display unit 404.

  Further, the server 500 in the present embodiment stores augmented reality information, and the portable information device 200 functioning as an augmented reality display device acquires the augmented reality information from the server 500. For this reason, the user of the portable information device 200 can browse related information included in the augmented reality information registered in the server 500 by a user different from the user.

  In the augmented reality display system 1 according to the present embodiment, basic information is registered in the server 500 by the portable information device 200 that functions as a basic information registration device, and the basic information is related to the basic information by the MFP 100 that functions as an augmented reality information registration device. Augmented reality information in which information and area information are added is registered. For this reason, portable information device 200 that functions as an augmented reality display device can display related information registered by operating MFP 100 by the user who uses it or a user other than the user. Therefore, related information registered by an arbitrary user can be displayed in association with the subject.

  The MFP 100 can register augmented reality information using image data read by the document reading unit 130 as related information.

  Further, the MFP 100 can register augmented reality information using data specified by the user from among data stored in the HDD 115 or data stored in a computer connected to the network 2.

  In the above-described embodiment, the portable information device 200 is described as an example of the augmented reality information display device. However, the augmented reality display process illustrated in FIGS. 20 and 21 is performed by the portable information device 200. It goes without saying that the invention can be specified as an augmented reality information display program that causes the CPU 201 of the portable information device 200 to execute the actual information display method or the augmented reality information display method.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

1 augmented reality display system, 2 network, 3 wireless station, 100 MFP, 200, 200A, 200B portable information device, 300 PC, 400 HMD, 500 server, 110 main circuit, 111 CPU, 112 communication I / F unit, 113 ROM 114 RAM, 115 HDD 115 Facsimile unit, 117 External storage device, 118 CD-ROM, 120 Automatic document feeder, 130 Document reading unit, 140 Image forming unit, 150 Paper feed unit, 155 Post-processing unit, 160 Operation panel, 161 Display unit, 163 operation unit, 201 CPU, 202 camera, 203 flash memory, 204 wireless communication unit, 205 call unit, 206 display unit, 207 operation unit, 208 wireless LAN I / F, 209 position detection unit, 210 direction detection unit, 211 External storage device, 401 CPU , 402 Camera, 403 Flash memory, 404 Display unit, 405 Wireless LAN I / F, 501 CPU, 502 ROM, 503 RAM, 504 HDD, 505 Communication unit, 506 Display unit, 507 Operation unit, 51 Production-side content acquisition unit, 53 Area determination unit, 55 related information acquisition unit, 57 augmented reality information generation unit, 59 registration unit, 61 position reception unit, 63 basic information acquisition unit, 65 generation content display unit, 67 display region reception unit, 69 preview unit, 71 Input receiving unit, 73 Document reading control unit, 75 Data acquisition unit, 251 Imaging control unit, 253 Position acquisition unit, 255 Augmented reality information acquisition unit, 257 Display control unit, 258 Reality display area determination unit, 259 Imaging control unit, 261 Position reference acquisition unit, 263 subject reference acquisition unit, 271 different part extraction unit, 273 process execution unit, 275 notification unit, 277 update control unit, 281 division unit, 283 comparison unit, 285 different part determination unit, 287 basic information registration unit, 551 basic information registration request reception unit, 553 basic information registration unit, 555 basic information transmission request reception unit 557 Basic information extraction unit, 559 Basic information transmission unit, 561 Augmented reality information registration request reception unit, 563 Augmented reality information registration unit, 565 Augmented reality information transmission request reception unit, 567 Augmented reality information extraction unit, 568 Augmented reality information transmission Department.

Claims (15)

Imaging means for imaging a subject and outputting a first image;
Augmented reality information obtaining means for obtaining augmented reality information including a second image having the same subject as the subject in the obtained first image;
The augmented reality information includes, in addition to the second image, region information indicating a display region determined at a position relative to the subject in the second image, and related information related to the subject. ,
A real display area determining means for determining a real display area whose relative position with respect to the subject in the first image is the same as the relative position between the subject in the second image and the display area;
Display control means for displaying the related information included in the augmented reality information in the determined reality display area;
A different part extracting means for extracting different parts different between the second image and the first image;
An augmented reality display device comprising: a process execution unit that executes a predetermined process when a different part is extracted by the different part extraction unit.   The augmented reality display device according to claim 1, wherein the processing execution unit includes an update control unit that updates the second image of the augmented reality information with the first image.   The augmented reality display device according to claim 1, wherein the processing execution unit includes a notification unit that notifies a user that the second image is different from the first image.   The difference portion extraction means is a comparison means for comparing a part of the first image output by the imaging means picking up the subject with at least a part of the second image included in the acquired augmented reality information. The augmented reality display device according to claim 1, comprising: The different part extracting means includes a dividing means for dividing the first image into a plurality of partial images;
Comparing means for comparing each of the plurality of partial images with a portion of the second image;
Of the plurality of partial images, a partial image surrounded by a plurality of partial images similar to a part of the second image and a partial image not similar to a part of the second image is determined as a different part. The augmented reality display device according to claim 4, further comprising: a different-part determining unit. The augmented reality information further includes position information indicating a geographical position,
The augmented reality display device further includes position acquisition means for acquiring a geographical position where the augmented reality display device actually exists as a current position,
The augmented reality information acquisition means further includes a position reference acquisition means for acquiring augmented reality information associated with a geographical position within a predetermined range from the acquired current position based on the acquired current position. An augmented reality display device according to any one of claims 1 to 5.   The augmented reality display according to claim 6, wherein the augmented reality information acquiring unit acquires the augmented reality information based on a first image output from the imaging unit when the current position cannot be acquired by the position acquiring unit. apparatus. It further comprises display means for displaying an image,
The display control means includes composite image generation means for generating a composite image in which the related information included in the augmented reality information is superimposed on the first image,
The augmented reality display device according to claim 1, wherein the generated composite image is displayed on the display unit. Arranged at a position determined with reference to the optical axis of the imaging means, further comprising a transmissive display means,
The actual display area determining means determines the actual display area on the display surface of the display means based on the first image output by the imaging means,
The augmented reality display device according to any one of claims 1 to 7, wherein the display control means displays related information included in the augmented reality information in an actual form display area in a display surface of the display means. A communication control means capable of communicating with the server storing the augmented reality information;
The augmented reality display device according to claim 1, wherein the augmented reality information acquisition unit acquires the augmented reality information from the server. An augmented reality display system comprising the augmented reality display device according to claim 10 and a basic information registration device and an augmented reality information registration device capable of communicating with the server,
The basic information registration device includes a basic imaging unit that images a subject and outputs an image;
Basic position acquisition means for acquiring the geographical position where the basic information registration device actually exists as a current position;
The basic information for registering basic information including the second image and the current position acquired by the basic position acquisition unit in the server is an image output by the basic imaging unit imaging the subject. A registration means,
The augmented reality information registration device includes a position receiving unit that receives designation of a geographical position;
Basic information acquisition means for acquiring, from the server, basic information including position information in which the received geographical position is within a predetermined range;
Generation-side display means for displaying a second image included in the acquired basic information;
Display area accepting means for accepting designation of a display area in the second image;
Related information acquisition means for acquiring related information;
Augmented reality information generating means for generating augmented reality information in which the acquired related information and area information indicating the display area received by the display area receiving means are added to the basic information;
An augmented reality display system comprising: registration means for registering the generated augmented reality information in the server. The augmented reality information registration device further includes document reading means for outputting image data obtained by optically reading a document and converting it into electronic data,
The augmented reality display system according to claim 11, wherein the related information acquisition unit acquires image data output from the document reading unit as the related information. The augmented reality information registration device further includes data storage means for storing data,
The augmented reality display system according to claim 11 or 12, wherein the related information acquisition unit acquires data specified by a user from the data stored in the data storage unit as the related information. An imaging step of acquiring a first image obtained by imaging a subject;
An augmented reality information acquisition step of acquiring augmented reality information including a second image having the same subject as the subject in the acquired first image;
The augmented reality information includes, in addition to the second image, region information indicating a display region determined at a position relative to the subject in the second image, and related information related to the subject. ,
A real display area determining step for determining a real display area in which a relative position with respect to a subject in the first image is the same as a relative position between the subject in the second image and the display area;
A display control step of displaying the related information included in the augmented reality information in the determined reality display area;
A different part extraction step of extracting different parts different between the second image and the first image;
An augmented reality display method comprising: a process execution step of executing a predetermined process when a different part is extracted in the different part extraction step. An imaging step of acquiring a first image obtained by imaging a subject;
An augmented reality information acquisition step of acquiring augmented reality information including a second image having the same subject as the subject in the acquired first image;
The augmented reality information includes, in addition to the second image, region information indicating a display region determined at a position relative to the subject in the second image, and related information related to the subject. ,
A real display area determining step for determining a real display area in which a relative position with respect to a subject in the first image is the same as a relative position between the subject in the second image and the display area;
A display control step of displaying the related information included in the augmented reality information in the determined reality display area;
A different part extraction step of extracting different parts different between the second image and the first image;
An augmented reality display program for causing a computer to execute a process execution step of executing a predetermined process when a different part is extracted in the different part extraction step.

Images