JP2016081313A - Image processing system and method of displaying virtual reality - Google Patents

Abstract

PROBLEM TO BE SOLVED: To notify other users of a modified portion.SOLUTION: An image processing device includes a substituting unit for converting original data into substitution data and an output control unit for outputting the substitution data. A server includes; an associated portion reply unit 559 that, upon receipt of a transmission request containing an AR tag and acquired data from any one of a plurality of portable information devices, transmits an associated portion associated with the AR tag contained in the transmission request to the portable information device that has transmitted the transmission request; an added portion extraction unit 563 that extracts an added portion that is present in the acquired data but not present in the substitution data; and an associated portion updating unit 571 that updates the associated portion by adding the added portion to the associated portion associated with the AR tag contained in the transmission request.SELECTED DRAWING: Figure 9

Description

  The present invention relates to an image processing system and a virtual reality display method, and in particular, an image processing system that displays an image corresponding to an AR tag in a real space imaged by a camera, and a virtual executed by the image processing system. The present invention relates to a reality display method.

  In recent years, technologies related to augmented reality have been developed. For example, Japanese Patent Laid-Open No. 2013-09287 discloses an acquisition unit that acquires information about a plurality of markers included in an image, and a display unit that superimposes a plurality of virtual objects respectively corresponding to the plurality of markers on the image. Display control means for controlling the display to be displayed on the display, and determination means for determining whether or not the plurality of virtual objects are in a specific combination, and the plurality of virtual objects are in a specific combination by the determination means The display control means controls the plurality of virtual objects to be superimposed on the image and displayed on the display unit in a specific arrangement corresponding to the specific combination. An image processing apparatus is described.

In the conventional image processing apparatus, since the virtual object is paired with the marker, if the virtual object is changed, the changed virtual object that is paired with the marker is displayed. However, when making the same change to a plurality of related virtual objects, there is a problem that the same change must be made for each of the plurality of virtual objects.
JP2013-09287A

  The present invention has been made to solve the above-described problems, and one of the objects of the present invention is to provide an image processing system capable of notifying another user of a changed portion.

  Another object of the present invention is to provide a virtual reality display method capable of notifying another user of a changed portion.

  In order to achieve the above object, according to one aspect of the present invention, an image processing system includes an image forming apparatus, a plurality of portable information devices, and a server, and the image forming apparatus includes: Replacement means for converting the original data into replacement data obtained by replacing at least a related portion of the original data with an AR tag, original identification information for identifying the original data, an AR tag, An AR tag registration unit that transmits registration information including a relevant part and replacement data to the server, and an output control unit that outputs the replacement data. The server receives the registration information from the image forming apparatus. Depending on the situation, the AR tag association means for associating the original data, the AR tag, the related part, and the replacement data included in the registration information, A In response to receiving a transmission request including a tag and acquired data from any of a plurality of portable information devices, a related part associated with the AR tag included in the transmission request is transmitted among the plurality of portable information devices. And a related part reply means for transmitting to the portable information device that transmitted the message, and an addition for extracting an additional part that exists in the acquired data included in the transmission request but does not exist in the replacement data associated with the AR tag included in the transmission request And a related part updating unit that updates the related part by adding the additional part to the related part associated with the AR tag included in the transmission request. Acquisition data acquisition means for acquiring replacement data output by the apparatus or data obtained by changing a part of the replacement data as acquisition data; AR tag extracting means for extracting the AR tag from the data, related part request means for transmitting a transmission request including the AR tag extracted from the acquired data and the acquired data to the server, and after transmitting the transmission request, the AR from the server Related part receiving means for receiving a related part associated with a tag, and display control means for displaying an image of the received related part at a position determined based on the position in the acquired data of the AR tag extracted from the acquired data And comprising.

  According to this aspect, replacement data output by the image forming apparatus or data obtained by changing a part of the replacement data is acquired as acquisition data by each of the plurality of portable information devices, and an AR tag is extracted from the acquisition data. When the related part associated with the AR tag is received from the server, an image of the related part is displayed at a position determined based on the position in the acquired data. For this reason, since the related part is displayed at the position defined by the AR tag in the acquired data obtained in the portable information device, the related part can be displayed in the acquired data. When the additional part is extracted from the acquired data received from the portable information device by the server, the related part is updated by adding the additional part to the related part associated with the AR tag. For this reason, since the related part after update is displayed in another portable information device that acquires replacement data, it is possible to notify another user of an additional part in which the replacement data has been changed. As a result, an image processing system capable of notifying another user of the changed part can be provided.

  Preferably, the image forming apparatus further includes data association registration means for registering, in the server, original association information in which the original data is associated with another original data different from the original data. When the first related part is updated by the partial updating unit, the first original data associated with the first related part and the second original associated with the original association information registered by the image forming apparatus Related original specifying means for specifying data, and when the second original data is specified by the related original specifying means, the related part updating means uses the second part extracted by the additional part extracting means as the second original data. Second by adding to the second related part associated with the original data Including the interlocking update means for updating the relevant portions.

  According to this aspect, when the first original data and the second original data associated by the image forming apparatus are specified, the second related portion associated with the second original data is set as the second original data. The second related part is updated by adding to the second related part associated with the data. For this reason, the same additional portion as the additional portion added to the related portion corresponding to the first original data can be added to the related portion corresponding to the second original data. As a result, the same additional part as the additional part added to one related part can be added to the other related part.

  Preferably, the image forming apparatus further includes a data acquisition unit that acquires data, and the data association registration unit associates each of the plurality of pages included in the acquired data with each other as a plurality of original data. Means.

  According to this aspect, since multiple pieces of original data included in the same data are related, the changes made to the replacement data corresponding to the original data of any page of the same data are replaced with the original data of other pages. Users can be notified of the data.

  Preferably, the data association registration unit includes a plurality of replacement data respectively output by the data output unit by a plurality of operations by the same user or a plurality of operations by a plurality of users belonging to the same group. The same user associating means for associating original data is included.

  According to this aspect, changes made to any of a plurality of replacement data output by a plurality of operations by the same user or a plurality of operations by one of a plurality of users belonging to the same group are changed to other replacement data. You can notify the viewer.

  Preferably, the data association registration unit selects the candidate data acquisition unit that acquires the original data from the server as candidate data and the original data corresponding to the replacement data output by the data output unit from the candidate data by the user. And operation association means for associating with the candidate data.

  According to this aspect, a change made to the replacement data corresponding to any of the plurality of original data associated by the user operating the image processing apparatus is notified to the user who sees the replacement data corresponding to the other original data. be able to.

  Preferably, in the server, the first related part is updated by the related part update unit based on the first transmission request received from the first portable information device among the plurality of portable information devices, and the original data specifying unit The second original data associated with the first original data associated with the first related part is identified with the second original data associated with the original association information, and is associated with the second original data by the linked update means. When the part is updated, the device specifying unit for specifying the second portable information device to which the second related part before update is transmitted by the related part reply unit among the plurality of portable information devices, and the device specifying unit When the second portable information device is specified, the second portable information device is updated in response to the second related part being updated by the interlock update means. A link transmission means for transmitting the updated second related part, and a display control means provided in the second portable information device that has received the second related part among the plurality of portable information devices, In response to receiving the updated second related part from the server after receiving the two related parts, the AR tag extracted from the picked-up image is received at a position determined with reference to the position in the picked-up image. In addition, it includes interlocking display control means for displaying the image of the second related part after the update.

  According to this aspect, the server extracts the additional portion from the acquired data received from the first portable information device, and first original data associated with the AR tag received from the first portable information device; When the second original data associated with the image forming apparatus is specified, the additional part is transmitted to the second portable information device to which the second related part associated with the second original data is transmitted. The In the second portable information device, the second related part and the additional part are displayed. For this reason, in the acquired data acquired in the first portable information device, an additional portion changed from the replacement data can be displayed on the second portable information device.

  Preferably, the data association registration unit included in the image forming apparatus is a priority registration unit that registers, in the server, priority information in which relative priorities are determined for the first original data and the second original data. The linked update means included in the server further includes priority comparison means for comparing the priority of the first original data with the priority of the second original data, and the comparison result by the priority comparison means is the first comparison result. The second related part is updated on condition that the comparison result is 1.

  According to this aspect, when the priority of the first original data is equal to or higher than the priority of the second original data, the second related part can be updated.

  According to another aspect of the present invention, a virtual reality display method is a virtual reality display method executed in an image processing system including an image forming apparatus, a plurality of portable information devices, and a server. In the apparatus, a replacement step of converting the original data into replacement data in which at least a related portion in the original data is replaced with an AR tag, original identification information for identifying the original data, and the AR tag And an AR tag registration step for transmitting registration information including the relevant part and replacement data to the server, and an output control step for outputting replacement data. The server receives registration information from the image forming apparatus. AR tag association that associates original data included in registration information, AR tag, related part, and replacement data In response to receiving a transmission request including a step, an AR tag, and acquired data from any of the plurality of portable information devices, a related portion associated with the AR tag included in the transmission request is displayed as a plurality of portable information devices. And a related partial reply step for transmitting to the portable information device that has transmitted the transmission request, and an additional portion that is present in the acquired data included in the transmission request but not present in the replacement data associated with the AR tag included in the transmission request And a related part update step for updating the related part by adding the additional part to the related part associated with the AR tag included in the transmission request, and each of the plurality of portable information devices. In addition, replacement data output by the image forming apparatus or data obtained by changing a part of the replacement data is acquired as acquisition data. A data acquisition step, an AR tag extraction step for extracting an AR tag from the acquisition data, an associated partial request step for transmitting a transmission request including the AR tag extracted from the acquisition data and the acquisition data to the server, and a transmission request After that, the related part receiving step for receiving the related part associated with the AR tag from the server, and the image of the received related part at the position determined based on the position in the acquired data of the AR tag extracted from the acquired data. And a display control step for displaying.

  If this situation is followed, the virtual reality display method which can notify a change part to another user can be provided.

It is a figure which shows an example of the whole outline | summary of the augmented reality display system in embodiment of this invention. 1 is a perspective view showing an external appearance of an MFP in the present embodiment. 2 is a block diagram illustrating an example of a hardware configuration of an MFP according to the present embodiment. FIG. It is a block diagram which shows an example of the outline | summary of the hardware constitutions of the portable information device in this Embodiment. It is a block diagram which shows an example of the hardware constitutions of HMD in this Embodiment. It is a block diagram which shows an example of the hardware constitutions of the server in this Embodiment. 2 is a block diagram illustrating an example of an overview of functions of a CPU provided in an MFP. FIG. It is a figure which shows an example of a priority setting screen. It is a block diagram which shows an example of the outline | summary of the function of CPU with which a server is provided. It is a block diagram which shows an example of the outline | summary of the function of CPU with which the portable information device in this Embodiment is provided. It is a figure which shows an example of 1st original data. It is a figure which shows an example of 1st replacement data. It is a figure which shows an example of 2nd original data. It is a figure which shows an example of 2nd replacement data. It is a figure which shows an example of the paper when a character or a figure is added to the paper on which the image of the replacement data is formed. It is a 1st figure which shows an example of the image displayed on a portable information device. It is a 2nd figure which shows an example of the image displayed on a portable information device. It is a flowchart which shows an example of the flow of a replacement data generation process. It is a flowchart which shows an example of the flow of a registration information generation process. It is a flowchart which shows an example of the flow of an original correlation information production | generation process. It is a flowchart which shows an example of the flow of a virtual reality realization process. It is a flowchart which shows an example of the flow of an interlocking transmission process. It is a flowchart which shows an example of the flow of an augmented reality display process.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. In the following description, the same parts are denoted by the same reference numerals. Their names and functions are also the same. Therefore, detailed description thereof will not be repeated.

  FIG. 1 is a diagram showing an example of an overall outline of an augmented reality display system according to an embodiment of the present invention. Referring to FIG. 1, an augmented reality display system 1 includes a wireless station 3, an MFP (Multi Function Peripheral) 100, portable information devices 200, 200 </ b> A, 200 </ b> B, and a personal computer (hereinafter referred to as a “computer”). 3, a head mounted display (hereinafter referred to as “HMD”) 400, and a server 500.

  MFP 100 functions as an image processing apparatus, and includes a document reading function for reading a document, an image forming function for forming an image on a recording medium such as paper based on image data, and a facsimile transmission / reception function for transmitting / receiving facsimile data. ing. In this embodiment, the MFP 100 is described as an example of an image processing apparatus. However, instead of the MFP 100, any apparatus having a function of processing an image may be a PC 300, a server 500, or the like. May be.

  The PC 300 is a general computer. The PC 300 is installed with a printer driver program for controlling the MFP 100.

  The portable information devices 200, 200A, and 200B are examples of information processing devices. The portable information devices 200, 200A, and 200B are used by being carried by users, such as PDAs (Personal Digital Assistants) and smartphones, and have a function of storing data and a wireless LAN function. In addition, portable information devices 200, 200A, and 200B can install application programs in the same manner as PC 300. Since the portable information devices 200, 200A, and 200B have the same hardware configuration and functions, the portable information device 200 will be described as an example here unless otherwise specified.

  The HMD 400 has a shape of glasses and is used by being worn by a user. The HMD 400 has an imaging function for imaging a subject, a display function for displaying an image on a lens portion of glasses, and a wireless LAN function. A user wearing the HMD 400 can visually recognize the subject through the lens and at the same time visually recognize the image displayed on the lens.

  The network 2 is a local area network (LAN), and the connection form may be wired or wireless. Further, the network 2 is not limited to a LAN, and may be a network using a public switched telephone network (Public Switched Telephone Networks). Further, the network 2 may be a wide area network (WAN) such as the Internet.

  The MFP 100 can transmit and receive data to and from the wireless station 3, the PC 300, and the server 500 via the network 2. The wireless station 3 is a relay device of the network 2 and communicates with the portable information devices 200, 200A, 200B and the HMD 400 having a wireless LAN communication function to connect the portable information devices 200, 200A, 200B and the HMD 400 to the network 2. To do. Therefore, each of portable information devices 200, 200 </ b> A, 200 </ b> B and HMD 400 can send and receive data to / from MFP 100, PC 300, and server 500 via wireless station 3 and network 2.

  FIG. 2 is a perspective view showing an appearance of the MFP according to the present embodiment. FIG. 3 is a block diagram illustrating an example of the hardware configuration of the MFP according to the present embodiment. 2 and 3, MFP 100 includes a main circuit 110, a document reading unit 130 for reading a document, an automatic document transport device 120 for transporting a document to document reading unit 130, and a document reading unit. An image is formed by an image forming unit 140 for forming an image on a sheet or the like based on image data output by the manuscript 130 reading and outputting a document, a paper feeding unit 150 for supplying the image to the image forming unit 140, and an image. A post-processing unit 155 that processes paper and an operation panel 160 as a user interface are included.

  The post-processing unit 155 executes a sorting process for sorting and discharging one or more sheets on which images are formed by the image forming unit 140, a punching process for punch hole processing, and a staple process for driving staple needles.

  The main circuit 110 includes a CPU 111, a communication interface (I / F) unit 112, a ROM 113, a RAM 114, a hard disk drive (HDD) 115 as a mass storage device, a facsimile unit 116, and a CD-ROM 118. External storage device 117. CPU 111 is connected to automatic document feeder 120, document reading unit 130, image forming unit 140, paper feeding unit 150, post-processing unit 155, and operation panel 160, and controls the entire MFP 100.

  The ROM 113 stores a program executed by the CPU 111 or data necessary for executing the program. The RAM 114 is used as a work area when the CPU 111 executes a program. The RAM 114 temporarily stores read data (image data) continuously sent from the document reading unit 130.

  Operation panel 160 is provided on the upper surface of MFP 100 and includes a display unit 161 and an operation unit 163. The display unit 161 is a display device such as a liquid crystal display (LCD) or an organic ELD (Electro-Luminescence Display), and displays an instruction menu for the user, information about acquired image data, and the like. The operation unit 163 includes a hard key unit 167 including a plurality of keys, and accepts input of various instructions, data such as characters and numbers by user operations corresponding to the keys. The operation unit 163 further includes a touch panel 165 provided on the display unit 161.

  Communication I / F unit 112 is an interface for connecting MFP 100 to network 2. CPU 111 communicates with portable information devices 200, 200 </ b> A, 200 </ b> B, PC 300, HMD 400, and server 500 via communication I / F unit 112 to transmit / receive data. The communication I / F unit 112 can communicate with a computer connected to the Internet via the network 2.

  The facsimile unit 116 is connected to the public switched telephone network (PSTN) and transmits facsimile data to the PSTN or receives facsimile data from the PSTN. The facsimile unit 116 stores the received facsimile data in the HDD 115 or outputs it to the image forming unit 140. The image forming unit 140 prints the facsimile data received by the facsimile unit 116 on a sheet. Further, the facsimile unit 116 converts the data stored in the HDD 115 into facsimile data, and transmits the facsimile data to a facsimile machine connected to the PSTN.

  The external storage device 117 is loaded with a CD-ROM (Compact Disk ROM) 118. The CPU 111 can access the CD-ROM 118 via the external storage device 117. The CPU 111 loads the program recorded on the CD-ROM 118 attached to the external storage device 117 to the RAM 114 and executes it. The medium for storing the program executed by the CPU 111 is not limited to the CD-ROM 118, but an optical disc (MO (Magnetic Optical Disc) / MD (Mini Disc) / DVD (Digital Versatile Disc)), IC card, optical card, It may be a semiconductor memory such as a mask ROM, EPROM (Erasable Programmable ROM), or EEPROM (Electrically EPROM).

  Further, the program executed by the CPU 111 is not limited to the program recorded on the CD-ROM 118, and the program stored in the HDD 115 may be loaded into the RAM 114 and executed. In this case, another computer connected to the network 2 may rewrite the program stored in the HDD 115 of the MFP 100, or may write a new program. Further, MFP 100 may download a program from another computer connected to network 2 and store the program in HDD 115. The program here includes not only a program directly executable by the CPU 111 but also a source program, a compressed program, an encrypted program, and the like.

  In the augmented reality display system 1, data is transmitted and received between the MFP 100, the portable information devices 200, 200 </ b> A, 200 </ b> B, the PC 300, the HDM 400, and the server 500. As a protocol for transmitting and receiving data, any protocol can be used as long as the transmission source can be specified in the receiving apparatus. As a protocol for transmitting and receiving data, for example, HTTP (Hyper Text Transfer Protocol), FTP (File Transfer Protocol), SMPT (Simple Mail Transfer Protocol), POP (Post Office Protocol), and the like are used.

  FIG. 4 is a block diagram illustrating an example of an outline of a hardware configuration of the portable information device according to the present embodiment. Referring to FIG. 4, portable information device 200 is connected to CPU 201 for controlling overall portable information device 200, camera 202, flash memory 203 for storing data in a nonvolatile manner, and call unit 205. A wireless communication unit 204; a display unit 206 that displays information; an operation unit 207 that accepts user operations; a wireless LAN I / F 208; a position detection unit 209; an orientation detection unit 210; and an external storage device 211. Including.

  The wireless communication unit 204 performs wireless communication with a mobile phone base station connected to a telephone communication network. The wireless communication unit 204 connects the portable information device 200 to a telephone communication network and enables a call using the call unit 205. The radio communication unit 204 decodes a voice signal obtained by demodulating the radio signal received from the mobile phone base station and outputs the decoded signal to the call unit 205. Further, the wireless communication unit 204 encodes the voice input from the call unit 205 and transmits it to the mobile phone base station. The call unit 205 includes a microphone and a speaker, outputs the voice input from the wireless communication unit 204 from the speaker, and outputs the voice input from the microphone to the wireless communication unit 204. Further, the wireless communication unit 204 is controlled by the CPU 201, connects the portable information device 200 to an electronic mail server, and transmits and receives electronic mail.

  The camera 202 includes a lens and a photoelectric conversion element. The light focused by the lens is imaged on the photoelectric conversion element. The photoelectric conversion element photoelectrically converts the received light and outputs image data to the CPU 201. The photoelectric conversion element is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, or the like.

  The display unit 206 is a display device such as a liquid crystal display (LCD) or an organic ELD, and displays an instruction menu for a user, information about acquired image data, and the like. The operation unit 207 includes a plurality of keys, and accepts input of various instructions, data such as characters and numbers by user operations corresponding to the keys.

  The wireless LAN I / F 208 is an interface for communicating with the wireless station 3 and connecting the portable information device 200 to the network 2. By registering the IP (Internet Protocol) addresses of HMD 200, server 500, MFP 100, and PC 300 in portable information device 200, portable information device 200 can communicate with HMD 400, server 500, MFP 100, and PC 300. Data transmission / reception becomes possible. In this embodiment, a case where portable information device 200 communicates with MFP 100, PC 300, HMD 400, and server 500 using wireless LAN I / F 208 will be described as an example, but communication is performed using another communication method. You may do it. Specifically, when portable information device 200, MFP 100, PC 300, HMD 400, and server 500 are equipped with a short-range wireless device such as Bluetooth (registered trademark), for example, portable information device 200 is MFP 100, PC 300, HMD 400. , One-to-one communication with any of the servers 500 may be performed. In addition, the portable information device 200 is connected to any one of the MFP 100, the PC 300, the HMD 400, and the server 500 by a wire such as a USB (Universal Serial Bus) cable, and the portable information device 200 is connected to the MFP 100, the PC 300, the HMD 400, the server. You may make it communicate with either of 500 one to one.

  The flash memory 203 stores a program executed by the CPU 201 or data necessary for executing the program. The CPU 201 loads a program recorded in the flash memory 203 to a RAM included in the CPU 201 and executes it.

  The position detection unit 209 detects the current position of the portable information device 200. Specifically, the position detection unit 209 is a global positioning system receiver, and measures the current position by receiving radio waves from a plurality of GPS satellites. The position detection unit 209 outputs a value indicating the measured current position, for example, latitude and longitude to the CPU 201.

  The direction detection unit 210 detects the direction in which the camera 202 of the portable information device 200 faces. Specifically, the orientation detection unit 210 is a geomagnetic sensor that measures geomagnetism. The direction detection unit 210 outputs the detected direction to the CPU 201. The direction detection unit 210 is installed in the portable information device 200 so as to be able to detect the imaging direction of the camera 202, in other words, the direction in which the optical axis of the lens of the camera 202 faces. The direction detection unit 210 is not limited to a geomagnetic sensor, and a gyro sensor or the like may be used.

  The external storage device 211 is detachable from the portable information device 200, and a CD-ROM 211A storing a remote operation program can be attached thereto. The CPU 201 can access the CD-ROM 211A via the external storage device 211. The CPU 201 can load a remote operation program recorded on the CD-ROM 211A attached to the external storage device 211 into a RAM included in the CPU 201 and execute it.

  Note that the program recorded in the flash memory 203 or the CD-ROM 211A has been described as a program executed by the CPU 201, but a program in which another computer connected to the network 2 rewrites the program stored in the flash memory 203, Alternatively, it may be a new program written additionally. Further, the portable information device 200 may be a program downloaded from another computer connected to the network 2. The program here includes not only a program directly executable by the CPU 201 but also a source program, a compressed program, an encrypted program, and the like.

  The medium for storing the program executed by the CPU 201 is not limited to the CD-ROM 211A, and may be a semiconductor memory such as an optical disk (MO / MD / DVD), an IC card, an optical card, a mask ROM, an EPROM, or an EEPROM. Good.

  FIG. 5 is a block diagram showing an example of the hardware configuration of the HMD in the present embodiment. Referring to FIG. 5, HMD 400 in the present embodiment includes a CPU 401 for controlling the entire HMD 400, a camera 402, a flash memory 403 for storing data in a nonvolatile manner, and a display unit 404 for displaying information. And wireless LAN I / F 405.

  The camera 402 includes a lens and a photoelectric conversion element. The light focused by the lens is imaged on the photoelectric conversion element. The photoelectric conversion element photoelectrically converts the received light and outputs image data to the CPU 401. The photoelectric conversion element is a CMOS (Complementary Metal Oxide Semiconductor) sensor, a CCD (Charge Coupled Device) sensor, or the like. Although an example in which the camera 402 captures a still image is described here, a video camera that captures a moving image may be used. When a video camera is used, one of a plurality of frames included in the moving image may be handled as a still image.

  The display unit 404 is a liquid crystal display (LCD) made of a transparent member, and is embedded in the lens portion of the HMD 400. The display surface of the display unit 404 is arranged at a position determined with reference to the optical axis of the camera 402. Specifically, the imaging range of the camera 402 is set to be the same as the field of view when the user wears the HMD 400. For this reason, the image obtained by capturing with the camera 402 is the same as the image in the field of view actually viewed by the user. For this reason, the position in the user's field of view can be specified from the image output by the camera 402. The display surface of the display unit 404 is set to be the same as the field of view when the user wears the HMD 400 in the lens portion. For this reason, an image can be displayed at an arbitrary position in the user's field of view.

  The wireless LAN I / F 405 is an interface for communicating with the wireless station 3 and connecting the HMD 400 to the network 2. By registering the IP (Internet Protocol) address of the server 500 in the flash memory 403, the HMD 400 can communicate with the server 500 and transmit / receive data. In this embodiment, the case where the HMD 400 communicates with the server 500 using the wireless LAN I / F 405 will be described as an example, but communication may be performed using another communication method. Specifically, when the HMD 400 and the server 500 are mounted with a short-range wireless device such as Bluetooth (registered trademark), for example, the HMD 400 may communicate with the server 500 on a one-to-one basis.

  The flash memory 403 stores a program executed by the CPU 401 or data necessary for executing the program. The CPU 401 loads the program recorded in the flash memory 403 to the RAM provided in the CPU 401 and executes it.

  FIG. 6 is a block diagram illustrating an example of a hardware configuration of the server according to the present embodiment. Referring to FIG. 6, server 500 has a CPU 501 for controlling server 500 as a whole, ROM 502 for storing a program to be executed by CPU 501, RAM 503 used as a work area for CPU 501, and non-volatile storage of data. HDD 504 for storing information, a communication unit 505 for connecting CPU 501 to network 2, a display unit 506 for displaying information, an operation unit 507 for accepting input of user operations, and an external storage device 508.

  The CPU 501 loads a program stored in the ROM 502 or the HDD 504 to the RAM 502 and executes it.

  The external storage device 508 can be loaded with a CD-ROM 508A storing a program. The CPU 501 can access the CD-ROM 508A via the external storage device 508. The CPU 501 can load a program recorded on the CD-ROM 508A into the RAM 502 and execute it.

  In addition, although the program recorded on ROM502, HDD504, or CD-ROM508A was demonstrated as a program which CPU501 performs, the other computer connected to the network 2 rewritten the program memorize | stored in HDD504, or It may be a new program written additionally. Further, the server 500 may be a program downloaded from another computer connected to the network 2. The program here includes not only a program directly executable by the CPU 501 but also a source program, a compressed program, an encrypted program, and the like.

  The medium for storing the program executed by the CPU 501 is not limited to the CD-ROM 508A, and may be a semiconductor memory such as an optical disk (MO / MD / DVD), IC card, optical card, mask ROM, EPROM, or EEPROM. Good.

  FIG. 7 is a block diagram illustrating an example of an overview of the functions of the CPU provided in the MFP. The functions shown in FIG. 7 are functions formed in CPU 111 when CPU 111 provided in MFP 100 executes a program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIG. 7, CPU 111 includes a data acquisition unit 51 that acquires original data, a replacement unit 53 that converts original data into replacement data, an AR tag registration unit 55, and an output control unit 57 that outputs replacement data. A user authentication unit 59 that authenticates a user who operates MFP 100, a data association registration unit 61 that associates original data with other original data, and a priority registration unit 63.

  The data acquisition unit 51 acquires original data, and outputs the acquired original data to the replacement unit 53, the AR tag registration unit 55, and the data association registration unit 61. The original data may be data stored in the HDD 115, or data stored in the PC 300 or the server 500. The format of the data is not limited, but may be application data generated by the PC 300 executing the application program or image data. The application program is not particularly limited, and includes, for example, a document editing program for generating a document, an image editing program for editing an image, a spreadsheet program, a program for generating presentation data, a database program, and the like.

  The replacement unit 53 generates replacement data in which the part specified by the user in the original data is used as a related part, and the related part is replaced with an AR (Augmented Reality) tag. Since the related portion is a portion specified by the user in the original data, it includes characters, graphics or photographs, or a combination thereof, although it varies depending on the original data. The AR tag is a predetermined image, and can be distinguished from other images and other AR tags. Note that the AR tag may be a two-dimensional barcode such as a QR code (registered trademark) (Quick Response Code). The related part includes position information indicating a relative position of the related part with reference to the AR tag arranged in the replacement data.

  For example, if the user designates a part of the original data as the related part, the original data is converted into replacement data. Even if the replacement data is output, the related portion is replaced by the AR tag, and therefore the related portion cannot be known from the replacement data. By specifying a part of the original data as the related part, the user can prevent another user who sees the replacement data from understanding the related part. The replacement unit 53 outputs the replacement data to the output control unit 57 and outputs a set of the related part, the AR tag in which the related part is replaced, and the replacement data to the AR tag registration unit 55.

  User authentication unit 59 authenticates a user who operates MFP 100. If the login screen is displayed on the display unit 161 and the user identification information and password input by the user to the operation unit 163 are received and they match the combination of the user identification information and password stored in the HDD 115 in advance, Authenticate the user. The user authentication unit 59 outputs user identification information of the authenticated user to the output control unit 57.

  Output control unit 57 outputs the replacement data input from replacement unit 53 to an output destination determined by an operation input to operation unit 163 by a user operating MFP 100. The output control unit 57 outputs an output completion signal including the user identification information input from the user authentication unit 59 to the data association registration unit 61 in response to outputting the replacement data. The output control unit 57 includes an image formation control unit 65, a data transmission unit 67, and a facsimile transmission control unit 69. The image formation control unit 65 controls the image formation unit 140 to form an image of replacement data on a sheet.

  The data transmission unit 67 transmits the replacement data to another computer via the communication I / F unit 112. Although the transmission protocol is not particularly limited, for example, file transfer by FTP (File Transfer Protocol), transmission of e-mail with attached replacement data or changed replacement data. Here, as an example of the case where the data transmission unit 67 transmits the replacement data, a case where the replacement data is transmitted to the portable information device 200 will be described as an example.

  The facsimile transmission control unit 69 controls the facsimile unit 116 to transmit the replacement data to another facsimile apparatus. In a facsimile apparatus that receives replacement data, an image of replacement data is formed on a sheet.

  The AR tag registration unit 55 receives original data from the data acquisition unit 51, and receives a set of related portions, AR tags, and replacement data from the replacement unit 361. The AR tag registration unit 55 transmits registration information including original identification information for identifying original data, a related portion, an AR tag, and replacement data to the server 500. Specifically, the registration information is transmitted to the server 500 via the communication I / F unit 112.

  The data association registration unit 61 receives original data from the data acquisition unit 51. The data association registration unit 61 associates the original data with another original data different from the original data. Specifically, original association information including two or more pieces of original identification information for identifying two or more pieces of original data is transmitted to server 500 via communication I / F unit 112. Further, the data association registration unit 61 outputs the original association information to the priority registration unit 63. The data association registration unit 61 includes an identical data association unit 71, an identical user association unit 73, a candidate data acquisition unit 75, and an operation association unit 77.

  When the original data input from the data acquisition unit 51 is one page of data consisting of a plurality of pages, the same data association unit 71 converts the original data input from the data acquisition unit 51 into data including the original data. It associates with the original data of one or more other pages among the plurality of pages. In other words, the same data association unit 71 associates a plurality of original data, which are a plurality of pages included in one data, with each other. For example, when the data consists of the first page, the second page, and the third page, the original data of the first page and the original data of the second page are associated with each other, and the original data of the first page and the original data of the third page And the original data on the second page and the original data on the third page.

  The same user association unit 73 associates a plurality of pieces of original data respectively corresponding to a plurality of pieces of replacement data output based on an instruction from the same user by the output control unit 57. After the original data is input from the data acquisition unit 51 and the output completion signal including the user identification information is input from the output control unit 57, the same user association unit 73 receives the user identification information included in the output completion signal. Based on the operation by the user, it is determined that replacement data obtained by replacing the original data has been output. The same user association unit 73 stores history information including original identification information for identifying original data and user identification information, and in response to an output completion signal being input from the output control unit 57. The original data specified by the original identification information included in the history information including the user identification information included in the output completion signal with reference to the history information, and the original data related to the original data input from the data acquisition unit 51 To decide. For example, when replacement data in which original data A is replaced based on an operation by user A is formed and replacement data in which original data B is replaced based on an operation by user A is formed, original data A and original data Associate with B. Further, the same user association unit 73 associates a plurality of original data respectively corresponding to a plurality of replacement data output based on instructions from a plurality of users belonging to the same group by the output control unit 57. For example, when user A and user B belong to the same group, replacement data in which original data A is replaced based on an operation by user A is formed as an image, and replacement data in which original data B is replaced based on an operation by user B When the image is formed, the original data A and the original data B are associated with each other.

  The candidate data acquisition unit 75 acquires one or more original data as candidates from the server 500. Since the registration information is transmitted to the server 500 by the AR tag registration unit 55, the server 500 stores original data included in the registration information. The candidate data acquisition unit 75 requests the server 500 to transmit candidate data, and acquires one or more original data stored in the server 500 as candidate data. The candidate data acquisition unit 75 outputs the acquired one or more candidate data to the operation association unit 77.

  The operation association unit 77 associates the original data input from the data acquisition unit 51 with candidate data selected by the user from one or more candidate data input from the candidate data acquisition unit 75.

  The priority registration unit 63 receives the original association information from the data association registration unit 61. The priority registration unit 63 gives a priority to the original data specified by two or more pieces of original identification information included in the original association information. The priority determines relative superiority or inferiority with respect to the two original data. For example, with respect to two original data, there are three ways: one has a higher priority than the other, one has a lower priority than the other, and both have the same priority. When the original data is 3 or more, the relationship between the priorities of the two sets of original data is the same as the above three types. The priority registration unit 63 transmits priority information including original identification information and priority of each of two or more original data to the server 500 via the communication I / F unit 112.

  FIG. 8 is a diagram illustrating an example of a priority setting screen. The priority setting screen 700 illustrated in FIG. 8 illustrates an example of a case where the priority setting screen 700 is displayed when the first original data and the second original data are associated with each other. Here, the original identification information “document X” of the first original data is used, and the original identification information “document Y” of the second original data is used.

  Referring to FIG. 8, priority setting screen 700 includes first original data thumbnail 701, second original data thumbnail 702, selection buttons 711 to 714, and determination button 715. The thumbnail is a reduced image of the data image.

  The original identification information “document X” of the first original data is arranged in the vicinity of the thumbnail 701 of the first original data, and the original identification information “document of the second original data in the vicinity of the thumbnail 702 of the second original data. Y "is arranged.

  Any one of the selection buttons 711 to 714 can be selected. The selection button 711 is a button for setting the priority of the first original data and the priority of the second original data to be the same. The selection button 712 is a button for setting the priority of the first original data to a value higher than the priority of the second original data. The selection button 713 is a button for setting the priority of the first original data to a value lower than the priority of the second original data. The selection button 714 is a button for not setting the priority of the first original data and the priority of the second original data.

  The determination button 715 is a button for inputting an operation for determining priority. When the determination button 715 is instructed in a state where any of the selection buttons 711 to 714 is selected, the priorities of the first original data and the second original data are determined.

  FIG. 9 is a block diagram illustrating an example of an overview of the functions of the CPU provided in the server. The functions shown in FIG. 9 are formed in the CPU 501 when the CPU 501 provided in the server 500 loads the program stored in the ROM 502 or the HDD 504 to the RAM 502 and executes it. 9, CPU 501 provided in server 500 includes an AR tag association unit 551, a registration information extraction unit 553, a transmission request reception unit 555, an original association unit 557, an associated partial reply unit 559, A related original identification unit 561, an additional part extraction unit 563, a device identification unit 565, a linked transmission unit 567, a priority setting unit 569, a related part update unit 571, and a linked update unit 573 are included.

  AR tag associating unit 551 receives registration information from MFP 100, associates original data included in registration information, AR tag, related portion, and replacement data with the received registration information as registration information extraction unit It outputs to 553. The AR tag association unit 551 controls the communication unit 505, receives registration information transmitted from the MFP 100, and stores the received registration information in the HDD 504, whereby the original data included in the registration information, the AR tag And the related part and the replacement data are associated with each other, and the registration information is output to the registration information extraction unit 553.

  In the transmission request receiving unit 555, the communication unit 505 receives a transmission request from any one of the portable information devices 200, 200 </ b> A, 200 </ b> B, or the HMD 400. Here, a case where a transmission request is received from portable information device 200 will be described as an example. The transmission request includes an AR tag and acquired data. The acquired data is data acquired by the portable information device 200 that has transmitted the transmission request. The acquired data is data obtained, for example, by the portable information device 200 capturing an image of a sheet on which the MFP 100 forms an image of replacement data using the camera 202 as a subject. The acquired data is replacement data when portable information device 200 receives replacement data transmitted by MFP 100. In response to receiving the transmission request, the transmission request receiving unit 55 outputs the AR tag included in the transmission request to the registration information extracting unit 553, and outputs the acquired data included in the transmission request to the additional portion extracting unit 563. , Device identification information for identifying the device that has transmitted the transmission request, here, device identification information of portable information device 200 is output to related partial reply section 559.

  The registration information extraction unit 553 receives the AR tag input from the transmission request receiving unit 555 in response to the input of the AR tag from the transmission request receiving unit 555, the original data associated by the AR tag association unit 551, Determine relevant parts and replacement data. Specifically, registration information including the same AR tag as the AR tag input from the transmission request receiving unit 555 is extracted from the registration information stored in the HDD 504, and the original identification information included in the extracted registration information Get part and replacement data. The registration information extraction unit 553 outputs the pair of the original identification information and the related part to the related part reply unit 559, outputs the original identification information to the related original specifying unit 561, and adds the pair of the original identification information and the replacement data The data is output to the partial extraction unit 563.

  The related part reply unit 559 receives a set of the original identification information and the related part from the registered information extraction unit 553 and transmits a transmission request from the transmission request receiving unit 555, here, the device identification information of the portable information device 200 Is entered. The related part reply unit 559 controls the communication unit 505 to transmit the related part to the portable information device 200 specified by the device identification information. In addition, the related part reply unit 559 outputs a set of original identification information and device identification information to the device identification unit 565.

  Original association unit 557 receives original association information from MFP 100, associates two or more original data respectively specified by two or more original identification information included in the received original association information, and A set of original association information is output to the related original identification unit 561. The original association unit 557 controls the communication unit 505 to receive the original association information transmitted from the MFP 100 and store the received original association information in the HDD 504, so that 2 included in the original association information. Two or more original data respectively specified by the above original identification information are associated with each other, and the original association information is output to the related original specifying unit 561.

  The related original identification unit 561 receives original identification information from the registered information extraction unit 553 and receives original related information from the original association unit 557. The original related information includes two or more pieces of original identification information. The related original specifying unit 561 specifies the original data specified by the original identification information input from the registered information extraction unit 553 and the original data related by the original related information as related original data. The related original identification unit 561 extracts original related information including original identification information input from the registration information extraction unit 553 from the original related information stored in the HDD 504, and is included in the extracted original related information 2. The above original identification information is acquired. The related original specifying unit 561 specifies one or more pieces of original identification information excluding the original identification information input from the registered information extraction unit 553 among the two or more pieces of acquired original identification information as the original identification information of the related original data. To unit 565 and interlocking update unit 571.

  The device specifying unit 565 receives a set of original identification information and device identification information from the related part reply unit 559, and receives original identification information of one or more related original data from the related original specifying unit 561. When the combination of the original identification information and the device identification information is input from the related part reply unit 559, the device specifying unit 565 stores them in the HDD 504. When the original identification information of one or more related original data is input from the related original specifying unit 561, the device specifying unit 565 refers to the set of the original identification information and the device identification information stored in the HDD 504 and refers to the related original. The device identification information paired with the original identification information of the related original data input from the specifying unit 561 is specified. Specifically, the device specifying unit 565 has the same original identification as the original identification information of the related original data input from the related original specifying unit 561 from the set of the original identification information and the device identification information stored in the HDD 504. A set including information is extracted, and device identification information included in the extracted set is specified. When a plurality of pieces of original identification information are input from the related original identification unit 561, the device identification unit 565 identifies device identification information paired with the original identification information for each of the plurality of original identification information. The device specifying unit 565 outputs the combination of the original identification information of the related original data and the device identification information that is paired with the original identification information to the linked update unit 573, and the device identification information that is paired with the original identification information to the linked transmission unit 567. Output.

  The additional portion extraction unit 563 receives the combination of the original identification information of the original data and the replacement data from the registration information extraction unit 553, and receives the acquired data from the transmission request reception unit 555. The additional portion extraction unit 563 extracts a portion that exists in the acquired data but does not exist in the replacement data as an additional portion. What is necessary is just to extract an additional part from the difference of replacement data and acquisition data. The additional part extraction unit 56 outputs a set of the extracted additional part and the original identification information of the original data input from the registration information extraction unit 553 to the related part update unit 571. The additional portion includes image information included in the acquired data and position information indicating the relative position of the additional portion based on the AR tag included in the acquired data.

  A priority setting unit 569 receives priority information from the MFP 100, sets priority for each of two or more original data respectively specified by two or more original identification information included in the priority information, and sets priority information. Is output to the interlocking transmission unit 567. The priority information defines original identification information and priority for each of two or more original data. The priority setting unit 569 controls the communication unit 505, receives priority information transmitted from the MFP 100, and stores the received priority information in the HDD 504, whereby two or more originals included in the priority information are stored. Priorities are set for each of two or more original data specified by the identification information, and the priority information is output to the interlocking transmission unit 567.

  The related part update unit 571 receives a set of the additional part and the original identification information from the additional part extraction unit 563. The related part update unit 571 updates the related part by adding the additional part to the related part. Specifically, the registration information including the same original identification information as the original identification information input from the additional portion extraction unit 563 is extracted from the registration information stored in the HDD 504, and is included in the extracted registration information. Update relevant parts. The related part includes position information indicating a relative position with respect to the AR tag, and the additional part includes position information indicating a relative position with respect to the AR tag. The related part update unit 571 determines the relative position between the related part and the additional part using the position information included in the related part and the position information included in the additional part, and is determined for the related part. Update the relevant part by placing the additional part in a relative position. When the related part update unit 571 updates the related part, the related part update unit 571 outputs a pair of the additional part and the original identification information paired therewith to the interlocking update unit 573.

  The linked update unit 573 receives original identification information of related original data from the related original identification unit 561, and receives an additional part and original identification information from the related part update unit 571. In response to the input of the additional part from the related part update part 571, the interlock update unit 573 updates the related part by adding the additional part to the related part associated with the related original data. Specifically, the registration information including the same original identification information as the original identification information of the related original data input from the related original identification unit 561 is extracted from the registration information stored in the HDD 504, and the extracted registration is extracted. Update relevant parts in the information.

  The interlock update unit 573 includes a rank comparison unit 581. The rank comparison unit 581 receives the combination of the additional part and the original identification information from the related part update unit 571 and the original identification information of the related original data from the related original identification unit 561. The original identification information input from the related part update unit 571 is original identification information of original data. The rank comparison unit 581 refers to the priority set for each original data by the priority setting unit 569, and compares the priority of the original data with the priority of the related original data.

  The interlock update unit 573 updates the related portion associated with the related original data on the condition that the priority of the original data is equal to or higher than the priority of the related original data. If the priority of the original data is lower than the priority of the related original data, the interlock update unit 573 does not update the related portion associated with the related original data. The interlock update unit 573 outputs a transmission instruction to the interlock transmission unit 567 when the related part associated with the related original data is updated. The transmission instruction includes an additional part.

  The interlock transmission unit 567 receives device identification information from the device specifying unit 565 and a transmission instruction from the interlock update unit 573. The interlocking transmission unit 567 controls the communication unit 505 in response to the transmission instruction input from the interlocking update unit 573, and the device identification information input from the device specifying unit 565 includes the additional part included in the transmission instruction. To the device specified in

  FIG. 10 is a block diagram illustrating an example of an outline of functions of a CPU provided in the portable information device according to the present embodiment. The functions shown in FIG. 10 are functions formed in the CPU 201 when the CPU 201 included in the portable information device 200 executes the augmented reality display program stored in the flash memory 203. Referring to FIG. 10, the CPU 201 obtains an acquisition data acquisition unit 253, an AR tag extraction unit 255, a related part request unit 257 that requests the server 500 to transmit a related part, and a relation that receives a related part from the server 500. A partial receiving unit 259 and a display control unit 251 for controlling the display unit 206 are included.

  The acquisition data acquisition unit 253 acquires data. Here, the data acquired by the acquisition data acquisition unit 253 is referred to as acquisition data. The acquisition data acquisition unit 253 outputs the acquired acquisition data to the AR tag extraction unit 255, the related part request unit 257, and the display control unit 251.

  Acquired data acquisition unit 253 controls wireless LAN I / F 208 and receives data transmitted by MFP 100 as acquired data. When MFP 100 transmits the acquired data by e-mail, the MFP 100 receives the e-mail and acquires the acquired data included in the received e-mail. The acquired data includes data obtained by changing the data received by the wireless LAN I / F 208 from the MFP 100. For example, when the user who operates MFP 100 edits data received from MFP 100 using an application program, acquisition data acquisition unit 253 acquires data after editing data received from MFP 100 as acquisition data. To do. The acquired data acquisition unit 253 includes an imaging control unit 261 that controls the camera 202. The imaging control unit 261 controls the camera 202 and acquires a captured image output by the camera 202 imaging the subject. When the camera 202 captures and outputs a sheet on which a replacement data image is formed by the MFP 100 as a subject, the acquisition data acquisition unit 253 acquires a captured image output by the camera 202 imaging the sheet as acquisition data. . The user may draw handwritten characters, figures, and the like on the sheet on which the replacement data image is formed by the MFP 100. In this case, the captured image that the camera 202 captures and outputs the subject includes images of handwritten characters and graphics in addition to the replacement data image.

  The AR tag extraction unit 255 extracts an AR tag from the acquisition data input from the acquisition data acquisition unit 253. The AR tag extraction unit 255 outputs the extracted AR tag to the related part request unit 257.

  The related part request unit 257 receives the acquisition data from the acquisition data acquisition unit 253 and the AR tag from the AR tag extraction unit 255. The related part request unit 257 controls the wireless LAN I / F 208 and transmits a transmission request to the server 500. The transmission request includes acquired data and an AR tag. The IP address of server 500 may be stored in advance in flash memory 203 of portable information device 200. When receiving the transmission request, the server 500 that receives the transmission request returns a related part associated with the AR tag included in the transmission request.

  The related part receiving unit 259 controls the wireless LAN I / F 208 to receive the related part from the server 500. The related part receiving unit 259 outputs the related part to the display control unit 251. The related part includes an image and position information indicating a relative position of the related part with reference to the AR tag.

  The display control unit 251 displays an image of the acquisition data on the display unit 206 in response to the acquisition data input from the acquisition data acquisition unit 253. When the related part is input from the related part receiving unit 259, the display control unit 251 displays an image of the related part on the display unit 206. Based on the position information included in the related portion, the display control unit 251 specifies the position where the image of the related portion is arranged based on the AR tag included in the image of the acquired data displayed on the display unit 206. The display control unit 251 superimposes and displays the image of the related portion at a position specified with reference to the AR tag in the acquired data image.

  The wireless LAN I / F 208 may receive the additional part from the server 500 after the related part is displayed by the display control unit 251. The display control unit 251 includes an interlocking display control unit 263. When receiving the additional portion from the server 500, the interlocking display control unit 263 displays an image of the additional portion on the display unit 206. The additional portion includes an image and position information indicating a relative position of the additional portion with reference to the AR tag. Based on the position information included in the additional portion, the interlocking display control unit 263 specifies the position where the image of the additional portion is to be arranged with reference to the AR tag included in the acquired data image displayed on the display unit 206. . The interlocking display control unit 263 superimposes and displays the image of the additional portion at the position specified on the basis of the AR tag in the acquired data image.

  Here, a case where MFP 100 converts data received from PC 300 into replacement data and forms an image will be described as an example. The data that MFP 100 receives from PC 300 includes first original data that is the first page and second original data that is the second page. In the first original data and the second original data, the case where the priority of the first original data is set higher than the priority of the second original data will be described as an example.

  FIG. 11 is a diagram illustrating an example of the first original data. Referring to FIG. 11, first original data 610 includes a character string 611 of “Confidential” and an area 613 where a bar graph is represented. Here, a case where the area 613 is designated as a related part by the user will be described as an example.

  FIG. 12 is a diagram illustrating an example of the first replacement data. The first replacement data is data obtained by replacing the related portion 613 of the first original data 610 with an AR tag. Referring to FIG. 12, first replacement data 610A includes a character string 611 of “Confidential” and an AR tag 613A.

  FIG. 13 is a diagram illustrating an example of second original data. Referring to FIG. 13, second original data 620 includes a character string 621 of “Confidential” and an area 623 where a band graph is represented. Here, a case where the area 623 is designated as a related part by the user will be described as an example.

  FIG. 14 is a diagram illustrating an example of second replacement data. The second replacement data is data in which the related portion 623 of the second original data 620 is replaced with an AR tag. Referring to FIG. 14, first replacement data 620A includes a character string 621 of “Confidential” and an AR tag 623A.

  When MFP 100 forms the image of first replacement data 610A on a sheet, MFP 100 includes first original identification information of first original data 610, AR tag 613A, related portion 613, and replacement data 610A. The registration information is transmitted to the server 500. When MFP 100 forms the image of second replacement data 620A on a sheet, MFP 100 includes second original identification information of first original data 620, AR tag 623A, related portion 623, and replacement data 620A. The second registration information is transmitted to the server 500. Further, since the first original data and the second original data are the first page and the second page included in the same data, the MFP 100 includes the first original identification information of the first original data, the second original data, and the second page. Original association information including the second original identification information of the original data is transmitted to the server 500. Further, since the priority of the first original data is set by the user higher than the priority of the second original data, the MFP 100 can identify the original identification information of each of the first original data and the second original data, Priority information in which the priority of one original data is higher than the second original data priority is transmitted to the server 500. The server 500 stores the first registration information, the second registration information, the original association information and the priority information in the HDD 504.

  At this stage, when the user A operates the portable information device 200 to capture a sheet on which the image of the first replacement data 610A is formed by the camera 202, the portable information device 200 uses the first replacement data 610A. A captured image obtained by capturing an image with the camera 202 is used as first acquired data, and a first transmission request including the AR tag 613A extracted from the first acquired data and the first acquired data is sent to the server 500. Send to. The server 500 extracts the first registration information including the same AR tag 613A as the AR tag 613A included in the first transmission request, and returns the related portion 613 included in the first registration information to the portable information device 200. . Therefore, the portable information device 200 displays the related portion 613 returned from the server 500 at a position determined based on the AR tag 613A in the first acquired data (captured image). Therefore, the same image as the first original data 610 shown in FIG. 11 is displayed on the display unit 206 of the portable information device 200.

  On the other hand, when the user B operates the portable information device 200A to capture a sheet on which the image of the second replacement data 620A is formed by the camera 202, the portable information device 200A displays the image of the second replacement data 620A. A captured image obtained by capturing with the camera 202 is used as second acquired data, and a second transmission request including the AR tag 623A extracted from the second acquired data and the second acquired data is transmitted to the server 500. To do. The server 500 extracts the second registration information including the same AR tag 623A as the AR tag 623A included in the second transmission request, and returns the related portion 623 included in the second registration information to the portable information device 200A. . Therefore, portable information device 200A displays related portion 623 returned from server 500 at a position determined based on AR tag 623A in the second acquired data (captured image). Therefore, the same image as the second original data 620 shown in FIG. 13 is displayed on the display unit 206 of the portable information device 200A.

  Next, a case will be described as an example where user A draws characters or figures by handwriting on a sheet on which an image of the first replacement data 610A is formed.

  FIG. 15 is a diagram illustrating an example of a sheet when characters or figures are added to the sheet on which the replacement data image is formed. Here, a case where the user A draws a character by handwriting on a sheet on which an image of the first replacement data 610A is formed is shown. Referring to FIG. 15, paper 610 </ b> B includes a character string 611 of “Confidential”, a related portion 613, and a region 615 where handwritten characters are represented.

  When the user A operates the portable information device 200 and images the paper 610B with the camera 202, the portable information device 200 uses the captured image obtained by imaging the paper with the camera 202 as the third acquisition data, and The third transmission request including the AR tag 613A extracted from the third acquired data and the third acquired data is transmitted to the server 500. The server 500 extracts the first registration information including the same AR tag 613A as the AR tag 613A included in the third transmission request, and returns the related portion 613 included in the first registration information to the portable information device 200. . Therefore, portable information device 200 displays related portion 613 returned from server 500 at a position determined based on AR tag 613A in the third acquired data (captured image). FIG. 16 shows an example of an image displayed on the portable information device 200.

  FIG. 16 is a first diagram illustrating an example of an image displayed on the portable information device. Referring to FIG. 16, an image 610 </ b> C includes a character string 611 of “Confidential”, a related portion 613, and a region 615 where handwritten characters are represented.

  On the other hand, the server 500 compares the third acquisition data included in the third transmission request with the first replacement data included in the first registration information, and exists in the third acquisition data. Determine additional parts that do not exist in the replacement data. Here, an area 615 in which handwritten characters that are present in the third acquired data 610B but not present in the first replacement data 610A are represented is determined as an additional portion. Furthermore, the server 500 specifies the first original data 610 specified by the first original identification information included in the first registration information and the second original data 620 related by the original association information. Further, the server 500 sets the second registration information including the second original identification information of the second original data 620 on the condition that the priority of the first original data is equal to or higher than the priority of the second original data. And the device that transmitted the second related portion 623A included in the second registration information is identified. Here, since the priority of the first original data is higher than the priority of the second original data and the second related portion 623A is transmitted to the portable information device 200A, the portable information device 200A is specified. Then, server 500 transmits additional portion 615 to portable information device 200A.

  When user B adds a handwritten character to the sheet on which the second replacement data 620 is formed, the server 500 specifies the second original identification information included in the second registration information. Original data 620 and the first original data 610 associated with the original association information are specified, but the priority of the second original data is lower than the priority of the first original data. The additional part is not transmitted to the portable information device 200 without extracting the first registration information including the first original identification information of the data 610. Therefore, by setting priority to a plurality of related original data, it is possible to determine whether or not to display an additional part added to one side on the other side.

  When the portable information device 200A receives the additional portion 615 from the server 500 while displaying the second related portion 623 received from the server 500, the portable information device 200A converts the additional portion 615 into the second acquired data (captured image). ) Is displayed at a position determined based on the AR tag 623A. FIG. 17 shows an example of an image displayed on portable information device 200A.

  FIG. 17 is a second diagram illustrating an example of an image displayed on the portable information device. Referring to FIG. 17, an image 620 </ b> C includes a character string 621 of “Confidential”, a related portion 623, and an additional portion 615 that is an area where handwritten characters are represented. Therefore, an image obtained by adding an additional portion 615 to the same image as the second original data 620 illustrated in FIG. 13 is displayed on the display unit 206 of the portable information device 200A.

  Thus, the user B can visually recognize the additional portion 615 added by the user A by handwriting. For this reason, the additional part added to at least one among the some original data relevant to each other can be notified to the user who browses other relevant original data.

  FIG. 18 is a flowchart illustrating an example of the flow of replacement data generation processing. The replacement data generation process is a process executed by CPU 111 when CPU 111 provided in MFP 100 executes a replacement data generation program stored in ROM 113, HDD 115, or CD-ROM 118. Referring to FIG. 18, CPU 111 authenticates a user who operates MFP 100 (step S01). A login screen is displayed on the display unit 161, and user identification information and a password input by the user to the operation unit 163 are accepted. If they are stored in the HDD 115 in advance, the user is authenticated.

  In the next step S02, data is acquired. The data acquired in step S02 may be data stored in advance in the HDD 115, print data received from the PC 300, or application data. Application data is data generated when the PC 300 executes an application program.

  In the next step S03, it is determined whether or not the data acquired in step S02 includes a plurality of pages. If the data includes a plurality of pages, the process proceeds to step S04. If the data acquired in step S02 does not include a plurality of pages, the data is set as original data to be processed, and the process proceeds to step S08.

  In step S04, one of the plurality of pages is selected as original data to be processed, and the process proceeds to step S05. In step S05, a registration information generation process is executed, and the process proceeds to step S06. Although details of the registration information generation process will be described later, it includes a process of converting original data set as a processing target into converted replacement data.

  In step S06, an original association information generation process is executed, and the process proceeds to step S07. Details of the original association information generation processing will be described later. In step S07, it is determined whether or not there is a page not selected as a processing target. If there is a page that is not selected as a processing target, the process returns to step S04. If there is no page, the process proceeds to step S10.

  In step S08, a registration information generation process is executed for the original data set as a processing target, and the process proceeds to step S09. In step S09, an original association information generation process is executed, and the process proceeds to step S10.

  In step S10, it is determined whether an output instruction has been accepted. When the operation unit 163 receives an operation of an output instruction by the user, it is determined that the output instruction has been received. For example, when a user presses a key to which a print instruction or a transmission instruction is assigned in advance among a plurality of keys provided in the hard key unit 167, the output instruction is accepted. The process waits until an output instruction is accepted (NO in step S10). If an output instruction is accepted (YES in step S10), the process proceeds to step S11. In step S11, the process branches depending on the output destination specified by the output instruction. If the output instruction indicates printing, the process is output to step S12. If the output instruction indicates data transmission, the process proceeds to step S14. For example, when a user presses a key assigned with a print instruction among a plurality of keys included in the hard key unit 167 in step S10, the process proceeds to step S12, and the key assigned a transmission instruction is pressed by the user. If so, the process proceeds to step S14.

  In step S12, the replacement data is printed, and the process proceeds to step S13. Specifically, the replacement data image generated in step S05 is formed on a sheet. In step S14, the replacement data is transmitted, and the process proceeds to step S13. Specifically, the replacement data generated in step S05 is transmitted to another computer via the communication I / F unit 112. The destination for transmitting the replacement data may be specified by the user using the operation unit 163. For example, when the portable information device 200 is designated by the user, the replacement data is transmitted to the portable information device 200.

  In the next step S13, the history information is stored and the process is terminated. The history information includes user identification information and original identification information. The user identification information is user identification information of the user authenticated in step S01. The original identification information is data identification information for identifying original data to be processed in the registration information generation process executed in step S05 or step S08. Specifically, the original data identification information is data identification information of each of a plurality of original data respectively corresponding to a plurality of pages when the data acquired in step S02 includes a plurality of pages, and is acquired in step S02. When the data includes one page, it is data identification information of the data.

  FIG. 19 is a flowchart illustrating an example of the flow of registration information generation processing. The registration information generation process is a process executed in step S05 or step S08 in FIG. Before the registration information generation process is executed, the original data to be processed is specified. Referring to FIG. 19, CPU 111 accepts designation of related parts in the original data (step S21). For example, if an image of original data is displayed on the display unit 161 and the user inputs an operation for designating an arbitrary region in the displayed image to the operation unit 163, the designated region is accepted as a related portion. The process waits until designation of a related part is accepted (NO in step S21). If designation of a related part is accepted (YES in step S21), the process proceeds to step S22. The related part includes position information in the original data.

  In step S22, an AR tag is determined. An arbitrary AR tag is determined from the AR tags prepared in advance. In the next step S23, the AR tag is associated with the related part. In step S21, the related part which is the area designated by the user in the original data is associated with the AR tag determined in step S22.

  In the next step S24, replacement data is generated. The replacement data is data obtained by replacing the relevant part of the original data with the AR tag. Specifically, the CPU 111 generates replacement data by replacing the related portion specified by the user in step S21 in the original data with the AR tag determined in step S22.

  In step S25, registration information is generated. The registration information includes original identification information for identifying original data, a related portion, an AR tag, and replacement data. Then, the registration information is registered in the server, and the process ends. The registration information generated in step S25 is transmitted to the server 500 via the communication I / F unit 112.

  FIG. 20 is a flowchart illustrating an example of the flow of the original association information generation process. The original association information generation process is a process executed in step S06 or step S09 in FIG. Before the original association information generation process is executed, the original data to be processed and the user who operates the MFP 100 are specified. Referring to FIG. 20, CPU 111 determines whether or not the original data is a part of data including a plurality of pages (step S31). If the original data is part of the data, the process proceeds to step S32. If not, the process proceeds to step S34. In step S32, another page is set as related original data, and the process proceeds to step S33. Among a plurality of pages included in the same data, data of each of one or more pages other than the original data page is set as related original data. In step S33, original association information is generated, and the process proceeds to step S34. The original association information includes two pieces of original identification information and associates two pieces of original data. Here, the original data is associated with the original data set as the related original data in step S32. When a plurality of related original data are set, the same number of pieces of original association information as the plurality of related original data are generated.

  In step S34, the same history information is extracted by the user or group. History information with the same user is history information including user identification information of the user who operates MFP 100 among the history information stored in HDD 115. The history information with the same group is history information including any of user identification information of each of one or more users belonging to the same group as the user who operates MFP 100 among the history information stored in HDD 115. The HDD 115 stores group data obtained by classifying a plurality of users into one or more groups, and uses the group data to identify one or more users belonging to the same group as the user who operates the MFP 100. For example, consider a case where user A who operates MFP 100 designates first original data and causes MFP 100 to execute a process of printing replacement data obtained by replacing the first original data with an AR tag. Prior to user A, user B belonging to the same group as user A designates the second original data, and causes MFP 100 to execute a process of printing replacement data in which the second original data is replaced with an AR tag. In this case, history information including the user identification information of the user B and the original identification information of the second original data is stored in the HDD 115. Therefore, when user A designates the first original data and causes MFP 100 to execute the process of printing the replacement data obtained by replacing the first original data with the AR tag, the user identification information of user B and the second The history information including the original identification information of the original data is extracted.

  In step S35, it is determined whether history information has been extracted. If the history information is extracted, the process proceeds to step S36. If not, the process proceeds to step S38. In step S36, the output original data is set as related original data. The output original data is the original data specified by the original identification information included in the history information extracted in step S34. In step S37, original association information is generated, and the process proceeds to step S38. The original data is associated with the original data set as the related original data in step S36. When a plurality of pieces of history information are extracted, the same number of original association information as the plurality of pieces of history information is generated.

  In step S38, the server is requested for candidate data. Transmission of candidate data is transmitted to the server 500 via the communication I / F unit 112. The server 500 that is requested to transmit candidate data returns the candidate data. In the server 500, candidate data can be arbitrarily determined. In the next step S39, when the communication I / F unit 112 receives one or more candidate data from the server 500, a candidate list from which one or more candidate data can be selected is displayed on the display unit 161. It is determined whether candidate data has been selected by the user (step S40). The process waits until one or more candidate data items are selected from the candidate list by the user. When one or more candidate data items are selected, the process proceeds to step S41. In step S41, the selected candidate data is set as related original data. In step S42, original association information is generated, and the process proceeds to step S43. The original data is associated with the candidate data set as the related original data in step S41. When a plurality of candidate data is extracted, the same number of original association information as the plurality of selected candidate data is generated.

  In step S43, it is determined whether or not original association information has been generated. If at least one piece of original association information has been generated, the process proceeds to step S44. If none has been generated, the process returns to the replacement data generation process. In step S44, a priority setting screen is displayed on display 161. The priority setting screen is, for example, the priority setting screen shown in FIG.

  In the next step S45, priority is accepted. If the user instructs one of the selection buttons 711 to 714 according to the priority setting screen and indicates the decision button 715, the priority is accepted. Then, priority information is generated (step S46). The priority information includes two sets of original identification information and priority. Specifically, the priority information includes a set of first original identification information and priority for the first original data, and a set of second original identification information and priority for the second original data. And including.

  In the next step S47, the original association information generated in step S33, step S37 or step S42 is transmitted to the server 500, and the process proceeds to step S48. In step S48, the priority information generated in step S46 is transmitted to server 500, and the process returns to the replacement data generation process.

  FIG. 21 is a flowchart illustrating an example of the flow of the virtual reality realization process. The virtual reality realization process is a process executed by the CPU 501 when the CPU 501 provided in the server 500 loads the virtual reality realization program stored in the ROM 502 or the HDD 504 to the RAM 502 and executes it. Referring to FIG. 21, CPU 501 determines whether or not registration information has been received. It is determined whether communication unit 505 has received registration information from MFP 100. If registration information is received, the process proceeds to step S52; otherwise, the process proceeds to step S53. In step S52, the received registration information is stored in HDD 504, and the process proceeds to step S53.

  In step S53, it is determined whether or not the original association information has been received. It is determined whether communication unit 505 has received the original association information from MFP 100. If the original association information is received, the process proceeds to step S54; otherwise, the process proceeds to step S55. In step S54, the received original association information is stored in HDD 504, and the process proceeds to step S55.

  In step S55, it is determined whether priority information has been received. It is determined whether communication unit 505 has received priority information from MFP 100. If priority information is received, the process proceeds to step S56; otherwise, the process proceeds to step S57. In step S56, the received priority information is stored in HDD 504, and the process proceeds to step S57.

  In step S57, it is determined whether a transmission request has been received. It is determined whether communication unit 505 has received a transmission request from any of portable information devices 200, 200A, 200B or HMD 400. The transmission request includes acquired data and an AR tag. If a transmission request is received, the process proceeds to step S58; otherwise, the process returns to step S51. Here, a case where a transmission request is received from portable information device 200 will be described as an example.

  In step S58, registration information including the AR tag included in the received transmission request is extracted from the registration information stored in HDD 115 in step S52. In the next step S59, it is determined whether or not registration information has been extracted. If the registration information is extracted, the process proceeds to step S60. If not, the process returns to step S51. In step S60, the original data is specified. The original data specified by the original identification information included in the extracted registration information is specified.

  In the next step S61, the related part is returned, and the process proceeds to step S62. Via the communication unit 505, the related part included in the extracted registration information is transmitted to the device that has transmitted the transmission request, here, the portable information device 200. In step S62, the transmission history is stored in HDD 504, and the process proceeds to step S63. The transmission history includes the original identification information of the original data specified in step S60 and the device identification information of the device that has transmitted the transmission request received in step S57.

  In step S63, the acquired data included in the transmission request is compared with the replacement data included in the registration information. And it is judged whether the additional part was extracted (step S64). The additional portion is a portion that exists in the acquired data but does not exist in the replacement data. If the additional part is extracted, the process proceeds to step S65. If the additional part cannot be extracted, the process returns to step S51.

  In step S65, the related part of the original data is updated, and the process proceeds to step S66. The related part is updated by adding the additional part extracted in step S64 to the related part included in the registration information extracted in step S58.

  In step S66, the interlock update process is executed, and the process returns to step S51.

  FIG. 22 is a flowchart illustrating an example of the flow of linked update processing. The linked update process is a process executed in step S65 of FIG. Before the linked update process is executed, the original data to be processed is specified, and the additional part is determined. Referring to FIG. 22, CPU 501 determines whether or not original association information exists (step S71). Original association information including original identification information of original data to be processed is extracted from the original association information stored in the HDD 115. If such original association information is extracted, the process proceeds to step S72. If not, the process returns to the virtual reality implementation process. In step S72, related original data is specified. Of the two pieces of original identification information included in the original association information extracted in step S71, the original data specified by the original identification information different from the original identification information of the original data to be processed is specified as the related original data. In the next step S73, it is determined whether or not a transmission history exists. It is determined whether or not a transmission history including original identification information of related original data is stored in HDD 504. If there is a transmission history, the process proceeds to step S74; otherwise, the process returns to the virtual reality implementation process.

  In step S74, the related device is specified. The device specified by the device identification information included in the transmission history including the original identification information of the related original data is specified as the related device. Here, a case where portable information device 200A is specified as a related device will be described as an example.

  In step S75, it is determined whether priority information exists. From the priority information stored in the HDD 115, priority information including original identification information of the original data to be processed and the related original data is extracted. If such priority information is extracted, the process proceeds to step S76. If not, the process proceeds to step S78.

  In step S76, the priorities are compared. The priority of the original data to be processed determined by the priority information extracted in step S75 is compared with the priority of the related original data (step S77). Then, it is determined whether or not the comparison result is the first comparison result. Here, the first comparison result is a case where the priority of the original data to be processed is equal to or higher than the priority of the related original data. If the comparison result of the two priorities is the first comparison result, the process proceeds to step S78; otherwise, the process returns to the virtual reality implementation process.

  In step S78, the related part of the related original data is specified. Of the registration information stored in the HDD 115 in step S52, the registration information including the original identification information of the related original data specified in step S72 is extracted, and the related part included in the extracted registration information is specified.

  In step S79, the related part of the related original data is updated, and the process proceeds to step S80. The related part is updated by adding the additional part extracted in step S64 to the related part specified in step S78. Specifically, the related part included in the registration information extracted in step S78 is updated.

  In step S80, the additional part is transmitted to the related device, and the process returns to the virtual reality implementation process. Via the communication unit 505, the additional part is transmitted to the related device identified in step S74, here, the portable information device 200A.

  FIG. 23 is a flowchart illustrating an example of the flow of the augmented reality display process. The augmented reality display process is a process executed by the CPU 201 when the CPU 201 included in the portable information device 200 executes the augmented reality display program stored in the flash memory 203. Referring to FIG. 23, CPU 201 determines whether or not a document has been imaged (step S201). If the camera 202 has captured the subject, the process proceeds to step S202. If not, the process proceeds to step S203. Here, the subject of the camera 202 is a sheet on which an image of replacement data is formed by the MFP 100. In step S202, the captured image output by the camera 202 imaging the subject is set as acquired data, and the process proceeds to step S205.

  In step S203, it is determined whether wireless LAN I / F 208 has received data from the outside. If data is received, the process proceeds to step S204; otherwise, the process returns to step S201. Here, a case where replacement data is received from MFP 100 will be described as an example. In step S204, the replacement data received by wireless LAN I / F 208 from MFP 100 is set as acquired data, and the process proceeds to step S205.

  In step S205, the acquired data image set in step S202 or step S204 is displayed on display 206, and the process proceeds to step S206. In step S206, it is determined whether an AR tag has been extracted from the acquired data. If the AR tag is extracted, the process proceeds to step S207; otherwise, the process ends.

  In step S207, a transmission request is transmitted to server 500. The transmission request includes acquired data and an AR tag. Then, the wireless LAN I / F 208 determines whether or not the relevant part has been received from the server 500 (step S208). The process waits until the related part is received. If the related part is received, the process proceeds to step S209. In step S209, the AR tag included in the acquired data is replaced with the related part, and the process proceeds to step S210. The related part that replaces the AR tag is the related part acquired in step S208. Specifically, using the position information included in the related part, the replacement target area is determined based on the position and size of the AR tag placed in the acquired data, and the related part is placed in the determined replacement target area.

  In step S210, the related part is displayed, and the process proceeds to step S211. Specifically, the acquired data in which the related portions are arranged is displayed on the display unit 206. As a result, the original data image in which the AR tag portion of the acquired data is replaced with the related portion is displayed on the display unit 206.

  In step S211, it is determined whether an additional portion has been received. If the wireless LAN I / F 208 receives an additional portion from the server 500, the process proceeds to step S212. If not, the process ends. In step S212, the additional part is displayed on display unit 206, and the process ends. Using the position information included in the additional part, the replacement target area is determined based on the position and size of the AR tag arranged in the acquired data, and the additional part is placed in the determined replacement target area.

<Modification>
In the embodiment described above, the portable information device 200 has been described as an example of the augmented reality display device. However, the HMD 400 may function as the augmented reality display device. In this case, the CPU 401 included in the HMD 400 has the same function as that of the CPU 201 included in the portable information device 200 illustrated in FIG. 10, but the display control unit 251 is different.

  The display control unit 251 included in the CPU 401 included in the HMD 400 displays a related portion on the display unit 404 without displaying the captured image on the display unit 404 in response to the input of the captured image from the imaging control unit 261. The user wearing the HMD 400 actually sees the same image as the captured image through the display surface of the display unit 404. For this reason, the display control unit 251 specifies an area determined by the position information included in the related portion on the display surface of the display unit 404 with reference to the AR tag included in the captured image, and displays the display surface of the display unit 404. The image of the relevant part is displayed in the area specified in the area. Similarly, the interlocking display control unit 263 specifies an area determined by the position information included in the additional portion on the display surface of the display unit 404 with reference to the AR tag included in the captured image. The image of the additional portion is displayed in the area specified on the display surface.

  In the modification, the CPU 401 provided in the HMD 400 executes the augmented reality display process shown in FIG. 23, but steps S205, S210, and S212 are different. In step S205, the CPU 401 provided in the HMD 400 does not display the acquired data image. In step S210, the CPU 401 specifies an area determined by position information included in the related part on the display surface of the display unit 404 with reference to the AR tag included in the captured image, and specifies an image of the related part. And the process proceeds to step S211. In step S212, the CPU 401 specifies an area determined by the position information included in the additional portion on the display surface of the display unit 404 with reference to the AR tag included in the captured image, and specifies an image of the additional portion. Display in the selected area and finish the process.

  As described above, in the image processing system 1 according to the present embodiment, the MFP 100 converts the original data into replacement data in which a related part, which is at least a part of the original data, is replaced with an AR tag, Registration information including original identification information for identifying original data, an AR tag, a related part, and replacement data is registered in the server, and the replacement data is output. In response to receiving registration information from MFP 100, server 500 associates original identification information included in the registration information, an AR tag, a related portion, and replacement data, and includes an AR tag and acquired data. In response to receiving the request from the first portable information device of portable information devices 200, 200A, 200B or HMD 400, the related part associated with the AR tag included in the transmission request is transmitted to the first portable information device. To do. Furthermore, the server 500 extracts an additional portion that exists in the acquired data included in the transmission request but does not exist in the replacement data associated with the AR tag included in the transmission request, and the additional portion includes the AR tag included in the transmission request. Update the related part by adding to the related part associated with. Each of portable information devices 200, 200A, 200B or HMD 400 acquires replacement data output from MFP 100 or data obtained by changing a part of replacement data as acquisition data, extracts an AR tag from the acquisition data, When the related portion associated with the AR tag is received, an image of the related portion is displayed at a position determined based on the position of the AR tag in the acquired data. For this reason, for example, since the related portion is displayed at the position determined by the AR tag in the acquired data obtained in the portable information device 200, the related portion can be displayed in the acquired data. When the additional part is extracted from the acquired data received from the portable information device 200 by the server, the related part is updated by adding the additional part to the related part associated with the AR tag. For this reason, for example, in another portable information device 200A that acquires replacement data, the updated related portion is displayed, so that an additional portion in which a certain user has changed replacement data can be notified to another user. .

  Further, MFP 100 registers original association information in which the original data is associated with another original data different from the original data in the server, and server 500 further updates the first related portion when the first related portion is updated. The first original data associated with one related portion and the second original data associated with the original association information registered by MFP 100 are identified, and the additional portion is associated with the second original data Update the second related part by adding to the second related part. Therefore, when the first original data and the second original data associated with MFP 100 are specified, the second related portion associated with the second original data is updated. Therefore, the same additional part as the additional part added to the related part corresponding to the first original data can be added to the related part corresponding to the second original data. As a result, the same additional part as the additional part added to one related part can be added to the other related part.

  In addition, MFP 100 associates each of a plurality of pages included in the acquired data with each other as a plurality of original data. For this reason, it is possible to notify a user who sees the replacement data of the original data of another page of a change made to the replacement data corresponding to the original data of any page of the same data.

  Further, MFP 100 associates a plurality of original data respectively corresponding to a plurality of replacement data output by a plurality of operations by the same user or a plurality of operations by a plurality of users belonging to the same group. For this reason, changes made to any of multiple replacement data output by multiple operations by the same user or multiple operations by any of multiple users belonging to the same group can be made available to users who view other replacement data. You can be notified.

  In addition, MFP 100 acquires original data as candidate data from the server, and associates original data corresponding to the output replacement data with candidate data selected by the user from the candidate data. Therefore, a change made to the replacement data corresponding to any of the plurality of original data associated by the user who operates MFP 100 can be notified to the user who sees the replacement data corresponding to the other original data.

  Further, the server 500 is a case where an additional part is extracted from acquired data received from the first portable information device, for example, the portable information device 200, and is associated with the AR tag received from the portable information device 200. When the first original data and the second original data related by the MFP 100 are specified, the second portable information device to which the second related part related to the second original data is transmitted, for example, The additional part is transmitted to portable information device 200A. In portable information device 200A, the second related part and the additional part are displayed. For this reason, in the acquired data acquired in the portable information device 200, an additional portion changed from the replacement data can be displayed on the portable information device 200A. As a result, the change added to the replacement data corresponding to the first original data can be notified to the user who views the replacement data corresponding to the second original data.

  Further, when the priority of the first original data is equal to or higher than the priority of the second original data, the server 500 transmits the additional portion to the portable information device 200A that is the second portable information device. The change added to the replacement data corresponding to the original data is not notified to the user who views the replacement data corresponding to the first original data. Therefore, whether or not to display the additional portion can be determined based on the priority of the related original data.

  In the above-described embodiment, the image processing system 1 has been described. However, the replacement data generation process illustrated in FIGS. 18 to 20 is executed by the MFP 100, and the virtual reality implementation process illustrated in FIGS. 21 and 22 is performed. It goes without saying that the invention can be understood as a virtual reality display method that is executed by the server 500 and causes the portable information device 200, 200A, 200B or HMD 400 to execute the augmented reality display processing shown in FIG.

  The embodiment disclosed this time should be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

<Appendix>
(1) The output control means included in the image forming apparatus includes an image formation control means for forming an image of the replacement data on a sheet,
Each of the plurality of portable information devices further includes imaging means for imaging a subject,
The acquisition data acquisition unit included in each of the plurality of portable information devices is an imaging control that acquires, as acquisition data, a captured image obtained by the imaging unit imaging a sheet on which an image is formed by the image forming apparatus as a subject. Including means,
The image processing system according to claim 1, wherein the AR tag extraction unit included in each of the plurality of portable information devices analyzes an image of the acquired data and extracts an AR tag.
(2) The image processing system according to claim 1, wherein the output control means included in the image forming apparatus further includes data transmission means for transmitting the replacement data to the portable information device.
(3) The image processing system according to (1), wherein the data transmission unit includes an e-mail transmission unit that transmits an e-mail including the replacement data.
(4) The image processing system according to claim 1, wherein the output control means included in the image forming apparatus further includes a facsimile transmission means for transmitting the replacement data by facsimile.

  1 augmented reality display system, 2 network, 3 wireless station, 100 MFP, 200, 200A, 200B portable information device, 300 PC, 400 HMD, 500 server, 110 main circuit, 111 CPU, 112 communication I / F unit, 113 ROM 114 RAM, 115 HDD 115 Facsimile unit, 117 External storage device, 118 CD-ROM, 120 Automatic document feeder, 130 Document reading unit, 140 Image forming unit, 150 Paper feed unit, 155 Post-processing unit, 160 Operation panel, 161 Display unit, 163 operation unit, 201 CPU, 202 camera, 203 flash memory, 204 wireless communication unit, 205 call unit, 206 display unit, 207 operation unit, 208 wireless LAN I / F, 209 position detection unit, 210 direction detection unit, 211 outside Storage device, 401 CPU, 402 Camera, 403 Flash memory, 404 Display unit, 405 Wireless LAN I / F, 501 CPU, 502 ROM, 503 RAM, 504 HDD, 505 Communication unit, 506 Display unit, 507 Operation unit, 51 Data acquisition Part, 53 replacement part, 55 tag registration part, 55 transmission request reception part, 56 additional part extraction part, 57 output control part, 59 user authentication part, 61 data association registration part, 63 priority registration part, 65 image formation control 67, data transmission unit, 69 facsimile transmission control unit, 71 identical data association unit, 73 identical user association unit, 75 candidate data acquisition unit, 77 operation association unit, 251 display control unit, 253 acquisition data acquisition unit, 255 Tag Extraction Unit, 257 Related Part Request Unit, 259 Related Part Reception , 261 Imaging control unit, 263 Interlock display control unit, 267 Interlock transmission unit, 361 Replacement unit, 551 Tag association unit, 553 Registration information extraction unit, 555 Transmission request reception unit, 557 Original association unit, 559 Related partial reply unit , 561 related original specifying unit, 563 additional part extracting unit, 565 device specifying unit, 567 linked transmission unit, 568 priority setting unit, 571 related part updating unit, 573 linked updating unit, 581 rank comparison unit.

Claims (8)

An image processing system including an image forming apparatus, a plurality of portable information devices, and a server,
The image forming apparatus includes a replacement unit that converts the original data into replacement data in which a related part that is at least a part of the original data is replaced with an AR tag;
AR tag registration means for transmitting registration information including original identification information for identifying the original data, the AR tag, the related portion, and the replacement data to the server;
Output control means for outputting the replacement data,
The server includes an AR tag association unit that associates the original data, the AR tag, the related portion, and the replacement data included in the registration information in response to the registration information being received from the image forming apparatus. ,
In response to receiving a transmission request including an AR tag and acquired data from any of the plurality of portable information devices, a related portion associated with the AR tag included in the transmission request is changed to the plurality of portable information devices. Related partial reply means for transmitting to the portable information device that has transmitted the transmission request,
An additional part extracting means for extracting an additional part that exists in the acquisition data included in the transmission request but does not exist in the replacement data associated with the AR tag included in the transmission request;
Associated part update means for updating the associated part by adding the added part to the associated part associated with the AR tag included in the transmission request;
Each of the plurality of portable information devices includes acquisition data acquisition means for acquiring replacement data output by the image forming apparatus or data obtained by changing a part of the replacement data as acquisition data;
An AR tag extracting means for extracting an AR tag from the acquired data;
Related portion requesting means for transmitting a transmission request including the AR tag extracted from the acquired data and the acquired data to the server;
Associated part receiving means for receiving an associated part associated with the AR tag from the server after transmitting the transmission request;
An image processing system comprising: display control means for displaying an image of the received related portion at a position determined based on a position in the acquired data of the AR tag extracted from the acquired data. The image forming apparatus further comprises data association registration means for registering in the server original association information that associates original data with another original data different from the original data,
The server further includes:
When the first related part is updated by the related part updating unit, the first related data associated with the first related part is associated with the original association information registered by the image forming apparatus. Related original specifying means for specifying the second original data,
The related part updating means, when the second original data is specified by the related original specifying means, the additional part extracted by the additional part extracting means is associated with the second original data. The image processing system according to claim 1, further comprising linked update means for updating the second related part by adding to two related parts. The image forming apparatus further includes data acquisition means for acquiring data,
The image processing system according to claim 2, wherein the data association registration unit includes the same data association unit that associates each of a plurality of pages included in the acquired data as a plurality of original data.   The data association registration means includes a plurality of originals respectively corresponding to a plurality of replacement data output by the data output means by a plurality of operations by one user or a plurality of operations by a plurality of users belonging to the same group. The image processing system according to claim 2, further comprising the same user association means for associating data. The data association registration means includes candidate data acquisition means for acquiring original data as candidate data from the server;
The image processing system according to claim 2, further comprising operation association means for associating original data corresponding to the replacement data output by the data output means with candidate data selected by the user from among the candidate data. . In the server, the first related part is updated by the related part updating unit based on a first transmission request received from the first portable information device among the plurality of portable information devices, and the original data specifying unit The first original data associated with the first related portion and the second original data associated with the original association information are specified by the association update means and associated with the second original data. When the second related part is updated, the device specification for specifying the second portable information device to which the second related part before the update is transmitted by the related part return means among the plurality of portable information devices Means,
When the second portable information device is specified by the device specifying means, the second portable information device is updated to the second portable information device in response to the update of the second related part by the linked update means. Further comprising interlocking transmission means for transmitting the second related part of
The display control means included in the second portable information device that has received the second related part of the plurality of portable information devices displays the second related part, and then displays the updated second information from the server. In response to receiving the two related parts, the received image of the second related part after update is received at a position determined based on the position of the AR tag extracted from the captured image in the captured image. The image processing system according to claim 2, further comprising interlocking display control means for displaying. The data association registration unit included in the image forming apparatus is a priority registration unit that registers, in the server, priority information that defines relative priorities for the first original data and the second original data. Further,
The linked update means included in the server includes priority comparison means for comparing the priority of the first original data with the priority of the second original data,
The image processing system according to claim 6, wherein the second related part is updated on condition that the comparison result by the priority comparison unit is a first comparison result. A virtual reality display method executed in an image processing system including an image forming apparatus, a plurality of portable information devices, and a server,
A replacement step of converting the original data into replacement data obtained by replacing the related data, which is at least a part of the original data, with an AR tag;
An AR tag registration step of transmitting registration information including original identification information for identifying the original data, the AR tag, the related portion, and the replacement data to the server;
An output control step for outputting the replacement data; and
An AR tag associating step associating the original data, the AR tag, the related portion, and the replacement data included in the registration information in response to the server receiving the registration information from the image forming apparatus; ,
In response to receiving a transmission request including an AR tag and acquired data from any of the plurality of portable information devices, a related portion associated with the AR tag included in the transmission request is changed to the plurality of portable information devices. A related partial reply step for transmitting to the portable information device that has transmitted the transmission request,
An additional part extracting step of extracting an additional part that exists in the acquisition data included in the transmission request but does not exist in the replacement data associated with the AR tag included in the transmission request;
Updating a related part by adding the additional part to a related part associated with an AR tag included in the transmission request; and
An acquisition data acquisition step of acquiring, as acquisition data, replacement data output by the image forming apparatus, or data obtained by changing a part of the replacement data, in each of the plurality of portable information devices;
An AR tag extraction step of extracting an AR tag from the acquired data;
An associated partial requesting step of transmitting a transmission request including the AR tag extracted from the acquired data and the acquired data to the server;
Receiving a related part associated with the AR tag from the server after transmitting the transmission request; and
A virtual reality display method for executing a display control step of displaying the received image of the related portion at a position determined based on a position in the acquired data of the AR tag extracted from the acquired data.

Images