JP2016197300A - Ar supporting system, content providing method, and computer program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a content corresponding to an AR marker to each user more desirably than before without requiring users to share one printed matter if any one of the users does not have a printed matter.SOLUTION: A confidential content 52 is displayed as indicated below so that the confidential content 52 will overlap with a printed matter 5 with a calling marker 57 and a nonconfidential content 51 described. When a terminal device sends the calling marker 57, it will be determined that the users of the terminal device has the printed matter 5. If the users do not have the printed matter 5, the nonconfidential content 51 and the confidential content 52 will be distributed to the terminal device. If the users have the printed matter 5, the confidential content 52 alone will be distributed.SELECTED DRAWING: Figure 6

Description

  The present invention relates to AR (Augmented Reality) technology.

  In recent years, AR (Augmented Reality) technology has begun to spread. AR is generally translated as “augmented reality”.

  According to the AR technology, when a printed material with an AR marker attached is photographed with a terminal device such as a smartphone or a wearable computer, the content corresponding to the AR marker is displayed on the printed material or its image. As a result, the user can feel as if the real space has been expanded.

  AR technology is also used to hide confidential information. That is, content to be classified (confidential content) among a plurality of contents constituting the document is replaced with an AR marker. The confidential content and the AR marker are stored in the server in association with each other, and the document in which the confidential content is replaced with the AR marker is printed on paper and distributed to the user. The user can download and display the confidential content from the server by photographing the paper with the terminal device. At this time, it appears to the user as if the confidential content is printed on the paper.

  In addition, the following methods have been proposed as methods for providing information. When the browsing count for a predetermined homepage browsed by the viewer via the network and the terminal device for the viewer connected thereto is counted in the cooperative service providing server, and the browsing count reaches a predetermined browsing count set in advance As a presentation of specific information selected from predetermined predetermined information or a plurality of predetermined specific information, a specific home page can be browsed or various kinds of information are specially provided. Or information for providing prizes such as goods and vouchers (Patent Document 1).

JP 2001-325515 A

  If a printed matter to which an AR marker is attached is distributed to a plurality of users one by one, these users can simultaneously browse content corresponding to the AR marker. For example, such printed matter is distributed to the participants one by one before the meeting, and brought to the meeting at the meeting. Accordingly, it is possible to safely distribute the printed material in advance.

  However, participants who have suddenly joined the meeting may not have received such printed matter. The printed matter may be distributed to the participants immediately before the meeting starts or during the meeting. However, if there is no photocopier nearby, the user must go to the place where the photocopier is installed, delaying the start of the conference or interrupting the conference. Even if there is a copier nearby, some delay or interruption may occur.

  Therefore, this participant may have other participants who have brought the printed material show the printed material from the side. Then, the participant's terminal device downloads and displays the content corresponding to the AR marker.

  However, each participant wants to view the printed material in a free position. In addition, a participant who has been distributed with a printed material may add a memo to the printed material, but it is difficult to add a memo when a participant who does not have the printed material is looking from the side. Therefore, it is not very desirable to view one printed matter by a plurality of people.

  In the present invention, in view of such problems, even when there is a user who does not have a printed matter, content corresponding to the AR marker is conventionally provided to each user without sharing a single printed matter among a plurality of users. It aims at providing more suitably.

  An AR (Augmented Reality) support system according to an aspect of the present invention includes a plurality of terminals that realize an AR by displaying the second content so as to overlap the paper on which the marker and the first content are written. An AR support system that provides the device with the second content, and when the marker is transmitted from any of the plurality of terminal devices, the user of the terminal device that is the transmission source of the marker Determining means for determining whether the user is the owner of the paper, and a terminal device used by the user among the plurality of terminal devices when the user is determined not to be the owner. A first transmission means for transmitting both the first content and the second content to a certain allocation terminal device; and when the user is determined to be the owner, Having a second transmission means for transmitting the second content only of the assigned terminal the first content and the second content to the user's device.

  Preferably, storage means for storing the number of copies issued the paper, and the determination means until the number of terminal devices that have transmitted the marker among the plurality of terminal devices reaches the number of copies. The user is determined to be the owner, and after the number of copies reaches the number of copies, the user is determined not to be the owner.

  Alternatively, the determination unit determines that the user is the owner if the user's assigned terminal device is registered in a predetermined database in advance, and determines that the user is not the owner if the user is not registered. . Or you may discriminate | determine that the said user is not the said owner, when the said marker continues being detected for the predetermined time in the said allocation terminal device of the said user. Alternatively, the user may be determined not to be the owner when the marker is detected a predetermined number of times intermittently at the user's assigned terminal device.

  Alternatively, the first transmission means makes an inquiry to the terminal device of the user determined not to be the owner of the plurality of terminal devices, and when there is a predetermined answer to the inquiry The first content and the second content are transmitted.

  According to the present invention, even if there is a user who does not have a printed matter on which a marker is written, the content corresponding to the marker is more suitable for each user than before, without sharing a single printed matter among a plurality of users. Can be provided.

It is a figure which shows the example of the whole structure of a document browsing system. 2 is a diagram illustrating an example of a hardware configuration of an image forming apparatus. FIG. It is a figure which shows the example of the hardware constitutions of an air tag server. It is a figure which shows the example of the hardware constitutions of a tablet type terminal device. It is a figure which shows the example of the hardware constitutions of a spectacles type terminal device. It is a figure which shows the example of printed matter. It is a figure which shows the example of a whole document. 2 is a diagram illustrating an example of a functional configuration of an image forming apparatus. FIG. It is a figure which shows the example of a functional structure of an air tag server. It is a figure which shows the example of a functional structure of a terminal device. It is a figure which shows the example of a whole document memory | storage part. It is a figure which shows the example of an air tag memory | storage part. It is a figure which shows the example of a browsing condition memory | storage part. It is a sequence diagram which shows the example of the flow of a process in a certain meeting of each apparatus. It is a sequence diagram which shows the example of the flow of a process in a certain meeting of each apparatus. It is a sequence diagram which shows the example of the flow of a process in a certain meeting of each apparatus. 6 is a flowchart illustrating an example of the overall processing flow of the image forming apparatus. It is a flowchart which shows the example of the flow of the whole process of an air tag server. It is a flowchart which shows the example of the flow of the whole process of a terminal device. It is a figure which shows the example of distributor data. It is a figure which shows the example of delivery reference | standard data.

  FIG. 1 is a diagram illustrating an example of the overall configuration of the document browsing system 100. FIG. 2 is a diagram illustrating an example of a hardware configuration of the image forming apparatus 1. FIG. 3 is a diagram illustrating an example of a hardware configuration of the air tag server 2. FIG. 4 is a diagram illustrating an example of a hardware configuration of the tablet terminal device 3T. FIG. 5 is a diagram illustrating an example of a hardware configuration of the glasses-type terminal device 3M. FIG. 6 is a diagram illustrating an example of the printed material 5. FIG. 7 is a diagram illustrating an example of the entire document 50.

  As shown in FIG. 1, the document browsing system 100 includes an image forming apparatus 1, an air tag server 2, a terminal device 3, a communication line 4, and the like. The document browsing system 100 manages data related to a document (document) and provides a user with a document browsing service.

  The image forming apparatus 1, the air tag server 2, and the terminal device 3 can communicate with each other via the communication line 4. As the communication line 4, the Internet, a so-called LAN (Local Area Network) line, a cellular phone line, or the like is used.

  As shown in FIG. 2, the image forming apparatus 1 includes a central processing unit (CPU) 10a, a random access memory (RAM) 10b, a read only memory (ROM) 10c, an auxiliary storage device 10d, a touch panel display 10e, and an operation key panel 10f. , NIC (Network Interface Card) 10g, modem 10h, scan unit 10i, print unit 10j, and the like.

  The touch panel display 10e displays a screen showing a message for the user, a screen for the user to input a command or information, a screen showing a result of processing executed by the CPU 10a and the like. Further, the touch panel display 10e sends a signal indicating the touched position to the CPU 10a.

  The operation key panel 10f is a so-called hardware keyboard, and includes a numeric keypad, a start key, a stop key, a function key, and the like.

  The NIC 10g communicates with the air tag server 2 or the terminal device 3 by a protocol such as TCP / IP (Transmission Control Protocol / Internet Protocol).

  The modem 10h exchanges image data with a facsimile terminal using a protocol such as G3.

  The scan unit 10i generates image data by scanning the set paper.

  The print unit 10j prints an image read by the scan unit 10i or an image received from another device on a sheet.

  The ROM 10c or the auxiliary storage device 10d stores a program for realizing the above-described functions such as copying. Further, a document management program 1P is stored. The document management program 1P is a program for registering and providing a document.

  These programs are loaded into the RAM 10b as necessary and executed by the CPU 10a. A hard disk, SSD (Solid State Drive), or the like is used as the auxiliary storage device 10d.

  The air tag server 2 stores data for displaying the air tag and provides it to the terminal device 3 as appropriate. The “air tag” generally means information virtually attached to the actual object or an image thereof. A guide or description attached to a subject (such as a building or an exhibit) photographed through a camera is an example of an air tag.

  A so-called server machine is used as the air tag server 2. Alternatively, a so-called cloud server may be used.

  As shown in FIG. 3, the air tag server 2 includes a CPU 20a, a RAM 20b, a ROM 20c, an auxiliary storage device 20d, a NIC 20e, and the like.

  The NIC 20e communicates with the image forming apparatus 1 or the terminal device 3 using a protocol such as TCP / IP.

  An air tag management program 2P is stored in the ROM 20c or the auxiliary storage device 20d. The air tag management program 2P is loaded into the RAM 20b and executed by the CPU 20a. A hard disk or SSD is used as the auxiliary storage device 20d.

  The terminal device 3 displays all or part of the document based on data provided from the image forming apparatus 1 or the air tag server 2. As the terminal device 3, a personal computer, a tablet computer, a smartphone, a glasses-type computer, or the like is used.

  The “glasses-type computer” is a kind of wearable computer, and is a computer that is worn on the face like glasses such as Google Glass of Google.

  Hereinafter, a case where a tablet computer or a glasses-type computer is used as the terminal device 3 will be described as an example. The former and the latter may be described separately from “tablet type terminal device 3T” and “glasses type terminal device 3M”, respectively.

  As shown in FIG. 4, the tablet-type terminal device 3T includes a CPU 30a, a RAM 30b, a ROM 30c, a flash memory 30d, a touch panel display 30e, an operation button group 30f, a digital camera 30g, a wireless communication device 30h, and the like.

  The touch panel display 30e is configured by a flat display (for example, a liquid crystal display) and a touch panel. The flat display displays various images. Especially used for displaying documents. The touch panel detects the position touched by the user and notifies the CPU 30a of the position.

  The operation button group 30f includes a button for returning to the home screen, a button for confirming something, a button for adjusting the volume, a button for switching on / off the power, and the like.

  The digital camera 30g captures various images. In particular, it is used for photographing paper for viewing documents.

  The wireless communication device 30h communicates with the image forming device 1 or the air tag server 2 by a protocol such as TCP / IP via a base station that configures the communication line 4. As a communication method of the wireless communication device 30h, for example, Wi-Fi, LTE (Long Term Evolution), W-CDMA (Wideband-Code Division Multiple Access), PHS (Personal Handy-phone System), or the like is used.

  A client program 3P is stored in the ROM 30c or the flash memory 30d. The client program 3P is loaded into the RAM 30b and executed by the CPU 30a.

  As shown in FIG. 5, the eyeglass-type terminal device 3M uses a CPU 31a, a RAM 31b, a ROM 31c, a flash memory 31d, a display 31e, an operation button group 31f, a digital camera 31g, a wireless communication device 31h, and the like.

  The roles of the CPU 31a, RAM 31b, ROM 31c, flash memory 31d, display 31e, operation button group 31f, digital camera 31g, and wireless communication device 31h are the CPU 30a, RAM 30b, ROM 30c, flash memory 30d, and touch panel of the tablet terminal device 3T, respectively. This is basically the same as the display 30e, the operation button group 30f, the digital camera 30g, and the wireless communication device 30h.

  However, the method for reproducing an image is different between the touch panel display 30e and the display 31e. As described above, a flat display such as a liquid crystal display is used for the touch panel display 30e. On the other hand, the display 31e reproduces an image by emitting light from a projector provided at the base of the handle to a prism provided near the end of the lens.

  The client program 3P is stored in the ROM 31c or the flash memory 31d, loaded into the RAM 31b, and executed by the CPU 31a.

  Incidentally, the document browsing system 100 can not only simply provide a file such as PDF (Portable Document Format) but also reproduce the document by AR (Augmented Reality) technology. At this time, the printed material 5 (see FIG. 6) is used. The printed material 5 is distributed to a plurality of users.

  Hereinafter, a case where the document browsing system 100 is used in a certain company will be described as an example.

  Each employee, that is, each user has one terminal device 3. Hereinafter, the terminal devices 3 possessed by the user U1, the user U2, the user U3,... Are distinguished from the “terminal device 31”, the “terminal device 32”, the “terminal device 33”,.

  As shown in FIG. 6A, the non-confidential content 51 and the calling marker 57 are printed on the printed matter 5. The non-confidential content 51 is a non-confidential content among the contents constituting a document (hereinafter referred to as “whole document 50”) described in one page as shown in FIG. The call marker 57 is used to call the confidential content 52 from the air tag server 2. The confidential content 52 is confidential content among the contents constituting the entire document 50.

  When the user wears the glasses-type terminal device 3M and sees the printed matter 5, the data of the confidential content 52 is downloaded from the air tag server 2 based on the calling marker 57, and the confidential content 52 is not displayed as shown in FIG. It looks as if it is printed on the printed material 5 together with the confidential content 51. The same appears when the printed product 5 is viewed while being photographed with the tablet terminal device 3T.

  Thus, according to the document browsing system 100, AR is realized. An AR marker is used as the calling marker 57.

  Furthermore, according to the document browsing system 100, the entire document 50 can be provided even to a user who does not have the printed matter 5.

  Hereinafter, a mechanism for providing the entire document 50 to each of the user having the printed matter 5 and the user not having the printed matter 5 will be described.

  FIG. 8 is a diagram illustrating an example of a functional configuration of the image forming apparatus 1. FIG. 9 is a diagram illustrating an example of a functional configuration of the air tag server 2. FIG. 10 is a diagram illustrating an example of a functional configuration of the terminal device 3.

  According to the document management program 1P, the whole document acquisition unit 101, the confidential part designation receiving unit 102, the marker issuing unit 103, the whole document registration unit 104, the whole document storage unit 105, the air tag registration command unit 106, and the printed matter generation unit shown in FIG. Functions such as 107, document data transmission unit 121, transmission destination storage unit 122, display end necessity determination unit 123, and display end command unit 124 are realized in the image forming apparatus 1.

  According to the air tag management program 2P, an air tag registration unit 201, an air tag storage unit 202, a user attribute determination unit 203, an air tag data transmission unit 204, a viewer addition unit 205, a browsing status storage unit 206, an entire document transmission command unit shown in FIG. Functions such as 207 and the browser deletion unit 208 are realized in the air tag server 2.

  According to the client program 3P, functions such as a marker detection unit 301, an air tag request unit 302, an air tag display unit 303, an entire document display unit 304, and a display end notification unit 305 shown in FIG.

[Preparation of data and printed materials for viewing]
FIG. 11 is a diagram illustrating an example of the entire document storage unit 105. FIG. 12 is a diagram illustrating an example of the air tag storage unit 202. FIG. 13 is a diagram illustrating an example of the browsing status storage unit 206.

  In the image forming apparatus 1, the entire document storage unit 105 stores image data 60 for displaying each entire document 50 in association with the calling marker 57, as shown in FIG. 11.

  In the air tag server 2, as shown in FIG. 12, the air tag storage unit 202 stores image data 62 for displaying each air tag, that is, confidential content 52 in association with the calling marker 57. The

  The image data 60 and the image data 62 are prepared by the following method, for example. The user creates the entire document 50 with an application such as spreadsheet software or word processor software installed in the personal computer. Then, the data of the entire document 50 is converted into image data in a format such as PDF, GIF (Graphics Interchange Format), or JPEG (Joint Photographic Experts Group), and uploaded to the image forming apparatus 1.

  Alternatively, the user may prepare the image data of the entire document 50 in the image forming apparatus 1 by causing the scan unit 10 i of the image forming apparatus 1 to read a printed matter on which the entire document 50 is printed.

  In the image forming apparatus 1, the entire document acquisition unit 101 acquires image data prepared by the user as image data 60.

  The confidential part designation receiving unit 102 allows the user to designate the confidential content 52 in the entire document 50 as follows.

  The confidential part designation receiving unit 102 displays the entire document 50 on the touch panel display 10e based on the image data 60 acquired by the entire document acquisition unit 101. Here, the user designates a part to be concealed by surrounding it with a finger. Furthermore, the required number of copies of the printed matter 5 (hereinafter referred to as “issued number Sa”) is designated.

  Then, the confidential part designation accepting unit 102 accepts the designated part as the range of the confidential content 52. Furthermore, the designated number of copies Sa is accepted.

  The marker issuing unit 103 issues a unique calling marker 57 to the confidential content 52 for which the range specification has been received by the confidential part specification receiving unit 102.

  The overall document registration unit 104 stores the image data 60 acquired by the overall document acquisition unit 101 in the overall document storage unit 105 in association with the calling marker 57 issued by the marker issuing unit 103. As a result, the image data 60 is registered in the entire document storage unit 105.

  The air tag registration command unit 106 commands the air tag server 2 to register the confidential content 52 whose range specification has been received by the confidential part specification reception unit 102. At this time, the image data of the confidential content 52 is transmitted as image data 62 to the air tag server 2 together with the calling marker 57 issued by the marker issuing unit 103 and the number of copies Sa specified by the user. As the format of the image data 62, PDF, GIF, JPEG, or the like is used.

  In the air tag server 2, when the air tag registration unit 201 receives the image data 62 and the calling marker 57 together with a registration command, the air tag storage unit 202 stores them in association with each other. As a result, the image data 62 is registered in the air tag storage unit 202.

  The browsing status storage unit 206 stores browsing status data 63 for each confidential content 52 as shown in FIG. When the image data 62 of the confidential content 52 is newly registered in the air tag storage unit 202, the browsing status data 63 of the confidential content 52 is generated by the air tag registration unit 201 and stored in the browsing status storage unit 206.

  “Number of copies” of the browsing status data 63 is the number of copies of the printed matter 5 of the confidential content 52. That is, it means how many sheets of the confidential content 52 are printed. The initial value is the number of issued copies Sa received from the image forming apparatus 1 (that is, specified by the user). The “number of browsing terminal devices” is the number of terminal devices 3 on which the confidential content 52 is currently displayed by the image data 62. The initial value is “0”. The “browsing terminal device identifier” is an identifier of the terminal device 3 on which the confidential content 52 is currently displayed by the image data 62. Initially, no identifier of any terminal device 3 is shown. An IP (Internet Protocol) address or a MAC (Media Access Control) address is used as the browsing terminal device identifier.

  In the image forming apparatus 1, the printed material generation unit 107 generates the printed material 5 with the number of copies Sa specified by the user as follows.

  The printed material generation unit 107 generates the printed material 5 as follows. The confidential content 52 is deleted from the entire document 50. A call marker 57 is arranged at a predetermined position (for example, the center) of the portion where the confidential content 52 is deleted. Then, the whole document 50 subjected to these processes is printed on the white paper with the number of copies Sa by the print unit 10j. Thereby, the printed matter 5 with the number of copies Sa is generated.

[Document provision and display processing]
The user points the digital camera 30g or 31g of his / her terminal device 3 toward the printed surface of the printed matter 5. Then, an image of the printing surface is taken by the digital camera 30g or 31g. In the tablet terminal device 3T, the photographed image is displayed on the touch panel display 30e.

  In the terminal device 3, the marker detection unit 301 detects the calling marker 57 from the captured image when shooting is started. During imaging, the marker detection unit 301 performs detection every predetermined time (for example, 1 second).

  When the marker detection unit 301 detects the calling marker 57, the air tag requesting unit 302 requests the air tag server 2 for an air tag corresponding to the calling marker 57. At this time, the calling marker 57 is notified to the air tag server 2.

  In the air tag server 2, the user attribute determination unit 203 determines the attribute of the user who is the request source of the air tag. Specifically, it is determined whether or not this user is the owner of the printed matter 5. The determination method is as follows.

  The user attribute discriminating unit 203 reads the browsing status data 63 corresponding to the calling marker 57 notified from the terminal device 3 together with the request from the browsing status storage unit 206 (see FIG. 13), and indicates the browsing status data 63. The number of terminal devices being browsed is compared with the number of printed copies. If the number of browsing terminal devices is less than the number of copies, it is determined that this user is the owner of the printed matter 5. If the number of terminal devices being browsed is equal to or greater than the number of copies, it is determined that this user is not the owner of the printed matter 5.

  When the user attribute determining unit 203 determines that this user is the owner of the printed matter 5, the air tag data transmitting unit 204 and the viewer adding unit 205 execute the following processing.

  The air tag data transmission unit 204 reads out the image data 62 corresponding to the calling marker 57 notified from the user terminal device 3 from the air tag storage unit 202 (see FIG. 12). Then, the image data 62 is transmitted to the terminal device 3.

  The browsing person adding unit 205 uses the identifier (for example, IP address) of the terminal device 3 as the “browsing terminal device identifier” of the browsing status data 63 corresponding to the calling marker 57 described in the browsing status storage unit 206. "And" 1 "is added to" Number of browsing terminal devices ".

  On the other hand, if the user attribute determination unit 203 determines that the user is not the owner of the printed matter 5, the entire document transmission command unit 207 sends the image data 60 to the image forming apparatus 1 so as to transmit the image data 60 to the terminal device 3. Command. At this time, the calling marker 57 and the identifier of the terminal device 3 are notified to the image forming apparatus 1.

  In the image forming apparatus 1, when the document data transmission unit 121 receives a request from the air tag server 2, the image data 60 corresponding to the calling marker 57 notified together with the request is stored in the entire document storage unit 105 (see FIG. 11). Read from inside. Then, based on the identifier notified with the request, the read image data 60 is transmitted to the terminal device 3.

  Note that the document data transmission unit 121 may inquire about the necessity of transmission of the image data 60 to the terminal device, and may transmit the image data 60 when a reply that it is necessary is received.

  The transmission destination storage unit 122 stores the calling marker 57 and the identifier of the terminal device 3 in association with each other.

  In the terminal device 3, when the air tag display unit 303 receives the image data 62 from the air tag server 2, the confidential content 52 represented by the image data 62 is displayed on the touch panel display 30e or the display 31e as follows.

  In the case of the tablet-type terminal device 3T, the air tag display unit 303 detects the position of the calling marker 57 in the image (image of the printed matter 5) taken by the digital camera 30g and displayed by the touch panel display 30e. Then, the confidential content 52 is displayed in an overlapping manner so that the center of the confidential content 52 coincides with this position in the image of the printed matter 5.

  According to the above processing, for example, when the entire printed matter 5 shown in FIG. 6A is photographed, the confidential content 52 is displayed on the image of the printed matter 5 as shown in FIG. Is done.

  On the other hand, in the case of the glasses-type terminal device 3M, the air tag display unit 303 detects the position of the calling marker 57 in the image photographed by the digital camera 31g. Then, the confidential content 52 is displayed on the display 31e so that the center of the confidential content 52 coincides with this position.

  According to the above processing, for example, when the user sees the entire printed matter 5 shown in FIG. 6A while wearing the glasses-type terminal device 3M, it is almost the same as the state shown in FIG. 6B. The printed matter 5 is visible.

  When the entire document display unit 304 receives the image data 60 from the image forming apparatus 1, the entire document display unit 304 displays the entire document 50 represented by the image data 60 on the touch panel display 30e or the display 31e.

[End of browsing]
The air tag display unit 303 starts measuring time when the calling marker 57 is no longer detected by the marker detection unit 301 after starting to display the confidential content 52. Then, when a predetermined time (for example, 1 minute) elapses without detecting the calling marker 57 again, the display of the confidential content 52 is terminated.

  When the display of the confidential content 52 is completed, the display end notification unit 305 notifies the air tag server 2 to that effect. At this time, a call marker 57 corresponding to the confidential content 52 is transmitted to the air tag server 2.

  In the air tag server 2, when the viewer deletion unit 208 receives the calling marker 57 together with the notification of the end of display, the browsing device identifier of the browsing status data 63 corresponding to the calling marker 57 is used as the terminal device that is the transmission source. 3 IP address is deleted. Further, the number of browsing terminal devices is reduced by “1”.

  In the image forming apparatus 1, the display end necessity determining unit 123 monitors the browsing status data 63 stored in the air tag server 2 for the number of browsing terminal devices of “0”. When the browsing status data 63 that is “0” is found, the entire document in the terminal device 3 in which the IP address is stored in the transmission destination storage unit 122 in association with the calling marker 57 of the browsing status data 63. It is determined that the display of 50 should be ended.

  The display end command unit 124 instructs the terminal device 3 to end the display of the entire document 50. After the instruction, the IP address and the call marker 57 are deleted from the transmission destination storage unit 122.

  The air tag server 2 monitors the browsing status data 63 in which the number of terminal devices being browsed becomes “0”, and notifies the image forming apparatus 1 of the calling marker 57 of the browsing status data 63 from the air tag server 2. May be.

  Alternatively, regardless of the number of browsing terminal devices, a command to end display of the entire document 50 is input to the terminal device 3 of any user, and the command is transmitted from the terminal device 3 to the image forming apparatus 1. In this case, the display end command unit 124 may determine that the display of the entire document 50 should be ended.

  In the terminal device 3, the entire document display unit 304 ends the display of the entire document 50 when receiving an instruction to end the display of the entire document 50 from the image forming apparatus 1. This user may be a user who has the printed matter 5 or a user who does not have it.

〔Example of use〕
FIG. 14 to FIG. 16 are sequence diagrams showing an example of the flow of processing in a certain conference of each device.

  Next, one form of use of the document browsing system 100 will be described by taking the following case as an example with reference to FIGS. A printed matter 5 (see FIG. 6A) is distributed in advance to each of five users U1 to U5 as a distributed item for the conference. Thereafter, the user U6 suddenly decided to participate in the conference, but the printed matter 5 of the user U6 is not prepared.

  The meeting organizer prepares the meeting materials as the entire document 50. The image forming apparatus 1 acquires the image data 60 of the entire document 50 (# 701 in FIG. 14) and issues a new call marker 57 (# 702). Then, the image data 60 is registered in the entire document storage unit 105 (# 703).

  Further, the organizer designates the confidential part of the entire document 50 and designates “5” as the number of copies Sa.

  When the image forming apparatus 1 receives the designation of the confidential portion and the issuance number Sa (# 704), the image forming apparatus 1 extracts the portion as the confidential content 52 and instructs the air tag server 2 to register the confidential content 52. (# 705). At this time, the image data 62 of the confidential content 52 is generated and transmitted to the air tag server 2 together with the calling marker 57 issued in step # 702 and the accepted number of copies Sa.

  Upon receiving the registration command (# 721), the air tag server 2 associates the image data 62 and the calling marker 57 transmitted together with the command and registers them in the air tag storage unit 202 (see FIG. 12) (# 722). ). Further, the browsing status data 63 is generated and registered in the browsing status storage unit 206 (see FIG. 13) (# 723). The browsing status data 63 indicates the call marker 57 and the number of copies Sa that have been transmitted.

  In addition, the image forming apparatus 1 generates the printed matter 5 by printing the non-confidential content 51 and the calling marker 57 on the white paper with the number of copies Sa (that is, 5 sheets) (# 706).

  Then, the organizer distributes one printed matter 5 to each of the users U1 to U5. With the above processing and work, preparation for the conference is completed.

  Users U1 to U5 gather at the meeting venue. When the meeting starts, the user prints his / her printed material 5 through his / her terminal device 3. For example, the user U <b> 1 among the five users first browses the printed matter 5 through the terminal device 31.

  The terminal device 31 starts photographing the printed material 5 (# 741). When the calling marker 57 is detected (# 742), an air tag is requested from the air tag server 2 (# 743). At this time, the detected call marker 57 is transmitted to the air tag server 2. After that, the terminal device 31 may detect the same calling marker 57, but the request may be made only once when it is detected for the first time after the start of photographing. The same applies to the other terminal devices 3.

  When the air tag server 2 receives the request (# 724), the requesting user (user U1 in this example) prints the printed product 5 based on the browsing status data 63 corresponding to the call marker 57 transmitted along with the request. It is determined whether or not the user is an owner (# 725). Specifically, if the number of terminal devices being browsed indicated in the browsing status data 63 is less than the number of printed copies, it is determined that the user U1 is the owner of the printed matter 5. In this example, since the number of copies is “5” and the number of currently viewed terminal devices is “0”, it is determined that the user is the owner of the printed matter 5.

  When the air tag server 2 determines that it is the owner of the printed matter 5, it reads out the image data 62 corresponding to the calling marker 57 from the air tag storage unit 202 (# 726) and transmits it to the terminal device 31 (# 727). Then, the IP address of the terminal device 31 is added to the browsing terminal device identifier of the browsing status data 63, and “1” is added to the number of browsing terminal devices (# 728 in FIG. 15).

  Upon receiving the image data 62 (# 744), the terminal device 31 displays the confidential content 52 based on the image data 62 (# 745).

  Similarly, the users U2 to U5 browse their own printed matter 5 via their terminal devices 32 to 35. Then, the terminal devices 32 to 35 perform the same processing as the terminal device 31 (# 741 to # 745). At this time, the air tag server 2 also performs the same processing based on requests from the terminal devices 32 to 35 (# 724 to # 728).

  User U6 joins the meeting with a delay. However, since the user does not have the printed material 5, he / she asks one of the users U1 to U5 to show the printed material 5. At this time, the printed matter 5 is viewed through its own terminal device 3, that is, the terminal device 36.

  As with the terminal device 31, the terminal device 36 starts photographing the printed matter 5 (# 761), and when detecting the calling marker 57 (# 762), requests the air tag server 2 for an air tag (# 763). At this time, the detected call marker 57 is transmitted to the air tag server 2.

  When the air tag server 2 receives the request (# 729), the requesting user (user U6 in this example) prints the printed matter 5 based on the browsing status data 63 corresponding to the calling marker 57 transmitted along with the request. It is determined whether or not the user is an owner (# 730). The determination method is the same as in step # 725.

  Since the number of terminal devices being browsed has already reached the number of printed copies, the air tag server 2 determines that the user U6 is not the owner of the printed matter 5.

  Therefore, the air tag server 2 requests the image forming apparatus 1 to transmit the image data 60 to the terminal device 36 (# 732). At this time, the calling marker 57 received from the terminal device 36 and the IP address of the terminal device 36 are transmitted to the image forming apparatus 1.

  When receiving the request (# 707), the image forming apparatus 1 reads the image data 60 corresponding to the calling marker 57 transmitted with the request from the entire document storage unit 105 (# 708). The read image data 60 is transmitted to the IP address transmitted with the request (# 709). Then, the call marker 57 and the IP address are associated with each other and recorded in the transmission destination storage unit 122 (# 710).

  Upon receiving the image data 60 (# 764), the terminal device 36 displays the entire document 50 based on the image data 60 (# 766).

  As a result, the user U6 can view the confidential content 52 without having the printed matter 5. In addition, since the non-confidential content 51 is also included in the entire document 50, the non-confidential content 51 can be viewed even after the printed matter 5 deviates from the shooting range of the terminal device 36.

  When it becomes unnecessary to refer to the printed material 5, the user U <b> 1 stops viewing the printed material 5. Therefore, the printed material 5 is out of the shooting range of the terminal device 31.

  When the terminal device 31 stops detecting the calling marker 57 from the printed matter 5 and a predetermined time has elapsed (# 746), the terminal device 31 ends the display of the confidential content 52 (# 747). Then, the air tag server 2 is notified that the display has been completed (# 748 in FIG. 16). At this time, the calling marker 57 is transmitted to the air tag server 2.

  When the air tag server 2 receives the notification (# 733), the air tag server 2 deletes the terminal device 31 from the browsing terminal device identifier of the browsing status data 63 corresponding to the calling marker 57 transmitted along with the notification, and the number of browsing terminal devices. Is decreased by "1" (# 734).

  Users U2 to U5 also act in the same way as user U1. Therefore, the terminal devices 32 to 35 perform the same processing as the terminal device 31 (# 746 to # 748), and the air tag server 2 performs the same processing in response to the notification from the terminal devices 32 to 35 (# 733, # 734). As a result, the number of browsing terminal devices becomes “0”.

  When the image forming apparatus 1 detects the browsing status data 63 in which the number of browsing terminal devices is “0”, or one of the users U1 to U5 inputs a display end command (# 749). When this command is transmitted from any of the terminal devices 31 to 35 (# 750, # 711), it is determined that the display of the entire document 50 should be terminated (# 712). (Destination) In other words, the terminal device 36 is instructed to end the display (# 713).

  Upon receiving the command from the image forming apparatus 1 (# 766), the terminal device 36 ends displaying the entire document 50 (# 767).

  FIG. 17 is a flowchart illustrating an example of the overall processing flow of the image forming apparatus 1. FIG. 18 is a flowchart showing an example of the overall processing flow of the air tag server 2. FIG. 19 is a flowchart illustrating an example of the overall processing flow of the terminal device 3.

  Next, the overall processing flow of each of the image forming apparatus 1, the air tag server 2, and the terminal device 3 will be described with reference to flowcharts.

  The image forming apparatus 1 executes the processing shown in FIG. 17 in response to a user operation or a command from another apparatus.

  When the user prepares a new whole document 50 and inputs it to the image forming apparatus 1 (Yes in # 801), the image forming apparatus 1 issues a new call marker 57 (# 802), and the image of the whole document 50 is obtained. The data 60 is registered in the entire document storage unit 105 (see FIG. 11) in association with the calling marker 57 (# 803).

  Further, when the user designates the confidential range and the number of copies Sa to be issued, the image forming apparatus 1 accepts the designation (# 804) and instructs the air tag server 2 to register the range as the confidential content 52 ( # 805). At this time, the image data 62 of the confidential content 52, the designated number of copies Sa, and the calling marker 57 issued in step # 802 are transmitted to the air tag server 2.

  Alternatively, when the air tag server 2 requests that the image data 60 be transmitted to a certain terminal device 3 (Yes in # 806), the image forming apparatus 1 reads the terminal device 3 from the entire document storage unit 105 ( # 807) and transmit to the terminal device 3 (# 808). Then, the IP address of the terminal device 3 is associated with the calling marker 57 of the image data 60 and registered in the transmission destination storage unit 122 (# 809).

  Alternatively, when an instruction to end the display of a certain entire document 50 is transmitted from the terminal device 3 (# 810), the image forming apparatus 1 determines that the display of the entire document 50 should be ended, The terminal device 3 that is the transmission destination of the image data 60 of the entire document 50 is instructed to end the display (# 811). Then, the call marker 57 of the terminal device 3 and the entire document 50 is deleted from the transmission destination storage unit 122 (# 812). Note that when the number of terminal devices being browsed in a certain browsing status data 63 becomes “0”, the display of the entire document 50 related to the browsing status data 63 may be instructed to end.

  The image forming apparatus 1 appropriately executes the processes of steps # 801 to # 812 while providing the distribution service for the entire document 50.

  The air tag server 2 executes the process shown in FIG. 18 in response to a command from another device.

  When a command for registering a new confidential content 52 (air tag) is transmitted from the image forming apparatus 1 together with the image data 62 of the confidential content 52, the calling marker 57, and the number of copies Sa (Yes in # 821). The air tag server 2 registers the image data 62 in association with the calling marker 57 in the air tag storage unit 202 (see FIG. 12), and browsing status data indicating the calling marker 57, the number of copies Sa, and the like. 63 is newly generated and registered in the browsing status storage unit 206 (see FIG. 13) (# 822).

  Alternatively, when data of confidential content 52, that is, image data 62 is requested from the terminal device 3 (Yes in # 823), the air tag server 2 determines whether the user of the terminal device 3 is the owner of the printed matter 5 or not. Is discriminated (# 824).

  If it is determined that it is the owner of the printed matter 5 (Yes in # 825), the air tag server 2 reads the image data 62 from the air tag storage unit 202 and transmits it to the terminal device 3 (# 826). On the other hand, if it is determined that it is not the owner (No in # 825), the air tag server 2 transmits the image data 60 of the entire document 50 including the confidential content 52 to the terminal device 3 so as to transmit the image data 60. 1 is requested (# 827). The image data 60 and the image data 62 can be identified based on the calling marker 57 transmitted from the terminal device 3 together with the request.

  Alternatively, when a display end notification is transmitted from the terminal device 3 (Yes in # 828), the air tag server 2 deletes the IP address of the terminal device 3 and “1” from the number of browsing terminal devices. Is subtracted (# 829).

  The image forming apparatus 1 appropriately executes the processes of steps # 821 to # 829 while providing the air tag distribution service.

  The terminal device 3 executes the process shown in FIG. 19 in response to a user operation or a command from another transmission.

  The terminal device 3 starts photographing when the user performs a predetermined operation (# 841). When the call marker 57 is detected from the captured image (Yes in # 842), the image data 62 of the confidential content 52 (air tag) corresponding to the call marker 57 is requested to the air tag server 2 (# 843). ).

  As a result, when the image data 62 is received (Yes in # 844), the terminal device 3 displays the confidential content 52 based on the image data 62 (# 845).

  Thereafter, the terminal device 3 continues shooting. When the calling marker 57 is not detected from the photographed image for a predetermined time (No in # 846), the display of the confidential content 52 is terminated (# 847). Or you may complete | finish when a user performs predetermined | prescribed operation. Then, the air tag server 2 is notified that the display has been completed (# 848).

  On the other hand, when the image data 60 is received as a result of the request in step # 843 (No in # 844), the terminal device 3 displays the entire document 50 based on the image data 60 (# 849).

  Thereafter, when a display end command is transmitted from the image forming apparatus 1 (Yes in # 850), the terminal device 3 ends the display of the entire document 50 (# 851).

  Alternatively, when the user instructs to end the display of the entire document 50 (Yes in # 852), the terminal device 3 instructs the image forming apparatus 1 to that effect (# 853).

  According to the present embodiment, even when there is a user who does not have the printed matter 5, the content corresponding to the AR marker is provided to each user more suitably than before without sharing the printed matter 5 among a plurality of users. Can do.

  FIG. 20 is a diagram illustrating an example of the distributor data 64. FIG. 21 is a diagram illustrating an example of the distribution reference data 65.

  In the present embodiment, the user attribute determination unit 203 (see FIG. 9) of the air tag server 2 determines whether or not the user who has requested the air tag (confidential content 52) has the printed matter 5 for calling the air tag. This was determined by comparing the number of terminal devices being browsed in the browsing status data 63 (see FIG. 13) corresponding to the marker 57 with the number of copies. However, it may be determined by the following method.

  Distributor data 64 is stored in the browsing status storage unit 206 instead of the browsing status data 63. In the distributor data 64, as shown in FIG. 20, the IP address of the terminal device 3 of the user who distributed the printed matter 5 is shown.

  When an air tag is requested from a terminal device 3 of a certain user, the user attribute determination unit 203 displays the printed matter 5 if the IP address of the terminal device 3 is indicated in the distributor data 64 of the air tag. It is determined that it has. If not shown, it is determined that the printed matter 5 is not provided.

  In the present embodiment, the entire document transmission command unit 207 transmits the image data 60 to the terminal device 3 of the user when the user attribute determination unit 203 determines that the user does not have the printed material 5. 1 may be commanded according to the number of times the terminal device 3 has detected the call marker 57 or the time it has been detected.

  Specifically, as shown in FIG. 21, distribution reference data 65 is prepared in the browsing status storage unit 206 instead of the browsing status data 63 in the air tag server 2. In the distribution reference data 65, the reference access count and the reference detection time, and the calling marker 57 of the confidential content 52 are shown.

  When the terminal device 3 first detects the calling marker 57 during shooting, the terminal device 3 requests the air tag server 2 for an air tag (confidential content 52) corresponding to the calling marker 57. Thereafter, while the detection of the calling marker 57 continues, the terminal device 3 continues to notify the air tag server 2 to that effect.

  The entire document transmission command unit 207 displays image data to the terminal device 3 of the user when the reference detection time indicated by the distribution reference data 65 corresponding to the call marker 57 and the call marker 57 continue to be notified. The image forming apparatus 1 is instructed to transmit 60. That is, when the user continues to look at the printed matter 5, the command is issued. As a result, the image forming apparatus 1 transmits the image data 60 corresponding to the calling marker 57 to the terminal device 3. When the terminal device 3 receives the image data 60, the terminal device 3 stops displaying the confidential content 52 and displays the entire document 50. Note that the fact that the user is determined not to be the owner of the printed matter 5 by the user attribute determination unit 203 may be added as a condition for a command to the image forming apparatus 1.

  Alternatively, after making a request to the air tag server 2, the terminal device 3 continues to detect the calling marker 57 and does not make a request or notification. However, once the call marker 57 cannot be detected and then the call marker 57 is detected again, a request is made. That is, access to the confidential content 52 is attempted again.

  When the number of requests based on the call marker 57 reaches the reference access count indicated by the distribution reference data 65 corresponding to the call marker 57, the entire document transmission command unit 207 sends the request to the terminal device 3 of the user. The image forming apparatus 1 is instructed to transmit the image data 60. As a result, the image data 60 corresponding to the calling marker 57 is transmitted to the terminal device 3 and the entire document 50 is displayed. Note that the fact that the user is determined not to be the owner of the printed matter 5 by the user attribute determination unit 203 may be added as a condition for a command to the image forming apparatus 1.

  In the present embodiment, the image forming apparatus 1 unconditionally transmits the image data 60 to the terminal device 3 of the user who is determined not to own the printed matter 5, but a restriction may be provided. The image data 60 may be transmitted only when the user or administrator who registered the image data 60 turns on the setting. Alternatively, it may be transmitted only when the IP address of the terminal device 3 is registered in a predetermined database.

  When the terminal device 3 that has transmitted the calling marker 57 to the air tag server 2 exceeds the number of copies Sa, the air tag server 2 determines whether or not the data of the entire document 50, that is, the image data 60 is necessary for all of these terminal devices 3. You may inquire. Then, it determines that the user of the terminal device 3 that has transmitted a reply indicating that it is necessary is a user who does not have the printed matter 5, and instructs the image forming apparatus 1 to transmit the image data 60 to the terminal device 3. do it. Note that the confidential content 52 data, that is, the image data 62 may be transmitted to each terminal device 3 until the number of copies Sa is exceeded.

  For a user who does not own the printed material 5, the image forming apparatus 1 may further generate the printed material 5 by the number of terminal devices 3 to which the image data 60 is transmitted. The generation may be performed at the end of the conference, or may be performed at the end of the display of the entire document 50. Alternatively, it may be performed every time the image data 60 is transmitted. Alternatively, it may be performed when a print command from the terminal device 3 is in a certain time zone (for example, a time zone until a predetermined time elapses after the provision of the image data 60). The predetermined time is, for example, 10 minutes. The administrator may arbitrarily set it.

  Then, the number of copies of the browsing status data 63 corresponding to the printed matter 5 stored in the air tag storage unit 202 is increased by the number of newly generated copies of the printed matter 5.

  When a plurality of confidential contents 52 are included in one whole document 50, a unique calling marker 57 may be issued for each of the confidential contents 52. In this case, the air tag server 2 transmits only the image data 62 of the confidential content 52 corresponding to the detected call marker 57 to the terminal device 3 of the user who is determined to be the owner of the printed matter 5. On the other hand, the image forming apparatus 1 transmits the image data 60 corresponding to the detected call marker 57 to the terminal device 3 of the user who is determined not to be the owner of the printed matter 5.

  In the present embodiment, when the number of browsing terminal devices is “0” or when a command is issued from the terminal device 3, the display of the entire document 50 is terminated on the terminal device 3 of the user who is not the owner of the printed matter 5. . However, a display time limit may be determined for each entire document 50, and the display may be terminated when the time limit has passed.

  The function of the air tag server 2 may be provided in the image forming apparatus 1. The image forming apparatus 1 may transmit not only the image data 60 but also the image data 62.

  In addition, the configuration of the whole or each part of the document browsing system 100, the image forming apparatus 1, the air tag server 2, and the terminal device 3, the processing content, the processing order, the configuration of the database, and the like can be appropriately changed in accordance with the spirit of the present invention. it can.

1 Image forming device (AR support system)
2 Air tag server (AR support system)
107 Print generation unit (issuing means)
121 Document data transmission unit (first transmission means)
124 Display end command section (command means)
203 User attribute discrimination unit (discrimination means)
204 Air tag data transmitter (second transmitter)
206 Browsing status storage unit (storage means)
3 Terminal device 5 Printed matter (paper)
51 Non-confidential content (first content)
52 Confidential content (second content)
57 Calling marker

Claims (15)

AR support for providing the second content to a plurality of terminal devices that realize AR (Augmented Reality) by displaying the second content so that the marker and the first content are overlapped on the sheet on which the marker is written. A system,
A determining unit that determines whether or not a user of a terminal device that is a transmission source of the marker is the owner of the paper when the marker is transmitted from any one of the plurality of terminal devices. When,
When it is determined that the user is not the owner, both the first content and the second content are sent to an allocation terminal device that is a terminal device used by the user among the plurality of terminal devices. A first transmission means for transmitting;
Second transmitting means for transmitting only the second content of the first content and the second content to the user terminal device when the user is determined to be the owner. When,
An AR support system comprising: Storage means for storing the number of copies issued the paper,
The determination means determines that the user is the owner until the number of terminal devices that have transmitted the marker among the plurality of terminal devices reaches the number of copies, and the number reaches the number of copies. After determining that the user is not the owner,
The AR support system according to claim 1. The determining means determines that the user is the owner if the user's assigned terminal device is registered in a predetermined database in advance, and determines that the user is not the owner if not registered.
The AR support system according to claim 1. The first transmission means transmits the first content and the second content when the marker is continuously detected for a predetermined time in the allocation terminal device.
The AR support system according to claim 1. The first transmission means intermittently transmits the first content and the second content when the marker is detected a predetermined number of times in the allocation terminal device;
The AR support system according to claim 1. The first transmission means makes an inquiry to the terminal device of the user determined not to be the owner of the plurality of terminal devices, and when there is a predetermined answer to the inquiry, Transmitting the first content and the second content;
The AR support system according to any one of claims 1 to 5. When there is a predetermined command from any of the plurality of terminal devices, the first content and the second content are terminated so as to end displaying the first content and the second content. Command means for instructing the terminal device transmitted by the first transmission means,
The AR support system according to any one of claims 1 to 6. The first content and the second content are transmitted by the first transmission means so that the display of the first content and the second content is terminated when a predetermined time has elapsed. Command means for instructing the terminal device
The AR support system according to any one of claims 1 to 6. When all of the terminal devices to which the second transmission unit of the plurality of terminal devices has transmitted the second content have finished displaying the second content, the first content and the Command means for instructing the terminal device to which the first content and the second content are transmitted by the first transmission means so as to finish displaying the second content;
The AR support system according to any one of claims 1 to 6. Printing the first content and the second content on paper for the number of terminal devices to which the first content and the second content of the plurality of terminal devices are transmitted by the first transmission means. Issuing means for issuing the paper by
The AR support system according to any one of claims 1 to 9. The issuing means issues the paper after being instructed by the instruction means to finish displaying the first content and the second content;
The AR support system according to claim 10. Storage means for storing the number of copies issued the paper,
When the number of terminal devices that have transmitted the marker out of the plurality of terminal devices exceeds the number of copies, the determination unit makes an inquiry to each of the terminal devices that have transmitted, and in response to the inquiry Determining that the user of the terminal device that has made a predetermined answer is not the owner;
The AR support system according to claim 1. Providing content that provides the second content to a plurality of terminal devices that realize AR (Augmented Reality) by displaying the second content so that the marker and the first content overlap the paper on which the first content is written A method,
When the marker has been transmitted from any of the plurality of terminal devices, it is determined whether the user of the terminal device that is the transmission source of the marker is the owner of the paper,
When it is determined that the user is not the owner, both the first content and the second content are sent to an allocation terminal device that is a terminal device used by the user among the plurality of terminal devices. Send
When it is determined that the user is the owner, only the second content of the first content and the second content is transmitted to the allocation terminal device of the user.
A content providing method characterized by comprising: A computer that provides the second content to a plurality of terminal devices that realize AR (Augmented Reality) by displaying the second content so that the marker and the first content are overlaid on the sheet on which the marker is written. A computer program used,
In the computer,
When the marker is transmitted from any one of the plurality of terminal devices, a first of determining whether or not the user of the terminal device that is the transmission source of the marker is the owner of the paper Execute the process,
When it is determined that the user is not the owner, both the first content and the second content are sent to an allocation terminal device that is a terminal device used by the user among the plurality of terminal devices. Execute the second process to send,
When it is determined that the user is the owner, a third process of transmitting only the second content of the first content and the second content to the allocation terminal device of the user To execute,
A computer program characterized by the above. The second process is a process of requesting another transmission to transmit the first content and the second content to the allocation terminal device.
The computer program according to claim 14.

Images