JP6291989B2 - Content display device and control program for content display device - Google Patents

Description

  The present invention relates to a content display device and a control program for the content display device, and more specifically, a content display device that displays AR content associated with an AR (Augmented Reality) code, and a control program for the content display device About.

  In recent years, interest in security has increased, and leakage of confidential information such as confidential information and personal information held by companies has become a problem. In order to deal with these problems, companies have taken measures such as providing access authority to confidential information that has been digitized and restricting bringing in and taking out recording media. In the case of computerized confidential information, it is possible to guard to some extent by implementing measures using the IT (Information Technology) mechanism as described above.

  On the other hand, in conferences and meetings held at companies and other organizations, printed materials in which confidential information is printed on paper are often distributed to participants as materials. In this case, it is difficult to manage the taking out and copying of the printed material on which the confidential information is printed, and it is not possible to guard against these actions by the participants using the IT mechanism as described above. This fact makes it difficult to maintain security.

  As a technique for preventing leakage of confidential information from printed matter, when a user who is the owner of the printed matter browses a printed matter with an AR code displayed by wearing a glasses-type wearable display or the like, the wearable display is A technique for acquiring an AR code image, accessing an AR server linked to the AR code, and receiving AR content of confidential information from the AR server has been proposed. In this technique, when the wearable display receives the AR content, the AR content is displayed on the display of the lens portion of the glasses so that the AR content is superimposed on the real image of the printed material viewed by the user.

  Conventional techniques relating to the AR code are disclosed, for example, in Patent Documents 1 and 2 below.

  Patent Document 1 below discloses an image processing apparatus that avoids a situation in which, when an AR application is provided to a user, environment recognition becomes unstable and the use of the AR application is hindered. The image processing apparatus includes an image acquisition unit that acquires an input image generated by imaging the real space using the imaging device, and the real space based on the position of one or more feature points that appear in the input image. A recognition unit that recognizes a relative position and orientation with the imaging device, an application unit that provides an augmented reality application using the recognized relative position and orientation, and a recognition process executed by the recognition unit. A display control unit that superimposes a guidance object that guides a user who operates the imaging apparatus on the input image according to the distribution of the feature points so as to be stabilized.

  Patent Document 2 below discloses a method that can provide an augmented reality service even when a marker is not accurately recognized. This method considers the process of constructing an augmented reality service screen using peripheral information, the process of checking at least one marker that can be recognized in consideration of the augmented reality service screen, and at least one marker that can be recognized. A guide screen and a process of additionally displaying a guide screen on a part of the augmented reality service screen.

JP 2013-225245 A JP 2013-109768 A

  A user wearing a wearable display may add (add) an image such as a memo by hand to an actual document (paper document or electronic document) while the AR content is displayed on the display of the wearable display. . In this case, if the AR code displayed in the actual document is overwritten, the wearable display camera cannot recognize the image of the AR code, and the wearable display cannot acquire the AR content. As a result, user convenience has been reduced.

  The problem that content cannot be acquired when an image is added to a code is not a problem that occurs only in a wearable display, but a problem that can occur in all devices that display content associated with a code.

  SUMMARY An advantage of some aspects of the invention is to provide a content display device and a content display device control program capable of improving convenience.

  A content display device according to one aspect of the present invention includes a document image acquisition unit that acquires an image of a document on which a first code is displayed, and a first content that acquires content associated with the first code from a server. An acquisition means, a first content display means for displaying the content acquired by the first content acquisition means at a position determined based on the position of the first code, and a display by the first content display means If the detection means detects the addition of an image to the first code, and the detection means detects the addition of the image, the first code after the addition of the image is used as the second code. Registration means for registering with the server in association with the second content acquisition means for acquiring content associated with the second code from the server, and second container. Content acquired by Tsu acquisition means, and a second content display means for displaying the determined position based on the position of the second code.

  Preferably, in the content display device, the first content display means superimposes the content acquired by the first content acquisition means on a real image of the document on which the first code viewed by the document viewer is displayed. As described above, the content acquired by the first content acquisition unit is displayed, and the second content display unit displays the content acquired by the second content acquisition unit by the second viewer of the document. The content acquired by the second content acquisition means is displayed so that the code is superimposed on the real image of the displayed document.

  Preferably, in the content display device, the document image acquisition unit includes at least one of an electronic document image displayed on the display and a printed document image as the document image on which the first code is displayed. To get.

  Preferably, in the content display device, the document image acquisition unit acquires an image of the electronic document displayed on the display as the document image on which the first code is displayed.

  Preferably, in the content display device, the document image acquisition unit acquires an image of the printed document as the image of the document on which the first code is displayed.

  Preferably, in the content display device, the detection unit detects addition of an image to the first code based on an image of the first code captured by the camera.

  Preferably, in the content display device, the detection unit detects addition of an image to the first code by touching a display that is a touch panel.

  Preferably, in the content display device, the document image acquisition unit acquires an image of the electronic document displayed on the display as the image of the document on which the first code is displayed, and the first content display unit includes the first content display unit. The content acquired by the first content acquisition unit is displayed on the display so that the content acquired by the content acquisition unit is superimposed on the image of the document on which the first code is displayed. The display unit displays the content acquired by the second content acquisition unit on the display so that the content acquired by the second content acquisition unit is superimposed on the image of the document on which the second code is displayed. To do.

  A content display device according to another aspect of the present invention is an image of a document on which a first code is displayed, and includes a document image acquisition unit that acquires an image of an electronic document displayed on a display, and a first code First content acquisition means for acquiring associated content from the server, and first content for displaying the content acquired by the first content acquisition means at a position determined based on the position of the first code When displayed by the display means and the first content display means, the detection means for detecting the addition of the image to the first code, and when the detection means detects the addition of the image, the first code is displayed. A corrected image display unit that displays a corrected image obtained by correcting the image of the document that has been corrected to include a second code different from the first code, and a detection unit that displays the corrected image. When the addition of the second code is detected, the second code, the shift direction and the shift amount of the second code with respect to the first code, the registration means for registering in the server in association with the content, and the second code A second content acquisition unit that acquires content from the server, and a second content display unit that displays the content acquired by the second content acquisition unit at a position determined based on the shift direction and shift amount of the second code. Content display means.

  In the content display device, the corrected image is preferably an image corrected so as not to include the first code.

  Preferably, the content display device further includes storage means for individually storing the first code and the image added to the first code when the addition of the image is detected by the detection means.

  Preferably, in the content display device, the document image acquisition unit repeatedly acquires an image of the document on which the first code is displayed until the detection unit detects addition of an image, and the first content acquisition unit includes: When the document image acquisition unit newly acquires an image of the document on which the first code is displayed, the content associated with the first code is acquired again from the server, or the first content acquisition unit While the first code is included in the image acquired by the document image display means, the content associated with the first code is not acquired again from the server.

  A control program for a content display device according to still another aspect of the present invention acquires a document image acquisition step for acquiring an image of a document on which a first code is displayed, and acquires content associated with the first code from a server. A first content acquisition step; a first content display step for displaying the content acquired in the first content acquisition step at a position determined based on the position of the first code; and a first content display A detection step for detecting the addition of an image to the first code when displayed in steps, and a detection of the addition of an image in the detection step, the first code after the image is added is a second code A registration step of registering in the server in association with the content as the code, and the content associated with the second code in the server And a second content display step for displaying the content acquired in the second content acquisition step at a position determined based on the position of the second code on the computer. Let it run.

  A control program for a content display device according to still another aspect of the present invention is a document image acquisition step of acquiring an image of an electronic document displayed on a display, the document image displaying a first code, A first content acquisition step of acquiring content associated with one code from the server, and the content acquired in the first content acquisition step are displayed at a position determined based on the position of the first code. When displaying in the first content display step and the first content display step, the detection step for detecting the addition of the image to the first code, and the detection of the addition of the image in the detection step, the first A corrected image obtained by correcting the image of the document on which the code is displayed to include a second code different from the first code is displayed. When the addition of an image is detected in the corrected image display step displayed on the ray and the detection step, the server associates the second code and the shift direction and shift amount of the second code with respect to the first code with the content. A registration step of registering in the second code, a second content acquisition step of acquiring the content associated with the second code from the server, and the content acquired in the second content acquisition step of the second code moving direction and Causing the computer to execute a second content display step of displaying at a position determined based on the amount of movement.

  ADVANTAGE OF THE INVENTION According to this invention, the control program of the content display apparatus and content display apparatus which can improve the convenience can be provided.

It is a figure which shows the notional structure of the AR content display system in the 1st Embodiment of this invention. It is a block diagram which shows the structure of the AR content display system in the 1st Embodiment of this invention. It is a figure which shows notionally the basic operation | movement of the AR content display system in the 1st Embodiment of this invention. It is a sequence diagram which shows the basic operation | movement of AR content display system in the 1st Embodiment of this invention. In the 1st Embodiment of this invention, it is a figure which shows notionally the first half part of operation | movement of AR content display system when an image is added with respect to AR code. In the 1st Embodiment of this invention, it is a figure which shows typically the AR code information table which AR server memorize | stores. In the 1st Embodiment of this invention, it is a figure which shows notionally the latter half part of operation | movement of AR content display system when an image is added with respect to AR code. FIG. 6 is a sequence diagram showing an operation of the AR content display system when an image is added to the AR code in the first embodiment of the present invention. It is a figure explaining the effect of the 1st Embodiment of this invention. It is a figure which shows notionally the operation | movement of AR content display system in the modification of the 1st Embodiment of this invention. In the 2nd Embodiment of this invention, it is a figure which shows notionally the operation | movement of AR content display system when an image is added with respect to AR code. In the 2nd Embodiment of this invention, it is a sequence diagram which shows operation | movement of AR content display system when an image is added with respect to AR code.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  In the following embodiments, a case will be described in which a content display device includes a wearable display and a portable terminal unless otherwise specified. The content display device may be composed of a wearable display alone or a mobile terminal alone. The content display device may be a portable terminal other than a wearable display such as a mobile phone or a smartphone. Further, the content display device may be a digital camera, a PC (Personal Computer), or the like, in addition to the mobile terminal.

  [First Embodiment]

    (Configuration of AR content display system)

  First, the configuration of the AR content display system will be described.

  FIG. 1 is a diagram showing a conceptual configuration of an AR content display system according to the first embodiment of the present invention.

  Referring to FIG. 1, the AR content display system includes a wearable display 100, an AR server 200 (an example of a server), a portable terminal 300, and a network 400. The wearable display 100 and the mobile terminal 300 are wirelessly connected to the network 400 through an access point AP connected to the network 400, for example, and the AR server 200 is wired to the network 400, for example. Thereby, the wearable display 100, the AR server 200, and the mobile terminal 300 are connected to each other through the network 400.

  The network 400 uses a dedicated line such as a wired or wireless LAN (Local Area Network). The network 400 may be the Internet. The network 400 connects various devices using a TCP / IP (Transmission Control Protocol / Internet Protocol) protocol. Devices connected to the network 400 can exchange various data with each other.

  Wearable display 100 is a glasses-type display worn on the head. A camera 170 for photographing the front (acquiring a front image) is provided at the front of the wearable display 100. In addition, a transmissive display 160 is provided in the lens portion of the glasses in the wearable display 100.

  A touch panel 360 is provided on one surface (surface visible in FIG. 1) of the mobile terminal 300, and a camera 370 is provided on the other surface (surface not visible in FIG. 1). The touch panel 360 includes a touch pad and a display (monitor).

  Note that the method of connecting the wearable display 100, the AR server 200, and the mobile terminal 300 to the network 400 may be wired connection or wireless connection.

  FIG. 2 is a block diagram showing the configuration of the AR content display system according to the first embodiment of the present invention.

  Referring to FIG. 2, wearable display 100 communicates with a central processing unit (CPU) 110 that controls wearable display 100 as a whole and an external device such as AR server 200 via network 400 in accordance with an instruction from CPU 110. A network I / F 120 to be performed, a ROM (Read Only Memory) 130 that stores a control program executed by the CPU 110, a RAM (Random Access Memory) 140 that is a working memory of the CPU 110, and a storage unit 150 that stores various types of information A transmissive display 160 that displays various types of information, a camera 170 that captures an image, and an operation unit 180 that receives various operations on the wearable display 100.CPU 110 is connected to each of network I / F 120, ROM 130, RAM 140, storage unit 150, display 160, camera 170, and operation unit 180. The storage unit 150 stores, for example, information related to the user of the wearable display 100 and information for receiving authentication from the AR server 200 (for example, a user ID (Identification) and a password).

  The AR server 200 is composed of, for example, a PC. The AR server 200 includes a CPU 210 that controls the entire AR server 200, a network I / F 220 that communicates with an external device such as the wearable display 100 via the network 400 in accordance with an instruction from the CPU 210, and a control program that the CPU 210 executes. ROM 230 for storing, RAM 240 as a working memory of CPU 210, and a storage unit 250 for storing various information. The CPU 210 is connected to each of the network I / F 220, the ROM 230, the RAM 240, and the storage unit 250. The storage unit 250 stores an AR code information table and an AR content (an example of content). The AR code information table is a table that associates the AR code with the name of the AR content corresponding to the AR code.

  The portable terminal 300 includes a CPU 310 that controls the entire portable terminal 300, a network I / F 320 that communicates with an external device such as the wearable display 100 via the network 400 in accordance with an instruction from the CPU 310, and a control program that the CPU 310 executes. A RAM 330 that is a working memory of the CPU 110, a storage unit 350 that stores various information, a touch panel 360 that displays various information and detects a touch, and a camera 370 that performs imaging. It is out. The CPU 310 is connected to each of the network I / F 320, the ROM 330, the RAM 340, the storage unit 350, the touch panel 360, and the camera 370. The storage unit 350 stores, for example, information related to the user of the mobile terminal 300 and information for receiving authentication from the AR server 200 (for example, a user ID and a password).

    (Basic operation of AR content display system)

  Next, the operation of the AR content display system in the present embodiment will be described.

  FIG. 3 is a diagram conceptually showing the basic operation of the AR content display system in the first embodiment of the present invention.

  Referring to FIG. 3A, an AR code C1 (an example of a first code such as a QR code (registered trademark)) is displayed on a paper document DC or an electronic document DI. The paper document DC is a paper document on which characters and the like are printed. The electronic document DI is a document electronically displayed on the touch panel 360 of the mobile terminal 300. The AR code C1 may be an arbitrary image. For example, information necessary for acquiring AR content (AR server URL (Uniform Resource Locator) information, AR code ID information, etc.) is encoded. It may be.

  The original document that is the source of the paper document DC or the electronic document DI includes confidential information (confidential information). This original document is a paper document DC printed with the confidential information hidden by the AR content area RG, and is displayed on the touch panel 360 of the portable terminal 300 with the confidential information hidden by the AR content area RG. The result is an electronic document DI. The AR content area RG is provided at a position where confidential information is to be displayed in the paper document DC. The AR code C1 is displayed at the center of the AR content area RG. In other words, the paper document DC or the electronic document DI is obtained by replacing the confidential information of the original document with the AR code C1.

  It is assumed that the user of the wearable display 100 browses the paper document DC or the electronic document DI with the wearable display 100 attached. In this case, wearable display 100 acquires an image of paper document DC or electronic document DI with camera 170 and recognizes AR code C1 included in the acquired image. Next, the wearable display 100 accesses a predetermined AR server 200 and transmits the user ID and password of the user of the wearable display 100 and the AR code C1 to the AR server 200. Upon receiving the user ID and password, the AR server 200 authenticates the wearable display 100. This authentication may be omitted. After authenticating the wearable display 100, the AR server 200 transmits the AR content associated with the received AR code C1 from the AR code information table to the wearable display 100.

  The wearable display 100 may acquire the AR content associated with the AR code C1 from the AR server. The wearable display 100, for example, accesses the AR server 200 specified based on the information included in the recognized AR code C1, and transmits the ID included in the AR code C1 to the AR server 200. The AR content associated with the ID may be acquired from the AR server.

  Referring to FIG. 3B, when receiving (acquiring) AR content IM, wearable display 100 displays received AR content IM on display 160. The AR content IM is confidential information, and is shown here as a graph. Wearable display 100 displays AR content IM at a position determined based on the position of AR code C1. Specifically, the wearable display 100 is arranged such that the position of the AR code C1 in the real image of the paper document DC or the electronic document DI viewed by the user of the wearable display 100 through the wearable display 100 and the center of the AR content IM overlap ( The AR content IM is superimposed and displayed (projected) so that the AR content region RG and the AR content IM match. As a result, the user of the wearable display 100 can view the AR content IM together with the real image of the paper document DC or the electronic document DI as shown in FIG.

  The AR content IM is preferably displayed in a semi-transparent manner in order to facilitate the addition of an image by the user of the wearable display 100 as will be described later. When displaying in a semi-transparent state, the AR code C1 may be hidden by the AR content IM by making the portion overlapping the AR code C1 in the AR content IM non-transparent.

  Polling between the wearable display 100 and the AR server 200 may be performed at predetermined intervals (for example, 0.1 s). In this case, the wearable display 100 repeatedly acquires the image of the paper document DC or the electronic document DI until the addition of the image ADI is detected, and newly acquires the image of the paper document DC or the electronic document DI on which the AR code C1 is displayed. When acquired, the AR content IM associated with the AR code C1 may be acquired again from the AR server 200.

  Further, polling between the wearable display 100 and the AR server 200 may not be performed while the AR code C1 is included in the image acquired by the wearable display 100 after being performed once. In this case, the wearable display 100 may not reacquire the AR content IM associated with the AR code C1 from the AR server 200 while the image acquired by the wearable display 100 includes the AR code C1. .

  FIG. 4 is a sequence diagram showing a basic operation of the AR content display system according to the first embodiment of the present invention.

  Referring to FIG. 4, when the wearable display 100 detects (acquires) the AR code, the wearable display 100 requests the AR server 200 to transmit the AR content corresponding to the AR code. Upon receiving the request, the AR server 200 responds to the wearable display 100 with AR content corresponding to the AR code. When the wearable display 100 receives the AR content, the wearable display 100 superimposes and displays the AR content on the real image of the paper document or electronic document being browsed by the user of the wearable display 100.

    (Operation of AR content display system when image is added to AR code)

  Next, the operation of the AR content display system when an image is added to the AR code will be described.

  FIG. 5 is a diagram conceptually showing the first half of the operation of the AR content display system when an image is added to the AR code in the first embodiment of the present invention.

  Referring to FIG. 5A, the user of wearable display 100 displays an image ADI such as a note on paper document DC or electronic document DI in a state where AR content IM is displayed on display 160 of wearable display 100. Add by hand.

  Specifically, when an image ADI is added to the paper document DC, the image ADI is an image actually drawn on the paper document DC using a writing tool such as a pen. When the image ADI is added to the electronic document DI, the mobile terminal 300 detects the movement of the touch pen or the finger that has touched the touch panel 360 with the touch panel 360, and the detected movement locus is displayed on the touch panel 360. It is an image. Any image ADI may be added, for example, a line, a character, or a symbol.

  FIG. 5B is an enlarged view of the AR code C1 in the paper document DC or the electronic document DI in FIG. Referring to FIG. 5B, when the image ADI is added onto the AR code C1, the image of the AR code C1 acquired by the wearable display 100 with the camera 170 is the image of the original AR code C1 (image ADI). The image is different from the image of the AR code C1 before the is added. As a result, the wearable display 100 cannot acquire the content associated with the AR code C1 from the AR server 200.

  Therefore, wearable display 100 detects whether or not image ADI is added to AR code C1 based on the image of AR code C1 captured by camera 170 when AR content IM is displayed. Wearable display 100 detects whether or not an image ADI has been added by detecting whether or not a pen tip or a finger has entered the area of AR code C1, for example. When the addition of the image ADI to the AR code C1 is detected, the wearable display 100 acquires the image of the AR code C1 to which the image ADI is added, transmits the acquired image to the AR server 200, and FIG. 2, the AR server 200 is requested to register this image as a new AR code C2 (the wearable display 100 itself may register a new AR code C2 in the AR server 200).

  When the image ADI is added to the electronic document DI displayed on the touch panel 360 of the portable terminal 300, the portable terminal is used instead of detecting whether the image ADI is added to the AR code C1 using the camera 170. You may detect by the presence or absence of the contact to 300 touch panels 360.

  When detecting the presence / absence of the addition of the image ADI to the AR code C1 based on the presence / absence of the touch on the touch panel 360 of the mobile terminal 300, the mobile terminal 300 transmits the AR code C2 to the AR server 200 instead of the wearable display 100. You may request registration.

  When detecting whether or not the image ADI is added to the AR code C1 based on whether or not the touch panel 360 of the mobile terminal 300 is touched, the mobile terminal 300 notifies the wearable display 100 of the detection and also detects the image ADI. May be transmitted to the wearable display 100. In this case, the wearable display 100 transmits an image obtained by combining the image ADI and the AR code C1 to the AR server 200 as a new AR code C2, and requests the AR server 200 to register the AR code C2.

  Furthermore, when detecting whether or not the image ADI is added to the AR code C1 based on whether or not the touch panel 360 of the mobile terminal 300 is touched, the mobile terminal 300 simply notifies the wearable display 100 of the detection. May be. In this case, wearable display 100 acquires an image of AR code C1 to which image ADI is added by camera 170, transmits the acquired image to AR server 200, and registers the transmitted image as new AR code C2. Request to the AR server 200.

  FIG. 6 is a diagram schematically showing an AR code information table stored in the AR server according to the first embodiment of this invention.

  Referring to FIG. 6A, an AR code, an AR code ID, and a corresponding AR content name are associated with the AR code information table.

  For example, an ID “0010” is attached to the AR code C1, and an AR content named “0001.pdf” is associated with the AR code C1. When receiving the AR code C1 (or the ID of the AR code C1) from the wearable display 100, the AR server 200 identifies the AR content named “0001.pdf” by referring to the AR code information table, and this AR The content is transmitted to the wearable display 100.

  Referring to FIG. 6B, when receiving a request for registration of an image and a new AR code from wearable display 100 or portable terminal 300, AR server 200 uses the received image as a new AR code as an AR code. Register in the information table. At the time of registration, the AR server 200 associates the received image with the AR content associated with the original AR code (the AR content most recently transmitted to the wearable display 100) and adds a new ID. At this time, the AR server 200 may notify the wearable display 100 of a new ID.

  For example, when receiving a request for registration of a new AR code C2 from the wearable display 100, the AR server 200 associates an AR content named “0001.pdf” with the image of the AR code C2, and sets a new “0011”. The image of the AR code C2 is registered with a unique ID.

  FIG. 7 is a diagram conceptually showing the latter half of the operation of the AR content display system when an image is added to the AR code in the first embodiment of the present invention.

  Referring to FIG. 7A, after registering a new AR code C2, wearable display 100 acquires an image of paper document DC or electronic document DI with camera 170, and is included in the acquired image. The AR code C2 is recognized, and the AR code C2 is transmitted to the AR server 200. The AR server 200 extracts the AR content associated with the received AR code C2 from the AR code information table, and transmits the extracted AR content to the wearable display 100.

  With reference to FIG. 7B, wearable display 100 displays received AR content IM on display 160. The AR content IM is the same as the AR content associated with the AR code C1. Wearable display 100 displays AR content IM at a position determined based on the position of AR code C2. Specifically, the wearable display 100 is configured such that the position of the AR code C2 in the real image of the paper document DC or the electronic document DI viewed by the user of the wearable display 100 through the wearable display 100 overlaps the center of the AR content IM ( The AR content IM is superimposed and displayed so that the AR content region RG and the AR content IM match. As a result, as shown in FIG. 7C, the user of the wearable display 100 can continue to browse the AR content IM along with the real image of the paper document DC or the electronic document DI even after adding the image ADI.

  FIG. 8 is a sequence diagram showing the operation of the AR content display system when an image is added to the AR code in the first embodiment of the present invention. The “detection unit” shown in FIGS. 8 and 12 means a configuration for detecting the addition of an image to the AR code. Specifically, the camera 170 of the wearable display 100 or the touch panel 360 of the mobile terminal 300 is used. is there.

  Referring to FIG. 8, the operation of the AR content display system until the AR content corresponding to the AR code before the image is added is the same as that in FIG. 3, and therefore the description thereof is repeated here. Absent.

  After the wearable display 100 displays the AR content superimposed on the paper document or electronic document, when the detection unit detects overwriting (addition of an image) to the AR code, the detection unit detects the overwritten AR code (image addition). The AR server 200 is requested to register the new AR code). Upon receiving the registration request, the AR server 200 registers a new AR code in the AR code information table. The AR server 200 notifies the wearable display 100 of the completion of registration.

  When receiving the notification of registration completion, wearable display 100 requests AR server 200 to transmit AR content corresponding to the new AR code. Upon receiving the request, the AR server 200 responds to the wearable display 100 with AR content corresponding to the new AR code. When the wearable display 100 receives the AR content, the wearable display 100 superimposes and displays the AR content on the real image of the paper document or electronic document being browsed by the user of the wearable display 100.

  FIG. 9 is a diagram for explaining the effect of the first exemplary embodiment of the present invention.

  Referring to FIG. 9, according to the present embodiment, when image ADI is added on AR code C1, AR code C1 to which image ADI is added is associated with AR code C1 as AR code C2. It is registered in association with the AR content IM. Thereby, even after the image ADI is added to the AR code C1, the wearable display 100 can acquire and display the AR content IM. As a result, convenience can be improved.

  In the above-described operation of the AR content display system, when the electronic document DI is displayed on the mobile terminal 300 and the addition of the image ADI to the AR code C1 of the electronic document DI is detected by the wearable display 100, the mobile terminal 300 The mobile terminal 300 may be replaced by any display device in general, and may not have a function of performing communication with the wearable display 100. In this case, the wearable display 100 serves alone as a content display device. In addition, when the wearable display 100 detects the addition of the image ADI to the AR code C1 of the paper document DC, the portable terminal 300 is not necessary, and the wearable display 100 alone serves as a content display device.

    (Modification of the first embodiment)

  Subsequently, a modification of the present embodiment will be described. In the present modification, the mobile terminal 300 alone serves as a content display device.

  FIG. 10 is a diagram conceptually showing the operation of the AR content display system in the modification of the first embodiment of the present invention.

  Referring to FIG. 10A, a user of mobile terminal 300 takes a paper document DC on which characters and the like are printed with camera 370 of mobile terminal 300 and electronically displays the image DII (paper document DC). Assume that an image (an image of an electronic document) is displayed on the touch panel 360 of the mobile terminal 300. In this case, the mobile terminal 300 recognizes the AR code C1 included in the image acquired by the camera 370.

  Referring to FIG. 10B, mobile terminal 300 acquires AR content IM associated with AR code C1 from AR server 200, and displays AR content IM on touch panel 360 so as to be superimposed on image DII. To do.

  Referring to FIG. 10C, the user of the mobile terminal 300 adds an image ADI such as a memo to the paper document DC by handwriting while the AR content IM is displayed on the touch panel 360 of the mobile terminal 300. To do. When the mobile terminal 300 detects the addition of the image ADI to the AR code C1 with the camera 370, the AR server 200 registers the image of the AR code C1 with the image ADI added as a new AR code C2. Request. The AR server 200 registers the AR code C2.

  Referring to FIG. 10D, after the registration of the AR code C2 is completed, the mobile terminal 300 acquires the AR content IM associated with the AR code C2 from the AR server 200 and is superimposed on the image DII. The AR content IM is displayed on the touch panel 360.

  [Second Embodiment]

  In this embodiment, an example in which a wearable display displays a new AR code when an image such as a memo is added on the AR code of the electronic document will be described.

  FIG. 11 is a diagram conceptually showing the operation of the AR content display system when an image is added to the AR code in the second embodiment of the present invention.

  Referring to FIG. 11A, as in the case of the first embodiment, wearable display 100 is an image of electronic document D1 on which AR code C1 is displayed, and is displayed on touch panel 360. An image of the document is acquired, AR content IM associated with the AR code C1 is acquired from the AR server 200, and displayed at a position determined based on the position of the AR code C1. Next, the user of wearable display 100 adds handwritten image ADI such as memo to electronic document DI displayed on portable terminal 300 in a state where AR content IM is displayed on display 160 of wearable display 100. To do. The image ADI is an image in which the mobile terminal 300 detects the movement of the touch pen or the finger touching the touch panel 360 with the touch panel 360 and the detected movement locus is displayed on the touch panel 360.

  Referring to FIG. 11B, wearable display 100 detects whether image ADI is added to AR code C1 based on the image of AR code C1 captured by camera 170. Wearable display 100 detects whether or not an image ADI has been added by detecting whether or not a pen tip or a finger has entered the area of AR code C1, for example. When the addition of the image ADI to the AR code C1 is detected, the wearable display 100 displays on the touch panel 360 a corrected image AI obtained by correcting the image of the electronic document DI to include a new AR code C3 different from the AR code C1. Display.

  When the addition of the image ADI to the AR code C1 is detected, the wearable display 100 or the portable terminal 300 preferably stores the AR code C1 and the image ADI separately. The corrected image AI is preferably an image corrected so as not to include the AR code C1.

  The wearable display 100 transmits the new AR code C3 and the shift direction and shift amount ΔD of the new AR code C3 with respect to the original AR code C1 to the AR server 200, and registers the new AR code C3. Requests to the AR server 200. The shift direction and the shift amount ΔD are expressed as ΔD = (x, y) when the shift amounts in the x-axis direction and the y-axis direction in FIG. 11B are “x” and “y”, respectively. The

  When the image ADI is added to the electronic document DI displayed on the touch panel 360 of the portable terminal 300, the portable terminal is used instead of detecting whether the image ADI is added to the AR code C1 using the camera 170. You may detect by the presence or absence of the contact to 300 touch panels 360.

  When detecting whether or not the image ADI is added to the AR code C1 based on whether or not the touch panel 360 of the mobile terminal 300 is touched, the mobile terminal 300 displays the new AR code C3 on the touch panel 360 and the new AR code. The AR server 200 may be requested to register C3 and the shift direction and shift amount ΔD of the new AR code C3 with respect to the original AR code C1 to the AR server 200 and register the new AR code C3.

  When detecting whether or not the image ADI is added to the AR code C1 based on whether or not the touch panel 360 of the mobile terminal 300 is touched, the mobile terminal 300 displays a new AR code C3 on the touch panel 360 and adds a new The AR code C3 and the shift direction and shift amount ΔD of the new AR code C3 with respect to the original AR code C1 may be transmitted to the wearable display 100. In this case, wearable display 100 transmits new AR code C3 and the shift direction and shift amount ΔD of new AR code C3 with respect to original AR code C1 to AR server 200, and registers new AR code C3. Request to the AR server 200.

  Furthermore, when detecting whether or not the image ADI is added to the AR code C1 based on whether or not the touch panel 360 of the mobile terminal 300 is touched, the mobile terminal 300 simply notifies the wearable display 100 of the detection. May be. In this case, the wearable display 100 displays a new AR code C3 different from the AR code C1 on the touch panel 360 of the mobile terminal 300, and the new AR code C3 and the new AR code C3 with respect to the original AR code C1. The shift direction and the shift amount ΔD are transmitted to the wearable display 100, and the AR server 200 is requested to register a new AR code C3.

  Referring to FIG. 11C, when receiving a request for registration of an image and a new AR code from wearable display 100 or portable terminal 300, AR server 200 uses the received image as a new AR code C3. It is registered in the AR code information table together with the shift direction and the shift amount ΔD. At the time of registration, the AR server 200 associates an AR content (here, an AR content named “0001.pdf”) associated with the AR code C1, which is the original AR code, with the received image, and creates a new ID. Is attached. At this time, the AR server 200 may notify the wearable display 100 of a new ID.

  Referring to FIG. 11D, after registering a new AR code C3, wearable display 100 acquires an image of electronic document DI with camera 170, and AR code C3 included in the acquired image is acquired. Recognize and transmit the AR code C3 to the AR server 200. The AR server 200 extracts the AR content associated with the received AR code C3 from the AR code information table, and transmits the extracted AR content to the wearable display 100. The AR server 200 also transmits information on the shift direction and shift amount ΔD of the AR code C3 to the wearable display 100 together with the AR content.

  Wearable display 100 displays received AR content IM at a position determined based on the shift direction and shift amount ΔD on display 160. Specifically, the wearable display 100 determines the center of the AR content IM from the position of the AR code C3 in the real image of the paper document DC or the electronic document DI to be browsed and the reciprocal of the shift direction and the shift amount ΔD (that is, (−x, − The AR content IM is displayed at the position shifted by y)). As a result, as shown in FIG. 11D, the user of the wearable display 100 can continue browsing the AR content IM together with the real image of the electronic document DI even after adding the image ADI. By displaying the AR content at the position determined based on the shift direction and the shift amount ΔD in the center of the AR content IM, the position of the AR content IM displayed based on the AR code C1 and the AR code C3 The position of the displayed AR content IM is the same.

  FIG. 12 is a sequence diagram showing the operation of the AR content display system when an image is added to the AR code in the second embodiment of the present invention.

  Referring to FIG. 12, the operation of the AR content display system until the AR content corresponding to the AR code before the image is added is the same as that in FIG. 3, and therefore the description thereof is repeated here. Absent.

  After the wearable display 100 superimposes and displays the AR content on the electronic document, when the detection unit detects the pen tip intrusion into the AR code area, the detection unit sends a new AR code to the mobile terminal 300. , As well as the shift direction and shift amount, and requests display of a new AR code. When the mobile terminal 300 receives the request, the mobile terminal 300 displays a new AR code at a position shifted from the AR code position by the received shift direction and shift amount. The detection unit transmits a new AR code, a shift direction and a shift amount to the AR server 200, and requests registration of the new AR code. When the AR server 200 receives the registration request, the AR server 200 registers a new AR code, a shift direction, and a shift amount in the AR code information table. The AR server 200 notifies the wearable display 100 of the completion of registration.

  When receiving the notification of registration completion, wearable display 100 requests AR server 200 to transmit AR content corresponding to the new AR code. Upon receiving the request, the AR server 200 responds to the wearable display 100 with the AR content corresponding to the new AR code, the shift direction, and the shift amount. When the wearable display 100 receives the AR content, the wearable display 100 displays the AR content superimposed on the real image of the electronic document displayed by the mobile terminal 300. The content and position of the AR content displayed based on the new AR code are the same as the content and position of the AR content displayed based on the original AR code.

  Since the operations of the AR content display system and the configuration of the AR content display system other than those described above are the same as those in the first embodiment, description thereof will not be repeated.

  According to the present embodiment, when an image is added on the original AR code, a new AR code is displayed, and the new AR code is associated with the AR content associated with the original AR code. And register. Thereby, even after an image is added on the original AR code, the wearable display can acquire and display the AR content based on the new AR code. As a result, convenience can be improved.

  Also, by registering the shift direction and shift amount of the new AR code with respect to the original AR code together with the new AR code, the position of the AR content displayed based on the new AR code can be changed to the original AR code. It can be made the same as the position of the AR content displayed based on it.

  [Others]

  The wearable display may acquire a document image printed on a whiteboard or the like in addition to a document printed on a printed matter or an electronic document image displayed on a mobile terminal.

  The processing in the above-described embodiment may be performed by software or may be performed using a hardware circuit. It is also possible to provide a program for executing the processing in the above-described embodiment, and record the program on a recording medium such as a CD-ROM, a flexible disk, a hard disk, a ROM, a RAM, or a memory card and provide it to the user. You may decide to do it. The program is executed by a computer such as a CPU. The program may be downloaded to the apparatus via a communication line such as the Internet.

  The above-described embodiment is to be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

100 Wearable display 110, 210, 310 CPU (Central Processing Unit)
120, 220, 320 Network I / F
130, 230, 330 ROM (Read Only Memory)
140, 240, 340 RAM (Random Access Memory)
150, 250, 350 Storage unit 160 Display 170 Camera 180 Operation unit 200 AR (Augmented Reality) server 300 Mobile terminal 360 Touch panel 370 Camera 400 Network ADI Added image AI corrected image AP Access point C1, C2, C3 AR code DC paper Document DI Electronic document DII Paper document image or electronic document IM AR content RG AR content area ΔD Shift amount

Claims (14)

Document image acquisition means for acquiring an image of the document on which the first code is displayed;
First content acquisition means for acquiring content associated with the first code from a server;
First content display means for displaying the content acquired by the first content acquisition means at a position determined based on the position of the first code;
Detection means for detecting addition of an image to the first code when displayed by the first content display means;
A registration unit that registers the first code after the image is added as a second code in the server in association with the content when the detection unit detects the addition of the image;
Second content acquisition means for acquiring the content associated with the second code from the server;
A content display device comprising: a second content display unit configured to display the content acquired by the second content acquisition unit at a position determined based on the position of the second code. The first content display unit is configured such that the content acquired by the first content acquisition unit is superimposed on a real image of the document on which the first code viewed by a document viewer is displayed. Displaying the content acquired by the first content acquisition means;
The second content display unit is configured such that the content acquired by the second content acquisition unit is superimposed on a real image of the document on which the second code viewed by the viewer of the document is displayed. The content display device according to claim 1, wherein the content acquired by the second content acquisition unit is displayed.   The document image acquisition unit acquires at least one of an image of an electronic document displayed on a display and an image of a printed document as an image of a document on which the first code is displayed. 3. The content display device according to 1 or 2.   The content display apparatus according to claim 3, wherein the document image acquisition unit acquires an image of an electronic document displayed on the display as an image of the document on which the first code is displayed.   The content display apparatus according to claim 3, wherein the document image acquisition unit acquires an image of a printed document as an image of the document on which the first code is displayed.   The content display device according to claim 1, wherein the detection unit detects addition of an image to the first code based on an image of the first code captured by a camera.   The content display apparatus according to claim 4, wherein the detection unit detects addition of an image to the first code by touching the display that is a touch panel. The document image acquisition means acquires an image of an electronic document displayed on a display as an image of a document on which the first code is displayed,
The first content display unit includes the first content acquisition unit such that the content acquired by the first content acquisition unit is superimposed on an image of a document on which the first code is displayed. Display the content acquired at the display,
The second content display means includes the second content acquisition means so that the content acquired by the second content acquisition means is superimposed on an image of a document on which the second code is displayed. The content display device according to claim 1, wherein the content acquired in step (1) is displayed on the display. A document image acquisition means for acquiring an image of an electronic document displayed on a display, the document image displaying a first code;
First content acquisition means for acquiring content associated with the first code from a server;
First content display means for displaying the content acquired by the first content acquisition means at a position determined based on the position of the first code;
Detection means for detecting addition of an image to the first code when displayed by the first content display means;
When addition of an image is detected by the detection means, a corrected image obtained by correcting an image of a document displaying the first code so as to include a second code different from the first code is displayed on the display. Corrected image display means to be displayed on,
Registration that registers the second code and the shift direction and shift amount of the second code relative to the first code in the server in association with the content when the detection unit detects addition of an image Means,
Second content acquisition means for acquiring the content associated with the second code from the server;
A content display device comprising: a second content display unit configured to display the content acquired by the second content acquisition unit at a position determined based on a shift direction and a shift amount of the second code. .   The content display device according to claim 9, wherein the corrected image is an image corrected so as not to include the first code.   11 or 10, further comprising storage means for individually storing the first code and an image added to the first code when addition of an image is detected by the detection means. The content display device described in 1. The document image acquisition unit repeatedly acquires an image of the document on which the first code is displayed until the detection unit detects addition of an image,
The first content acquisition unit obtains a content associated with the first code from the server when the document image acquisition unit newly acquires an image of a document on which the first code is displayed. Reacquire or
The first content acquisition unit does not re-acquire the content associated with the first code from the server while the image acquired by the document image display unit includes the first code. The content display device according to claim 1. A document image acquisition step of acquiring an image of the document on which the first code is displayed;
A first content acquisition step of acquiring content associated with the first code from a server;
A first content display step of displaying the content acquired in the first content acquisition step at a position determined based on the position of the first code;
A detection step of detecting addition of an image to the first code when displayed in the first content display step;
A registration step of registering in the server in association with the content, the second code as the first code after the addition of the image when the addition of the image is detected in the detection step;
A second content acquisition step of acquiring from the server the content associated with the second code;
Control of a content display device that causes a computer to execute a second content display step of displaying the content acquired in the second content acquisition step at a position determined based on the position of the second code program. A document image obtaining step of obtaining an image of the electronic document displayed on the display, the document image displaying the first code;
A first content acquisition step of acquiring content associated with the first code from a server;
A first content display step of displaying the content acquired in the first content acquisition step at a position determined based on the position of the first code;
A detection step of detecting addition of an image to the first code when displayed in the first content display step;
When addition of an image is detected in the detection step, a corrected image obtained by correcting an image of a document on which the first code is displayed so as to include a second code different from the first code is displayed on the display. A corrected image display step to be displayed on
Registration that registers the second code and the shift direction and shift amount of the second code with respect to the first code in association with the content when the addition of an image is detected in the detection step. Steps,
A second content acquisition step of acquiring from the server the content associated with the second code;
A content for causing a computer to execute a second content display step for displaying the content acquired in the second content acquisition step at a position determined based on a moving direction and a moving amount of the second code. Display device control program.