JP2015194954A - Display controller and control program for display controller - Google Patents

Display controller and control program for display controller Download PDF

Info

Publication number
JP2015194954A
JP2015194954A JP2014073150A JP2014073150A JP2015194954A JP 2015194954 A JP2015194954 A JP 2015194954A JP 2014073150 A JP2014073150 A JP 2014073150A JP 2014073150 A JP2014073150 A JP 2014073150A JP 2015194954 A JP2015194954 A JP 2015194954A
Authority
JP
Japan
Prior art keywords
image
display
superimposed
display mode
document
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Granted
Application number
JP2014073150A
Other languages
Japanese (ja)
Other versions
JP6303723B2 (en
Inventor
小澤 開拓
Kaitaku Ozawa
開拓 小澤
南 猛
Takeshi Minami
猛 南
高橋 健一
Kenichi Takahashi
健一 高橋
松原 賢士
Kenji Matsubara
賢士 松原
義明 澁田
Yoshiaki Shibuta
義明 澁田
陽介 谷口
Yosuke Taniguchi
陽介 谷口
敦史 田村
Atsushi Tamura
敦史 田村
Original Assignee
コニカミノルタ株式会社
Konica Minolta Inc
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by コニカミノルタ株式会社, Konica Minolta Inc filed Critical コニカミノルタ株式会社
Priority to JP2014073150A priority Critical patent/JP6303723B2/en
Publication of JP2015194954A publication Critical patent/JP2015194954A/en
Application granted granted Critical
Publication of JP6303723B2 publication Critical patent/JP6303723B2/en
Application status is Active legal-status Critical
Anticipated expiration legal-status Critical

Links

Abstract

A display control device and a control program for the display control device are provided. A wearable display serving as a display control device acquires an image of a printed material DC on which a marker MK including information related to an AR (Augusted Reality) server is displayed, and acquires AR data from the AR server. The wearable display 100 displays an image in which the image of the AR data in the first display mode is superimposed on the image of the printed matter DC. The wearable display 100 is the first when the image data of the printed material DC includes at least one of the memorandum MR added to the printed material DC and the object other than the printed material DC in the AR data area RG. The image SI1 obtained by superimposing the image IM1 of the AR data of the second display mode different from the display mode of FIG. [Selection] Figure 4

Description

  The present invention relates to a display control device and a control program for the display control device. More specifically, the present invention relates to a display control device and a display control device for displaying a superimposed image acquired from a server so as to be superimposed on a document image. It relates to the control program.

  In recent years, interest in security has increased, and leakage of confidential information (confidential data) such as confidential information and personal information held by companies has become a problem. In order to deal with these problems, companies have taken measures such as providing access authority to confidential information that has been digitized and restricting bringing in and taking out recording media. In the case of computerized confidential information, it is possible to guard to some extent by implementing measures using the IT (Information Technology) mechanism as described above.

  On the other hand, in conferences and meetings held at companies and other organizations, printed materials in which confidential information is printed on paper are often distributed to participants as materials. In this case, it is difficult to manage the taking out and copying of the printed material on which the confidential information is printed, and it is not possible to guard against these actions by the participants using the IT mechanism as described above. This fact makes it difficult to maintain security.

As a technology to prevent leakage of confidential information from the printed material, when the terminal of the printed material's terminal is accessed through an AR (Augmented Reality) marker displayed on the printed material, the AR A technique for distributing confidential information as AR data to a terminal has been proposed.

  FIG. 15 is a diagram schematically showing a conventional technique using an AR marker.

  Referring to FIG. 15, in this technique, the image forming apparatus outputs a printed matter (paper document) DC as shown in FIG. The printed product DC includes an AR marker MK. The printed material DC originally contains confidential information, but the confidential information is hidden by the AR marker MK. Therefore, the owner of the printed material DC (hereinafter sometimes referred to as the owner) cannot grasp the content of the confidential information only by looking at the printed material DC.

  When viewing confidential information, the owner reads the AR marker MK with a portable communication terminal such as a glasses-type wearable display 1100. The wearable display 1100 that has read the AR marker MK accesses an AR server (not shown) and acquires AR data from the AR server. As shown in FIG. 15B, the wearable display 1100 displays, on the display unit, an image SI in which the image IM of the AR data is superimposed on the image of the printed matter DC. The image IM of AR data is displayed at a display position determined based on the position of the AR marker MK. As a result, the owner can see the image IM of AR data that is confidential information through the wearable display 1100.

  Conventional techniques relating to the AR marker are disclosed in, for example, the following Patent Documents 1 to 4.

  Patent Document 1 below captures an image of a basic document with an AR tag, extracts an AR tag from the captured image, acquires augmented reality additional information associated with the AR tag from a server device, and A portable communication terminal that displays augmented reality additional information superimposed on a captured image is disclosed.

  Patent Document 2 listed below discloses a portable terminal device that causes a display device to display an image captured by an imaging device. The mobile terminal device includes: an AR processing unit that displays an AR operation panel superimposed on a captured image based on AR operation panel data when an AR marker of the image forming apparatus is detected in the captured image; A communication processing unit for acquiring AR operation panel data from the server device and transmitting a request corresponding to a user operation on the AR operation panel to the server device when the AR marker of the image forming apparatus is detected in .

  Patent Document 3 below discloses an information processing system including an information processing device and a search device. The information processing apparatus includes an imaging unit, an imaging magnification changing unit that changes the imaging magnification, and a display unit that combines and displays the captured image and the tag display image. The information processing device transmits information including the position, imaging magnification, imaging direction, and imaging angle of view to the search device. The search device searches for tag information for generating a tag display image from the transmitted information by the search unit, and transmits predetermined tag information to the information processing device. The tag display image generation unit of the information processing device generates a tag display image displayed by the display unit from the transmitted tag information.

  Patent Document 4 below discloses an image composition device that converts an AR marker in a real video acquired by a camera into an AR object and displays the AR marker in the video. When the AR marker in the real image is a signboard or sign indicating an emergency or warning, the image composition device displays the AR object transparently and copies the real image corresponding to the AR object. The image is superimposed on the foreground of the real image.

JP 2013-026922 A JP 2013-025638 A JP 2012-216935 A JP 2012-164157 A

  The conventional technique related to the AR marker has a problem of low convenience. This will be described below.

  FIG. 16 is a diagram schematically illustrating an image SI100 of the printed material DC displayed on the conventional wearable display 1100 when a memo is present around the AR marker MK in the image of the printed material DC.

  Referring to FIG. 16A, it is assumed that the owner adds memo MR around AR marker MK in printed material DC, and then sees printed product DC on wearable display 1100. In this case, as shown in FIG. 16B, in the image SI100 displayed on the display unit of the wearable display 1100, the AR data image IM100 is displayed on the added note writing MR, and the added note writing is performed. MR is hidden.

  FIG. 17 is a diagram schematically showing an image SI101 of the printed product DC displayed on the conventional wearable display 1100 when the owner tries to add a note to the printed product DC.

  Referring to FIG. 17A, a case where the owner tries to add a note to the image IM101 of the AR data included in the image SI101 while viewing the image SI101 displayed on the display unit of the wearable display 1100. Suppose. In this case, the pen point PN to which a note is to be added is displayed on the display unit of the wearable display 1100. However, as shown in FIG. 17B, when the pen tip PN enters the area of the AR data image IM101, the pen tip PN is covered with the AR data image IM101, and the pen tip PN is hidden.

  As described above, when an AR data image corresponding to the AR marker is displayed on a mobile communication terminal or the like, notes and objects existing under the AR data image cannot be seen, resulting in a decrease in convenience.

  SUMMARY An advantage of some aspects of the invention is to provide a display control device and a control program for the display control device that are highly convenient.

  A display control device according to one aspect of the present invention includes a first image acquisition unit that acquires an image of a document on which a marker including information about a server is displayed, and a second image that acquires superimposed image data from the server. An acquisition unit, a first display unit that displays a superimposed image of the first display mode on a document image, and a region in which the superimposed image of the first display mode is superimposed on the document image. A first discriminating unit for discriminating whether or not at least one of a memorandum added to the document and an object other than the document exists, and a superimposed image of the first display mode Different from the first display mode when the first discriminating unit determines that at least one of a memo added to the document and an object other than the document exists in the matching area. Second And a second display means for displaying by superimposing the superimposed image of the display mode for the document image.

  Preferably, in the above display control device, after the superimposed image of the first display mode is displayed on the first display unit, the first determination unit is an area in which the superimposed image of the first display mode is superimposed. It is determined whether or not at least one of a memo added to the document and an object other than the document exists, and the document is within the region where the superimposed images of the first display mode are superimposed. The first display means displays the superimposed image of the first display mode when it is determined by the first determination means that neither the memo written added to the object nor the object other than the document exists. Keep doing.

  Preferably, in the display control device, the first determination means includes at least one of a memo written added to the document and a tip of the writing instrument in a region where the superimposed images of the first display mode are superimposed. Determine if it exists.

  Preferably, in the above display control apparatus, when the superimposed image of the second display mode is displayed by the second display unit, the writing instrument that existed in the region where the superimposed image of the first display mode is superimposed And a second discriminating means for discriminating whether or not the tip of the first display mode has moved out of the region where the superimposed image of the first display mode is superimposed, and the tip of the writing tool overlaps the superimposed image of the first display mode. When it is determined by the second determining means that the image has moved out of the area, it is determined whether or not the note added to the document exists in the area where the superimposed image of the first display mode is superimposed. And a third discriminating means for detecting that the memorandum added to the document exists in the region where the superimposed images of the first display mode are superimposed. In addition, the second display means is a polymerization of the second display mode. Continue to display the image.

  Preferably, in the above display control device, when the third determining means determines that there is no note added to the document in the region where the superimposed image of the first display mode is superimposed, The second display unit stops displaying the superimposed image in the second display mode, and the first display unit displays the superimposed image in the first display mode.

  Preferably, in the display control apparatus, the second display means includes at least one of a method of making at least a part of the superimposed image of the first display mode transparent, and a position and a size of the superimposed image of the first display mode. At least one of a method of changing one of the methods, and a method of adding at least one of an image of a memo added to a document and an image of an object other than a document to the superimposed image of the first display mode A creation unit that creates a superimposed image of the second display mode using any one of the methods, and a post-creation display unit that displays the superimposed image of the second display mode created by the creation unit.

  Preferably, in the display control device, the creation unit creates the first display mode superimposed image by using a method of making at least a part of the superimposed image of the first display mode transparent. In the superposed image of the display mode, at least a portion overlapping with the marker is not made transparent.

  Preferably, in the display control apparatus, the superimposed image of the second display mode is a memo written in a region where the superimposed image of the first display mode is superimposed, and an object other than the document. The display mode is such that at least one of them can be seen.

  A control program for a display control device according to another aspect of the present invention includes a first image acquisition step for acquiring an image of a document on which a marker including information related to a server is displayed, and a first image acquisition step for acquiring superimposed image data from the server. Two image acquisition steps; a first display step for displaying the superimposed image of the first display mode on the image of the document; and a superimposed image of the first display mode in the document image. A first determination step for determining whether or not at least one of a memorandum added to a document and an object other than a document exists in the overlapping region; In the first determination step, it is determined that at least one of a memo added to the document and an object other than the document exists in the region where the images are superimposed. The case, to execute the second display step includes a first display mode for displaying superposed against the superimposed image document images of different second display mode on the computer.

  ADVANTAGE OF THE INVENTION According to this invention, the control program of a display control apparatus and a display control apparatus with high convenience can be provided.

It is a block diagram which shows the structure of the AR data delivery system in one embodiment of this invention. It is a figure which shows notionally the basic operation | movement of the AR data delivery system in one embodiment of this invention. The figure which shows typically image SI displayed when neither memo writing nor objects other than printed matter DC exist in AR data area RG in the image of printed matter DC in one embodiment of this invention. It is. In an embodiment of the present invention, an image SI1 created by using the first change method that is displayed when a memo is present in the AR data area RG in the image of the printed matter DC is schematically shown. FIG. It is a figure which shows typically image SI1 produced using the 1st change method displayed when objects other than printed matter DC exist in AR data area RG in the image of printed matter DC. In one embodiment of the present invention, it is a diagram illustrating the operation of wearable display 100 after wearable display 100 displays image SI1. 5 is a flowchart showing the operation of wearable display 100 when the first change method is employed in an embodiment of the present invention. In an embodiment of the present invention, an image SI2 created by using the second change method, which is displayed when a memo is present in the AR data area RG in the image of the printed matter DC, is schematically shown. FIG. In an embodiment of the present invention, an image SI1 created by using the second change method is displayed when an object other than the printed material DC exists in the AR data area RG in the image of the printed material DC. It is a figure shown typically. In one embodiment of the present invention, it is a diagram for explaining the operation of wearable display 100 after wearable display 100 displays image SI2. In one embodiment of the present invention, it is a flowchart showing the operation of wearable display 100 when the second change method is adopted. In one embodiment of the present invention, an image SI3A created using the third change method, which is displayed when a memo is present in the AR data area RG in the image of the printed matter DC, is schematically shown. FIG. In an embodiment of the present invention, an image SI3B created using the third change method displayed when an object other than the printed material DC exists in the AR data area RG in the image of the printed material DC. It is a figure shown typically. In one embodiment of the present invention, it is a flowchart showing the operation of wearable display 100 when the third change method is adopted. It is a figure which shows schematically the prior art using AR marker. It is a figure which shows typically image SI100 of printed matter DC which the conventional wearable display 1100 displays, when the memo writing exists in the circumference | surroundings of AR marker MK in the image of printed matter DC. FIG. 10 is a diagram schematically showing an image SI101 of a printed material DC displayed on a conventional wearable display 1100 when an owner tries to add a note to the printed material DC.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings.

  In the following embodiment, a case where the display control device is a wearable display will be described. The display control device may be a mobile communication terminal other than a wearable display such as a mobile phone or a smartphone. The display control apparatus may be a server, an image forming apparatus, a digital camera, or a PC (Personal Computer) that communicates with the mobile communication terminal.

  [Configuration of AR data distribution system]

  First, the configuration of the AR data distribution system will be described.

  FIG. 1 is a block diagram showing a configuration of an AR data distribution system according to an embodiment of the present invention.

  Referring to FIG. 1, the AR data distribution system includes a wearable display 100 (an example of a display control device), an AR server 200 (an example of a server), and a network 300. The wearable display 100 and the AR server 200 are connected to each other through the network 300.

  The network 300 uses a dedicated line such as a wired or wireless LAN (Local Area Network). The network 300 connects various devices using a TCP / IP (Transmission Control Protocol / Internet Protocol) protocol. Devices connected to the network 300 can exchange various data with each other. Wearable display 100 and AR server 200 may be connected to each other through the Internet.

  Wearable display 100 is a glasses-type display worn on the head. The wearable display 100 includes a CPU (Central Processing Unit) 110 that controls the entire wearable display 100, a network I / F 120 that communicates with an external device such as the AR server 200 via the network 300 in accordance with an instruction from the CPU 110, A ROM (Read Only Memory) 130 that stores a control program executed by the CPU 110, a RAM (Random Access Memory) 140 that is a working memory of the CPU 110, a storage unit 150 that stores various types of information, and various types of information are displayed. A display unit 160, a camera 170 that captures an image, and an image composition unit 180 that superimposes an image of AR data are included. CPU 110 is connected to each of network I / F 120, ROM 130, RAM 140, storage unit 150, display unit 160, camera 170, and image composition unit 180. The storage unit 150 stores information related to the owner of the wearable display 100 and information for receiving authentication from the AR server 200 (for example, a user ID (Identification) and a password).

  The AR server 200 is composed of, for example, a PC. The AR server 200 includes a CPU 210 that controls the entire AR server 200, a network I / F 220 that communicates with an external device such as the wearable display 100 via the network 300 in accordance with an instruction from the CPU 210, and a control program that the CPU 210 executes. ROM 230 for storing, RAM 240 as a working memory of CPU 210, and a storage unit 250 for storing various information. The CPU 210 is connected to each of the network I / F 220, the ROM 230, the RAM 240, and the storage unit 250. The storage unit 250 stores an AR marker ID and AR data associated with the AR marker ID as AR server management information.

  [Operation of AR data distribution system]

  Next, the operation of the AR data distribution system in the present embodiment will be described.

  FIG. 2 is a diagram conceptually showing the basic operation of the AR data distribution system in one embodiment of the present invention.

  Referring to FIG. 2, printed matter DC is a paper document on which characters and the like are printed, and AR marker MK (for example, QR code (registered trademark)) is displayed. The marker MK includes, for example, information related to the AR server 200 (for example, the URL (Uniform Resource Locator) of the AR server 200), information on the ID of the AR marker, and the like.

  When the owner views the printed product DC using the wearable display 100, the camera 170 of the wearable display 100 acquires an image of the printed product DC (process PR1). The wearable display 100 recognizes the AR marker MK included in the image of the printed material DC (process PR2). Next, the wearable display 100 accesses the AR server 200 based on the information included in the AR marker MK, and transmits the user ID and password of the owner and the ID information of the AR marker to the AR server 200 (processing) PR3). Upon receiving the user ID and password, the AR server 200 authenticates the wearable display 100. After authenticating the wearable display 100, the AR server 200 extracts AR data associated with the received ID of the AR marker from the AR server management information (process PR4). Subsequently, the AR server 200 transmits the extracted AR data to the wearable display 100 (process PR5). When the wearable display 100 receives (acquires) the AR data, the image IM of the first display mode (an example of the superimposed image of the first display mode) is superimposed on the image of the printed product DC. The image SI is displayed on the display unit 160 (glasses lens unit) (processing PR6). The display mode of the AR data image IM is a default display mode of the received AR data image. As a result, the owner can see the image of the AR data through the wearable display 100.

  The wearable display 100 displays the image SI and then writes a note added to the printed material DC and an object other than the printed material DC (for example, the tip of a writing tool (pen nib)) in the AR data area (confidential data area) RG. When none of the above exists, the image IM of the AR data in the first display mode is continuously displayed.

  FIG. 3 schematically shows an image SI that is displayed when there is no memo and any object other than the printed material DC in the AR data area RG in the image of the printed material DC in the embodiment of the present invention. FIG.

  With reference to Fig.3 (a), the confidential information is contained in the document data used as the origin of printed matter DC. This document data is printed as a printed material DC in a state where the confidential information is hidden by the AR data area RG. The AR data area RG is provided at a position where confidential information is to be displayed on the printed material DC. The AR data area RG includes an area where the AR marker MK is displayed and a plain (opaque) area around the AR marker MK.

  The wearable display 100 changes the display mode of the image of the AR data when neither the memo pad added to the print product DC nor the object other than the print product DC exists in the AR data area RG in the image of the print product DC. Without continuing, the image SI shown in FIG. The image SI is obtained by superimposing the image IM of the AR data on the AR data area RG in the image of the printed matter DC.

  On the other hand, after display of image SI, wearable display 100 has either a memo written added to printed material DC or an object other than printed material DC in AR data area (confidential data area) RG. If so, the image of the AR data in the first display mode is changed to the image of the AR data in the second display mode, and the image of the AR data after the change is displayed.

  Specifically, the wearable display 100 created and created an image of the AR data in the second display mode using the following first change method, second change method, or third change method. An image of AR data is displayed.

    (First change method)

  The first changing method is a method of making at least a part of an image of AR data in the first display mode transparent, and creating an image of AR data in the second display mode.

  FIG. 4 shows an image SI1 created by using the first change method displayed when a memo is present in the AR data area RG in the image of the printed matter DC in the embodiment of the present invention. FIG.

  Referring to FIG. 4A, when the memo MR added to the printed matter DC exists in the AR data area RG in the image of the printed matter DC, the wearable display 100 is shown in FIG. The image SI1 is displayed on the display unit 160. The image SI1 is obtained by superimposing the image IM1 of the AR data in the second display mode on the AR data region RG of the image of the printed matter DC.

  The AR data image IM1 is created by a method (by the first changing method) in which at least a part of the AR data image IM in the first display mode is made transparent.

  The image IM1 of the AR data is preferably created by a method of making at least a portion where the memo MR is present so that the memo MR can be seen.

  The AR data image IM1 is created by making the periphery of the AR marker MK in the AR data image IM transparent (transparent image) and not making at least the portion PT overlapping the AR marker MK in the AR data image IM transparent. It is preferable. Thereby, it is possible to prevent the AR marker MK from being seen in the image SI1.

  FIG. 5 is a diagram schematically showing an image SI1 created using the first change method, which is displayed when an object other than the printed material DC exists in the AR data area RG in the image of the printed material DC. It is.

  Referring to FIG. 5A, when the owner tries to add a memo to the AR data image IM while viewing the image SI of the printed material DC, the AR data area RG in the image of the printed material DC The pen tip PN of the writing instrument (an example of the tip of the writing instrument) enters. Also in this case, the wearable display 100 displays the image SI1 shown in FIG. The image SI1 is the same as the image SI1 shown in FIG. The image IM1 of the AR data is preferably created by a method of making at least a portion where the pen tip PN exists transparent so that the pen tip PN can be seen.

  After the pen tip PN enters the AR data area RG in the image of the printed matter DC and the wearable display 100 displays the image SI1, the wearable display 100 may further perform the following operation.

  FIG. 6 is a diagram for explaining the operation of the wearable display 100 after the wearable display 100 displays the image SI1 in the embodiment of the present invention.

  Referring to FIG. 6A, after the wearable display 100 displays the image SI1, the nib PN existing in the AR data area RG in the image of the printed matter DC moves out of the AR data area RG. Is assumed. In this case, the wearable display 100 determines whether or not a note added to the printed material DC exists in the AR data area RG in the image of the printed material DC.

  When the memo MR added to the printed material DC does not exist in the AR data area RG in the image of the printed material DC, the wearable display 100 stops displaying the image SI1. The wearable display 100 displays the image SI on the display unit 160 instead of the image SI1. Thereby, the image of AR data can be returned to the default display state.

  With reference to FIG. 6B, when the memo MR added to the printed matter DC exists in the AR data area RG in the image of the printed matter DC, the wearable display 100 continues to display the image SI1. Thereby, even after the owner moves the nib PN out of the AR data area RG, it is possible to maintain a state in which the memo MR added to the printed material DC can be seen.

  FIG. 7 is a flowchart showing the operation of wearable display 100 when the first change method is adopted in the embodiment of the present invention.

  Referring to FIG. 7, when CPU 110 of wearable display 100 captures an image of a printed material using camera 170 (S101), CPU 110 determines whether an AR marker is included in the image of the printed material (S103).

  If the AR marker is not included in the printed image in step S103 (NO in S103), the CPU 110 proceeds to the process of step S101 because it is not necessary to acquire AR data.

  In step S103, when it is determined that the AR marker is included in the printed image (YES in S103), the CPU 110 accesses the AR server 200 based on the information included in the AR marker. It requests the AR data to be transmitted (S105). Next, the CPU 110 determines whether or not AR data has been acquired from the AR server (S107).

  If it is determined in step S107 that AR data has been acquired from the AR server (YES in S107), the CPU 110 displays an image of the AR data in the first display mode on the display unit 160 (S109). Subsequently, the CPU 110 determines whether or not a pen tip or a memo is present in the AR data area in the image of the printed product DC (S111).

  If it is determined in step S107 that AR data is not acquired from the AR server (NO in S107), or if it is determined in step S111 that neither the nib nor the memo is present in the AR data area (NO in S111). ), The CPU 110 ends the process.

  If it is determined in step S111 that at least one of the pen tip and the memo is present in the AR data area (YES in S111), the CPU 110 determines the AR in the image of the AR data in the first display mode. A portion (peripheral portion) other than the portion overlapping with the marker MK is changed to transmissive display (transparent image) (S113), and the AR data image in the second display mode, which is the changed AR data image, is displayed on the display unit 160. It is displayed (S115). Next, the CPU 110 determines whether or not the pen tip exists in the AR data area (S117).

  If it is determined in step S117 that the pen tip does not exist in the AR data area (NO in S117), the CPU 110 ends the process.

  If it is determined in step S117 that the pen tip is present in the AR data area (YES in S117), the CPU 110 determines whether the pen tip has moved out of the AR data area (S119). CPU 110 repeats the process of step S119 until it is determined that the pen tip has moved out of the AR data area.

  If it is determined in step S119 that the pen tip has moved out of the AR data area (YES in S119), the CPU 110 determines whether or not a note is present in the AR data area (S121).

  If it is determined in step S121 that a memo is present in the AR data area (YES in S121), CPU 110 displays an image of the AR data in the second display mode that is the image of the AR data after the change. Is maintained (S123), and the process is terminated.

  If it is determined in step S121 that no memorandum is present in the AR data area (NO in S121), the CPU 110 displays an image of the AR data in the first display mode that is an image of the original AR data. 160 (S125), and the process ends.

    (Second change method)

  The second changing method is to create an AR data image in the second display mode by changing at least one of the position and size of the AR data image in the first display mode.

  FIG. 8 shows an image SI2 created by using the second changing method, which is displayed when a note is present in the AR data area RG in the image of the printed matter DC in the embodiment of the present invention. FIG.

  With reference to FIG. 8A, when the memo MR added to the printed matter DC is present in the AR data area RG in the image of the printed matter DC, the wearable display 100 is shown in FIG. 8B. The image SI2 is displayed on the display unit 160. The image SI2 is obtained by superimposing the image IM2 of the AR data in the second display mode on a part of the AR data area RG of the image of the printed matter DC.

  The AR data image IM2 is created by the method of changing the position and size of the AR data image IM1 in the first display mode (by the second changing method). The AR data image IM2 may be created by a method of changing only one of the position and size of the AR data image IM.

  The image IM2 of the AR data is preferably set to a position and size so that the memo MR can be seen. In FIG. 8B, the vertical length and the horizontal length of the AR data image IM2 are reduced to about 70% of the vertical length and the horizontal length of the AR data image IM1. ing. The center point of the image IM2 of the AR data has moved to the lower left side compared to the center point of the AR data region RG. As a result, the memo MG written on the right side of the center point of the AR data area RG can be seen.

  The image IM2 of the AR data is preferably arranged so as to overlap with the AR marker MK so as to be within the image of the printed matter DC. Thereby, it is possible to prevent the AR marker MK from being seen in the image SI2.

  FIG. 9 is created using the second change method displayed when an object other than the printed material DC exists in the AR data area RG in the image of the printed material DC in the embodiment of the present invention. It is a figure which shows typically image SI1.

  Referring to FIG. 9A, even when the pen tip PN of the writing instrument enters the AR data area RG in the image of the printed material DC, the wearable display 100 displays the image SI2 shown in FIG. 160. The image SI2 is the same as the image SI2 shown in FIG. The image IM1 of the AR data is preferably created by a method of making at least a portion where the pen tip PN exists transparent so that the pen tip PN can be seen.

  After the pen tip PN enters the AR data area RG in the image of the printed matter DC and the wearable display 100 displays the image SI2, the wearable display 100 may further perform the following operation.

  FIG. 10 is a diagram for explaining the operation of the wearable display 100 after the wearable display 100 displays the image SI2 in the embodiment of the present invention.

  Referring to FIG. 10A, after the wearable display 100 displays the image SI2, the nib PN existing in the AR data area RG in the image of the printed matter DC moves out of the AR data area RG. Is assumed.

  In the above-described case, when the memo MR added to the printed matter DC does not exist in the AR data area RG in the image of the printed matter DC, the wearable display 100 stops displaying the image SI2. Wearable display 100 displays image SI on display unit 160 instead of image SI2. Thereby, the image of AR data can be returned to the default display state.

  Referring to FIG. 10B, in the above-described case, when the memorandum MR added to the printed material DC exists in the AR data area RG in the image of the printed material DC, the wearable display 100 displays the image. Continue to display SI2. Thereby, even after the owner moves the nib PN out of the AR data area RG, it is possible to maintain a state in which the memo MR added to the printed material DC can be seen.

  FIG. 11 is a flowchart showing the operation of wearable display 100 when the second changing method is adopted in the embodiment of the present invention.

  Referring to FIG. 11, CPU 110 of wearable display 100 first executes steps S201 to S211 corresponding to steps S101 to S111 in the flowchart shown in FIG. 7. The description of these processes will not be repeated.

  If it is determined in step S211 that at least one of the pen tip and the memo is present in the AR data area (YES in S211), the CPU 110 fits within the image of the printed matter (paper document). The position and size of the AR data image in the first display mode are changed, and the changed AR data image that is the AR data image in the second display mode is created (S213). Next, the CPU 110 displays the changed AR data image on the display unit 160 (S215). After that, CPU 110 executes processing after step S217 corresponding to the processing after step S117 in the flowchart shown in FIG. The description of these processes will not be repeated.

  If it is determined in step S211 that neither the pen tip nor the memo writing exists in the AR data area (NO in S211), the CPU 110 ends the process.

    (Third change method)

  In the third change method, at least one of a memo written added to the printed material and an image of an object other than the printed material is added to the image of the AR data in the first display mode. An image of AR data in a display mode is created.

  FIG. 12 shows an image SI3A created by using the third change method displayed when a memo is present in the AR data area RG in the image of the printed matter DC in the embodiment of the present invention. FIG.

  With reference to FIG. 12A, when the memo MR added to the printed matter DC is present in the AR data area RG in the image of the printed matter DC, the wearable display 100 is shown in FIG. The image SI3A is displayed on the display unit 160. The image SI3A is obtained by superimposing the AR data image IM3A of the second display mode on the AR data area RG of the image of the printed matter DC.

  The AR data image IM3A is created by the method of adding (combining) the image of the note MG to the AR data image IM of the first display mode (by the third changing method).

  FIG. 13 is a diagram schematically illustrating an image SI3B created by using the third change method, which is displayed when an object other than the printed material DC exists in the AR data area RG in the image of the printed material DC. It is.

  With reference to FIG. 13A, when the pen tip PN of the writing instrument enters the AR data area RG in the image of the printed matter DC, the wearable display 100 displays the image SI3B shown in FIG. indicate. The image SI3B is obtained by superimposing the AR data image IM3B of the second display mode on the AR data region RG of the image of the printed matter DC.

  The AR data image IM3B is created by the method of adding (combining) the image of the pen tip PN to the AR data IM of the first display mode (by the third change method).

  The AR data images IM3A and IM3B may be created by the wearable display 100. When the wearable display 100 creates the image IM3A or IM3B of AR data, the wearable display 100 transmits the created image IM3A or IM3B of AR data to the AR server 200, and the AR server 200 stores the stored AR data. The received AR data may be updated to that of the image IM3A or IM3B.

  The AR data images IM3A and IM3B may be created by the AR server 200. When the AR server 200 creates the AR data image IM3A or IM3B, the wearable display 100 transmits the memo or pen tip image data included in the printed DC image to the AR server 200, and the memo or pen tip. The AR data with the added image is received from the AR server 200. When the AR server 200 creates the AR data image IM3A or IM3B, the AR server 200 may update the stored AR data to the created AR data image IM3A or IM3B.

  After the wearable display 100 displays the image SI3B, the AR data area in the image of the printed matter DC when the pen point PN existing in the AR data area RG in the image of the printed matter DC moves out of the AR data area RG. When the memo added to the printed material DC does not exist in the RG, the wearable display 100 displays the image SI on the display unit 160 instead of the image SI3B. When the memo MR added to the printed product DC exists in the AR data area RG in the image of the printed product DC, the wearable display 100 continues to display the image SI3B.

  FIG. 14 is a flowchart showing the operation of the wearable display 100 when the third changing method is adopted in the embodiment of the present invention.

  Referring to FIG. 14, CPU 110 of wearable display 100 first executes steps S301 to S311 corresponding to steps S101 to S111 in the flowchart shown in FIG. 7. The description of these processes will not be repeated.

If it is determined in step S311 that at least one of the pen tip and the memo is present in the AR data area (YES in S311), the CPU 110
The image of the AR data in the first display mode is combined with the image of the memo pad or the pen tip, and the image of the AR data after the synthesis that is the image of the AR data in the second display mode is created (S313). . Next, the CPU 110 updates the AR data stored in the AR server 200 to the combined AR data (S314). Next, the CPU 110 displays an image of the AR data after synthesis on the display unit 160 (S315). Thereafter, CPU 110 executes processing after step S317 corresponding to the processing after step S117 in the flowchart shown in FIG. The description of these processes will not be repeated.

  If it is determined in step S311 that neither the pen tip nor the memo writing exists in the AR data area (NO in S311), the CPU 110 ends the process.

  [Effect of the embodiment]

  According to the above-described embodiment, when either one of the memo written added to the printed material and the object other than the printed material exists in the AR data area in the image of the printed material, Superimposes an image of AR data in a different display mode on an image of printed matter. As a result, the owner can see both the AR data image and the notes and objects (such as the pen tip) existing in the AR data area, and the AR data image is also printed on the printed matter. You can add notes without any special operations in the same way as normal images. As a result, convenience can be improved.

  In particular, a method for making at least a part of an image of AR data in the first display mode transparent, a method for changing at least one of the position and size of the image of AR data in the first display mode, and the first display The AR data of the second display mode is used by using at least one of a method of adding a memo written to the document and an image of an object other than the document to the image of the AR data of the mode. By creating an image, it becomes easier to see the memo and pen tip existing in the AR data area.

  [Others]

  The wearable display may obtain a document image printed on a printed material, a document image drawn on a whiteboard, or a document image displayed on a display such as a tablet.

  The wearable display includes a memo written in the AR data area and an object other than the printed material without displaying an image obtained by superimposing the image of the AR data in the first display mode on the image of the printed material. Whether one of them is present or not, and based on the determination result, it is determined whether to overlay the image of the AR data in the first display mode or the second display mode. May be.

  The wearable display displays the image of the AR data in the second display form when the pen tip enters the AR data area in the image of the printed matter or when the finger of the owner enters the image of the printed matter. May be displayed in a superimposed manner.

  Each process of the flowcharts shown in FIGS. 7, 11, and 14 may be executed by the AR server instead of being executed by the wearable display.

  The first to third changing methods can be appropriately combined. For example, by combining the first change method and the second change method, the image of the AR data in the first display form is made transparent, and the position and size of the AR data image in the first display form are changed. The image may be used as an image of AR data in the second display mode.

  The processing in the above-described embodiment may be performed by software or may be performed using a hardware circuit. It is also possible to provide a program for executing the processing in the above-described embodiment, and record the program on a recording medium such as a CD-ROM, a flexible disk, a hard disk, a ROM, a RAM, or a memory card and provide it to the user. You may decide to do it. The program is executed by a computer such as a CPU. The program may be downloaded to the apparatus via a communication line such as the Internet.

  The above-described embodiment is to be considered as illustrative in all points and not restrictive. The scope of the present invention is defined by the terms of the claims, rather than the description above, and is intended to include any modifications within the scope and meaning equivalent to the terms of the claims.

100, 1100 Wearable display 110 CPU (Central Processing Unit)
120,220 Network I / F
130,230 ROM (Read Only Memory)
140,240 RAM (Random Access Memory)
150, 250 Storage unit 160 Display unit 170 Camera 180 Image composition unit 200 AR (Augmented Reality) server 300 Network DC printed material IM, IM1, IM2, IM3A, IM3B, IM100, IM101 AR data image MK AR marker MR Note PN pen Prior PR1, PR2, PR3, PR4, PR5, PR6 Processing PT AR data image part RG AR data area SI, SI1, SI2, SI3A, SI3B, SI100, SI101 Image of AR data superimposed

Claims (9)

  1. First image acquisition means for acquiring an image of a document on which a marker including information about the server is displayed;
    Second image acquisition means for acquiring superimposed image data from the server;
    First display means for displaying the superimposed image of the first display mode superimposed on the image of the document;
    Whether at least one of a note added to the document and an object other than the document exists in an area where the superimposed image of the first display mode is overlapped in the image of the document First determining means for determining whether or not,
    The first determination is made when at least one of a memo written added to the document and an object other than the document exists in a region where the superimposed images of the first display mode are superimposed. A second display means for displaying the superimposed image in a second display mode different from the first display mode on the image of the document when it is discriminated by the means, Display control device.
  2. After the superimposed image of the first display mode is displayed on the first display unit, the first determination unit is in a region where the superimposed image of the first display mode is superimposed. Determining whether there is at least one of a note added to the document and an object other than the document;
    The first discriminating unit discriminates that neither a memo added to the document nor an object other than the document exists in the region where the superimposed images of the first display mode are superimposed. 2. The display control apparatus according to claim 1, wherein the first display unit continues to display the superimposed image in the first display mode when the display is performed.
  3.   Whether the first discriminating means includes at least one of a note added to the document and a tip of a writing instrument in a region where the superimposed images of the first display mode are superimposed. The display control device according to claim 1, wherein the display control device determines whether or not.
  4. When the superimposed image of the second display mode is displayed on the second display means, the tip of the writing instrument that existed in the region where the superimposed image of the first display mode is superimposed is Second discriminating means for discriminating whether or not the superimposed image of the first display mode has moved out of the region to be superimposed;
    When the second discriminating unit determines that the tip of the writing instrument has moved out of the region where the superimposed image of the first display mode is superimposed, the superimposed image of the first display mode is superimposed. A third discriminating unit for discriminating whether or not a note added to the document exists in the matching area;
    The second display when the third determining means detects that a memo added to the document is present in the region where the superimposed images of the first display mode are superimposed. The display control apparatus according to claim 3, wherein the means continues to display the superimposed image in the second display mode.
  5.   The second display when the third determining means determines that the note added to the document does not exist in the region where the superimposed images of the first display mode are superimposed. 5. The display according to claim 4, wherein the means stops displaying the superimposed image in the second display mode, and the first display means displays the superimposed image in the first display mode. Control device.
  6. The second display means changes at least one of a method of making at least a part of the superimposed image of the first display mode transparent, and a position and a size of the superimposed image of the first display mode. And a method of adding at least one of an image of a note added to the document and an image of an object other than the document to the superimposed image of the first display mode Creating means for creating the superimposed image of the second display mode using at least one method;
    The display control apparatus according to claim 1, further comprising a post-creation display unit that displays the superimposed image of the second display mode created by the creation unit.
  7.   The creating means creates the first display when creating the superimposed image of the second display mode using a method of making at least a part of the superimposed image of the first display mode transparent. The display control apparatus according to claim 6, wherein at least a portion overlapping the marker in the superimposed image of the aspect is not transparent.
  8.   The superimposed image in the second display mode is a memo written in the area where the superimposed image in the first display mode is superimposed, and an object other than the document. The display control device according to claim 1, wherein the display control device is in a display mode in which at least one of them can be seen.
  9. A first image acquisition step of acquiring an image of a document on which a marker including information about the server is displayed;
    A second image acquisition step of acquiring superimposed image data from the server;
    A first display step of displaying the superimposed image of the first display mode superimposed on the image of the document;
    Whether at least one of a note added to the document and an object other than the document exists in an area where the superimposed image of the first display mode is overlapped in the image of the document A first determination step of determining whether or not,
    The first determination is made when at least one of a memo written added to the document and an object other than the document exists in a region where the superimposed images of the first display mode are superimposed. When determined in step, the computer executes a second display step for displaying the superimposed image in a second display mode different from the first display mode on the image of the document. A display control device control program.
JP2014073150A 2014-03-31 2014-03-31 Display control device and display control device control program Active JP6303723B2 (en)

Priority Applications (1)

Application Number Priority Date Filing Date Title
JP2014073150A JP6303723B2 (en) 2014-03-31 2014-03-31 Display control device and display control device control program

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
JP2014073150A JP6303723B2 (en) 2014-03-31 2014-03-31 Display control device and display control device control program

Publications (2)

Publication Number Publication Date
JP2015194954A true JP2015194954A (en) 2015-11-05
JP6303723B2 JP6303723B2 (en) 2018-04-04

Family

ID=54433884

Family Applications (1)

Application Number Title Priority Date Filing Date
JP2014073150A Active JP6303723B2 (en) 2014-03-31 2014-03-31 Display control device and display control device control program

Country Status (1)

Country Link
JP (1) JP6303723B2 (en)

Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006126936A (en) * 2004-10-26 2006-05-18 Canon Inc Image processing method and image processing device
JP2013026922A (en) * 2011-07-22 2013-02-04 Kyocera Document Solutions Inc Image formation system, information processing device, image formation device, and computer program

Patent Citations (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2006126936A (en) * 2004-10-26 2006-05-18 Canon Inc Image processing method and image processing device
JP2013026922A (en) * 2011-07-22 2013-02-04 Kyocera Document Solutions Inc Image formation system, information processing device, image formation device, and computer program

Also Published As

Publication number Publication date
JP6303723B2 (en) 2018-04-04

Similar Documents

Publication Publication Date Title
KR102010221B1 (en) Smartphone-based methods and systems
JP4679661B1 (en) Information presenting apparatus, information presenting method, and program
JP3834766B2 (en) Man machine interface system
US9778464B2 (en) Shape recognition device, shape recognition program, and shape recognition method
US20060204135A1 (en) Image output apparatus, image output method and image output program
US8055061B2 (en) Method and apparatus for generating three-dimensional model information
KR20120075487A (en) Sensor-based mobile search, related methods and systems
KR20120127655A (en) Intuitive computing methods and systems
US20140091984A1 (en) Method and apparatus for providing an indication regarding content presented to another user
US20150070347A1 (en) Computer-vision based augmented reality system
US9536350B2 (en) Touch and social cues as inputs into a computer
AU2006233222B2 (en) Conference system
US20130174213A1 (en) Implicit sharing and privacy control through physical behaviors using sensor-rich devices
CN103620600B (en) Method and apparatus for enabling virtual tags
US20100317399A1 (en) Content sharing methods and systems
US20100208033A1 (en) Personal Media Landscapes in Mixed Reality
US20150040074A1 (en) Methods and systems for enabling creation of augmented reality content
CN100479530C (en) Mixed reality space image generation method and mixed reality system
JP2010512693A (en) System and method for data addition, recording and communication
KR101108743B1 (en) Method and apparatus for holographic user interface communication
EP2499635B1 (en) Augmented reality system, method and apparatus for displaying an item image in a contextual environment
JP6105092B2 (en) Method and apparatus for providing augmented reality using optical character recognition
JP5852135B2 (en) Superimposed annotation output
US9182815B2 (en) Making static printed content dynamic with virtual data
EP2508975B1 (en) Display control device, display control method, and program

Legal Events

Date Code Title Description
A621 Written request for application examination

Free format text: JAPANESE INTERMEDIATE CODE: A621

Effective date: 20161221

A977 Report on retrieval

Free format text: JAPANESE INTERMEDIATE CODE: A971007

Effective date: 20170810

A131 Notification of reasons for refusal

Free format text: JAPANESE INTERMEDIATE CODE: A131

Effective date: 20170829

A521 Written amendment

Free format text: JAPANESE INTERMEDIATE CODE: A523

Effective date: 20171023

TRDD Decision of grant or rejection written
A01 Written decision to grant a patent or to grant a registration (utility model)

Free format text: JAPANESE INTERMEDIATE CODE: A01

Effective date: 20180206

A61 First payment of annual fees (during grant procedure)

Free format text: JAPANESE INTERMEDIATE CODE: A61

Effective date: 20180219

R150 Certificate of patent or registration of utility model

Ref document number: 6303723

Country of ref document: JP

Free format text: JAPANESE INTERMEDIATE CODE: R150