JP6264982B2 - Information display device, information display system, and program - Google Patents

Description

  The present invention relates to an information display device, an information display system, and a program.

  In recent years, mobile terminals such as smart phones and tablet terminals have been used for business. For example, when business documents including confidential information are viewed using a mobile terminal while traveling using public transportation means such as a train. There is. In this case, as a technique for preventing leakage of confidential information due to so-called peeping, the display content being viewed by a third party behind, a technique for reducing the brightness of the display area of the confidential information in the mobile terminal (for example, Patent Document 1). ) And a technique for displaying confidential information on a portable terminal in a concealed state (for example, Patent Document 2).

JP 2011-133661 A JP 2011-118842 A

  It is an object of the present invention to allow a user to browse information without reducing leakage due to peeping of information displayed on an information terminal device and without reducing the visibility of the user himself / herself.

  An information display device according to the present invention includes a display unit, a photographing unit, and a reference position member provided on a front surface facing the display unit when a user wearing the wearable display device views the display unit. A connecting means for connecting a wearable display device equipped with the transmissive display screen, which is formed so that the transmissive display screen is positioned in front of the user's eyes when the user is worn; and a display image; A dividing unit that divides the first image into a second image including a portion other than the first image of the display image; and the photographing unit when the wearable display device is at a reference position with respect to the information terminal device. Acquisition means for acquiring reference information relating to the wearable display device from imaging data of the wearable display device taken by the user, and the wearable display worn by a user who views the display means when the display image is displayed The equipment The display image is formed by the first image and the second image that are divided and displayed on the display means and the transmissive display screen, respectively, based on the imaging data acquired by being shaded and the reference information. And correcting means for correcting the second image, and controlling the display of the first image on the display means and the display of the second image corrected by the correction means on the transmissive display screen. Display control means.

  When the display image includes confidential information, the dividing means prevents the confidential information from being read only by the first image, or the confidential information is included only in the second image. It is divided into two images.

  Further, the correcting means is based on the relationship between the size of the reference position member included in the imaging data obtained by photographing by the photographing means and the size of the reference position member obtained from the reference information. The size difference between the first image and the second image is corrected by adjusting the size of the second image generated by the division.

  Further, the correcting means is based on the relationship between the position of the reference position member included in the imaging data obtained by photographing by the photographing means and the position of the reference position member obtained from the reference information. A positional shift between the first image and the second image is corrected by adjusting a display position of the second image generated by the division on the transmissive display screen.

  In addition, the correction unit may perform the division based on a relationship between the size of the wearable display device included in the imaging data obtained by photographing by the photographing unit and the size of the wearable display device obtained from the reference information. The inclination of the second image with respect to the first image is corrected by distorting the second image generated by the division by the means.

  An information display system according to the present invention includes a wearable display device formed so that a mounted transmissive display screen is positioned in front of a user's eyes when the user is worn, and information including a photographing unit and a display unit. And the wearable display device has a reference position member provided on the front surface facing the display means when a user wearing the wearable display device looks at the display means. The information terminal device includes: a connecting unit that connects the wearable display device; and a dividing unit that divides the display image into a first image and a second image that includes a portion other than the first image of the display image. Obtaining means for acquiring reference information relating to the wearable display device from imaging data of the wearable display device photographed by the photographing means when the wearable display device is at a reference position with respect to the information terminal device; The above The display means and the transmissive display screen based on the imaging data and the reference information acquired by photographing the wearable display device worn by the user who views the display means when a display image is displayed Correction means for correcting the second image so that the display image is formed by the first image and the second image displayed separately, and the first image is displayed on the display means. Display control means for controlling to display the second image corrected by the correction means on the transmissive display screen.

  In addition, a height indicating member for indicating the height of the wearable display device is disposed at each of upper and lower ends of the wearable display device.

  In addition, a width indicating member for indicating the width of the wearable display device is disposed on each of left and right ends of the wearable display device.

  The program according to the present invention includes a computer provided with photographing means, a display means, and a reference position provided on the front surface facing the display means when a user wearing the wearable display device views the display means. A connecting means for connecting a wearable display device equipped with the transmissive display screen, wherein the transmissive display screen is positioned in front of the user's eyes when the user wears the display. Dividing means for dividing the first image into a second image composed of a portion other than the first image of the display image; and the photographing means when the wearable display device is at a reference position with respect to the information terminal device Acquisition means for acquiring reference information relating to the wearable display device from imaging data of the wearable display device taken by the user, the wearer wearing the user who sees the display means when the display image is displayed The first image and the second image that are divided and displayed on the display means and the transmissive display screen, respectively, based on the imaging data acquired by photographing the type display device and the reference information. Correction means for correcting the second image so that a display image is formed, displaying the first image on the display means, and displaying the second image corrected by the correction means on the transmissive display screen It is for functioning as a display control means for controlling.

  According to the first aspect of the present invention, the leakage of information due to the peeping of information displayed on the information terminal device is reduced, and the visibility of the person when the user uses the wearable display device is reduced. The user can browse the information without letting it go.

  According to the second aspect of the present invention, leakage of confidential information can be prevented.

  According to the third aspect of the present invention, it is possible to correct an image size shift between the first image and the second image associated with the distance relationship between the information display device and the wearable display device.

  According to the fourth aspect of the present invention, it is possible to correct the positional deviation between the first image and the second image associated with the positional relationship of the wearable display device with respect to the information display device.

  According to the fifth aspect of the present invention, it is possible to correct the inclination of the second image relative to the first image accompanying the inclination of the wearable display device with respect to the information display device.

  According to the sixth aspect of the present invention, the leakage of information due to the peeping of information displayed on the information terminal device is reduced, and the visibility of the person when the user uses the wearable display device is reduced. The user can browse the information without letting it go.

  According to the seventh aspect of the invention, the width of the wearable display device can be easily detected from the photographing data.

  According to the eighth aspect of the present invention, the height of the wearable display device can be easily detected from the photographing data.

  According to the invention described in claim 9, while reducing leakage of information due to peeping of information displayed on the information terminal device, the visibility of the person when the user uses the wearable display device is reduced. The user can browse the information without letting it go.

It is the block block diagram which showed Embodiment 1 of the information display apparatus which concerns on this invention. FIG. 2 is a hardware configuration diagram of a computer forming the information display device in the first embodiment. 6 is a diagram showing an example of a business document in Embodiment 1. FIG. 6 is a diagram illustrating an example of a divided image of a business document displayed on an operation panel according to Embodiment 1. FIG. 6 is a diagram illustrating an example of a divided image of a business document displayed on the HMD in Embodiment 1. FIG. FIG. 6 is a diagram illustrating an example of a divided image of a business document in which another business document and the business document are displayed on an operation panel and an HMD, respectively, in the first embodiment. 6 is a diagram illustrating an example of a divided image of a business document in which another business document and the business document in Embodiment 1 are displayed on an operation panel and an HMD, respectively, and are divided by another division method. FIG. 3 is a flowchart showing display processing in the first embodiment. It is the flowchart which showed the image division process in FIG. It is the conceptual diagram which showed an example of the display of the memo document editing application displayed on the operation panel of a general information display apparatus. It is the conceptual diagram which showed an example of the divided image displayed on HMD as information to be kept secret among the images shown in the example of a display of FIG. It is the conceptual diagram which showed an example of the divided image displayed on an operation panel as information which does not need to be concealed among the images shown in the example of a display of FIG. It is the conceptual diagram which showed an example of the display by the mail application displayed on the operation panel of a general information display apparatus. It is the conceptual diagram which showed an example of the divided image displayed on HMD as information to be kept secret among the images shown in the example of a display of FIG. It is the conceptual diagram which showed an example of the divided image displayed on an operation panel as information which does not need to be concealed among the images shown in the example of a display of FIG. 3 is a block configuration diagram of an information display device in Embodiment 2. FIG. 10 is a diagram showing a positional relationship between an operation panel of the information display device and a camera lens in Embodiment 2. FIG. 6 is a perspective view of an HMD used in Embodiment 2. FIG. (A) is a plan view when the HMD shown in FIG. 18 is viewed from above, and (b) is a side view when the HMD shown in FIG. 18 is viewed from the side. 6 is a diagram showing a positional relationship between an HMD and an information display device when in a reference positional relationship in Embodiment 2. FIG. 10 is a flowchart showing display processing in the second embodiment. It is the flowchart which showed the image correction process in FIG. FIG. 10 is a diagram illustrating imaging data acquired in an image correction process in the second embodiment. FIG. 10 is a diagram illustrating a vertical positional relationship between a specific mark image and a specific mark on a camera image in the second embodiment. FIG. 10 is a diagram illustrating a horizontal inclination of the HMD with respect to the information display device in the second embodiment. FIG. 10 is a diagram illustrating a vertical inclination of the HMD with respect to the information display device in the second embodiment.

  Hereinafter, preferred embodiments of the present invention will be described with reference to the drawings.

Embodiment 1 FIG.
FIG. 1 is a block configuration diagram showing an embodiment of an information display system according to the present invention. The information display system according to the present embodiment includes a head mounted display (hereinafter referred to as “HMD”) 9 and an information display device 10. The information display device 10 is assumed to be a portable computer suitable for use in a public place such as a smart phone or a tablet terminal.

  The HMD 9 is a wearable display device that is worn on the user's head, and has a glasses-like shape formed so that a transmissive display screen is positioned in front of the user's eyes when the user wears the device. It is assumed that the equipment used will be used. Strictly speaking, the image is displayed on a display screen (liquid crystal panel) mounted on the HMD 9, but in the following description, it is also described as “display on the HMD 9” for convenience.

  FIG. 2 is a hardware configuration diagram of a computer forming the information display device 10 according to the present embodiment. The computer forming the information display apparatus 10 in the present embodiment can be realized by a general-purpose hardware configuration that has existed in the past. That is, as shown in FIG. 2, the computer includes the CPU 1, ROM 2, RAM 3, hard disk drive (HDD) 4, network controller 5 provided as communication means, and operation panel 6 provided as user interface means on the internal bus 7. Connected and configured.

  Returning to FIG. 1, the information display device 10 according to the present exemplary embodiment includes an image acquisition unit 11, an image division unit 12, an HMD connection unit 13, a display control unit 14, and a processing control unit 15. Note that components not used in the description of the present embodiment are omitted from FIG.

  The image acquisition unit 11 acquires an image to be displayed on the information display system. The image dividing unit 12 is provided as a dividing unit, and the image acquired by the image acquiring unit 11 is a first image and a second image made up of parts other than the first image so that the images cannot be easily read only by the respective images. Divide into In addition, when the image acquired by the image acquisition unit 11 is an image including confidential information (confidential information), the confidential information cannot be read only by the first image, or the confidential information is second. You may divide | segment into a 1st image and a 2nd image so that it may be included only in an image. The HMD connection unit 13 is provided as a connection unit, and connects the HMD 9 to the information display device 10 so as to be communicable. The connection form may be wired or wireless. The display control unit 14 is provided as a display control unit, and controls to display the first image on the operation panel 6 that is a display unit and to display the second image on the HMD 9. The process control unit 15 performs operation control of each component 11 to 14. Also, processing necessary for performing operation control, specifically, document type discrimination, current position detection, and the like are performed.

  Each component 11-15 in the information display apparatus 10 is implement | achieved by cooperation operation | movement with the program which operate | moves with the computer which forms the information display apparatus 10, and CPU1 mounted in the computer.

  Further, the program used in this embodiment can be provided not only by communication means but also by storing it in a computer-readable recording medium such as a CD-ROM or DVD-ROM. The program provided from the communication means or the recording medium is installed in the computer, and various processes are realized by the CPU of the computer sequentially executing the program.

  Next, before describing the display processing of the document image in the present embodiment, how the document image is displayed as a result of performing the display processing in the present embodiment will be described first.

  FIG. 3 is a diagram illustrating an example of a business document including information to be kept secret that is selected by the user as a display target on the information display device 10. In the present embodiment, for the sake of convenience, the “business document” means a document including information that is desired to be kept secret in order to prevent information leakage to a third party. This is not limited to the so-called “business” document, and may be a document including contents that the user does not want to be seen, such as a personal mail document or memo created by the user. The information display device 10 according to the present embodiment generates the divided images shown in FIGS. 4 and 5 in response to the display request for the business document illustrated in FIG. 3 by the user. The divided image as the first image shown in FIG. 4 is a divided image to be displayed on the operation panel 6, and the divided image as the second image shown in FIG. 5 is a divided image to be displayed on the HMD 9. is there. Even if each of the divided images is browsed independently, the contents described in the business document cannot be easily confirmed. By the way, in the transmissive HMD 9, the black portion of the divided image is transmitted, so that the portion ahead of the black portion is visible. Therefore, if the user views the operation panel 6 so that the black portion shown in FIG. 5 is aligned with the displayed character portion shown in FIG. 4, the divided image displayed on the HMD 9 is displayed. When the divided images displayed on the operation panel 6 appear to be combined, the business document shown in FIG. 3 is formed as an image visible to the user. As described above, the divided image displayed on the operation panel 6 via the HMD 9 on which the divided image shown in FIG. 5 is displayed is in a display state that cannot be easily read only by the divided image displayed on the operation panel 6. By seeing, only the user can read the business document.

  3 to 5 show business documents made up of character strings, while FIGS. 6 and 7 show cases of business documents made up of figures. FIG. 6 shows an example in which the business document 21 is divided into a grid and each divided image is displayed on the operation panel 6 and the HMD 9. FIG. 7 shows an example in which the business document 21 is divided into a central part and its peripheral part, and a split image including the central part is displayed on the HMD 9 and a split image including the peripheral part is displayed on the operation panel 6. Has been. In the image division method illustrated in FIG. 7, there is a high possibility that confidential information is included only in the HMD 9 and the confidential information can be visually recognized only by the display content of the HMD 9, but the display content on the HMD 9 is peeped by a third party. The secret information will not be leaked.

  As described above, the present embodiment is characterized in that the confidential information cannot be read only by the divided image displayed by dividing the business document including the confidential information and displayed on the operation panel 6. For this purpose, it is necessary to divide the confidential information and display it on the operation panel 6 and the HMD 9 or to divide the business document so that the confidential information is included only in the divided images displayed on the HMD 9.

  First, in the former case, when the confidential information is a character string as illustrated in FIG. 3, the character string is arranged by performing image analysis, for example, a straight line passing through the lower end of each character included in the character string is defined as a reference line. To do. And it divides into two parts up and down. That is, the character string is divided into upper and lower parts based on the center. Of course, this is an example of a division method, and the character may be further divided into three or more divisions so that the characters cannot be read more reliably. Or you may divide | segment into a grid | lattice form as illustrated in FIG. 4 and 5, the boundary of the divided image is a straight line, but may be a wavy line or a curve. For example, in the case of horizontal writing, it may be divided in the vertical direction. In the case where the confidential information is a graphic as illustrated in FIGS. 6 and 7, the area may be divided into a lattice shape as illustrated in FIGS. In any case, in this embodiment, even if one divided image is referred to, the character string is divided so that the visibility is lowered and the character string is difficult to read.

  In the latter case, when it can be grasped that the display area of the confidential information is predetermined in the business document image, the second image is formed so as to include at least an image corresponding to the display area of the confidential information, and the HMD 9 May be displayed.

  Further, when character strings and figures are mixed in the confidential information, a predetermined division method, for example, a method of always dividing into a lattice shape may be adopted, or image analysis is performed to Each display area of the graphic may be specified, and the above-described division process may be performed for each specified area.

  3 to 6 show examples in which the image to be displayed is divided almost evenly, but the divided image displayed on the operation panel 6 is divided so as to be smaller than the divided image displayed on the HMD 9. Then, the visibility with only the operation panel 6 may be further reduced.

  Next, display processing for displaying an image on the information display system in this embodiment will be described with reference to the flowchart shown in FIG.

  First, a user of an information display system that displays an image designates a document to be displayed by operating the operation panel 4 of the information display device 10 with the HMD 9 mounted on the head in advance. The document to be displayed may be a document stored in the HDD 4 of the information display device 10 or a document obtainable via a network. The image acquisition unit 11 acquires a document designated by the user. When the document acquired by the image acquisition unit 11 is not a business document (N in Step 110), the display control unit 14 displays the document as it is on the operation panel 6 (Step 150). That is, if the document is not a business document, the entire document may be displayed on the operation panel 6 as usual without performing the processing described later on the acquired document. Whether a business document is a business document may be determined based on the rule when the business document is managed according to a predetermined rule such as a storage location or a document name. In addition, it may be determined whether the document is a business document based on the type of application program that performs document browsing or document editing, such as a document creation application program, mail browsing, or creation application program. Further, it may be configured that the user instructs the document to be a business document by operating the operation panel 6.

  When the document acquired by the image acquisition unit 11 is a business document (Y in step 110), the processing control unit 15 subsequently determines whether the current position where the user is trying to display the business document is a public place. Judge whether. This may be determined by comparing the current position and the position information of the office when the information display device 10 is connected to an in-house network or has a GPS function. When it is determined that the user is not in a public place (N in Step 120), the display control unit 14 displays the document on the operation panel 6 (Step 150).

  On the other hand, when it is determined that the user is in a public place (Y in step 120), the image dividing unit 12 divides the document image as follows (step 130). The flow of the image division process will be briefly described with reference to the flowchart shown in FIG. A public place refers to a place where confidential information may be leaked due to the possibility that a business document displayed on the operation panel 6 can be viewed by a third party. Further, the user may operate the operation panel 6 to instruct whether or not the user is in a public place.

  By the way, the HMD 9 includes a binocular type in which the display screen is located in front of both eyes and a monocular type in which the display screen is located only in one eye. It can be known in advance which type of HMD 9 connected to the information display device 10 is. When the HMD 9 is a binocular type (Y in step 131), the image dividing unit 12 divides the document image into three (step 132), and when the HMD 9 is a monocular type (N in step 131), the image dividing unit 12 divides the document image into two. (Step 133).

  By the way, the present embodiment is characterized in that the document image to be displayed is divided as described above, and the divided images are displayed on the operation panel 6 and the HMD 9 respectively. When the HMD 9 is a binocular type, there are three screens including a display screen corresponding to each of the left and right eyes and the operation panel 6, so the document image is divided into three. That is, this is because different divided images are displayed on both screens of the HMD 9. However, for the sake of brevity, the description will be made assuming that the image of the same content is displayed on both display screens even if the HMD 9 is a binocular type unless otherwise specified. That is, the description will be made assuming that the HMD 9 is divided into two even if it is a binocular type.

  Subsequently, when the business document to be image-divided is a document consisting of a character string (Y in step 134), the image dividing unit 12 performs character division such as bi-division or tri-division using the lower end of the character as a reference line as described above On the other hand, if the document to be image-divided is a document including a figure (N in step 134), the figure is divided such as dividing into a grid as described above (step 135). Step 136). At this time, an area for seeing through the display content of the operation panel 6 included in the divided image displayed on the HMD 9 is processed into black. This is because the black display on the transmissive HMD becomes transparent. On the other hand, the area corresponding to the display position of the HMD 9 included in the divided image displayed on the operation panel 6 is processed into white.

  Here, for convenience, the business document is branched depending on the document composed of character strings or figures. However, when the character strings and the figures are mixed, the business document image is displayed according to a predetermined rule as described above. What is necessary is just to divide | segment an image by dividing | segmenting or analyzing a business document image and specifying each area | region of a character string and a figure.

  When the entire confidential information is displayed on the operation panel 6, there is a possibility that the confidential information may be leaked by being peeped at by a third party in the same public place. Therefore, when dividing the image, the image dividing unit 12 divides the confidential information so that it is displayed only on the display screen of the HMD 9 as described above. Alternatively, the entire confidential information is divided so as not to be displayed on the operation panel 6 so that the confidential information cannot be read from the display content of the operation panel 6.

  When the image dividing unit 12 divides the business document image as described above to form the operation panel and HMD divided images, the display control unit 14 displays the operation panel divided images on the operation panel 6. The HMD divided images are respectively displayed on the HMD 9 (step 140).

  The user interprets the business document by aligning the divided images displayed on the operation panel 6 and the HMD 9 as described above. On the other hand, even if a third party behind the user looks into the operation panel 6, the contents of the document cannot be read only with the divided image of the business document.

  If the image to be divided is a color image, it may be divided by color (RGB) instead of area division as described above, and only a single color may be displayed on the operation panel 6.

  In addition, when the business document to be displayed includes GUI (graphical user interface) parts that can be operated by the user and used for turning pages or inputting characters, the GUI parts are displayed by the user when displayed on the HMD 9. Operation becomes difficult. Therefore, an image including a GUI component may be displayed on the operation panel 6 of the information display device 10 that allows easy operation of the GUI component.

  The image division process in this example will be described. FIGS. 10 to 15 are conceptual diagrams used to explain screen display when editing a document using an application program for editing a memo document in the information display device 10. FIG. 10 is a display example in the memo document editing application displayed on the operation panel 6 of the general information display device 10. The GUI component constituting the displayed information is a display screen including “memo” as the application program name display field 101, a character information input field 102, and a virtual keyboard 103 for inputting characters. The user operates the virtual keyboard 103 displayed on the operation panel 6 to display the characters input in the character information input field 102. In the present embodiment, the image dividing unit 12 generates a divided image composed of information to be concealed and displays it on the HMD 9, generates a divided image composed of information that does not need to be concealed and displays it on the operation panel 6.

  FIG. 11 is an example of a divided image displayed on the HMD 9 as information to be kept secret. The display mode by this divided image is as follows. Since the character input field 112 displays the result of character input, it is displayed on the HMD 9 because it is information that is not desired to be viewed by a third party. Of the character input keys constituting the virtual keyboard 113, the character input keys that are actually input by the user using the operation panel 6 are displayed on the HMD 9. This is to prevent a third party from seeing characters that are actually input. Character input keys that are not operated by the user may be hidden on the HMD 9 or may be configured to display only the border of the virtual keyboard 113 as shown. In FIG. 12, the area not displayed is indicated by a broken line. The same applies to the drawings used in the following description.

  FIG. 12 is an example of a divided image displayed on the operation panel 6 as information that does not need to be concealed. The display mode by this divided image is as follows. The application program name display field 121 displays “new memo” because it is not information that should be kept secret. Since the information to be concealed is displayed in the character input field 122, the input image is not included in the divided image displayed on the operation panel 6, and only the frame line is displayed. In FIG. 12, the frame line is indicated by a broken line because it is a non-display portion. Since the virtual keyboard 123 is information for the user to operate the operation panel 6, a divided image is configured to be displayed on the operation panel 6. Here, not all the character input keys of the virtual keyboard 123 are displayed on the operation panel 6, but a portion overlapping with the character input keys displayed on the HMD 9 may be hidden. Further, when a character input key that is a selection candidate is displayed in a field that is not displayed on the operation panel 6 when one of the character input keys of the virtual keyboard 123 is selected by the user, for example, “A” When “” is selected, “U” is displayed in the input field 122 that is not displayed on the operation panel 6. In this case, the character input key “U” applied to the input field 122 is displayed on the HMD 9 and the virtual keyboard. The character input keys “A”, “I”, “E”, and “O” displayed in the display area 123 may be displayed on the operation panel 6, respectively. Similarly, a divided image may be formed in the mail application.

  FIG. 13 is a display example in the mail application displayed on the operation panel 6 of the general information display device 10. As GUI parts constituting the displayed information, an application program name display field 131, an input field 132 for information such as a destination, a subject, and a body, an operation button 133 for inputting instructions such as sending, storing, and discarding mail are input. It is a display screen comprising a conversion candidate display field 134 for displaying a character conversion candidate and a virtual keyboard 135 for inputting a character. The user can input characters by operating the virtual keyboard 135 displayed on the operation panel 6, or can instruct mail processing by operating the operation buttons 133. In the present embodiment, the image dividing unit 12 generates a divided image composed of information to be concealed and displays it on the HMD 9, generates a divided image composed of information that does not need to be concealed and displays it on the operation panel 6.

  FIG. 14 is an example of a divided image displayed on the HMD 9 as information to be kept secret. The display mode by this divided image is as follows. In the character input field 142, since the result of the character input by the user using the virtual keyboard 145 is displayed, it is displayed on the HMD 9 because the information is not desired to be viewed by a third party. The conversion candidate display field 144 indicating the conversion candidates for the character string input from the virtual keyboard 145 is a display for the user to select and input from the displayed candidate character strings, and information to be concealed. Therefore, it is displayed on the HMD 9. The application program name display field 141, the operation buttons 143, and the virtual keyboard 145 other than these may be hidden, or the divided image may be configured so that only the frame line is displayed on the HMD 9.

  FIG. 15 is an example of a divided image displayed on the operation panel 6 as information that does not need to be concealed. The display mode by this divided image is as follows. The application program name display field 151 displays “Create Mail” because it is not information that should be kept secret. In the character input field 152, information to be concealed is displayed for the input destination and the mail text information, so the divided image displayed on the operation panel 6 does not include the input content and only the frame line is displayed. I am doing so. Further, since the selected conversion candidate character string is displayed in the conversion candidate display field 154, it is information to be kept secret and is not displayed. The operation buttons 153 and the virtual keyboard 155 for inputting instructions such as sending, storing, and discarding mail are information for the user to operate the operation panel 6, and configure a divided image to be displayed on the operation panel 6. .

  In the present embodiment, the information display device according to the present invention is described as an example applied to a mobile terminal such as a smart phone or a tablet terminal. However, the present invention is not necessarily limited by the presence or absence of portability. Any computer that can display information may be realized by a desktop personal computer (PC) or the like.

  Alternatively, a plurality of HMDs 9 may be connected to the information display device 10 and the same divided image may be displayed on the plurality of HMDs 9 so that a plurality of users can view the same business document at the same time.

Embodiment 2. FIG.
FIG. 16 is a block diagram showing an embodiment of the information display system in the present embodiment. Note that the same components as those in the first embodiment are denoted by the same reference numerals, and description thereof is omitted as appropriate. The information display system in the present embodiment has a configuration in which a camera image acquisition unit 16, a reference information acquisition unit 17, an image correction unit 18, and a reference information storage unit 19 are added to the configuration shown in the first embodiment. Yes. The information display device 10 used in the present embodiment is assumed to be a computer such as a smart phone or a tablet terminal equipped with a camera as a photographing means. An example of the information display device 10 is shown in FIG. 17, and the lens 8 of the camera in the present embodiment is the same surface as the operation panel 6 for displaying the first image, in other words, the surface facing the HMD 9 worn by the user. It is arranged.

  The camera image acquisition unit 16 acquires imaging data (hereinafter also referred to as “camera image”) by the camera. The reference information acquisition unit 17 is provided as an acquisition unit, acquires reference information about the HMD 9 from the image data of the HMD 9 taken by the camera when the HMD 9 is at the reference position with respect to the information display device 10, and a reference information storage unit 19 is registered. The image correction unit 18 is provided as a correction unit, and causes the camera to photograph the user-mounted HMD 9 viewing the operation panel 6 when a business document image is displayed, and the image data and reference information storage unit obtained by the photographing. Based on the reference information stored in 19, the business document image is formed by the first image and the second image that are displayed separately on the operation panel 6 of the information display device 10 and the transmissive display screen of the HMD 9. The second image is corrected. The display control unit 14 in the present embodiment performs control so that the second image corrected by the image correction unit 18 is displayed on the HMD 9.

  The information display device 10 may have the same hardware configuration as that of the first embodiment except that a camera is required. The components 16 to 18 in the information display device 10 are realized by a cooperative operation of a computer that forms the information display device 10 and a program that operates on the CPU 1 mounted on the computer. The reference information storage unit 19 is realized by the HDD 4, but may be realized by the RAM 3.

  By the way, the image dividing unit 12 forms divided images to be displayed on the information display device 10 and the HMD 9 respectively. In this embodiment as well, a case where the business document image to be displayed is divided into a first image for the information display device 10 and a second image for the HMD 9 will be described as an example. The divided image formed by the image dividing unit 12 simply divides one image. Therefore, in the first embodiment, the user connects the operation panel 6 so that the black portion shown in FIG. 5 matches the displayed character portion shown in FIG. 4, that is, the first image and the second image are combined. In the above description, the business document shown in FIG. However, in practice, for the user, fine adjustment of the image positions displayed on the HMD 9 and the information display device 10 may be troublesome or difficult. In addition, it may be easier for the user to browse the business document by looking at the top or slightly tilting his / her face, for example, depending on the habit when reading the business document from the front. For this reason, even if the divided images are simply displayed, there is a possibility that the boundary between the first image and the second image is slightly shifted even if the alignment is performed.

  Therefore, in the present embodiment, image processing for correcting the second image according to the positional relationship between the HMD 9 and the information display device 10 (the positional relationship between the first image and the second image) is different from that of the first embodiment. By applying the second image, the second image matches the first image and is displayed on the HMD 9.

  18 is a perspective view of the HMD 9 used in the present embodiment. FIG. 19A is a schematic plan view when the HMD 9 is viewed from above, and FIG. 19B is a view of the HMD 9 viewed from the side. FIG. The HMD 9 in the present embodiment is provided with a reference position member (hereinafter referred to as “specific mark”) 91 for making it possible to know the position of the HMD 9 taken by the camera when analyzing the camera image. . In the present embodiment, it is arranged in the middle of both eyes (the center of the glasses) so as to simplify the calculation described later, but it is not necessary to limit to this. In the present embodiment, the specific mark 91 has a spherical shape so that there is no distortion on the camera image even when the user views the operation panel 6 of the information display apparatus 10 from a position shifted vertically and horizontally. It is not necessary to limit to this spherical shape as long as the shape is difficult to occur. Further, although only shown in FIG. 19A, width indicating members 02 a for indicating the width of the HMD 9 may be disposed at the left and right ends of the HMD 9. Further, although only shown in FIG. 19B, a height indicating member 92 b for indicating the height of the HMD 9 may be disposed on each of the upper and lower ends of the HMD 9.

  In the present embodiment, reference information needs to be prepared before the document image display process is executed. Therefore, the reference information acquisition process performed by the reference information acquisition unit 17 will be described.

  FIG. 20 is a diagram showing the relationship between the specific mark 91 provided on the HMD 9, the camera lens 8 of the information display device 10, and the camera image 20. The specific mark 91 is provided on the front surface of the HMD 9 facing the operation panel 6 when the user wearing the HMD 9 views the operation panel 6 of the information display device 10. First, the HMD 9 is disposed at a location where the center of the specific mark 91 is separated from the center of the lens 8 by a reference distance (for example, 1000 mm) from the information display device 10, and the reference information acquisition unit 17 has the captured image 22 of the specific mark 91. The camera is caused to shoot so as to be in the center of the camera image 20. Then, the reference information acquisition unit 17 registers data relating to the image of the specific mark 91 in the camera image 20, for example, the diameter “a” in the reference information storage unit 19 as reference information related to the HMD 9 from the camera image 20. In the present embodiment, the diameter is used as the reference information. However, as long as the data is related to the image of the specific mark 91 in the camera image 20, the number of pixels, the radius, the circumference, the area, and the like in the camera image 20 may be used. Furthermore, the image size of the second image displayed on the HMD 9 is also registered in the reference information storage unit 19.

  Further, as described later, since it is necessary to obtain the width (in fact, the length from the specific mark 91 to one end) and the height of the HMD 9 from the camera image 20, the entire HMD 9 is photographed and the camera image is captured. It is desirable to obtain the width and height of the HMD 9 at the reference position 20 and register it in the reference information storage unit 19 as reference information.

  Next, display processing for displaying a document image on the information display system according to the present embodiment will be described with reference to the flowchart shown in FIG. The present embodiment features an image correction process (step 210) for correcting the second image (image for display on the HMD 9) formed by the division in the image division process (step 130) in the first embodiment. Other processing may be the same as in the first embodiment. Therefore, the characteristic second image correction processing in the present embodiment will be described with reference to the flowchart shown in FIG. 22, and description of other processing will be omitted as appropriate.

  When the divided image is displayed in step 130 and the business document image is displayed in step 140, the processing control unit 15 activates the camera image acquisition unit 16 and photographs the HMD 9 mounted on the user viewing the operation panel 6 before that. Then, a camera image is acquired (step 211). An example of the acquired camera image is shown in FIG. As shown in FIG. 23A, the camera image 20 includes an image of the specific mark 91 (hereinafter “specific mark image”) 24. As in this example, the specific mark image 24 is not necessarily photographed at the center point 23 of the camera image 20, that is, the user places the operation panel 6 of the information display device 10 in front of the face. The information display device 10 is often located at a position shifted vertically and horizontally. Further, it is considered that there is an individual difference in the distance between the information display device 10 and the HMD 9. Therefore, it is necessary to adjust (enlarge / reduce) the image size which varies depending on the vertical / horizontal shift or the distance between the information display device 10 and the HMD 9. Specifically, the image correction unit 18 obtains a deviation amount necessary for correction as follows.

  FIG. 23B is a diagram in which auxiliary lines and the like are added to the camera image 20 illustrated in FIG. The image correcting unit 18 analyzes the camera image 20 to obtain the angle D of the specific mark image 24 and the distance D from the center point 23 in advance as a left-handed north reference with the north (upward) direction set to 0 °.

  FIG. 24 is a diagram illustrating a vertical positional relationship between the specific mark image 24 and the specific mark 91 on the camera image 20. First, the image correction unit 18 has the center of the lens 8 when the positional relationship is a linear distance A from the information display device 10 (strictly, the center of the lens 8) to the HMD 9 (strictly, the center of the specific mark 91). A vertical shift amount (distance Cv) from (the center point 23 of the camera image 20) to the center of the specific mark 91 is obtained. First, the diameter a ′ of the specific mark image 24 in the camera image 20 is obtained by analyzing the camera image 20, and the diameter a ′ decreases as the distance from the information display device 10 to the HMD 9 increases. In other words, it is inversely proportional to the distance. The straight line distance A = 1000 × a / a ′ can be calculated. Here, a is the diameter of the specific mark image 24 on the camera image 20 measured at the reference distance. Since the vertical distance cv from the center point 23 to the specific mark image 24 in the camera image 20 is a known value of Dsinγ, the vertical distance Cv = cv × a / a ′ can be calculated. Similarly, a horizontal shift amount (distance Ch) from the center of the lens 8 (the center point 23 of the camera image 20) to the center of the specific mark 91 is obtained. Since the horizontal distance ch from the center point 23 to the specific mark image 24 in the camera image 20 is a known value in Dcosγ, it can be calculated as a vertical distance Ch = ch × a / a ′. As described above, the image correction unit 18 calculates the vertical and horizontal shift amounts Cv and Ch of the specific mark image 24 with respect to the center point 23 (the center of the specific mark image in the reference image) of the camera image 20 (step 212). ).

  Subsequently, the image correction unit 18 obtains the inclination of the HMD 9 with respect to the information display device 10. FIG. 25 is a diagram illustrating the horizontal inclination of the HMD 9 with respect to the information display device 10. In FIG. 25, the length α is the length from the position of the specific mark 91 of the HMD 9 that is a distance A away from the information display device 10 (strictly, further separated from the radius of the specific mark 91) to the left end. The length β is the length of the spectacle portion calculated from a camera image obtained by capturing the spectacle portion having the length α. The length α is obtained in advance as reference information. Assuming that the horizontal tilt angle of the HMD 9 is represented by θ, β = α cos θ holds, and therefore the angle θ is calculated from this equation.

  FIG. 26 is a diagram illustrating the vertical inclination of the HMD 9 with respect to the information display device 10. In FIG. 26, the length η is the length from the position of the specific mark 91 to the lower end of the HMD 9 that is a distance A away from the information display device 10 (strictly, the radius of the specific mark 91 is further separated). The length ζ is the length of the spectacle portion calculated from a camera image obtained by capturing the spectacle portion having the length η. The length η is obtained in advance as reference information. Assuming that the inclination angle of the HMD 9 in the vertical direction is represented by δ, ζ = ηcos δ holds, and therefore the angle δ is calculated from this equation. The image correction unit 18 calculates the vertical and horizontal inclinations of the HMD 9 with respect to the information display device 10 as described above (step 213). Note that either step 212 or step 213 may be performed first.

  Subsequently, the image correction unit 18 has the diameter a of the captured image 22 of the specific mark 91 obtained at the reference distance, the diameter a ′ of the specific mark image 24 in the camera image 20 obtained at the timing of displaying the business document image, and the above-mentioned. Correction is performed so as to adjust the second image displayed on the HMD 9 using the deviation amounts Cv and Ch and the inclination angles θ and δ calculated in the processing (step 214).

  First, the image correction unit 18 enlarges / reduces the second image based on the distance A. Specifically, when the image size of the second image at the reference distance is S, the corrected image size S ′ = S × a ′ / a is calculated. That is, when the HMD 9 is away from the reference distance from the information display device 10, the image size of the second image is reduced and displayed on the HMD 9. Based on the relationship between the size of the specific mark 91 included in the camera image 20 obtained at the timing of displaying the business document image and the size of the specific mark 91 obtained from the reference information, the image correcting unit 18 By adjusting the size, the image size difference between the divided image (first image) displayed on the information display device 10 and the divided image (second image) displayed on the HMD 9 is corrected.

  Further, the image correction unit 18 shifts the display position of the second image displayed on the display screen of the HMD 9 from the standard position for display in the vertical direction Cv and the horizontal direction Ch. In the example of FIG. 23, the fact that the specific mark image 24 is shifted to the upper right from the center point 23 means that the user moves the information display device 10 at the lower left position with respect to the reference position shown in FIG. Therefore, the display position of the second image on the HMD 9 is shifted to the lower left and displayed. The image correcting unit 18 in this way, based on the relationship between the position of the specific mark 91 included in the camera image 20 obtained at the timing of displaying the business document image and the position of the specific mark 91 obtained from the reference information. The positional deviation between the first image and the second image is corrected by adjusting the display position in the HMD 9.

  Furthermore, the image correction unit 18 corrects the second image displayed on the display screen of the HMD 9 so as to be distorted. Specifically, since the second image generated by the image dividing unit 12 is an image when viewed from the front, the second image is corrected by an angle θ in the horizontal direction from the state when viewed from the front. To do. Similarly, by correcting the second image so as to be distorted by the angle δ in the vertical direction, a second image in a state of being viewed from the front is formed. The image correction unit 18 performs the first processing based on the relationship between the size (width and height) of the HMD 9 included in the camera image 20 obtained at the timing of displaying the business document image and the size of the HMD 9 obtained from the reference information. The inclination of the second image with respect to the first image is corrected by distorting the two images.

  When the second image is corrected as described above, the display control unit 14 uses the operation panel divided image (first image) as the operation panel 6 and the HMD divided image corrected by the image correction unit 18. Each (second image) is displayed on the HMD 9 (step 140).

  In the present embodiment, since the second image displayed on the HMD 9 is corrected according to the positional relationship between the information display device 10 and the HMD 9 as described above, the information display device when reading a business document is performed. The business document is displayed in a state in which the shift between the first image displayed on the information display device 10 and the second image displayed on the HMD 9 has disappeared regardless of the position of the hand holding 10. Become.

  1 CPU, 2 ROM, 3 RAM, 4 hard disk drive (HDD), 5 network controller, 6 operation panel, 7 internal bus, 9 head mounted display (HMD), 10 information display device, 11 image acquisition unit, 12 image division unit , 13 HMD connection unit, 14 display control unit, 15 processing control unit, 16 camera image acquisition unit, 17 reference information acquisition unit, 18 image correction unit, 19 reference information storage unit.

Claims (9)

Display means;
Photographing means;
When the user wearing the wearable display device looks at the display means, the user has a reference position member provided on the front surface facing the display means. Connecting means for connecting a wearable display device mounted with the transmissive display screen, formed to be located in front of
Dividing means for dividing the display image into a first image and a second image composed of a portion other than the first image of the display image;
Obtaining means for acquiring reference information about the wearable display device from imaging data of the wearable display device imaged by the photographing means when the wearable display device is at a reference position with respect to the information terminal device;
Based on the imaging data and the reference information acquired by photographing the wearable display device worn by the user who views the display means when the display image is displayed, the display means and the transmissive display Correction means for correcting the second image so that the display image is formed by the first image and the second image displayed separately on each screen;
Display control means for controlling to display the first image on the display means and to display the second image corrected by the correction means on the transmissive display screen;
An information display device comprising:   When the display image includes confidential information, the dividing means prevents the confidential information from being read only by the first image, or the confidential information is included only in the second image. The information display device according to claim 1, wherein the information display device is divided into two images.   The correcting means is based on the division by the dividing means based on the relationship between the size of the reference position member included in the imaging data obtained by photographing by the photographing means and the size of the reference position member obtained from the reference information. The information display device according to claim 1, wherein a shift in image size between the first image and the second image is corrected by adjusting a size of the generated second image.   The correcting means is based on the division by the dividing means based on the relationship between the position of the reference position member included in the imaging data obtained by photographing by the photographing means and the position of the reference position member obtained from the reference information. 2. The information display according to claim 1, wherein a positional shift between the first image and the second image is corrected by adjusting a display position of the generated second image on the transmissive display screen. apparatus.   The correcting means is based on the relationship between the size of the wearable display device included in the imaging data obtained by photographing by the photographing means and the size of the wearable display device obtained from the reference information. The information display apparatus according to claim 1, wherein the second image generated by the division is distorted to correct an inclination of the second image with respect to the first image. A wearable display device formed so that the mounted transmissive display screen is positioned in front of the user's eyes when worn by the user;
An information terminal device having photographing means and display means;
Have
The wearable display device has a reference position member provided on the front surface facing the display means when a user wearing the wearable display device looks at the display means,
The information terminal device
Connection means for connecting the wearable display device;
Dividing means for dividing the display image into a first image and a second image composed of a portion other than the first image of the display image;
Obtaining means for acquiring reference information about the wearable display device from imaging data of the wearable display device imaged by the photographing means when the wearable display device is at a reference position with respect to the information terminal device;
Based on the imaging data and the reference information acquired by photographing the wearable display device worn by the user who views the display means when the display image is displayed, the display means and the transmissive display Correction means for correcting the second image so that the display image is formed by the first image and the second image displayed separately on each screen;
Display control means for controlling to display the first image on the display means and to display the second image corrected by the correction means on the transmissive display screen;
An information display system comprising:   7. The information display system according to claim 6, wherein a height indicating member for indicating the height of the wearable display device is provided at each of upper and lower ends of the wearable display device.   The information display system according to claim 6, wherein a width indicating member for indicating a width of the wearable display device is disposed on each of left and right ends of the wearable display device. A computer equipped with shooting means
Display means,
When the user wearing the wearable display device looks at the display means, the user has a reference position member provided on the front surface facing the display means. Connecting means for connecting a wearable display device mounted with the transmissive display screen, formed to be positioned in front of
A dividing unit that divides the display image into a first image and a second image including a portion other than the first image of the display image;
Acquisition means for acquiring reference information about the wearable display device from imaging data of the wearable display device taken by the photographing means when the wearable display device is at a reference position with respect to the information terminal device;
Based on the imaging data and the reference information acquired by photographing the wearable display device worn by the user who views the display means when the display image is displayed, the display means and the transmissive display Correction means for correcting the second image so that the display image is formed by the first image and the second image displayed separately on each screen;
Display control means for controlling the display of the first image on the display means and the display of the second image corrected by the correction means on the transmissive display screen;
Program to function as.

Images