WO2023248338A1 - ウェアラブル端末、連係表示システム及び連係表示方法 - Google Patents

ウェアラブル端末、連係表示システム及び連係表示方法 Download PDF

Info

Publication number
WO2023248338A1
WO2023248338A1 PCT/JP2022/024683 JP2022024683W WO2023248338A1 WO 2023248338 A1 WO2023248338 A1 WO 2023248338A1 JP 2022024683 W JP2022024683 W JP 2022024683W WO 2023248338 A1 WO2023248338 A1 WO 2023248338A1
Authority
WO
WIPO (PCT)
Prior art keywords
wearable terminal
display
information
wearable
camera
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2022/024683
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
宏司 中森
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Maxell Ltd
Original Assignee
Maxell Ltd
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Maxell Ltd filed Critical Maxell Ltd
Priority to JP2024528143A priority Critical patent/JP7717279B2/ja
Priority to PCT/JP2022/024683 priority patent/WO2023248338A1/ja
Priority to US18/876,693 priority patent/US20250371201A1/en
Priority to CN202280097004.9A priority patent/CN119365846A/zh
Publication of WO2023248338A1 publication Critical patent/WO2023248338A1/ja
Anticipated expiration legal-status Critical
Priority to JP2025121822A priority patent/JP2025142227A/ja
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/70—Protecting specific internal or peripheral components, in which the protection of a component leads to protection of the entire computer
    • G06F21/82—Protecting input, output or interconnection devices
    • G06F21/84—Protecting input, output or interconnection devices output devices, e.g. displays or monitors
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F21/00—Security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F21/30—Authentication, i.e. establishing the identity or authorisation of security principals
    • G06F21/31—User authentication
    • G06F21/34—User authentication involving the use of external additional devices, e.g. dongles or smart cards
    • G06F21/35—User authentication involving the use of external additional devices, e.g. dongles or smart cards communicating wirelessly
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0481—Interaction techniques based on graphical user interfaces [GUI] based on specific properties of the displayed interaction object or a metaphor-based environment, e.g. interaction with desktop elements like windows or icons, or assisted by a cursor's changing behaviour or appearance
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06V—IMAGE OR VIDEO RECOGNITION OR UNDERSTANDING
    • G06V20/00—Scenes; Scene-specific elements
    • G06V20/50—Context or environment of the image
    • G06V20/52—Surveillance or monitoring of activities, e.g. for recognising suspicious objects
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F2221/00—Indexing scheme relating to security arrangements for protecting computers, components thereof, programs or data against unauthorised activity
    • G06F2221/03—Indexing scheme relating to G06F21/50, monitoring users, programs or devices to maintain the integrity of platforms
    • G06F2221/032—Protect output to user by software means

Definitions

  • the present invention relates to a wearable terminal, a linked display system, and a linked display method, and particularly relates to a display technology in which a wearable display terminal and a non-wearable display terminal are linked.
  • Patent Document 1 states that ⁇ a first device coupled to a first display and an image sensor has a second display different from the first display. receiving output data from the device; the output data representing content displayable by the second device on a second display; the first device using the image sensor to determining a position of a second display and displaying content on the first display based on output data received from the second device and the determined position of the second display relative to the first device. (excerpt)".
  • the display screen of a non-wearable terminal such as a desktop computer, notebook computer, or tablet can be viewed by anyone other than the user of the non-wearable terminal.
  • a non-wearable terminal such as a desktop computer, notebook computer, or tablet
  • Patent Document 1 it is possible to use the display technology of Patent Document 1, but in Patent Document 1, if the user makes an arbitrary selection, or if it is determined that the user is looking at a PC or mobile terminal using an HMD, The screen image is simply displayed on the HMD screen as an alternative to match the screen position of the PC or mobile device, and information leakage is prevented depending on the location and surrounding conditions where the user uses the non-wearable device such as a PC. The point of view of doing so has not been taken into consideration. Therefore, security cannot be maintained simply by displaying the output data of the first device on the second device.
  • the present invention has been made in view of the above points, and aims to improve the security of information by preventing others from peeking at the screen.
  • the present invention includes the configurations described in the claims.
  • FIG. 1 is a schematic configuration diagram of an alternative display system according to the present embodiment.
  • FIG. 2 is a schematic configuration diagram of a non-wearable terminal.
  • FIG. 3 is an external view of a wearable terminal, in which (a) shows an HMD and (b) shows AR glasses.
  • FIG. 2 is a functional block diagram of an alternative display system.
  • FIG. 7 is a diagram illustrating the field of view of another person and the user during alternative display when a notebook computer is used as a non-wearable terminal.
  • FIG. 7 is a diagram illustrating the field of view of another person and the user during alternative display when a desktop personal computer is used as a non-wearable terminal.
  • FIG. 1 is a schematic configuration diagram of an alternative display system according to the present embodiment.
  • FIG. 2 is a schematic configuration diagram of a non-wearable terminal.
  • FIG. 3 is an external view of a wearable terminal, in which (a) shows an HMD and (b) shows AR glasses.
  • FIG. 2 is a
  • FIG. 7 is a diagram illustrating the field of view of another person and the user during alternative display when a mobile terminal is used as a non-wearable terminal.
  • FIG. 4 is a diagram illustrating an example of alternative display control based on location information and information regarding the surrounding environment as information regarding the location, where (a) is a non-private office space, (b) is your own room, and (c) is an office. Here is an example of its usage.
  • FIG. 7 is a diagram illustrating an example of alternative display control based on positional information of a place and information about the surrounding environment as information about the place, and an example of alternative display control depending on the presence or absence of an obstruction.
  • FIG. 6 is a diagram illustrating an example of alternative display control according to the depth of a shielding object, where (a) shows the user's field of view when the shielding object does not extend to the front;
  • FIG. 4 is a diagram illustrating an example of alternative display control in a space where a surveillance camera is installed as an information leakage medium, in which (a) is an example in which alternative display is performed, and (b) is an example in which alternative display is not performed.
  • 12 is a diagram showing a situation where no surveillance camera is installed in the same space as FIG. 11.
  • FIG. 6 is a diagram illustrating an example of alternative display control in which someone other than the user is used as the information leakage medium.
  • FIG. 4 is a diagram illustrating an example of alternative display control when another person is located further away than the first distance; (a) is an example in which alternative display is performed, and (b) is an example in which alternative display is not performed.
  • FIG. 7 is a diagram illustrating an example where another person is located further away than the first distance and alternative display is not performed.
  • An example of a user setting screen for the alternative display function is shown. This is a user setting screen for detailed alternative display functions. This is the affiliation information registration setting screen. It is a figure which shows the example of alternative display control rule information.
  • 2 is a flowchart (first half) showing the flow of processing of the alternative display system according to the present embodiment.
  • 2 is a flowchart (second half) showing the flow of processing of the alternative display system according to the present embodiment.
  • 7 is a flowchart showing a process flow of alternative display control based on location information and information regarding the surrounding environment as information regarding the location.
  • 7 is a flowchart illustrating a process flow of alternative display control based on a surveillance camera as an information leakage medium.
  • 12 is a flowchart illustrating a process flow of alternative display control based on a person other than the user as the information leakage medium.
  • 3 is a flowchart illustrating a process flow of alternative display control based on a person's affiliation.
  • 7 is a flowchart illustrating a process flow of alternative display control based on a person's affiliation.
  • FIG. 1A is a schematic configuration diagram of an alternative display system according to this embodiment.
  • FIG. 1B is a schematic configuration diagram of a non-wearable terminal.
  • FIG. 2 is an external view of the wearable terminal, in which (a) shows an HMD and (b) shows AR glasses.
  • the alternative display system 10 shown in FIG. 1A is configured by wirelessly connecting a wearable terminal 1 and a non-wearable terminal 2.
  • the wireless communication connection may be made by directly connecting the wearable terminal 1 and the non-wearable terminal 2 using short-range wireless communication such as Bluetooth (registered trademark), or by indirectly connecting the wearable terminal 1 and the non-wearable terminal 2 via a LAN. good.
  • the wearable terminal 1 used in the alternative display system 10 may be any terminal whose screen can be viewed by only the user wearing the wearable terminal 1, and specific examples include the HMD 100 shown in FIG. There is an AR glass 100s shown in b).
  • the wearable terminal 1 includes a processor 101 (corresponding to a first processor), a memory 102, a storage 103, an operation input interface 104, a camera 106, a positioning sensor 107, a distance sensor 108, an acceleration sensor 109, and a gyro sensor. 110, a geomagnetic sensor 111, a display 112 (corresponding to a first display), a microphone 113, a speaker 114, a timer 115, a network communication device 116, and a BT communication device 117 (BT: Bluetooth (registered trademark)). 118 to each other.
  • BT Bluetooth
  • the storage 103 stores an alternative display program 120, and the processor 101 develops and stores this alternative display program 120 in the memory 102 and executes it.
  • the storage 103 stores data 130 generated in the process and result of executing the alternative display program 120.
  • the data 130 includes location information 131, information leakage medium information 132, and alternative display control information 133.
  • the location information 131, the information leakage medium information 132, and the alternative display control information 133 are respectively set on a setting screen 400 (see FIG. 16), a user setting screen 410 (FIG. 17(a)), and an affiliation information registration setting screen 420 (described later).
  • FIG. 17(b)) the affiliation information details screen 430 (FIG. 18) is input.
  • the distance sensor 108 is a sensor that outputs distance information to a real object to be measured.
  • it may be a sensor that calculates a distance using an image with parallax, such as a stereo camera, to a feature point such as a contour of a real object and outputs distance information.
  • the distance may be measured by irradiating a measurement target with a light beam two-dimensionally, such as a TOF (Time of Flight) method.
  • TOF Time of Flight
  • the network communication device 116 is a communication device that realizes a communication function of wireless communication using the Wi-Fi (registered trademark) standard or the like.
  • the network communication device 116 may be communicatively connected to the server 4 via a LAN or external network 5.
  • the update program may be downloaded from the server 4.
  • the BT communication device 117 is a communication device using Bluetooth (registered trademark), and is a non-wearable terminal 2 that performs alternative display processing, such as a personal computer (including a notebook computer and a desktop computer), a mobile terminal (including a smartphone and a tablet). ) to perform short-range wireless communication.
  • a personal computer including a notebook computer and a desktop computer
  • a mobile terminal including a smartphone and a tablet.
  • the display 112 is a transmissive display.
  • the user views the virtual reality object displayed on the display 112 superimposed on the real world viewed through the display 112 .
  • the non-wearable terminal 2 is a terminal whose screen can be viewed by someone other than the user of the wearable terminal 1, and includes, for example, a desktop computer, a notebook computer, a smartphone, or a tablet connected to a monitor.
  • the non-wearable terminal 2 includes a processor 201 (corresponding to a second processor), a memory 202, a storage 203, an operation input interface 204, a camera 206, a display 212 (corresponding to a second display), and a network. It includes a communication device 216 and a BT communication device 217, which are connected to each other via a bus 218.
  • the non-wearable terminal 2 When a smartphone or tablet is used as the non-wearable terminal 2, it has a ranging sensor, a positioning sensor, etc. in addition to the above configuration, but the non-wearable terminal 2 used in the alternative display system 10 has a ranging sensor, a positioning sensor, etc. It is omitted in FIG. 1B because it is not necessarily necessary to include.
  • the non-display switching program 140 stored in the storage 203 changes the display data that is currently being displayed or will be displayed on the display 212.
  • the display data is hidden and the display data is sent to the wearable terminal 1.
  • the alternative display program 120 is installed in the wearable terminal 1, and the wearable terminal 1 determines whether or not to execute alternative display control.
  • the non-display switching program 140 installed in the non-wearable terminal 2 mainly executes display control to hide the display 212 of the non-wearable terminal 2 in response to a control signal from the wearable terminal 1.
  • FIG. 3 is a functional block diagram of the alternative display system.
  • the alternative display program 120 includes an image recognition section 121, an obstructing object determination section 122, a non-display area determination section 123, an information leakage medium determination section 124, an alternative display control section 125, a display data reception section 126, and a virtual reality object display section 127. include.
  • the image recognition unit 121 analyzes images captured by the cameras 106 and 206 mounted on the wearable terminal 1 or the non-wearable terminal 2, and performs object recognition processing.
  • the shielding object determination unit 122 determines the height, placement position, non-transparency, etc. of the shielding object when the shielding object is imaged as a subject.
  • the non-display area determining unit 123 determines a non-display area according to the obstructing object based on the determination result by the obstructing object determining unit 122.
  • the information leakage medium determining unit 124 determines whether the subject corresponds to an information leakage medium.
  • the alternative display control unit 125 determines the start and end of alternative display, and the necessity of alternative display control based on position information and image information.
  • the display data receiving unit 126 receives display data for performing an alternative display from the non-wearable terminal 2.
  • the virtual reality object display unit 127 detects the position of the display 212 of the non-wearable terminal 2 based on the image captured by the camera 106 of the wearable terminal 1, and displays the display data on the wearable terminal 1 so as to overlap the display 212 of the non-wearable terminal 2. is displayed on the display 112 of. Thereby, the display data is displayed on the display 112 as a virtual reality object.
  • the non-display switching program 140 includes a display switching section 141 and a display data transmitting section 142.
  • the display switching unit 141 switches the display 212 between non-display and display based on a control signal from the wearable terminal 1 instructing the start and end of alternative display.
  • the display data transmitting unit 142 transmits display data for performing an alternative display to the wearable terminal 1.
  • FIG. 4 is a diagram showing the field of view of another person and the user during alternative display using a notebook computer.
  • FIG. 5 is a diagram showing the field of view of another person and the user during alternative display using a desktop computer.
  • FIG. 6 is a diagram showing the field of view of another person and the user during alternative display using a mobile terminal.
  • the display 212A of the notebook computer 2A is hidden, and the information that the user originally wants to see is not displayed. Therefore, the display 212 of the notebook computer 2A is a non-display screen in the field of view seen by others.
  • Notebook computer 2A transmits display data 300 that was supposed to be displayed on display 212A to wearable terminal 1.
  • the wearable terminal 1 detects the position of the display 212A, and displays the display data 300 on the display 112 of the wearable terminal 1 so as to overlap the position where the display 212A can be seen through the display 112. Therefore, in the view of the user wearing the wearable terminal 1, the display data 300 appears to be displayed on the display 212A of the notebook computer 2A.
  • FIG. 5 shows a state in which a desktop personal computer (not shown) and the wearable terminal 1 are connected for communication, and the display 212B connected to the desktop personal computer is displayed on the wearable terminal 1 as an alternative.
  • FIG. 6 shows a state in which the mobile terminal 2C and the wearable terminal 1 are connected for communication, and the display 212C of the mobile terminal 2C is displayed on the wearable terminal 1 instead.
  • the display data 300 displayed on the display 112 of the wearable terminal 1 is superimposed on the display 212 of the non-wearable terminal 2 in the real world viewed through the display 112 of the wearable terminal 1.
  • the user visually confirms the information.
  • the display 212 of the non-wearable terminal 2 is hidden, it appears to the user that the display data 300 is being displayed on the display 212 of the non-wearable terminal 2, but the display data 300 is not visible to others. .
  • FIG. 7 is a diagram showing an example of alternative display control based on location information and information regarding the surrounding environment as information regarding the location, where (a) is a non-private office space, (b) is your own room, (c) shows an example of use in an office.
  • FIG. 7(a) is an example of use without prior registration, such as in a non-private office space. In this case, an alternative display is performed.
  • Figure 7 (b) is an example of use in your own room (pre-registered)
  • Figure 7 (c) is an example of use in your office (pre-registered).
  • Information security is considered to be high because of the obligation of confidentiality. Therefore, images showing the location information and surrounding environment of places with high information security are acquired by the positioning sensor 107 and the camera 106 and registered in advance, and the alternative display function is controlled so as not to operate in the pre-registered places.
  • the alternative display function is controlled to operate in locations that are not pre-registered. Thereby, the alternative display function can be controlled based on location information and images of the pre-registered location and the location where the user is currently located, as location information.
  • FIG. 8 is a modification of alternative display control based on location information and information on the surrounding environment as information on the place, and alternative display control based on the presence or absence of an obstruction for the non-wearable terminal 2 as information on the surrounding environment. It is a figure which shows an example.
  • non-transparent shields 320 such as partitions or walls
  • Fig. 8 2 shows the state of the display 212 of the non-wearable terminal 2
  • the presence or absence of a partition is determined by analyzing the image captured by the camera 106 of the wearable terminal 1 or the camera 206 of the non-wearable terminal 2, and if it is determined that there is a shield 320 that cannot be seen by the display 212, an alternative display is not displayed. may be controlled.
  • FIG. 9 is a diagram showing an example of alternative display control according to the height of a shielding object for the non-wearable terminal 2, where (a) shows the user's field of view when the shielding object is not high enough, and (b) shows the user's field of view.
  • FIG. 10 is a diagram showing an example of alternative display control according to the depth of an obstruction to the non-wearable terminal 2, where (a) shows the case where the obstruction does not reach the front, and (b) shows the user's field of view. .
  • the area where the display 212 is visible from the outside changes depending on the height and depth of the obstruction (in FIGS. 9 and 10, the display 212 of the non-wearable terminal 2 is shown in a ), the information security of the locations differs.
  • the height of the non-transparent part of the shield 320a is low, information security is higher compared to a case where there is no shield, but information security is lower when compared to a case where the shield is sufficiently high.
  • an alternative display of the application screen related to the area where the display 212 of the non-wearable terminal 2 can be seen from the left and right sides may be performed.
  • the height of the shield 320a is lower than the upper end of the display 212 of the non-wearable terminal 2. Therefore, the application screen related to the upper half of the non-wearable terminal 2 is hidden.
  • the user wearing the wearable terminal 1 can see the display data 300 of the non-wearable terminal 2 on the display 212 of the non-wearable terminal 2 as shown in FIG. 9(b).
  • the application screens alternatively displayed in 1 appear to overlap.
  • the application screen 301 is displayed on the display 212 of the non-wearable terminal 2, it is not displayed on the wearable terminal 1.
  • the user views the application screen 301 actually displayed on the display 212 of the non-wearable terminal 2 through the display 112 of the wearable terminal 1 .
  • FIG. 10 is a diagram illustrating an example of alternative display control according to the depth of the shielding object for the non-wearable terminal 2, and the height of the partition as the shielding object 320b is equal to the upper end of the non-wearable terminal 2. Or, although it is taller than that, the length in the depth direction is short, and the partition does not cover the area from the front of the desk to the center in the depth direction. Therefore, since the lower left of the non-wearable terminal 2 can be shielded from the partition, the application screen 301 is displayed without performing an alternative display, but the application screen above and to the right of the half of the display 212 is displayed as an alternative.
  • the user wearing the wearable terminal 1 can see the display data 300 of the non-wearable terminal 2 on the display 212 of the non-wearable terminal 2 as shown in FIG. 10(b).
  • the application screens alternatively displayed in 1 appear to overlap.
  • the application screen 301 is displayed on the display 212 of the non-wearable terminal 2, it is not displayed on the wearable terminal 1.
  • the user views the application screen 301 actually displayed on the display 212 of the non-wearable terminal 2 through the display 112 of the wearable terminal 1 .
  • alternative display control can be performed based on information regarding the transparency and shape of the shielding object for the non-wearable terminal 2 as information regarding the surrounding environment of the place where the user operates the non-wearable terminal 2. Due to the shape, it is possible to control the display 212 of the non-wearable terminal 2 so that only the application screen that spans a partial area that is difficult to cover is displayed as an alternative. If the screen image quality and other performance of the wearable terminal 1 such as an HMD or AR glasses is poorer than that of the non-wearable terminal 2 such as a PC, the alternative display may become a factor that impairs the user-friendliness of the non-wearable terminal 2. There is. Therefore, by limiting the area of the screen where alternative display is performed to only the necessary portion, it is possible to improve information security while suppressing a decrease in the user's usability of the non-wearable terminal 2.
  • FIG. 11 is a diagram showing an example of alternative display control in a space where a surveillance camera is installed, in which (a) is an example in which alternative display is performed, and (b) is an example in which alternative display is not performed.
  • FIG. 12 is a diagram showing a situation where no surveillance camera is installed in the same space as FIG. 11.
  • a surveillance camera 350 is installed in the living room of the user who is operating the non-wearable terminal 2.
  • the surveillance camera 350 is an example of an information leakage medium.
  • the display data displayed on the screen of the display 212 of the non-wearable terminal 2 will be reflected in the image of the surveillance camera 350, resulting in information leakage. Connect.
  • alternative display control may be performed based on the presence or absence of the surveillance camera 350 as information regarding the information leakage medium.
  • a surveillance camera 350 captures an image taken by the camera 206 mounted on the non-wearable terminal 2. Therefore, since there is a possibility that the screen of the display 212 of the non-wearable terminal 2 will be reflected in the image of the surveillance camera 350, an alternative display is performed.
  • the camera 106 of the wearable terminal 1 may take an image. In that case, the relationship between the positions and orientations of the display 212 of the non-wearable terminal 2 and the surveillance camera 350 is analyzed based on the image captured by the camera 106 and the distance measurement result measured by the distance measurement sensor 108.
  • the surveillance camera 350 is captured in the image captured by the camera 106 of the wearable terminal 1, but the surveillance camera 350 is not captured in the captured image by the camera 206 mounted on the non-wearable terminal 2. Therefore, in FIG. 11(b), it can be determined that the orientation (photographing direction) of the surveillance camera 350 is such that the screen of the display 212 cannot be imaged, and therefore no alternative display is performed.
  • Figure 12 shows the same space as Figure 11, but no surveillance cameras are installed. Therefore, when the presence or absence of the surveillance camera 350 is determined based on the image captured by the camera 206 of the non-wearable terminal 2 or the camera 106 of the wearable terminal 1, it is determined that there is no surveillance camera 350, and no alternative display is performed.
  • alternative display control can be performed as information regarding the information leakage medium according to the presence or absence of the surveillance camera 350 existing around the user or the non-wearable terminal 2 and the direction of the lens of the surveillance camera 350 (photographing direction).
  • FIG. 13 is a diagram illustrating an example of alternative display control in which someone other than the user is used as the information leakage medium.
  • the non-wearable terminal 2 measures the distance to the other person.
  • the non-wearable terminal 2 is equipped with a distance measurement sensor, distance measurement information from the distance measurement sensor may be used. If a distance measurement sensor is not installed, the approximate distance may be estimated from the image of the camera 206. In this case, for example, if the ratio between the user's face size and the other person's face size is a predetermined ratio, it is determined that the distance to the user and the other person are approximately equal based on the non-wearable terminal 2. Good too.
  • the camera 106 and distance measurement sensor 108 of the wearable terminal 1 may perform imaging and distance measurement. In that case, the relationship between the position and orientation of the display 212 of the non-wearable terminal 2 and the person 360 other than the user is analyzed based on the image captured by the camera 106 and the distance measurement result measured by the distance measurement sensor 108.
  • the first distance range is approximately equal to the distance between the user and the non-wearable terminal 2.
  • FIG. 14 is a diagram showing an example of alternative display control when the other person 360 is located at a location farther than the first distance. (a) is an example in which alternative display is performed, and (b) is an example in which alternative display is not performed.
  • FIG. 14(a) shows a case where a stranger 360 is located further away than the first distance range, and the face of the stranger 360 is in a position where the display 212 of the non-wearable terminal 2 can be seen. If the image captured by the camera 206 of the non-wearable terminal 2 or the camera 106 of the wearable terminal 1 is analyzed and someone else can recognize the image, an alternative display is performed.
  • FIG. 14(b) shows a case where the other person 360 is located further away than the first distance range and the face of the other person 360 cannot be seen from the display 212 of the non-wearable terminal 2.
  • the display 212 is not visible to the other person 360 because it is in the shadow of the user, and there is a high possibility that the information cannot be read, so no alternative display is performed.
  • FIG. 15 is a diagram illustrating an example in which the other person 360 is located further away than the first distance and no alternative display is performed.
  • the other person 360 is away from the first distance range. Furthermore, since the user's back is facing the non-wearable terminal 2, the data displayed on the display 212 cannot be read. Therefore, no alternative display is performed in this case.
  • alternative display control can be performed based on the presence/absence, position, and orientation of the other person 360 existing around the user or the non-wearable terminal 2 as information regarding the information leak medium.
  • FIG. 16 shows an example of a user setting screen for the alternative display function.
  • the main items are (1) “Non-applied (OFF)” (2) “Always applied (ON)” (3) “Timely application” There is a pattern to apply.
  • (3-2-1) Automatic selection Analyzes the image captured by the camera 206 of the non-wearable terminal 2 or the camera 106 of the wearable terminal 1, and automatically selects the information leak medium for alternative display (Automatic setting: default By default, all registered information leakage media (e.g. surveillance cameras, people) are automatically identified and alternatively displayed based on the registered information leakage media image.) Thereby, it is possible to control the information leakage medium to be automatically identified and alternatively displayed.
  • all registered information leakage media e.g. surveillance cameras, people
  • FIG. 17 is a detailed user setting screen for the alternative display function.
  • FIG. 17(b) When the "affiliation information registration setting" button 411 is pressed, the affiliation information registration setting screen 420 shown in FIG. 17(b) opens. On the affiliation information registration setting screen 420, edit the affiliation information of each person and the device used, and press the "registration setting" button 421 to proceed to the affiliation information details screen 430 in FIG. 18.
  • FIG. 17(b) and FIG. 18 show an example where information regarding "affiliation of person/equipment used" is classified by the same organization, but information regarding "affiliation of person/equipment used” is shown.
  • the method of classification is not limited to this example; for example, it may be classified by person or by device used.
  • FIG. 18 shows the affiliation information registration setting screen.
  • Each item on the affiliation information details screen 430 will be explained below.
  • Affiliation name For Optional affiliation name.
  • Affiliation information Specific affiliation (company (including information on departments, etc.)), family, clubs/circles, etc. In order to perform alternative display control as shown in FIG. 19, which will be described later, it is desirable that affiliation information can be defined in several stages.
  • Applicable person A person who corresponds to this organization (register person identification information such as images and voices that can uniquely identify the person).
  • Equipment used Equipment used by our organization (PCs, mobile terminals, etc.).
  • Specific confidential characters for non-applicable persons ...specific confidential characters ( ⁇ section, ⁇ meeting, etc.
  • the affiliation information details screen 430 is registered after editing the following items: (1) affiliation name, (2) affiliation information, (3) applicable person, (4) equipment used, and (5) specific confidential characters for non-applicable persons. "Registration setting buttons" 431, 432, 433, 434, and 435 are provided for this purpose.
  • (6-1) “Identification of non-applicable persons (user criteria)” It is determined whether the affiliation of a user using a PC or the like is the same as the affiliation of other people around the user, and alternative display control is performed based on the determination result.
  • (6-2) “Identification of non-applicable persons (device standards used)” It is determined whether the organization (affiliation) that manages equipment such as a PC used by the user is the same as the affiliation of other people around the user, and alternative display control is performed based on the determination result.
  • FIG. 19 shows which of the "Affiliation of Person/Equipment Used" on the screen of FIG.
  • FIG. 4 is a diagram showing an example of alternative display control rule information applied when one of the following is selected.
  • FIG. 20 is a flowchart (first half) showing the flow of processing of the alternative display system according to this embodiment
  • FIG. 21 is a flowchart (second half) showing the flow of processing of the alternative display system according to this embodiment.
  • the alternative display control unit 125 checks the setting information (S101).
  • step S108 the process of determining the necessity of alternative display in step S108, which will be described later, is performed in the order set in "priority setting" in FIG. Also get the configuration information. If a priority is not set but a plurality of application conditions are set, settings may be made so that the necessity determination using the application conditions is performed in a predetermined order.
  • the alternative display control unit 125 transmits alternative display start/continuation information to the non-wearable terminal 2 (S102).
  • the alternative display start/continuation information also includes request information for data necessary to determine the application conditions of the alternative display specified in the setting information (see FIG. 16). For example, if “surveillance camera” is included in “applicable target (information leakage medium) automatic identification", an imaging request by the non-wearable terminal 2 is included.
  • the non-wearable terminal 2 receives the alternative display start/continuation information (S103), and based on it, collects information used to determine whether the alternative display is necessary (S104).
  • the non-wearable terminal 2 if it is necessary to take an image using the camera 206 mounted on the non-wearable terminal 2 and to measure the distance to the person who is the information leakage medium, the non-wearable terminal 2 measures the distance to the imaged person (S104). It is transmitted to the wearable terminal 1 (S105).
  • the display 212 of the non-wearable terminal 2 is switched to non-display after sending and receiving the alternative display start/continuation information (S102, S103) (skip to S112).
  • the wearable terminal 1 also collects information used to determine whether an alternative display is necessary (S106).
  • the wearable terminal 1 in addition to imaging, distance measurement, and position information from the positioning sensor 107, the user's voice is collected by the microphone 113 to the extent necessary for determining whether or not an alternative display is necessary.
  • the wearable terminal 1 collects information about the user's location collected by the wearable terminal 1 and the non-wearable terminal 2 (location information detected by the positioning sensor 107, images captured by the cameras 106 and 206), and information about surrounding information leakage media ( (images captured by the cameras 106 and 206), user information (user affiliation information read from the setting information of the wearable terminal 1, audio picked up by the microphone 113, etc.), and device information received from the non-wearable terminal 2 ( S107) and is used for the process of determining whether or not an alternative display is necessary (S108). Details of the process for determining the necessity of alternative display will be described later (S108).
  • the process of determining whether or not an alternative display is necessary includes "alternative display control based on location information and information on the surrounding environment as information regarding the location” ( Figure 22), and “alternative display control based on surveillance cameras as information leakage media” ( Figure 22). ” ( Figure 23), “Alternative display control based on someone other than the user as the information leakage medium” ( Figure 24), “Alternative display control based on the affiliation of the person” ( Figure 25), “Alternative display control based on the affiliation of the person 26). These are processed according to the priority information read in S101.
  • the wearable terminal 1 specifies a screen area for an alternative display, and sends alternative display instruction information indicating the screen area to the non-wearable terminal. 2 (S110).
  • the non-wearable terminal 2 receives the alternative display instruction information (S111), and switches the screen area for alternative display to non-display (S112). Furthermore, the hidden display data 300 is transmitted to the wearable terminal 1 (S113), and the wearable terminal 1 receives it (S114).
  • the wearable terminal 1 displays the received information regarding the alternative display on the display 112 of the wearable terminal 1 (S115). If the alternative display is to be continued (S116: No), the process returns to step S102, and the non-wearable terminal 2 and the wearable terminal 1 collect information used to determine whether or not the alternative display is necessary. Note that if "always applied" is selected, the process returns to step S112.
  • alternative display end information is transmitted from the wearable terminal 1 to the non-wearable terminal 2 (S117), and the non-wearable terminal 2 receives it (S118).
  • the non-wearable terminal 2 returns from non-display to normal display (S119), and also stops transmitting display data to the wearable terminal 1. Then, the alternative display control of the wearable terminal 1 and the non-wearable terminal 2 is ended.
  • FIG. 22 is a flowchart showing the process flow of alternative display control based on location information and information regarding the surrounding environment as information regarding the location.
  • the alternative display control unit 125 acquires the user's position information measured by the positioning sensor 107, an image showing the surrounding environment taken by the camera 206 or 106, and distance information measured by the distance measurement sensor 108, and refers to the setting information. It is determined whether the current position or the surrounding environment is a place where alternative display control is applied (S201). If the determination result is negative (S201: No), the result of the process for determining whether or not the alternative display is necessary is outputted (S205) so as to continue to the process for not performing the alternative display (S109: No).
  • the alternative display control unit 125 refers to the setting information. If there is a condition setting based on an obstruction (S202: Yes), the image recognition unit 121 analyzes the image captured by the camera 106 of the wearable terminal 1 or the camera 206 of the non-wearable terminal 2, detects an obstruction, and determines the obstruction. The unit 122 determines an area of the screen to be hidden according to the obstruction (S203). Then, the process result for determining whether or not the alternative display is necessary is outputted (S205) so as to proceed to the process for executing the alternative display (S109: Yes). In the image analysis described above, the image recognition unit 121 detects the imaging area of the shielding object reflected in the image. Then, the shielding object determination unit 122 determines the area of the screen to be hidden according to the height, non-transparency, and placement position of the shielding object.
  • the entire surface of the display 212 of the non-wearable terminal 2 is determined as a non-display area (S204), and the alternative display control unit 125 performs a process for executing an alternative display (S109). :Yes), the result of the process for determining the necessity of alternative display is output (S205).
  • FIG. 23 is a flowchart showing the process flow of alternative display control based on the surveillance camera as the information leakage medium.
  • the image recognition unit 121 analyzes the image captured by the camera 106 of the wearable terminal 1 or the image captured by the camera 106 of the non-wearable terminal 2 to detect a surveillance camera, and the information leakage medium determination unit 124 detects the location of the surveillance camera. Depending on the orientation, it is determined whether the surveillance camera corresponds to an information leakage medium (S301). If the determination result is negative (S301: No), the result of the process for determining whether or not the alternative display is necessary is outputted (S303) so as to continue to the process for not performing the alternative display (S109: No).
  • the information leakage medium determination unit 124 determines to hide the entire screen of the display 212 of the non-wearable terminal 2 (S302), and the alternative display control unit 125 executes the alternative display.
  • the result of the process for determining whether or not an alternative display is necessary is output (S303) so as to proceed to the process (S109: Yes).
  • FIG. 24 is a flowchart showing the process flow of alternative display control based on someone other than the user as the information leakage medium.
  • the image recognition unit 121 analyzes the image captured by the camera 206 of the non-wearable terminal 2 or the camera 106 of the wearable terminal 1 and determines whether someone other than the user is captured (S401). If the determination result is negative (S401: No), the result of the process for determining whether the alternative display is necessary is outputted (S405) so as to continue to the process for not performing the alternative display (S109: No).
  • the information leakage medium determining unit 124 determines that the distance from the non-wearable terminal 2 to the other person is within the first distance range for determining whether the information on the display 212 can be read. (S402). If the determination result is affirmative (S402: Yes), the information leakage medium determination unit 124 determines to hide the entire screen of the display 212 (S403), and the alternative display control unit 125 performs processing when performing alternative display. The process result for determining whether or not an alternative display is necessary is outputted (S405) so as to proceed to (S109: Yes).
  • the information leakage medium determination unit 124 determines that if the other person's face is facing the display 212 (S404: Yes) It is determined that the entire screen is to be hidden (S403), and the alternative display control unit 125 outputs the result of the process for determining whether or not the alternative display is necessary (S405) so as to proceed to the process for executing the alternative display (S109: Yes). ).
  • the information leakage medium determining unit 124 determines that if the other person's face is not facing the display 212 (S404: No), the result of the process for determining whether the alternative display is necessary is outputted (S405) so as to proceed to the process for not performing the alternative display (S109: No).
  • FIG. 25 is a flowchart showing the process flow of alternative display control based on the affiliation of a person.
  • the image recognition unit 121 analyzes the image captured by the camera 206 of the non-wearable terminal 2 or the camera 106 of the wearable terminal 1 and determines whether someone other than the user is captured (S501). If the determination result is negative (S501: No), the result of the process for determining whether or not the alternative display is necessary is outputted (S503) so as to continue to the process for not performing the alternative display (S109: No).
  • the alternative display control unit 125 refers to the affiliation information set on the affiliation information details screen 430 (see FIG. 18) and determines whether the user's affiliation is the same as another person's affiliation. A judgment is made (S502). Then, the alternative display control unit 125 refers to the alternative display control rule information 440 (see FIG. 19), determines the content of alternative display control, and sets the alternative display to proceed to the process for executing the alternative display (S109: Yes). The result of the display necessity determination process is output (S503).
  • FIG. 26 is a flowchart showing the process flow of alternative display control based on the affiliation of a person.
  • the image recognition unit 121 analyzes the image captured by the camera 206 of the non-wearable terminal 2 or the camera 106 of the wearable terminal 1 and determines whether someone other than the user is captured (S601). If the determination result is negative (S601: No), the process result for determining whether or not the alternative display is necessary is outputted (S603) so as to continue to the process for not performing the alternative display (S109: No).
  • the alternative display control unit 125 refers to the affiliation information set on the affiliation information details screen 430 (see FIG. 18) and identifies the organization (affiliation) that manages the non-wearable terminal 2. The identity with another person's affiliation is determined (S602). Then, the alternative display control unit 125 refers to the alternative display control rule information 440 (see FIG. 19), determines the content of alternative display control, and sets the alternative display to proceed to the process for executing the alternative display (S109: Yes). The result of the display necessity determination process is output (S603).
  • the display of the non-wearable terminal is switched to a dark screen or a masked screen depending on the location where the user uses the non-wearable terminal and the surrounding situation (need for alternative display), thereby preventing information leakage.
  • the display of the non-wearable terminal is switched to a dark screen or a masked screen depending on the location where the user uses the non-wearable terminal and the surrounding situation (need for alternative display), thereby preventing information leakage.
  • the alternative display program 120 is installed on the wearable terminal 1, but the alternative display program 120 is installed on the non-wearable terminal 2, and after the wearable terminal 1 and the non-wearable terminal 2 are connected, the non-wearable terminal 2 can perform alternative display control. It may be determined whether or not execution is necessary.
  • non-display processing for the display 212 of the non-wearable terminal 2 in this embodiment may be considered to include brightness control, masking processing, mosaic processing, etc. of the display 212 of the non-wearable terminal 2.
  • each processing example may be independent programs, or a plurality of programs may constitute one application program. Furthermore, the order in which each process is performed may be changed.
  • Some or all of the functions of the present invention described above may be realized in hardware by, for example, designing an integrated circuit, a general-purpose processor, or a specific-purpose processor.
  • a processor includes transistors and other circuits and is considered a circuit or processing circuit.
  • the functions may be realized by software by having a processor unit or the like interpret and execute an operating program for realizing each function. Hardware and software may be used together.
  • control lines and information lines shown in the figures are those considered necessary for explanation, and do not necessarily show all control lines and information lines on the product. In reality, almost all components may be considered to be interconnected.
  • the embodiment includes the following aspects.
  • (Form of the first viewpoint) It is a wearable terminal, display and a communication device that receives display data from a non-wearable terminal; comprising a processor;
  • the processor includes: An alternative method of transmitting a control signal that causes the non-wearable terminal to hide the received display data via the communication device, receiving the display data of the non-wearable terminal, and displaying the display data on the display. perform display control, Wearable device.
  • a linked display system including a wearable terminal and a non-wearable terminal, Wearable devices are a first display; a first communication device that receives display data of the non-wearable terminal; a first processor; Non-wearable devices are a second display; a second communication device that transmits display data to the wearable terminal; a second processor;
  • the first processor transmits a control signal for hiding the display data to the non-wearable terminal via the first communication device, receives the display data of the non-wearable terminal, and displays the display data on the first display. executing alternative display control to display the display data on;
  • the second processor receives the control signal via the second communication device, hides the display data on the second display, and transmits the display data to the wearable terminal.
  • Linked display system is a first display; a first communication device that receives display data of the non-wearable terminal; a first processor;
  • Non-wearable devices are a second display; a second communication device that transmits display data to the wearable terminal; a second processor;
  • a linked display method executed between a wearable terminal and a non-wearable terminal comprising: transmitting a control signal from the wearable terminal to the non-wearable terminal to hide display data displayed on the display of the non-wearable terminal; Hiding the display data on a display of the non-wearable terminal and transmitting the display data to the wearable terminal; the wearable terminal receiving the display data from the non-wearable terminal; Displaying the display data on a display included in the wearable terminal; Linked display methods including.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Hardware Design (AREA)
  • Computer Security & Cryptography (AREA)
  • Software Systems (AREA)
  • Multimedia (AREA)
  • Human Computer Interaction (AREA)
  • Computer Networks & Wireless Communication (AREA)
  • User Interface Of Digital Computer (AREA)
  • Controls And Circuits For Display Device (AREA)
PCT/JP2022/024683 2022-06-21 2022-06-21 ウェアラブル端末、連係表示システム及び連係表示方法 Ceased WO2023248338A1 (ja)

Priority Applications (5)

Application Number Priority Date Filing Date Title
JP2024528143A JP7717279B2 (ja) 2022-06-21 2022-06-21 ウェアラブル端末、連係表示システム及び連係表示方法
PCT/JP2022/024683 WO2023248338A1 (ja) 2022-06-21 2022-06-21 ウェアラブル端末、連係表示システム及び連係表示方法
US18/876,693 US20250371201A1 (en) 2022-06-21 2022-06-21 Wearable terminal, linked display system, and linked display method
CN202280097004.9A CN119365846A (zh) 2022-06-21 2022-06-21 穿戴式终端、联动显示系统以及联动显示方法
JP2025121822A JP2025142227A (ja) 2022-06-21 2025-07-22 ウェアラブル端末による表示制御方法

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2022/024683 WO2023248338A1 (ja) 2022-06-21 2022-06-21 ウェアラブル端末、連係表示システム及び連係表示方法

Publications (1)

Publication Number Publication Date
WO2023248338A1 true WO2023248338A1 (ja) 2023-12-28

Family

ID=89379568

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2022/024683 Ceased WO2023248338A1 (ja) 2022-06-21 2022-06-21 ウェアラブル端末、連係表示システム及び連係表示方法

Country Status (4)

Country Link
US (1) US20250371201A1 (https=)
JP (2) JP7717279B2 (https=)
CN (1) CN119365846A (https=)
WO (1) WO2023248338A1 (https=)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008036005A (ja) * 2006-08-03 2008-02-21 Toshiba Corp 医用画像表示装置及び医用画像表示方法
JP2015172884A (ja) * 2014-03-12 2015-10-01 株式会社東芝 電子機器、システム及び方法
JP2016130916A (ja) * 2015-01-13 2016-07-21 コニカミノルタ株式会社 操作表示装置、画像形成装置、プログラム、操作表示システム

Family Cites Families (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR101991133B1 (ko) * 2012-11-20 2019-06-19 마이크로소프트 테크놀로지 라이센싱, 엘엘씨 헤드 마운트 디스플레이 및 그 제어 방법
US9536106B2 (en) * 2013-10-08 2017-01-03 D.R. Systems, Inc. System and method for the display of restricted information on private displays
US20160239091A1 (en) 2015-02-12 2016-08-18 Qualcomm Incorporated Controlled display of content on wearable displays
WO2021237383A1 (en) * 2020-05-23 2021-12-02 Citrix Systems, Inc. Sensitive information obfuscation during screen share
US12147553B2 (en) * 2020-09-17 2024-11-19 Acronis International Gmbh Systems and methods for providing access to data based on privacy of a user location

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2008036005A (ja) * 2006-08-03 2008-02-21 Toshiba Corp 医用画像表示装置及び医用画像表示方法
JP2015172884A (ja) * 2014-03-12 2015-10-01 株式会社東芝 電子機器、システム及び方法
JP2016130916A (ja) * 2015-01-13 2016-07-21 コニカミノルタ株式会社 操作表示装置、画像形成装置、プログラム、操作表示システム

Also Published As

Publication number Publication date
JP2025142227A (ja) 2025-09-30
JP7717279B2 (ja) 2025-08-01
JPWO2023248338A1 (https=) 2023-12-28
US20250371201A1 (en) 2025-12-04
CN119365846A (zh) 2025-01-24

Similar Documents

Publication Publication Date Title
US11315336B2 (en) Method and device for editing virtual scene, and non-transitory computer-readable storage medium
US8502780B1 (en) Head mount display and method for controlling the same
EP3857851B1 (en) Method, device, and medium for indicating a security status of an internet of things device
US20160352661A1 (en) Video communication method and apparatus
JP2012174208A (ja) 情報処理装置、情報処理方法、プログラム及び端末装置
US20190095654A1 (en) Method and apparatus for displaying application interface
CN108459788B (zh) 一种图片显示方法及终端
US20180144546A1 (en) Method, device and terminal for processing live shows
CN110032417A (zh) 会话入口屏蔽方法、装置、设备及存储介质
CN111479275B (zh) 可疑设备的检测方法、装置、设备及存储介质
CN106896917B (zh) 辅助用户体验虚拟现实的方法及装置、电子设备
JP2019117483A (ja) 情報処理装置、制御方法、及びプログラム
WO2023248338A1 (ja) ウェアラブル端末、連係表示システム及び連係表示方法
CN111556189B (zh) 通话的方法、装置、电子设备及介质
EP4250744A1 (en) Display terminal, communication system, method for displaying, method for communicating, and carrier means
CN114143280B (zh) 会话显示方法、装置、电子设备及存储介质
CN114816600A (zh) 会话消息显示方法、装置、终端及存储介质
US12541834B2 (en) Display terminal, display method, and recording medium
US20230368399A1 (en) Display terminal, communication system, and non-transitory recording medium
CN115981509A (zh) 信息管理方法、装置、客户端及存储介质
CN115840506A (zh) 基于可穿戴设备的数据交互方法、装置、设备及介质
CN116339498A (zh) 用户信息获取方法及装置、电子设备、存储介质
HK40026330A (en) Method and apparatus for detecting suspicious device, device and storage medium
HK40026330B (en) Method and apparatus for detecting suspicious device, device and storage medium
CN112788277A (zh) 信息处理装置及计算机可读介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 22947901

Country of ref document: EP

Kind code of ref document: A1

WWE Wipo information: entry into national phase

Ref document number: 2024528143

Country of ref document: JP

WWE Wipo information: entry into national phase

Ref document number: 202280097004.9

Country of ref document: CN

WWE Wipo information: entry into national phase

Ref document number: 18876693

Country of ref document: US

NENP Non-entry into the national phase

Ref country code: DE

WWP Wipo information: published in national office

Ref document number: 202280097004.9

Country of ref document: CN

122 Ep: pct application non-entry in european phase

Ref document number: 22947901

Country of ref document: EP

Kind code of ref document: A1

WWP Wipo information: published in national office

Ref document number: 18876693

Country of ref document: US