JP7453300B2 - Display control system, display device, and display control method - Google Patents

Description

The present invention relates to a display control system, a display device, and a display control method.

In recent years, as devices such as sensors have become smaller, wearable transmissive HMDs (Head Mounted Displays) called smart glasses have been commercialized and are used to support work such as factory work and equipment maintenance work. Generally, a user wearing a transparent HMD is able to check and operate work procedures by projecting and displaying information related to work support on a transparent virtual screen while looking at the user's own work target. Become.

In addition to displaying a virtual screen, a transmissive HMD may be equipped with a function of detecting a user's line of sight direction and position using an acceleration sensor, a gyro sensor, or the like. Using this function, the display position of information can be adjusted when displaying information on the virtual screen. For example, as in Patent Document 1, by detecting the position information of the virtual screen using various sensors and further setting a drawing priority in advance for the drawing objects to be displayed on the virtual screen, multiple drawing objects can interfere in the same place. A method has been devised to adjust the position so that it is not displayed.

Japanese Patent Application Publication No. 2015-115034

As described in the background art, a user wearing a transmissive HMD can perform his/her own work while viewing multiple pieces of information superimposed on a virtual screen without interfering with it. However, currently, the display position and display content of the information displayed on the virtual screen are defined in advance under uniform conditions.For example, when highly specialized work is performed by a non-specialist user, When a user who is familiar with the task performs the task, displaying the same content in the same display position may lead to a lack of understanding of the task itself and a decrease in work efficiency.

Furthermore, with the evolution of IT (Information Technology) technology, users can now efficiently acquire a large amount of information by using a recommendation function based on the hit rate of search words and a function to retain past browsing history information. However, in order to work with such a large amount of information, it is necessary to display many screens, and of course there is a problem in that there is a limit to the number of display devices that can be installed. Furthermore, presenting a lot of information on a display device is highly likely to lead to information leakage.

An object of the present invention is to provide a technology that can adjust the display of information according to user characteristics.

A display control system according to one aspect of the present invention includes a first display device that can be worn by a user, and a second display device that can communicate with the first display device, The first display device includes a first display section and a first control section that displays a first screen that is a virtual screen on the first display section, and the second display device includes: , a second display unit, and a second control unit that displays a second screen, which is an actual screen, on the second display unit, and the first control unit or the second control unit , adjusting the display method of the first screen and/or the second screen on the first display section and/or the second display section based on characteristic information determined as the user's characteristics; The display control system is configured as a display control system characterized by the following.

Further, a display device according to one aspect of the present invention is a display device that can be worn by a user, and includes a first display portion and a second screen that displays a first screen that is a virtual screen on the first display portion. 1 control unit, the first control unit controls the first screen having the first screen and/or the second display unit based on characteristic information determined as the user's characteristics. adjusting a method of displaying a second screen, which is an actual screen displayed on a second display device capable of communicating with the display device, on the first display section and/or the second display section. It is configured as a characteristic display device.

Further, in one aspect of the present invention, the present invention can also be understood as a display control method performed by the display control system.

According to one aspect of the present invention, it is possible to adjust the display of information according to user characteristics.

A diagram showing an example of a display device of a transmissive HMD Diagram showing an example of a display device of a transmissive HMD (another example) Diagram showing an example of a display device of a transmissive HMD (another example) Diagram showing one configuration of a display device of a transmissive HMD Diagram showing one configuration of an information processing device Diagram showing the configuration of a cloud information processing device Diagram showing the processing flow when starting to use the display device Diagram showing a processing flow for determining screen display conditions of a display device Diagram showing a processing flow for determining screen display conditions of a display device Diagram showing a processing flow for learning the screen display conditions of a display device Diagram showing a configuration of display device management data (user information) Diagram showing one structure of display device management data (work history) Diagram showing a configuration of display device management data (skill information) Diagram showing one configuration of display device management data (priority mode) Diagram showing an example of a screen displayed by a display device Diagram showing an example of a screen displayed by a display device Diagram showing an example of a screen displayed by a display device Diagram showing one configuration of an information processing device A diagram showing an example of linking an image captured by a display device to display content of an information processing device

In the following embodiments, when necessary for convenience, the explanation will be divided into multiple sections or embodiments, but unless otherwise specified, they are not unrelated to each other, and one does not differ from the other. This is related to variations, details, supplementary explanations, etc. of some or all of the above.

In addition, in the following embodiments, when referring to the number of elements (including numbers, numerical values, amounts, ranges, etc.), we also refer to cases where it is specifically specified or where it is clearly limited to a specific number in principle. However, it is not limited to the specific number, and may be greater than or less than the specific number.

Furthermore, in the embodiments described below, the constituent elements (including elemental steps, etc.) are not necessarily essential, unless explicitly stated or when they are considered to be clearly essential in principle. Needless to say.

Similarly, in the following embodiments, when referring to the shape, positional relationship, etc. of components, etc., the shape or positional relationship of the components, etc., is substantially the same, unless explicitly stated or it is clearly considered otherwise in principle. This shall include things that approximate or are similar to, etc. This also applies to the above numerical values and ranges.

Furthermore, in all the drawings for explaining the embodiments, the same members are designated by the same reference numerals in principle, and repeated explanation thereof will be omitted.
Embodiments of the present invention will be described below with reference to the drawings.

Below, we will show examples of display devices and display systems for transmissive HMDs, and show that by efficiently displaying a large amount of information on multiple screens, it is possible to realize a display device that can effectively support the user's work. .

First, the configuration of a transmissive HMD display device and a display system using the display device in this embodiment will be described.

FIG. 1A(a) shows a schematic diagram of a display device 101 (first display device) of a transmissive HMD in this embodiment. The display device 101 includes a calculation function, an operation function, and a display function, and includes a sensor unit 102 having a plurality of sensors, a display unit 103 (first display unit) that displays information, a light source unit 104, an operation unit 105, and an overall control unit. It has a section 106. The display device 101 of the transmissive HMD does not need to be glasses-shaped as long as it can be worn on the head, and the display unit 103 may be one-eyed. The sensor unit 102 may be installed, for example, at an end of the display device 101 instead of at the center, or may be installed separately at a plurality of locations. Further, the light source section 104 is installed so that it can emit light when the sensor section 102 is used or when the surrounding environment is dark. For example, if the irradiation range of the light source is narrow, it is possible to widen the irradiation range by installing multiple light sources, or by installing them at both ends.

The operation unit 105 may be a button, switch, or touch pad that allows the user to turn on/off the power of the display device 101, make various settings, and input data, and may have a communication function and cooperate with an external input device. It is also possible to do so. The overall control unit 106 includes a CPU (Central Processing Unit), a memory, a power supply, and a communication module. These may exist independently or may be integrated with the sensor section 102 and the operation section 105.

In this embodiment, it is assumed that the user performs work only on the display device 101, and a large amount of information is presented in space by displaying a plurality of virtual screens on the display device 101.

Further, as shown in FIG. 1A(b), the display device 101 of the transmissive HMD may have only a display function, and a separate device 108 connected via a communication line 107 may have a calculation function and an operation function. By dividing the functions and installing them in separate devices, it is possible to reduce the size and weight of the device that is worn on the head. The communication line 107 may be a wired communication or a wireless communication. In the case of wireless communication, there is no need for a communication line connecting the display unit 101 and the separate device 108, and the user can perform work without worrying about the existence of the communication line, which improves usability.

FIG. 1B shows a schematic diagram of another form of the display device 101. This example shows an example in which the display function is expanded in cooperation with another device 108. As an example of the separate device 108, a projector can be used in addition to a PC (Personal Computer) 108a, a mobile terminal or a smartphone 108b. Specifically, via the communication line 107, the display device 101 of the transmissive HMD and another device 108 (second display device) share and display the screen to be displayed. For example, if there are three screens to be displayed, the display device 101 displays two on the display unit 103 (virtual screen area) and one on the display unit 108a (real screen area) of the mobile terminal 108. The real screen area may be the display section 108b of the PC. For operations on each display section, individual operation sections may be used, or one of the operation sections may be used.

In this embodiment, it is possible to increase the amount of information presented to the user, and at the same time, it is possible to save space. In addition, it is also possible to control the display to either the virtual screen area or the real screen area depending on the "access level" and "type (with diagrams, only documents, etc.)" of the data to be displayed, the type of application used, etc. . For example, customer information and confidential information may be displayed in the virtual screen area, and applications such as communication apps that share the screen with multiple people may be displayed in the real screen area. Further, it is possible to share a virtual screen area among a plurality of display devices 101 via the communication line 107, or to display a different virtual screen area on each display device.

FIG. 1C shows a schematic diagram of another form of the display device 101. The display device 101 and the separate device 108 are connected to a cloud service 110 on the Internet 109 via a communication line 107. The cloud service 110 is provided by a plurality of devices, and provides various services such as an image recognition service, a translation service, and an information search service to the display device 101 and another device 108 for free or for a fee. The separate device 108 may be located in the same location as the display device 101, may be located in the same location as the cloud service 110, or may be located in separate locations.

In this embodiment, in order to perform display control on the real screen and virtual screen shown in FIG. It becomes possible to set the display method and display contents.

FIG. 2 is a block diagram showing an example of the configuration of display device 101 according to this embodiment. This example is an example (ie, FIG. 1A(a)) in which a display device worn on the head has an operation function and an arithmetic processing function integrated.

The display device 101 includes a sensor group 102, a display section 103 including a display control section 207, display sections 208a and 208b, a light source section 104, an operation section 105 including an input operation section 205, an audio input/output section 206, and a control section 103. The overall control unit 106 includes a communication unit 201, a communication unit 202, a power supply unit 203, an AI engine group 209, and a storage unit 210.

The control unit 201 includes a CPU and a GPU (Graphics Processing Unit), and performs calculations regarding control processing of the entire display device 101 and real-time image processing.

The communication unit 202 communicates control commands and various data with other devices and cloud services via the communication line 107 and the Internet 109. It is equipped with one or more communication methods such as LAN (Local Area Network), WAN (Wide Area Network), WiFi, Bluetooth (registered trademark), LPWA (Low-Power Wide Area), and USB (Universal Serial Bus). The communication unit 202 is also used to exchange operation commands when an external operation unit 205 is used.

The power supply unit 203 is a built-in or external battery.

The operation unit 205 is one or more input devices and device control drivers that allow the user to operate the display device 101 using buttons, touch panels, switches, dials, external mice, and keyboards.

The audio input/output unit 206 is a built-in or external microphone or earphone.

The display control unit 207 controls the position and timing of screens displayed on the display units 208a and 208b.

The display units 208a and 208b display screens on the display unit 103 (virtual screen) according to instructions from the display control unit 207.

In the AI engine group 209, AI (Artificial Intelligence) estimates and judges various events using parameters and threshold values learned through machine learning and deep learning.

The storage unit 210 includes a nonvolatile memory such as a ROM (Read Only Memory) or a FROM (Flash ROM), and stores various parameters and programs. The storage unit 210 stores programs for operating the authentication unit 2101, display generation unit 2102, display management unit 2103, skill management unit 2104, and user status management unit 2105.

The authentication unit 2101 performs access restrictions such as personal authentication, device authentication, and service authentication between the display device 101, another device 108, and the cloud service 110.

The display generation unit 2102 generates screen contents and display position information to be displayed on the display units 208a and 208b via the display control unit 207.

The display management unit 2103 manages, via the display control unit 207, the number and size of screens displayed on the display units 208a and 208b, the association between the screens and applications, the active status, and the like.

The skill management unit 2104 stores static information such as the user's job type, work history, years of experience, and qualifications held, as well as information such as the types of applications and services used, the types and frequency of operations on the terminal used, and work history. Generate and manage dynamic information. The static information can also be said to be user attribute information regarding skills. It can also be said that the dynamic information is usage information regarding the target for which the user's skill is to be demonstrated. Furthermore, these static information and dynamic information can be said to be information indicating characteristics related to the user's skills.

The user status management unit 2105 acquires and manages the working status, fatigue level, etc. of the user wearing the display device 101 using a vital measurement unit 2115, which will be described later.

The sensor group 102 includes various sensors such as an image sensor, a GPS (Global Positioning System), an acceleration sensor, a gyro sensor, and an illumination meter, and includes an imaging unit 2111, an imaging unit 2112, an inertial measurement unit 2113, an environment measurement unit 2114, and a vital sensor. It has a measuring section 2115.

The imaging unit 2111 and the imaging unit 2112 capture still images and videos. The imaging unit 2111 targets the user's field of view, and the imaging unit 2112 targets the surrounding environment including the user himself.

The inertial measurement unit 2113 includes an acceleration sensor, a gyro sensor, a geomagnetic sensor, and a GPS, and acquires and estimates the user's gaze direction, gaze position, posture, and current position.

The environment measurement unit 2114 includes an illumination meter, a temperature/hygrometer, a microphone, and the like, and acquires and estimates information regarding the user's surrounding environment.

The vital measurement unit 2115 includes a heart rate meter, a thermometer, a blood pressure meter, a pulse meter, an electroencephalogram, and the like, and acquires the user's physical information and estimates the user's physical condition and emotions.

Note that the storage unit 210 and the AI engine group 209 may be provided outside the present system. For example, the storage unit 210 and the AI engine group 209 may be provided in a server device connected to this system via a network, and may be accessed from this system as needed. The same applies to the storage unit and the AI engine group shown in FIGS. 3 and 4, which will be described later.

The case shown in FIG. 1A (a) described above assumes that the user uses all the necessary information by displaying it anywhere in the virtual screen area while going out, at home, or working hands-free. The user performs input operations via the communication unit 202 and the operation unit 205 using a Bluetooth (registered trademark) or wireless USB mouse, keyboard, remote control, or voice.

In the case of FIG. 1A(b) described above, the display device 101 and the separate device 108 are connected via the communication unit 202. The display device 101 includes at least the display control section 207 in FIG. Equipped with an integrated circuit (chip). It is assumed that the other devices are installed in a separate device 108.

In the case of FIG. 1B, the display device 101 may have a configuration similar to that of FIG. 2, or may have a configuration similar to that of FIG. 1A(b) described above.

The user can use the configuration shown in FIGS. 1A (a) and 1A (b) outside the office, and seamlessly switch to the configuration shown in FIG. 1B when returning to the office. Furthermore, after working in the form shown in FIG. 1B at the office, the user can shift to the form shown in FIG. 1A (a) when going out. For example, the control unit 201 identifies the user's current location using various sensors installed in the display device 101, which will be described later, and determines whether the user's current location is within the office. When the control unit 201 determines that the user's current location is within the office, the control unit 201 performs the processing flow of FIG. 5, which will be described later, automatically or in accordance with the user's instructions.

FIG. 3 is a block diagram showing an example of the configuration diagram 1B of the separate device 108 according to the present embodiment. This example is an example in which screen display is carried out in cooperation with and sharing with the display device 101.

The separate device 108 includes a control unit 301, a communication unit 302, a time management unit 303, a power supply unit 304, an operation unit 305, an audio input/output unit 306, a display control unit 307, a display unit 308, an AI engine group 309, a storage unit 310, It has a sensor group 311.

The AI engine group 309 has an AI engine that has performance equal to or higher than that of the AI engine group 209 of the display device 101, or has a different function or algorithm.

In addition to the same sensors as the sensor group 211 of the display device 101, the sensor group 311 includes an eye tracking unit 3112 that tracks the user's line of sight in real time. The eye tracking unit 3112 may be the imaging unit 2111.

The storage unit 310 includes an authentication unit 3101, a display generation unit 3102, a display management unit 3103, a skill management unit 3104, a user state management unit 3105, and an optimal expression conversion unit 3106.

The authentication unit 3101 and the authentication unit 2101 of the display device 101 authenticate whether each device is a legally manufactured device and whether the correct user is accessing/using the device.

The display generation unit 3102 generates a screen image to be displayed on the display unit 308 or information for generating the image. Further, the display generation unit 3102 generates a screen image or an image to be displayed on the display units 208a and 208b of the display device 101 connected via the communication unit 302 according to instructions from the control unit 201 and the display management unit 2103. generate information.

The display management unit 3103 displays the display screen generated by the display generation unit 3102 on the display unit 308 or on the display device 101 connected via the communication unit 302 in order to expand the area of the display unit 308. or both. If the display management unit 3103 determines that the display screen or information is to be displayed on the display device 101, the display screen or information created by the display generation unit 3102 is passed to the display device 101 via the control unit 301 and the communication unit 302. The control unit 201 of the display device 101 passes the received display screen to the display control unit 207 and displays it on the display units 208a and 208b.

Like the skill management unit 2104 of the display device 101, the skill management unit 3104 generates and manages information regarding the user's abilities and skills. This information may have the same content as the skill management section 2104, or may have different content.

The user status management unit 3105 manages the user's physical status and work status using information acquired using the imaging unit 3111 and the eye tracking unit 3112 of the sensor group 311. All or part of the information managed by the user status management unit 2105 of the display device 101 may be acquired via the communication line 107.

The optimal expression conversion unit 3106 matches the user's skills and status managed by the skill management unit 3104 and user status management unit 3105, or the display target (display unit 308 or display device 101) determined by the display management unit 3103. Convert what is displayed on the screen. For example, when the user status management unit 3105 determines that the user is highly fatigued, the optimal expression conversion unit 3106 increases the font size displayed on the display unit 308 or decreases the virtual screen area of the display device 101. . To determine whether the degree of fatigue is high, for example, the user condition management unit 3105 compares information regarding the user's condition obtained from the sensor group 311 with a predetermined threshold, and determines whether the information regarding the user's condition is If it is larger than the threshold, it may be determined that the degree of fatigue is high. As the threshold value, for example, an average value of information regarding past user states can be used.

FIG. 4 is a block diagram showing an example of the configuration of cloud service 110 according to this embodiment.

The cloud service 110 includes a control unit 401, a communication unit 402, a time management unit 403, a power supply unit 404, an operation unit 405, an audio input/output unit 406, a display control unit 407, a display unit 408, an AI engine group 409, and a storage unit 410. have.

Elements other than the storage unit 410 are the same as the display device 101 and other devices 108 in FIGS. 2 and 3. Note that the time management unit 403 is a timer or a clock.

The storage unit 410 includes a user management unit 4101, a screen display condition learning unit 4102, a skill learning unit 4103, and a work history management unit 4104.

The user management unit 4101 creates and manages information regarding users who use this cloud service.

A screen display condition learning unit 4102 uses an AI engine group to learn conditions related to screen display for each user managed by the user management unit, and obtains optimal arrangement conditions.

The skill learning unit 4103 uses an AI engine group to learn conditions related to work-related knowledge and experience for each user managed by the user management unit, and acquires the user's skill conditions.

The work history management unit 4104 creates and manages information regarding the work performed for each user managed by the user management unit.

FIG. 5 shows a processing flow when a user starts using the display device 101 according to this embodiment. Data and screen examples necessary for the processing will be explained using FIGS. 9A to 9D and FIG. 10.

This processing flow assumes that the user wears the display device 101 while using another device 108 (FIG. 1A). Further, assume the case shown in FIG. 1B described above.

When the control unit 201 of the display device 101 detects that the user is wearing the display device 101, the control unit 201 uses the authentication unit 2101 to perform user authentication processing. The authentication process uses password input using the operation unit 205, voice input using the voice input/output unit 206, user's physical information (pulse wave, brain wave) acquired from the vitals measurement unit 2115, and imaging units 2111 and 2112. Using one or more pieces of information such as a fingerprint or face, it is determined whether the user is a user registered in advance in the authentication unit 2101.

FIG. 9A shows an example of user information 900. User information 900 includes a user ID 901, password 902, affiliation 903, position 904, and terminal number 905. In this step, a user ID 901 and a password 902 are used (step 501). By performing step 501, the user and the display device 101 are linked.

Next, the control unit 201 uses the inertial measurement unit 2113 to acquire the user's current position, uses the environment measurement unit 2114 to acquire the temperature, illuminance, and noise at the user's current position, and uses the imaging unit 2111 to acquire the user's current position. An image of the user's line of sight is acquired (step 502). By performing step 502, various information regarding the environment around the user's current location is acquired.

The control unit 201 identifies the user's location using the location information registered in advance in the authentication unit 2101 (for example, the affiliation 903 in the user information 900) and the information acquired in step 502. In addition, the control unit 201 uses the image recognition function of the AI engine group 209 to select the real screen area used by the user for work (display of a PC or smartphone, projector screen, digital signage) (step 503). By performing step S503, the pairing between the display device 101 and the real screen area that is in the line of sight of the user via the display device 101 is completed.

If the real screen area is detected in step 503, the control unit 201 determines whether the real screen area is the one registered in the authentication unit 2101 (for example, the information access level 9009 in the user information 900 or the terminal number 905). Determine. For example, this may be the case when a user uses a PC shared by multiple people in his or her office (step 504).

In step 504, if the control unit 201 determines that the real screen area is the one registered in the authentication unit 2101 (step 504; Yes), the control unit 201 displays the real screen area via the authentication unit 2010 and the communication unit 202. The authentication unit 3101 of the other device 108 that is connected to the other device 108 authenticates whether the device is a legitimate device and whether or not information can be exchanged with each other (step 505). By performing step 505, the display device 101 and the separate device 108 displaying the real screen area paired in step 503 are mutually authenticated as legitimate devices.

If the authentication is successful, the control unit 201 uses the display management unit 2103 and the display control unit 207 to determine whether or not the user is currently working on the display device 101, and if the user is currently working, saves the work. It is determined whether or not (step 506). By performing step 506, the screen being worked on is temporarily saved. Whether or not work is in progress may be determined, for example, based on whether or not the screen displayed in the real screen area is active.

When the information currently being worked on is saved in step 506 (step 506; Yes), the control unit 201 synchronizes the screen display contents via the control unit 301 of the separate device 108 and the communication unit. For example, if the file being worked on on the display device 101 is also displayed on another device 108, the control unit 201 performs processing such as updating it or saving it with a different file name as necessary (step 507).

Thereafter, the control unit 201 determines whether or not it is necessary to organize the screens to be displayed with the control unit 301 of the separate device 108. The determination procedure will be described later (step 508).

If it is determined that the information is to be organized by both parties (step 508; Yes), the control unit 201 of the display device 101 shares the information displayed on each screen with the control unit 301 of the separate device 108. Specifically, the work history 9000 in FIG. 9B is created and updated between the display management unit 2103 of the display device 101 and the display management unit 3013 of another separate device 108.

The work history 9000 includes a work ID 9001, implementation date 9002, time 9003, terminal number 9004, work name 9005, application information 9006, screen information 9007, input operation information 9008, and information access level 9009.

The work ID 9001 is an identifier of a work that the user is currently performing/performed during a predetermined period including the present. The execution date 9002, the time 9003, and the terminal number 9004 indicate the execution date of the work, the time used, and the terminal number. A work name 9005 is a classification name of the work, and the classification name can be defined according to the user's affiliation and position. Application information 9006 is the classification name of the application used for the task. The screen information 9007 includes the number of screens displayed when using the application information 9006, the display size, the position of the screen in the real screen area 1001 (display section 308) and the virtual screen areas 1002 and 1003 (display sections 208a and 208b), Indicates the screen reference/update frequency, etc. Input operation information 9008 indicates the operation method used for the task. The information access level 9009 indicates the security level of information used and referenced when performing the work.

The display device 101 and the separate device 108 maintain and manage the work history 9000 related to their own terminal number 9004, and also create the information currently being displayed for each screen. The work history 9000 may share the same information between both devices, or may be held and managed by either device (step 509).

FIG. 10 shows an example in which screens are displayed on a real screen area 1001, which is the display of a PC assuming another device 108, and on virtual screen areas 1002 and 1003 displayed by the display device 101, respectively. For example, the display unit 208a of the display device 101 displays the virtual screen area 1002, and the display unit 208b of the display device 101 displays the virtual screen area 1003. The control unit 201 of the display device 101 uses the inertial measurement unit 2113, the imaging unit 2111, the imaging unit 2112, and the AI engine group 209 to determine the three-dimensional position of the real screen area 1001 that is visible to the user, and determines the three-dimensional position of the real screen area 1001 that does not overlap with that position. The virtual screen areas 1002 and 1003 are arranged as follows. Here, in order to clarify the storage location of the screen content to be displayed, the control unit 201 clearly displays a mark 1004 somewhere on the screen of the display device 101, for example. In FIG. 10, two screens displayed in the virtual screen area 1002 are marked with a mark V (Virtual) indicating that they are virtual screens, and these screens are virtual screens (that is, the display device It can be seen that 101 is the screen being displayed. In addition, while the front screen 1001a displayed in the real screen area 1001 is marked with a mark V indicating that it is a virtual screen, the rear screen 1001b is not marked with the mark. It can be seen that is a virtual screen displayed by the display device 101, and the latter is a real screen displayed by another device 108. In this way, the user can easily understand whether the display screen is a screen displayed in the virtual screen area or a screen displayed in the real screen area (step 510).

As described above, when the user wears the display device 101, the user can detect the real screen area existing around the eyes, automatically connect to the real screen area, and display the virtual screen area around the real screen area. This allows the amount of information to be displayed to be increased without increasing the real screen area.

Here, although the control unit 201 of the display device 101 detects the other device 108 in step 503 above, the control unit 301 of the other device 108 detects the display device 101 using the imaging unit 3111 and the AI engine group 309. It's okay.

In addition, in the screen example of FIG. 10, the screens to be displayed are distributed to the display device 101 and the separate device 108, but the screen displayed by the separate device 108 is distributed to both the real screen area 1001 and the virtual screen areas 1002 and 1003. It may be displayed. Further, the screens displayed in the real screen area 1001 and the virtual screen areas 1002 and 1003 may be limited to only the screens displayed by the display device 101. Furthermore, the device to be used may be determined by the screen area, such as the real screen area 1001 and the virtual screen area 1003 being screens displayed by another device 108, and the virtual screen area 1002 displaying the screen displayed by the display device 101.

Furthermore, although this processing flow assumes that the user attaches the display device 101 while using the separate device 108, the same procedure can be performed even if the separate device 108 is started while the display device 101 is attached. be.

FIG. 6 shows a condition determination processing flow (step 508 in FIG. 5) in which the display device 101 or the separate device 108 according to this embodiment organizes the arrangement of screens to be displayed. This processing flow assumes that the user wears the display device 101 while using another device 108. Data necessary for the processing will be explained using FIGS. 9A to 9D.

Here, in this condition determination processing flow, it is determined which of the display device 101 and the separate device 108 should perform the determination, according to the setting value of the priority mode 9200 in FIG. 9D. The priority mode 9200 is held and managed by the display management units 2103 and 3103. The priority mode 9200 has a user ID 9201, a priority display target 9202, and a priority control target 9203. The priority display target 9202 is an item for determining which device should be displayed on the screen with priority. In this item, either another device 108 (“real screen area”) to be displayed or the display device 101 (“virtual screen area”) to be displayed is set. Similarly, the priority control target 9203 is an item for determining which device's operation should be prioritized. As in the case of the priority display target 9202, either another device 108 (“real screen area”) or the display device 101 (“virtual screen area”) is set in this item. In this example, both are "real screen areas", that is, separate devices 108.

The control unit 301 of the separate device 108 uses the environment measurement unit 3114 to measure the temperature, humidity, illuminance, noise, etc. around the user. Alternatively, information measured by the environment measurement unit 2114 of the display device 101 may be acquired (step 601).

Further, the control unit 301 acquires information about the screen currently displayed in the real screen area 1001 and the virtual screen areas 1002 and 1003 from its own display management unit 3103 and the control unit 301 of the display device 101. That is, the control unit 301 extracts from the work history 9000 the currently displayed information whose end time is not listed (or whose window is active) at time 9003. Then, the control unit 301 obtains the layout of those screens (all displayed, overlapping, hidden, etc.) (step 602).

Next, the control unit 301 acquires skill information 9100 shown in FIG. 9 from the skill management unit 3104 (step 603). Note that in this embodiment, information regarding the user's skill is used to explain how to adjust the display of a virtual screen and a real screen according to the skill level, but information regarding the user other than the user's skill level ( For example, the display of the virtual screen or the real screen may be adjusted according to the user using information indicating attributes representing user characteristics such as the user's age, gender, and nationality.

The skill information 9100 includes a user ID 9101, year of employment 9102, work history information 9103, qualification information 9104, and job-specific characteristics 9105.

Work history information 9103 is information indicating the user's current and past work history, and stores one or more pieces of work data in which a work ID is associated with an evaluation of the work identified by the work ID. .

Qualification information 9104 is information regarding qualifications that the user holds or has held in the past.

The work-specific characteristics 9105 include the number of screens displayed and the average/maximum/minimum value of the screen size for the work described in the work name 9005, application information 9006, and input operation information 9008 in the work history 9000. Shows statistical values and settings such as apps used, input operation time, average work time, etc. For example, the skill management unit 3104 searches for user information that includes the same user ID 9101 from the user information 900, and reads the terminal number 905 included in the searched user information. The skill management unit 3104 reads the work history that includes the same terminal number 9004 as the read terminal number 905 from the work history 9000, and reads the time 9003, work name 9005, application information 9006, and screen information included in the read work history. The settings are made by reading information such as 9007 and input operation information 9008 and calculating the above-mentioned statistical values and setting values by summing up the read information.

Using the currently displayed screen information acquired in step 602 and the job-specific characteristics 9105 acquired in step 603, the control unit 301 specifies the current work content (step 604). To specify the work content, for example, the control unit 301 refers to the application information 9007 (for example, document creation software) in the work history 9000, identifies the application that the user is working on, and determines the work-specific characteristics of the identified application. Read 9105. The control unit 301 determines the work content of the application currently being worked on based on the read work-specific characteristics 9105 and information such as the time 9003, work name 9005, application information 9006, screen information 9007, input operation information 9008, etc. of the work history 9000. (For example, information regarding the presence or absence of keyboard input, input content, and input time) is aggregated and output. The currently working application includes the screen of the active application as well as the screens of other applications currently being displayed in the operation screen area.

Next, the control unit 301 sets the display counter (Cnt) to the current number of display screens (Dn) (step 605), and determines whether the display counter (Cnt) is greater than or equal to the maximum number of screens (Dm). is determined (step 606). For example, the display counter may be set by the control unit 301 to the maximum value or average value of the number of screens stored as the task-specific feature 9105 of the skill information 9100. By setting the number of screens, the user can normally set the number of screens to be displayed simultaneously in the operation screen area.

If the control unit 301 determines that the display counter has reached the maximum value, that is, if it determines that the following processing has been performed for all displayed screens (step 606; Yes), the control unit 301 performs the specified processing in step 604. Among the work contents, the work whose reference frequency is less than the minimum value at time 9003 of the work history 9000 or the work whose screen size is less than the minimum value from the screen information 9007 is extracted (step 607). That is, the control unit 301 determines whether the operations performed from the most recent work time to the present in the work history for the currently displayed screen are less than a predetermined minimum value, or the currently displayed screen. If the screen size of the screen is less than a predetermined minimum value, it is determined that the task is infrequently used and has a low priority for the user, and the task is identified.

If the control unit 301 determines in step 607 that the corresponding work has been extracted (step 607; Yes), it instructs the display management unit 3103 to hold screen information regarding the work (step 608). On the other hand, if the control unit 301 determines in step 607 that the corresponding work could not be extracted (step 607; No), the control unit 301 retains the screen information with the lowest reference frequency (step 610).

After that, the control unit 301 decrements the display counter (Cnt) by -1 and repeats steps 606 to 609 until the display counter becomes less than the maximum value (Dm).

When the control unit 301 determines that the display counter (Cnt) has become less than the maximum value (Dm) (step 606; Yes), the control unit 301 determines whether the display counter is equal to the number of display screens (Dn) (step 606; Yes). step 611). If the values are different in step 611, this indicates that the number of screens has reached the maximum value and steps 606 to 609 have been executed one or more times.

If the control unit 301 determines that the values are different in step 611 (step 611; Yes), it instructs the display management unit 3103 to set a virtual screen use flag for the screen information held in step 608 or step 610 (step 611; Yes). 612). Then, the control unit 301 notifies the display device 101 to display the work screen for which the virtual screen use flag is set in the virtual screen area of the display device 101. The control unit 201 of the display device 101 instructs the display management unit 2103 to display the corresponding work screen in the virtual screen area (step 613).

From the above, if the number of screens to be displayed in the real screen area increases, if the screen size of the work target is small, or if there are screens that are being displayed but are referenced less frequently, those screens will be moved to the real screen area. By moving the information from the virtual screen area to the virtual screen area, it is possible to increase the amount of information displayed in the real screen area, which is the main work target.

Note that the order of steps 601 to 603 may be changed. Further, when the display priority 9202 of the priority mode 9200 is "virtual screen area", the processing flow of FIG. 6 may be executed by the display device 101 side.

FIG. 7 shows another example condition determination processing flow different from FIG. 6. Similar to FIG. 6, this processing flow will also be described for the case where the user wears the display device 101 while using another device 108. Steps 701 to 704 perform the same processing as steps 601 to 604 described above, so the processing from step 705 onward will be described.

Next, the control unit 301 of the separate device 108 initializes the access restriction counter (Acnt) (step 705), refers to the information access level 9009 of the work history 9000 related to the work extracted in step 704, and displays it on the screen. Determine whether the information is access-restricted. The information access level can be arbitrarily defined, for example, "0: No access restriction," "1: Internal access limited," "2: Internal internal access limited," and "3: Team access limited." If the information access level 9009 in the work history 9000 regarding the work identified in step 704 is "2: internal access only", it is determined that this work has access restrictions (step 706).

Here, the control unit 301 may estimate the current position from the environmental information acquired in step 701 and incorporate the relationship between the current position and the information access level into the determination in step 706. For example, if the information access level 9009 is "2: Internal access limited" and the current location is estimated to be an office, the control unit 301 determines that the work does not have access restrictions. If the information access level 9009 is "2: Internal access limited" and the current location is estimated to be outside the office, it is determined that the work is access restricted.

If the control unit 301 determines that access is restricted in step 706 (step 706; Yes), it determines that use at the current location is not desirable and instructs the display management unit 3103 to change the screen related to the work. The information is retained (step 707). Then, the control unit 301 increments the access restriction counter (Acnt) by +1 (step 708). The control unit 301 performs the processing from steps 706 to 708 for all screens currently being displayed.

If it is determined in step 706 that there is no access restriction (step 706; No), the control unit 301 determines that all the tasks with access restrictions have been extracted, and furthermore, there are tasks that use communication apps. It is determined whether or not (step 709). Communication apps refer to applications such as chat tools and video conferencing that allow multiple users to send and receive information from multiple networked locations.

If the control unit 301 determines in step 709 that someone is using a communication app (step 709; Yes), the control unit 301 displays the previous screen displayed by the app, a summary of the communication, and a transcription of the audio. , determines content to be displayed by extending the current display screen, such as indirect content that is not used in the current communication (step 710), and sets a communication application use flag (step 711). That is, when using a communication application, the control unit 301 expands and displays the summary screen that aggregates the history of individual communications (for example, one round trip conversation) for a certain period of time in chats, conversations, etc. Create the content and set the communication app use flag for the summary screen. As shown below, the summary screen for which the communication application usage flag has been set changes from being displayed in the real screen area to being displayed in the virtual screen area. The communication application usage flag is set to 0 when there is no content to be expanded and displayed, and is set to a value of 1 or more in other cases.

Next, if the control unit 301 determines that there is no communication application in use in step 709 (step 709; No), or if the communication application usage flag is set in step 711, the control unit 301 adds a graphic application to the work. It is determined whether there is one in use (step 712). The graphic application refers to an application that creates 2D/3D design drawings, circuit diagrams, and layout diagrams, such as CAD (Computer-Aided Design) and image production (step 712).

If the control unit 301 determines in step 712 that there is a graphic application using a graphic application (step 712; Yes), the control unit 301 enlarges the screen displayed by the application or changes the screen from 2D (3D) to 3D (2D). In addition to the converted screen, the content to be displayed by expanding the current display screen is determined, such as indirect content that is not used in the current graphic display (step 713), and a graphic application usage flag is set (step 714). . As shown below, the screen for which the graphic application use flag has been set changes from being displayed in the real screen area to being displayed in the virtual screen area. The graphic application usage flag is set to 0 when there is no content to be expanded and displayed, and is set to a value of 1 or more in other cases.

The control unit 301 checks the set values of the access control counter, the communication application use flag, and the graphic application use flag, and if any of the set values is 1 or more, sets a virtual screen use flag for the corresponding display screen ( In step 715), the display device 101 is notified to display on the virtual screen area of the display device 101. The control unit 201 of the display device 101 instructs the display management unit 2103 to display the corresponding work screen (step 716).

From the above, if there is display content that requires security consideration, such as customer information or management information, among the screens displayed according to the skill level, those screens will be displayed in the virtual screen area. This prevents anyone other than the user from viewing the displayed content, making it possible to prevent information leakage. FIG. 11 shows an example screen.

FIG. 11 is a diagram showing an example of displaying a screen with access restrictions in a virtual display area. In FIG. 11, a virtual screen area 1002 displays two screens that display information with access restrictions (other company information, management information), and a virtual screen area 1003 displays other information with access restrictions ( This shows that three screens displaying customer information A to C) are displayed. Further, on the display device 101, which is a real screen area, a screen 1001a, which is a virtual screen, and a screen 1001b, which is a real screen, are displayed, as in the case of FIG. Therefore, the user U1 wearing the display device 101 can view both the screen 1001a and the screen 1001b that are not subject to access restrictions displayed in the real screen area 1001, as well as the screen with access restrictions; User U2 who is not wearing display device 101 will not be able to view the latter screen.

Additionally, when using chat tools, there is a limit to the amount of information that can be displayed on one screen, and the content is updated in real time, making it difficult to remember what was exchanged a while ago. Therefore, by displaying the content that can no longer be displayed on the screen in the virtual screen area, it is possible to immediately refer to the previous exchange. Furthermore, by creating a summary of the content that cannot be displayed on the screen using the AI engine group 309 and constantly displaying it in the virtual screen area, it is possible to easily understand the content of the communication. FIG. 12 shows an example screen.

FIG. 12 is a diagram illustrating an example of displaying a screen on which the communication application usage flag is set in the virtual display area. In FIG. 12, a virtual screen area 1002 displays a screen S1 that displays summary information when interacting with the chat tool, and a virtual screen area 1003 displays individual chat screens S3 that have been exchanged in the past using the chat tool. , S4 are displayed. Furthermore, the latest chat screen S1 is displayed as a real screen on the display device 101, which is a real screen area. Therefore, a user wearing the display device 101 can check the latest chat content on the real screen area, while viewing chat summary information and past chat content as needed on the virtual screen areas displayed on both sides. You can check the content of the latest chat and concentrate on it. Further, as in the case of FIG. 11, the contents of the screens S1 to S3 displayed in the virtual screen area are not seen by a third party.

Here, the condition determination processing flows of FIGS. 6 and 7 may be used in combination, or may be used individually for each task.

FIG. 8 shows a data generation process flow (a) and a learning process flow (b) for learning the judgment conditions (steps 603 and 703) shown in FIGS. 6 and 7. Specifically, it refers to the generation/update processing of the work history 9000 and the generation/update processing of the job-specific characteristics 9105 of the skill information 9100. In this embodiment, the data generation processing flow (a) when the display device 101 is used will be described, but the same applies to the other device 108.

When the user turns on the power of the display device 101, the control unit 201 acquires the start time from a timer (not shown) (step 801), and obtains information regarding the work managed by the work application or schedule application that starts automatically when the power is turned on. (For example, the work ID 9001, implementation date 9002, time 9003, terminal number 9004, work name 9005, application information 9006, and access level 9009 of the work history 9000) are acquired (step 802). The implementation date 9002 and start time 9003 may be set from the startup time obtained in step 801. After initializing the display counter (Vcnt), the control unit 201 determines whether the display device 101 is currently powering down (step 804). The reason for determining whether or not the power is being shut down is to perform the following processing while the user is starting up the display device 101.

When the control unit 201 determines that the power is not being shut down (step 804; No), the control unit 201 records and updates the information regarding the currently running application acquired in step 802 in the work history 9000 (step 805), and updates the information to the work history 9000 (step 805). It is determined whether it exists in the work history 9000 (step 806). Here, when the power is turned on, in step 805, the information regarding the application acquired in step 802 is recorded and updated in the work history 9000, but since the new application does not exist in the work history 9000, it is temporarily stored in the memory.

When the control unit 201 determines that the information acquired in step 806 does not exist in the work history 9000 (step 806; No), the control unit 201 increments the display counter (Vcnt) by +1 (step 807), and controls the operation unit for the corresponding work application. 205 and the audio input/output unit 206 (for example, input operation information 9008 of the work history 9000). For example, the control unit 201 counts the number of mouse clicks and keyboard types per unit time (step 808).

In addition, the control unit 201 acquires the number of display screens and display screen size of the work application managed by the display management unit 2103, starts detecting whether the number or size of screens has been changed, and changes the detected number of screens and display screen size. Screen information (for example, screen information 9007 of work history 9000) is acquired (step 809). Then, the control unit 201 uses the display management unit 2103 to create a work history 9000 regarding the work application for the new application, assigns an arbitrary work ID 9001, and sets the implementation date 9002 and start time 9003 obtained in step 802. do. Further, the terminal number 9004 is set to its own terminal number (for example, the terminal number HMD-8 of the display device 101), and the work name 9005 and application information 9006 are set to the information obtained in step 805. Then, the control unit 201 sets, in the screen information 9007, the number of display screens and the screen size around the unit time when detection was started in step 809. Similarly, the control unit 201 sets, in the input operation information 9008, the number of mouse and keyboard inputs per unit time whose counting was started in step 808 (step 810).

If the control unit 201 determines that the information acquired in step 806 exists in the work history 9000 (step 806; Yes), it determines whether the corresponding work application has ended (step 811), and if it has not ended. If it is determined (step 811; No), the process returns to step 804, and the work history 9000 is recorded and updated until shutdown.

On the other hand, if the control unit 201 determines in step 811 that the corresponding work application has ended (step 811; Yes), the control unit 201 obtains the current time (step 812) and sets the time 9003 (end time) of the corresponding work history 9000. After setting, final updating the screen information 9007 and input operation information 9008 (step 813), and decrementing the display counter (Vcnt) by -1 (step 814), the process returns to step 804.

The control unit 201 continues the processing from steps 804 to 814 while the power is turned on. If the user instructs a shutdown, the one or more created work histories 9000 are stored in the display management unit 2103 as display learning data (step 815).

By performing such processing to record and update the work history 9000, it is possible to record and update the work history 9000, so that it is possible to record and update the work history 9000, so that it is possible to record and update the work history 9000, so that it is possible to record and update the work history 9000. You can obtain the original data for learning, such as whether the

Next, a learning process flow (b) when using the display device 101 will be described. The work history 9000 as display learning data stored in the display management unit 2103 in step 815 is used periodically by the control unit 201 as follows.

The control unit 201 acquires the display learning data from the display management unit 2103 at an arbitrary timing (step 820), and generates data to be input to the AI engine group 209 (step 821). Specifically, input data is generated using two or more items in the work history 9000 other than the work ID 9001 and one or more items in the skill information 9101 other than the user ID 9101.

After that, the control unit 201 acquires learning parameters held by the AI engine group 2109 (step 822), and executes machine learning using the learning parameters and the AI engine group 2109 (step 823). Here, general methods such as statistical methods and deep learning are used as machine learning methods.

The control unit 201 registers the parameters obtained as a result of learning in step 823 in the AI engine group 2109 as new learning parameters (step 824), and stores the results of using the learning parameters in the task-specific features 9105 of the skill information 9100. Store (step 825).

From the above, by creating and updating thresholds for the number of screens displayed, display size, and reference frequency for each work application, the screen display position can be optimized according to the user's skills and work content, and a large amount of information can be stored. It becomes possible to work while displaying the information effectively. Here, the learning processing flow in FIG. 8(b) may be executed on the cloud side using the configuration in FIG. 1C.

Examples of display devices and display systems for transmissive HMDs are shown below, and the user's work can be made more effective by easily linking still images or moving images acquired by the display device with display content on other information processing devices. We show that it is possible to realize a display device that can support the

FIG. 13 is a block diagram showing a configuration of a transmissive HMD display device 1300 in this embodiment. Although this example assumes the same form as FIG. 1A(a), it can also be applied to FIGS. 1B and 1C described in Example 1.

In addition to the configuration of the display device 101, the display device 1300 includes a content recognition section 1301 and a link management section 1302.

The content identification unit 1301 includes text and image information included in the real screen area displayed by the separate device 108, content being displayed on the display unit 208, or content being recorded/recorded by the imaging unit 2111, and their position information. get. Specifically, this information is acquired using OCR (Optical Character Recognition/Reader) or image recognition using the AI engine group 209.

The link management unit 1302 associates the text and image information acquired by the content identification unit 1301 with another real screen area or the content being displayed, and manages the link information. Note that an eye tracking section 3112 (detection section) may be used instead of the imaging section 2112.

For example, FIG. 14 shows a case where a moving image 1401 taken by the user on the display device 1300 is linked to an arbitrary part of the work application being displayed on the separate device 108. In FIG. 14, the control unit 201 of the display device 1300 reads three moving images 1401 captured by the imaging units 2111 and 2112, arranges (copies) them in the virtual screen area 1400, and presses a playback button to play back the moving images 1401. 1402 and a link button 1403 for linking (paste) the moving image 1401 to another device 108. In the following, as shown in FIGS. 5 to 7, the control unit 201 controls the number of screens to be displayed in the virtual screen area of the display device 101 and the number of screens to be displayed in the separate device 108 according to attributes representing the skill level and characteristics of the user. This shows the case where the number of screens displayed in the real screen area is adjusted.

A user wearing the display device 1300 uses the imaging units 2111 and 2112 to capture a moving image of operating an actual object C1 such as a home appliance or experimental device installed in a location other than the office, or to record a paper document for reference. and the contents of the manual C2. Then, the display device 1300 displays a list screen of images photographed by the imaging units 2111 and 2112 in the virtual screen area 1400 displayed by the display unit 208. The number of images displayed on the list screen is determined according to attributes representing the user's skill level and characteristics.

When the user selects any moving image 1401a on the list screen using the operation unit 105 and receives a playback instruction from the playback button 1402, the control unit 201 uses the display generation unit 2102 and the display management unit 2103 to Play the selected image.

Thereafter, the user returns to his or her desk in the office and moves the cursor to any work application displayed on the display 1001 or to any position in the work application using the operation unit 305 or the audio input/output unit 306 of the separate device 108. When the control unit 201 receives a press of the link button 1401 on the list screen from the user via the operation unit 105, the selected image is moved to the position of the cursor on the screen displayed on the display 1001 and inserted. be done.

Specifically, when the control unit 201 receives a press of the link button 1403 on the list screen from the user, the control unit 201 transmits the selected moving image 1401a to the work application running on the separate device 108 via the communication unit 202. V. The moving image to be transmitted may be edited on the display device 1300.

The control unit 301 of the separate device 108 that has received the moving image 1401a starts a work application and copies the moving image data to a buffer area that can be shared by multiple applications, such as a clipboard. When the control unit 301 of the separate device 108 receives an instruction from the user to paste the buffer area on the work application by pressing the link button 1402 of the display device 1300, the control unit 301 inserts a moving image 1404 at the position of the cursor.

With the configuration and method described above, users can convert information on various media that they refer to into data at any time, and easily associate it with the content they are currently working on.

Here, in the above embodiment, the user selects the pasting position by moving the cursor using the operation unit 305 and the audio input/output unit 306 of the separate device 108, but as described above, the eye tracking of the separate device 108 The pasting position may be selected using the user's line of sight position determined by the section 3112. Further, in the above embodiment, the moving image itself captured by the display device 1300 is pasted, but any information indicating the location of the moving image, such as link information such as a path name to the location where the moving image is stored, may be used. The location where the moving image is stored may be, for example, the storage unit 210 of the display device 1300 or the storage unit 310 of the separate device 108.

In this way, according to each of the above embodiments, it is possible to adjust the display method of the screen when displaying information corresponding to the skills and situation such as the characteristics of the user. On the other hand, it is possible to adjust the display position and display content according to the user's skills and work content, allowing the user to perform work efficiently and with high precision. Furthermore, it is possible to effectively increase the amount of information displayed to the user and at the same time save space, thereby appropriately supporting the user's understanding of the displayed content.

DESCRIPTION OF SYMBOLS 101...Display device, 102...Sensor part, 103...Display part, 104...Light source part, 105...Operation part, 106...Communication line, 107...PC, 108...Mobile terminal, 109...Internet, 110...Cloud service 201...Control Department, 202...Communication department, 203...Time management section, 204...Power supply section, 205...Operation section, 206...Audio input/output section, 207...Display control section, 208...Display section, 209...AI engine group, 210...Storage 211...Sensor group 2101...Authentication unit, 2102...Display generation unit, 2013...Display management unit, 2014...Skill management unit, 2015...User state management unit, 2016...Optimum expression conversion unit 2111...Imaging unit, 2113...Inertia Measurement unit, 2114...Environment measurement unit, 2115...Vital measurement unit, 3112...Eye tracking unit, 4101...User management unit, 4102...Screen display condition learning unit, 4103...Skill learning unit, 4104...Work history management unit 900...User Information, 9000...Work history, 9100...Skill information

Claims (8)

A display device wearable by a user, the display device comprising:
a first display section;
a skill management department that manages information regarding user skills and characteristics;
a first control unit that displays a virtual screen on the first display unit;
a communication unit that communicates with other devices;
The first control unit includes:
Adjusting at least one of the number of screens displayed on the first display unit, display position, display size, and work content according to the user's skills and work content managed by the skill management unit, and
When sharing the screen with multiple people via the communication section, the first display includes a summary screen that aggregates communication history for a certain period of time, and a screen that displays indirect content that is not used in the current communication. to be displayed in the
A display device characterized by: The skill management department is
As information regarding the user's skills,
At least information indicating the work history and work history of the user, or information on the number of screens displayed for the work performed by the user, screen size, applications used, type and time of input operation, and work time. manage a piece of information,
The display device according to claim 1, characterized in that: The display device includes a vital measurement unit,
The vitals measurement unit acquires physical information of the user and estimates at least one of the user's physical condition, emotions, and fatigue level.
The display device according to claim 1, characterized in that: A display control system comprising a first display device wearable by a user and a second display device capable of communicating with the first display device,
The first display device includes:
a first display section;
a first control unit that displays a first screen that is a virtual screen on the first display unit;
The second display device includes:
a second display section;
a skill management department that manages information regarding user skills and characteristics;
a second control unit that displays a second screen that is a real screen on the second display unit;
a communication unit that communicates with other devices;
The second control unit includes:
When adjusting the display method of the first screen and/or the second screen on the first display section and/or the second display section,
Adjusting at least one of the number, display position, display size, and work content of the first screen and/or the second screen according to the skill and work content of the user managed by the skill management unit. With,
When sharing the screen with multiple people via the communication section, the first display includes a summary screen that aggregates communication history for a certain period of time, and a screen that displays indirect content that is not used in the current communication. to be displayed in the
A display control system characterized by: The skill management department is
As information regarding the user's skills,
At least information indicating the work history and work history of the user, or information on the number of screens displayed for the work performed by the user, screen size, applications used, type and time of input operation, and work time. manage a piece of information,
5. The display control system according to claim 4. The second display device includes a user status management section,
The user condition management unit manages the physical condition and work condition of the user,
The second control unit controls the first screen and/or the second screen based on the user's skills managed by the skill management unit and the user's physical condition managed by the user status management unit. adjusting the display method on the first display section and/or the second display section;
5. The display control system according to claim 4. The first display device includes a vital measurement unit,
The vitals measuring unit acquires physical information of the user and estimates at least one of the user's physical condition, emotions, and fatigue level,
The first control unit transmits information acquired or estimated by the vitals measurement unit to the second display device,
The user state management unit manages the user's physical state and work state, the user's physical information received from the first display device, and the estimated information,
The second control unit controls the first screen and/or the second screen based on the user's skills managed by the skill management unit and the user's physical condition managed by the user status management unit. adjusting the display method on the first display section and/or the second display section;
7. The display control system according to claim 6 . The second control unit includes:
If it is determined that the degree of fatigue of the user is high based on the physical information of the user managed by the user condition management unit and the estimated information, display on the first display unit and/or the second display unit. adjust the text and screen size,
The display control system according to claim 7, characterized in that:

Images