JP2024055512A - Display method and system - Google Patents

Abstract

The arrangement of information can be easily determined.
[Solution] The display method displays registration information associated with a parent object in three-dimensional space together with the parent object from the subjective viewpoint of an observer, and includes an energy map generation process that creates one or more energy maps, which are data on two-dimensional planes that indicate areas suitable for placing the registration information, at the respective viewpoint positions of one or more observers, a reference position determination process that sets a reference position for displaying the registration information at a second position in the energy map that has lower energy than the first position, and a drawing process that draws the registration information drawn in an area including the parent object and the reference position from the subjective viewpoint of the observer.
[Selected Figure] Figure 1

Description

The present invention relates to a display method and a display system.

Configurations for simultaneously viewing objects and information in a three-dimensional space are widely known. Patent Document 1 discloses an information processing device that includes a control unit that controls the display of a character string represented by text information related to an object included in a content image displayed in a display area, and the control unit determines a placement area in the display area in which the character string is placed based on the position of the object relative to the display area.

International Publication No. 2019/181488

The invention described in Patent Document 1 leaves room for improvement in the method for determining the display position of information.

A display method according to a first aspect of the present invention is a display method for displaying registration information associated with a parent object in a three-dimensional space together with the parent object from an observer's subjective viewpoint, and includes an energy map generation process for creating one or more energy maps, which are data on a two-dimensional plane indicating an area suitable for arranging the registration information, at each of the viewpoint positions of one or more of the observers, a reference position determination process for setting a reference position for displaying the registration information at a second position in the energy map having lower energy than a first position, and a drawing process for drawing the registration information drawn in an area including the parent object and the reference position from the observer's subjective viewpoint.
A display system according to a second aspect of the present invention is a display system that displays registration information associated with a parent object in a three-dimensional space together with the parent object from an observer's subjective viewpoint, and includes: an energy map generation unit that creates one or more energy maps, which are data on a two-dimensional plane that indicates an area suitable for arranging the registration information, at each viewpoint position of one or more of the observers; a reference position determination unit that sets a reference position for displaying the registration information at a second position in the energy map that has lower energy than a first position; and an image generation unit that renders the registration information drawn in an area including the parent object and the reference position from the observer's subjective viewpoint.

The present invention makes it easy to determine the placement of information.

Overall configuration of the information presentation system FIG. 1 shows an example of an object table. FIG. 1 is a diagram showing an example of an information table and a user table. Hardware configuration diagram of a computing device representing a server and a client A diagram explaining the situational setting that is the premise for explaining energy images. Diagram explaining energy images Flowchart showing the process of the information placement unit Flowchart showing energy image generation processing Flowchart showing reference position determination processing Flowchart showing individual vector calculation processing Flowchart showing the evaluation energy calculation process Overall configuration diagram of an information presentation system according to a first modified example Overall configuration diagram of an information presentation system according to a second modified example 1 is a diagram showing an overall configuration of an information presentation system according to a second embodiment of the present invention; A diagram showing the worker Diagram showing the worker's head 11 is a flowchart showing an overview of a process performed by an information presentation system according to a second embodiment of the present invention. FIG. 13 is a diagram showing an example of a work record table. FIG. 1 shows an example of an image table and an audio table. FIG. 1 shows an example of a procedure table and a work posture table. Detailed configuration diagram of the recording section and post-mortem analysis section A flowchart showing a risk calculation process by a risk judgment program. FIG. 13 is a diagram showing a display of registration information in a browsing phase according to the second embodiment; A diagram showing an example of a display template A diagram showing the replay explanation video seen from behind the worker A diagram showing the replay explanation video from the worker's side perspective. 13 is a flowchart showing an outline of a process performed by an information presentation system according to a third embodiment.

-First embodiment-
A first embodiment of the display method will be described below with reference to FIGS.

In this embodiment, a method of presenting information when viewing a virtual space from a first-person perspective is described. However, as shown in a modified example described later, this is also effective in cases where a computer image is superimposed on a real-world image, a so-called augmented reality. In this embodiment, the registration information presented to the user is displayed in an area called an information presentation window. However, this name is for convenience, and the window may be made transparent to display only the registration information. For ease of explanation, the information presentation window is described as rectangular, but it may also be polygonal or elliptical.

(overall structure)
FIG. 1 is an overall configuration diagram of an information presentation system 1. The information presentation system 1 includes a server 10 and one or more clients 20. The server 10 and each client 20 are connected by a wireless or wired network. The distance between the server 10 and the clients 20 does not matter, and they may be in the same room or several thousand km apart. The distance between the clients 20 does not matter either. Although three clients 202 and only one user U are shown in FIG. 1, there is no limit to the number of clients 20 and users U included in the information presentation system 1. The same number of clients 20 and users U are included in the information presentation system 1. In the following, the user U is also called an "observer."

Each client 20 is used by an individual user U who uses an input device 961 and an image display device 962. The user U freely moves around and views the virtual space (hereinafter also referred to as a "specific space") generated by the server 10. That is, the server 10 generates an image of the virtual space from the viewpoint of each user U, i.e., the first-person viewpoint of the user U, and transmits it to each client 20. The client 20 transmits movement commands and viewpoint operation commands by the user U to the server 10. The server 10 updates the position and line of sight of each user based on the movement commands and viewpoint operation commands transmitted from the client 20, and generates an image of the virtual space corresponding to the updated viewpoint and transmits it to each client 20.

The server 10 comprises a table update unit 11, an information placement unit 12, an image generation unit 13, and a storage device 14. An object table 1000, an information table 1100, and a user table 1200 are stored in the storage device 14. All data stored in the storage device 14 is pre-recorded, except for the viewpoint position 1204 and line of sight direction 1205 in the user table 1200. The table update unit 11 updates the user table 1200 based on movement commands and line of sight operation commands sent from the client 20.

The information placement unit 12 places the registration information stored in the information table 1100 in the virtual space. The information placement unit 12 includes an energy map generation unit 12a and a reference position determination unit 12b. The detailed operation of the information placement unit 12 will be described later. The image generation unit 13 generates an image of the virtual space based on the viewpoint position and line of sight direction of each user U and transmits it to each client 20.

The storage device 14 is a non-volatile information storage device such as a hard disk drive. The object table 1000 stores data such as the position, size, and appearance of objects that exist in the virtual space. The data stored in the object table 1000 is read by the image generation unit 13. The information table 1100 stores various pieces of information in the virtual space as registered information. The data stored in the information table 1100 is read by the image generation unit 13 when it is placed in the virtual space by the information placement unit 12. In other words, if the data stored in the object table 1000 is within the field of view of each user U, it is used for the rendering process of the image generation unit 13.

The user table 1200 stores data such as an avatar, which is the appearance in the virtual space for each user U, and the viewpoint position and viewpoint direction. The viewpoint position and viewpoint direction in the user table 1200 are updated as appropriate by the table update unit 11. Note that a predetermined fixed value is adopted as the viewing angle of the user U, so that the field of view of the user U can be calculated from the viewpoint position and viewpoint direction. Specific examples of the object table 1000, the information table 1100, and the user table 1200 will be described later.

The client 20 includes a command transmission unit 21 and a video output unit 22. The command transmission unit 21 transmits movement commands and gaze operation commands to the server 10 based on the output of an input device 961 operated by the user U. The video output unit 22 receives an image of the virtual space from the server 10 and outputs the received image to the input device 961.

The user U operates the input device 961 and the image display device 962. The input device 961 transmits operation commands for the user U's movement and gaze in the virtual space to the client 20. The input device 961 is a combination of a mouse and keyboard, a joystick, or an external camera and computing unit. When a mouse and keyboard are used, when the user U presses a specific key on the keyboard, a command to move forward, backward, left, or right is output to the client 20, and when the user U operates the mouse, a command to move the gaze is output to the client 20. When a joystick is used, the movement command and gaze movement are output to the client 20 depending on the tilt direction of the two analog sticks provided on the joystick.

When an external camera and computing unit are used, the camera observing the movements of user U recognizes the movements of user U and the movements of the user U's head, and movement commands and gaze movements are output to client 20. In this case, the camera may be attached integrally to user U, or may be fixed in a position where it can capture an image of user U. The image display device 962 is, for example, a stationary liquid crystal display or a head-mounted display. The functions of the computing unit described here may be realized by client 20.

(table)
2 is a diagram showing an example of an object table 1000. The object table 1000 has a plurality of records, and each record has fields of an element model ID 1001, 3D model data 1002, an item name 1003, a bounding box 1004, a center position 1005, an importance 1006, a related keyword tag 1007, and a saved information ID 1008. The element model ID 1001 stores an element model ID that is an identifier unique to an object. The 3D model data 1002 stores a URI indicating a location where a 3D model of the object is saved. The item name 1003 stores a name of the object. That is, although there may be a plurality of records with the same item name 1003 in the object table 1000, there are no records with the same element model ID 1001.

The bounding box 1004 stores two three-dimensional coordinates that are diagonals of a rectangular parallelepiped region that contains the object. The center position 1005 stores the three-dimensional coordinates that are the center position of the object. The importance 1006 stores the importance of the object. Note that in this embodiment, the importance 1006 is expressed as an integer value, with a larger value indicating greater importance, but the importance 1006 may also be a ranking result such as "A", "B", or "C". The related keyword tag 1007 stores keywords related to the object. The related saved information ID 1007 stores a saved information ID that is an identifier for saved information related to the object.

Figure 3 shows an example of the information table 1100 and the user table 1200. The information table 1100 has multiple records, and each record has fields for a saved information ID 1101, importance 1102, template ID 1103, keyword list 1104, description 1105, image 1106, related saved information 1107, and parent model ID 1108. The saved information ID 1101 stores an identifier for registered information. The importance 1102 stores the importance of the registered information.

Template ID 1103 stores the identifier of the template when displaying the registered information. Keyword list 1104 stores key sites related to the registered information. Description 1105 stores text that explains the registered information. Image 1106 stores a URI indicating the location where an image related to the registered information is saved. Related saved information 1107 stores the saved information ID of other registered information related to that registered information. Parent model ID 1108 stores the element model ID 1001 of the object with which the registered information is associated.

The user table 1200 has multiple records, and each record has fields for a user ID 1201, a user name 1202, an avatar model 1203, a viewpoint position 1204, a line of sight direction 1205, and a depth preference 1206. The user ID 1201 stores an identifier for identifying the user. The user name 1202 stores the name of the user. The avatar model 1203 stores a URI indicating the location where a 3D model showing the user's appearance is saved. The viewpoint position 1204 stores three-dimensional coordinates indicating the user's latest viewpoint position. The line of sight direction 1205 stores data indicating the latest line of sight direction of the user U, such as the yaw angle, roll angle, and pitch angle based on a specified posture. The depth preference 1206 stores the distance at which each user U prefers to display information.

(Hardware configuration)
4 is a hardware configuration diagram of a computing device 40 representing the server 10 and the client 20. Although the hardware configurations of the server 10 and the client 20 are not necessarily the same, the server 10 and the client 20 have in common the hardware of the computing device 40 described below. The computing device 40 includes a CPU 41 which is a central processing unit, a ROM 42 which is a read-only storage device, a RAM 43 which is a readable and writable storage device, an input/output interface 44 which is a contact point with a user interface, and a communication device 45. The CPU 41 loads a program stored in the ROM 42 into the RAM 43 and executes it to perform various calculations.

The arithmetic unit 40 may be realized by a field programmable gate array (FPGA), which is a rewritable logic circuit, or an application specific integrated circuit (ASIC), which is an integrated circuit for a specific application, instead of the combination of the CPU 41, ROM 42, and RAM 43. The arithmetic unit 40 may also be realized by a combination of different configurations, for example, a combination of the CPU 41, ROM 42, RAM 43, and an FPGA, instead of the combination of the CPU 41, ROM 42, and RAM 43.

The input/output interface 44 is a general-purpose communication port and video signal output device to which various peripheral devices can be connected. For example, a mouse, keyboard, joystick, camera, display, etc. can be connected to the input/output interface 44. The communication device 45 is a communication module compatible with, for example, IEEE802.3 or IEEE802.11. The table update unit 11 and information placement unit 12 of the server 10 are realized by a program executed by the CPU 41. The video generation unit 13 is realized by a program executed by the CPU 41 and the communication device 45. The command transmission unit 21 of the client 20 is realized by a program executed by the CPU 41, the input/output interface 44, and the communication device 45. Similarly, the video output unit 22 is realized by a program executed by the CPU 41, the input/output interface 44, and the communication device 45.

(Energy image)
The energy image will be described with reference to Fig. 5 and Fig. 6. Specifically, Fig. 5 is a diagram for explaining a situation setting that is a premise for explaining the energy image, and Fig. 6 is a diagram for explaining the energy image. In Fig. 5, an object 901 is arranged in front of the line of sight of the user u in the virtual space, and an object 902 is arranged on the left side. Also, an information presentation window 904 in which information related to the object 901 is presented is arranged on the right side of the object 901. In Fig. 5, the dashed line indicates the field of view of the user u in the virtual space. In this case, the image generation unit 13 generated the scene image 905 shown in the lower part of Fig. 5 from the subjective viewpoint of the user u in the virtual space. Assuming a situation in which the information placement unit 12 places another information presentation window related to the information presentation window 904 immediately after this scene image 905 is generated, the energy image will be described with reference to the next Fig. 6.

The energy image 80 is an image used in calculations to determine the position at which the next information presentation window will be displayed from the viewpoint of each user U. The energy image 80 is a collection of data for each grid-like region. However, the energy image is merely called an "image" for convenience, and does not have to be an image file such as a bitmap or JPEG. The energy image 80 can also be called an "energy map." In this embodiment, each pixel in the energy image has a value of, for example, -100 to +100. The numerical value of each pixel is called the "energy amount." The energy image indicates that the smaller the pixel's numerical value, i.e., the lower the energy amount, the more suitable it is for arranging an information presentation window. For example, a position with an energy amount of -10 is more suitable for arranging an information presentation window than a position with an energy amount of zero.

The energy image 80 is created based on the object map 81, the target map 82, and the adjustment map 83. In FIG. 6, areas without color indicate the initial value of zero, positive values are indicated by diagonal hatching, and negative values are indicated by point cloud hatching. Note that point cloud hatching is also called "point filling" or "dot hatching." The number of pixels in the energy image 80, the object map 81, the target map 82, and the adjustment map 83 are the same.

The object map 81 is a map in which a large positive value is set in the area where an object that is present in the field of view and has an importance higher than a predetermined threshold value is present. However, the objects referred to here also include existing information presentation windows. In the object map 81 shown in FIG. 6, a large positive value is set in the position where each object was located in the scene image 905. Since the object map 81 defines a positive amount of energy, a newly placed information presentation window is placed so as to avoid the area set in the object map 81.

The object map 82 is a map in which a negative value is set for an object to which a newly placed information presentation window is related. For example, a negative value is set for a perfect circle or ellipse that is centered on the center of the related object and that contains the object. Note that the star in the object map 82 shown in Figure 6 indicates the center of the object 901. The object map 82 allows the new information presentation window to be placed near the related object.

The adjustment map 83 is set when a new information presentation window is related to another existing information presentation window. In the adjustment map 83, negative values are set to the top, bottom, left, and right of the related existing information presentation window 904. No particular value is set for the position of the information presentation window 904, which remains at the initial value of zero. The adjustment map 83 makes it easier for the new information presentation window to be positioned near the other existing information presentation window, specifically above, below, left, or right.

The negative values set in the adjustment map 83 do not have to be the same. For example, the right side of the existing information presentation window may be set to the largest negative value, the left side to the next largest negative value, the top to the third largest negative value, and the bottom to the smallest negative value. With this skewed distribution, the new information presentation window is most likely to be placed at the bottom of the existing information presentation window among the top, bottom, left, and right.

These three maps, specifically the object map 81, target map 82, and adjustment map 83, are superimposed and then blurred to form the energy image 80. Note that if the new information presentation window is not related to any other existing information presentation window, there is no adjustment map 83, so the object map 81 and target map 82 are superimposed and blurred to form the energy image 80. However, due to drawing restrictions, the blurring process in the energy image 80 cannot be shown in Figure 6. Note that superimposing maps means adding up the values of corresponding pixels. The blurring process can be performed using Gaussian blur, for example.

Energy graph 911 shown at the bottom of energy image 80 is a diagram showing the amount of energy at the position of dashed line 910 in energy image 80. Reference characters 912 to 915 in energy graph 911 are reference characters indicating each region in the horizontal direction in the figure. The corresponding region in energy image 80 for region 912 is white, i.e., zero. However, adjacent region 913 is set to a large positive value in object map 81, so the blurring process causes a smooth increase from region 912 to region 913. Region 914 is set to a large positive value in object map 81, a negative value in target object map 82, and a negative value in adjustment map 83, so the energy amount is smaller than that of region 913.

(flowchart)
7 is a flowchart showing the processing of the information placement unit 12. First, in step S301, the information placement unit 12 identifies the current scene, i.e., the location and the users associated with it. For example, a specific range in the virtual space is designated as the current location from outside, and the information placement unit 12 identifies users whose viewpoint positions 1204 in the user table 1200 are within that range. In the following step S302, the information placement unit 12 lists the registered information to be displayed.

There is no upper limit to the number of registration information items to be displayed, but for example, the 10 most important items are listed among the registration information items of objects located at the location identified in step S301. The information placement unit 12 may list all registration information items whose importance 1102 is higher than a predetermined threshold and whose parent object is within a predetermined distance from the current location of the user U. In the following step S303, the information placement unit 12 selects one of the unprocessed registration information items from the registration information items listed in step S302 as information to be processed. In the following step S304, the information placement unit 12 lists the users identified in step S301.

In the following step S305, the information placement unit 12 selects one of the unprocessed users from among the users listed in step S304 as a user to be processed, and proceeds to step S306. In step S306, the energy map generation unit 12a of the information placement unit 12 performs an energy image generation process, and proceeds to step S307. Details of the energy image generation process will be described later with reference to FIG. 8. In step S307, the information placement unit 12 determines whether all users listed in step S304 have been processed. If the information placement unit 12 determines that all users have been processed, it proceeds to step S308, and if it determines that there is an unprocessed user, it returns to step S305.

In step S308, the reference position determination unit 12b of the information placement unit 12 executes a reference position determination process and proceeds to step S309. In step S309, the information placement unit 12 places an information presentation window for each user U based on the reference position determined in step S803. However, the information placement unit 12 does not need to place the information presentation window in the same position and in the same orientation for all users U, as long as the information presentation window having a spatial extent is placed so as to include the reference position. In other words, the normal of the information presentation window may be parallel to or coincident with the line of sight of each user U, and the position may be adjusted by referring to the energy image for each user U.

In the next step S310, the information placement unit 12 determines whether all of the registered information listed in step S302 has been processed. If the information placement unit 12 determines that all of the registered information listed has been processed, it ends the process shown in FIG. 7, and if it determines that there is unprocessed registered information, it returns to step S303.

Figure 8 is a flowchart showing the energy image generation process by the information placement unit 12, and also shows details of step S306 in Figure 7. This flowchart is executed in a state where the target user and the target information have been specified. In other words, the process shown in Figure 8 is executed a number of times equal to the product of the number of target users and the number of target information. This flowchart is executed by the energy map generation unit 12a of the information placement unit 12. First, in step S321, the information placement unit 12 refers to the user table 1200 to specify the viewpoint position and line of sight direction of the target user. In the following step S322, the information placement unit 12 calculates a projection matrix indicating the line of sight of the target user in the virtual space based on the viewpoint position and line of sight direction specified in step S321.

In the next step S323, the information placement unit 12 generates an object map 81 for avoiding objects with high importance. Specifically, the information placement unit 12 applies a projection matrix to the bounding box 1004 of an object that exists at the location specified in step S301 and has an importance 1006 higher than a predetermined threshold, and generates an object map 81 in which a large positive value is set in that area. Note that the value set in the projection position of each object may be set to a larger value as the importance of each object increases. In the next step S324, the information placement unit 12 generates an object map 82 for approaching the parent object of the processing target information. Specifically, the information placement unit 12 applies a projection matrix to the center position 1005 of an object that has the value of the parent model ID 1108 of the processing target information as the element model ID 1001, and generates an object map 82 in which a negative value is set in a perfect circle or ellipse area centered on the projected position.

In the next step S325, the information placement unit 12 generates an adjustment map 83 for adjusting the position with the existing parent window. Specifically, the information placement unit 12 identifies the parent window by referring to the related saved information ID 1107 in the information to be processed, and sets negative values to the top, bottom, left, and right of the display position. In the next step S326, the information placement unit 12 overlays the three maps generated in steps S323 to S325, and further applies a blurring process to generate an energy image 80, and ends the process shown in FIG. 8. In the following, the energy image 80 generated in this step is called Euw. The subscript "u" is the user identifier, and the subscript "w" is the identifier of the information presentation window. Note that a specific example of steps S323 to S326 is as shown in FIG. 6.

9 is a flowchart showing the reference position determination process by the information placement unit 12, and is also a diagram showing details of step S308 in FIG. 7. This flowchart is executed by the reference position determination unit 12b of the information placement unit 12. First, in step S330, if the information to be processed is already displayed, the information placement unit 12 records the three-dimensional coordinates that are the current reference position as the past display position p_old. If the information to be processed is not yet displayed, no processing is performed. In the following step S331, the information placement unit 12 sets the tentative reference position to the initial position. If the information to be processed is already displayed, this initial position is the reference position, and if the information to be processed is not yet displayed, the initial position is set to the origin (0,0,0) or the center position 1005 of the parent object of the presentation information to be processed. In the following step S332, the information placement unit 12 lists the users identified in step S301.

In the following step S333, the information placement unit 12 selects one of the unprocessed users from the users listed in step S333 as a user to be processed, and proceeds to step S334. In step S334, the information placement unit 12 executes an individual vector calculation process, which will be described later. In the individual vector calculation process, a three-dimensional vector that moves the optimal provisional reference position for each user is calculated. In the following step S335, the information placement unit 12 determines whether all of the registered information listed in step S332 has been processed. If the information placement unit 12 determines that all of the registered information listed has been processed, the process proceeds to step S336, and if the information placement unit 12 determines that there is unprocessed registered information, the process returns to step S333.

In step S336, the information placement unit 12 adds up the individual vectors of all users U. In the above-mentioned step S334, a three-dimensional vector is calculated as an individual vector for each user to be processed, so in this step, these three-dimensional vectors are simply added together. In the following step S337, the information placement unit 12 moves the tentative reference position according to the summation result in step S336. For example, if the tentative reference position immediately before executing step S337 is (100, 50, 10) and the summation result in step S336 is (0.2, -0.3, 0.1), the new tentative reference position will be (100.2, 49.7, 10.1).

In the next step S338, the information placement unit 12 calculates the evaluation energy En at the new tentative reference position. However, if this is the first time step S338 is executed and the information to be processed is already displayed, the information placement unit 12 also calculates the evaluation energy E0 at the previous display position p_old. The method of calculating the evaluation energy En will be described later. In the next step S339, the information placement unit 12 determines whether the amount of change in the evaluation energy En, i.e., the absolute value of the difference between the evaluation energy at the previous tentative reference position and the evaluation energy at the current tentative reference position, is equal to or less than a threshold value. If the information placement unit 12 determines that the amount of change in the evaluation energy En is equal to or less than a predetermined threshold value, it determines that the calculation has ended and ends the process shown in FIG. 9, but if it determines that the amount of movement of the tentative reference position is greater than the predetermined threshold value, it returns to step S332.

Figure 10 is a flowchart showing the individual vector calculation process, and also shows details of step S334 in Figure 9. First, in step S341, the information placement unit 12 calculates a tentative reference position in the energy image of the user to be processed. The above-mentioned projection matrix can be used for this calculation. In the following step S342, the information placement unit 12 determines whether the calculated tentative reference position exists in the energy image. If the information placement unit 12 determines that the tentative reference position exists in the energy image 80, the process proceeds to step S343, and if the information placement unit 12 determines that the tentative reference position does not exist in the energy image 80, the process proceeds to step S350.

In step S343, the information placement unit 12 reads the amount of energy at the tentative reference position in the energy image 80 and records it as a current value Ec. In the following step S344, the information placement unit 12 moves the tentative reference position in the energy image 80 by small distances in multiple directions and reads the amount of energy at the position after the movement. These multiple directions are, for example, a total of eight directions including up, down, left, right, and diagonal directions in between in the energy image 80. In the following step S345, the information placement unit 12 determines the direction that results in the smallest amount of energy among the amounts of energy read in step S344 as the optimal direction of movement, and calculates the reduction rate dE. The reduction rate dE is calculated by the following formula 1, where Em is the minimum value of the amount of energy read in step S344.

dE = (Ec-Em)/Ec... (Equation 1)

In the next step S346, the information placement unit 12 calculates the distance d_tgt from the tentative reference position to the parent object of the information to be processed. Specifically, the distance d_tgt is the shortest distance between the viewpoint position 1204 and the bounding box 1004 of the parent object of the information to be processed. In the next step S347, the information placement unit 12 calculates the evaluation distance Ed using the following Equations 2 to 4.

Ed = E_0 * (d_pref - d) ... (Formula 2)

Ed += E_1 if d < d_min ... (Equation 3)

Ed - = E_2 if d > (d_tgt - D_tgt_near) ... (Equation 4)

In equations 2 to 4, E_0 to E_2 are predetermined constants, d is the distance between the viewpoint position 1204 and the provisional reference position to which the initial value was set in step S331 and which was moved in step S337, d_pref is the value of depth preference 1206 in user table 1200, and d_min is a predetermined threshold value for limiting close distances. Also, D_tgt_near in equation 4 is the distance to another object that is closer to the parent object of the information to be processed than is seen from the viewpoint position 1204. Equations 2 to 4 can be explained as follows. That is, the approximate value of the evaluation distance Ed is determined by equation 2, and if d is smaller than d_min, equation 3 is corrected, and if d is larger than the difference between d_tgt and D_tgt_near, equation 4 is corrected.

In the next step S348, the information placement unit 12 calculates a vector Vz pointing from the viewpoint position of the processing target user in the user table 1200 to the parent object of the processing target information. In the next step S349, the information placement unit 12 calculates an individual vector. This individual vector Vbest is calculated using the following formula 2, and the process shown in FIG. 10 is terminated. Note that the single "x" in formula 5 is written as an operation symbol indicating multiplication.

Vbest = dE x Vxy + Ed x Vz ... (Equation 5)

In step S350, the information placement unit 12 sets the individual vectors to zero and ends the process shown in FIG. 10.

Figure 11 is a flowchart showing the evaluation energy calculation process, and also shows details of step S338 in Figure 9. First, in step S361, the information placement unit 12 initializes a variable UE to zero. In the following step S362, the information placement unit 12 lists the users identified in step S301. In the following step S363, the information placement unit 12 selects one of the unprocessed users from the users listed in step S362 as a user to be processed, and proceeds to step S364. In step S364, the information placement unit 12 identifies a tentative reference position on the energy image for the user to be processed.

In the next step S365, the information placement unit 12 adds the energy of the tentative reference position to the variable EU. That is, the energy of each user at the tentative reference position is added to the variable EU. In the next step S366, the information placement unit 12 determines whether or not all users have been processed, and if it is determined that all users have been processed, the process proceeds to step S367, and if it is determined that there is an unprocessed user, the process returns to step S363.

In step S367, the information placement unit 12 determines whether or not a past display position exists. The determination in this step is also a determination of whether or not the information to be processed has already been displayed. If the information placement unit 12 determines that a past display position exists, the process proceeds to step S368, and if it determines that a past display position does not exist, the process proceeds to step S369. In step S368, the information placement unit 12 sets the sum of the variable EU, which was repeatedly added in step S365, and the function E shown in Equation 6 as the evaluation energy En, and ends the process shown in FIG. 11.

E(p_new, p_old) = (p_new - p_old)^2 x E_constant ... (Equation 6)

In equation 6, p_new is the tentative reference position, and p_old is the previously displayed position. That is, equation 6 multiplies the square of the distance in three-dimensional space between the tentative reference position and the previously displayed position by a predetermined constant. In step S369, the information placement unit 12 sets the variable EU, which was repeatedly added in step S365, itself as the evaluation energy En, and ends the process shown in FIG. 11.

According to the above-described first embodiment, the following advantageous effects can be obtained.
(1) The display method executed by the information presentation system 1 displays registration information registered in the registration information table 1100, associated with an object in a three-dimensional space, together with the object from the subjective viewpoint of the observer. This display method includes an energy map generation process ( FIG. 8 ) for creating one or more energy images, which are data on a two-dimensional plane showing an area suitable for arranging the registration information, at the viewpoint positions of one or more users U, a reference position determination process ( FIG. 9 ) for setting a reference position for displaying the registration information at a second position in the energy image that has lower energy than the first position, and a drawing process executed by the image generation unit 13 for drawing the registration information drawn in an area including the parent object and the reference position from the subjective viewpoint of the observer. Therefore, the arrangement of the registration information can be easily determined.

(2) In the energy map generation process, the amount of energy is set high by assigning a positive amount of energy to the area of the parent object as shown in the object map 81, and the amount of energy is set low by assigning a negative amount of energy to the periphery of the parent object as shown in the target object map 82. (3) In the energy map generation process, the amount of energy is set high by assigning a positive amount of energy to the area of other registered information that is already displayed as shown in the object map 81. (4) In the energy map generation process, the amount of energy is set low by assigning a negative amount of energy to the area near the area where other registered information related to the registered information is already displayed as shown in the adjustment map 83. (5) In the energy map generation process, the energy of the area of an object other than the parent object and whose set importance is greater than a predetermined threshold is set high. Therefore, the arrangement of the registered information can be easily determined by setting the energy high in the area where the registered information is not to be displayed by superimposing it and setting the energy low in the area where the registered information is to be displayed.

(6) In the reference position determination process, the reference position is determined so that the sum of the energy amounts at the reference positions of each energy map is minimized. Specifically, in steps S343 to S345 of FIG. 10, a desired direction in which to move the tentative reference position is determined for each energy map, and this vector amount is added up in S336 of FIG. 9 to move the tentative reference position.

(7) In the reference position determination process, a tentative reference position having three-dimensional coordinates is set in advance (S331), a position on each energy map corresponding to the tentative reference position is identified (S341), and a movement vector of the tentative reference position is set in each energy map so that the total energy amount at the position corresponding to the tentative reference position is reduced (S345). Furthermore, in the reference position determination process, the tentative reference position is updated by adding up the movement vectors calculated in each energy map for the tentative reference position (S336, S337), and repeated calculations are performed with the tentative reference position being the reference position when the change in the total energy amount at the position corresponding to the tentative reference position before and after updating the tentative reference position becomes equal to or less than a threshold (S339: YES).

(8) In the reference position determination process, the reference position is determined taking into consideration an evaluation distance Ed based on the difference between the distance d from the viewpoint position to the reference position and the distance d_pref preferred by the observer, and a vector Vz pointing from the viewpoint position toward the parent object. Therefore, the distance direction from the reference position can also be adjusted for each user U.

(9) If the registered information is already displayed based on the previous reference position (S367: YES), in the reference position determination process, the reference position is determined so that the sum of the energy amounts at the reference positions of each energy map and the value based on the distance between the previous reference position and the reference position is small (S368). This makes it possible to prevent the display position of the existing information presentation window from changing significantly.

(Variation 1)
In the first embodiment described above, the server 10 renders the video for each client 20. However, the server 10 may transmit the reference position of the information presentation window to each client 20 without performing the rendering, and each client 20 may perform the rendering.

Figure 12 is an overall configuration diagram of an information presentation system 1A in Modification 1. In the function-saving server 10A, the image generator 13 is removed from the functions of the server 10 in the first embodiment. According to Modification 1, the amount of communication from the function-saving server 10A to the client 20 can be reduced.

(Variation 2)
13 is an overall configuration diagram of an information presentation system 1B in Modification 2. The information presentation system 1B is composed of a plurality of second extended clients 20B, and each of the second extended clients 20B has not only the function of the client 20 in the first embodiment but also the function of the server 10. However, the command transmission unit 21 transmits the position information of each user to the other second extended clients 20B, not to the server 10. The other configurations and processes are the same as those in the first embodiment, and therefore will not be described. According to Modification 2, the server 10 that aggregates data is not required.

(Variation 3)
In the individual vector calculation process, movement in the depth direction relative to the energy image does not need to be taken into consideration. In this case, steps S346 to S348 in Fig. 10 are omitted, and Vbest is calculated as the product of dE and Vxy in step S349.

(Variation 4)
In step S347, the information placement unit 12 calculates the evaluation distance Ed using Equations 2 to 4. However, the information placement unit 12 may calculate the evaluation distance Ed using only Equation 2 without using Equations 3 and 4.

(Variation 5)
In the above-described first embodiment, an example of viewing a virtual space has been described. However, the configuration described in the first embodiment may be used to view a real space and superimpose a computer image on a real-world image, so-called augmented reality. However, the input device 961 in this modification is a device that detects the movement of the user U in the real world. Specifically, the input device 961 is an accelerometer that detects the movement of the user U, and a motion sensor that is attached to the body of the user U and detects the movement of the joints. However, a fixed camera that photographs the user U from a distance may be used as the input device 961. In this modification, the image generating unit 13 does not need to draw an object, and only needs to draw the registration information. In this modification, the image display device 962 is a transparent display that allows the back to be seen.

(Variation 6)
In the first embodiment described above, only one energy image 80 is generated for each user U at the same time. However, multiple energy images 80 of the user U at the same time may be created. That is, the information placement unit 12 may estimate the position and gaze direction of the user U after a predetermined time, for example, 5 seconds, based on the time-series change of the viewpoint position 1204 in the user table 1200, and further create an energy image 80 based on the viewpoint position of the user U at the estimated destination. This energy image 80 is also used in the reference position determination process (S308), so the reference position is determined taking into consideration the future position of the user U.

According to this modified example, the following effects can be obtained.
(10) In the energy map generation process, the movement of the user U is estimated, and an energy map is generated based on the viewpoint position of the observer at the estimated destination. Therefore, when the user U moves, the reference position does not change significantly, and the user U can comfortably refer to the registration information.

--Second embodiment--
A second embodiment of the information presentation system will be described with reference to Figures 14 to 26. In the following description, the same components as in the first embodiment are given the same reference numerals, and differences will be mainly described. Points that are not specifically described are the same as in the first embodiment. This embodiment differs from the first embodiment mainly in that data recording is also performed.

Figure 14 is an overall configuration diagram of an information presentation system 1C in the second embodiment. In addition to the configuration of the information presentation system 1 in the first embodiment, it includes a fixed camera 971, a microphone 972, earphones 973, and a wearable camera 974. The server 10C further includes a recording unit 16, a post-analysis unit 17, and a context evaluation unit 18, and the storage device 14 further stores a work record table 1300, an image table 1400, an audio table 1500, and a procedure table 1600.

The worker W wearing the microphone 972, earphones 973, and wearable camera 974 may be the same as or different from the user U. For ease of explanation, the person who is in the specific space when data is recorded is referred to as the worker W. For convenience of drawing, only one worker W is shown in FIG. 14, but there may be two or more workers W. The recording unit 16, the post-mortem analysis unit 17, and the context evaluation unit 18 are realized by the CPU 41 expanding a program stored in the ROM 42 into the RAM 43 and executing it, similar to the table update unit 11, etc.

The object table 1000 and the procedure table 1600 are stored in advance in the storage device 14. The registration information table 1100 is created by the recording unit 16 and the post-analysis unit 17. The work record table 1300, the image table 1400, the audio table 1500, and the work posture table 1700 are created by the recording unit 16. The viewpoint position 1204 and the line of sight direction 1205 of the user table 1200 are updated by the table update unit 11, and other information of the user table 1200 is created in advance.

The worker W performs work in a specific space, which is a space in which the objects listed on the object table 1000 exist. The specific space is, for example, the interior of a building, a bus, a train, a passenger car, a ship, etc. The wearable camera 974 is, for example, an action camera attached to smart glasses or a helmet worn by the worker W. The wearable camera 974 may further include a function for calculating depth information, which is distance in the depth direction. The depth information may be realized by a combination of a laser range finder and a camera, may be realized using images captured by multiple cameras installed at a predetermined baseline distance apart, or may be realized using multiple images captured by a single camera at different times.

The fixed camera 971 is one or more cameras. The image captured by the fixed camera 971 is hereinafter referred to as a "fixed camera image." The fixed camera 971 is placed in a space where the object described on the object table 1000 exists. The installation position and installation angle of the fixed camera 971 are known. The voice data recorded by the microphone 972 and the image captured by the wearable camera 974 (hereinafter referred to as a "subjective image") are sent to the server 10C. The server 10C can output voice to the earphone 973. The microphone 972, the earphone 973, and the wearable camera 974 are linked in advance to the worker W who uses them. For example, when the server 10C receives voice data from the microphone 972, it can determine which worker W the voice belongs to. The fixed camera image and the subjective image may be either a still image or a video.

Figure 15 shows the appearance of worker W, and corresponds to the registration phase described below. In Figure 15, worker W is on a train, working by touching a work object with his or her hands. Although it is not shown in Figure 15 due to drawing limitations, worker W has a microphone 972, earphones 973, and a wearable camera 974 fixed to his or her head. Fixed cameras 971 are placed on the upper left and right sides of worker W, and capture images of worker W.

Figure 16 is a diagram showing the head of worker W. As described above, worker W has a microphone 972, earphones 973, and a wearable camera 974 fixed to his head. In Figure 16, the wearable camera 974 captures an image from the subjective viewpoint of worker W, that is, an image of worker W reaching out his hand to touch the work object.

Figure 17 is a flowchart showing an overview of the processing of the information presentation system 1C in the second embodiment. In the second embodiment, the recording unit 16 first records the work of the worker W (S401) and registers the work log in the database, i.e., the storage device 14 (S402). Recording of the work and registration in the storage device 14 are repeated until the work by the worker W is completed. When the work by the worker W is completed (S403: YES), detection of items requiring attention and assignment of evaluation KPIs are performed (S404), and the results are written to the storage device 14 (S405). Below, S401 to S405 are referred to as the registration phase. When the registration phase is completed, narrowing of the displayed information (S406) and display of items requiring attention (S407) are repeated.

Figure 18 is a diagram showing an example of a work record table 1300. The work record table 1300 has multiple records, and each record has fields for work class 1301, work name 1302, work implementation registration ID 1303, object class 1304, object 3D model 1305, individual object ID 1306, individual object installation position 1307, object placement angle 1308, worker 1309, start time 1310, end time 1311, interruption time list 1312, image list 1313, work voice data 1314, and work voice command 1315.

Work class 1301 stores a work class number indicating the content of the work. Work name 1302 stores the name of the work. Work execution registration ID 1303 stores an identifier for the work record. In other words, when the same type of work is performed, the same values are stored in the aforementioned work class 1301 and work name 1302, but a different value is set for work execution registration ID 1303 for each work.

The object class 1304 stores the class of the object of the work. The object 3D model 1305 stores the name of the 3D model of the object to be worked on or the file name. The object individual ID 1306 stores the individual ID of the object to be worked on. The object individual installation position 307 stores the installation position of the object to be worked on. The object placement angle 1308 stores the placement angle of the object to be worked on. The worker 1309 stores the name or full name of the worker. The start time 1310 stores the time when the work started. The end time 1311 stores the time when the work ended. The interruption time list 1312 stores a list of the times when the work was interrupted. The image list 1313 stores a list of image IDs of images taken during work. The work voice data 1314 stores the voice ID of the voice data recorded during work. The work voice command 1315 stores the identifier of the work voice command.

Figure 19 shows an example of an image table 1400 and an audio table 1500. The image table 1400 has multiple records, and each record has fields for an image ID 1401, an image URL 1402, an image type 1403, a photographing device ID 1404, a photographing user ID 1405, a photographing time 1406, and a work class 1407. The image ID 1401 stores an image identifier. The work implementation registration ID 1303 and the image ID 1401 can be associated with each other by the image ID 1401 contained in the image list 1313 field in the work record table 1300.

Image URL 1402 stores the URL where the image is saved. Image type 1403 stores the type of image. Photography device ID 1404 stores the identifier of the device used to photograph the image. Photography user ID 1405 stores the ID of the user who photographed the image. Photography time 1406 stores the time the image was photographed. Work class 1407 stores a work class number indicating the content of the work. Work class 1407 is the same type of information as work class 1301 in work record table 1300.

The audio table 1500 has multiple records, and each record has fields for audio ID 1501, audio URL 1502, audio type 1503, recording device ID 1504, recording user ID 1505, recording time 1506, and work audio command 1507. The audio ID 1501 stores an identifier for the audio data. The audio ID 1501 included in the work audio data 1314 field in the work record table 1300 allows the work execution registration ID 1303 to be associated with the audio ID 1501. The audio URL 1502 stores the URL where the audio data is saved.

Audio type 1503 stores the type of audio data. Recording device ID 1504 stores the identifier of the device used to record the audio data. Recording user ID 1505 stores the ID of the user who recorded the audio data. Recording time 1506 stores the time when the audio data was recorded. Note that recording time 1506 may store any of the time when recording started, the time when recording ended, or the time when recording started and ended. Work audio command 1507 stores an identifier of the command for the work audio.

Figure 20 shows an example of a procedure table 1600 and a work posture table 1700. The procedure table 1600 has multiple records, and each record has fields for work class 1601, work name 1602, start condition 1603, target location 1604, and CAD model change 1605. The work class number indicating the content of the work is stored in the work class 1601. The name of the work is stored in the work name 1602. The start condition 1603 stores the work class that must be completed before the work can be started. If there is no work class that must be completed beforehand, "none" is stored. The target location 1604 stores ** indicating the target location of the work. The CAD model change 1605 stores data before and after the CAD model that changes due to the work.

The work posture table 1700 has multiple records, and each record has fields for a user ID 1701, acquisition time 1702, a work ID 1703, a position 1704, a base angle 1705, an estimated motion work 1706, first joint information 1707, and second joint information 1708. The user ID 1701 stores the identifier of the user who is the worker. The acquisition time 1702 stores the time when the posture information was acquired. The work ID 1703 stores the work implementation registration ID 1303 of the work record table 1300, which is an identifier of the work.

Position 1704 stores the user's position. Base angle 1705 stores a base angle. Estimated motion task 1706 stores the name of the task that is estimated to be performed by the user. First joint information 1707 and second joint information 1708 store angle information of each joint. That is, task posture table 1700 stores the position and posture at one or more times when a worker W performs a task.

(Registration phase)
21 is a detailed configuration diagram of the recording unit 16 and the post-analysis unit 17. The recording unit 16 includes a hand posture extraction program 161, a target part estimation program 162, a task content estimation program 163, a user position and posture estimation program 164, and a fixed camera position and angle estimation program 165. The hand posture extraction program 161 extracts the posture of the hand from the subjective image. The target part estimation program 162 estimates the target part from the subjective image. The task content estimation program 163 estimates the task content from the estimated object, the posture of the hand, and the depth distance in the subjective image. The user position and posture estimation program 164 estimates the position and posture of the worker W during the task from the fixed camera image and records it in the task posture table 1700. The fixed camera position and angle estimation program 165 estimates the shooting position and angle of the subjective image from the position of the fixed camera 971 and the position of the target part.

Furthermore, the recording unit 16 detects the start and end of work using the subjectively photographed images and records the times in the work record table 1300. At this time, the recording unit 16 may output a voice from the earphones 973 of the worker W as follows. That is, when the start of work is detected, the recording unit 16 outputs "Work start confirmed at work object: ddd, start time: ddd", and when the end of work is detected, the recording unit 16 outputs "Work end confirmed at work object: ddd, start time: ddd". The worker W may speak into the microphone 972, "Work object ddd, start work on ddd" or "Finish work", and the recording unit 16 may recognize the content of the speech by voice recognition and record the time of the speech in the work record table 1300.

The post-analysis unit 17 includes a KPI generation program 171, a risk judgment program 172, and a work abnormality judgment program 173. The KPI generation program 171 generates KPIs (Key Performance Indicators) from the sensing data registered in the work group. The risk judgment program 172 uses the KPIs to judge the presence or absence of registered risks. The work abnormality judgment program 173 uses the KPIs to judge abnormal values of work.

The KPI generation program 171 of the post-analysis unit 17 creates KPIs using the recorded data. For example, any of the following eight can be used as KPIs: (1) Time required for the task. (2) Task time for each step, if the task is divided into multiple steps. (3) Similarity with other cases based on the color and shape of the object. (4) ID of the object, if one has been set. (5) Number of failures when processing of the object is confirmed. (6) Error in the task position of the object. (7) Maximum, minimum, rate of change, and other values of torque fluctuation during the task, if it can be calculated using a separate sensor, etc. (8) Values obtained by analyzing the sound spectrum during the task. However, the method of calculating KPIs is not limited to these.

Of these calculation methods, image processing and deep learning can be used to find KPIs from images. Also known is a method of coloring image elements that have influenced the KPI (Explainable). The post-analysis unit 17 is not limited to the content described so far, and can calculate KPIs using various known methods.

The risk judgment program 172 of the post-analysis unit 17 uses this KPI to calculate the potential risk using one of the following two methods, and enters the value of the potential risk in the importance 1102 column of the registration information table 1100. The first method is irregularity detection by clustering. In this case, the same or similar tasks are first grouped, and if the KPI being evaluated is significantly different from other cases within the same group, it is determined that there is a risk that the work results contain problems. The magnitude of the risk can be evaluated by the size of the deviation of the outlier.

The second method is risk detection using a supervised model. In this case, if the KPI to be evaluated is included in a class similar to the problematic cases, it is determined that there is a risk that the work results contain problems. Specifically, identical or similar work is grouped, and cases that contain problem elements are picked out from the work results. The problematic cases are then grouped according to the problematic case group tags. Furthermore, a discriminator is created that separates cases that are included in the problematic cases from cases that are not included in the problematic cases. Any discrimination method such as support vector machines, random forests, and deep learning can be used for this discriminator. This discriminator can also calculate an index including the accuracy of the discrimination. When a new sample is obtained, the discriminator is used to determine whether or not it is included in the problematic case group.

Figure 22 is a flowchart showing the risk calculation process by the risk judgment program 172. The process shown in Figure 22 is executed in a state where the work result for which the risk is to be judged (hereinafter referred to as the "judgment target result") has been identified. The risk judgment program 172 first determines whether to use the first method or the second method in step S381. If the risk judgment program 172 uses the first method, it proceeds to step S382, and if the risk judgment program 172 uses the second method, it proceeds to step S387. In step S382, the risk judgment program 172 calls up data classified into a group that is the same as or similar to the judgment target result.

In the next step S383, the risk judgment program 172 judges whether the KPI of the judgment result deviates from the group's KPI, i.e., the KPI of the other data called up in step S382. If the risk judgment program 172 judges that the KPI of the judgment result deviates from the group's KPI, it proceeds to step S384, and if it judges that the KPI of the judgment result does not deviate from the group's KPI, it proceeds to step S386. In step S384, the risk judgment program 172 determines the magnitude of the risk as the magnitude of the deviation of the outlier. Specifically, it determines the magnitude of the risk as the product of the fixed value assigned to that KPI and the standard deviation of the KPI.

In the next step S385, the risk judgment program 172 adds a new record to the registration information table 1100 and ends the process shown in FIG. 22. The magnitude of risk calculated in step S384 is stored in the importance 1102 column of this new record. In step S386, the risk judgment program 172 determines that there is no risk in the judgment target result, and ends the process shown in FIG. 22 without adding any information to the registration information table 1100.

In step S387, the risk judgment program 172 classifies the judgment target results using the discriminator described above, and calls up the data of the group classified as having similar judgment target results. In the following step S388, the risk judgment program 172 judges whether the called group is a problematic group. If the group is a problematic group, the risk judgment program 172 proceeds to step S389, and if the group is not a problematic group, the risk judgment program 172 proceeds to step S386. In step S389, the risk judgment program 172 sets the index of the accuracy of the judgment output by the discriminator as the magnitude of risk. In the following step S390, the risk judgment program 172 adds a new record to the registration information table 1100 and ends the process shown in FIG. 22. The magnitude of risk calculated in step S389 is stored in the importance 1102 column of this new record.

When machine learning is used, the content of the description associated with the test data is used for the description 1105 among the information registered in the registration information table 1100. When a determination is made by clustering that a specific KPI has an abnormal value, the name of the KPI and the fluctuation range of the KPI are recorded as a character string. An example of this character string is "Time required: 100 seconds or more." When the machine learning system has a function for automatically generating warning messages, the post-mortem analysis unit 17 saves the generated warning message in the description 1105. When the related risk description spans multiple windows, the post-mortem analysis unit 17 sets a parent-child relationship in the display window and enters the related saved information ID in the related saved information ID 1107.

(Browse phase)
In the viewing phase, for example, the user U, who is the viewer, refers to the registered information table 1100 and displays the areas that require inspection. This operation can be used for quality assurance activities after development and inspection activities during maintenance. However, since a large amount of information can be registered in the registered information table 1100, selection is necessary. The user U may explicitly select the registered information to be displayed, or the context evaluation unit 18 may evaluate the context and the image generation unit 13 may display the registered information without the user U making an explicit selection. Instead of immediately displaying the registered information, the presence of the registered information may be notified to the user U with a marker, and the registered information may be displayed when the user U selects the marker.

Figure 23 is a diagram showing the display of registered information in the browsing phase of the second embodiment. In Figure 23, a 3D CAD model of an object existing in a specific space and registered information are displayed. This registered information can be considered as the analysis result performed in the registration phase. Registered information with high importance may be emphasized. Specifically, it may be displayed in a conspicuous color such as red, the shape of the marker may be changed, or the size of the marker may be increased. The registered information may be displayed using a display template, which will be described next.

24 is a diagram showing an example of a display template. The display template includes a saved information ID display field 981, a potential risk display field 982, an explanatory text display field 983, a related keyword display field 984, an image display area 985, and an option display field 986. The image generating unit 13 reads necessary information from the registered information table 1100 and applies it to the display template. For example, in the case of a risk estimated from clustering, the explanatory text display field 983 displays "KPI {value} is in {value} state. This state is {value} σ away from the general case." In addition, in the case of a risk estimated from machine learning, the following is displayed: "KPI {value} is in {value} state. This may be classified as a problem case {value} state. (Confidence {value})."

The window displaying each piece of registered information may further include buttons for referencing other information. That is, a button for checking related images, a button for displaying a separation graph of KPIs, and a button for displaying a comparison with similar cases. Furthermore, a button may be provided for displaying images of the same object taken at different times in the image display area 985.

The context evaluation unit 18 may increase the importance 1102 of registered information having a parent object near the current location of the user U or registered information related to the content of the user U's utterance above the threshold value listed in step S302 of the information drawing process. For example, the context evaluation unit 18 acquires the voice of the user U, searches for data converted into text by a machine learning method, and if there is content that matches a keyword registered in the keyword list 1104, increases the importance 1102 by a value corresponding to the weight associated with each keyword. In addition, the user U may associate the content of the utterance with the keywords in advance, and machine learning may be performed on the conversation data using the associated correspondence as correct answer data to create a machine learning model that appropriately interprets the utterance of the user U.

When the user moves within the virtual space and approaches one of the markers, a summary subtitle of the marker is displayed. The subtitle can be, for example, the class name of the problem case, or the type of KPI that indicates an outlier. Furthermore, if the user explicitly selects one of these using a pointing device or eye-gaze input, or if the degree of match between the conversation context and keyword list 1104 exceeds a certain value, a summary screen is displayed. For example, the above-mentioned display template can be used for this summary screen. The summary screen can be displayed in a window that fits into a relatively small area in the 3D space, and can perform processing such as reading aloud if specified by the user. The location where the summary screen is displayed uses the method shown in the first embodiment.

Furthermore, if user U explicitly selects one of the summary screens using a pointing device or eye gaze input, or if the conversation context match exceeds a certain value, a related window is opened. For this related window, a template screen like that shown in FIG. 24 is prepared as an HTML template, and a warning window is created by filling in the necessary parts with the character information and numerical information registered in 1100. Within the window, a button is placed to call up a detailed display of similar cases and related information.

Figure 25 is a diagram showing a replay explanation video seen from a viewpoint behind the worker W, and Figure 26 is a diagram showing a replay explanation video seen from a viewpoint from the side of the worker W. If the user U selects to present detailed information, a reproduced model of the work display is displayed in the virtual space, as shown in Figures 25 and 26.

The image generating unit 13 uses the data recorded in the working posture table 1700 to display a 3D model that reproduces the posture of the worker W. The image generating unit 13 may also use a 3D model that reflects joint information in this 3D displayed worker model, or may use point cloud data based on a depth map of a fixed camera, i.e., point group data. The subjective image captured by the worker W is displayed, as shown in FIG. 25, making it clear from which position the image was captured. This display shows a semi-transparent square with the shooting positions as its vertices, and presents the subjectively captured image at the bottom to express the target position and shooting direction.

However, if this rectangle overlaps with other objects and becomes difficult to see, or if the image size is too small or too large to see, a window displaying the information is placed at a low energy position using the method described in the first embodiment. As shown in Figure 25, the vertices of the rectangle, which are pseudo shooting positions, are set at positions different from the actual shooting positions, and leader lines are drawn from the pseudo shooting positions to reference the true shooting positions.

According to the above-described second embodiment, the following advantageous effects can be obtained.
(11) The display method executed by the information presentation system 1C includes a selection process executed by the context evaluation unit 18 to select registered information to be displayed based on the content of the user U's utterance and the distance from the user U to the parent object. Therefore, necessary information can be presented from a huge amount of registered information without imposing a burden on the user U.

(Modification of the second embodiment)
The importance 1102 recorded in the registration information table 1100 may be a fixed value, and a mechanism may be provided for dynamically calculating the relevance with a context indicating the current intention of the user U to use the information, and the display granularity of the information display may be adjusted according to the importance, relevance, and distance from the information target. The display granularity of the information may be determined taking into account the size of the display window that occupies the arrangement position when each display granularity is adopted, and the apparent size of the display window for the user U.

--Third embodiment--
A third embodiment of the information presentation system will be described with reference to FIG. 27. In the following description, the same components as those in the second embodiment are given the same reference numerals, and differences will be mainly described. Points that are not particularly described are the same as those in the second embodiment. This embodiment differs from the first embodiment mainly in that a third party who visits a specific space also records data. In this embodiment, this third party is called a visitor V to distinguish him from the worker W and user U described above.

The configuration of the information presentation system in this embodiment is the same as that in the second embodiment. When a visitor V who visits a specific space discovers something, such as a malfunction or a scratch, in a certain location (hereinafter referred to as a "location of interest"), the visitor V adds data to the registration information table 1100 stored in the storage device 14. For example, the visitor V starts a recording app on his or her smartphone, takes a photo of the location of interest, inputs a comment by voice or text, and registers information about the location of the photo in the registration information table 1100. Three methods for identifying and recording locations of interest are explained below.

In the first method, a 3D model and an explanatory diagram are provided to the visitor V, and the visitor V himself/herself manually inputs the points of interest into the registration information table 1100. In other words, the information presentation system does not perform a process to identify points of interest.

In the second technique, a fixed camera is present that captures images of the interior of a specific space, and a visitor V in the captured images is identified by the technique used to identify a worker W in the second embodiment. The recording app instructs the visitor V to move their smartphone while the fixed camera is capturing images, and the visitor V moves the smartphone. The recording app obtains the output of an acceleration sensor built into the smartphone, identifies the timing at which the smartphone was moved, and outputs the output to a device that processes the images from the fixed camera. The device identifies the person moving the smartphone in the image at the timing when the acceleration of the smartphone changes as visitor V, and identifies the point of interest from the position and orientation of the smartphone held by visitor V.

In the third method, a marker whose position can be uniquely identified from the image is first set in a specific space, such as a character string indicating a seat number, an AR marker, or a two-dimensional code. The position of this marker is recorded in advance in a recording app. Visitor V uses the recording app to capture a picture of the marker near the location in question, and then moves the camera from there to the problem position to capture another picture. The recording app tracks changes in the camera parameters from changes in the captured image, and estimates the location where the picture was taken.

Figure 27 is a flowchart showing an outline of the processing of the information presentation system in the third embodiment. First, the visitor V starts the recording application (S411), records and photographs the points of interest (S412), and registers the photographing location (S413). These data are then recorded in the storage device 14. Any of the above three methods may be used as a specific method for identifying and recording points of interest. The visitor V can register any number of points of interest, and when recording is completed (S415: YES), the next process is performed in the same manner as in the second embodiment. That is, attention-requiring items are detected and assigned evaluation value KPIs (S416), and the database is updated (S417), and the process shown in Figure 27 ends.

According to the third embodiment described above, a visitor V who visits a specific space can add new data to the registration information table 1100.

In each of the above-described embodiments and variations, the functional block configurations are merely examples. Several functional configurations shown as separate functional blocks may be configured together, or a configuration shown in a single functional block diagram may be divided into two or more functions. In addition, some of the functions of each functional block may be provided by other functional blocks.

In each of the above-described embodiments and variations, the program may be stored in the storage device 14. The calculation device 40 may also be provided with an input/output interface (not shown), and the program may be loaded from another device when necessary via the input/output interface and a medium available to the calculation device 40. Here, the medium refers to, for example, a storage medium that is detachable from the input/output interface, or a communication medium, i.e., a network such as a wired, wireless, or optical network, or a carrier wave or digital signal that propagates through the network. Also, some or all of the functions realized by the program may be realized by a hardware circuit or FPGA.

The above-mentioned embodiments and modifications may be combined with each other. Although various embodiments and modifications have been described above, the present invention is not limited to these. Other aspects that are conceivable within the scope of the technical concept of the present invention are also included within the scope of the present invention.

REFERENCE SIGNS LIST 1, 1A, 1B, 1C... Information presentation system 10... Server 11... Table update unit 12... Information placement unit 12a... Energy map generation unit 12b... Reference position determination unit 13... Image generation unit 20... Client U... User V... Visitor

Claims (11)

A display method for displaying registration information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of an observer, comprising:
an energy map generation process for generating one or more energy maps, which are data relating to a two-dimensional plane indicating an area suitable for arranging the registration information, at each of the viewpoint positions of the one or more observers;
a reference position determination process for setting a reference position for displaying the registration information at a second position in the energy map, the second position having a lower energy amount than a first position;
a rendering process for rendering the registration information rendered in an area including the parent object and the reference position from a subjective viewpoint of the observer. The display method according to claim 1,
In the energy map generation process, a positive amount of energy is assigned to an area of the parent object, and a negative amount of energy is assigned to an area around the parent object. The display method according to claim 1,
In the energy map generation process, a positive energy amount is assigned to an area of other registered information that is already displayed. The display method according to claim 1,
In the energy map generation process, a negative amount of energy is assigned to an area adjacent to an area in which other registered information related to the registered information is already displayed. The display method according to claim 1,
A display method, wherein in the reference position determination process, the reference positions are determined so that a sum of energy amounts at the reference positions of each of the energy maps is small. The display method according to claim 5,
The reference position determination process includes:
A temporary reference position having three-dimensional coordinates is set in advance;
identifying positions on the energy map corresponding to the tentative reference positions;
a movement vector of the tentative reference position is set so that a total energy amount of a position corresponding to the tentative reference position is reduced in each of the energy maps;
updating the tentative reference position by adding up the movement vectors calculated in each of the energy maps to the tentative reference position;
a display method for performing repeated calculations with the tentative reference position set as the reference position when a change in the total amount of energy at a position corresponding to the tentative reference position before and after updating the tentative reference position becomes equal to or smaller than a threshold value. The display method according to claim 5,
A display method in which the reference position determination process further determines the reference position by taking into account a distance based on the difference between the distance from the viewpoint position to the reference position and a distance preferred by the observer, and a vector from the viewpoint position toward the parent object. The display method according to claim 5,
a display method in which, when the registered information is already displayed based on a previous reference position which is the reference position, in the reference position determination process, the reference position is determined so that the sum of the sum of the energy amounts at the reference positions of each of the energy maps and a value based on the distance between the previous reference position and the reference position is small. The display method according to claim 1,
The energy map generating process further includes estimating a movement of the observer, and further generating the energy map based on the viewpoint position of the observer at the estimated movement destination. The display method according to claim 1,
a context generation process for generating a context;
A display method further comprising a selection process for selecting the registration information to be displayed based on the context and the distance from the observer to the parent object. A display system that displays registration information associated with a parent object in a three-dimensional space together with the parent object from a subjective viewpoint of an observer,
an energy map generating unit that generates one or more energy maps, which are data relating to a two-dimensional plane that indicates an area suitable for arranging the registration information, at each of the viewpoint positions of the one or more observers;
a reference position determination unit that sets a reference position for displaying the registration information at a second position in the energy map that has a lower energy amount than a first position;
an image generating unit that renders the registration information rendered in an area including the parent object and the reference position from a subjective viewpoint of the observer.

Images