JP2023060799A - Content display device, content display program, content display method, and content display system - Google Patents

Abstract

To provide a technique that can achieve determination work for an object using MR technology with a less expensive device.SOLUTION: According to an embodiment, a content display device includes: a parameter calculation unit 101 that calculates, as initial parameters, the initial position and initial orientation of the device using the position of a real object as the origin, based on a captured image of the real object; a parameter update unit 102 that calculates the displacement of the position and orientation of the device, and updates the position and orientation of the device using the initial position and initial orientation as a start point; a display processing unit 104 that superimposes and displays content disposed in a virtual space using the position of the real object as the origin on the real object; and a determination processing unit 105 that changes a determination state for a virtual object when the virtual object included in the displayed content is selected.SELECTED DRAWING: Figure 4

Description

TECHNICAL FIELD Embodiments of the present invention relate to techniques for displaying content.

Conventionally, MR (Mixed Reality) technology that superimposes the real world and virtual world and AR (Augmented Reality) technology that adds virtual information to the real world are known. Attempts have also been made to introduce MR and AR technologies into work sites, but they require equipment that is not general-purpose, such as head-mounted displays. .

In addition, as a related technology, a mixed reality space image acquisition process for acquiring a mixed reality space image based on an image of a first imaging unit worn by a worker and a virtual object image based on the position and orientation of the first imaging unit an event information acquisition step of acquiring event information on the virtual object performed by the worker; a second user-specified position/posture information acquisition step of acquiring viewpoint position/posture information of the instructor; and a generating step of generating a virtual object image according to the event information based on the first mode of presenting the first image to the instructor and a second mode of presenting the virtual object image to the instructor. An information processing method is known (Patent Document 1).

JP 2006-293605 A

One of the challenges is to use less expensive equipment to perform determination work on objects using MR technology.

The embodiments of the present invention have been made to solve the above-described problems, and it is an object of the present invention to provide a technology capable of realizing determination work on an object using MR technology with a less expensive device. and

In order to solve the above-described problems, an embodiment of the present invention provides a content display device for displaying content including at least one or more virtual objects, wherein the position of the real object is determined based on a captured image of the real object. a parameter calculation unit that calculates, as initial parameters, an initial position and an initial orientation of the content display device taken as an origin; A parameter updating unit that updates the position and orientation of a content display device, a display processing unit that superimposes and displays the content placed in a virtual space with the position of the real object as the origin, and the content to be displayed. and a determination processing unit that changes a determination state for the virtual object when a selection operation is performed on the virtual object included in the virtual object.

1 is a schematic diagram showing the configuration of a content display system according to a first embodiment; FIG. 2 is a block diagram showing the hardware configuration of the content display device according to the first embodiment; FIG. 2 is a block diagram showing the hardware configuration of the content management device according to the first embodiment; FIG. 1 is a block diagram showing the functional configuration of a content display device according to a first embodiment; FIG. 2 is a block diagram showing the functional configuration of the content management device according to the first embodiment; FIG. 4 is a flowchart showing content generation processing according to the first embodiment; 4 is a diagram showing content management information according to the first embodiment; FIG. It is a figure which shows the object information which concerns on 1st Embodiment. 4 is a flow chart showing the operation of the content display device according to the first embodiment; FIG. 2 is a schematic diagram showing a real object according to the first embodiment; FIG. FIG. 4 is a schematic diagram for explaining calculation of an initial position and an initial orientation according to the first embodiment; 4 is a flowchart showing content display processing according to the first embodiment; FIG. 2 is a schematic diagram showing content superimposed on a real object according to the first embodiment; FIG. FIG. 3 is a schematic diagram showing a virtual object that is a determination target according to the first embodiment; 4 is a flowchart showing determination processing according to the first embodiment; FIG. 4 is a schematic diagram showing a determined virtual object according to the first embodiment; FIG. 4 is a flowchart showing output processing according to the first embodiment; It is a schematic diagram showing a determination result display screen according to the first embodiment. It is a figure which shows the determination result information which concerns on 1st Embodiment. FIG. 2 is a schematic diagram showing a real object according to the first embodiment; FIG. FIG. 2 is a diagram showing a content display device that displays content superimposed on a real object according to the first embodiment; FIG. It is a figure which shows the content display apparatus which displays the pin object which concerns on 1st Embodiment. 9 is a flowchart showing content generation processing according to the second embodiment; FIG. 10 is a diagram showing object information according to the second embodiment; FIG. 9 is a flow chart showing the operation of the content display device according to the second embodiment; 9 is a flowchart showing variable processing according to the second embodiment; FIG. 11 is a schematic diagram showing a real object according to the second embodiment; FIG. FIG. 10 is a diagram showing a content display device that displays a virtual object superimposed on a real object according to the second embodiment; FIG. 10 is a diagram showing a content display device that displays a changed virtual object according to the second embodiment; FIG. 9 is a flowchart showing content generation processing according to the second embodiment; FIG. 13 is a diagram showing object information according to the third embodiment; FIG. FIG. 13 is a diagram showing a hierarchical virtual object group according to the third embodiment; FIG.

BEST MODE FOR CARRYING OUT THE INVENTION Hereinafter, embodiments of the present invention will be described with reference to the drawings.

<First Embodiment>
(Hardware configuration)
A hardware configuration of the content display system according to the first embodiment will be described. FIG. 1 is a schematic diagram showing the configuration of a content display system according to this embodiment. 2 and 3 are block diagrams showing the hardware configurations of the content display device and the content management device, respectively.

As shown in FIG. 1 , the content display system 1 according to this embodiment includes a content display device 10 and a content management device 20 . In this embodiment, the content display device 10 is a terminal device used by a user who browses content related to real objects, and is configured as a portable terminal device, specifically a tablet-type terminal device. The content management device 20 is a device that manages content viewed using the content display device 10, and is configured as a PC (Personal Computer) in this embodiment. Note that the content display device 10 may be configured as a smartphone or a head-mounted display, or may be configured as a device capable of executing display on a display or projection by a projector when a determination function described later is not used. . Also, the content management device 20 may be replaced by a local environment or a cloud environment.

A real object is an object on which arbitrary work is performed by the user, and is a real object to which a marker, which will be described in detail later, is attached. Such real objects include, for example, a device that requires inspection of each part, a device that determines the color scheme of each part, and the like. In this embodiment, the description will be made assuming that the real object is a device that requires inspection of each part. The content display device 10 connects to a network via a wireless LAN (Local Area Network) installed at a work site where real objects are arranged, and communicates with the content management device 20 via the network. Communication between the content display device 10 and the content management device 20 may be performed via a wired connection using a USB (Universal Serial Bus) cable or the like.

As shown in FIG. 2, the content display device 10 includes, as hardware, a CPU (Central Processing Unit) 11, a RAM (Random Access Memory) 12, a storage device 13, a touch panel 14, a network I/F (Interface) 15, and a camera. 16 and a depth sensor 17 . The CPU 11 and RAM 12 cooperate to execute various functions, and the storage device 13 stores various data used for processing executed by the various functions. The touch panel 14 is an input/output device that has a display and a touch sensor and is provided on the front side of the content display device 10 . The network I/F 15 performs wireless communication with the content management device 20 . Communication between the content management device 20 and the content management device 10 may be performed via a wired connection using a USB (Universal Serial Bus) cable or the like.

The camera 16 is provided on the back side of the content display device 10, and is an imaging device that captures two-dimensional images and videos using visible light. The depth sensor 17 is provided on the back side of the content display device 10, and is a sensor that measures the distance from the content display device 10 to surrounding objects by projecting a laser beam and receiving the reflected light with a light receiving element. be. In this embodiment, the depth sensor 17 measures the distance from the content display device 10 to surrounding objects using a ToF (Time of Flight) method that converts the delay time of the light receiving pulse to the light emitting pulse into a distance. It is assumed that the sensor is converted into a three-dimensional point group. It is assumed that the imaging direction of the camera 16 and the measurement direction of the depth sensor 17 are the same.

The content management device 20 includes, as hardware, a CPU 21, a RAM 22, a storage device 23, an input/output I/F 24, and a network I/F 25, as shown in FIG. The CPU 21 and RAM 22 cooperate to execute various functions, and the storage device 23 stores various data used for processing executed by the various functions. The input/output I/F 24 inputs and outputs data to and from an input device such as a keyboard and an output device such as a display connected to the content management device 20 . The network I/F 25 performs wired communication and wireless communication with other devices including the content display device 10 .

(Functional configuration)
Functional configurations of the content display device and the content management device will be described. 4 and 5 are block diagrams showing the functional configurations of the content display device and the content management device according to this embodiment, respectively.

As shown in FIG. 4, the content display device 10 has, as functions, a parameter calculation unit 101, a parameter update unit 102, a content reception unit 103, a display processing unit 104, a determination processing unit 105, an output processing unit 106, and a determination result transmission unit. 107.

The parameter calculation unit 101 recognizes the marker in the captured image of the real object captured by the camera 16, and based on the recognized marker, the initial position and the initial orientation of the content display device 10 at the marker coordinates with the marker as the origin. Calculate as The parameter update unit 102 calculates the displacement of the position and orientation of the content display device 10 for each frame based on the displacement between the frames before and after the three-dimensional point cloud acquired by the depth sensor 17, and based on this displacement, , update the position and orientation of the content display device 10 starting from the initial position and initial orientation calculated by the parameter calculation unit 101 .

The content receiving unit 103 receives content transmitted from the content management device 20 . The display processing unit 104 displays the content received by the content receiving unit 103 on the touch panel 14 based on the position and orientation of the content display device 10 . The determination processing unit 105 performs content determination processing based on the user's selection operation on the touch panel 14 . The output processing unit 106 outputs determination result information indicating the result of determination processing by the determination processing unit 105 . The determination result transmission unit 107 transmits determination result information output by the determination processing unit 105 to the content management apparatus 20 .

As shown in FIG. 5, the content management device 20 includes a conversion unit 201, a management unit 202, a content transmission unit 203, and a determination result reception unit 204 as functions. A conversion unit 201 converts three-dimensional data related to a real object into content. Note that CAD (Computer Aided Design) data, which is design information of a real object, can be cited as three-dimensional data converted into content. A management unit 202 manages the content converted by the conversion unit 201 . The content transmission unit 203 transmits content managed by the management unit 202 to the content display device 10 . The determination result receiving unit 204 receives determination result information transmitted by the content display device 10 .

(content generation processing)
Content generation processing will be described. FIG. 6 is a flowchart showing content generation processing. 7 and 8 are diagrams showing content management information and object information, respectively. In FIG. 6, it is assumed that 3D data related to real objects has already been input to the content management apparatus.

As shown in FIG. 6, the conversion unit 201 first selects 3D data to be converted from the 3D data input to the content management device 20 (S101).

Here, 3D data converted into content will be described. The 3D data to be converted is, for example, design data of a device to be inspected as a real object. This 3D data includes at least one or more virtual objects representing each part of the device as a 3D model. When 3D data contains multiple virtual objects, the multiple virtual objects are classified by the creator of the 3D data according to criteria such as placement positions and types of parts, and are grouped by division. do.

Prior to content generation processing, at least a portion of groups included in 3D data are set as conversion targets, which are virtual objects to be converted into content, and at least a portion of the conversion target groups are virtual objects subjected to determination processing. is set as a determination target.

After selecting the 3D data, the conversion unit 201 generates a content ID, which is an identifier that uniquely identifies the content, for the 3D data being selected. and the file path of the 3D data are associated with each other to generate content information (S102).

Next, the conversion unit 201 selects an unselected group in the 3D data being selected (S103), and determines whether or not the group being selected is set as a conversion target (S104).

If the selected group is to be converted (S104, YES), the conversion unit 201 determines whether the selected group is set as a determination target (S105).

If the selected group is to be determined (S105, YES), the conversion unit 201 converts each virtual object included in the selected group into an object including an object ID that is an identifier that uniquely identifies the virtual object. Information is generated, and the determination target flag in the object information is set to "ON" (S106).

The object information is information in which an object ID, a content ID, a creator ID, a name, a group ID, and a determination target flag are associated with each other, as shown in FIG. The content ID indicates the content to which the corresponding virtual object belongs. The creator ID is an identifier that uniquely indicates the creator of content. The name indicates the name given by the creator to the corresponding virtual object. A group ID is an identifier that uniquely indicates a group to which a corresponding virtual object belongs. The determination target flag is a flag indicating whether or not the corresponding virtual object is a determination target. be. Further, the conversion unit 201 may change the setting flag of the virtual object selected by the user to "OFF" after setting the determination target flag to "ON" for all the virtual objects included in the group.

After the operation of step S106, the conversion unit 201 determines whether or not an unselected group exists in the 3D data being selected (S107).

If there is no unselected group (S107, NO), the conversion unit 201 terminates content generation processing for the input 3D data. On the other hand, if an unselected group exists (S107, YES), the conversion unit 201 selects an unselected group in the 3D data being selected again (S103).

Further, in step S105, if the selected group is not the determination target (S105, NO), the conversion unit 201 assigns an identifier that uniquely identifies the virtual object to each of the virtual objects included in the selected group. Object information including a certain object ID is generated, and the determination target flag in the object information is set to "OFF" (S108). After the operation of step S108, the conversion unit 201 determines whether or not an unselected group exists in the 3D data being selected (S107).

Also, in step S104, if the group being selected is not the conversion target (S104, NO), the conversion unit 201 deletes the virtual objects belonging to the group being selected from the 3D data being selected (S109). After the operation of step S109, the conversion unit 201 determines whether or not an unselected group exists in the 3D data being selected (S107).

In this way, according to the content generation process, content is generated by deleting unnecessary virtual objects as content from the input 3D data and making some of the virtual objects subject to determination. The content includes 3D data from which unnecessary virtual objects have been deleted, content information that assigns a content ID to the 3D data, and object information that assigns an object ID to each virtual object included in the 3D data. 3D data, content information, and object information are all managed by the management unit 202 .

(Operation of content display device)
The operation of the content display device will be described. FIG. 9 is a flow chart showing the operation of the content display device. FIG. 10 is a schematic diagram showing a real object. FIG. 11 is a schematic diagram for explaining calculation of the initial position and initial orientation. It is assumed that, prior to the operation shown in FIG. 9, the content related to the real object has been transmitted to the content display device by the content management device.

As shown in FIG. 9, first, the parameter calculator 101 determines whether or not the marker M is recognized in the image captured by the camera 16 (S201).

As shown in FIG. 10, the marker M is a two-dimensional image attached to the real object RO, and defines the origin of the virtual space where the content is arranged in the real space. Examples of such a marker M include a two-dimensional image attached to the real object RO, a two-dimensional image-like unevenness stamped on the real object, and the like. Note that the marker M does not necessarily have to be attached to the real object RO, and may be attached to the floor surface, wall surface, ceiling surface, etc. around the real object RO.

Alternatively, the origin of the virtual space may be determined in the real space by a three-point survey method or a fixed position method without using the marker M. The trilateration method is a method in which at least three shapes of a real object are selected, and a plane connecting the selected locations is used instead of the marker M. FIG. In the fixed position method, the arrangement position of the real object and the use position of the content display system 1, specifically, the initial position of the content display device 10 are fixed in advance, and the content display system 1 is always used from the same position. This is a method of setting the origin of the virtual space in the real space. In this way, the origin of the virtual space may be set in the real space by any method.

When the marker M is not recognized (S201, NO), the parameter calculation unit 101 again determines whether the marker M is recognized in the image captured by the camera 16 (S201). On the other hand, when the marker M is recognized (S201, YES), the parameter calculation unit 101 calculates the initial position and initial orientation of the content display device 10 with the marker M as the origin as parameters (S202).

Here, the calculation of the initial position and initial orientation will be described. As shown in FIG. 11, when the imaging range S including the marker M added to the real object RO is captured by the camera 16, the positional relationship between the marker M and the content display device 10 causes the marker m on the captured image to be Scaling and deformation occur. The parameter calculation unit 101 calculates the relative position and orientation of the content display device 10 with respect to the marker M from the size and shape of the marker m based on the marker information indicating the shape and size of the marker M serving as a reference. Furthermore, the parameter calculation unit 101 calculates the position of the content display device 10 on three axes (x, y, z in the drawing) with the marker M as the origin, and the rotation axis ( By calculating the rotation of the content display device 10 around x1, y1, z1), the initial position and initial attitude of the content display device 10 are calculated. In this embodiment, the center point of the marker M is the origin O, and the center of the camera 16, specifically the center point of the image sensor, is the origin I of the rotation axis.

After calculating the parameters, the parameter update unit 102 updates the current position and current orientation of the content display device 10 based on the displacement of the three-dimensional feature point group detected by the depth sensor 17 (S203).

Here, updating of the current position and current attitude will be described. The parameter updating unit 102 calculates the displacement of the position and orientation of the content display device 10 based on the displacement of the three-dimensional feature point group that changes in chronological order, and converts this displacement to the initial position and initial orientation or the previously updated position and orientation. The current position and current attitude of the content display device 10 are successively updated by accumulating the attitude. It should be noted that a known method such as SLAM (Simultaneous Localization and Mapping) or SfM (Structure from Motion) is used to calculate the displacement of self-position and self-orientation based on the displacement of the three-dimensional feature point group. Alternatively, the VSLAM (Visual SLAM) method may be used to calculate the displacement of the position and posture of the content display device 10 based on the captured image captured by the camera 16 .

After the current position and orientation are updated, content display processing is executed (S204), determination processing is executed (S205), and output processing is executed (S206). Content display processing, determination processing, and output processing will be described later.

After executing the output process, the display processing unit 104 determines whether or not the user has given an end instruction to end the display of the content via the touch panel 14 (S207). If the end instruction is given (S207, YES), the operation of the content display device 10 is ended. On the other hand, if the end instruction is not issued (S207, NO), the parameter updating unit 102 updates the current position and current orientation of the content display device 10 based on the displacement of the three-dimensional feature point group detected by the depth sensor 17 again. Update (S203).

Thus, according to the content display device 10, the origin O of the virtual space can be determined based on a portion of the real object RO. Note that the calculation of the initial position and the initial orientation by the parameter calculation unit 101 may be performed in response to an instruction from the user via the touch panel 14, or may be performed at each predetermined cycle. It may be done every M imaging. According to the content display device 10 configured in this way, the deviation of the content from the real object RO is reduced.

(Content display processing)
Content display processing will be described. FIG. 12 is a flowchart showing content display processing. FIG. 13 is a schematic diagram showing content superimposed on a real object. FIG. 14 is a schematic diagram showing a virtual object to be determined. It should be noted that in the content display process shown in FIG. 12, it is assumed that the content related to the real object has been selected in advance by the user.

As shown in FIG. 12, the display processing unit 104 determines whether or not the user has given a display target instruction to select a group to be displayed (S301).

When a display target instruction is given (S301, YES), the display processing unit 104 changes the display target so that the group selected by the user is displayed on the touch panel 14 (S302).

After the display target is changed, or when the display target is not instructed (S301, NO), the display processing unit 104 determines whether the user has issued a display color instruction to change the display color of the selected group. Determine (S303).

If a display color instruction is given (S303, YES), the display processing unit 104 changes the display color of the virtual object belonging to the selected group to the specified color (S304).

After changing the display color or when no display color instruction is given (S303, NO), the display processing unit 104 determines whether or not the user has given a transparency instruction to change the transparency of the selected group. (S305).

If the transparency is specified (S305, YES), the display processing unit 104 changes the virtual objects belonging to the selected group to the specified transparency (S306).

After changing the transparency, or when the transparency is not instructed (S305, NO), the display processing unit 104 selects the virtual object of the group designated to be displayed among the contents arranged with respect to the origin O of the marker M. is displayed on the touch panel 14 in the specified display color and transparency (S307), and the display process is terminated.

In this embodiment, as shown in FIG. 13, the relative position of the content with respect to the origin O in the virtual space is set in advance so that the content is displayed on the touch panel 14 with substantially the same posture and size as the real object. shall be Further, when the group designated as the display target includes the virtual object as the determination target, as shown in FIG. Displayed in a different color than other virtual objects.

The display of the content is either MR (Mixed Reality) display in which the content is superimposed on the image captured by the camera 16 or AR (Augmented Reality) display in which only the content is displayed, depending on the setting made by the user. It shall be possible to switch between

(Determination process)
Determination processing will be described. FIG. 15 is a flowchart showing determination processing. FIG. 16 is a schematic diagram showing a determined virtual object.

As shown in FIG. 15, the determination processing unit 105 first determines whether the group displayed on the touch panel 14, more specifically, the group specified as the display target in the content display process described above includes the determination object. (S401).

If the displayed group includes a determination object (S401, YES), the determination processing unit 105 determines whether or not the user has selected the determination object (S402). Here, the determination processing unit 105 determines that the determination object is selected when a tap operation is performed on the determination object on the touch panel 14 .

If a determination object is selected (S402, YES), the determination processing unit 105 changes the determination state of the selected determination object (S403), and the determination process ends.

Here, the determination state will be explained. As shown in FIG. 16, the determination object JO is changed to a plurality of different selection states (JO1/JO2) according to the user's selection operation. In this embodiment, the judgment object is judged to be either "normal" or "abnormal". FIG. 16 shows an undetermined determination object JO0, a determination object JO1 determined to be "normal", and a determination object JO2 determined to be "abnormal". The determination objects JO0, JO1, and JO2 are displayed on the touch panel 14 by the display processing unit 104 in different display colors. This allows the user to easily recognize the determination state of the determination object. Note that determination states other than "normal" and "abnormal" include, for example, "unknown" and "temporary suspension". Further, when the determined state is "abnormal", the abnormal state can be separately recorded as character string information or still image information, as will be described later. By doing so, as will be described in detail later as an example of use, the user can register the judgment result as a record and complete the report work to the superior.

The determination state of the determination object JO is changed by the number of selection operations on the determination object JO, which is the number of tap operations on the determination object JO in this embodiment. Specifically, when the number of tap operations is 0, it is in an undetermined state, when the number of tap operations is 1, it is in a "normal" determination state, and when the number of tap operations is 2, it is in an "abnormal" state. "It will be in a judgment state. If a further tap operation is performed on the determination object in the “abnormal” determination state, the determination object can be put in the undetermined state. If the determination state includes "normal", "abnormal", "unknown", and "temporarily suspended", it is undetermined when the number of tap operations is 0, and when the number of tap operations is 1 If the number of tap operations is 2, it becomes an "abnormal" judgment state, if the number of tap operations is 3, it becomes an "unknown" judgment state, and if the number of tap operations is 4, it becomes "temporary suspension" When the judgment state is entered and a tap operation is further performed, an undetermined state can be entered. Further, when the determined state is "abnormal", the abnormal state can be separately recorded as character string information or still image information, as will be described later. By doing so, as will be described in detail later as an example of use, the user can register the judgment result as a record and complete the report work to the superior. Further, the determination state of the determination object JO may be changed by operating a determination state change screen including options such as "normal", "abnormal", "unknown", and "temporary hold". This determination state change screen is superimposed and displayed on the main screen by tapping once on the determination object JO whose determination state is to be changed. When the determination state is changed to "abnormal", the user can separately record the abnormal state as character string information or still image information on the determination state change screen, as will be described later. If the save button or cancel button is pressed after "normal" is selected, or after "abnormal" is selected and character string information is added, judgment is made after the changed judgment state is saved. The state change screen shall be closed.

Note that at least one determination state of the determination object JO should exist, and labels for a plurality of determination states can be set by the user. This allows the content display device 10 to be used for various purposes.

If the judgment object is not included in the objects displayed in step S401 (S401, NO), or if the judgment object is not selected (S402, NO), the judgment process ends.

(output processing)
Output processing will be described. FIG. 17 is a flowchart showing output processing. FIG. 18 is a schematic diagram showing a judgment result display screen. FIG. 19 is a diagram showing determination result information.

As shown in FIG. 17, the display processing unit 104 determines whether or not the user has issued a determination result display instruction (S501).

When the determination result display instruction is given (S501, YES), the display processing unit 104 causes the touch panel 14 to display the determination result for the determination object (S502). The determination result is displayed on the determination result display screen.

The determination result display screen includes, as shown in FIG. 18, the number of determination targets, the number of normal determinations, the number of abnormal determinations, the undetermined number, and an abnormality determination list. The determination target count indicates the total number of determination objects included in the content. The number of normal judgments indicates the total number of judgment objects judged to be “normal”. The number of determined abnormalities indicates the total number of determined objects determined to be "abnormal". The undetermined number indicates the total number of determination objects that have not been determined among all determination objects. According to the judgment result display screen, the user can confirm whether there is any check omission, and the quality and productivity of the judgment work can be improved.

The abnormality determination list shows, in a list, details of each determination object for which an "abnormal" determination has been made. A record in the abnormality determination list includes fields of an object ID, a comment, and an attached file. The comment is text input by the user on the determination object indicated by the corresponding object ID on the determination result display screen. The attached file indicates the file name attached by the user to the determination object indicated by the corresponding object ID on the determination result display screen. As files to be attached, media files such as image files, moving image files, and audio files related to the corresponding determination object are assumed, but any files may be attached.

After displaying the determination result, the output processing unit 106 determines whether or not the user has instructed to output the determination result (S503).

When the determination result output instruction is given (S503, YES), the output processing unit 106 outputs determination result information (S504), and the determination result transmission unit 107 transmits the output determination result information to the content management apparatus 20. (S505), and the output process ends. The determination result information is managed by the management unit 202 in the content management device 20 .

The judgment result information is a file containing an abnormality judgment list, and as shown in FIG. User IDs indicating ten users, determination results for determination objects, determination dates and times indicating dates and times when determinations were made, comments, and attached files are associated with each other. The determination result information is output as a file indicating a data structure such as an XML (eXtensible Markup Language) file, a JSON (Javascript (registered trademark) Object Notation) file, a CSV (Comma-Separated Values) file, or the like. The determination result information is transmitted together with the attached file indicated in this determination result information.

If no judgment result display instruction is given (S501, NO) or if no judgment result output instruction is given (S503, NO), the output process is terminated.

Displaying a list of the determination results allows the user to easily grasp the determination results for the real object. Also, by outputting the determination result as a file, the determination result for the real object can be easily shared in the organization.

In the above-described embodiment, the content display device 10 superimposes and displays the content CN on the real object RO. On the other hand, as shown in FIG. 21, the content CN may be arranged and displayed in the missing portion of the real object RO. That is, the content display device 10 may arrange the content CN at a position not overlapping the real object RO in the virtual space.

Further, as shown in FIG. 22, the content display device 10 may display a pin object P formed in a pin shape pointing to a predetermined position so as to be superimposed on the real object RO. This pin object P is a virtual object pointing to a predetermined location in the virtual space. A user can register a new pin object P by selecting a predetermined location in the virtual space via the touch panel 14 . In addition, the user can associate additional information, which can include text, images, etc., with the new pin object P, and can modify the additional information associated with the existing pin object P. The display processing unit 104 causes the touch panel 14 to display the pin object P, and when the pin object P is selected, causes the touch panel 14 to display additional information associated with the selected pin object P.

(First usage example)
A first usage example will be described. A first usage example is a usage example when the content display system is used for confirming the positions of welding points.

Conventionally, confirmation of the position of a welding point is performed by sticking a sheet of paper of actual size with a hole punched at the position of the welding point or a gabari on the object to be welded. The worker who checks the position performs quality inspections such as whether the welding point position is correct, whether the welding point size is correct, and whether the welding point depth (whether the required number of metal plates are welded) is correct. It is up to the operator to check whether all welding points have been confirmed. After confirmation, the worker reports the confirmation result to the superior by separately inputting the confirmation result on a PC or the like.

When the content display system 1 is used to confirm the position of the welding point, the worker can confirm the position of the welding point by the content created according to the object to be welded, so advance preparation such as gabari is not required. In addition, since the determination process is performed for each determination object indicating the welding point, it is possible to confirm without omission. In addition, since the judgment results are accumulated as data, it is easy to report to superiors, analyze the locations and patterns of welding defects, and use the judgment results for improvement activities related to welding work. Become.

(Second usage example)
A second usage example will be described. A second usage example is a usage example when the content display system is used for confirmation work after painting.

Conventionally, a confirmation operation is performed after paint has been applied to an object to be painted. In the confirmation work, it is confirmed whether or not the type of coating material, coating position, and coating range are correct. In confirming the coating range, it is confirmed whether the coating is applied to the required range and whether the coating does not protrude outside the coating range. After the confirmation work, the worker reports the confirmation result to the superior by separately inputting the confirmation result on a PC or the like.

When the content display system 1 is used for confirmation work after painting, advance preparation such as a checklist for confirmation is not required, and the type, application position, and application size of the coating material are confirmed by the content created in accordance with the coating work. can do. Since the determination process is performed for each determination object indicating the application position, it is possible to confirm without omission. In addition, since the judgment results are accumulated as data, it is easy to report to superiors, analyze the locations and patterns of frequent coating defects, and use the judgment results for improvement activities related to coating work. becomes.

(Third usage example)
A third usage example will be described. A third example of use is a case where the content display system is used to verify the assembly of a mold and a press machine.

Conventionally, when verifying the assembly of a die and a press machine, a full-size sheet or guard is prepared in advance for each die and press machine on which the die is installed, and the sheet is placed on the upside-down inverted press die. and gabari are attached to the mold. The operator checks whether the cushion pin or the protective member that protects the cushion pin does not interfere with the base portion of the press die (in the shape of a grid). Since there is a difference in the shape of each press, if interference occurs between cushion pins and protective members, the operator removes the interference by scraping the base portion of the mold as necessary. In general, verification work and interference prevention work are performed by different workers. there is For this reason, who performed what kind of work at what time is recorded on paper or left unrecorded.

When the content display system 1 is used for verifying the assembly of a mold and a press machine, it is possible to confirm the positions of cushion pins and the like as virtual objects without the need for advance preparation such as gabari. By performing determination processing for each cushion pin as a determination object, the operator can check without omission. Alternatively, since the judgment results can be accumulated as data, it is possible to analyze the data after the fact, analyze the locations and patterns of frequent interference, and utilize them for improvement activities.

In addition, even if different workers are involved in the verification work and the interference prevention work, outputting the judgment results as data makes it possible to communicate what kind of work is required for which part. You can keep a record of when, what part, and what kind of work was done. Since judgment results and transmitted records can be accumulated as data, data analysis can be performed after the fact, and the locations and patterns of many defects can be analyzed and used for improvement activities.

(Fourth usage example)
A fourth usage example will be described. A fourth example of use is a case where the content display system is used for mold maintenance work.

Conventionally, in regular mold maintenance work, a checker removes the mold from the machine and visually checks the place where the mold is to be repaired by welding, cutting, spring replacement, cushion replacement, or the like. Furthermore, the repair workers have been performing repair work on the mold based on communication from the confirmation workers using verbal, text, photographs, or the like. However, since there is a time difference between the time of confirmation and the time of repair, the content of repair becomes ambiguous, and confirmation of the repaired part and the content of repair is wasteful.

When the content display system 1 is used for mold maintenance work, the confirmation operator clearly uses various media files to determine what should be done and what should be done for which location of the mold using the determination object. can be shown. Furthermore, repair workers can clearly understand where, what, and how, and can perform repair work without time lag.

In addition, since it is possible to record who repaired which part when and, it is possible to accumulate the confirmation results, transmitted information, and repair work records as data. It is possible to analyze places and patterns, etc. and use them for improvement activities. Furthermore, by feeding back the results of such analysis to the design and development departments, it is possible to reduce the number of defects and make designs that are easy to repair. In addition, the analysis results can be used to adjust the schedule of regular maintenance as preventive maintenance, and the occurrence of irregular maintenance as post-maintenance can be reduced.

(Fifth usage example)
A fifth usage example will be described. A fifth usage example is a usage example in which the content display system is used to designate an application range in painting.

In the past, in order to indicate which application material should be applied to which position, a special tool including a cover made of plastic or metal was arranged, and masking tape was attached around it. Since the design is changed many times in the prototype stage, it is necessary to recreate the special equipment according to the design change. Since it takes time to manufacture the special equipment, there was a time lag between the design change and the masking work.

By using the content display system 1 to specify the application range and superimposing the content based on the design data of the application target on the application target, the application position and the application range are clarified, and the masking tape is pasted without a special tool. be able to. Furthermore, time and cost are reduced because special tools are manufactured.

(Sixth usage example)
A sixth usage example will be described. A sixth usage example is a usage example when the content display system is used for layout planning of machinery and equipment.

In the past, when new machinery was installed or when machinery was replaced, it was determined whether the machinery could be installed in the factory building based on design drawings. In this case, consideration of the interference with the building and other machinery that may occur when the machinery operates, the flow line of workers engaged in the building, the flow line of the material transport vehicle, etc. There was a need to review the placement of

When the content display system 1 is used for layout planning of machinery and equipment, it is possible to arrange newly installed virtual machinery and equipment with respect to existing factory buildings and machinery. It is possible to sufficiently consider and confirm interference with equipment, the flow line of workers, the flow line of material transport vehicles, and the like.

As described above, according to the content display system 1, since a general-purpose tablet terminal or PC can be used as the content display device 10 and the content management device 20, determination of an object using MR technology is possible. Work can be done more cheaply.

<Second embodiment>
A content display system according to the second embodiment will be described. 23, 25, and 26 are flow charts showing content generation processing, operation of the content display device, and variable processing, respectively, according to this embodiment. FIG. 24 is a diagram showing object information according to this embodiment. FIG. 27 is a schematic diagram showing a real object according to this embodiment. FIG. 28 is a diagram showing a content display device that displays a virtual object superimposed on a real object according to this embodiment.

In the content display device 10 according to the present embodiment, the scale of the virtual object displayed superimposed on the real object can be changed for each of the three axes (x, y, z, see FIG. 11) of the virtual space. It differs from the first embodiment. In this embodiment, the content includes a variable object, which is a virtual object whose scale can be expanded or reduced.

In order to generate such a variable object, the content generation process according to the present embodiment differs from the first embodiment in that step S111 is executed after steps S106, S108, and S109, as shown in FIG. is different. In step S111, the conversion unit 201 of the content management device 20 determines whether or not the selected group is set as a variable target.

If the group being selected is a variable target (S111, YES), the conversion unit 201 sets the variable target flag in the object information of each virtual object included in the group being selected to "ON" (S112, YES), The process of step S107 is executed. On the other hand, if the group being selected is not a variable target (S112, NO), the conversion unit 201 sets the variable target flag in the object information of each virtual object included in the group being selected to "OFF". Execute the process.

As shown in FIG. 24, the object information according to the present embodiment generated in this manner is similar to the first embodiment in that the object ID is further associated with the variable target flag, the x scale, the y scale, and the z scale. different from the form. The variable target flag indicates whether or not the corresponding virtual object is variable target. The x-scale, y-scale, and z-scale all have an initial value of 0, and input values ranging from a negative value as the minimum value to a positive value as the maximum value. Objects are shrunk and corresponding virtual objects are magnified if a positive value is entered. The x-scale indicates the scale of the corresponding virtual object in the x-axis direction. The y scale indicates the scale of the corresponding virtual object in the y-axis direction. The z-scale indicates the scale of the corresponding virtual object in the z-axis direction. For example, if a negative value is entered for z-scale, the corresponding virtual object will only shrink in the z-axis direction.

As shown in FIG. 25, the content display apparatus 10 according to the present embodiment differs from the first embodiment in that variable processing is executed as processing in step S208 before output processing is performed. In the variable processing, as shown in FIG. 26, first, the display processing unit 104 determines whether or not a variable object, which is a virtual object whose variable target flag is set to "ON", is included in the content being displayed. (S601).

If a variable object is included (S601, YES), the display processing unit 104 determines whether or not a variable object has been selected by the user (S602). It is determined whether or not the user has issued a variable instruction to vary the scale in the selected axial direction (S603). If a variable instruction has been issued (S603, YES), the display processing unit 104 changes the scale of the variable object based on the variable instruction (S604).

If no variable object is included (S601, NO), if no variable object is selected (S602, NO), or if no variable instruction is given (S603, NO), the variable processing ends.

As a specific example, a variable object whose scale is reduced will be described with reference to FIGS. 27 to 29. FIG. In this specific example, as shown in FIG. 27, the real object RO is a substantially rectangular parallelepiped object having a hole H formed in its upper surface, and the hole H is a non-through hole having a bottom surface B formed therein. Also, as shown in FIG. 28, it is assumed that a columnar variable object VO extending in the z-axis direction is displayed superimposed on the real object RO. Also, the variable object VO is supposed to imitate a part to be inserted into the hole H. As shown in FIG.

The variable instruction described above is made using the variable operation unit VC, which is a GUI displayed on the touch panel. The variable operation unit VC includes three check boxes for selecting the axis for varying the scale of the variable object VO, and a slider UI that can operate the knob in the horizontal direction. When the knob is moved to the left, the variable object VO is reduced, and when the knob is moved to the right, the variable object VO is enlarged.

In FIG. 28, the z-axis (vertical direction in FIG. 28) is selected as the axis for varying the scale of the variable object VO, and the knob is located in the center, so the scale of the variable object VO is not changed. The variable object VO is positioned to fit into the hole H, but the user can determine whether the bottom surface B of the hole H is covered with the variable object VO and the variable object VO interferes with the bottom surface B of the hole H. I can't judge.

As shown in FIG. 29, when the knob of the slider UI in the variable operation unit VC is moved leftward, the variable object VO is reduced in the z-axis direction. As a result, both the variable object VO and the bottom surface B of the hole H are displayed on the touch panel 14, and the user can determine whether the variable object VO interferes with the bottom surface B of the hole H or not.

According to the content display device 10 according to the present embodiment, as in the above-described third usage example, when used for verifying the assembly of a mold and a press machine, a cushion is applied to a mold having a plurality of concave portions. By reducing the scale of the virtual object that imitates the pin in the recessed direction of the recess, it is possible to easily check the degree of interference between the bottom of the recess of the mold and the cushion pin.

<Third Embodiment>
A content display system according to the third embodiment will be described. FIG. 30 is a flowchart showing content generation processing according to this embodiment. FIG. 31 is a diagram showing object information according to this embodiment. FIG. 32 is a diagram showing a hierarchical virtual object group according to this embodiment.

The content display device 10 according to the present embodiment differs from the first embodiment in that it can hierarchize and handle a plurality of virtual objects included in content. In this embodiment, the content includes a layered object that is a virtual object associated with the group to which the group to which the content belongs belongs.

In order to generate such a hierarchical object, as shown in FIG. 30, the content generation process according to the present embodiment is different from the first embodiment in that step S121 is executed after steps S106, S108, and S109. different from In step S121, the conversion unit 201 of the content management device 20 determines whether or not the selected group is set as a structuring target.

If the selected group is to be structured (S121, YES), the conversion unit 201 converts the parent ID (described later) in the object information of each virtual object included in the selected group to these virtual objects. A group ID indicating the group to which the group belongs is set (S122, YES), and the process of step S107 is executed. On the other hand, if the selected group is not a structuring target (S122, NO), the conversion unit 201 executes the processing of empty step S107 without changing the parent ID in the object information. The initial value of the parent ID is a null value or a group ID indicating a root group containing all virtual objects in the content. Also, the parent ID is set to a group specified by the user prior to execution of the content generation process, or is set based on structural information included in the input 3D data.

As shown in FIG. 31, the object information according to the present embodiment generated in this way differs from the first embodiment in that the object ID is further associated with the parent ID described above. A group ID indicating a group to which the group indicated by the corresponding group ID belongs is set in the parent ID.

According to object information including such parent IDs, as shown in FIG. 32, virtual objects can be hierarchized and handled. FIG. 32 shows six nuts A to F as virtual objects. G1 is set in the parent ID in the object information of each of the six nuts A to F, G2 is set in the group ID in the object information of each of the nuts A to C, and G2 is set in the object information of each of the nuts D to F. G3 is set as the group ID. With this, all nuts including three nuts belonging to "area 1" and three nuts belonging to "area 2" different from "area 1" can be expressed. Also, a group belonging to another group may be set as the parent ID, and the virtual object may be hierarchized in three or more hierarchies.

Structuring the virtual objects in this way allows for more flexibility in making changes to multiple virtual objects. For example, when changing the display color of nuts as shown in FIG. 32, first, the display color of all nuts is changed to red, and among the nuts belonging to "area 1", the display color of only nut B is changed to blue. , and change the display color of all nuts belonging to "Area 2" to yellow. In this way, structuring virtual objects makes it easier to change some virtual objects belonging to the same group, compared to the case where all virtual objects are individually changed.

When the content display device 10 according to the present embodiment is used for confirmation work after painting as in the above-described second usage example, a large number of application positions can be determined not only by the application location and the application material, but also by the area in charge and the application material. It is possible to classify each person in charge, and it is possible to reduce mistakes in painting work. In addition, the operator can hide the checked application positions on the content display device 10 and display only the unchecked ones, thereby improving the checking efficiency.

While embodiments of the invention have been described, these embodiments are provided by way of example and are not intended to limit the scope of the invention. These novel embodiments can be embodied in various other forms, and various omissions, replacements, and modifications can be made without departing from the scope of the invention. These embodiments and their modifications are included in the scope and gist of the invention, and are included in the scope of the invention described in the claims and its equivalents.

10 content display device 101 parameter calculation unit 102 parameter update unit 104 display processing unit 105 determination processing unit 106 output processing unit

Claims (19)

A content display device for displaying content including at least one or more virtual objects,
a parameter calculation unit that calculates, as initial parameters, an initial position and an initial orientation of the content display device with the position of the real object as the origin, based on the captured image of the real object;
a parameter updating unit that calculates displacement of the position and orientation of the content display device and updates the position and orientation of the content display device with the initial position and the initial orientation as origins; and a virtual object with the position of the real object as the origin. a display processing unit that superimposes and displays the content arranged in space on the real object;
A content display device, comprising: a determination processing unit that changes a determination state for a virtual object included in displayed content when a selection operation is performed on the virtual object. 2. The content display device according to claim 1, wherein the determination processing unit changes the determination state for the virtual object according to the number of selection operations for the virtual object. 3. The content display device according to claim 1, wherein the determination state for the virtual object includes at least normal state and abnormal state. 4. The content display device according to claim 3, further comprising an output processing unit that outputs determination result information regarding the virtual object determined to be abnormal. 5. The content display device according to claim 4, wherein the determination result information associates an object ID, which is an identifier indicating a virtual object determined to be abnormal, with a user ID indicating a user who changed the determination state. 6. The content display device according to claim 5, wherein the determination result information further associates the object ID with a date and time when the determination state was changed. 7. The content display device according to claim 5, wherein said determination result information further associates said object ID with a text for a virtual object indicated by said object ID. 8. The content display device according to claim 5, wherein said determination result information further associates said object ID with a media file. 9. The parameter calculator according to any one of claims 1 to 8, wherein the parameter calculator calculates an initial position and an initial orientation of the content display device based on the captured image of the real object, based on an instruction operation by a user. 1. The content display device according to item 1. 9. The parameter calculator according to any one of claims 1 to 8, wherein the parameter calculator calculates an initial position and an initial orientation of the content display device based on the captured image of the real object at each predetermined cycle. The content display device according to . 1. The parameter calculation unit calculates an initial position and an initial orientation of the content display device based on a captured image of the real object each time a characteristic portion of the real object is captured. Item 9. The content display device according to any one of Item 8. 12. The content display according to any one of claims 1 to 11, wherein the display processing unit displays a pin object, which is a pin-shaped virtual object pointing to a location designated by the user in the virtual space. Device. the pin object is associated with additional information including at least one of text and media files;
13. The content display device according to claim 12, wherein when the pin object is selected, the display processing unit displays additional information associated with the pin object. 1. The display processing unit enlarges or reduces the scale of the selected virtual object in the direction of an axis selected from among three axes in the virtual space, based on a user's instruction. Item 13. The content display device according to any one of Items 12 to 13. The virtual object is associated with a group ID indicating a group to which the virtual object belongs, and a parent ID indicating another group to which the group indicated by the group ID belongs. The content display device according to any one of 1. A content display program for displaying content including at least one or more virtual objects,
the computer,
a parameter calculation unit that calculates, as initial parameters, an initial position and an initial orientation of the device with the position of the real object as the origin, based on the captured image of the real object;
a parameter updating unit that calculates the displacement of the position and orientation of the device, and updates the position and orientation of the device with the initial position and the initial orientation as starting points; and a virtual space with the position of the real object as the origin. a display processing unit that superimposes and displays the arranged content on the real object;
A content display program that functions as a determination processing unit that changes a determination state for a virtual object when a virtual object included in displayed content is selected. A content display method for displaying content including at least one or more virtual objects,
the computer
calculating, as initial parameters, an initial position and an initial orientation of the device with the position of the real object as the origin, based on the captured image of the real object;
calculating the displacement of the position and orientation of the device, updating the position and orientation of the device with the initial position and the initial orientation as starting points;
displaying the content arranged in a virtual space with the position of the real object as the origin, superimposed on the real object;
A content display method for changing a determination state for a virtual object included in displayed content when a selection operation is performed on the virtual object. A content display system comprising: a content display device displaying content including at least one or more virtual objects; and a content management device communicably connected to the content display device,
The content management device
a conversion unit that converts input three-dimensional data into content information including the three-dimensional data and a content ID that is an identifier that uniquely identifies the three-dimensional data;
a content transmission unit that transmits the converted content information to the content display device;
The content display device is
a parameter calculation unit that calculates, as initial parameters, an initial position and an initial orientation of the content display device with the position of the real object as the origin, based on the captured image of the real object;
a parameter updating unit that calculates the displacement of the position and orientation of the content display device and updates the position and orientation of the content display device with the initial position and the initial orientation as starting points;
a content receiving unit that receives content information transmitted by the content management device;
a display processing unit that superimposes and displays the content arranged in a virtual space with the position of the real object as an origin based on the received content information on the real object;
A content display system, comprising: a determination processing unit that changes a determination state for a virtual object included in displayed content when a selection operation is performed on the virtual object. A content display method executed by a content display device displaying content including at least one or more virtual objects and a content management device communicably connected to the content display device, comprising:
The content management device
converting the input three-dimensional data into content information including the three-dimensional data and a content ID that is an identifier uniquely indicating the three-dimensional data;
transmitting the converted content information to the content display device;
The content display device is
calculating, as initial parameters, an initial position and an initial orientation of the content display device with the position of the real object as the origin, based on the captured image of the real object;
calculating a displacement of the position and orientation of the content display device, updating the position and orientation of the content display device with the initial position and the initial orientation as starting points;
receiving content information transmitted by the content management device;
displaying the content arranged in a virtual space with the position of the real object as an origin based on the received content information, superimposed on the real object;
A content display method for changing a determination state for a virtual object included in displayed content when a selection operation is performed on the virtual object.

Images